| Message ID | 20240210230442.3633663-1-richard.purdie@linuxfoundation.org |
|---|---|
| State | Accepted, archived |
| Commit | 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9 |
| Headers | show |
| Series | [1/2] runqueue: Optimise taskname lookups in next_buildable_task | expand |
On Sat, 2024-02-10 at 23:04 +0000, Richard Purdie via lists.openembedded.org wrote: > "time bitbake world -n" shows a lot of time being spent trying to execute the > setscene tasks when many of them are being skipped due to not being present. > > The challenge is the "return True" back into the mainloop restarts the search > back at the start of the task list particularly when there are large numbers of > deferred tasks due to hard dependencies. > > In this case we can use continue to continue within the list at the expense of > not returning so quickly to loop back through the main loop. This does significantly > improve performance in real world usage scenarios and we can probably afford the > potential loop starvation. > > Thanks for the suggestion from Alexander Kanavin. > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > lib/bb/runqueue.py | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py > index 95cab5132f..85669b80ee 100644 > --- a/lib/bb/runqueue.py > +++ b/lib/bb/runqueue.py > @@ -2204,7 +2204,9 @@ class RunQueueExecute: > if nexttask in self.sqdata.outrightfail: > logger.debug2('No package found, so skipping setscene task %s', nexttask) > self.sq_task_failoutright(nexttask) > - return True > + # Technically we should return True here but that would then start the task list back at the start > + # which is really bad for performance ("time bitbake world -n"). Instead, keep our place in the tasklist. > + continue > if nexttask in self.sqdata.unskippable: > logger.debug2("Setscene task %s is unskippable" % nexttask) > task = nexttask Sadly this doesn't work: https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6354 Cheers, Richard
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index c8392346a8..95cab5132f 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -270,11 +270,11 @@ class RunQueueScheduler(object): best = None bestprio = None for tid in buildable: - taskname = taskname_from_tid(tid) - if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]): - continue prio = self.rev_prio_map[tid] if bestprio is None or bestprio > prio: + taskname = taskname_from_tid(tid) + if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]): + continue stamp = self.stamps[tid] if stamp in self.rq.build_stamps.values(): continue
A quick profile of bitbake world showed 147 million calls to taskname_from_tid(). The next_buildable_task function is performance senstive so move the call inside the if block to reduce the number of calls and speed the code up. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- lib/bb/runqueue.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)