[bitbake-devel,01/26] runqueue: Tweak buildable variable handling in scheduler

Submitted by Richard Purdie on July 10, 2019, 11:53 p.m. | Patch ID: 162950

Details

Message ID 20190710235420.23825-1-richard.purdie@linuxfoundation.org
State Accepted
Commit e851169acfebba404514135bf512e6f045739a13
Headers show

Commit Message

Richard Purdie July 10, 2019, 11:53 p.m.
Work off a copy of the 'buildable' class variable, allowing easier
future code changes.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/runqueue.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

Patch hide | download patch | download mbox

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 704e309b93..b19b524e77 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -142,7 +142,8 @@  class RunQueueScheduler(object):
         Return the id of the first task we find that is buildable
         """
         self.buildable = [x for x in self.buildable if x not in self.rq.runq_running]
-        if not self.buildable:
+        buildable = self.buildable
+        if not buildable:
             return None
 
         # Filter out tasks that have a max number of threads that have been exceeded
@@ -158,8 +159,8 @@  class RunQueueScheduler(object):
             else:
                 skip_buildable[rtaskname] = 1
 
-        if len(self.buildable) == 1:
-            tid = self.buildable[0]
+        if len(buildable) == 1:
+            tid = buildable[0]
             taskname = taskname_from_tid(tid)
             if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
                 return None
@@ -174,7 +175,7 @@  class RunQueueScheduler(object):
 
         best = None
         bestprio = None
-        for tid in self.buildable:
+        for tid in buildable:
             taskname = taskname_from_tid(tid)
             if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
                 continue

Comments

Richard Purdie July 11, 2019, 11:33 a.m.
On Thu, 2019-07-11 at 10:57 +0000, Jacob Kroon wrote:
> Exciting to see steps towards the sstate hash equivalence thing!
> 
> I tested master-next, 6ff6485fa9195055ab6057527c6388f6c013cb8d, and
> got:
> 
> Sstate summary: Wanted 1152 Found 579 Missed 573 Current 0 (50%
> match,
> 0% complete)
> NOTE: Executing Tasks
> Traceback (most recent call last):
>    File "<snip>/bitbake/lib/bb/ui/knotty.py", line 480, in main
>      termfilter.updateFooter()
>    File "<snip>/bitbake/lib/bb/ui/knotty.py", line 291, in
> updateFooter
>      content = self.main_progress.update(progress)
>    File "<snip>/bitbake/lib/progressbar/progressbar.py", line 256, in
> update
>      raise ValueError('Value out of range')
> ValueError: Value out of range
> 
> Restarted the build, and now its running...

Thanks, I think this is the UI not understanding both sets of tasks run
in parallel. There is a patch added to -next and on the list which
hopefully should stop the above happening (I couldn't reproduce it
though).

Cheers,

Richard
Richard Purdie July 11, 2019, 4:21 p.m.
On Thu, 2019-07-11 at 01:05 +0100, Richard Purdie wrote:
> These changes set up the foundations so that we can work on two other
> significant problems:
> 
> a) reuse of sstate artefacts between multiconfig build tasks

I have spent a long time thinking about this so now the codebase was
ready it turned out not to be too hard to make it work. Patches aren't
tested apart from with the unittest but I've posted them and we can
take them from here.

> b) adapting the task queue 'on the fly' if the output of a task
> matches a previous output and hence we have sstate hash equivalency
> 
> Solving these would be great for OE!
> 
> I will follow up with more details on the future plans for the above
> and this code soon.

Adapting to new hashes "on the fly" is definitely going to be the
harder challenge.

This will involve the following steps:

a) Knowing there are new unitask hashes available and propagating that
information

b) Figuring out which tasks should now have their sstate retested

c) Test for sstate existence (if it doesn't exist, no point in going
further)

d) Ensure any normal tasks covered by the sstate task are not running,
wait if any are

e) Remove all those normal tasks from access by the normal task running
code

f) Enable those tasks in the scenequeue code

We have no existing code which moves things "backward" in the state
engine from normal task to scenequeue so this will be new ground, but
should be feasible.

I'm going to wait for the current patches to settle before I look into
this any further though.

Cheers,

Richard
Khem Raj July 12, 2019, 12:57 a.m.
Hi Richard

Good changes I believe. Can you push them to a temp branch on bitbake repo ?

On Thu, Jul 11, 2019 at 9:21 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2019-07-11 at 01:05 +0100, Richard Purdie wrote:
> > These changes set up the foundations so that we can work on two other
> > significant problems:
> >
> > a) reuse of sstate artefacts between multiconfig build tasks
>
> I have spent a long time thinking about this so now the codebase was
> ready it turned out not to be too hard to make it work. Patches aren't
> tested apart from with the unittest but I've posted them and we can
> take them from here.
>
> > b) adapting the task queue 'on the fly' if the output of a task
> > matches a previous output and hence we have sstate hash equivalency
> >
> > Solving these would be great for OE!
> >
> > I will follow up with more details on the future plans for the above
> > and this code soon.
>
> Adapting to new hashes "on the fly" is definitely going to be the
> harder challenge.
>
> This will involve the following steps:
>
> a) Knowing there are new unitask hashes available and propagating that
> information
>
> b) Figuring out which tasks should now have their sstate retested
>
> c) Test for sstate existence (if it doesn't exist, no point in going
> further)
>
> d) Ensure any normal tasks covered by the sstate task are not running,
> wait if any are
>
> e) Remove all those normal tasks from access by the normal task running
> code
>
> f) Enable those tasks in the scenequeue code
>
> We have no existing code which moves things "backward" in the state
> engine from normal task to scenequeue so this will be new ground, but
> should be feasible.
>
> I'm going to wait for the current patches to settle before I look into
> this any further though.
>
> Cheers,
>
> Richard
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Richard Purdie July 12, 2019, 8:13 a.m.
Hi Khem,

On Thu, 2019-07-11 at 17:57 -0700, Khem Raj wrote:
> Hi Richard
> 
> Good changes I believe. Can you push them to a temp branch on bitbake
> repo ?

They're in master-next!

Cheers,

Richard