Patchwork SetScene tasks hang forever?

login
register
mail settings
Submitter Richard Purdie
Date May 9, 2012, 8:32 p.m.
Message ID <1336595524.2494.77.camel@ted>
Download mbox | patch
Permalink /patch/27403/
State New
Headers show

Comments

Richard Purdie - May 9, 2012, 8:32 p.m.
Hi Rich,

You might like to try the change below as I think it might address the problem.

Cheers,

Richard

bitbake/runqueue: Fix 'full' stamp checking to be more efficient and cache results

This should fix issues where bitbake would seemingly lock up when checking
certain configurations of stampfiles.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
Rich Pixley - May 9, 2012, 11:20 p.m.
My rebuild completed in 5 seconds.

Thank you very much!

I'd rather not fork bitbake locally.  Can I expect this patch to show up 
in git://git.openembedded.org/bitbake sometime soon?

--rich

On 5/9/12 13:32 , Richard Purdie wrote:
> Hi Rich,
>
> You might like to try the change below as I think it might address the problem.
>
> Cheers,
>
> Richard
>
> bitbake/runqueue: Fix 'full' stamp checking to be more efficient and cache results
>
> This should fix issues where bitbake would seemingly lock up when checking
> certain configurations of stampfiles.
>
> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index b870caf..48433be 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -875,7 +875,7 @@ class RunQueue:
>               bb.msg.fatal("RunQueue", "check_stamps fatal internal error")
>           return current
>
> -    def check_stamp_task(self, task, taskname = None, recurse = False):
> +    def check_stamp_task(self, task, taskname = None, recurse = False, cache = {}):
>           def get_timestamp(f):
>               try:
>                   if not os.access(f, os.F_OK):
> @@ -915,6 +915,9 @@ class RunQueue:
>           t1 = get_timestamp(stampfile)
>           for dep in self.rqdata.runq_depends[task]:
>               if iscurrent:
> +                if dep in cache:
> +                    iscurrent = cache[dep]
> +                    continue
>                   fn2 = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[dep]]
>                   taskname2 = self.rqdata.runq_task[dep]
>                   stampfile2 = bb.build.stampfile(taskname2, self.rqdata.dataCache, fn2)
> @@ -931,7 +934,9 @@ class RunQueue:
>                           logger.debug(2, 'Stampfile %s<  %s', stampfile, stampfile2)
>                           iscurrent = False
>                       if recurse and iscurrent:
> -                        iscurrent = self.check_stamp_task(dep, recurse=True)
> +                        iscurrent = self.check_stamp_task(dep, recurse=True, cache=cache)
> +                        cache[dep] = iscurrent
> +        cache[task] = iscurrent
>           return iscurrent
>
>       def execute_runqueue(self):
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Richard Purdie - May 9, 2012, 11:32 p.m.
On Wed, 2012-05-09 at 16:20 -0700, Rich Pixley wrote:
> My rebuild completed in 5 seconds.
> 
> Thank you very much!
> 
> I'd rather not fork bitbake locally.  Can I expect this patch to show up 
> in git://git.openembedded.org/bitbake sometime soon?

Yes, I just wanted to confirm we were fixing the right problem...

I'll post it to the bitbake list and merge in a day or two assuming no
negative feedback.

I still wonder if that stamp mode is useful but that is a different
discussion.

Cheers,

Richard
Rich Pixley - May 10, 2012, midnight
On 5/9/12 16:32 , Richard Purdie wrote:
> On Wed, 2012-05-09 at 16:20 -0700, Rich Pixley wrote:
>> My rebuild completed in 5 seconds.
>>
>> Thank you very much!
>>
>> I'd rather not fork bitbake locally.  Can I expect this patch to show up
>> in git://git.openembedded.org/bitbake sometime soon?
> Yes, I just wanted to confirm we were fixing the right problem...
>
> I'll post it to the bitbake list and merge in a day or two assuming no
> negative feedback.
I'm running more thorough tests as I type.
> I still wonder if that stamp mode is useful but that is a different
> discussion.
If I'm understanding your other message, then no, it's probably not.

My only concern is that in the A depends on B depends on C case, 
changing C forces a rebuild of A.  I don't really care which mechanism 
makes that happen.

--rich

Patch

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index b870caf..48433be 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -875,7 +875,7 @@  class RunQueue:
             bb.msg.fatal("RunQueue", "check_stamps fatal internal error")
         return current
 
-    def check_stamp_task(self, task, taskname = None, recurse = False):
+    def check_stamp_task(self, task, taskname = None, recurse = False, cache = {}):
         def get_timestamp(f):
             try:
                 if not os.access(f, os.F_OK):
@@ -915,6 +915,9 @@  class RunQueue:
         t1 = get_timestamp(stampfile)
         for dep in self.rqdata.runq_depends[task]:
             if iscurrent:
+                if dep in cache:
+                    iscurrent = cache[dep]
+                    continue
                 fn2 = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[dep]]
                 taskname2 = self.rqdata.runq_task[dep]
                 stampfile2 = bb.build.stampfile(taskname2, self.rqdata.dataCache, fn2)
@@ -931,7 +934,9 @@  class RunQueue:
                         logger.debug(2, 'Stampfile %s < %s', stampfile, stampfile2)
                         iscurrent = False
                     if recurse and iscurrent:
-                        iscurrent = self.check_stamp_task(dep, recurse=True)
+                        iscurrent = self.check_stamp_task(dep, recurse=True, cache=cache)
+                        cache[dep] = iscurrent
+        cache[task] = iscurrent
         return iscurrent
 
     def execute_runqueue(self):