Patchwork sstate.bbclass: Ensure machine specific stamps are only wiped for the current task

login
register
mail settings
Submitter Richard Purdie
Date Oct. 10, 2011, 2:16 p.m.
Message ID <1318256216.23801.37.camel@ted>
Download mbox | patch
Permalink /patch/12975/
State New, archived
Headers show

Comments

Richard Purdie - Oct. 10, 2011, 2:16 p.m.
On Mon, 2011-10-10 at 13:45 +0200, Martin Jansa wrote:
> On Sun, Oct 09, 2011 at 08:11:45PM +0200, Koen Kooi wrote:
> > 
> > Op 9 okt. 2011, om 20:10 heeft Martin Jansa het volgende geschreven:
> > 
> > > On Sun, Oct 09, 2011 at 01:25:09PM +0200, Koen Kooi wrote:
> > >> 
> > >> Op 6 okt. 2011, om 16:19 heeft Richard Purdie het volgende geschreven:
> > >> 
> > >>> sstate was being a little too ethusiastic about removing stamp files and
> > >>> was removing stamp files for other machines when it shouldn't have been.
> > >>> 
> > >>> This patch teaches sstate about machine specific stamp extensions and
> > >>> allows it to only remove the current task's stampfiles.
> > >> 
> > >> Not sure if it's related to this problem or not but sstate is still not working as intented for me when switching machines, it keeps rebuilding the toolchain when switching machine. What I did this morning:
> > >> 
> > >> 1) wipe out tmp, sstate-cache, pseudodone
> > >> 2) MACHINE=beagleboard bitbake u-boot
> > >> 3) cp conf/machine/beagleboard.conf conf/machine/brokensstate.conf
> > >> 4) MACHINE=brokensstate bitbake u-boot
> > > 
> > > Same here.. :/ (building 3 armv7 machines.. always "from scratch").
> > 
> > To eliminate varables: Angstrom/shr/aurora/micro/other/all of the above?
> 
> DISTRO=shr
> 
> MACHINE=nokia900 bitbake shr-image -> builds from scratch, finished ok
> MACHINE=palmpre bitbake shr-image -> builds from scratch, finished ok
> MACHINE=palmpre2 bitbake shr-image -> builds from scratch, finished ok
> 
> without wiping anything (same tmp, same sstate-cache, same pseudodone)
> 
> MACHINE=palmpre bitbake shr-image -> builds from scratch, again!, finished ok
> MACHINE=palmpre2 bitbake shr-image -> builds from scratch, again!, finished ok
> MACHINE=nokia900 bitbake shr-image -> builds from scratch, again!, finished ok
> 
> palmpre and palmpre2 are "the same"
> $ cat meta-smartphone/meta-palm/conf/machine/palmpre.conf | grep -v ^#
> require conf/machine/include/palmpre.inc
> 
> $ cat meta-smartphone/meta-palm/conf/machine/palmpre2.conf | grep -v ^#
> require conf/machine/include/palmpre.inc
> 
> nokia900 is also armv7a-vfp-neon (that's why all packages except
> MACHINE_ARCH were always reused without rebuilding in OE-classic).
> 
> I have tried to debug sigdata files to see why and also tried to add
> MACHINEOVERRIDES[vardepsexclude] = "MACHINE"
> as RP suggested, but sofar haven't found solution at least to reuse
> already built sstate packages for armv7a-vfp-neon.

Also as a tip for debugging this, you can do what is in the following
commit (patch also inline below):

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip9&id=1261cc69b152f33b934c2448020dfa3c95f312c6

which will mean bitbake -n generates stamp files for a given bitbake
run. You can then quickly do:

MACHINE=A bitbake -n foo
MACHINE=B bitbake -n foo

for example and then look for cases where two stamp sigdata files were
generated. If you see them you can "bitbake-diffsigs A B" and I'll be
very interested in the results of that and "bitbake-diffsigs A"...

Cheers,

Richard

bitbake: Allow easier dry run stamp debugging

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
Koen Kooi - Oct. 10, 2011, 2:37 p.m.
Op 10 okt. 2011, om 16:16 heeft Richard Purdie het volgende geschreven:

> On Mon, 2011-10-10 at 13:45 +0200, Martin Jansa wrote:
>> On Sun, Oct 09, 2011 at 08:11:45PM +0200, Koen Kooi wrote:
>>> 
>>> Op 9 okt. 2011, om 20:10 heeft Martin Jansa het volgende geschreven:
>>> 
>>>> On Sun, Oct 09, 2011 at 01:25:09PM +0200, Koen Kooi wrote:
>>>>> 
>>>>> Op 6 okt. 2011, om 16:19 heeft Richard Purdie het volgende geschreven:
>>>>> 
>>>>>> sstate was being a little too ethusiastic about removing stamp files and
>>>>>> was removing stamp files for other machines when it shouldn't have been.
>>>>>> 
>>>>>> This patch teaches sstate about machine specific stamp extensions and
>>>>>> allows it to only remove the current task's stampfiles.
>>>>> 
>>>>> Not sure if it's related to this problem or not but sstate is still not working as intented for me when switching machines, it keeps rebuilding the toolchain when switching machine. What I did this morning:
>>>>> 
>>>>> 1) wipe out tmp, sstate-cache, pseudodone
>>>>> 2) MACHINE=beagleboard bitbake u-boot
>>>>> 3) cp conf/machine/beagleboard.conf conf/machine/brokensstate.conf
>>>>> 4) MACHINE=brokensstate bitbake u-boot
>>>> 
>>>> Same here.. :/ (building 3 armv7 machines.. always "from scratch").
>>> 
>>> To eliminate varables: Angstrom/shr/aurora/micro/other/all of the above?
>> 
>> DISTRO=shr
>> 
>> MACHINE=nokia900 bitbake shr-image -> builds from scratch, finished ok
>> MACHINE=palmpre bitbake shr-image -> builds from scratch, finished ok
>> MACHINE=palmpre2 bitbake shr-image -> builds from scratch, finished ok
>> 
>> without wiping anything (same tmp, same sstate-cache, same pseudodone)
>> 
>> MACHINE=palmpre bitbake shr-image -> builds from scratch, again!, finished ok
>> MACHINE=palmpre2 bitbake shr-image -> builds from scratch, again!, finished ok
>> MACHINE=nokia900 bitbake shr-image -> builds from scratch, again!, finished ok
>> 
>> palmpre and palmpre2 are "the same"
>> $ cat meta-smartphone/meta-palm/conf/machine/palmpre.conf | grep -v ^#
>> require conf/machine/include/palmpre.inc
>> 
>> $ cat meta-smartphone/meta-palm/conf/machine/palmpre2.conf | grep -v ^#
>> require conf/machine/include/palmpre.inc
>> 
>> nokia900 is also armv7a-vfp-neon (that's why all packages except
>> MACHINE_ARCH were always reused without rebuilding in OE-classic).
>> 
>> I have tried to debug sigdata files to see why and also tried to add
>> MACHINEOVERRIDES[vardepsexclude] = "MACHINE"
>> as RP suggested, but sofar haven't found solution at least to reuse
>> already built sstate packages for armv7a-vfp-neon.
> 
> Also as a tip for debugging this, you can do what is in the following
> commit (patch also inline below):
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip9&id=1261cc69b152f33b934c2448020dfa3c95f312c6
> 
> which will mean bitbake -n generates stamp files for a given bitbake
> run. You can then quickly do:
> 
> MACHINE=A bitbake -n foo
> MACHINE=B bitbake -n foo
> 
> for example and then look for cases where two stamp sigdata files were
> generated. If you see them you can "bitbake-diffsigs A B" and I'll be
> very interested in the results of that and "bitbake-diffsigs A"...
> 
> Cheers,
> 
> Richard
> 
> bitbake: Allow easier dry run stamp debugging
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py

This seems to be against poky instead of bitbake and doesn't apply against bitbake master:

koen@dominion:/OE/tentacle/sources/bitbake$ patch -p2 < dry 
patching file lib/bb/build.py
Hunk #1 succeeded at 257 (offset 2 lines).
Hunk #2 succeeded at 319 (offset 2 lines).
Hunk #3 succeeded at 360 (offset 2 lines).
patching file lib/bb/runqueue.py
Hunk #1 FAILED at 1156.
1 out of 1 hunk FAILED -- saving rejects to file lib/bb/runqueue.py.rej
Martin Jansa - Oct. 10, 2011, 2:41 p.m.
On Mon, Oct 10, 2011 at 04:37:51PM +0200, Koen Kooi wrote:
> 
> Op 10 okt. 2011, om 16:16 heeft Richard Purdie het volgende geschreven:
> 
> > On Mon, 2011-10-10 at 13:45 +0200, Martin Jansa wrote:
> >> On Sun, Oct 09, 2011 at 08:11:45PM +0200, Koen Kooi wrote:
> >>> 
> >>> Op 9 okt. 2011, om 20:10 heeft Martin Jansa het volgende geschreven:
> >>> 
> >>>> On Sun, Oct 09, 2011 at 01:25:09PM +0200, Koen Kooi wrote:
> >>>>> 
> >>>>> Op 6 okt. 2011, om 16:19 heeft Richard Purdie het volgende geschreven:
> >>>>> 
> >>>>>> sstate was being a little too ethusiastic about removing stamp files and
> >>>>>> was removing stamp files for other machines when it shouldn't have been.
> >>>>>> 
> >>>>>> This patch teaches sstate about machine specific stamp extensions and
> >>>>>> allows it to only remove the current task's stampfiles.
> >>>>> 
> >>>>> Not sure if it's related to this problem or not but sstate is still not working as intented for me when switching machines, it keeps rebuilding the toolchain when switching machine. What I did this morning:
> >>>>> 
> >>>>> 1) wipe out tmp, sstate-cache, pseudodone
> >>>>> 2) MACHINE=beagleboard bitbake u-boot
> >>>>> 3) cp conf/machine/beagleboard.conf conf/machine/brokensstate.conf
> >>>>> 4) MACHINE=brokensstate bitbake u-boot
> >>>> 
> >>>> Same here.. :/ (building 3 armv7 machines.. always "from scratch").
> >>> 
> >>> To eliminate varables: Angstrom/shr/aurora/micro/other/all of the above?
> >> 
> >> DISTRO=shr
> >> 
> >> MACHINE=nokia900 bitbake shr-image -> builds from scratch, finished ok
> >> MACHINE=palmpre bitbake shr-image -> builds from scratch, finished ok
> >> MACHINE=palmpre2 bitbake shr-image -> builds from scratch, finished ok
> >> 
> >> without wiping anything (same tmp, same sstate-cache, same pseudodone)
> >> 
> >> MACHINE=palmpre bitbake shr-image -> builds from scratch, again!, finished ok
> >> MACHINE=palmpre2 bitbake shr-image -> builds from scratch, again!, finished ok
> >> MACHINE=nokia900 bitbake shr-image -> builds from scratch, again!, finished ok
> >> 
> >> palmpre and palmpre2 are "the same"
> >> $ cat meta-smartphone/meta-palm/conf/machine/palmpre.conf | grep -v ^#
> >> require conf/machine/include/palmpre.inc
> >> 
> >> $ cat meta-smartphone/meta-palm/conf/machine/palmpre2.conf | grep -v ^#
> >> require conf/machine/include/palmpre.inc
> >> 
> >> nokia900 is also armv7a-vfp-neon (that's why all packages except
> >> MACHINE_ARCH were always reused without rebuilding in OE-classic).
> >> 
> >> I have tried to debug sigdata files to see why and also tried to add
> >> MACHINEOVERRIDES[vardepsexclude] = "MACHINE"
> >> as RP suggested, but sofar haven't found solution at least to reuse
> >> already built sstate packages for armv7a-vfp-neon.
> > 
> > Also as a tip for debugging this, you can do what is in the following
> > commit (patch also inline below):
> > 
> > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip9&id=1261cc69b152f33b934c2448020dfa3c95f312c6
> > 
> > which will mean bitbake -n generates stamp files for a given bitbake
> > run. You can then quickly do:
> > 
> > MACHINE=A bitbake -n foo
> > MACHINE=B bitbake -n foo
> > 
> > for example and then look for cases where two stamp sigdata files were
> > generated. If you see them you can "bitbake-diffsigs A B" and I'll be
> > very interested in the results of that and "bitbake-diffsigs A"...
> > 
> > Cheers,
> > 
> > Richard
> > 
> > bitbake: Allow easier dry run stamp debugging
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> 
> This seems to be against poky instead of bitbake and doesn't apply against bitbake master:
> 
> koen@dominion:/OE/tentacle/sources/bitbake$ patch -p2 < dry 
> patching file lib/bb/build.py
> Hunk #1 succeeded at 257 (offset 2 lines).
> Hunk #2 succeeded at 319 (offset 2 lines).
> Hunk #3 succeeded at 360 (offset 2 lines).
> patching file lib/bb/runqueue.py
> Hunk #1 FAILED at 1156.
> 1 out of 1 hunk FAILED -- saving rejects to file lib/bb/runqueue.py.rej

Same problem here.. seems like diff is bigger then it was before:
both current master

OE @ ~/projects $ diff -rq poky/bitbake/ bitbake/ | grep -v pyc | grep -v pyo | grep -v .git
Only in bitbake/: MANIFEST.in
Only in bitbake/: TODO
Only in poky/bitbake/bin: bitbake-runtask
Only in bitbake/: classes
Only in bitbake/: conf
Files poky/bitbake/lib/bb/__init__.py and bitbake/lib/bb/__init__.py differ
Files poky/bitbake/lib/bb/build.py and bitbake/lib/bb/build.py differ
Files poky/bitbake/lib/bb/cooker.py and bitbake/lib/bb/cooker.py differ
Files poky/bitbake/lib/bb/parse/parse_py/BBHandler.py and bitbake/lib/bb/parse/parse_py/BBHandler.py differ
Files poky/bitbake/lib/bb/pysh/pyshtables.py and bitbake/lib/bb/pysh/pyshtables.py differ
Files poky/bitbake/lib/bb/runqueue.py and bitbake/lib/bb/runqueue.py differ
Only in poky/bitbake/lib/bb: shell.py
Only in bitbake/: setup.py

Patch

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 8937f08..efa9ffe 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -255,7 +255,7 @@  def _task_data(fn, task, d):
     data.expandKeys(localdata)
     return localdata
 
-def _exec_task(fn, task, d, quieterr):
+def _exec_task(fn, task, d, quieterr, dryrun=False):
     """Execute a BB 'task'
 
     Execution of a task involves a bit more setup than executing a function,
@@ -317,10 +317,13 @@  def _exec_task(fn, task, d, quieterr):
     event.fire(TaskStarted(task, localdata), localdata)
     try:
         for func in (prefuncs or '').split():
-            exec_func(func, localdata)
-        exec_func(task, localdata)
+            if not dryrun:
+                exec_func(func, localdata)
+        if not dryrun:
+            exec_func(task, localdata)
         for func in (postfuncs or '').split():
-            exec_func(func, localdata)
+            if not dryrun:
+                exec_func(func, localdata)
     except FuncFailed as exc:
         if not quieterr:
             logger.error(str(exc))
@@ -355,13 +358,13 @@  def _exec_task(fn, task, d, quieterr):
 
     return 0
 
-def exec_task(fn, task, d):
+def exec_task(fn, task, d, dryrun=False):
     try: 
         quieterr = False
         if d.getVarFlag(task, "quieterrors") is not None:
             quieterr = True
 
-        return _exec_task(fn, task, d, quieterr)
+        return _exec_task(fn, task, d, quieterr, dryrun)
     except Exception:
         from traceback import format_exc
         if not quieterr:
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index aca06b5..e53da56 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1156,8 +1156,7 @@  class RunQueueExecute:
                     logger.critical(str(exc))
                 os._exit(1)
             try:
-                if not self.cooker.configuration.dry_run:
-                    ret = bb.build.exec_task(fn, taskname, the_data)
+                ret = bb.build.exec_task(fn, taskname, the_data, self.cooker.configuration.dry_run)
                 os._exit(ret)
             except:
                 os._exit(1)