| Message ID | 20231026103320.2751277-1-alex@linutronix.de |
|---|---|
| State | Accepted, archived |
| Commit | 7a7d76aa8a8d590ebc99156f9f4b9535cdf868c7 |
| Headers | show |
| Series | [1/3] selftest/sstatetests: add tests for 'bitbake -S printdiff' | expand |
Hello Alex, On Thu, 26 Oct 2023 12:33:18 +0200 "Alexander Kanavin" <alex.kanavin@gmail.com> wrote: > 'bitbake -S printdiff' is a useful diagnostic facility for finding out > why sstate is not being reused, but until now it had no tests that would > ensure it works. This commit adds three basic scenarios: > > 1. make a change in a really basic, common recipe that is at the very root > of dependency trees (quilt-native), and ensure that change is correctly discovered when > building an image. > > 2. make a change in gcc-source recipe, which is somewhat special > (operates in work-shared), and ensure that gcc-runtime builds track > that down as well. > > 3. make a change in base_do_configure() definition from base.bbclass, > which is not recipe-specific, but affects many basic recipes, and ensure that > is correctly reported as well. > > The test itself actually runs twice: > - first against a fully populated build directory, where > the printdiff code is guaranteed to find the correct previous > stamp that can be compared with in a predictable manner. > > - then in an empty build directory where the printdiff code > goes to look in the sstate cache, and so the existence of the > previous signature can be tested, but not the difference with it > (what the exact difference would be is unpredictable as the > sstate cache is indeed shared between many builds). > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > .../base-do-configure-modified.bbclass | 3 + > .../gcc-source/gcc-source_%.bbappend | 2 + > .../quilt-native/quilt-native_%.bbappend | 2 + > meta/lib/oeqa/selftest/cases/sstatetests.py | 113 ++++++++++++++++++ Is this different from the series you sent on the 23rd? There is no vN tag and no ChangeLog, so it's not obvious. Luca
On Thu, 26 Oct 2023 at 22:02, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote: > Is this different from the series you sent on the 23rd? There is no > vN tag and no ChangeLog, so it's not obvious. Only the third patch should be different, and it has core-image-weston removed (as it was causing missing objects, as reported by you). I'll try to include changes under --- in each commit going forward. Alex
Hello Alex, On Fri, 27 Oct 2023 07:29:28 +0200 "Alexander Kanavin" <alex.kanavin@gmail.com> wrote: > On Thu, 26 Oct 2023 at 22:02, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote: > > Is this different from the series you sent on the 23rd? There is no > > vN tag and no ChangeLog, so it's not obvious. > > Only the third patch should be different, and it has core-image-weston > removed (as it was causing missing objects, as reported by you). I'll > try to include changes under --- in each commit going forward. That will be useful here, thanks! Without it, at least a v<N> in the subject would be good to help keeping track of the current iteration. Luca
On Thu, 2023-10-26 at 12:33 +0200, Alexander Kanavin wrote: > > 'bitbake -S printdiff' is a useful diagnostic facility for finding out > > why sstate is not being reused, but until now it had no tests that would > > ensure it works. This commit adds three basic scenarios: > > > > 1. make a change in a really basic, common recipe that is at the very root > > of dependency trees (quilt-native), and ensure that change is correctly discovered when > > building an image. > > > > 2. make a change in gcc-source recipe, which is somewhat special > > (operates in work-shared), and ensure that gcc-runtime builds track > > that down as well. > > > > 3. make a change in base_do_configure() definition from base.bbclass, > > which is not recipe-specific, but affects many basic recipes, and ensure that > > is correctly reported as well. > > > > The test itself actually runs twice: > > - first against a fully populated build directory, where > > the printdiff code is guaranteed to find the correct previous > > stamp that can be compared with in a predictable manner. > > > > - then in an empty build directory where the printdiff code > > goes to look in the sstate cache, and so the existence of the > > previous signature can be tested, but not the difference with it > > (what the exact difference would be is unpredictable as the > > sstate cache is indeed shared between many builds). > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> There have been autobulilder failures from the do_configure base change and we've not been quite sure why. The summary below took quite a bit of digging but seems quite simple when written up like this! If you dig into a failure like: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2379/steps/15/logs/stdio you see messages like: The differences between the current build and any cached tasks start at the following tasks: /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_collect_spdx_deps virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_configure virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/zlib/zlib_1.3.bb:do_collect_spdx_deps virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_collect_spdx_deps /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_prepare_recipe_sysroot which seems odd. Taking the first one, if quilt-native has built, you'd expect those to be present. I went onto the arm builder and tried a "bitbake quilt-native -S printdiff -DDD": DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/4d/8d/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:4d8da10af56c40d61adb5febb96f583a399ceb276d0dc13e61971f1eb0b7d81d_create_runtime_spdx.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/85/79/sstate:quilt-native::0.67:r0::11:85794fd6d83b61cb273e4d14f6e3581d7fa4ec743b455f1d21786cff032c262c_patch.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b8/fb/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:b8fbbfbafeb7cdf333f1766d6b3c94822a21d42145e17377bede29e44a86a14d_prepare_recipe_sysroot.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fe/ae/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:feae04b2abf592e3df83d0dcac1f74399e342ba99c718975d34b8a27e07eeca5_configure.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f5/98/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:f598fad5807fabde02cd16e5f7f26b486d0b5f87dec6ff3c3d5378ba68780ff1_install.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/e2/a4/sstate:quilt-native::0.67:r0::11:e2a41fd44aa7736bd318ec9c4a6b91e4b80563ba213035afe3df3cb96fdf7aaf_fetch.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/da/07/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:da07e6362c0b01fc2798456aa1319e2343a556e33d0857f90a129701eaa722b2_deploy_source_date_epoch.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/96/sstate:quilt-native::0.67:r0::11:7396e70fb1865087409e454eb8bacae3bcb4add6218b3cd707d2e96880b72936_populate_lic.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/8e/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:9f8e9e5303322240b173483312d885144dfdf2228698868857a281b510ec3214_recipe_qa.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fd/c4/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:fdc40ea4bffb4485bdbb3c38a13b4e5d8d94d1c2cdb66c8659fe5f758a3530a3_populate_sysroot.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/ac/69/sstate:quilt-native::0.67:r0::11:ac69437e7d349349b5f1b2df79203d3ade24e59dc32f16cdf03e6dc92dd3fad6_unpack.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/94/9b/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:949b4472b14403843ae5219789e371979cbc9a53c7fa250033f55157186d902e_compile.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/1b/da/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:1bdafcc9e6b4209bfbdf4b83f8b44e0cc3cc26cdeebe7d713a9f7fcba4b02d48_create_spdx.tar.zst.siginfo and the sstate for quilt-native_0.67.bb:do_collect_spdx_deps is indeed missing, there is only x86: $ ls /srv/autobuilder/autobuilder.yocto.io/pub/sstate/*/72/12/sstate\:quilt-native\:*collect_spdx_deps.tar.zst.siginfo /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:x86_64-linux:0.67:r0:x86_64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo The normal build works since it doesn't care about the siginfo files, just the actual sstate it needs and as a covered task, it just skips over this. This highlights two issues: a) Why is the single sstate siginfo file gone? The most likely explanation is that it wasn't accessed recently and was deleted by the cleanup/ageing code. This makes sense since the code in question never downloads it, only checks it exists and uses that as a marker. Our usage "accesses" are in a different codepath to the parsing check. b) This means -S printdiff doesn't quite do what you'd expect since it doesn't skip "covered" tasks in the same way a normal build would. To reproduce the test failure you should be able to just delete that siginfo file from your local sstate. Cheers, Richard
On Mon, 6 Nov 2023 at 13:48, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > This highlights two issues: > > a) Why is the single sstate siginfo file gone? The most likely > explanation is that it wasn't accessed recently and was deleted by the > cleanup/ageing code. This makes sense since the code in question never > downloads it, only checks it exists and uses that as a marker. Our > usage "accesses" are in a different codepath to the parsing check. > > b) This means -S printdiff doesn't quite do what you'd expect since it > doesn't skip "covered" tasks in the same way a normal build would. > > To reproduce the test failure you should be able to just delete that > siginfo file from your local sstate. It is tempting to then not use the 'live' AB cache, and run the tests against private synthetic cache like the most of the rest of sstatetests does. Should we? Alex
On Mon, 2023-11-06 at 14:07 +0100, Alexander Kanavin wrote: > On Mon, 6 Nov 2023 at 13:48, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > This highlights two issues: > > > > a) Why is the single sstate siginfo file gone? The most likely > > explanation is that it wasn't accessed recently and was deleted by the > > cleanup/ageing code. This makes sense since the code in question never > > downloads it, only checks it exists and uses that as a marker. Our > > usage "accesses" are in a different codepath to the parsing check. > > > > b) This means -S printdiff doesn't quite do what you'd expect since it > > doesn't skip "covered" tasks in the same way a normal build would. > > > > To reproduce the test failure you should be able to just delete that > > siginfo file from your local sstate. > > It is tempting to then not use the 'live' AB cache, and run the tests > against private synthetic cache like the most of the rest of > sstatetests does. Should we? These tests are already very very slow and slowing down builds overall as they're taking 4000s to run. The reason is that unlike the others, they force a full build, probably due to the hashequiv disabling. We need these workflows to work in the real world and I think we need to think of better ways to fix them and make them more performant if we can too. Cheers, Richard
On Mon, 2023-11-06 at 12:47 +0000, Richard Purdie via lists.openembedded.org wrote: > On Thu, 2023-10-26 at 12:33 +0200, Alexander Kanavin wrote: > > > 'bitbake -S printdiff' is a useful diagnostic facility for finding out > > > why sstate is not being reused, but until now it had no tests that would > > > ensure it works. This commit adds three basic scenarios: > > > > > > 1. make a change in a really basic, common recipe that is at the very root > > > of dependency trees (quilt-native), and ensure that change is correctly discovered when > > > building an image. > > > > > > 2. make a change in gcc-source recipe, which is somewhat special > > > (operates in work-shared), and ensure that gcc-runtime builds track > > > that down as well. > > > > > > 3. make a change in base_do_configure() definition from base.bbclass, > > > which is not recipe-specific, but affects many basic recipes, and ensure that > > > is correctly reported as well. > > > > > > The test itself actually runs twice: > > > - first against a fully populated build directory, where > > > the printdiff code is guaranteed to find the correct previous > > > stamp that can be compared with in a predictable manner. > > > > > > - then in an empty build directory where the printdiff code > > > goes to look in the sstate cache, and so the existence of the > > > previous signature can be tested, but not the difference with it > > > (what the exact difference would be is unpredictable as the > > > sstate cache is indeed shared between many builds). > > > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > There have been autobulilder failures from the do_configure base change > and we've not been quite sure why. The summary below took quite a bit > of digging but seems quite simple when written up like this! > > If you dig into a failure like: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2379/steps/15/logs/stdio > > you see messages like: > > The differences between the current build and any cached tasks start at the following tasks: > /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_collect_spdx_deps > virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_configure > virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/zlib/zlib_1.3.bb:do_collect_spdx_deps > virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_collect_spdx_deps > /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_prepare_recipe_sysroot > > which seems odd. Taking the first one, if quilt-native has built, you'd > expect those to be present. > > I went onto the arm builder and tried a "bitbake quilt-native -S printdiff -DDD": > > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/4d/8d/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:4d8da10af56c40d61adb5febb96f583a399ceb276d0dc13e61971f1eb0b7d81d_create_runtime_spdx.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/85/79/sstate:quilt-native::0.67:r0::11:85794fd6d83b61cb273e4d14f6e3581d7fa4ec743b455f1d21786cff032c262c_patch.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b8/fb/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:b8fbbfbafeb7cdf333f1766d6b3c94822a21d42145e17377bede29e44a86a14d_prepare_recipe_sysroot.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fe/ae/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:feae04b2abf592e3df83d0dcac1f74399e342ba99c718975d34b8a27e07eeca5_configure.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f5/98/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:f598fad5807fabde02cd16e5f7f26b486d0b5f87dec6ff3c3d5378ba68780ff1_install.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/e2/a4/sstate:quilt-native::0.67:r0::11:e2a41fd44aa7736bd318ec9c4a6b91e4b80563ba213035afe3df3cb96fdf7aaf_fetch.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/da/07/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:da07e6362c0b01fc2798456aa1319e2343a556e33d0857f90a129701eaa722b2_deploy_source_date_epoch.tar.zst.siginfo > DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/96/sstate:quilt-native::0.67:r0::11:7396e70fb1865087409e454eb8bacae3bcb4add6218b3cd707d2e96880b72936_populate_lic.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/8e/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:9f8e9e5303322240b173483312d885144dfdf2228698868857a281b510ec3214_recipe_qa.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fd/c4/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:fdc40ea4bffb4485bdbb3c38a13b4e5d8d94d1c2cdb66c8659fe5f758a3530a3_populate_sysroot.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/ac/69/sstate:quilt-native::0.67:r0::11:ac69437e7d349349b5f1b2df79203d3ade24e59dc32f16cdf03e6dc92dd3fad6_unpack.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/94/9b/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:949b4472b14403843ae5219789e371979cbc9a53c7fa250033f55157186d902e_compile.tar.zst.siginfo > DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/1b/da/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:1bdafcc9e6b4209bfbdf4b83f8b44e0cc3cc26cdeebe7d713a9f7fcba4b02d48_create_spdx.tar.zst.siginfo > > and the sstate for quilt-native_0.67.bb:do_collect_spdx_deps is indeed missing, there is only x86: > > $ ls /srv/autobuilder/autobuilder.yocto.io/pub/sstate/*/72/12/sstate\:quilt-native\:*collect_spdx_deps.tar.zst.siginfo > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:x86_64-linux:0.67:r0:x86_64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo > > The normal build works since it doesn't care about the siginfo files, > just the actual sstate it needs and as a covered task, it just skips > over this. > > This highlights two issues: > > a) Why is the single sstate siginfo file gone? The most likely > explanation is that it wasn't accessed recently and was deleted by the > cleanup/ageing code. This makes sense since the code in question never > downloads it, only checks it exists and uses that as a marker. Our > usage "accesses" are in a different codepath to the parsing check. > > b) This means -S printdiff doesn't quite do what you'd expect since it > doesn't skip "covered" tasks in the same way a normal build would. > > To reproduce the test failure you should be able to just delete that > siginfo file from your local sstate. I had a look into b), where I needed to teach the runqueue differences code about setscene covered tasks. A really hacky way to test that would be: diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index bb999c9dc1..3592130046 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1697,7 +1697,7 @@ class RunQueue: tocheck.add(tid) - valid_new = self.validate_hashes(tocheck, self.cooker.data, 0, True, summary=False) + valid_new = self.validate_hashes(tocheck, self.cooker.data, 0, True, summary=True) # Tasks which are both setscene and noexec never care about dependencies # We therefore find tasks which are setscene and noexec and mark their @@ -1720,8 +1722,21 @@ class RunQueue: if tid not in valid_new and tid not in noexec: invalidtasks.add(tid) + self.sqdata = SQData() + self.sqdata.cantskip = set() + self.sqdata.tasks_scenequeue_done = set() + self.sqdata.sq_buildable = set() + build_scenequeue_data(self.sqdata, self.rqdata, None, self.cooker, None, self.sqdata) + + for tid in self.rqdata.runq_setscene_tids: + if tid not in valid_new: + continue + for dep in self.sqdata.sq_covered_tasks[tid]: + if dep in invalidtasks: + invalidtasks.remove(dep) + found = set() for tid in invalidtasks: processed = set() toprocess = set([tid]) while toprocess: @@ -2961,6 +2978,8 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): sqrq.sq_deferred[tid] = sqdata.hashes[h] bb.debug(1, "Deferring %s after %s" % (tid, sqdata.hashes[h])) + if not rq: + return update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True) # Compute a list of 'stale' sstate tasks where the current hash does not match the one the code needing some proper refactoring to allow that for an upstreamable patch. The trouble is that whilst it is improved, this still showed glitches: The differences between the current build and any cached tasks start at the following tasks: /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_prepare_recipe_sysroot virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/makedevs/makedevs_1.0.1.bb:do_configure /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_deploy_source_date_epoch virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/pigz/pigz_2.8.bb:do_configure virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_configure virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_configure /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_configure /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb:do_configure /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/gettext/gettext-minimal-native_0.22.bb:do_configure virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/zstd/zstd_1.5.5.bb:do_configure virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/unzip/unzip_6.0.bb:do_configure e.g. why the tzcode-native.bb:do_deploy_source_date_epoch issue? Further debugging with "bitbake tzcode-native -S printdiff -DDD": DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/e2/a4/sstate:quilt-native::0.67:r0::11:e2a41fd44aa7736bd318ec9c4a6b91e4b80563ba213035afe3df3cb96fdf7aaf_fetch.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b8/fb/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:b8fbbfbafeb7cdf333f1766d6b3c94822a21d42145e17377bede29e44a86a14d_prepare_recipe_sysroot.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/94/9b/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:949b4472b14403843ae5219789e371979cbc9a53c7fa250033f55157186d902e_compile.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/ee/ae/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:eeaecb5daeb66cd24558c8ad6f75a33cbc3a108d6c6e98ebcc2e28e73cc4978c_compile.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/83/6e/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:836eeaecc95081aa94d30ec4d61b30b325f92b9913c77190d0e22eeadb5b7d0b_create_runtime_spdx.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/8e/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:9f8e9e5303322240b173483312d885144dfdf2228698868857a281b510ec3214_recipe_qa.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/59/02/sstate:tzcode-native::2023c:r0::11:5902de31682091449c8d40162f35f4ef0bee1a4879b3743b565504fd4fb03f27_populate_lic.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/03/07/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:0307b7910df3aae19e2714a290786fd46213b5664ed082eb38658ee4d0a7295b_recipe_qa.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/85/79/sstate:quilt-native::0.67:r0::11:85794fd6d83b61cb273e4d14f6e3581d7fa4ec743b455f1d21786cff032c262c_patch.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/a7/fa/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:a7fa933ded6a2c1c8a60de136a9a46cd1788fb47c53e5a00cb981349ffd260e5_configure.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/ac/69/sstate:quilt-native::0.67:r0::11:ac69437e7d349349b5f1b2df79203d3ade24e59dc32f16cdf03e6dc92dd3fad6_unpack.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fe/ae/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:feae04b2abf592e3df83d0dcac1f74399e342ba99c718975d34b8a27e07eeca5_configure.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/03/0d/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:030dba86bef371a20bb779cef7cdd93c0c84eaa0fadfe1d74d8784cfc9980b2c_deploy_source_date_epoch.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f5/98/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:f598fad5807fabde02cd16e5f7f26b486d0b5f87dec6ff3c3d5378ba68780ff1_install.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2b/cf/sstate:tzcode-native::2023c:r0::11:2bcf4112bf22bd356e5352e9f6dde17040e9f9ebc4a60b4321112f501a6596bd_fetch.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/3b/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:9f3b9de096a3092769fc5c09a8f85de88aff486874b1c97150eb4cff252f5f03_install.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f6/97/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:f697fc1dffb4c3e7942e0a0e952978b50535cc61a8ed1bd1b0f5c9e1bc91d676_prepare_recipe_sysroot.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/66/4d/sstate:tzcode-native::2023c:r0::11:664de93a4c91a94e498ce6039aba6bd490699f18b4295163b78495524dda17b9_patch.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/91/da/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:91dadb158ecb0b8ac2178d8847bd8894bb8b95f84f53af47d617ef2c7eb2c3dd_populate_sysroot.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b0/30/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:b030f98a1bbe000c1b9eda75252084fd2e43239d55d0b410306ca5566ecf422b_collect_spdx_deps.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/da/07/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:da07e6362c0b01fc2798456aa1319e2343a556e33d0857f90a129701eaa722b2_deploy_source_date_epoch.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/c2/92/sstate:tzcode-native::2023c:r0::11:c2925547a4521cb90a5e8b48c4e1397115800337248e1cd21ef25e5ab88e8834_unpack.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fd/c4/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:fdc40ea4bffb4485bdbb3c38a13b4e5d8d94d1c2cdb66c8659fe5f758a3530a3_populate_sysroot.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/37/9d/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:379d43911b1203a6ff7232a38b8f1eb95bdc3df5924ee06ed4f3be84a691d864_create_spdx.tar.zst.siginfo so it seems that missing siginfo of tasks *before* do_configure would also cause the test to show issues. I'm going to try a patch touching sstate files to try and improve things. I do think the runqueue patches to improve printdiff may also be worthwhile as the above was much more readable, which is the end goal. Cheers, Richard
diff --git a/meta-selftest/classes/base-do-configure-modified.bbclass b/meta-selftest/classes/base-do-configure-modified.bbclass new file mode 100644 index 00000000000..3f96827a428 --- /dev/null +++ b/meta-selftest/classes/base-do-configure-modified.bbclass @@ -0,0 +1,3 @@ +base_do_configure:append () { + echo "this changes base_do_configure() definiton" +} diff --git a/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend new file mode 100644 index 00000000000..205720982cb --- /dev/null +++ b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend @@ -0,0 +1,2 @@ +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests. +include test_recipe.inc diff --git a/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend new file mode 100644 index 00000000000..205720982cb --- /dev/null +++ b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend @@ -0,0 +1,2 @@ +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests. +include test_recipe.inc diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index bdad9088d37..b96eacc9ad8 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -773,3 +773,116 @@ addtask tmptask2 before do_tmptask1 latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:] bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) self.assertEqual(recursecb_count,1) + +class SStatePrintdiff(SStateBase): + # FIXME: OEBasicHash setting is necessary for now as otherwise the following error can occur: + # ERROR: Can't find a task we're supposed to have written out? (hash: e79d70b9c2cc72030c1ce822525510699a1eeb1ddf5986271d3217422244366a)? + # The underlying issue should be investigated and addressed. + def run_test_printdiff_changerecipe(self, target, change_recipe, change_bbtask, change_content, expected_sametmp_output, expected_difftmp_output): + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff" +BB_SIGNATURE_HANDLER = "OEBasicHash" +""") + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff") + bitbake(target) + bitbake("-S none {}".format(target)) + bitbake(change_bbtask) + self.write_recipeinc(change_recipe, change_content) + result_sametmp = bitbake("-S printdiff {}".format(target)) + + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2" +BB_SIGNATURE_HANDLER = "OEBasicHash" +""") + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2") + result_difftmp = bitbake("-S printdiff {}".format(target)) + + self.delete_recipeinc(change_recipe) + for item in expected_sametmp_output: + self.assertIn(item, result_sametmp.output) + for item in expected_difftmp_output: + self.assertIn(item, result_difftmp.output) + + def run_test_printdiff_changeconfig(self, target, change_content, expected_sametmp_output, expected_difftmp_output): + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff" +BB_SIGNATURE_HANDLER = "OEBasicHash" +""") + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff") + bitbake(target) + bitbake("-S none {}".format(target)) + self.append_config(change_content) + result_sametmp = bitbake("-S printdiff {}".format(target)) + + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2" +BB_SIGNATURE_HANDLER = "OEBasicHash" +""") + self.append_config(change_content) + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2") + result_difftmp = bitbake("-S printdiff {}".format(target)) + + for item in expected_sametmp_output: + self.assertIn(item, result_sametmp.output) + for item in expected_difftmp_output: + self.assertIn(item, result_difftmp.output) + + + # Check if printdiff walks the full dependency chain from the image target to where the change is in a specific recipe + def test_image_minimal_vs_quilt(self): + expected_output = ("Task quilt-native:do_install couldn't be used from the cache because:", +"We need hash", +"most recent matching task was") + expected_sametmp_output = expected_output + ("Variable do_install value changed",'+ echo "this changes the task signature"') + expected_difftmp_output = expected_output + + self.run_test_printdiff_changerecipe("core-image-minimal", "quilt-native", "-c do_install quilt-native", +""" +do_install:append() { + echo "this changes the task signature" +} +""", +expected_sametmp_output, expected_difftmp_output) + + # Check if changes to gcc-source (which uses tmp/work-shared) are correctly discovered + def test_gcc_runtime_vs_gcc_source(self): + gcc_source_pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV'] + + expected_output = ("Task {}:do_preconfigure couldn't be used from the cache because:".format(gcc_source_pn), +"We need hash", +"most recent matching task was") + expected_sametmp_output = expected_output + ("Variable do_preconfigure value changed",'+ print("this changes the task signature")') + #FIXME: printdiff is supposed to find at least one preconfigure task signature in the sstate cache, but isn't able to + #expected_difftmp_output = expected_output + expected_difftmp_output = () + + self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn), +""" +python do_preconfigure:append() { + print("this changes the task signature") +} +""", +expected_sametmp_output, expected_difftmp_output) + + # Check if changing a really base task definiton is reported against multiple core recipes using it + def test_image_minimal_vs_base_do_configure(self): + expected_output = ("Task zstd-native:do_configure couldn't be used from the cache because:", +"Task texinfo-dummy-native:do_configure couldn't be used from the cache because:", +"Task ldconfig-native:do_configure couldn't be used from the cache because:", +"Task gettext-minimal-native:do_configure couldn't be used from the cache because:", +"Task tzcode-native:do_configure couldn't be used from the cache because:", +"Task makedevs-native:do_configure couldn't be used from the cache because:", +"Task pigz-native:do_configure couldn't be used from the cache because:", +"Task update-rc.d-native:do_configure couldn't be used from the cache because:", +"Task unzip-native:do_configure couldn't be used from the cache because:", +"Task gnu-config-native:do_configure couldn't be used from the cache because:", +"We need hash", +"most recent matching task was") + expected_sametmp_output = expected_output + ("Variable base_do_configure value changed",'+ echo "this changes base_do_configure() definiton"') + expected_difftmp_output = expected_output + + self.run_test_printdiff_changeconfig("core-image-minimal", +""" +INHERIT += "base-do-configure-modified" +""", +expected_sametmp_output, expected_difftmp_output)
'bitbake -S printdiff' is a useful diagnostic facility for finding out why sstate is not being reused, but until now it had no tests that would ensure it works. This commit adds three basic scenarios: 1. make a change in a really basic, common recipe that is at the very root of dependency trees (quilt-native), and ensure that change is correctly discovered when building an image. 2. make a change in gcc-source recipe, which is somewhat special (operates in work-shared), and ensure that gcc-runtime builds track that down as well. 3. make a change in base_do_configure() definition from base.bbclass, which is not recipe-specific, but affects many basic recipes, and ensure that is correctly reported as well. The test itself actually runs twice: - first against a fully populated build directory, where the printdiff code is guaranteed to find the correct previous stamp that can be compared with in a predictable manner. - then in an empty build directory where the printdiff code goes to look in the sstate cache, and so the existence of the previous signature can be tested, but not the difference with it (what the exact difference would be is unpredictable as the sstate cache is indeed shared between many builds). Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- .../base-do-configure-modified.bbclass | 3 + .../gcc-source/gcc-source_%.bbappend | 2 + .../quilt-native/quilt-native_%.bbappend | 2 + meta/lib/oeqa/selftest/cases/sstatetests.py | 113 ++++++++++++++++++ 4 files changed, 120 insertions(+) create mode 100644 meta-selftest/classes/base-do-configure-modified.bbclass create mode 100644 meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend create mode 100644 meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend