diff mbox series

[v2] selftest/sstatetests: add tests for 'bitbake -S printdiff'

Message ID 20231018084652.900785-1-alex@linutronix.de
State Accepted, archived
Commit 7a7d76aa8a8d590ebc99156f9f4b9535cdf868c7
Headers show
Series [v2] selftest/sstatetests: add tests for 'bitbake -S printdiff' | expand

Commit Message

Alexander Kanavin Oct. 18, 2023, 8:46 a.m. UTC 
'bitbake -S printdiff' is a useful diagnostic facility for finding out
why sstate is not being reused, but until now it had no tests that would
ensure it works. This commit adds three basic scenarios:

1. make a change in a really basic, common recipe that is at the very root
of dependency trees (quilt-native), and ensure that change is correctly discovered when
building an image.

2. make a change in gcc-source recipe, which is somewhat special
(operates in work-shared), and ensure that gcc-runtime builds track
that down as well.

3. make a change in base_do_configure() definition from base.bbclass,
which is not recipe-specific, but affects many basic recipes, and ensure that
is correctly reported as well.

The test itself actually runs twice:
- first against a fully populated build directory, where
the printdiff code is guaranteed to find the correct previous
stamp that can be compared with in a predictable manner.

- then in an empty build directory where the printdiff code
goes to look in the sstate cache, and so the existence of the
previous signature can be tested, but not the difference with it
(what the exact difference would be is unpredictable as the
sstate cache is indeed shared between many builds).

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../base-do-configure-modified.bbclass        |   3 +
 .../gcc-source/gcc-source_%.bbappend          |   2 +
 .../quilt-native/quilt-native_%.bbappend      |   2 +
 meta/lib/oeqa/selftest/cases/sstatetests.py   | 106 ++++++++++++++++++
 4 files changed, 113 insertions(+)
 create mode 100644 meta-selftest/classes/base-do-configure-modified.bbclass
 create mode 100644 meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend
 create mode 100644 meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend

Comments

Alexandre Belloni Oct. 18, 2023, 7:39 p.m. UTC | #1 
Hello Alex,

I had this failure on the oe-selftest-armhost builder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2290/steps/14/logs/stdio

Task texinfo-dummy-native:do_configure couldn't be used from the cache because:
  We need hash 76c87e43650c854836e1c9a95db39c16de49bf0271535a4da3f030056fcc3fb2, most recent matching task was 8e7ba727bc84dd547f1a52515f25189fddcbcb01634b52521741e389cae9103e
  Hash for task dependency texinfo-dummy-native:do_patch changed from 19ba8d4ae1894b1725dd9b7e90afb71139de2d72c73208c1bea4094fe1ce259d to 11fdb7aa5419cb36b11f0fa281f6e6ca5dcbcc2da515c4d1782b7c0b61ccc447
      Hash for task dependency texinfo-dummy-native:do_unpack changed from 8864a86b4edb8cf635c7b597596fb4bab900afa8648d486f7dfb00c9c8ce03c3 to 8165664451ef563a6c2c84a93abfa4cdf4d617ce1afe55a850fb71cf67b81a01
          basehash changed from 935b6ffebc04d72ae1d16476f82aee22c18c7aaeee340f1ab34d571c3e121147 to 1367421b45cdb92b9994acadd6503c4a0874535e1d175e9c39001998a1c8b862
          Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list
ERROR: Can't find a task we're supposed to have written out? (hash: e79d70b9c2cc72030c1ce822525510699a1eeb1ddf5986271d3217422244366a)?


Task gettext-minimal-native:do_configure couldn't be used from the cache because:
  We need hash ee11593c797a4579bfb48c61940f50ebd2c6da4883ca265d3231352fd5e248c6, most recent matching task was 140c1b528103db5f34ecd8db962b1ccc715a9da95573b9cb9d0441d7c604317c
  basehash changed from ade121cb5b2caa1f610daf254c4d19d28959630e468700c6a5dee6fc44ee8869 to 5190156579dda094eafc1f1df780def5d749972885a93ed7d952833d286f8177
  Variable base_do_configure value changed:
@@ -13,3 +13,4 @@
 		mkdir -p `dirname ${CONFIGURESTAMPFILE}`
 		echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE}
 	fi
+	echo "this changes base_do_configure() definiton"
Task perlcross-native:do_configure couldn't be used from the cache because:
  We need hash f557deda75fdaf738574f41d517b54338c609677a9a0dbb1774d811714f3b7e1, most recent matching task was 7838e1dc635c14048dee89663af3378861d33e779edb609e7a4c5f00fc5af70c
  Task dependencies changed from:
['AR', 'AS', 'B', 'BUILD_AR', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_FC', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'CC', 'CCLD', 'CFLAGS', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CXX', 'CXXFLAGS', 'DEBUG_BUILD', 'DEPENDS', 'ERROR_QA', 'EXTRA_NATIVE_PKGCONFIG_PATH', 'EXTRA_OECONF', 'FC', 'HOST_PREFIX', 'INSANE_SKIP', 'LC_ALL', 'LD', 'LDFLAGS', 'MAKE', 'MLPREFIX', 'NATIVE_PACKAGE_PATH_SUFFIX', 'NM', 'OBJCOPY', 'OBJDUMP', 'P', 'PACKAGECONFIG', 'PERL_HASH_SEED', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PN', 'PSEUDO_DISABLED', 'PV', 'PYTHONHASHSEED', 'QA_ERRORS_FOUND', 'QA_LOGFILE', 'RANLIB', 'READELF', 'RECIPE_SYSROOT_NATIVE', 'S', 'STAGING_BASE_LIBDIR_NATIVE', 'STAGING_BINDIR_NATIVE', 'STAGING_DATADIR', 'STAGING_DATADIR_NATIVE', 'STAGING_DIR_NATIVE', 'STAGING_ETCDIR_NATIVE', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STAGING_SBINDIR_NATIVE', 'STRINGS', 'STRIP', 'TZ', 'UNKNOWN_CONFIGURE_OPT_IGNORE', 'base_bindir', 'base_libdir', 'base_libdir_native', 'base_prefix', 'base_sbindir', 'bindir', 'bindir_native', 'datadir', 'datadir_native', 'do_qa_configure', 'docdir', 'exec_prefix', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'nonarch_base_libdir', 'nonarch_libdir', 'oe.qa.exit_if_errors', 'oe.qa.exit_with_message_if_errors', 'oe.qa.handle_error', 'oe.qa.write_error', 'oe.types.boolean', 'oe.utils.vartrue', 'oldincludedir', 'prefix', 'prefix_native', 'root_prefix', 'sbindir', 'sbindir_native', 'servicedir', 'sharedstatedir', 'sysconfdir', 'sysconfdir_native', 'systemd_system_unitdir', 'systemd_unitdir', 'systemd_user_unitdir']
to:
['AR', 'AS', 'B', 'BUILD_AR', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_FC', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'CC', 'CCLD', 'CFLAGS', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CXX', 'CXXFLAGS', 'DEBUG_BUILD', 'DEPENDS', 'DISABLE_STATIC', 'ERROR_QA', 'EXTRA_NATIVE_PKGCONFIG_PATH', 'EXTRA_OECONF', 'FC', 'HOST_PREFIX', 'INSANE_SKIP', 'LC_ALL', 'LD', 'LDFLAGS', 'MAKE', 'MLPREFIX', 'NATIVE_PACKAGE_PATH_SUFFIX', 'NM', 'OBJCOPY', 'OBJDUMP', 'P', 'PACKAGECONFIG', 'PERL_HASH_SEED', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PN', 'PSEUDO_DISABLED', 'PV', 'PYTHONHASHSEED', 'QA_ERRORS_FOUND', 'QA_LOGFILE', 'RANLIB', 'READELF', 'RECIPE_SYSROOT_NATIVE', 'S', 'STAGING_BASE_LIBDIR_NATIVE', 'STAGING_BINDIR_NATIVE', 'STAGING_DATADIR', 'STAGING_DATADIR_NATIVE', 'STAGING_DIR_NATIVE', 'STAGING_ETCDIR_NATIVE', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STAGING_SBINDIR_NATIVE', 'STRINGS', 'STRIP', 'TZ', 'UNKNOWN_CONFIGURE_OPT_IGNORE', 'base_bindir', 'base_libdir', 'base_libdir_native', 'base_prefix', 'base_sbindir', 'bindir', 'bindir_native', 'datadir', 'datadir_native', 'do_qa_configure', 'docdir', 'exec_prefix', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'nonarch_base_libdir', 'nonarch_libdir', 'oe.qa.exit_if_errors', 'oe.qa.exit_with_message_if_errors', 'oe.qa.handle_error', 'oe.qa.write_error', 'oe.types.boolean', 'oe.utils.vartrue', 'oldincludedir', 'prefix', 'prefix_native', 'root_prefix', 'sbindir', 'sbindir_native', 'servicedir', 'sharedstatedir', 'sysconfdir', 'sysconfdir_native', 'systemd_system_unitdir', 'systemd_unitdir', 'systemd_user_unitdir']
  basehash changed from 4866ba7810881112d6ce0cb85301756c02eb4b282604bae69516ec23d7b20acf to 297e5c7592ebd6378c8fc49fe98ffd9e675a04324346c5b50811c6a5eebbe78e
  List of dependencies for variable EXTRA_OECONF changed from 'frozenset()' to 'frozenset({'DISABLE_STATIC'})'
  Dependency on variable DISABLE_STATIC was added
  Variable EXTRA_OECONF value changed from '' to '${DISABLE_STATIC}'
ERROR: Can't find a task we're supposed to have written out? (hash: 999250461834bf74545962a8577bc26854a3d08bb2a96140d5efe314e0ef2d63)?


ERROR: Can't find a task we're supposed to have written out? (hash: e79d70b9c2cc72030c1ce822525510699a1eeb1ddf5986271d3217422244366a)?

On 18/10/2023 10:46:52+0200, Alexander Kanavin wrote:
> 'bitbake -S printdiff' is a useful diagnostic facility for finding out
> why sstate is not being reused, but until now it had no tests that would
> ensure it works. This commit adds three basic scenarios:
> 
> 1. make a change in a really basic, common recipe that is at the very root
> of dependency trees (quilt-native), and ensure that change is correctly discovered when
> building an image.
> 
> 2. make a change in gcc-source recipe, which is somewhat special
> (operates in work-shared), and ensure that gcc-runtime builds track
> that down as well.
> 
> 3. make a change in base_do_configure() definition from base.bbclass,
> which is not recipe-specific, but affects many basic recipes, and ensure that
> is correctly reported as well.
> 
> The test itself actually runs twice:
> - first against a fully populated build directory, where
> the printdiff code is guaranteed to find the correct previous
> stamp that can be compared with in a predictable manner.
> 
> - then in an empty build directory where the printdiff code
> goes to look in the sstate cache, and so the existence of the
> previous signature can be tested, but not the difference with it
> (what the exact difference would be is unpredictable as the
> sstate cache is indeed shared between many builds).
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  .../base-do-configure-modified.bbclass        |   3 +
>  .../gcc-source/gcc-source_%.bbappend          |   2 +
>  .../quilt-native/quilt-native_%.bbappend      |   2 +
>  meta/lib/oeqa/selftest/cases/sstatetests.py   | 106 ++++++++++++++++++
>  4 files changed, 113 insertions(+)
>  create mode 100644 meta-selftest/classes/base-do-configure-modified.bbclass
>  create mode 100644 meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend
>  create mode 100644 meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend
> 
> diff --git a/meta-selftest/classes/base-do-configure-modified.bbclass b/meta-selftest/classes/base-do-configure-modified.bbclass
> new file mode 100644
> index 00000000000..3f96827a428
> --- /dev/null
> +++ b/meta-selftest/classes/base-do-configure-modified.bbclass
> @@ -0,0 +1,3 @@
> +base_do_configure:append () {
> +	echo "this changes base_do_configure() definiton"
> +}
> diff --git a/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend
> new file mode 100644
> index 00000000000..205720982cb
> --- /dev/null
> +++ b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend
> @@ -0,0 +1,2 @@
> +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
> +include test_recipe.inc
> diff --git a/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend
> new file mode 100644
> index 00000000000..205720982cb
> --- /dev/null
> +++ b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend
> @@ -0,0 +1,2 @@
> +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
> +include test_recipe.inc
> diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
> index bdad9088d37..6ef339897bf 100644
> --- a/meta/lib/oeqa/selftest/cases/sstatetests.py
> +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
> @@ -773,3 +773,109 @@ addtask tmptask2 before do_tmptask1
>                  latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:]
>                  bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb)
>                  self.assertEqual(recursecb_count,1)
> +
> +class SStatePrintdiff(SStateBase):
> +    def run_test_printdiff_changerecipe(self, target, change_recipe, change_bbtask, change_content, expected_sametmp_output, expected_difftmp_output):
> +        self.write_config("""
> +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff"
> +""")
> +        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff")
> +        bitbake(target)
> +        bitbake("-S none {}".format(target))
> +        bitbake(change_bbtask)
> +        self.write_recipeinc(change_recipe, change_content)
> +        result_sametmp = bitbake("-S printdiff {}".format(target))
> +
> +        self.write_config("""
> +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2"
> +""")
> +        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2")
> +        result_difftmp = bitbake("-S printdiff {}".format(target))
> +
> +        self.delete_recipeinc(change_recipe)
> +        for item in expected_sametmp_output:
> +            self.assertIn(item, result_sametmp.output)
> +        for item in expected_difftmp_output:
> +            self.assertIn(item, result_difftmp.output)
> +
> +    def run_test_printdiff_changeconfig(self, target, change_content, expected_sametmp_output, expected_difftmp_output):
> +        self.write_config("""
> +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff"
> +""")
> +        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff")
> +        bitbake(target)
> +        bitbake("-S none {}".format(target))
> +        self.append_config(change_content)
> +        result_sametmp = bitbake("-S printdiff {}".format(target))
> +
> +        self.write_config("""
> +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2"
> +""")
> +        self.append_config(change_content)
> +        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2")
> +        result_difftmp = bitbake("-S printdiff {}".format(target))
> +
> +        for item in expected_sametmp_output:
> +            self.assertIn(item, result_sametmp.output)
> +        for item in expected_difftmp_output:
> +            self.assertIn(item, result_difftmp.output)
> +
> +
> +    # Check if printdiff walks the full dependency chain from the image target to where the change is in a specific recipe
> +    def test_image_minimal_vs_quilt(self):
> +        expected_output = ("Task quilt-native:do_install couldn't be used from the cache because:",
> +"We need hash",
> +"most recent matching task was")
> +        expected_sametmp_output = expected_output + ("Variable do_install value changed",'+    echo "this changes the task signature"')
> +        expected_difftmp_output = expected_output
> +
> +        self.run_test_printdiff_changerecipe("core-image-minimal", "quilt-native", "-c do_install quilt-native",
> +"""
> +do_install:append() {
> +    echo "this changes the task signature"
> +}
> +""",
> +expected_sametmp_output, expected_difftmp_output)
> +
> +    # Check if changes to gcc-source (which uses tmp/work-shared) are correctly discovered
> +    def test_gcc_runtime_vs_gcc_source(self):
> +        gcc_source_pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV']
> +
> +        expected_output = ("Task {}:do_preconfigure couldn't be used from the cache because:".format(gcc_source_pn),
> +"We need hash",
> +"most recent matching task was")
> +        expected_sametmp_output = expected_output + ("Variable do_preconfigure value changed",'+    print("this changes the task signature")')
> +        #FIXME: printdiff is supposed to find at least one preconfigure task signature in the sstate cache, but isn't able to
> +        #expected_difftmp_output = expected_output
> +        expected_difftmp_output = ()
> +
> +        self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn),
> +"""
> +python do_preconfigure:append() {
> +    print("this changes the task signature")
> +}
> +""",
> +expected_sametmp_output, expected_difftmp_output)
> +
> +    # Check if changing a really base task definiton is reported against multiple core recipes using it
> +    def test_image_minimal_vs_base_do_configure(self):
> +        expected_output = ("Task zstd-native:do_configure couldn't be used from the cache because:",
> +"Task texinfo-dummy-native:do_configure couldn't be used from the cache because:",
> +"Task ldconfig-native:do_configure couldn't be used from the cache because:",
> +"Task gettext-minimal-native:do_configure couldn't be used from the cache because:",
> +"Task tzcode-native:do_configure couldn't be used from the cache because:",
> +"Task makedevs-native:do_configure couldn't be used from the cache because:",
> +"Task pigz-native:do_configure couldn't be used from the cache because:",
> +"Task update-rc.d-native:do_configure couldn't be used from the cache because:",
> +"Task unzip-native:do_configure couldn't be used from the cache because:",
> +"Task gnu-config-native:do_configure couldn't be used from the cache because:",
> +"We need hash",
> +"most recent matching task was")
> +        expected_sametmp_output = expected_output + ("Variable base_do_configure value changed",'+	echo "this changes base_do_configure() definiton"')
> +        expected_difftmp_output = expected_output
> +
> +        self.run_test_printdiff_changeconfig("core-image-minimal",
> +"""
> +INHERIT += "base-do-configure-modified"
> +""",
> +expected_sametmp_output, expected_difftmp_output)
> -- 
> 2.39.2
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189370): https://lists.openembedded.org/g/openembedded-core/message/189370
> Mute This Topic: https://lists.openembedded.org/mt/102035446/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Alexander Kanavin Oct. 20, 2023, 1:12 p.m. UTC | #2 
On Wed, 18 Oct 2023 at 21:39, Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
> ERROR: Can't find a task we're supposed to have written out? (hash: e79d70b9c2cc72030c1ce822525510699a1eeb1ddf5986271d3217422244366a)?
> ERROR: Can't find a task we're supposed to have written out? (hash: 999250461834bf74545962a8577bc26854a3d08bb2a96140d5efe314e0ef2d63)?

This is proving tricky to reproduce. It happens fairly reliably in
oe-selftest-armhost, but so far never on regular oe-selftest, or in
standalone local build on an arm host where I run only the selftests
that are failing. It seems like one of those dreaded selftest things
that fail only in particular execution order with other tests.

The only idea I have right now is to run specially configured
oe-selftest-armhost which would run only a subset (e.g. just
sstatetests) and see if it can be narrowed down from there.

I'm also tempted to set this aside for now as it has all the signs of
protracted, painful debugging slog, and push the esdk/sstate build
registry subject, so there's something to show for it soon.

Alex
Richard Purdie Oct. 20, 2023, 1:44 p.m. UTC | #3 
On Fri, 2023-10-20 at 15:12 +0200, Alexander Kanavin wrote:
> On Wed, 18 Oct 2023 at 21:39, Alexandre Belloni
> <alexandre.belloni@bootlin.com> wrote:
> > ERROR: Can't find a task we're supposed to have written out? (hash: e79d70b9c2cc72030c1ce822525510699a1eeb1ddf5986271d3217422244366a)?
> > ERROR: Can't find a task we're supposed to have written out? (hash: 999250461834bf74545962a8577bc26854a3d08bb2a96140d5efe314e0ef2d63)?
> 
> This is proving tricky to reproduce. It happens fairly reliably in
> oe-selftest-armhost, but so far never on regular oe-selftest, or in
> standalone local build on an arm host where I run only the selftests
> that are failing. It seems like one of those dreaded selftest things
> that fail only in particular execution order with other tests.
> 
> The only idea I have right now is to run specially configured
> oe-selftest-armhost which would run only a subset (e.g. just
> sstatetests) and see if it can be narrowed down from there.
> 
> I'm also tempted to set this aside for now as it has all the signs of
> protracted, painful debugging slog, and push the esdk/sstate build
> registry subject, so there's something to show for it soon.

I did idly wonder if we need to set the sig handler to not use
hashequiv for these tests? I haven't checked if that would help though,
I'm just making a wild guess.

Cheers,

Richard
Alexander Kanavin Oct. 20, 2023, 2:34 p.m. UTC | #4 
On Fri, 20 Oct 2023 at 15:44, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I did idly wonder if we need to set the sig handler to not use
> hashequiv for these tests? I haven't checked if that would help though,
> I'm just making a wild guess.

I fired it up with BB_HASHSERVE = "auto" set inside the test, let's see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2307

Alex
Richard Purdie Oct. 20, 2023, 2:39 p.m. UTC | #5 
On Fri, 2023-10-20 at 16:34 +0200, Alexander Kanavin wrote:
> On Fri, 20 Oct 2023 at 15:44, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > I did idly wonder if we need to set the sig handler to not use
> > hashequiv for these tests? I haven't checked if that would help though,
> > I'm just making a wild guess.
> 
> I fired it up with BB_HASHSERVE = "auto" set inside the test, let's see:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2307

BB_SIGNATURE_HANDLER = "OEBasicHash"

is the thing I had in mind FWIW but that should be ok.

Cheers,

Richard
Alexander Kanavin Oct. 22, 2023, 11:51 a.m. UTC | #6 
On Fri, 20 Oct 2023 at 16:39, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > > I did idly wonder if we need to set the sig handler to not use
> > > hashequiv for these tests? I haven't checked if that would help though,
> > > I'm just making a wild guess.
> >
> > I fired it up with BB_HASHSERVE = "auto" set inside the test, let's see:
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2307
>
> BB_SIGNATURE_HANDLER = "OEBasicHash"
>
> is the thing I had in mind FWIW but that should be ok.

Seems like that ('auto') had no effect. :-(

Alex
Alexander Kanavin Oct. 22, 2023, 11:52 a.m. UTC | #7 
On Sun, 22 Oct 2023 at 13:52, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
> > > > I did idly wonder if we need to set the sig handler to not use
> > > > hashequiv for these tests? I haven't checked if that would help though,
> > > > I'm just making a wild guess.
> > >
> > > I fired it up with BB_HASHSERVE = "auto" set inside the test, let's see:
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2307
> >
> > BB_SIGNATURE_HANDLER = "OEBasicHash"
> >
> > is the thing I had in mind FWIW but that should be ok.
>
> Seems like that ('auto') had no effect. :-(

Hit send too quickly. Should I try with your suggestion?

Alex
Richard Purdie Oct. 22, 2023, 11:54 a.m. UTC | #8 
On Sun, 2023-10-22 at 13:52 +0200, Alexander Kanavin wrote:
> On Sun, 22 Oct 2023 at 13:52, Alexander Kanavin via
> lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
> wrote:
> > > > > I did idly wonder if we need to set the sig handler to not use
> > > > > hashequiv for these tests? I haven't checked if that would help though,
> > > > > I'm just making a wild guess.
> > > > 
> > > > I fired it up with BB_HASHSERVE = "auto" set inside the test, let's see:
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2307
> > > 
> > > BB_SIGNATURE_HANDLER = "OEBasicHash"
> > > 
> > > is the thing I had in mind FWIW but that should be ok.
> > 
> > Seems like that ('auto') had no effect. :-(
> 
> Hit send too quickly. Should I try with your suggestion?

It is worth a shot, yes. One causes a local hashequiv whereas the other
disables it entirely.

Cheers,

Richard
Alexander Kanavin Oct. 23, 2023, 9:17 a.m. UTC | #9 
On Sun, 22 Oct 2023 at 13:55, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > > Seems like that ('auto') had no effect. :-(
> >
> > Hit send too quickly. Should I try with your suggestion?
>
> It is worth a shot, yes. One causes a local hashequiv whereas the other
> disables it entirely.

Seems like it worked:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2314/steps/15/logs/stdio

Should I fold BB_SIGNATURE_HANDLER = "OEBasicHash" into this commit,
with a FIXME perhaps? Tools should work without special tweaks, so it
needs to be addressed.

Alex
Richard Purdie Oct. 23, 2023, 9:32 a.m. UTC | #10 
On Mon, 2023-10-23 at 11:17 +0200, Alexander Kanavin wrote:
> On Sun, 22 Oct 2023 at 13:55, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > > Seems like that ('auto') had no effect. :-(
> > > 
> > > Hit send too quickly. Should I try with your suggestion?
> > 
> > It is worth a shot, yes. One causes a local hashequiv whereas the other
> > disables it entirely.
> 
> Seems like it worked:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2314/steps/15/logs/stdio

Great!

> Should I fold BB_SIGNATURE_HANDLER = "OEBasicHash" into this commit,
> with a FIXME perhaps? Tools should work without special tweaks, so it
> needs to be addressed.

We should add the workaround into the commit and leave a note, yes.

Hashequiv is 'fun' since it means in a new build of something, the
build can track onto an older build/hash chain depending on whether it
matches it or not.

I've not looked into the test specifically and you're right, we want
people to be able to get diffs reliably regardless of whether it is
enabled or not. Whether the issue is with the test case or the tooling,
I'm not really sure. I suggested it as it was the most likely source of
the issues and definitely worth ruling out if it wasn't.

Note that the armhost case is interesting since the hash equivalence
mapping can be present but the sstate might not be in the native case
if it was built only on x86.

Cheers,

Richard
diff mbox series

Patch

diff --git a/meta-selftest/classes/base-do-configure-modified.bbclass b/meta-selftest/classes/base-do-configure-modified.bbclass
new file mode 100644
index 00000000000..3f96827a428
--- /dev/null
+++ b/meta-selftest/classes/base-do-configure-modified.bbclass
@@ -0,0 +1,3 @@ 
+base_do_configure:append () {
+	echo "this changes base_do_configure() definiton"
+}
diff --git a/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend
new file mode 100644
index 00000000000..205720982cb
--- /dev/null
+++ b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend
@@ -0,0 +1,2 @@ 
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
diff --git a/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend
new file mode 100644
index 00000000000..205720982cb
--- /dev/null
+++ b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend
@@ -0,0 +1,2 @@ 
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
index bdad9088d37..6ef339897bf 100644
--- a/meta/lib/oeqa/selftest/cases/sstatetests.py
+++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
@@ -773,3 +773,109 @@  addtask tmptask2 before do_tmptask1
                 latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:]
                 bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb)
                 self.assertEqual(recursecb_count,1)
+
+class SStatePrintdiff(SStateBase):
+    def run_test_printdiff_changerecipe(self, target, change_recipe, change_bbtask, change_content, expected_sametmp_output, expected_difftmp_output):
+        self.write_config("""
+TMPDIR = "${TOPDIR}/tmp-sstateprintdiff"
+""")
+        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff")
+        bitbake(target)
+        bitbake("-S none {}".format(target))
+        bitbake(change_bbtask)
+        self.write_recipeinc(change_recipe, change_content)
+        result_sametmp = bitbake("-S printdiff {}".format(target))
+
+        self.write_config("""
+TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2"
+""")
+        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2")
+        result_difftmp = bitbake("-S printdiff {}".format(target))
+
+        self.delete_recipeinc(change_recipe)
+        for item in expected_sametmp_output:
+            self.assertIn(item, result_sametmp.output)
+        for item in expected_difftmp_output:
+            self.assertIn(item, result_difftmp.output)
+
+    def run_test_printdiff_changeconfig(self, target, change_content, expected_sametmp_output, expected_difftmp_output):
+        self.write_config("""
+TMPDIR = "${TOPDIR}/tmp-sstateprintdiff"
+""")
+        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff")
+        bitbake(target)
+        bitbake("-S none {}".format(target))
+        self.append_config(change_content)
+        result_sametmp = bitbake("-S printdiff {}".format(target))
+
+        self.write_config("""
+TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2"
+""")
+        self.append_config(change_content)
+        self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2")
+        result_difftmp = bitbake("-S printdiff {}".format(target))
+
+        for item in expected_sametmp_output:
+            self.assertIn(item, result_sametmp.output)
+        for item in expected_difftmp_output:
+            self.assertIn(item, result_difftmp.output)
+
+
+    # Check if printdiff walks the full dependency chain from the image target to where the change is in a specific recipe
+    def test_image_minimal_vs_quilt(self):
+        expected_output = ("Task quilt-native:do_install couldn't be used from the cache because:",
+"We need hash",
+"most recent matching task was")
+        expected_sametmp_output = expected_output + ("Variable do_install value changed",'+    echo "this changes the task signature"')
+        expected_difftmp_output = expected_output
+
+        self.run_test_printdiff_changerecipe("core-image-minimal", "quilt-native", "-c do_install quilt-native",
+"""
+do_install:append() {
+    echo "this changes the task signature"
+}
+""",
+expected_sametmp_output, expected_difftmp_output)
+
+    # Check if changes to gcc-source (which uses tmp/work-shared) are correctly discovered
+    def test_gcc_runtime_vs_gcc_source(self):
+        gcc_source_pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV']
+
+        expected_output = ("Task {}:do_preconfigure couldn't be used from the cache because:".format(gcc_source_pn),
+"We need hash",
+"most recent matching task was")
+        expected_sametmp_output = expected_output + ("Variable do_preconfigure value changed",'+    print("this changes the task signature")')
+        #FIXME: printdiff is supposed to find at least one preconfigure task signature in the sstate cache, but isn't able to
+        #expected_difftmp_output = expected_output
+        expected_difftmp_output = ()
+
+        self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn),
+"""
+python do_preconfigure:append() {
+    print("this changes the task signature")
+}
+""",
+expected_sametmp_output, expected_difftmp_output)
+
+    # Check if changing a really base task definiton is reported against multiple core recipes using it
+    def test_image_minimal_vs_base_do_configure(self):
+        expected_output = ("Task zstd-native:do_configure couldn't be used from the cache because:",
+"Task texinfo-dummy-native:do_configure couldn't be used from the cache because:",
+"Task ldconfig-native:do_configure couldn't be used from the cache because:",
+"Task gettext-minimal-native:do_configure couldn't be used from the cache because:",
+"Task tzcode-native:do_configure couldn't be used from the cache because:",
+"Task makedevs-native:do_configure couldn't be used from the cache because:",
+"Task pigz-native:do_configure couldn't be used from the cache because:",
+"Task update-rc.d-native:do_configure couldn't be used from the cache because:",
+"Task unzip-native:do_configure couldn't be used from the cache because:",
+"Task gnu-config-native:do_configure couldn't be used from the cache because:",
+"We need hash",
+"most recent matching task was")
+        expected_sametmp_output = expected_output + ("Variable base_do_configure value changed",'+	echo "this changes base_do_configure() definiton"')
+        expected_difftmp_output = expected_output
+
+        self.run_test_printdiff_changeconfig("core-image-minimal",
+"""
+INHERIT += "base-do-configure-modified"
+""",
+expected_sametmp_output, expected_difftmp_output)