From patchwork Tue Oct 17 13:30:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 32452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7436CDB474 for ; Tue, 17 Oct 2023 13:30:58 +0000 (UTC) Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) by mx.groups.io with SMTP id smtpd.web11.215017.1697549457393830246 for ; Tue, 17 Oct 2023 06:30:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=fs9AHE24; spf=pass (domain: gmail.com, ip: 209.85.208.54, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-533c5d10dc7so9928785a12.3 for ; Tue, 17 Oct 2023 06:30:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697549456; x=1698154256; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=rhMyz5Qhu/mTqz9tIkgVze52Wu129gn4ot1H+CtUW/w=; b=fs9AHE24XZtvySndAbt+xXcKD63v2N5P3yHQ8WVDIDg5u8ayDg//wzyV1cWZcCfNHE 1rFwt1jKKi4XrTNUc14B4tK3ZwQlv9krTgX3NvHX8tiiwe/wKB3dn6AmiiKa+Cg3pySO aO5LvxJcFlyRKDbUfPS6FYWoBBLTqVJW4Dt0ESegnWi20n5OFi+elJ8EjcQJQWxFFZi4 SKfjff4XqfqR9cXvzGmWks0JVneAhyMix2FfthgBIGOAyzAWBwgp1mUSfsvjX1fry0j8 CMwzKw8GPivipeN0Adq8Mo+dJrUjKt2BIDFWJ0u8hLX816uXTUGrEBWbr1tWsreJqgYP AE1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697549456; x=1698154256; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=rhMyz5Qhu/mTqz9tIkgVze52Wu129gn4ot1H+CtUW/w=; b=o+hvGk1etzKuqoKlTbw0uS0gl5AMgylMm2rqzv7dQ7L+EI+3EmDRB6w2RKFhve7Whj xQa/9Jh00dGGeVfXx0p6Vv6kaYncF4t9ZYLQdb3YGVgUhPKyCSaob1bu0dWam4DJ7v4x gNTBRP6i4wZUzGk59m2ycDRr4Ie5JgfQR0yLUKor/illD+rga3fF/p3nh8z2tDXj2dxL dnZ56HTpzvlHSXjYO09v6PCy8uewgSW7Cwb0A60hfLvpuiz4gNNR1GtLOk02LRFwymna yVQpYQ2bcggGtyuHp1Mdfq2L/k7R43PUVaU4A33RPBAhwazOefYhyCY9ebI9nKDBR++t OysA== X-Gm-Message-State: AOJu0Ywl4bBQksNv9OUukKlE/32iCckDlJTput7C+BrXkMgZ12IoFPuN GKZl52YtTy8McF1fpKEpKV7cS5ymD0A= X-Google-Smtp-Source: AGHT+IG6p2jvScEReZDO99fLcY6wN9RAZeMgBEZQqWKobFic7BJQV6zKujBdQwI03n9XhDJMMbOzjQ== X-Received: by 2002:a05:6402:4410:b0:530:bd6b:7a94 with SMTP id y16-20020a056402441000b00530bd6b7a94mr2183296eda.24.1697549455183; Tue, 17 Oct 2023 06:30:55 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id s22-20020a50ab16000000b0053e589016a7sm956576edc.16.2023.10.17.06.30.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 06:30:54 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 1/3] lib/oe/sstatesig.py: dump locked.sigs.inc only when explicitly asked via -S lockedsigs Date: Tue, 17 Oct 2023 15:30:48 +0200 Message-Id: <20231017133050.147000-1-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 17 Oct 2023 13:30:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189331 This was writing out locked-sigs.inc into cwd with every 'bitbake -S' invocation. When the intent is only to to get task stamps (-S none), or print the difference between them (-S printdiff), the file is unnecessary clutter. A couple of selftests were however relying on this, so they're adjusted to explicitly request the file. eSDK code calls dump_lockedsigs() separately via oe.copy_buildsystem.generate_locked_sigs() and so isn't affected. Signed-off-by: Alexander Kanavin --- meta/lib/oe/sstatesig.py | 7 ++++--- meta/lib/oeqa/selftest/cases/archiver.py | 2 +- meta/lib/oeqa/selftest/cases/signing.py | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index 5bf1697e727..63d792efe9d 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -142,9 +142,10 @@ class SignatureGeneratorOEBasicHashMixIn(object): super().set_taskdata(data[3:]) def dump_sigs(self, dataCache, options): - sigfile = os.getcwd() + "/locked-sigs.inc" - bb.plain("Writing locked sigs to %s" % sigfile) - self.dump_lockedsigs(sigfile) + if 'lockedsigs' in options: + sigfile = os.getcwd() + "/locked-sigs.inc" + bb.plain("Writing locked sigs to %s" % sigfile) + self.dump_lockedsigs(sigfile) return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options) diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py index 3fa59fff510..3cb888c5067 100644 --- a/meta/lib/oeqa/selftest/cases/archiver.py +++ b/meta/lib/oeqa/selftest/cases/archiver.py @@ -141,7 +141,7 @@ class Archiver(OESelftestTestCase): pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV'] # Generate the tasks signatures - bitbake('mc:mc1:%s mc:mc2:%s -c %s -S none' % (pn, pn, task)) + bitbake('mc:mc1:%s mc:mc2:%s -c %s -S lockedsigs' % (pn, pn, task)) # Check the tasks signatures # To be machine agnostic the tasks needs to generate the same signature for each machine diff --git a/meta/lib/oeqa/selftest/cases/signing.py b/meta/lib/oeqa/selftest/cases/signing.py index 322e753ed3b..18cce0ba258 100644 --- a/meta/lib/oeqa/selftest/cases/signing.py +++ b/meta/lib/oeqa/selftest/cases/signing.py @@ -191,7 +191,7 @@ class LockedSignatures(OESelftestTestCase): bitbake(test_recipe) # Generate locked sigs include file - bitbake('-S none %s' % test_recipe) + bitbake('-S lockedsigs %s' % test_recipe) feature = 'require %s\n' % locked_sigs_file feature += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n' From patchwork Tue Oct 17 13:30:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 32454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00F1BCDB482 for ; Tue, 17 Oct 2023 13:30:59 +0000 (UTC) Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by mx.groups.io with SMTP id smtpd.web10.215662.1697549458144574874 for ; Tue, 17 Oct 2023 06:30:58 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=OoL0k3hC; spf=pass (domain: gmail.com, ip: 209.85.218.52, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-99c3d3c3db9so915216266b.3 for ; Tue, 17 Oct 2023 06:30:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697549456; x=1698154256; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=L+orJAU5qKbCFhIw84Np4f5XTDZeBKrdiF19t45j6f0=; b=OoL0k3hCLgtvhDuzVb+nemvFcW8POkEjkn0rZwuU2MX+Hy24b+yJ2NLtjTs33wccWt 2/Xs0yphblENwGNmfYkY5G20M3u6kYSrMo/VpEEh9RBeh5p3UWRpnC0tsboWL7MywUIe tk9k3xWfM262JRpgkoJxF8ofFFjCErBDXmM/nSDFDbPbqM4MZmGXc3SuP8RtNNNgjWLO nC44HRrb+tos8dn4rUnsEr9Vn+oy8s5v1XstnRCe5kak4+DWuEFpeJVoxnmWU8lc/FQA ezCkcoe1ZNHfTUZvPXixGgjYyl30nrlcnPHfLknAr7Z16d3wXdvA8Y3Zo9WrTGSY8O8A ejnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697549456; x=1698154256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L+orJAU5qKbCFhIw84Np4f5XTDZeBKrdiF19t45j6f0=; b=gqen5UK6nQqIjdWgTEktyg7zW97iH7TowXh3mxBgeuzbDTFYcYgpd4upewEqXJVvI+ VN6vHri3a9b9hmSOHboGkdfRnGG4zKLqpomNl+uHEBVWt3HhCZBGqPLbfnfTHwjPgcJo ez2llzDB67CDqiu0rGBCmH61BdNbxOMuUAJx+Gc2fR55EqhzBUj81ZT2QOdOmzBGzsd6 jVg2m2eWmbrKFT0kWPdYVkjUPYeJw8eFkRsET4bOlY6HXc3s0P0FINpnihd+74279V3B Y9vdKEnVz1ZsE1PVJU5dgMxb1+O5reWMUBCPfiJKOLgXF++O30Ejohf9ux5mscN+LwB7 iZgA== X-Gm-Message-State: AOJu0YwoXjOAogeo+GORbzthWtHvz5JyksMWBpXl3ZLQMu8PpOrpoL9/ oUbslizQJJf6S793sMGDJNavLWUNhEw= X-Google-Smtp-Source: AGHT+IG2AU60cJ74bAk3uOL9jEOJjq23W/luffSq8UpNQbEnQRAwPuBgfw6PrOFI6/HgCqjaBRgvUw== X-Received: by 2002:a17:907:d8a:b0:9a1:f10d:9746 with SMTP id go10-20020a1709070d8a00b009a1f10d9746mr1724851ejc.20.1697549456129; Tue, 17 Oct 2023 06:30:56 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id s22-20020a50ab16000000b0053e589016a7sm956576edc.16.2023.10.17.06.30.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 06:30:55 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 2/3] selftest/sstatetests: add tests for 'bitbake -S printdiff' Date: Tue, 17 Oct 2023 15:30:49 +0200 Message-Id: <20231017133050.147000-2-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231017133050.147000-1-alex@linutronix.de> References: <20231017133050.147000-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 17 Oct 2023 13:30:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189332 'bitbake -S printdiff' is a useful diagnostic facility for finding out why sstate is not being reused, but until now it had no tests that would ensure it works. This commit adds three basic scenarios: 1. make a change in a really basic, common recipe that is at the very root of dependency trees (quilt-native), and ensure that change is correctly discovered when building an image. 2. make a change in gcc-source recipe, which is somewhat special (operates in work-shared), and ensure that gcc-runtime builds track that down as well. 3. make a change in base_do_configure() definition from base.bbclass, which is not recipe-specific, but affects many basic recipes, and ensure that is correctly reported as well. The test itself actually runs twice: - first against a fully populated build directory, where the printdiff code is guaranteed to find the correct previous stamp that can be compared with in a predictable manner. - then in an empty build directory where the printdiff code goes to look in the sstate cache, and so the existence of the previous signature can be tested, but not the difference with it (what the exact difference would be is unpredictable as the sstate cache is indeed shared between many builds). Signed-off-by: Alexander Kanavin --- .../base-do-configure-modified.bbclass | 3 + .../gcc-source/gcc-source_%.bbappend | 2 + .../quilt-native/quilt-native_%.bbappend | 2 + meta/lib/oeqa/selftest/cases/sstatetests.py | 104 ++++++++++++++++++ 4 files changed, 111 insertions(+) create mode 100644 meta-selftest/classes/base-do-configure-modified.bbclass create mode 100644 meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend create mode 100644 meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend diff --git a/meta-selftest/classes/base-do-configure-modified.bbclass b/meta-selftest/classes/base-do-configure-modified.bbclass new file mode 100644 index 00000000000..3f96827a428 --- /dev/null +++ b/meta-selftest/classes/base-do-configure-modified.bbclass @@ -0,0 +1,3 @@ +base_do_configure:append () { + echo "this changes base_do_configure() definiton" +} diff --git a/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend new file mode 100644 index 00000000000..205720982cb --- /dev/null +++ b/meta-selftest/recipes-test/gcc-source/gcc-source_%.bbappend @@ -0,0 +1,2 @@ +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests. +include test_recipe.inc diff --git a/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend new file mode 100644 index 00000000000..205720982cb --- /dev/null +++ b/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend @@ -0,0 +1,2 @@ +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests. +include test_recipe.inc diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index bdad9088d37..29e31f2ef9f 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -773,3 +773,107 @@ addtask tmptask2 before do_tmptask1 latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:] bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) self.assertEqual(recursecb_count,1) + +class SStatePrintdiff(SStateBase): + def run_test_printdiff_changerecipe(self, target, change_recipe, change_bbtask, change_content, expected_sametmp_output, expected_difftmp_output): + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff" +""") + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff") + bitbake(target) + bitbake("-S none {}".format(target)) + bitbake(change_bbtask) + self.write_recipeinc(change_recipe, change_content) + result_sametmp = bitbake("-S printdiff {}".format(target)) + + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2" +""") + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2") + result_difftmp = bitbake("-S printdiff {}".format(target)) + + self.delete_recipeinc(change_recipe) + for item in expected_sametmp_output: + self.assertIn(item, result_sametmp.output) + for item in expected_difftmp_output: + self.assertIn(item, result_difftmp.output) + + def run_test_printdiff_changeconfig(self, target, change_content, expected_sametmp_output, expected_difftmp_output): + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff" +""") + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff") + bitbake(target) + bitbake("-S none {}".format(target)) + self.append_config(change_content) + result_sametmp = bitbake("-S printdiff {}".format(target)) + + self.write_config(""" +TMPDIR = "${TOPDIR}/tmp-sstateprintdiff-2" +""") + self.append_config(change_content) + self.track_for_cleanup(self.topdir + "/tmp-sstateprintdiff-2") + result_difftmp = bitbake("-S printdiff {}".format(target)) + + for item in expected_sametmp_output: + self.assertIn(item, result_sametmp.output) + for item in expected_difftmp_output: + self.assertIn(item, result_difftmp.output) + + + # Check if printdiff walks the full dependency chain from the image target to where the change is in a specific recipe + def test_image_minimal_vs_quilt(self): + expected_output = ("Task quilt-native:do_install couldn't be used from the cache because:", +"We need hash", +"most recent matching task was") + expected_sametmp_output = expected_output + ("Variable do_install value changed",'+ echo "this changes the task signature"') + expected_difftmp_output = expected_output + + self.run_test_printdiff_changerecipe("core-image-minimal", "quilt-native", "-c do_install quilt-native", +""" +do_install:append() { + echo "this changes the task signature" +} +""", +expected_sametmp_output, expected_difftmp_output) + + # Check if changes to gcc-source (which uses tmp/work-shared) are correctly discovered + def test_gcc_runtime_vs_gcc_source(self): + expected_output = ("Task gcc-source-13.2.0:do_preconfigure couldn't be used from the cache because:", +"We need hash", +"most recent matching task was") + expected_sametmp_output = expected_output + ("Variable do_preconfigure value changed",'+ print("this changes the task signature")') + #FIXME: printdiff is supposed to find at least one preconfigure task signature in the sstate cache, but isn't able to + #expected_difftmp_output = expected_output + expected_difftmp_output = () + + self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure gcc-source-13.2.0", +""" +python do_preconfigure:append() { + print("this changes the task signature") +} +""", +expected_sametmp_output, expected_difftmp_output) + + # Check if changing a really base task definiton is reported against multiple core recipes using it + def test_image_minimal_vs_base_do_configure(self): + expected_output = ("Task zstd-native:do_configure couldn't be used from the cache because:", +"Task texinfo-dummy-native:do_configure couldn't be used from the cache because:", +"Task ldconfig-native:do_configure couldn't be used from the cache because:", +"Task gettext-minimal-native:do_configure couldn't be used from the cache because:", +"Task tzcode-native:do_configure couldn't be used from the cache because:", +"Task makedevs-native:do_configure couldn't be used from the cache because:", +"Task pigz-native:do_configure couldn't be used from the cache because:", +"Task update-rc.d-native:do_configure couldn't be used from the cache because:", +"Task unzip-native:do_configure couldn't be used from the cache because:", +"Task gnu-config-native:do_configure couldn't be used from the cache because:", +"We need hash", +"most recent matching task was") + expected_sametmp_output = expected_output + ("Variable base_do_configure value changed",'+ echo "this changes base_do_configure() definiton"') + expected_difftmp_output = expected_output + + self.run_test_printdiff_changeconfig("core-image-minimal", +""" +INHERIT += "base-do-configure-modified" +""", +expected_sametmp_output, expected_difftmp_output) From patchwork Tue Oct 17 13:30:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 32453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7479C46CA1 for ; Tue, 17 Oct 2023 13:30:58 +0000 (UTC) Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) by mx.groups.io with SMTP id smtpd.web11.215019.1697549458410150411 for ; Tue, 17 Oct 2023 06:30:58 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YLCQywiR; spf=pass (domain: gmail.com, ip: 209.85.208.43, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f43.google.com with SMTP id 4fb4d7f45d1cf-53b32dca0bfso11856570a12.0 for ; Tue, 17 Oct 2023 06:30:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697549457; x=1698154257; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AcKAnDyVRCIBxuqzcaZImLHZq8MMpRqvw3lyrSKNkhM=; b=YLCQywiRV+0RRJ5zOXneIlhbuXgvGGbPXzvhXryHDUVtsCz2h+8sqwz4rydI4tOiMw WvXmjv6OtxRtsCO66CfjKHyvdk5osd9PRITEPSbw7/ppQ2RDAb5a7IGdZCa7LjkUFnu2 Wr0SL2TNWhVdnhuePRRBh5Q3bgWkJmvBEKnUzuwoGOS13D2WGDSEETZO3+MZ7+cvry2L HYp/zpEGcYIaKAfuOBiQOyLUy/LLe9hh93U01yczRI5IYGuFJzbkgHlSWaGbUojXulpf O4k+rnApZDKKJLhNeVrfosW/nvoX+lbGRTdUahwFNC0TlWeVin730lHR8YALW2KP4wQC YSLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697549457; x=1698154257; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AcKAnDyVRCIBxuqzcaZImLHZq8MMpRqvw3lyrSKNkhM=; b=R8BnuDw0O8ac9KMr9zFCeuIF5/4Nz7O0X6+wWUsT0mU6AFJhmGPXKmpKDcHk/npU53 tQK+Mv4rYz1o+Ns51bsGJvr79pixFVSF8f33ShaXvgQmkq9WRLUdrcY6/ETOeh5AKGAn JGrro5YehgzbvhIjk9lVGNrzR4mPpcZ7//KUBNsoxYTT0S+dc56jvRdxpWHzzaZcOWNU YNZ+IFbhDMlkiXs6E5Hpf5RMFxPwY9/Ej1UZdiIPE9YKKuJGYuIl+5ZoBDiIP8gPT2qV e22gHynw+8RINy1hz9KcRQSUdozsx4VtUdWeeK5jI/EQyJgRK/bG4YqbEZ7E1Z+gzH+X JT1g== X-Gm-Message-State: AOJu0YwTlMPi3KfxSHuYBCbcPpGs6g6M+cuEKniT8kqbmEQIc5hgfUVm PH3E4pI0THXZJy4ylUDWYXUpt08ovhk= X-Google-Smtp-Source: AGHT+IHv9CrkO3ZPWAQjr1vNY9aO1UpOqM72eoiaErEW1r3ZFkVMinB7QhReKShU0DCRug0FQSiumQ== X-Received: by 2002:a50:9996:0:b0:523:2e23:a0bf with SMTP id m22-20020a509996000000b005232e23a0bfmr1710197edb.11.1697549456702; Tue, 17 Oct 2023 06:30:56 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id s22-20020a50ab16000000b0053e589016a7sm956576edc.16.2023.10.17.06.30.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 06:30:56 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 3/3] scripts/bitbake-whatchanged: remove Date: Tue, 17 Oct 2023 15:30:50 +0200 Message-Id: <20231017133050.147000-3-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231017133050.147000-1-alex@linutronix.de> References: <20231017133050.147000-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 17 Oct 2023 13:30:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189333 This is not documented or tested, and indeed hasn't been producing useful reports for some time. The script works by redirecting STAMPS_DIR into a separate location, then running bitbake -S none, then comparing the two sets of stamp filenames with regexes: Match the stamp's filename group(1): PE_PV (may no PE) group(2): PR group(3): TASK group(4): HASH stamp_re = re.compile("(?P.*)-(?Pr\d+)\.(?Pdo_\w+)\.(?P[^\.]*)") Then there's some code that finds out what changed in the above between the two sets. Messing about with STAMPS_DIR like that isn't supported, and will either do nothing, or remove the original stamps. Also stamp filenames aren't really a 'public API'. For finding out the changes between two builds, 'bitbake -s printdiff' is a supported and tested option. It may be a bit too verbose, but that can be more easily fixed than rewriting bitbake-whatchanged into a working state. Signed-off-by: Alexander Kanavin --- scripts/bitbake-whatchanged | 318 ------------------------------------ 1 file changed, 318 deletions(-) delete mode 100755 scripts/bitbake-whatchanged diff --git a/scripts/bitbake-whatchanged b/scripts/bitbake-whatchanged deleted file mode 100755 index cdb730dbdb9..00000000000 --- a/scripts/bitbake-whatchanged +++ /dev/null @@ -1,318 +0,0 @@ -#!/usr/bin/env python3 -# -# Copyright (c) 2013 Wind River Systems, Inc. -# -# SPDX-License-Identifier: GPL-2.0-only -# - -import os -import sys -import getopt -import shutil -import re -import warnings -import subprocess -import argparse - -scripts_path = os.path.abspath(os.path.dirname(os.path.abspath(sys.argv[0]))) -lib_path = scripts_path + '/lib' -sys.path = sys.path + [lib_path] - -import scriptpath - -# Figure out where is the bitbake/lib/bb since we need bb.siggen and bb.process -bitbakepath = scriptpath.add_bitbake_lib_path() -if not bitbakepath: - sys.stderr.write("Unable to find bitbake by searching parent directory of this script or PATH\n") - sys.exit(1) -scriptpath.add_oe_lib_path() -import argparse_oe - -import bb.siggen -import bb.process - -# Match the stamp's filename -# group(1): PE_PV (may no PE) -# group(2): PR -# group(3): TASK -# group(4): HASH -stamp_re = re.compile("(?P.*)-(?Pr\d+)\.(?Pdo_\w+)\.(?P[^\.]*)") -sigdata_re = re.compile(".*\.sigdata\..*") - -def gen_dict(stamps): - """ - Generate the dict from the stamps dir. - The output dict format is: - {fake_f: {pn: PN, pv: PV, pr: PR, task: TASK, path: PATH}} - Where: - fake_f: pv + task + hash - path: the path to the stamp file - """ - # The member of the sub dict (A "path" will be appended below) - sub_mem = ("pv", "pr", "task") - d = {} - for dirpath, _, files in os.walk(stamps): - for f in files: - # The "bitbake -S" would generate ".sigdata", but no "_setscene". - fake_f = re.sub('_setscene.', '.', f) - fake_f = re.sub('.sigdata', '', fake_f) - subdict = {} - tmp = stamp_re.match(fake_f) - if tmp: - for i in sub_mem: - subdict[i] = tmp.group(i) - if len(subdict) != 0: - pn = os.path.basename(dirpath) - subdict['pn'] = pn - # The path will be used by os.stat() and bb.siggen - subdict['path'] = dirpath + "/" + f - fake_f = tmp.group('pv') + tmp.group('task') + tmp.group('hash') - d[fake_f] = subdict - return d - -# Re-construct the dict -def recon_dict(dict_in): - """ - The output dict format is: - {pn_task: {pv: PV, pr: PR, path: PATH}} - """ - dict_out = {} - for k in dict_in.keys(): - subdict = {} - # The key - pn_task = "%s_%s" % (dict_in.get(k).get('pn'), dict_in.get(k).get('task')) - # If more than one stamps are found, use the latest one. - if pn_task in dict_out: - full_path_pre = dict_out.get(pn_task).get('path') - full_path_cur = dict_in.get(k).get('path') - if os.stat(full_path_pre).st_mtime > os.stat(full_path_cur).st_mtime: - continue - subdict['pv'] = dict_in.get(k).get('pv') - subdict['pr'] = dict_in.get(k).get('pr') - subdict['path'] = dict_in.get(k).get('path') - dict_out[pn_task] = subdict - - return dict_out - -def split_pntask(s): - """ - Split the pn_task in to (pn, task) and return it - """ - tmp = re.match("(.*)_(do_.*)", s) - return (tmp.group(1), tmp.group(2)) - - -def print_added(d_new = None, d_old = None): - """ - Print the newly added tasks - """ - added = {} - for k in list(d_new.keys()): - if k not in d_old: - # Add the new one to added dict, and remove it from - # d_new, so the remaining ones are the changed ones - added[k] = d_new.get(k) - del(d_new[k]) - - if not added: - return 0 - - # Format the output, the dict format is: - # {pn: task1, task2 ...} - added_format = {} - counter = 0 - for k in added.keys(): - pn, task = split_pntask(k) - if pn in added_format: - # Append the value - added_format[pn] = "%s %s" % (added_format.get(pn), task) - else: - added_format[pn] = task - counter += 1 - print("=== Newly added tasks: (%s tasks)" % counter) - for k in added_format.keys(): - print(" %s: %s" % (k, added_format.get(k))) - - return counter - -def print_vrchanged(d_new = None, d_old = None, vr = None): - """ - Print the pv or pr changed tasks. - The arg "vr" is "pv" or "pr" - """ - pvchanged = {} - counter = 0 - for k in list(d_new.keys()): - if d_new.get(k).get(vr) != d_old.get(k).get(vr): - counter += 1 - pn, task = split_pntask(k) - if pn not in pvchanged: - # Format the output, we only print pn (no task) since - # all the tasks would be changed when pn or pr changed, - # the dict format is: - # {pn: pv/pr_old -> pv/pr_new} - pvchanged[pn] = "%s -> %s" % (d_old.get(k).get(vr), d_new.get(k).get(vr)) - del(d_new[k]) - - if not pvchanged: - return 0 - - print("\n=== %s changed: (%s tasks)" % (vr.upper(), counter)) - for k in pvchanged.keys(): - print(" %s: %s" % (k, pvchanged.get(k))) - - return counter - -def print_depchanged(d_new = None, d_old = None, verbose = False): - """ - Print the dependency changes - """ - depchanged = {} - counter = 0 - for k in d_new.keys(): - counter += 1 - pn, task = split_pntask(k) - if (verbose): - full_path_old = d_old.get(k).get("path") - full_path_new = d_new.get(k).get("path") - # No counter since it is not ready here - if sigdata_re.match(full_path_old) and sigdata_re.match(full_path_new): - output = bb.siggen.compare_sigfiles(full_path_old, full_path_new) - if output: - print("\n=== The verbose changes of %s.%s:" % (pn, task)) - print('\n'.join(output)) - else: - # Format the output, the format is: - # {pn: task1, task2, ...} - if pn in depchanged: - depchanged[pn] = "%s %s" % (depchanged.get(pn), task) - else: - depchanged[pn] = task - - if len(depchanged) > 0: - print("\n=== Dependencies changed: (%s tasks)" % counter) - for k in depchanged.keys(): - print(" %s: %s" % (k, depchanged[k])) - - return counter - - -def main(): - """ - Print what will be done between the current and last builds: - 1) Run "STAMPS_DIR= bitbake -S recipe" to re-generate the stamps - 2) Figure out what are newly added and changed, can't figure out - what are removed since we can't know the previous stamps - clearly, for example, if there are several builds, we can't know - which stamps the last build has used exactly. - 3) Use bb.siggen.compare_sigfiles to diff the old and new stamps - """ - - parser = argparse_oe.ArgumentParser(usage = """%(prog)s [options] [package ...] -print what will be done between the current and last builds, for example: - - $ bitbake core-image-sato - # Edit the recipes - $ bitbake-whatchanged core-image-sato - -The changes will be printed. - -Note: - The amount of tasks is not accurate when the task is "do_build" since - it usually depends on other tasks. - The "nostamp" task is not included. -""" -) - parser.add_argument("recipe", help="recipe to check") - parser.add_argument("-v", "--verbose", help = "print the verbose changes", action = "store_true") - args = parser.parse_args() - - # Get the STAMPS_DIR - print("Figuring out the STAMPS_DIR ...") - cmdline = "bitbake -e | sed -ne 's/^STAMPS_DIR=\"\(.*\)\"/\\1/p'" - try: - stampsdir, err = bb.process.run(cmdline) - except: - raise - if not stampsdir: - print("ERROR: No STAMPS_DIR found for '%s'" % args.recipe, file=sys.stderr) - return 2 - stampsdir = stampsdir.rstrip("\n") - if not os.path.isdir(stampsdir): - print("ERROR: stamps directory \"%s\" not found!" % stampsdir, file=sys.stderr) - return 2 - - # The new stamps dir - new_stampsdir = stampsdir + ".bbs" - if os.path.exists(new_stampsdir): - print("ERROR: %s already exists!" % new_stampsdir, file=sys.stderr) - return 2 - - try: - # Generate the new stamps dir - print("Generating the new stamps ... (need several minutes)") - cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir, args.recipe) - # FIXME - # The "bitbake -S" may fail, not fatal error, the stamps will still - # be generated, this might be a bug of "bitbake -S". - try: - bb.process.run(cmdline) - except Exception as exc: - print(exc) - - # The dict for the new and old stamps. - old_dict = gen_dict(stampsdir) - new_dict = gen_dict(new_stampsdir) - - # Remove the same one from both stamps. - cnt_unchanged = 0 - for k in list(new_dict.keys()): - if k in old_dict: - cnt_unchanged += 1 - del(new_dict[k]) - del(old_dict[k]) - - # Re-construct the dict to easily find out what is added or changed. - # The dict format is: - # {pn_task: {pv: PV, pr: PR, path: PATH}} - new_recon = recon_dict(new_dict) - old_recon = recon_dict(old_dict) - - del new_dict - del old_dict - - # Figure out what are changed, the new_recon would be changed - # by the print_xxx function. - # Newly added - cnt_added = print_added(new_recon, old_recon) - - # PV (including PE) and PR changed - # Let the bb.siggen handle them if verbose - cnt_rv = {} - if not args.verbose: - for i in ('pv', 'pr'): - cnt_rv[i] = print_vrchanged(new_recon, old_recon, i) - - # Dependencies changed (use bitbake-diffsigs) - cnt_dep = print_depchanged(new_recon, old_recon, args.verbose) - - total_changed = cnt_added + (cnt_rv.get('pv') or 0) + (cnt_rv.get('pr') or 0) + cnt_dep - - print("\n=== Summary: (%s changed, %s unchanged)" % (total_changed, cnt_unchanged)) - if args.verbose: - print("Newly added: %s\nDependencies changed: %s\n" % \ - (cnt_added, cnt_dep)) - else: - print("Newly added: %s\nPV changed: %s\nPR changed: %s\nDependencies changed: %s\n" % \ - (cnt_added, cnt_rv.get('pv') or 0, cnt_rv.get('pr') or 0, cnt_dep)) - except: - print("ERROR occurred!") - raise - finally: - # Remove the newly generated stamps dir - if os.path.exists(new_stampsdir): - print("Removing the newly generated stamps dir ...") - shutil.rmtree(new_stampsdir) - -if __name__ == "__main__": - sys.exit(main())