From patchwork Thu Dec 14 13:45:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A71AC46CA2 for ; Thu, 14 Dec 2023 13:46:08 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web11.22349.1702561567308503693 for ; Thu, 14 Dec 2023 05:46:07 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Bir1bOuw; spf=pass (domain: gmail.com, ip: 209.85.128.54, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-40c339d2b88so66467065e9.3 for ; Thu, 14 Dec 2023 05:46:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561566; x=1703166366; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=XJR42CGzTFw2B2GH1aFPyryGB0RLjblm/6vFcLY+p9s=; b=Bir1bOuwxnH5yenfU95FeIQt6QrlR1LQ14rzdcv9JveQnhJB/gQagYPU0QqvWpnn7v SOB2NalM+ahVElFyxwiUJhj64siULc844g3eqqWdVKCwBe9AwemSU2Cuw452mncz0Ftc P8ABcLKIHqp04Hucrjd0Q+jo6YxqODJ3IXI+ldNoTijEMmyCgiquY0/MVPWI2u0f3xrQ n6aF46X2qNUfAREsH0BUxCRATDtn6A5vvIgP3FVHNdb9uuvOtUGsqHxndNSXU7cCmOV3 J/n7JV+L/lHPlcs7lLcNlHvYh1V2NP0AVzXxyNaWy4a04qn5RcM1NCzenb3oMrrBUnsl p0uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561566; x=1703166366; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XJR42CGzTFw2B2GH1aFPyryGB0RLjblm/6vFcLY+p9s=; b=byXGvWxN3y1XbD1BU3LfM5e/fhahnq3lrid7Fc3ify8bd+24gKo84cyiekP7ZE3UQX yLwt5n7yTgtkLIwLR9vEH1IDy1ZFOIxgLeItxOee6aB0UZN0She3UXkKiTYSELbn9rd5 PZldnu/vl5IBixeDiwgCdOoOr77yzZvyv9BKcuMWqkffZEIa4+SSeggRL8O4Yu8vvCtc 7J95yyq4A/Za98vZ/JNaTlaAxWQnhoED1eH3splOCoy5uW9RcwCzcfM9tnkVU/UFX90N MRWVP68S91DMyaXwE1Y407CIHBOWoIU+QonWdlBFozyd72QjFq+AykigKOl/9l+0puC5 69YQ== X-Gm-Message-State: AOJu0YzpXgoyoENXXIFSRQ7pNC4rGqOAQWP2eHzZF0MhAGtrS2Xnn4Oq VCgMhgkXqyqYuAqx+ticsFWPg8DfBCM= X-Google-Smtp-Source: AGHT+IFeDGdCQgbKG07QkR8Xog+rvHVIepX8eQnBngvTyT9bjbvzFiIgup8XnDxIsq/BhP9rbhF5BA== X-Received: by 2002:a05:600c:3b06:b0:40b:2a15:9b30 with SMTP id m6-20020a05600c3b0600b0040b2a159b30mr4927963wms.1.1702561565477; Thu, 14 Dec 2023 05:46:05 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:05 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 1/9] oeqa/selftest/sstatetests: re-work CDN tests, add local cache tests Date: Thu, 14 Dec 2023 14:45:20 +0100 Message-Id: <20231214134528.1973602-1-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192371 With the rework of printdiff, it is not longer useful for checking absence of sstate objects in a remote http cache, as it would only report the top level missing signatures, and leave the recursive investigation to diffsigs (which relies on ability to list cache files - not available over http). The CDN check can be performed by simply running 'bitbake -DD -n' which is very verbose, but neverthless reports the amount of missing sstate objects and what they are in a way that can be programmatically extracted and checked (as suggested by RP). This also adds local sstate tests, as they can be useful to determine whether the missing cdn objects were never created or erroneously cleaned up, or if they were created but didn't propagate to cdn. [YOCTO #15303] Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/sstatetests.py | 99 ++++++++++++++------- 1 file changed, 67 insertions(+), 32 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 795cd5bd85f..f52ca77c09c 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -884,47 +884,82 @@ expected_sametmp_output, expected_difftmp_output) @OETestTag("yocto-mirrors") class SStateMirrors(SStateBase): - def check_bb_output(self, output, exceptions): - in_tasks = False - missing_objects = [] - checked_urls = [] - for l in output.splitlines(): - if "Testing URL" in l: - checked_urls.append(l.split()[3]) - if "The differences between the current build and any cached tasks start at the following tasks" in l: - in_tasks = True - continue - if "Writing task signature files" in l: - in_tasks = False - continue - if in_tasks: - recipe_task = l.split("/")[-1] - recipe, task = recipe_task.split(":") - for e in exceptions: - if e[0] in recipe and task == e[1]: + def check_bb_output(self, output, exceptions, check_cdn): + def is_exception(object, exceptions): + for e in exceptions: + if re.search(e, object): + return True + return False + + output_l = output.splitlines() + for l in output_l: + if l.startswith("Sstate summary"): + for idx, item in enumerate(l.split()): + if item == 'Missed': + missing_objects = int(l.split()[idx+1]) break else: - missing_objects.append(recipe_task) - self.assertTrue(len(missing_objects) == 0, "URLs checked:\n{}\nMissing objects in the cache:\n{}".format("\n".join(checked_urls), "\n".join(missing_objects))) - - def run_test_cdn_mirror(self, machine, targets, exceptions): - exceptions = exceptions + [[t, "do_deploy_source_date_epoch"] for t in targets.split()] - exceptions = exceptions + [[t, "do_image_qa"] for t in targets.split()] - self.config_sstate(True) - self.append_config(""" + self.fail("Did not find missing objects amount in sstate summary: {}".format(l)) + break + else: + self.fail("Did not find 'Sstate summary' line in bitbake output") + + failed_urls = [] + for l in output_l: + if "SState: Unsuccessful fetch test for" in l and check_cdn: + missing_object = l.split()[6] + elif "SState: Looked for but didn't find file" in l and not check_cdn: + missing_object = l.split()[8] + else: + missing_object = None + if missing_object: + if not is_exception(missing_object, exceptions): + failed_urls.append(missing_object) + else: + missing_objects -= 1 + + self.assertEqual(len(failed_urls), missing_objects, "Amount of reported missing objects does not match failed URLs: {}\nFailed URLs:\n{}".format(missing_objects, "\n".join(failed_urls))) + self.assertEqual(len(failed_urls), 0, "Missing objects in the cache:\n{}".format("\n".join(failed_urls))) + + def run_test(self, machine, targets, exceptions, check_cdn = True): + # sstate is checked for existence of these, but they never get written out to begin with + exceptions += ["{}.*image_qa".format(t) for t in targets.split()] + exceptions += ["{}.*deploy_source_date_epoch".format(t) for t in targets.split()] + exceptions += ["{}.*image_complete".format(t) for t in targets.split()] + exceptions += ["linux-yocto.*shared_workdir"] + # these get influnced by IMAGE_FSTYPES tweaks in yocto-autobuilder-helper's config.json (on x86-64) + # additionally, they depend on noexec (thus, absent stamps) package, install, etc. image tasks, + # which makes tracing other changes difficult + exceptions += ["{}.*create_spdx".format(t) for t in targets.split()] + exceptions += ["{}.*create_runtime_spdx".format(t) for t in targets.split()] + + if check_cdn: + self.config_sstate(True) + self.append_config(""" MACHINE = "{}" BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687" SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH" """.format(machine)) - result = bitbake("-D -S printdiff {}".format(targets)) - self.check_bb_output(result.output, exceptions) + else: + self.append_config(""" +MACHINE = "{}" +""".format(machine)) + result = bitbake("-DD -n {}".format(targets)) + bitbake("-S none {}".format(targets)) + self.check_bb_output(result.output, exceptions, check_cdn) def test_cdn_mirror_qemux86_64(self): - # Example: - # exceptions = [ ["packagegroup-core-sdk","do_package"] ] exceptions = [] - self.run_test_cdn_mirror("qemux86-64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions) + self.run_test("qemux86-64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions) def test_cdn_mirror_qemuarm64(self): exceptions = [] - self.run_test_cdn_mirror("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions) + self.run_test("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions) + + def test_local_cache_qemux86_64(self): + exceptions = [] + self.run_test("qemux86-64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions, check_cdn = False) + + def test_local_cache_qemuarm64(self): + exceptions = [] + self.run_test("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions, check_cdn = False) From patchwork Thu Dec 14 13:45:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A676C4167D for ; Thu, 14 Dec 2023 13:46:08 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web10.22364.1702561567667404765 for ; Thu, 14 Dec 2023 05:46:07 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=amiWcX31; spf=pass (domain: gmail.com, ip: 209.85.221.53, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-336447f240cso769938f8f.3 for ; Thu, 14 Dec 2023 05:46:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561566; x=1703166366; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DfrRFhVZKOUjru69f1VLa+i4i9GFlZAaEcvcdFYqslc=; b=amiWcX31h+0+vXxvrU4rL8p6vUi4qn3HjqeR9e6OkIOHiYXs2I3eK85wxWUWydilIl m/V5v80ViuOGoSkTrkEFHs4kgnum2PwwURvUGtSA/4dwiYBrXpQgZpZG559ovWnRc+2W rsOfbkID2W+qiS3zmnmCcTCoWrt0Zp3AHM9izm3bneLwFsQ2nyWE5A6qFXMRLzxmATtN 70Y7LnOa1y8k0JvoO598gWGdTRYaiZUtVzte+8diR9W9ZZtNZ88G9+waTTHcTv9CgkpU DGRHCCwt33FaNZWLUv+qNZS+L07BJ8C+TNM3FAtv/3Eg3/6aGF5Au/L9cXVI3FIYI/4/ XzNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561566; x=1703166366; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DfrRFhVZKOUjru69f1VLa+i4i9GFlZAaEcvcdFYqslc=; b=Uo8KeMEbnUfFcyffEXIj3qxX+8IbjAqsIVfb9taAwYUXZfGrNx47B5BuVRcSiGgQUa J0ANJd6oc9WWUSPqr1TGu9n+cw31vDOUGih9ba65E6TeoOrD/V2bgXQ3LrSc/s15Z0AC vaeJaq+tIUO3A8WLfqRdq1gULOUaarrWE55Zg8pFc+ucD6FhiTUReKbLHBPHiN6C3yLG zVLcw4CRveFDVF1SQpZs8wPHM2Qbq4xdwpWC0S15Vl+nA9MyHTbt0BOAMnXwGsiCPkIL 93tL4AY3dYEKSO/k6FR+2FOcczbRIoksXKlIBCOyCHjejKAvZ7GpziGmGGrj95PBvpjZ /ryw== X-Gm-Message-State: AOJu0Yzqv1C2FHvDVdOvlAP6/rSoQqtpazFH2pxeC/UEcNobbrMMcIW/ 4PfssdPHrXS0g3ZMMFblsgUVz1Opjfg= X-Google-Smtp-Source: AGHT+IHJ6n0KLNtR2ZSWdWQb4+LLRh6FcS1FEmgxw7vfeAeFc1sY/kRuIvdIqottJKoddKdnZ6YP+Q== X-Received: by 2002:a05:6000:b03:b0:336:4258:8dae with SMTP id dj3-20020a0560000b0300b0033642588daemr1161301wrb.21.1702561565930; Thu, 14 Dec 2023 05:46:05 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:05 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 2/9] gnu-config: delete do_compile task Date: Thu, 14 Dec 2023 14:45:21 +0100 Message-Id: <20231214134528.1973602-2-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192372 noexec flag has an unfortunate side effect of not writing out the .siginfo file into sstate (because that is done in sstate bbclass from a task-completed event handler). In the absence of the siginfo file, diffsigs code is unable to trace back the change in task signatures if the change is really basic and affects tasks coming ahead of gnu-config-native:do_compile. Signed-off-by: Alexander Kanavin --- meta/recipes-devtools/gnu-config/gnu-config_git.bb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta/recipes-devtools/gnu-config/gnu-config_git.bb b/meta/recipes-devtools/gnu-config/gnu-config_git.bb index 718f798a00e..7c5a34db4e3 100644 --- a/meta/recipes-devtools/gnu-config/gnu-config_git.bb +++ b/meta/recipes-devtools/gnu-config/gnu-config_git.bb @@ -19,7 +19,8 @@ UPSTREAM_CHECK_COMMITS = "1" CLEANBROKEN = "1" -do_compile[noexec] = "1" +deltask do_compile +addtask install after do_configure do_install () { install -d ${D}${datadir}/gnu-config \ From patchwork Thu Dec 14 13:45:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33D34C4167D for ; Thu, 14 Dec 2023 13:46:18 +0000 (UTC) Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mx.groups.io with SMTP id smtpd.web10.22366.1702561568378200049 for ; Thu, 14 Dec 2023 05:46:08 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Z0RyLc4F; spf=pass (domain: gmail.com, ip: 209.85.128.47, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-40c2c65e6aaso84775315e9.2 for ; Thu, 14 Dec 2023 05:46:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561567; x=1703166367; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v1hGX18iW1UuGNhwHD/jhuyxl5vy1fWOligRo3zCdQw=; b=Z0RyLc4FAuY/TRcHkLd3X54koDAmpoPurnh5XVda1wVV+3Tb+s+koDJoMBVB4Cwb1P sYABYXqXA2MVahcA/IHGS49DJDgZBopVDE3ulcyQw6oCCf81f6I4FfH1RpleD0/cqJTu AiXczQPNho1rU2jtH6Dxp/E+eU3KaVdlv+h6kTByt8Qwoyb7jCK3/h6xLZqjaDm3ZzVC ZoTN9ocjjPTCBU7RaUQRcYvIjeUii+OLehKl/qiXZcgfSdEaHDEppNRhQQ86I9BLxXAI is59ql9dJyT+gBoKL5RFJxEd7TMZj+lMzZbt9LwHZ84HNV3WhTeQOY1Ug7+1BQyIijpb coVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561567; x=1703166367; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v1hGX18iW1UuGNhwHD/jhuyxl5vy1fWOligRo3zCdQw=; b=gKN5nLr9J3ivX4VlHk+bGBP/oRq1XsX8sMCEo9Hn+rNOIs7m5h3bDXrZOsJjdhC/NG KZvhbM71knpcUQdqOK88rrv0M7y4FbYLyWKggZUPLAA500CVKJ1n+p4XSbrTNVGZAHZc Wkv2Et49kP+9gZYpEXVlYIe4+zB4EpJ3zWeL7o+dJqmfgiHflqO1S+jCyOynfEZ5dvJq Ykhvf4MtbeGejV2l4EITHCU+IzDJJHgsmzUVBqYYgsLhZMUYQ1ZMq/pHwfcxtVJQpU44 TsafvuHrvCroOkuP0jbfUFMR1euzdtvLD14vVsQkRTktBfZ5ydrnuSQUAJ9ijXOFEFb3 lbzg== X-Gm-Message-State: AOJu0YzjyZ/NlUiS+byWKywkn1oz/jDAZ8swGSn4Xyxrm2NE4P+1z2N0 aj5Q/PGHwYzaUqv9DBH2BehR/zS8x2c= X-Google-Smtp-Source: AGHT+IFE2+Gj+vl3LkVVocmcrYWABpepL0iD/juLDNRDOgMUJp+ccFmgdFXm8IV873WfydhzHOcQkQ== X-Received: by 2002:a7b:c40d:0:b0:40c:3150:3d73 with SMTP id k13-20020a7bc40d000000b0040c31503d73mr4546953wmi.247.1702561566372; Thu, 14 Dec 2023 05:46:06 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:06 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 3/9] bitbake/runqueue: initialize RunQueueExecute before printdiff rather than after Date: Thu, 14 Dec 2023 14:45:22 +0100 Message-Id: <20231214134528.1973602-3-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192373 printdiff needs setscene dependencies and they're available only through that object. Previously it was instantianted just after running printdiff, this moves the initilization to just prior. Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/runqueue.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 864708ee4a5..1f59d18b71a 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1555,6 +1555,11 @@ class RunQueue: ('bb.event.HeartbeatEvent',), data=self.cfgData) self.dm_event_handler_registered = True + self.rqdata.init_progress_reporter.next_stage() + self.start_worker() + self.rqdata.init_progress_reporter.next_stage() + self.rqexe = RunQueueExecute(self) + dump = self.cooker.configuration.dump_signatures if dump: self.rqdata.init_progress_reporter.finish() @@ -1566,11 +1571,6 @@ class RunQueue: self.state = runQueueComplete if self.state is runQueueSceneInit: - self.rqdata.init_progress_reporter.next_stage() - self.start_worker() - self.rqdata.init_progress_reporter.next_stage() - self.rqexe = RunQueueExecute(self) - # If we don't have any setscene functions, skip execution if not self.rqdata.runq_setscene_tids: logger.info('No setscene tasks') From patchwork Thu Dec 14 13:45:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33CD9C4332F for ; Thu, 14 Dec 2023 13:46:18 +0000 (UTC) Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by mx.groups.io with SMTP id smtpd.web10.22367.1702561568431675765 for ; Thu, 14 Dec 2023 05:46:08 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=E2/BNBuH; spf=pass (domain: gmail.com, ip: 209.85.128.41, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-40c317723a8so72267825e9.3 for ; Thu, 14 Dec 2023 05:46:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561567; x=1703166367; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZSIOQHxTW7Q1ug0nYc2sFbb8sNYhkwypp10lmfNd4UQ=; b=E2/BNBuHjWTb/2semOz2A1nJBKY3MQeK/eUUP7xrJSu1oGwJ9dOtWkkS0rCrMH/SB5 MyD5wCPbY81A4jPn2EGEoQluuX3YYQbweLzs/NcszIpTOMTka4AWVe5gkeQN5nDqoldN AgaUkCKpuNIK1l6SMjwkTsSANCVgMbWQCuZmM7zR8jIp4eQkqXYZEssz/nBvkeYtYuFR s1X1KPcXEvDvFY0XLkxI6Ty98MP0G7lRvgQKlrDsHtnXyhVFoSYADHvAv4B16B2EE407 p5NjitiJaMSCtgedkAHoSONy1LBDYEc/cMi0WwPjQN5uN4ZfBOOzBGoa+bFSvtzZVup2 2bxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561567; x=1703166367; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZSIOQHxTW7Q1ug0nYc2sFbb8sNYhkwypp10lmfNd4UQ=; b=Ka1YnzmTzxMEVXuZQiN1zOyRgeg/oU1MhuCSu02tftxr9j03lYbj/5wuoCG5utCqQj sK0ifLiR/zqC8s4CtXaWdqgkDGiB56PKop1H1p281BbcL2kzEX2XQ1/6czHUOE6ZSZj1 /nBdnLe383YnqLJgdkc5G4L7c7YBMZ4VbvMzimERebQkDXdGucZ1oxNTC0VkdB/+uRjU hYacNsSjesKonxjeA8LYVXUGvsEsHyBqqsxemc+n6ZRCi/7gSz47zE822MFjE/qdeATB i+DDeuzRzF7Nl3JelCnoY477P8TU3RXymVhkHP9UKT9O5Um50L50wPYxInnntiOSuO/x Z3WQ== X-Gm-Message-State: AOJu0YwkukoOAoLHgNUo115sisctvhqwVROJ1oJjY/4jzgSHoVEtHDMO 6HLZaT85JUUKRk6K1ip1DL8ckpHJwpM= X-Google-Smtp-Source: AGHT+IHOIjbFvMof4RT0277Qyg++xwfrHR819JEo8d6KdBIWdHkfjKd6pwfRDNjHA7rDnp6faPrKYA== X-Received: by 2002:a05:600c:21c1:b0:40c:78c:f864 with SMTP id x1-20020a05600c21c100b0040c078cf864mr4685164wmj.16.1702561566871; Thu, 14 Dec 2023 05:46:06 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:06 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 4/9] bitbake/runqueue: rework 'bitbake -S printdiff' logic Date: Thu, 14 Dec 2023 14:45:23 +0100 Message-Id: <20231214134528.1973602-4-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192374 Previously printdiff code would iterate over tasks that were reported as invalid or absent, trying to follow dependency chains that would reach the most basic invalid items in the tree. While this works in tightly controlled local builds, it can lead to bizarre reports against industrial-sized sstate caches, as the code would not consider whether the overall target can be fulfilled from valid sstate objects, and instead report missing sstate signature files that perhaps were never even created due to hash equivalency providing shortcuts in builds. This commit reworks the logic in two ways: - start the iteration over final targets rather than missing objects and stop at the first invalid object (1) - if a given object can be fulfilled from sstate, recurse only into its setscene dependencies; bitbake wouldn't care if dependencies for the actual task are absent, and neither should printdiff (1) there's a further improvement I'd like to make here: instead of stopping, recurse into dependencies, with the goal of finding and reporting the 'root' invalid objects. Otherwise tracking down the difference to its root relies on finding the most 'recent' signature in stamps or sstate in a different function later, and recursively comparing that to the current signature, which is unreliable on real world caches. For the sake of fixing the test failures I'd like to not delay the patchset any further for now. [YOCTO #15289] Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/runqueue.py | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 1f59d18b71a..61effe24fae 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1717,35 +1717,34 @@ class RunQueue: valid_new.add(dep) invalidtasks = set() - for tid in self.rqdata.runtaskentries: - if tid not in valid_new and tid not in noexec: - invalidtasks.add(tid) - found = set() - processed = set() - for tid in invalidtasks: + toptasks = set(["{}:{}".format(t[3], t[2]) for t in self.rqdata.targets]) + for tid in toptasks: toprocess = set([tid]) while toprocess: next = set() for t in toprocess: - for dep in self.rqdata.runtaskentries[t].depends: - if dep in invalidtasks: - found.add(tid) - if dep not in processed: - processed.add(dep) + if t not in valid_new and t not in noexec: + invalidtasks.add(t) + continue + if t in self.rqdata.runq_setscene_tids: + for dep in self.rqexe.sqdata.sq_deps[t]: next.add(dep) + continue + + for dep in self.rqdata.runtaskentries[t].depends: + next.add(dep) + toprocess = next - if tid in found: - toprocess = set() tasklist = [] - for tid in invalidtasks.difference(found): + for tid in invalidtasks: tasklist.append(tid) if tasklist: bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist)) - return invalidtasks.difference(found) + return invalidtasks def write_diffscenetasks(self, invalidtasks): From patchwork Thu Dec 14 13:45:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3ED1BC4167B for ; Thu, 14 Dec 2023 13:46:18 +0000 (UTC) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by mx.groups.io with SMTP id smtpd.web11.22350.1702561569123060730 for ; Thu, 14 Dec 2023 05:46:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=OCv6n/x5; spf=pass (domain: gmail.com, ip: 209.85.128.53, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-40c41df5577so49535855e9.0 for ; Thu, 14 Dec 2023 05:46:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561567; x=1703166367; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kFiRefpgRKlkCGzi5QCtI+ODA24uuFXrgdpr0+dEMMM=; b=OCv6n/x51Elq+8Xncvw6N01ATbzsOG5UYx4cDZc0AT1QUx1p/AIuNlM6JwRHW6U2md gYQejk+mwv1l2Raesu6cZpnWnGPAeksCmnSLoHLXye82/2tOcm8yYktsSLN7UVJJTOTE utvfK/xU0qNUiO0eberJT+BO7vW4xjOqhWnYUhrXchnGWVT42Aprjmook5KMSc2vWe4f GIVwagCY09TniZ6EB7hQY+Jp7c17+YrongEaaTWeJh9Us/IImfbZ6isLKtXgBsqLbOCB lBNG0/zmaVQe/CfkHDvFwrt7DpBcxbnsk7pSvFHFjYrjgtLhQi2p/y9VkbIyPpwuAum7 cUMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561567; x=1703166367; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kFiRefpgRKlkCGzi5QCtI+ODA24uuFXrgdpr0+dEMMM=; b=iCSpjpKQ1SkwEMByb5kPoGRa8Ebgh2H30T7934WRhulislQYaMhCNCl8rgAY9pRfw5 t9vgPK7Rtmrtp3vSmpnhAFG/bTDZZwCW7lfE0jdYtX+d4fUFQQbYvQsoSi4icFYm+T2Y f9CQdFoRbQD6ZhLWSlsWhiwxQyUI0/NDQokGBBDbHrin70dulmING4QPCTGzfYtfsySS ZVBVjZejgvZldGipkpMiA6SG88W87Soxwd2wygSZmSiPrrmTcoG8Yu8i8cNlcIi50zxX htdfrkiqlT4t44Ws0N/s2rjecw1U0mUo4IvRXCIKCNwjxlgt9CsvhrPeuOgc5XIBvYEF V0YA== X-Gm-Message-State: AOJu0Yx5Xq9ty0TRyEhZCgGG++yvQ55kP0/w+/mWbpr9LODh+YPBfw+a patA4NxZUd9G89ZZCEJEqklPGtYLiUI= X-Google-Smtp-Source: AGHT+IFawwrJO3Z2YvpzuNgvR3mcay1dUOFdEWeHohMZI48yIXf2fBihwRMGKn3M5jgOByG7h/DhXA== X-Received: by 2002:a05:600c:17d0:b0:40c:3617:ee1a with SMTP id y16-20020a05600c17d000b0040c3617ee1amr2422701wmo.142.1702561567372; Thu, 14 Dec 2023 05:46:07 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:07 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 5/9] selftest/sstatetests: fix up printdiff test to match rework of printdiff logic Date: Thu, 14 Dec 2023 14:45:24 +0100 Message-Id: <20231214134528.1973602-5-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192375 The rework changes the console output significantly: where previously printdiff would report the most basic invalid tasks it could find, with this change the most top-level tasks are reported instead. This is not a problem as diffsigs has its own task recursion facility, and will track down the actual change anyway. However the changes still need to be reflected in the selftests. Note that printdiff is run on one specific target task rather than overall do_build for a recipe. Otherwise find_siginfo would run glob.glob("*/*/*taskname*") against autobuilder sstate cache for each of those tasks (six or seven times) - this is an expensive operation taking several minutes, and it's best to do it only once per test. [YOCTO #15289] Signed-off-by: Alexander Kanavin --- .../perlcross_%.bbappend} | 0 meta/lib/oeqa/selftest/cases/sstatetests.py | 40 +++++++++---------- 2 files changed, 18 insertions(+), 22 deletions(-) rename meta-selftest/recipes-test/{quilt-native/quilt-native_%.bbappend => perlcross/perlcross_%.bbappend} (100%) diff --git a/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend b/meta-selftest/recipes-test/perlcross/perlcross_%.bbappend similarity index 100% rename from meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend rename to meta-selftest/recipes-test/perlcross/perlcross_%.bbappend diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index f52ca77c09c..f5b3437d86b 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -824,14 +824,17 @@ TMPDIR = "${{TOPDIR}}/tmp-sstateprintdiff-difftmp-{}" # Check if printdiff walks the full dependency chain from the image target to where the change is in a specific recipe - def test_image_minimal_vs_quilt(self): - expected_output = ("Task quilt-native:do_install couldn't be used from the cache because:", + def test_image_minimal_vs_perlcross(self): + expected_output = ("Task core-image-minimal:do_create_spdx couldn't be used from the cache because:", "We need hash", "most recent matching task was") - expected_sametmp_output = expected_output + ("Variable do_install value changed",'+ echo "this changes the task signature"') + expected_sametmp_output = expected_output + ( +"Hash for task dependency perlcross-native:do_install changed from", +"Variable do_install value changed", +'+ echo "this changes the task signature"') expected_difftmp_output = expected_output - self.run_test_printdiff_changerecipe("core-image-minimal", "quilt-native", "-c do_install quilt-native", + self.run_test_printdiff_changerecipe("core-image-minimal:do_create_spdx", "perlcross", "-c do_install perlcross-native", """ do_install:append() { echo "this changes the task signature" @@ -843,15 +846,15 @@ expected_sametmp_output, expected_difftmp_output) def test_gcc_runtime_vs_gcc_source(self): gcc_source_pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV'] - expected_output = ("Task {}:do_preconfigure couldn't be used from the cache because:".format(gcc_source_pn), + expected_output = ("Task gcc-runtime:do_package_write_rpm couldn't be used from the cache because:", "We need hash", "most recent matching task was") - expected_sametmp_output = expected_output + ("Variable do_preconfigure value changed",'+ print("this changes the task signature")') - #FIXME: printdiff is supposed to find at least one preconfigure task signature in the sstate cache, but isn't able to - #expected_difftmp_output = expected_output - expected_difftmp_output = () + expected_sametmp_output = expected_output + ("Hash for task dependency {}:do_preconfigure changed from".format(gcc_source_pn), +"Variable do_preconfigure value changed", +'+ print("this changes the task signature")') + expected_difftmp_output = expected_output - self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn), + self.run_test_printdiff_changerecipe("gcc-runtime:do_package_write_rpm", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn), """ python do_preconfigure:append() { print("this changes the task signature") @@ -861,22 +864,15 @@ expected_sametmp_output, expected_difftmp_output) # Check if changing a really base task definiton is reported against multiple core recipes using it def test_image_minimal_vs_base_do_configure(self): - expected_output = ("Task zstd-native:do_configure couldn't be used from the cache because:", -"Task texinfo-dummy-native:do_configure couldn't be used from the cache because:", -"Task ldconfig-native:do_configure couldn't be used from the cache because:", -"Task gettext-minimal-native:do_configure couldn't be used from the cache because:", -"Task tzcode-native:do_configure couldn't be used from the cache because:", -"Task makedevs-native:do_configure couldn't be used from the cache because:", -"Task pigz-native:do_configure couldn't be used from the cache because:", -"Task update-rc.d-native:do_configure couldn't be used from the cache because:", -"Task unzip-native:do_configure couldn't be used from the cache because:", -"Task gnu-config-native:do_configure couldn't be used from the cache because:", + expected_output = ("Task core-image-minimal:do_populate_lic_deploy couldn't be used from the cache because:", "We need hash", "most recent matching task was") - expected_sametmp_output = expected_output + ("Variable base_do_configure value changed",'+ echo "this changes base_do_configure() definiton "') + expected_sametmp_output = expected_output + ("Hash for task dependency gnu-config-native:do_configure changed from", +"Variable base_do_configure value changed", +'+ echo "this changes base_do_configure() definiton "') expected_difftmp_output = expected_output - self.run_test_printdiff_changeconfig("core-image-minimal", + self.run_test_printdiff_changeconfig("core-image-minimal:do_populate_lic_deploy", """ INHERIT += "base-do-configure-modified" """, From patchwork Thu Dec 14 13:45:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 490C2C10F13 for ; Thu, 14 Dec 2023 13:46:18 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.22351.1702561569440207416 for ; Thu, 14 Dec 2023 05:46:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=EVbixczW; spf=pass (domain: gmail.com, ip: 209.85.128.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-40c317723a8so72267945e9.3 for ; Thu, 14 Dec 2023 05:46:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561568; x=1703166368; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PDCuA8x0alZUuFuCGsWiaVoAEgDZCzlDf/8YVd2NLg0=; b=EVbixczWzYXk7e85e8LzR7Plv2gXdI003HkSHfu6CfyfNMtXFCgA+kc8MiRPzafsM0 A2KAkFLxjLhhpl1RYCGemBAUxZG2ToNV1rk0zEmMIbY7U19NUhlgggY3OvPImKUfkKvH 3qVi79Oilqs/h+b9fJCO3kX/uytiZp8nUHNFhVMcMF56nYmMFRs3vzjlIBeZGSFyodQJ 3nUIjZW7Q1xAQdBWnP9RICm09ncx/iLQRbzcpUa2NEJbwZYNa2wuEeo7ucSuTgz7rxZm 5REtLvInESVpCfkn1BfMHe41zQfwndpxPImtSm/fu59qafdxbHvY9nqS9FaHBfqnELSg ZwNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561568; x=1703166368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PDCuA8x0alZUuFuCGsWiaVoAEgDZCzlDf/8YVd2NLg0=; b=I9sRQ13zddl8dNjWCdNs1K2jcD6OOP/Z7CIsBoG5C2cmCKLCPZ5Yup5pNTA1db8FEh WRKTM4A4vqegOa+SHyNpFu48VBbTzoWRR/L1+uAzsTdLUqYDDP2tsyj+iJzw1dfg0qA4 UB1YKoeR5NUaQW+egGVDbhjj9GEablHHOSdgJ7RTYDud/f1m01yQ0kWd3ItzwU1Vh59C LBemFIh7ndLraJNhl0BZSwBENo9pAb4h1S5Q6FPoIrqLeNot2wgYXHaI2s+0niNNZDCq E9H9CgE40+AH374CVYFbPVoZyR5Qce4dsYyUEVKk45AJymGDOUDGR/rLpR4WuRnZHs0K TX8g== X-Gm-Message-State: AOJu0YzfHn6/aoUSdUZrcVqjU5LlqsSuzSO2J3N+w2dmhD+/NEKxTIi5 Ab2jhofAXSB41MOo6DfRmz3ToaY4gQw= X-Google-Smtp-Source: AGHT+IGGje97Xj/9QpZn5gBCzTBKy63SnNTkDaIM6MSg4wqBAENvtj6BufOOTx1VSSTDoXJi4N0LQw== X-Received: by 2002:a05:600c:492f:b0:40c:3464:f816 with SMTP id f47-20020a05600c492f00b0040c3464f816mr4854189wmp.51.1702561567836; Thu, 14 Dec 2023 05:46:07 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:07 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 6/9] sstatesig/find_siginfo: unify a disjointed API Date: Thu, 14 Dec 2023 14:45:25 +0100 Message-Id: <20231214134528.1973602-6-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192376 find_siginfo() returns two different data structures depending on whether its third argument (list of hashes to find) is empty or not: - a dict of timestamps keyed by path - a dict of paths keyed by hash This is not a good API design; it's much better to return a dict of dicts that include both timestamp and path, keyed by hash. Then the API consumer can decide how they want to use these fields, particularly for additional diagnostics or informational output. I also took the opportunity to add a binary field that tells if the match came from sstate or local stamps dir, which will help prioritize local stamps when looking up most recent task signatures. Signed-off-by: Alexander Kanavin --- meta/lib/oe/buildhistory_analysis.py | 2 +- meta/lib/oe/sstatesig.py | 31 +++++++++------------ meta/lib/oeqa/selftest/cases/sstatetests.py | 10 +++---- 3 files changed, 19 insertions(+), 24 deletions(-) diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py index b1856846b6a..4edad01580c 100644 --- a/meta/lib/oe/buildhistory_analysis.py +++ b/meta/lib/oe/buildhistory_analysis.py @@ -562,7 +562,7 @@ def compare_siglists(a_blob, b_blob, taskdiff=False): elif not hash2 in hashfiles: out.append("Unable to find matching sigdata for %s with hash %s" % (desc, hash2)) else: - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, collapsed=True) + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1]['path'], hashfiles[hash2]['path'], recursecb, collapsed=True) for line in out2: m = hashlib.sha256() m.update(line.encode('utf-8')) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index 8a97fb0c04b..0342bcdc87a 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -349,7 +349,6 @@ def find_siginfo(pn, taskname, taskhashlist, d): pn, taskname = key.split(':', 1) hashfiles = {} - filedates = {} def get_hashval(siginfo): if siginfo.endswith('.siginfo'): @@ -357,6 +356,12 @@ def find_siginfo(pn, taskname, taskhashlist, d): else: return siginfo.rpartition('.')[2] + def get_time(fullpath): + try: + return os.stat(fullpath).st_mtime + except OSError: + return None + # First search in stamps dir localdata = d.createCopy() localdata.setVar('MULTIMACH_TARGET_SYS', '*') @@ -372,24 +377,21 @@ def find_siginfo(pn, taskname, taskhashlist, d): filespec = '%s.%s.sigdata.*' % (stamp, taskname) foundall = False import glob + bb.debug(1, "Calling glob.glob on {}".format(filespec)) for fullpath in glob.glob(filespec): match = False if taskhashlist: for taskhash in taskhashlist: if fullpath.endswith('.%s' % taskhash): - hashfiles[taskhash] = fullpath + hashfiles[taskhash] = {'path':fullpath, 'sstate':False, 'time':get_time(fullpath)} if len(hashfiles) == len(taskhashlist): foundall = True break else: - try: - filedates[fullpath] = os.stat(fullpath).st_mtime - except OSError: - continue hashval = get_hashval(fullpath) - hashfiles[hashval] = fullpath + hashfiles[hashval] = {'path':fullpath, 'sstate':False, 'time':get_time(fullpath)} - if not taskhashlist or (len(filedates) < 2 and not foundall): + if not taskhashlist or (len(hashfiles) < 2 and not foundall): # That didn't work, look in sstate-cache hashes = taskhashlist or ['?' * 64] localdata = bb.data.createCopy(d) @@ -412,22 +414,15 @@ def find_siginfo(pn, taskname, taskhashlist, d): localdata.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/") filespec = '%s.siginfo' % localdata.getVar('SSTATE_PKG') + bb.debug(1, "Calling glob.glob on {}".format(filespec)) matchedfiles = glob.glob(filespec) for fullpath in matchedfiles: actual_hashval = get_hashval(fullpath) if actual_hashval in hashfiles: continue - hashfiles[hashval] = fullpath - if not taskhashlist: - try: - filedates[fullpath] = os.stat(fullpath).st_mtime - except: - continue + hashfiles[actual_hashval] = {'path':fullpath, 'sstate':True, 'time':get_time(fullpath)} - if taskhashlist: - return hashfiles - else: - return filedates + return hashfiles bb.siggen.find_siginfo = find_siginfo diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index f5b3437d86b..83abcdfdd62 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -765,14 +765,14 @@ addtask tmptask2 before do_tmptask1 hashes = [hash1, hash2] hashfiles = find_siginfo(key, None, hashes) self.assertCountEqual(hashes, hashfiles) - bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) + bb.siggen.compare_sigfiles(hashfiles[hash1]['path'], hashfiles[hash2]['path'], recursecb) for pn in pns: recursecb_count = 0 - filedates = find_siginfo(pn, "do_tmptask1") - self.assertGreaterEqual(len(filedates), 2) - latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:] - bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) + matches = find_siginfo(pn, "do_tmptask1") + self.assertGreaterEqual(len(matches), 2) + latesthashes = sorted(matches.keys(), key=lambda h: matches[h]['time'])[-2:] + bb.siggen.compare_sigfiles(matches[latesthashes[-2]]['path'], matches[latesthashes[-1]]['path'], recursecb) self.assertEqual(recursecb_count,1) class SStatePrintdiff(SStateBase): From patchwork Thu Dec 14 13:45:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 506C0C46CA2 for ; Thu, 14 Dec 2023 13:46:18 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web10.22369.1702561570156317089 for ; Thu, 14 Dec 2023 05:46:10 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=GWlrkC27; spf=pass (domain: gmail.com, ip: 209.85.128.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-40c2d50bfbfso3820675e9.0 for ; Thu, 14 Dec 2023 05:46:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561568; x=1703166368; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lrjmM/DQ7JA93nfNjkI606RQQXBY6OWU2zVVOo08hWM=; b=GWlrkC27tfWrlr0xM0pvDH8kock/RFgFtBXikH9/XxWlA5+560NRBGX6EbVZpsVWyu dMQKQe3Yoktl6+LTfbiiicVSAs9HYG7Uv6SXR8t1mGiSoHTYA4kYlfgN0oH6DilHYjmh iKUA3QkyciWd41d4mDDJKPnT4ENE7+pWgaYczbD3PTwWaw9HIcgCL4I9BQtVkrnaQG3d zVMtSbuKD/MLXDFYqgcQWXcMa6XRyK7OVE3UhSAXvUKQcsRbg1QPaIQrZn/dTbHF1mrP NGA6PuvU2B92Z2dr3EUhlMEDK3NWunTHOffvcO/kT+2f8YOCGIv0tsNl7Sa2qKy17iEG j5sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561568; x=1703166368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lrjmM/DQ7JA93nfNjkI606RQQXBY6OWU2zVVOo08hWM=; b=mzEoV50pfPdFTXXxoCwz1Y535jUrevSKOM4vdgDxHFna8U3qtPrbRObIkjqySERgGU fBxR9YXP3WR2txZ0ZxRbeRV1SzKitMkz1Rqmgl7/2GWcfSuxrXqrAjWh7Cd6aV6yUgIA kmoiaIkzt9ufAyuXx9YQ3wE0stEpoFMnuLh06K+wAPVqz9qzsqD/ZRVL9uQdJMgkhcxl XmGYiF/9KCcnZkMVgRShOpUu+EiYAiPbzlN0/I4lME1Sv+nzU1AMFmQDmMvWfS5yiObc ZSITa2NRXuRiaW8QnygRLQSonxAAzlbTUSXXAIpLuQY0K+L75zvU+IOMqGZaXyjgG+ey RI1A== X-Gm-Message-State: AOJu0Yz7AxFN14Ql1ZjA1Kz9vwwAfmh+V/7zlsTpNEKCEFIDRLiej612 Hu+6Js/KljwUAjqBWW/zyOdKI9Egfxs= X-Google-Smtp-Source: AGHT+IHbwO+eaDPt3ZUaKgpGpoU/EpM3fb9057o3WNqyq7zQDpy+28phas6SPzvuzZiCjlx9t3GzVg== X-Received: by 2002:a05:600c:3107:b0:40c:29a1:5559 with SMTP id g7-20020a05600c310700b0040c29a15559mr4949855wmo.67.1702561568321; Thu, 14 Dec 2023 05:46:08 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:08 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 7/9] bitbake-diffsigs/runqueue: adapt to reworked find_siginfo() Date: Thu, 14 Dec 2023 14:45:26 +0100 Message-Id: <20231214134528.1973602-7-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192377 In particular having 'time' explicitly used as a sorting key should make it more clear how the entries are being sorted. Signed-off-by: Alexander Kanavin --- bitbake/bin/bitbake-diffsigs | 11 +++++++---- bitbake/lib/bb/runqueue.py | 10 +++++----- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs index fe0f33eea17..a8f49191b0c 100755 --- a/bitbake/bin/bitbake-diffsigs +++ b/bitbake/bin/bitbake-diffsigs @@ -72,13 +72,16 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): elif sig2 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2)) sys.exit(1) - latestfiles = [sigfiles[sig1], sigfiles[sig2]] else: - filedates = find_siginfo(bbhandler, pn, taskname) - latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:] - if not latestfiles: + sigfiles = find_siginfo(bbhandler, pn, taskname) + latestsigs = sorted(sigfiles.keys(), key=lambda h: sigfiles[h]['time'])[-2:] + if not latestsigs: logger.error('No sigdata files found matching %s %s' % (pn, taskname)) sys.exit(1) + sig1 = latestsigs[0] + sig2 = latestsigs[1] + + latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']] return latestfiles diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 61effe24fae..ac5222514de 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1755,7 +1755,7 @@ class RunQueue: recout = [] if len(hashfiles) == 2: - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1]['path'], hashfiles[hash2]['path'], recursecb) recout.extend(list(' ' + l for l in out2)) else: recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) @@ -1769,14 +1769,14 @@ class RunQueue: h = self.rqdata.runtaskentries[tid].unihash matches = bb.siggen.find_siginfo(pn, taskname, [], self.cooker.databuilder.mcdata[mc]) match = None - for m in matches: - if h in m: - match = m + for m in matches.values(): + if h in m['path']: + match = m['path'] if match is None: bb.fatal("Can't find a task we're supposed to have written out? (hash: %s tid: %s)?" % (h, tid)) matches = {k : v for k, v in iter(matches.items()) if h not in k} if matches: - latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1] + latestmatch = matches[sorted(matches.keys(), key=lambda h: matches[h]['time'])[-1]]['path'] prevh = __find_sha256__.search(latestmatch).group(0) output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb) bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, most recent matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output)) From patchwork Thu Dec 14 13:45:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5925EC3DA79 for ; Thu, 14 Dec 2023 13:46:18 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web10.22370.1702561570367151091 for ; Thu, 14 Dec 2023 05:46:10 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YsYeoL4o; spf=pass (domain: gmail.com, ip: 209.85.128.50, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-40c3f68b69aso51463425e9.1 for ; Thu, 14 Dec 2023 05:46:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561569; x=1703166369; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9ZxyAw3eBngwKmEf9BauHy1LZgyKnwrRJBWZyYft3yo=; b=YsYeoL4or0wj7napdcuHvLqYVRveUb8FsWaOfrWm00e7uniIhCDaaYsDSjvSCXkGax d9zVwbfEmAA5E7EsVQ2mnez3rg99qj9kgn12hVpM6TDEXAxljRcSbql+e80M40qe3bEd xf7w7M4f+Pxh4snIlDBigfoo/8RVOe1KrgIEXyC1yUUTSU9uToZPYj7I1rdccjzhlAio Z9RxxtvtizWkt5k4PuB3dA4pWiWvU3FBNWa5bjaoTryrI8yNqFFma6lYkF1pAyYciS3l zTnHmP8nsy11QuVVe6HJU0xm9Ko6BMau63NKbo40EWaWNxKucSJr4DYHxo/YkzydRfim uvpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561569; x=1703166369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9ZxyAw3eBngwKmEf9BauHy1LZgyKnwrRJBWZyYft3yo=; b=tgf7bEorDZcodKlsPgX7PhJi+oTmGfYYAwYm/1O6UF5ueG5RCbfPyXewl3QoIB3Mu5 FQhkVaCmM8UUmGCowpsorB+j3gj/kyfd2HnHWdYYJf4i6MZKfrpRjnbE15dEvzZyoy77 XKVksATUIQ9itXEQx/DhjyBG9ElFWeMg1Ld7nz0QhKfXNuQ/55t3hQn1Rlkfv4c+TWXj XF5LfnkcyjdQe7Yxxr6HH3KXKB8INzrUHuPVIL62iYwo48/hTWVv/I80/CTR6F1aAv4Z ZI6GVqgHzDvPeBGz9emPRmMM3ptiAikAiwaKQVIcCOENEL2TtP6GoTfGkNPCdJhpjxBM RAaw== X-Gm-Message-State: AOJu0YzsCUQrdIhVx2/SmiW3HN/56jk1K0SI1Gi+tZJEOM5hB8ntq6ED l0E98k/kvoiYXNnbV9JNF+e1qrYMT78= X-Google-Smtp-Source: AGHT+IGr9ciU+yeaXX9TePsHDjwEvbL4/OuyxWRFCEGI087MEUuXCVau/b4UUfoeBVg4CITkmeakog== X-Received: by 2002:a05:600c:3ca2:b0:40c:50d5:f7a6 with SMTP id bg34-20020a05600c3ca200b0040c50d5f7a6mr2848604wmb.121.1702561568767; Thu, 14 Dec 2023 05:46:08 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:08 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 8/9] bitbake/runqueue: prioritize local stamps over sstate signatures in printdiff Date: Thu, 14 Dec 2023 14:45:27 +0100 Message-Id: <20231214134528.1973602-8-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192378 Even with the reworked printdiff code, sstate which is heavily used in parallel can throw races at the tests: if a new matching, but otherwise unrelated sstate signature appears between writing out local stamps and listing matching sstate files, then that signature will be deemed 'the latest' and the actual local stamp will be discarded. This change ensures the scenario does not happen. It also makes use of the reworked find_siginfo(), particularly the 'sstate' entry in returned results. Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/runqueue.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index ac5222514de..2d2b28b3c99 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1775,6 +1775,9 @@ class RunQueue: if match is None: bb.fatal("Can't find a task we're supposed to have written out? (hash: %s tid: %s)?" % (h, tid)) matches = {k : v for k, v in iter(matches.items()) if h not in k} + matches_local = {k : v for k, v in iter(matches.items()) if h not in k and not v['sstate']} + if matches_local: + matches = matches_local if matches: latestmatch = matches[sorted(matches.keys(), key=lambda h: matches[h]['time'])[-1]]['path'] prevh = __find_sha256__.search(latestmatch).group(0) From patchwork Thu Dec 14 13:45:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65C4FC46CA3 for ; Thu, 14 Dec 2023 13:46:18 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.22352.1702561570813518065 for ; Thu, 14 Dec 2023 05:46:11 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=asxWUoIc; spf=pass (domain: gmail.com, ip: 209.85.128.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-40c3ceded81so52913425e9.1 for ; Thu, 14 Dec 2023 05:46:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702561569; x=1703166369; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A+gM/aU97fLDxQi6vxpdZkgP5dImybBSIlpvW130m/o=; b=asxWUoIcOei4+sDIgR1gWW7R776rWc0dGLg+Xb1H2UTPCBVBGQwMOQ59P7UUzhIqzt Nyw+N1+hYoGl2I2oR4JYnhMzNRXybnoZxXwjEhzeT63xHvDxy/D8vxXq7iCXZl66ciyE m5IKUDChmwwaoCi867ZlUS3MjYanXSbYhFzgGi/Qvu+YCsr30135xiogg3UfCg08qGFk X88O4HQshevmsALn/MGcgDk8jMuov6yFh8JObSWB1DbfgD1M1343CRNmaLRTQr0pO0C7 R6bmubxX7PJK6qn/KWlK0rtBEbTYosI5wlQBJoRqWW9ntM1tx68PWGccmeVItLijPIG4 letg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702561569; x=1703166369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A+gM/aU97fLDxQi6vxpdZkgP5dImybBSIlpvW130m/o=; b=ULrqJQcWy78Eh9sMUeoE4/n4GMDJdkxRSE9j1cCv4SB4wuBj9m3IBFRScFGqZIxsmn AbXeufFwtpN3MRpBlnxE6MiVOzuvNvcPi2DTTSEK3cd1pNHB8xLuxa68ZPaDR2jJH87G D6AJPvGOdpCbLUd740gT5LC4ArUWZVflupsz3knLd53LYaMVW2FFXcoHrzPgyX54HDP8 Q/GDt9bj4wvp2gptHZWT5XdKzYaKfVVZPPmMJNHHTNFjTyCQcTtmUJpmj67Cs3VkY0Q+ Kx9/u3/FwtVt9ub9RAtdB7FDF2HI8A/zIN41vJdssyTXsY3lV5GSmV0kM/1zKjv8I6TR kEQA== X-Gm-Message-State: AOJu0YwuSmBgs8m2aDppYNAV5F7rCWLazT0i6yB6DBtRog6hz6q+P5Sh JTOQ8xR/4SmIJmN2YWe/I7R/MWMOq+c= X-Google-Smtp-Source: AGHT+IHpANTshGz1sdsXhZZwd7gbaurdQ5GBpAdZTlTG2L+kBYXVHcKiJyhwtfJvyqr44+X+UlGZHQ== X-Received: by 2002:a7b:cc86:0:b0:40b:5e21:ec24 with SMTP id p6-20020a7bcc86000000b0040b5e21ec24mr5352821wma.86.1702561569233; Thu, 14 Dec 2023 05:46:09 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c1d0f00b003feae747ff2sm27293710wms.35.2023.12.14.05.46.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 05:46:09 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 9/9] bitbake/runqueue: add debugging for find_siginfo() calls Date: Thu, 14 Dec 2023 14:45:28 +0100 Message-Id: <20231214134528.1973602-9-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231214134528.1973602-1-alex@linutronix.de> References: <20231214134528.1973602-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 14 Dec 2023 13:46:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192379 Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/runqueue.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 2d2b28b3c99..bb75e81c557 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1751,7 +1751,9 @@ class RunQueue: # Define recursion callback def recursecb(key, hash1, hash2): hashes = [hash1, hash2] + bb.debug(1, "Recursively looking for recipe {} hashes {}".format(key, hashes)) hashfiles = bb.siggen.find_siginfo(key, None, hashes, self.cfgData) + bb.debug(1, "Found hashfiles:\n{}".format(hashfiles)) recout = [] if len(hashfiles) == 2: @@ -1767,7 +1769,9 @@ class RunQueue: (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] h = self.rqdata.runtaskentries[tid].unihash + bb.debug(1, "Looking for recipe {} task {}".format(pn, taskname)) matches = bb.siggen.find_siginfo(pn, taskname, [], self.cooker.databuilder.mcdata[mc]) + bb.debug(1, "Found hashfiles:\n{}".format(matches)) match = None for m in matches.values(): if h in m['path']: