From patchwork Mon Dec 18 08:43:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAE9CC46CA2 for ; Mon, 18 Dec 2023 08:44:23 +0000 (UTC) Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by mx.groups.io with SMTP id smtpd.web10.39800.1702889055833327407 for ; Mon, 18 Dec 2023 00:44:16 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=IdQUk8Ix; spf=pass (domain: gmail.com, ip: 209.85.218.44, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a2363e4f996so23678166b.1 for ; Mon, 18 Dec 2023 00:44:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702889054; x=1703493854; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=nDHmBbAIJvsZgBWNxwA7mhh6S1bqL6N7QUu1IwG3P/I=; b=IdQUk8IxXXl5oMrOzfiYtUda0BvJR3cBO9P+cD8fuvI9Gkt+aEEJAsiaxgTqMqW25d fIY6OOW6ooZSVdkN7q7120GgncXjghgtYw64+mWqYWleoxJBC+uHcuftf49OBb8bFOhR /yA528EmPonM8k32YaC/ekvnB097jXPWVZ0Th87H0PTanmKON68QR5mKMpV0mwwMWa/Z wFqkMzygN2uh8jvmJT2YM6svDPUFqI068lWX3cN4ojDR+286bvJqQTw4TFi4wfMIDYCk 6ecUM9v9cVbKXflRwm1SDRmuQ+/peiJaiHfsF6kq1rhwklkwqt84PlgYM2kLBJbgv1/U iwYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702889054; x=1703493854; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nDHmBbAIJvsZgBWNxwA7mhh6S1bqL6N7QUu1IwG3P/I=; b=IMN3U8PxAlNx4b5Lwj+8NGPQZSQLFd+enzFt8hBIymm114180ljGKAqv/aWghcB0Cb Dkxld31/dX8wu0gDMCqjtDi1iqzS1YI6NC90SADxtYseIk1IrSpu6QxW8xdxHdB42q50 Fc9W9QZUoHfIubGqqbkmtyr8Km2TL+5WRfLH8BqvJd0m7VrJ7KIkQqkRYYFP2itmQ1Ej IQuSuzTTjsmPSp6Iu/btUi5q5l0vmIzdE27H4EfSke8lAoixBQ3fgZ8jV0nh8V3IyI8t U5Epmnb0QjvwwOk3tREtC57wBfTagIQenrEk195/PFc2icDug3AhEUblH8W0C7yTfV9Z moZQ== X-Gm-Message-State: AOJu0YxPgW1+s9pqLvdu3PtqE2zrK1zAJiOfV2cJZ/eVlIRh5fIK8lac /m5ENHRYYmZhpX0rhXBbgT5eJBL9640= X-Google-Smtp-Source: AGHT+IGzoUAKIF2+O2AP9vnyb47RVMb5u77wzeCc0gF8SW9SIzC62R2X+iW5WfyjvJgOSsS78AbY6w== X-Received: by 2002:a17:906:73d0:b0:a22:f58e:ed09 with SMTP id n16-20020a17090673d000b00a22f58eed09mr4097919ejl.153.1702889054055; Mon, 18 Dec 2023 00:44:14 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id vt6-20020a170907a60600b00a1ce98016b6sm14016299ejc.97.2023.12.18.00.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 00:44:13 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 1/7] bitbake/runqueue: rework 'bitbake -S printdiff' logic Date: Mon, 18 Dec 2023 09:43:57 +0100 Message-Id: <20231218084403.599015-1-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Dec 2023 08:44:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192612 Previously printdiff code would iterate over tasks that were reported as invalid or absent, trying to follow dependency chains that would reach the most basic invalid items in the tree. While this works in tightly controlled local builds, it can lead to bizarre reports against industrial-sized sstate caches, as the code would not consider whether the overall target can be fulfilled from valid sstate objects, and instead report missing sstate signature files that perhaps were never even created due to hash equivalency providing shortcuts in builds. This commit reworks the logic in two ways: - start the iteration over final targets rather than missing objects and try to recursively arrive at the root of the invalid object dependency. A previous version of this patch relied relies on finding the most 'recent' signature in stamps or sstate in a different function later, and recursively comparing that to the current signature, which is unreliable on real world caches. - if a given object can be fulfilled from sstate, recurse only into its setscene dependencies; bitbake wouldn't care if dependencies for the actual task are absent, and neither should printdiff I wrote a recursive function for following dependencies, as doing recursive algorithms non-recursively can result in write-only code, as was the case here. [YOCTO #15289] Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/runqueue.py | 41 ++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 24497c5c173..f54d9b85541 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1685,6 +1685,17 @@ class RunQueue: return def print_diffscenetasks(self): + def get_root_invalid_tasks(task, taskdepends, valid, noexec, visited_invalid): + invalidtasks = [] + for t in taskdepends[task].depends: + if t not in valid and t not in visited_invalid: + invalidtasks.extend(get_root_invalid_tasks(t, taskdepends, valid, noexec, visited_invalid)) + visited_invalid.add(t) + + direct_invalid = [t for t in taskdepends[task].depends if t not in valid] + if not direct_invalid and task not in noexec: + invalidtasks = [task] + return invalidtasks noexec = [] tocheck = set() @@ -1718,35 +1729,35 @@ class RunQueue: valid_new.add(dep) invalidtasks = set() - for tid in self.rqdata.runtaskentries: - if tid not in valid_new and tid not in noexec: - invalidtasks.add(tid) - found = set() - processed = set() - for tid in invalidtasks: + toptasks = set(["{}:{}".format(t[3], t[2]) for t in self.rqdata.targets]) + for tid in toptasks: toprocess = set([tid]) while toprocess: next = set() + visited_invalid = set() for t in toprocess: - for dep in self.rqdata.runtaskentries[t].depends: - if dep in invalidtasks: - found.add(tid) - if dep not in processed: - processed.add(dep) + if t not in valid_new and t not in noexec: + invalidtasks.update(get_root_invalid_tasks(t, self.rqdata.runtaskentries, valid_new, noexec, visited_invalid)) + continue + if t in self.rqdata.runq_setscene_tids: + for dep in self.rqexe.sqdata.sq_deps[t]: next.add(dep) + continue + + for dep in self.rqdata.runtaskentries[t].depends: + next.add(dep) + toprocess = next - if tid in found: - toprocess = set() tasklist = [] - for tid in invalidtasks.difference(found): + for tid in invalidtasks: tasklist.append(tid) if tasklist: bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist)) - return invalidtasks.difference(found) + return invalidtasks def write_diffscenetasks(self, invalidtasks): From patchwork Mon Dec 18 08:43:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F022C46CD8 for ; Mon, 18 Dec 2023 08:44:24 +0000 (UTC) Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by mx.groups.io with SMTP id smtpd.web11.40232.1702889056913233718 for ; Mon, 18 Dec 2023 00:44:17 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=glRgszOs; spf=pass (domain: gmail.com, ip: 209.85.208.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-550dd0e3304so3267963a12.1 for ; Mon, 18 Dec 2023 00:44:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702889055; x=1703493855; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FcSF4UtUs94nxHswCiwY5ym4j6x6+XsWShzrdjCHf4E=; b=glRgszOs47y5ZI4siXRJtWhLF+74um4TS0e6KRGFPGwfrqaafDtvWEFHPlzGk/mjr6 UKGRyWeBLEBJwQ3uTyZlEGSA3EhFA17XL6vyqkpTIRG1gDK4V1rKYZVxrBJm5AHN0wI2 b9q1vztBFdwRSEaSpAgnB4p+jvscFXWie+9NP5iqHLaeVLTIQ4p1yAyxIN20UwDqdXw9 UNSOm8SKvXj7QCUR8rwwzgCAXZjJCGEv/Np6MqP/KayHAXaeAQfI4+pQgUYacmvXFvgU o7i1cJAKvoqduFRaC/VfE/IT2R3xJNmfeSDtrkOwbRnEOegrxocS72755MZCcGZTLFbk u4lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702889055; x=1703493855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FcSF4UtUs94nxHswCiwY5ym4j6x6+XsWShzrdjCHf4E=; b=L4Agh1mUiJJvjy/0ABHXj9D9tLvX3hZ2VOShn1HNn5OaqePydhyQ0lw87LGHGNB0CJ fxx9wuCwm3RFhN2zyl5CTLQxy766Y+sj5gkXMIKUMt+Ao3CGWc1YjS9NQE0x38M6rh09 42xsLlSMNp1Tp9BgCSNbE0/JUQ39fzN7tWMgewxwjcktuAfS7Z38WSZqgoinkJRl6Dry iIMNZI7mOkOrTuCGBvX/NfkoGa/6DaUDlWrm8TiHcMJflnh9bKvdY+FkIAXkFmq6ORW9 Upe4HC4os8xjZtT3S4d7ARxIS8GaPdQRZxaMMJwFrMoZjBiy1UMyJO0LJ9fA0zdTGRLZ S7Fg== X-Gm-Message-State: AOJu0YxbIu0InfVOEPBeToV1fvvcxyjRT04wMlQC9/HPIs8MT8BcZeLF Xf5rqW5VcFAP9wrs8mKDyAAc7t6z9jE= X-Google-Smtp-Source: AGHT+IFyi/JkVk7uoQibtFw3I+cAmDUyMOfMFDOpUIkSYRbkV3NZKhIkhF9gZ/oeMft7DcZyCY8EoA== X-Received: by 2002:a17:906:1799:b0:a18:ad93:460d with SMTP id t25-20020a170906179900b00a18ad93460dmr7937216eje.69.1702889054584; Mon, 18 Dec 2023 00:44:14 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id vt6-20020a170907a60600b00a1ce98016b6sm14016299ejc.97.2023.12.18.00.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 00:44:14 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 2/7] selftest/sstatetests: fix up printdiff test to match rework of printdiff logic Date: Mon, 18 Dec 2023 09:43:58 +0100 Message-Id: <20231218084403.599015-2-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231218084403.599015-1-alex@linutronix.de> References: <20231218084403.599015-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Dec 2023 08:44:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192613 Other than the formatting changes, there are two functional ones: - use perlcross instead of quilt, as quilt is special in the sense of being excluded from task hash calculcations. perlcross is a full participant. - run the full test (local + sstate) for gcc do_preconfiure change as the necessary fix has been implemented (sstatesig/find_siginfo: special-case gcc-source when looking in sstate caches) Note that when several tasks are found to have changed (as is the case when base do_configure is adjusted), find_siginfo() runs glob.glob("*/*/*taskname*") against autobuilder sstate cache for each of those tasks (six or seven times). This is an expensive operation taking several minutes. I left it in for now, but if it's proven too slow the test would have to be reduced to checking a specific base recipe (e.g. zstd-native) rather than a distant image target. [YOCTO #15289] Signed-off-by: Alexander Kanavin --- .../perlcross_%.bbappend} | 0 meta/lib/oeqa/selftest/cases/sstatetests.py | 22 +++++++++++-------- 2 files changed, 13 insertions(+), 9 deletions(-) rename meta-selftest/recipes-test/{quilt-native/quilt-native_%.bbappend => perlcross/perlcross_%.bbappend} (100%) diff --git a/meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend b/meta-selftest/recipes-test/perlcross/perlcross_%.bbappend similarity index 100% rename from meta-selftest/recipes-test/quilt-native/quilt-native_%.bbappend rename to meta-selftest/recipes-test/perlcross/perlcross_%.bbappend diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index f827615ba1b..6af3c8f2178 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -824,14 +824,16 @@ TMPDIR = "${{TOPDIR}}/tmp-sstateprintdiff-difftmp-{}" # Check if printdiff walks the full dependency chain from the image target to where the change is in a specific recipe - def test_image_minimal_vs_quilt(self): - expected_output = ("Task quilt-native:do_install couldn't be used from the cache because:", + def test_image_minimal_vs_perlcross(self): + expected_output = ("Task perlcross-native:do_install couldn't be used from the cache because:", "We need hash", "most recent matching task was") - expected_sametmp_output = expected_output + ("Variable do_install value changed",'+ echo "this changes the task signature"') + expected_sametmp_output = expected_output + ( +"Variable do_install value changed", +'+ echo "this changes the task signature"') expected_difftmp_output = expected_output - self.run_test_printdiff_changerecipe("core-image-minimal", "quilt-native", "-c do_install quilt-native", + self.run_test_printdiff_changerecipe("core-image-minimal", "perlcross", "-c do_install perlcross-native", """ do_install:append() { echo "this changes the task signature" @@ -846,10 +848,10 @@ expected_sametmp_output, expected_difftmp_output) expected_output = ("Task {}:do_preconfigure couldn't be used from the cache because:".format(gcc_source_pn), "We need hash", "most recent matching task was") - expected_sametmp_output = expected_output + ("Variable do_preconfigure value changed",'+ print("this changes the task signature")') - #FIXME: printdiff is supposed to find at least one preconfigure task signature in the sstate cache, but isn't able to - #expected_difftmp_output = expected_output - expected_difftmp_output = () + expected_sametmp_output = expected_output + ( +"Variable do_preconfigure value changed", +'+ print("this changes the task signature")') + expected_difftmp_output = expected_output self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn), """ @@ -873,7 +875,9 @@ expected_sametmp_output, expected_difftmp_output) "Task gnu-config-native:do_configure couldn't be used from the cache because:", "We need hash", "most recent matching task was") - expected_sametmp_output = expected_output + ("Variable base_do_configure value changed",'+ echo "this changes base_do_configure() definiton "') + expected_sametmp_output = expected_output + ( +"Variable base_do_configure value changed", +'+ echo "this changes base_do_configure() definiton "') expected_difftmp_output = expected_output self.run_test_printdiff_changeconfig("core-image-minimal", From patchwork Mon Dec 18 08:43:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 116D5C46CD7 for ; Mon, 18 Dec 2023 08:44:24 +0000 (UTC) Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by mx.groups.io with SMTP id smtpd.web11.40234.1702889057376452112 for ; Mon, 18 Dec 2023 00:44:17 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=favvmKXt; spf=pass (domain: gmail.com, ip: 209.85.208.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-54f4f7e88feso3627723a12.3 for ; Mon, 18 Dec 2023 00:44:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702889056; x=1703493856; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JpAWXQbY8MP84gzbq4goyuUOCkPrrct4XgapFfia3y0=; b=favvmKXtTxqr3JeTDj/yRlrqkFnxr4vZ3jTdo6FIdFu8BXZO+X1tQHgKFRKOAjGEC+ cRkyyeFkB7IpMvpewiTI+B6WMnlCXXbKpUw3nz2k2qbSYaKwxvb2UcF5fhudk+pMPK3N FTpAj+hcbWXy9FdVqt6cxkJ7vTY0E+5tgi6IO3YRQK2sOoIBQ0S4hytViF+K4NYXnymL PgxfudGFaVLTxhqNyxtz3vvabeAVxV7SD3QIuiWjJz1wiEG2+ViXVYisahNiUUwDQ1pC 9WiFxxG4XWAvs+sCYT2oeWy16kbGAMhi5bh00KnQ+OG9GQq+0doOrMtZv64254Rs6XJE Hkrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702889056; x=1703493856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JpAWXQbY8MP84gzbq4goyuUOCkPrrct4XgapFfia3y0=; b=Z49HV4hIvh5uU1H0hpPejFXPWAp8FvSlTK9Ww3QM1GeaTst7X8H1gqJN7y/fRSR+wf qtxN6VO72CjNSfm2iIOjNqXuwCgav3/ZV2htjvVdBfqAVOgsDAlD3A+Wwn7DfY7wz2bc XWXOus5dAPJEDsH94TBneKsGnJ3+UOzjGS9F9S+OMvGBCdtkKhg8GWaewg4CPykouXdK 48UBYOWpw1MpUqQLozTFL0LTavT8IhKw4ZRM7iKBuZsxlpSaZke9KWsdmdHyJliUyNUD ZccAgVBZiSKG2vmdxWd4mSOiX+ONPRtdd+ugOgu3ZTOwhjFf+kyJ9SFmyUS/+kmjd9Wo nkvw== X-Gm-Message-State: AOJu0Yw8l281V4R+oenvND6yZpaIlqeNBgSa4PKcDmK1GdpDO9V+JxYM h814CFkEK72/qtVf9F8QNH3668OfAXI= X-Google-Smtp-Source: AGHT+IG8OiXcn8EJAalBzSNzbwTCkXsXPNjYZSzcqu+G9H7dRFx7DEeP5MI1X1sjWg9T3KYlv2/JwA== X-Received: by 2002:a17:906:73d0:b0:a23:482:ee6e with SMTP id n16-20020a17090673d000b00a230482ee6emr3513431ejl.20.1702889055085; Mon, 18 Dec 2023 00:44:15 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id vt6-20020a170907a60600b00a1ce98016b6sm14016299ejc.97.2023.12.18.00.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 00:44:14 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 3/7] sstatesig/find_siginfo: unify a disjointed API Date: Mon, 18 Dec 2023 09:43:59 +0100 Message-Id: <20231218084403.599015-3-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231218084403.599015-1-alex@linutronix.de> References: <20231218084403.599015-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Dec 2023 08:44:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192615 find_siginfo() returns two different data structures depending on whether its third argument (list of hashes to find) is empty or not: - a dict of timestamps keyed by path - a dict of paths keyed by hash This is not a good API design; it's much better to return a dict of dicts that include both timestamp and path, keyed by hash. Then the API consumer can decide how they want to use these fields, particularly for additional diagnostics or informational output. I also took the opportunity to add a binary field that tells if the match came from sstate or local stamps dir, which will help prioritize local stamps when looking up most recent task signatures. Signed-off-by: Alexander Kanavin --- meta/lib/oe/buildhistory_analysis.py | 2 +- meta/lib/oe/sstatesig.py | 31 +++++++++------------ meta/lib/oeqa/selftest/cases/sstatetests.py | 10 +++---- 3 files changed, 19 insertions(+), 24 deletions(-) diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py index b1856846b6a..4edad01580c 100644 --- a/meta/lib/oe/buildhistory_analysis.py +++ b/meta/lib/oe/buildhistory_analysis.py @@ -562,7 +562,7 @@ def compare_siglists(a_blob, b_blob, taskdiff=False): elif not hash2 in hashfiles: out.append("Unable to find matching sigdata for %s with hash %s" % (desc, hash2)) else: - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, collapsed=True) + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1]['path'], hashfiles[hash2]['path'], recursecb, collapsed=True) for line in out2: m = hashlib.sha256() m.update(line.encode('utf-8')) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index 8a97fb0c04b..0342bcdc87a 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -349,7 +349,6 @@ def find_siginfo(pn, taskname, taskhashlist, d): pn, taskname = key.split(':', 1) hashfiles = {} - filedates = {} def get_hashval(siginfo): if siginfo.endswith('.siginfo'): @@ -357,6 +356,12 @@ def find_siginfo(pn, taskname, taskhashlist, d): else: return siginfo.rpartition('.')[2] + def get_time(fullpath): + try: + return os.stat(fullpath).st_mtime + except OSError: + return None + # First search in stamps dir localdata = d.createCopy() localdata.setVar('MULTIMACH_TARGET_SYS', '*') @@ -372,24 +377,21 @@ def find_siginfo(pn, taskname, taskhashlist, d): filespec = '%s.%s.sigdata.*' % (stamp, taskname) foundall = False import glob + bb.debug(1, "Calling glob.glob on {}".format(filespec)) for fullpath in glob.glob(filespec): match = False if taskhashlist: for taskhash in taskhashlist: if fullpath.endswith('.%s' % taskhash): - hashfiles[taskhash] = fullpath + hashfiles[taskhash] = {'path':fullpath, 'sstate':False, 'time':get_time(fullpath)} if len(hashfiles) == len(taskhashlist): foundall = True break else: - try: - filedates[fullpath] = os.stat(fullpath).st_mtime - except OSError: - continue hashval = get_hashval(fullpath) - hashfiles[hashval] = fullpath + hashfiles[hashval] = {'path':fullpath, 'sstate':False, 'time':get_time(fullpath)} - if not taskhashlist or (len(filedates) < 2 and not foundall): + if not taskhashlist or (len(hashfiles) < 2 and not foundall): # That didn't work, look in sstate-cache hashes = taskhashlist or ['?' * 64] localdata = bb.data.createCopy(d) @@ -412,22 +414,15 @@ def find_siginfo(pn, taskname, taskhashlist, d): localdata.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/") filespec = '%s.siginfo' % localdata.getVar('SSTATE_PKG') + bb.debug(1, "Calling glob.glob on {}".format(filespec)) matchedfiles = glob.glob(filespec) for fullpath in matchedfiles: actual_hashval = get_hashval(fullpath) if actual_hashval in hashfiles: continue - hashfiles[hashval] = fullpath - if not taskhashlist: - try: - filedates[fullpath] = os.stat(fullpath).st_mtime - except: - continue + hashfiles[actual_hashval] = {'path':fullpath, 'sstate':True, 'time':get_time(fullpath)} - if taskhashlist: - return hashfiles - else: - return filedates + return hashfiles bb.siggen.find_siginfo = find_siginfo diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 6af3c8f2178..6da3ef3ca77 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -765,14 +765,14 @@ addtask tmptask2 before do_tmptask1 hashes = [hash1, hash2] hashfiles = find_siginfo(key, None, hashes) self.assertCountEqual(hashes, hashfiles) - bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) + bb.siggen.compare_sigfiles(hashfiles[hash1]['path'], hashfiles[hash2]['path'], recursecb) for pn in pns: recursecb_count = 0 - filedates = find_siginfo(pn, "do_tmptask1") - self.assertGreaterEqual(len(filedates), 2) - latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:] - bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) + matches = find_siginfo(pn, "do_tmptask1") + self.assertGreaterEqual(len(matches), 2) + latesthashes = sorted(matches.keys(), key=lambda h: matches[h]['time'])[-2:] + bb.siggen.compare_sigfiles(matches[latesthashes[-2]]['path'], matches[latesthashes[-1]]['path'], recursecb) self.assertEqual(recursecb_count,1) class SStatePrintdiff(SStateBase): From patchwork Mon Dec 18 08:44:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 116F9C4706C for ; Mon, 18 Dec 2023 08:44:24 +0000 (UTC) Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by mx.groups.io with SMTP id smtpd.web11.40233.1702889057294171911 for ; Mon, 18 Dec 2023 00:44:17 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PiIlKsdu; spf=pass (domain: gmail.com, ip: 209.85.218.43, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-a234205785dso119417866b.0 for ; Mon, 18 Dec 2023 00:44:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702889056; x=1703493856; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dKH98MYBC6ceZUzA3zyk4IIY+pdooUGAOo7iJfRaySc=; b=PiIlKsdup0Auz0ieTJUa/3XkvjdEZx+YjDnoFiYdtsukYP/TNhcpkAs+z2fb844mU6 tzHN81UJLSX+eePoBO36K0Zwe7KaTAGlwZ1AIonUC3sCkT/m4wyyjDqbxs3CdD8yS0X/ ybxRLzPUABG14mI34xUnSRgoOyfBID6haxRbI+B0oJS8758cJpTMwC9x1tYZeMU7ylYS I1swZjVak0dZ+yeFNCd5rswqOSIoG8o3cHTEo1oozRyX5btaHckSWjOTwfF6PYdVyNGp MmyQgk/1i+cXqqsOfNuR18cHZp6yBPkk4VJbBKk0o4m9VoNc2PT+tI9afs3seyV96dgs OFqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702889056; x=1703493856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dKH98MYBC6ceZUzA3zyk4IIY+pdooUGAOo7iJfRaySc=; b=AA3SJm/LjLwxQ4mC+PhBKFhYogom1WMTdNKGShcoTvn0e/RFtnXoiEv7kw4wjDCnA9 Z5e7hxWgxpKf5ZkmJU0ON2y4GgFRnLGUE2K96xIVDnonyVSzp1Ze8BZpETxalSjzlhLk Y6CuDJqjLkZ39ViP92YmzX/X96+u+WzYUTogCIZssRUA1OXqb3vkIGY4Qz6ymEidok0a 8uiH9vg3c7XJvYAoMtepbPWKUMK2uZWO0iqyBLbixZIueT4uaJ8EZ4cHNrwsH2uU2YWX CFxWg2E6pE0Qd8aY2dyKVWwnt8Vtk1FOWjApBcaK+LNzZZG9lBhIpLRLx8Z5Utj29nA0 CqGA== X-Gm-Message-State: AOJu0YzEYv4MzUBJuKjMIy4xjU4XHeYhqhA7BOzy5GNOaLYTglESpiEz Xg9dRYIRRtwQ2EfixdCL/qiFYQ3HLVw= X-Google-Smtp-Source: AGHT+IH9Ypp6oK2aWfYqEDqzUqTWpDbnaOUIYJ6C5uiEnzKtUnUZMg723g1DM0xrPOUWyXnGhKv2HQ== X-Received: by 2002:a17:907:720d:b0:a19:d40a:d222 with SMTP id dr13-20020a170907720d00b00a19d40ad222mr4798625ejc.238.1702889055555; Mon, 18 Dec 2023 00:44:15 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id vt6-20020a170907a60600b00a1ce98016b6sm14016299ejc.97.2023.12.18.00.44.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 00:44:15 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 4/7] bitbake-diffsigs/runqueue: adapt to reworked find_siginfo() Date: Mon, 18 Dec 2023 09:44:00 +0100 Message-Id: <20231218084403.599015-4-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231218084403.599015-1-alex@linutronix.de> References: <20231218084403.599015-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Dec 2023 08:44:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192614 In particular having 'time' explicitly used as a sorting key should make it more clear how the entries are being sorted. Signed-off-by: Alexander Kanavin --- bitbake/bin/bitbake-diffsigs | 11 +++++++---- bitbake/lib/bb/runqueue.py | 10 +++++----- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs index fe0f33eea17..a8f49191b0c 100755 --- a/bitbake/bin/bitbake-diffsigs +++ b/bitbake/bin/bitbake-diffsigs @@ -72,13 +72,16 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): elif sig2 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2)) sys.exit(1) - latestfiles = [sigfiles[sig1], sigfiles[sig2]] else: - filedates = find_siginfo(bbhandler, pn, taskname) - latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:] - if not latestfiles: + sigfiles = find_siginfo(bbhandler, pn, taskname) + latestsigs = sorted(sigfiles.keys(), key=lambda h: sigfiles[h]['time'])[-2:] + if not latestsigs: logger.error('No sigdata files found matching %s %s' % (pn, taskname)) sys.exit(1) + sig1 = latestsigs[0] + sig2 = latestsigs[1] + + latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']] return latestfiles diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index f54d9b85541..72d2feab27a 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1768,7 +1768,7 @@ class RunQueue: recout = [] if len(hashfiles) == 2: - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1]['path'], hashfiles[hash2]['path'], recursecb) recout.extend(list(' ' + l for l in out2)) else: recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) @@ -1782,14 +1782,14 @@ class RunQueue: h = self.rqdata.runtaskentries[tid].unihash matches = bb.siggen.find_siginfo(pn, taskname, [], self.cooker.databuilder.mcdata[mc]) match = None - for m in matches: - if h in m: - match = m + for m in matches.values(): + if h in m['path']: + match = m['path'] if match is None: bb.fatal("Can't find a task we're supposed to have written out? (hash: %s tid: %s)?" % (h, tid)) matches = {k : v for k, v in iter(matches.items()) if h not in k} if matches: - latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1] + latestmatch = matches[sorted(matches.keys(), key=lambda h: matches[h]['time'])[-1]]['path'] prevh = __find_sha256__.search(latestmatch).group(0) output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb) bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, most recent matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output)) From patchwork Mon Dec 18 08:44:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05F4DC46CD2 for ; Mon, 18 Dec 2023 08:44:24 +0000 (UTC) Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by mx.groups.io with SMTP id smtpd.web10.39801.1702889057665910752 for ; Mon, 18 Dec 2023 00:44:17 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PEWO9Es+; spf=pass (domain: gmail.com, ip: 209.85.218.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a2335d81693so217417166b.0 for ; Mon, 18 Dec 2023 00:44:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702889056; x=1703493856; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+yl0J49Lf+BNpHDbePEAizGhsMRxX2zjOOsRdhQrKtU=; b=PEWO9Es+LQ6mCwk1HtDOFpUU2roZj8X+owUCj93XjkoMFaCqXi0VewG81GxUxpK745 w/fw1AcyQkykFxTimaF0MK8Ok0fI8cmz/mMhcxdHPbslvo56omLOnXymXmieEdJTiu7r bllRvmp9iUJKuV4fBZc/a4wvxRWDalmDGuDKs5EbjzpoPm36q4mW8EjNToiuCjIBWdTc llXgcNbznl2/wjpNRQxYxvuNcShCDyhqaQE5l22PX8qF6HxOrsj5yVkrZl9PpEpk1iwf Re3oLdIWNxlyAL1r6MUIVoUzy65E1KAK/U6StAIsaixrScePlVgcFRo+xvRUCzO+abbl KRZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702889056; x=1703493856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+yl0J49Lf+BNpHDbePEAizGhsMRxX2zjOOsRdhQrKtU=; b=o6smxr3U6ofDaRMTpOKAqmkcGsd01O0MRkgD/rhD8Eakx8EDgfAFXEon3HKX3Kl1Z3 I7PqKbc2u5csIwVEbbM9QIPOEhpkoU3jY6HHv+X1eqT3UhMutq/TLMH8QVoSUFLv8pgJ m/PmspyBsMIP2YT3HkKnv6mB9qyrb/NJ9yI3RO/wRytCY4oW/G/w4YMTyLQFngHUXtuc J6UU0fFJy6FeCntQbjOgRRfTwjY1HfuIgtN2YH9iIhj3Q1WGf3soRnqa7edWvrUKoSE7 ovV2aWoOuEkvq5R8BrvBzt2Ulw3ADrVXxHzf9z3TaLwwyzVeDKM4C02KejJazAhZH9ym ecyQ== X-Gm-Message-State: AOJu0YwddG1OlC1spcYUOQC8xaxpijnrbZgODi3CjSRg1QMBKecOTkoP Yoptv/NqQA7aNNQ8facdh3s1ahl/P3g= X-Google-Smtp-Source: AGHT+IHpALCr+oKuc8BsP2qlD4kYCmquLsCUJs2Ym2oXXNpQ10Y0vC93Ivo077yvHQY3tjD/CCbpBg== X-Received: by 2002:a17:906:225d:b0:a0f:c538:4612 with SMTP id 29-20020a170906225d00b00a0fc5384612mr14270375ejr.28.1702889056001; Mon, 18 Dec 2023 00:44:16 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id vt6-20020a170907a60600b00a1ce98016b6sm14016299ejc.97.2023.12.18.00.44.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 00:44:15 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 5/7] bitbake/runqueue: prioritize local stamps over sstate signatures in printdiff Date: Mon, 18 Dec 2023 09:44:01 +0100 Message-Id: <20231218084403.599015-5-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231218084403.599015-1-alex@linutronix.de> References: <20231218084403.599015-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Dec 2023 08:44:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192616 Even with the reworked printdiff code, sstate which is heavily used in parallel can throw races at the tests: if a new matching, but otherwise unrelated sstate signature appears between writing out local stamps and listing matching sstate files, then that signature will be deemed 'the latest' and the actual local stamp will be discarded. This change ensures the scenario does not happen. It also makes use of the reworked find_siginfo(), particularly the 'sstate' entry in returned results. Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/runqueue.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 72d2feab27a..0fc26ccdf88 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1788,6 +1788,9 @@ class RunQueue: if match is None: bb.fatal("Can't find a task we're supposed to have written out? (hash: %s tid: %s)?" % (h, tid)) matches = {k : v for k, v in iter(matches.items()) if h not in k} + matches_local = {k : v for k, v in iter(matches.items()) if h not in k and not v['sstate']} + if matches_local: + matches = matches_local if matches: latestmatch = matches[sorted(matches.keys(), key=lambda h: matches[h]['time'])[-1]]['path'] prevh = __find_sha256__.search(latestmatch).group(0) From patchwork Mon Dec 18 08:44:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC1BAC46CD4 for ; Mon, 18 Dec 2023 08:44:23 +0000 (UTC) Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by mx.groups.io with SMTP id smtpd.web10.39802.1702889058288771178 for ; Mon, 18 Dec 2023 00:44:18 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Z0S2fn0f; spf=pass (domain: gmail.com, ip: 209.85.208.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-54c5d041c23so3039260a12.2 for ; Mon, 18 Dec 2023 00:44:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702889057; x=1703493857; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5Ko7dfscRWyOYsoLBA2bq9GBOUV048i1Gtasxg3jLXw=; b=Z0S2fn0fq06lqNN3Dlrj9f7SQ53q8CCX/QMp66JGXeiNtzlrOzrKY++irSK3vp3KML wNMnQVV0gFckngzacFwnDp5SU0acX+1ZpZKt+SQzG9NsmukXhdeRSIpNSdIstkXAgupy jDtecfYeuA0OK5EFP+tkq31rRjHfQ3aFyZi5LErEsdBHJtWdtKEkD+ic/r+UDkdC+xWG vmhxfePMK/pl3YzhexywhYmxCmOh66ySRPzEyYnL+gZnlaLDXgzuQCZ5dAAzSjWWRLMR 8h6p+neOIiq9UCuPlftw2HRa6xCk1V+F08YZhk15UgDCZxIDW9zy271t4tzXrSLn5NxK 8ZzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702889057; x=1703493857; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5Ko7dfscRWyOYsoLBA2bq9GBOUV048i1Gtasxg3jLXw=; b=wqcG3luedlQ5k4N/gZlUlUpL74LgzZgUFFIvw0lrgwIuyoyrV8JUd7WSYCDkiQIfss iBW/7uKStumPxWq2a74BtB+cHlpboqq2WVygRiM66L9L0ediJiFy0NfaApW12qtyVGal M1yD8wDBFKBrQfM+QxIlzM7JlC0y2QCTDf0aTRgcw8m8yco83Gta6fs9Rq/C00u6RKwg LZK1Vso9s2q6sIoOOWhQYsRlhKZhbIdECTYogTZCXHknpbAtffR8sytn7JepaakilNxo Aee1gCDnEQmIF+cbDvVsNT5rLpXP4UAICQk+MDHZ1xmtmI9x4JCyIBs1ueBNwAskvDCf b8gw== X-Gm-Message-State: AOJu0YxgG3hGXpPRdu7BCMYFeb3ICIE3tgoZD1uDqwAks6u72GU2GZB7 a+1wMRv6ORqkhAsV74nl07yWKIHLinU= X-Google-Smtp-Source: AGHT+IFgyTLbkEUSZ60wWMw0vjpKb9aVFOONJNS/59j+4iOyUkd3j02N2v87cNlQHyMjW1naMh7svg== X-Received: by 2002:a17:906:5184:b0:a1d:b9d2:2aea with SMTP id y4-20020a170906518400b00a1db9d22aeamr3014480ejk.280.1702889056501; Mon, 18 Dec 2023 00:44:16 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id vt6-20020a170907a60600b00a1ce98016b6sm14016299ejc.97.2023.12.18.00.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 00:44:16 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 6/7] bitbake/runqueue: add debugging for find_siginfo() calls Date: Mon, 18 Dec 2023 09:44:02 +0100 Message-Id: <20231218084403.599015-6-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231218084403.599015-1-alex@linutronix.de> References: <20231218084403.599015-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Dec 2023 08:44:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192617 Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/runqueue.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 0fc26ccdf88..51959231b72 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1764,7 +1764,9 @@ class RunQueue: # Define recursion callback def recursecb(key, hash1, hash2): hashes = [hash1, hash2] + bb.debug(1, "Recursively looking for recipe {} hashes {}".format(key, hashes)) hashfiles = bb.siggen.find_siginfo(key, None, hashes, self.cfgData) + bb.debug(1, "Found hashfiles:\n{}".format(hashfiles)) recout = [] if len(hashfiles) == 2: @@ -1780,7 +1782,9 @@ class RunQueue: (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] h = self.rqdata.runtaskentries[tid].unihash + bb.debug(1, "Looking for recipe {} task {}".format(pn, taskname)) matches = bb.siggen.find_siginfo(pn, taskname, [], self.cooker.databuilder.mcdata[mc]) + bb.debug(1, "Found hashfiles:\n{}".format(matches)) match = None for m in matches.values(): if h in m['path']: From patchwork Mon Dec 18 08:44:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 36533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9B54C35274 for ; Mon, 18 Dec 2023 08:44:23 +0000 (UTC) Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by mx.groups.io with SMTP id smtpd.web11.40235.1702889058632475346 for ; Mon, 18 Dec 2023 00:44:18 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YJQqMdhC; spf=pass (domain: gmail.com, ip: 209.85.218.41, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a235eb41251so46350366b.3 for ; Mon, 18 Dec 2023 00:44:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702889057; x=1703493857; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H1GVh7bIHF7CLeoOrEXTpECP0j/CvgbnNp+1vHC+XMg=; b=YJQqMdhCVDRZ9CAD0GhRIA7xikVpRG1DR3YTlCuS/5mXccL3LfIY/kUH334H6oUkn5 TGcQ1U0jmU0qI+BCWCUw1z+JSog4ZK/sK/qM08rjZFWQAhgYr1vJxiMA8ehLGT8Ix6wi bWxGIknJBj4LUX7OMYPlckFAuX4HQpB83pAb1cprp8ncWj7v5mQSPTCfL/iUr3C1ECqP UaxCCVveGJEVP6wQiXGn7A47svI4sBT4qyoZ2bB9Z6LoGyfvRWOjWepz6cCO7plSSO14 ZCBjqZ27M4PoKFSgi7ra1wVgR5dDDVW02UqLMrEKsS1gYSmkc+feWeZxwHOvXqqLx+bQ Y4BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702889057; x=1703493857; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H1GVh7bIHF7CLeoOrEXTpECP0j/CvgbnNp+1vHC+XMg=; b=IdEolkeJvp+rU4qYHO7jNsTZEig4EaPJCbtjh1ufPcE4hTkWWPihIeFtNhwMwxV8tB Sl2JoNgRSe0DTMEuvOe5kw1L7a6oPSvjuLQhhzjufXQ8FIjPy/BCq/XHtpDoc91zxoCY 0YCRSLSpZbyKEYkEpAtYS6Xe121tb6GGwvBq8Qr8ri7Z8+QXGTcG/bnKFEFc1Hx6sBXD n/7nYURcRlbJVePQGBZqAGjEssidMZJ1HGC/Oe5fY4+GbXj0gW3rc6xq5eCzuyN45z+j HUs/upYuma4T2MnHINdn/bxcf5PwkVKYuFbyM/u4+nWNuRdhwJ2ceOwhEFWwdLFKjNuH hwgg== X-Gm-Message-State: AOJu0Yyv1r7lNrf4h6N9/Vcu6pykUhSBDLRc9kdTVgU8lVd5jTpzgZS1 PjtAiOTb8LLDVt2uc+QaRbYwlKmmIeA= X-Google-Smtp-Source: AGHT+IEub0SxbomKZIbcWGE0pTPGbFDxDX276YT75TESujoo3B/DwnzW3/vyueE4E+f49jxgCGgFjw== X-Received: by 2002:a17:906:1708:b0:a19:a19b:4238 with SMTP id c8-20020a170906170800b00a19a19b4238mr4859378eje.163.1702889057111; Mon, 18 Dec 2023 00:44:17 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id vt6-20020a170907a60600b00a1ce98016b6sm14016299ejc.97.2023.12.18.00.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 00:44:16 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 7/7] selftest/sstatetest: re-enable gcc printdiff test Date: Mon, 18 Dec 2023 09:44:03 +0100 Message-Id: <20231218084403.599015-7-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231218084403.599015-1-alex@linutronix.de> References: <20231218084403.599015-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Dec 2023 08:44:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192618 Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/sstatetests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 6da3ef3ca77..62b7a79bbb6 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -842,7 +842,7 @@ do_install:append() { expected_sametmp_output, expected_difftmp_output) # Check if changes to gcc-source (which uses tmp/work-shared) are correctly discovered - def _test_gcc_runtime_vs_gcc_source(self): + def test_gcc_runtime_vs_gcc_source(self): gcc_source_pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV'] expected_output = ("Task {}:do_preconfigure couldn't be used from the cache because:".format(gcc_source_pn),