From patchwork Tue Feb 14 15:28:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19A7CC05027 for ; Tue, 14 Feb 2023 15:28:55 +0000 (UTC) Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by mx.groups.io with SMTP id smtpd.web10.10123.1676388533445745013 for ; Tue, 14 Feb 2023 07:28:53 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=a5mAeElX; spf=softfail (domain: sakoman.com, ip: 209.85.214.182, mailfrom: steve@sakoman.com) Received: by mail-pl1-f182.google.com with SMTP id w5so17379155plg.8 for ; Tue, 14 Feb 2023 07:28:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=rp86yPkudrRq6/6EkvY8obX3gbfSDtCSAiMRIEZ0eBU=; b=a5mAeElXQuqkkhC4VLYcZdLrkl1N0whgpV77IaQh4PCMaJRLCqAh6b5IdYM7ewhg8E Y3+eCMoqe52XcxzmySGBbhiibjoynE8TImG7g6r7/CKuQH2ulki6Pf16aLP1dWiS0gWK fhCw+MfnEP9qeYShRdVCL8NufiFgvgV/z45laJJMXaocMvyPO1sa0LpBu1NiXyadBojX kM+XG8AuCdM1bP52hJVkOrJldA4s7T20kYrBD/JgFdlGuwfbygrxg6JJkPTQhjSKrhtN nVsDAzJwcBjL2usiPAJuC7RtMhjQB033P+8HFBU1NRQqmeZG2UQUmAVeOFlr7lkt2fZz m53A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rp86yPkudrRq6/6EkvY8obX3gbfSDtCSAiMRIEZ0eBU=; b=PD0bQmTCIlMmfERy8Pdi/5TtE76Z8Z//WZnNtN3oyVxEhd4iLLyRDjLCJWng/3CSTK RnJOkvIRB45LVfBW4kRt9znTho4zMbxWGeiHpaTTWQPpZc1yd/LQLE0qUWhsEepw8x9n jVG2OXjo/CB6GA4nGUbv/5chE3aBvKuRLi3t/0mwNNEFVQeamXBaure3c3dahhqBqWJV 0GjVMDdKDFRXStJN/1uB0dwx7bj4jsfFzPJe9/2SOQtTLW5XzUTW77f2vNYQCJEuyei5 8B9Ug47Zk0/KlGaGOc51Tc3z2aPoWQKdLOJo0Thsc6zOGW2UxDqfuEt39ipeV20C32EX jD/g== X-Gm-Message-State: AO0yUKVKhGnbyw/EEDYdwv1AaE1gDJfVuHc175tkYcX4o/UI1x4Yyaib J0ons5n7HyqsJ5X7V/fmYNJFHNst204n1m+NP+U= X-Google-Smtp-Source: AK7set+lb6/Jy4QSgM1xLxgqKYZP+iSEiL1QaLfOY8hyXEs28VIilY5HeXxHhccirXzAQFN3Ne3mRA== X-Received: by 2002:a17:90b:4c09:b0:22b:b832:d32 with SMTP id na9-20020a17090b4c0900b0022bb8320d32mr2918239pjb.9.1676388532512; Tue, 14 Feb 2023 07:28:52 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id m3-20020a17090a5a4300b00233cde36909sm5624203pji.21.2023.02.14.07.28.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Feb 2023 07:28:52 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 1/4] siggen: Fix inefficient string concatenation Date: Tue, 14 Feb 2023 05:28:41 -1000 Message-Id: <590ae6fde9da75db3a368e5c0d47920696c33ebf.1676388410.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 Feb 2023 15:28:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14415 From: Etienne Cordonnier As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython has an optimization for statements in the form "a = a + b" or "a += b". It seems that this line does not get optimized, because it has a form a = a + b + c: data = data + "./" + f.split("/./")[1] For that reason, it does a copy of data for each iteration, potentially copying megabytes of data for each iteration. Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds instead of 45 seconds on my test setup where SRC_URI points to a big directory. Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython: "do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b" However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup: data = ''.join((data, "./", f.split("/./")[1])) I have changed the other lines to also use += for consistency only, however those were in the form a = a + b and were optimized already. Co-authored-by: JJ Robertson Signed-off-by: Etienne Cordonnier Signed-off-by: Richard Purdie (cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717) Signed-off-by: Steve Sakoman --- lib/bb/siggen.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 9a20fc8e..cea3a538 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -329,19 +329,19 @@ class SignatureGeneratorBasic(SignatureGenerator): data = self.basehash[tid] for dep in self.runtaskdeps[tid]: - data = data + self.get_unihash(dep) + data += self.get_unihash(dep) for (f, cs) in self.file_checksum_values[tid]: if cs: if "/./" in f: - data = data + "./" + f.split("/./")[1] - data = data + cs + data += "./" + f.split("/./")[1] + data += cs if tid in self.taints: if self.taints[tid].startswith("nostamp:"): - data = data + self.taints[tid][8:] + data += self.taints[tid][8:] else: - data = data + self.taints[tid] + data += self.taints[tid] h = hashlib.sha256(data.encode("utf-8")).hexdigest() self.taskhash[tid] = h From patchwork Tue Feb 14 15:28:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B900C64ED6 for ; Tue, 14 Feb 2023 15:29:05 +0000 (UTC) Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by mx.groups.io with SMTP id smtpd.web10.10125.1676388535337673768 for ; Tue, 14 Feb 2023 07:28:55 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=xCCI0Qyb; spf=softfail (domain: sakoman.com, ip: 209.85.216.47, mailfrom: steve@sakoman.com) Received: by mail-pj1-f47.google.com with SMTP id d13-20020a17090ad3cd00b0023127b2d602so15781446pjw.2 for ; Tue, 14 Feb 2023 07:28:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=yDZQZ5e5bNYaIXGrexir6Bw3vRchVsJq2enTL0Yef3E=; b=xCCI0QybNjpv6UziHUXf2qtHWEdPYMG44LEqNlyZcguePbtm+Gk0cETyO5ZXxEM6f8 Jr2u/K8jbOiVlM366e6dRe7BT6Xmf0DPKO1HyjmTTL1R45zpEyYNhxSSeR74k91E7abm t33Nzp6c4KeaP1IbD4fHYgvFYqwNHrM6uGVLjM7dg8+zEmqA7+WCenCJyRAR6+3GkKa0 17XTZlEkohr9zZP9yv/LwaUrTiZSbrorn+9jD7rgFfDqsvJ2xn1dO+oMHbEVOGsgyT0I JteR5GdBZReo5KGw1QI2zYWV/gtWvkd2P4+hwJzgzGzOIl0J01tq2JVZE+pE5IqWvf5/ 6h6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yDZQZ5e5bNYaIXGrexir6Bw3vRchVsJq2enTL0Yef3E=; b=cbQ44ek1NudPYBFEY5ZH6cT3D+wz3FBRSuW5nGxEkX2lvGZYnwqHmQmUNMHkcWpMsJ ns1lmD8rH2hhheokekkeDehkMFCSSxaOfZSpsPWoz4fWdUqJWEPsnFgXnPHGgLrhgSra Ep66I0+GmNNkCPKh1YQ/pKtWt2JzOOnuuPxgY1UaoayUAjG1sJDs3rbwrdfYf7JCHjaa YhG3dTOIRwCSnIEL4UD9sqv5m3jfV8GkR0wa2p0FZ/OI1js1lrriPZw2ifMycrxFZCis tDcBGw1nlM+rcS74Xup6UVW6sQagfuAgyJVQrRGF3CaBslZ+jQ6Gbp4lT6tqE5zs5avk twnw== X-Gm-Message-State: AO0yUKXYCfTib7lhH1HlC5TJ98mri1nXGGv15tO4A6vWm0Z3NB8YWNhw UBGSUy2BDQlpNwNgH/Zeh4+uU8cHisMdE7xir24= X-Google-Smtp-Source: AK7set/vbqI/FO6n8AMrsmi1Ksaw5x0j0+4ZMaz/PCZ65yELZ9zkOsSa0Q/cfA541BMTY5eBcwKVqA== X-Received: by 2002:a17:90b:1e0a:b0:231:70e:964e with SMTP id pg10-20020a17090b1e0a00b00231070e964emr2864847pjb.35.1676388534347; Tue, 14 Feb 2023 07:28:54 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id m3-20020a17090a5a4300b00233cde36909sm5624203pji.21.2023.02.14.07.28.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Feb 2023 07:28:53 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 2/4] utils/ply: Update md5 to better report errors with hashlib Date: Tue, 14 Feb 2023 05:28:42 -1000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 Feb 2023 15:29:05 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14416 From: Mark Hatle In the case where hashlib is not available, the try would fail and fall through resulting in a backtrace on the usage of the 'sig'. The backtrace itself was confusing and made it difficult to determine what went wrong. Update the import to be in it's own try block with an appropriate message to indicate what went wrong. Note, the current version of ply all of this code has been restructured so this is not applicable upstream. Additionally, some versions of hashlib don't appear to implement the second FIPS related argument. Detect this and support both versions. Signed-off-by: Mark Hatle Signed-off-by: Mark Hatle Signed-off-by: Luca Ceresoli (cherry picked from commit 484ab42f440070c0369b81f5c69da860fa47a798) Signed-off-by: Steve Sakoman --- lib/bb/utils.py | 7 ++++++- lib/ply/yacc.py | 7 +++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index 66a8a08c..bca4830f 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -545,7 +545,12 @@ def md5_file(filename): Return the hex string representation of the MD5 checksum of filename. """ import hashlib - return _hasher(hashlib.new('MD5', usedforsecurity=False), filename) + try: + sig = hashlib.new('MD5', usedforsecurity=False) + except TypeError: + # Some configurations don't appear to support two arguments + sig = hashlib.new('MD5') + return _hasher(sig, filename) def sha256_file(filename): """ diff --git a/lib/ply/yacc.py b/lib/ply/yacc.py index 767c4e46..381b50cf 100644 --- a/lib/ply/yacc.py +++ b/lib/ply/yacc.py @@ -2798,7 +2798,14 @@ class ParserReflect(object): def signature(self): try: import hashlib + except ImportError: + raise RuntimeError("Unable to import hashlib") + try: sig = hashlib.new('MD5', usedforsecurity=False) + except TypeError: + # Some configurations don't appear to support two arguments + sig = hashlib.new('MD5') + try: if self.start: sig.update(self.start.encode('latin-1')) if self.prec: From patchwork Tue Feb 14 15:28:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2105EC64ED8 for ; Tue, 14 Feb 2023 15:29:05 +0000 (UTC) Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by mx.groups.io with SMTP id smtpd.web10.10122.1676388529746511584 for ; Tue, 14 Feb 2023 07:28:56 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=tW/PL8VB; spf=softfail (domain: sakoman.com, ip: 209.85.216.42, mailfrom: steve@sakoman.com) Received: by mail-pj1-f42.google.com with SMTP id n20-20020a17090aab9400b00229ca6a4636so20782747pjq.0 for ; Tue, 14 Feb 2023 07:28:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=0fUiHwEMeOqlHcPpm+H23tMICC9YYKwKtWnpWghw/ZA=; b=tW/PL8VBaP0NPEo+VTrcV0JVxA0ObbV0fp1ely5cvfmgojSBBXvp9M89ZdzMv1ZBCw mbmpziCzEpWgb17WFhJCkPw2b/qBbbryVE+RhO3DNRGPL/qC1eC+kFm4kcJ7P3wTsFGv xCrA1BygTfh48qSFTtvJFzqV0j7mVNlsWGpfDoZGAEgpoUr0TMF7WNT8l6wMquv5u3FG gftjLwwmFb+1dCJUs3bPTDKdkkotB7zyp/pmlW4ZRX41VV4b0YXd8yuFPNQkc+c5pvqB czcPnCrRbLUamFUDQrestyQ2y/XcDQpm13y51v28WDYd1i6x1z7FOERcCY5MdMGVVCgu 3D/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0fUiHwEMeOqlHcPpm+H23tMICC9YYKwKtWnpWghw/ZA=; b=yntv7xiNE9y4MFiU+VAu+FB1CNcDG5zGf8DEtmanaXRpD9lBpB9szB3FwyriZi9HOs m7E2nnVqjSZZd3o4Ii5C2+XKIe2IlmVUs7qluq6Uv7gXvBZ372bLOwjpZ9icd/a9Vsp/ cBwosz8kmI4XL43w3ur4kwmyl2M7/XmABLeh9csaKHrbWeoIyzC3QrMG3cotqnnqybuZ FqiZlZ79aLyLT7UdQwjm+kiA++tx6XFyyHbZ/Uo1i00A46EY3cWLcIKOPgJqYyuK1O6C nAw5oMV9KiMLGmR09PMRtz6K5PcbUUInWh+rloUECaDSGoE5/G/pz+O8EMKAX0eY4R1C 3k8A== X-Gm-Message-State: AO0yUKUKuOn6NjDQXGXe3SZ2gKev63gJYhQYhDGMsTEsM9Mm1imvX0MD tArUtUgR5dWR+LxWxANOMu4YtkibFskXYuGNPB0= X-Google-Smtp-Source: AK7set9gGA0k9mcQLQuw5YFRgWgyv+WWX/qg+Sc3nJNXsR0uERaJCfNOuc5lFABN7sLLNPJun6zekA== X-Received: by 2002:a17:90b:33cc:b0:233:d91e:e64a with SMTP id lk12-20020a17090b33cc00b00233d91ee64amr2924791pjb.10.1676388536073; Tue, 14 Feb 2023 07:28:56 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id m3-20020a17090a5a4300b00233cde36909sm5624203pji.21.2023.02.14.07.28.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Feb 2023 07:28:55 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 3/4] bitbake-diffsigs: Make PEP8 compliant Date: Tue, 14 Feb 2023 05:28:43 -1000 Message-Id: <60e05043f83c73a34cd154193e5c40d18a3ed3db.1676388410.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 Feb 2023 15:29:05 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14417 From: Marius Kriegerowski This ignores flake8 rules: * E402 module level import not at top of file * E501 line too long Signed-off-by: Marius Kriegerowski Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie (cherry picked from commit e8b176de448dc387c7a578c92b52aef28591038f) Signed-off-by: Steve Sakoman --- bin/bitbake-diffsigs | 49 ++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/bin/bitbake-diffsigs b/bin/bitbake-diffsigs index cf4cc706..fe0f33ee 100755 --- a/bin/bitbake-diffsigs +++ b/bin/bitbake-diffsigs @@ -11,6 +11,7 @@ import os import sys import warnings + warnings.simplefilter("default") import argparse import logging @@ -27,6 +28,7 @@ logger = bb.msg.logger_create(myname) is_dump = myname == 'bitbake-dumpsig' + def find_siginfo(tinfoil, pn, taskname, sigs=None): result = None tinfoil.set_event_mask(['bb.event.FindSigInfoResult', @@ -52,6 +54,7 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None): sys.exit(2) return result + def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): """ Find the most recent signature files for the specified PN/task """ @@ -63,10 +66,10 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): if not sigfiles: logger.error('No sigdata files found matching %s %s matching either %s or %s' % (pn, taskname, sig1, sig2)) sys.exit(1) - elif not sig1 in sigfiles: + elif sig1 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig1)) sys.exit(1) - elif not sig2 in sigfiles: + elif sig2 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2)) sys.exit(1) latestfiles = [sigfiles[sig1], sigfiles[sig2]] @@ -88,9 +91,9 @@ def recursecb(key, hash1, hash2): recout = [] if not hashfiles: recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) - elif not hash1 in hashfiles: + elif hash1 not in hashfiles: recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1)) - elif not hash2 in hashfiles: + elif hash2 not in hashfiles: recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) else: out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color) @@ -110,36 +113,36 @@ parser.add_argument('-D', '--debug', if is_dump: parser.add_argument("-t", "--task", - help="find the signature data file for the last run of the specified task", - action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) + help="find the signature data file for the last run of the specified task", + action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) parser.add_argument("sigdatafile1", - help="Signature file to dump. Not used when using -t/--task.", - action="store", nargs='?', metavar="sigdatafile") + help="Signature file to dump. Not used when using -t/--task.", + action="store", nargs='?', metavar="sigdatafile") else: parser.add_argument('-c', '--color', - help='Colorize the output (where %(metavar)s is %(choices)s)', - choices=['auto', 'always', 'never'], default='auto', metavar='color') + help='Colorize the output (where %(metavar)s is %(choices)s)', + choices=['auto', 'always', 'never'], default='auto', metavar='color') parser.add_argument('-d', '--dump', - help='Dump the last signature data instead of comparing (equivalent to using bitbake-dumpsig)', - action='store_true') + help='Dump the last signature data instead of comparing (equivalent to using bitbake-dumpsig)', + action='store_true') parser.add_argument("-t", "--task", - help="find the signature data files for the last two runs of the specified task and compare them", - action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) + help="find the signature data files for the last two runs of the specified task and compare them", + action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) parser.add_argument("-s", "--signature", - help="With -t/--task, specify the signatures to look for instead of taking the last two", - action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig')) + help="With -t/--task, specify the signatures to look for instead of taking the last two", + action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig')) parser.add_argument("sigdatafile1", - help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", - action="store", nargs='?') + help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", + action="store", nargs='?') parser.add_argument("sigdatafile2", - help="Second signature file to compare", - action="store", nargs='?') + help="Second signature file to compare", + action="store", nargs='?') options = parser.parse_args() if is_dump: @@ -157,7 +160,8 @@ if options.taskargs: with bb.tinfoil.Tinfoil() as tinfoil: tinfoil.prepare(config_only=True) if not options.dump and options.sigargs: - files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1]) + files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], + options.sigargs[1]) else: files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1]) @@ -166,7 +170,8 @@ if options.taskargs: output = bb.siggen.dump_sigfile(files[-1]) else: if len(files) < 2: - logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (options.taskargs[0], options.taskargs[1])) + logger.error('Only one matching sigdata file found for the specified task (%s %s)' % ( + options.taskargs[0], options.taskargs[1])) sys.exit(1) # Recurse into signature comparison From patchwork Tue Feb 14 15:28:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B44DC64EC7 for ; Tue, 14 Feb 2023 15:29:05 +0000 (UTC) Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by mx.groups.io with SMTP id smtpd.web11.9973.1676388538676270739 for ; Tue, 14 Feb 2023 07:28:58 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=lxStBUkb; spf=softfail (domain: sakoman.com, ip: 209.85.216.50, mailfrom: steve@sakoman.com) Received: by mail-pj1-f50.google.com with SMTP id mg23so8883582pjb.0 for ; Tue, 14 Feb 2023 07:28:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=H84wM51lTXM1aKimvF/tPHvA7HUiyvpAY2WjU2XgebI=; b=lxStBUkbFu46A0fubQinYMw7+WPu4HyGSKWj0MLsCvzMDRiru51JLZqxE6qp4kKnry XiEsK46PZquNKTvpFXTlqqoD15xzK+b/3iABL+xepx5EufM90U882fBA2vKXWwMixNO0 7RQKR3z47cnlTByoUujGHv+/+b+dNwqx/lOODMMyVe5hD7EtTYFGvWVo3Znds8RkhBif 1dAmXEl3l6frxjEce7PN/quBTAODE7K4bK2mHZnK66Rx0C6jyEdtXZrrDdkD36urP/k0 UOA2BZoVrllkcFg1k+LQGvyBqmJq15Z210Q8D9afkJw8HfMTlG9sCO7exz5IInaRctLa yr7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H84wM51lTXM1aKimvF/tPHvA7HUiyvpAY2WjU2XgebI=; b=t1ShWrBE5Mb+r9zEkXZXaYMMSoe8Xhy0sY4VYgpYJLoqkf6lo/qVNenl7ijT7kmRuK N7kS7lQY8uwjexoDfiNgF/Z0MO1dgTZTf4Q2u4/BLKZw7FzPd7ABwtjTa4ifZbcqn+Zx ltT1x1i2zpscjocwkAKirW6C1fKJP3IYu0M9XvjLLJsstfJL6Kg/zM3ivVgj4LwV+gm4 Sf9t24Zlet/0lV/J6nDrjtZX1UiC70rwNCJA+Ys13q2C737Tbmf++dBjFrikFjTrD+K4 /c2GSLBXvxbfu55DUXS6t4B/GBwQpJSWd1D/gysXRnMbzJMQETn5kVZInJ6J1w/eJyfv WzfQ== X-Gm-Message-State: AO0yUKXMZ8TdjxGb2u6OXc0TXwkRjG3/5OH+Hrewaoc5Oihzhheka9vv 5bOPJSjYROkI7dNDuB5pTCUNs1OJjaNQFKdRTQE= X-Google-Smtp-Source: AK7set8o9JBsn672aC5kTpVNUhGguWLwEHbP9H0ShxQv8oChTSpPsuuSirkbxN3FBX0xe3i/hE4HzQ== X-Received: by 2002:a17:90b:4f81:b0:233:bc73:a926 with SMTP id qe1-20020a17090b4f8100b00233bc73a926mr2981863pjb.20.1676388537786; Tue, 14 Feb 2023 07:28:57 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id m3-20020a17090a5a4300b00233cde36909sm5624203pji.21.2023.02.14.07.28.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Feb 2023 07:28:57 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 4/4] bitbake-diffsigs: break on first dependent task difference Date: Tue, 14 Feb 2023 05:28:44 -1000 Message-Id: <89f13cd4a927a73de98998c27082c63b07671525.1676388410.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 Feb 2023 15:29:05 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14418 From: "Schmidt, Adriaan" compare_sigfiles() recursively calculates differences on all dependent tasks with changed hashes. This is done in arbitrary/alphabetical order, and only the last of those results is returned, while everything else is discarded. This changes the behavior to instead return the first difference and not calculate any more, which significantly speeds up diffs of tasks with many dependencies. Signed-off-by: Adriaan Schmidt Signed-off-by: Richard Purdie (cherry picked from commit ea6a676c9aa2864c2eff40eea41ba09ce903a651) Signed-off-by: Steve Sakoman --- lib/bb/siggen.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index cea3a538..0a9ce0ed 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -1028,6 +1028,7 @@ def compare_sigfiles(a, b, recursecb=None, color=False, collapsed=False): # If a dependent hash changed, might as well print the line above and then defer to the changes in # that hash since in all likelyhood, they're the same changes this task also saw. output = [output[-1]] + recout + break a_taint = a_data.get('taint', None) b_taint = b_data.get('taint', None)