From patchwork Mon Nov 6 16:43:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 33944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E53FAC4332F for ; Mon, 6 Nov 2023 16:43:51 +0000 (UTC) Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com [209.85.208.180]) by mx.groups.io with SMTP id smtpd.web11.59035.1699289024292329937 for ; Mon, 06 Nov 2023 08:43:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=e8IQelqp; spf=pass (domain: linuxfoundation.org, ip: 209.85.208.180, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-lj1-f180.google.com with SMTP id 38308e7fff4ca-2c54c8934abso66623061fa.0 for ; Mon, 06 Nov 2023 08:43:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1699289022; x=1699893822; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=ABSbCthUsh3EikMUMzObF10+P7j+8I2ilavVZ3UxJWE=; b=e8IQelqpvAhljbLZGdh+3Sv3DPth/e5mYTbLj7mJteUI8qipVzxYXpp5uYRTXsgLUc BskNMmoF+X5aokxJt/sJKDN6qrwlwn4qnYWY43IH2Dknyrr2Y1m6g2ibkC0r7Pcy1cWN 1LUDv12rj4Hks51w2zZ/OU70FrEU2b9M+y9Jg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699289022; x=1699893822; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ABSbCthUsh3EikMUMzObF10+P7j+8I2ilavVZ3UxJWE=; b=UAoUovwzlL+OagQQ0rNWxoLNI66EBK44feQkQqAG4oxk+YgUeqIpXWWk5VPvbfzUoT 5UIWOA8VuOT5Vy1zY/W4ml2Eibt3dajC6qDabyGMOChxwZzTtTC0kCFwQp+0Dl+y7rtG LhhNtwFdEA4in0OVKW1UhVlMcQXkNMg5k5DfuIa0HyIKOTvPMdLG4M4UZuQN0tc1NR4H OSlAu06E2TU5IOJjwJI0TkwfKGTexwPzccEBSEkMoIlvcNeJvysEV0aIxqDHB4JtLWSy LMX/GawDEZElg4zceH4JMZQkrvA+gbM6g3CsHIZicGYcVJ6RLpS53Ylx+s2pLYz8L+UL YJrw== X-Gm-Message-State: AOJu0YwNb2y8H5YcHDv63rjGCRySdN2zCDxkblRQt6bN8CNg9GX3Ii9c w/VhfWLkGOCfRvwJvrn0994nSEDp+ohyZm/0sGE= X-Google-Smtp-Source: AGHT+IFeXtCtKjQeDka3+IOGntdEcUggvDfDalghTLI3kBbue1+3w5QKjKsbuJWQ3xDhRKU4rYq20Q== X-Received: by 2002:a2e:9e4c:0:b0:2c5:19f2:4fde with SMTP id g12-20020a2e9e4c000000b002c519f24fdemr20707763ljk.23.1699289021577; Mon, 06 Nov 2023 08:43:41 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1ccc:c310:74c7:1597]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003fefb94ccc9sm13131383wmq.11.2023.11.06.08.43.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Nov 2023 08:43:41 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/3] sstate: Ensure sstate searches update file mtime Date: Mon, 6 Nov 2023 16:43:38 +0000 Message-Id: <20231106164340.1213043-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Nov 2023 16:43:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190235 Commands like "bitbake XXX -S printdiff" search for sstate files but don't download them. This means that local files aren't touched as the download code would do, meaning the sstate cleanup scripts can delete them. This can then lead to obtuse build failures. Have the search code touch local files in the same way as the main code paths would to avoid these files disappearing. Move the function to a common touch() function in lib/oe instead of duplicating code. Signed-off-by: Richard Purdie --- meta/classes-global/sstate.bbclass | 12 ++---------- meta/lib/oe/utils.py | 12 ++++++++++++ 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 2676f18e0a2..5b27a1f0f98 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -937,6 +937,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, sstatefile = d.expand("${SSTATE_DIR}/" + getsstatefile(tid, siginfo, d)) if os.path.exists(sstatefile): + oe.utils.touch(sstatefile) found.add(tid) bb.debug(2, "SState: Found valid sstate file %s" % sstatefile) else: @@ -1183,16 +1184,7 @@ python sstate_eventhandler() { if not os.path.exists(siginfo): bb.siggen.dump_this_task(siginfo, d) else: - try: - os.utime(siginfo, None) - except PermissionError: - pass - except OSError as e: - # Handle read-only file systems gracefully - import errno - if e.errno != errno.EROFS: - raise e - + oe.utils.touch(siginfo) } SSTATE_PRUNE_OBSOLETEWORKDIR ?= "1" diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index a3b1bb1087d..14a7d07ef01 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -7,6 +7,7 @@ import subprocess import multiprocessing import traceback +import errno def read_file(filename): try: @@ -528,3 +529,14 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root)) return total + +# Update the mtime of a file, skip if permission/read-only issues +def touch(filename): + try: + os.utime(filename, None) + except PermissionError: + pass + except OSError as e: + # Handle read-only file systems gracefully + if e.errno != errno.EROFS: + raise e From patchwork Mon Nov 6 16:43:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 33945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6233C4167B for ; Mon, 6 Nov 2023 16:43:51 +0000 (UTC) Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by mx.groups.io with SMTP id smtpd.web11.59036.1699289024391528865 for ; Mon, 06 Nov 2023 08:43:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=IW5+UI5V; spf=pass (domain: linuxfoundation.org, ip: 209.85.208.176, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2c4fdf94666so62518791fa.2 for ; Mon, 06 Nov 2023 08:43:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1699289022; x=1699893822; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=oTXpOLhFXoHXSH7iP+kc5OQjeF6U6M+NxEV2vdT615g=; b=IW5+UI5Vsn8M0313mPpb6T5PT2VxlhBti4U+usIqOGGbH5RYxGioAk96lSrV0nxFr9 HTdGUev8/T1eV4xyiQ/gS1z75R5vv36KEgx+d54a4gEuXxOf6nomB5K3HwRF55ytEAaG Lu06Hcz/ahkNm3xk4VZ3XCo/PyWLt0oeh/jcM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699289022; x=1699893822; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oTXpOLhFXoHXSH7iP+kc5OQjeF6U6M+NxEV2vdT615g=; b=rXXIYUsVVmXJFGzr5QuzxL/sjVxjxnRi41luSWSuj1Rxcaa1YF5nPmR+mqKkJeKW21 pnW0IK8IaUFovlbe1ikFReLjGEDdYrxYrJT+9ZOubCexP2oTXbVTzKYhVW33Djg68ZEh wlJ+mR/EddjuxG0+83JlQkd48tsuwixZdfYS0V/I4z+VsvyFlJ6C/D0Dvdb6YJ2oxskb jUnmxlTnTCTR1TDQzSBIbPqr5vV8SlnGQyhx+qDOO3UihtuK5Fxni10Ebm0a3QeweJbp spnh4YxO5iNLL1XjGkJIuiaO3hX7K1YFCiVSj5SGfEAoeNSMLi9v9DWzQQ4mUwAi2csz /rlA== X-Gm-Message-State: AOJu0Yxjvz0adUDzfe2oFJgmsA3Iks/yFXLAW1eBszkvyHWZdJPMI4IF pCP7KAIUKM6x1uyzZSW345lSArC5DgKlZtlLTmI= X-Google-Smtp-Source: AGHT+IHD5jz0hS7UrHANsZmhy+EhZDDzT783TLFA5hIpjaiuwH5GhBP13oUvT+DuEakDTplCb1VA7Q== X-Received: by 2002:a2e:3307:0:b0:2c6:ef64:3164 with SMTP id d7-20020a2e3307000000b002c6ef643164mr10264019ljc.17.1699289022255; Mon, 06 Nov 2023 08:43:42 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1ccc:c310:74c7:1597]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003fefb94ccc9sm13131383wmq.11.2023.11.06.08.43.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Nov 2023 08:43:41 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/3] insane: Move unpack tests to do_recipe_qa Date: Mon, 6 Nov 2023 16:43:39 +0000 Message-Id: <20231106164340.1213043-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231106164340.1213043-1-richard.purdie@linuxfoundation.org> References: <20231106164340.1213043-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Nov 2023 16:43:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190236 The SRC_URI tests are a better fit for the new do_recipe_qa task, move them there. Signed-off-by: Richard Purdie --- meta/classes-global/insane.bbclass | 46 ++++++++++++++---------------- 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 307a09d115e..020865138f5 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -1513,35 +1513,18 @@ Rerun configure task after fixing this.""" oe.qa.exit_if_errors(d) } -def unpack_check_src_uri(pn, d): - import re - - skip = (d.getVar('INSANE_SKIP') or "").split() - if 'src-uri-bad' in skip: - bb.note("Recipe %s skipping qa checking: src-uri-bad" % d.getVar('PN')) - return - - if "${PN}" in d.getVar("SRC_URI", False): - oe.qa.handle_error("src-uri-bad", "%s: SRC_URI uses PN not BPN" % pn, d) - - for url in d.getVar("SRC_URI").split(): - # Search for github and gitlab URLs that pull unstable archives (comment for future greppers) - if re.search(r"git(hu|la)b\.com/.+/.+/archive/.+", url) or "//codeload.github.com/" in url: - oe.qa.handle_error("src-uri-bad", "%s: SRC_URI uses unstable GitHub/GitLab archives, convert recipe to use git protocol" % pn, d) - python do_qa_unpack() { src_uri = d.getVar('SRC_URI') s_dir = d.getVar('S') if src_uri and not os.path.exists(s_dir): bb.warn('%s: the directory %s (%s) pointed to by the S variable doesn\'t exist - please set S within the recipe to point to where the source has been unpacked to' % (d.getVar('PN'), d.getVar('S', False), s_dir)) - - unpack_check_src_uri(d.getVar('PN'), d) } python do_recipe_qa() { - def test_missing_metadata(d): + import re + + def test_missing_metadata(pn, d): fn = d.getVar("FILE") - pn = d.getVar('BPN') srcfile = d.getVar('SRC_URI').split() # Check that SUMMARY is not the same as the default from bitbake.conf if d.getVar('SUMMARY') == d.expand("${PN} version ${PV}-${PR}"): @@ -1553,16 +1536,31 @@ python do_recipe_qa() { else: oe.qa.handle_error("missing-metadata", "Recipe {} in {} does not contain a HOMEPAGE. Please add an entry.".format(pn, fn), d) - def test_missing_maintainer(d): + def test_missing_maintainer(pn, d): fn = d.getVar("FILE") - pn = d.getVar("PN") if pn.endswith("-native") or pn.startswith("nativesdk-") or "packagegroup-" in pn or "core-image-ptest-" in pn: return if not d.getVar('RECIPE_MAINTAINER'): oe.qa.handle_error("missing-maintainer", "Recipe {} in {} does not have an assigned maintainer. Please add an entry into meta/conf/distro/include/maintainers.inc.".format(pn, fn), d) - test_missing_metadata(d) - test_missing_maintainer(d) + def test_srcuri(pn, d): + skip = (d.getVar('INSANE_SKIP') or "").split() + if 'src-uri-bad' in skip: + bb.note("Recipe %s skipping qa checking: src-uri-bad" % pn) + return + + if "${PN}" in d.getVar("SRC_URI", False): + oe.qa.handle_error("src-uri-bad", "%s: SRC_URI uses PN not BPN" % pn, d) + + for url in d.getVar("SRC_URI").split(): + # Search for github and gitlab URLs that pull unstable archives (comment for future greppers) + if re.search(r"git(hu|la)b\.com/.+/.+/archive/.+", url) or "//codeload.github.com/" in url: + oe.qa.handle_error("src-uri-bad", "%s: SRC_URI uses unstable GitHub/GitLab archives, convert recipe to use git protocol" % pn, d) + + pn = d.getVar('PN') + test_missing_metadata(pn, d) + test_missing_maintainer(pn, d) + test_srcuri(pn, d) oe.qa.exit_if_errors(d) } From patchwork Mon Nov 6 16:43:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 33946 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB167C0018A for ; Mon, 6 Nov 2023 16:43:51 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web10.59711.1699289025182276571 for ; Mon, 06 Nov 2023 08:43:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Ff29BUze; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.42, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-40806e4106dso28751605e9.1 for ; Mon, 06 Nov 2023 08:43:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1699289023; x=1699893823; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=GQWAgVG2OBBa1TX5CczJdBHhTg8KYSIRfXuR/mb+biA=; b=Ff29BUzeDvCJBKXWbUraEr9TTQlngm2RwKTt/+M4g5utcCqMDQ7tMkGU70xIFomtpf kYCSc2EJJxQCm8ofVqCf3d0l54E8GM5N6hLpBDU8pMhUbC4idlLvKWUCSJm2B0+0m9Bg 2q3mWiKknApuqM6zwe3l3c8FcuL06OFbjAl/s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699289023; x=1699893823; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GQWAgVG2OBBa1TX5CczJdBHhTg8KYSIRfXuR/mb+biA=; b=OwHmL+sdI13gfxLZ9YwogxJw7rCHoEbix7Hz0jy2ylaV1LnwC2uUXESLCJOyg/eqQ+ zKNiD7uTtmgVF0uK9LxhjAonQF+rVa3dlBbO4A/1LEvHdTDSM6aYCkKsS9weZZsTNQKG fE7C0r5p2P+26L5XpXAJWyt/FzIdXapLavjqih0Y9hH9qNGJ6WPg6ULXohV7MnEtKL0g Nb41XpJgpJxjDCR30MU56q6SPM6aViR1Zr8kjW5jb6pJ0FHpQf85oJXy1/ZBvd49UxjW 3sclIIHsUpHXRVIPvm8pwS89L2dbig0pxZ366RlEP352A3O0zutfx42+JUwlIkVT0N1i Vv0A== X-Gm-Message-State: AOJu0YwZ1DrJ+n+uWPcCygoP+0sdHW8MlYvWFjayHfk58p880JsKnQGH B3h8oAwLPYg2twMBPc3mQsGEu+dRILIe1AEx5+4= X-Google-Smtp-Source: AGHT+IHvtwc7pXSedb2jMcLC22pyoh9xZfacXgZa8omtS6s7lyVrSIQz1Kv6SlHFAvZPfRmQIsN4NQ== X-Received: by 2002:a05:600c:45c6:b0:405:3924:3cad with SMTP id s6-20020a05600c45c600b0040539243cadmr22201wmo.15.1699289023264; Mon, 06 Nov 2023 08:43:43 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1ccc:c310:74c7:1597]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003fefb94ccc9sm13131383wmq.11.2023.11.06.08.43.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Nov 2023 08:43:42 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/3] go-vendor: Minor style tweaks Date: Mon, 6 Nov 2023 16:43:40 +0000 Message-Id: <20231106164340.1213043-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231106164340.1213043-1-richard.purdie@linuxfoundation.org> References: <20231106164340.1213043-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Nov 2023 16:43:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190237 Drop a len() usage that isn't needed, drop a comment that isn't needed and use bb.fatal() to end the task with an error. Signed-off-by: Richard Purdie --- meta/classes/go-vendor.bbclass | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.bbclass index 5b017b0b9da..2426bddfba4 100644 --- a/meta/classes/go-vendor.bbclass +++ b/meta/classes/go-vendor.bbclass @@ -41,9 +41,6 @@ def go_src_uri(repo, version, path=None, subdir=None, \ return src_uri python do_vendor_unlink() { - - # We unlink - go_import = d.getVar('GO_IMPORT') source_dir = d.getVar('S') linkname = os.path.join(source_dir, *['src', go_import, 'vendor']) @@ -58,9 +55,8 @@ python do_go_vendor() { src_uri = (d.getVar('SRC_URI') or "").split() - if len(src_uri) == 0: - bb.error("SRC_URI is empty") - return + if not src_uri: + bb.fatal("SRC_URI is empty") default_destsuffix = "git/src/import/vendor.fetch" fetcher = bb.fetch2.Fetch(src_uri, d)