From patchwork Tue Sep 19 21:46:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 424D5CE79A8 for ; Tue, 19 Sep 2023 21:46:33 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web10.22617.1695159985276521142 for ; Tue, 19 Sep 2023 14:46:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=MKEIbaEB; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-3216ba1b01eso837299f8f.2 for ; Tue, 19 Sep 2023 14:46:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695159983; x=1695764783; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=uAnbdXaHah1ruKey2JGToMfh7fBQz2GVDoStz4MQTm0=; b=MKEIbaEBZ7UUX/faMl33CqvBhjoGFqlBhHzTrHbOseep4nTioRry5anTri+p5Tt9X3 Vc0bHC/KiZCpocm+h9aXcCuQ+A+JA5MQQmRvmdN9hCJWprDmmzFRSugcuHknUFN35+na VoGpqTZg5ZMLOTq7RZ7h2b/QkPH8XcnqAzR9k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695159983; x=1695764783; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uAnbdXaHah1ruKey2JGToMfh7fBQz2GVDoStz4MQTm0=; b=u5J+FZhDa34cVYyBLMRoN09Tl7JiVsL11fY6PMG2xh8Gi/WKDxIKDFri9VsW0VS1ba Fly7NSANt7aepP4FVXadb2olVZ4ouLjHix6P2M0Q4M+3HD7xoGc2m9LuLAWjYQ93ITMN ZPWAmQD/OLnerLgeivuOCK8IHASdLfw7Xzy7B3SjKaxlzvK9uH27G/3sSaEhSwCS2M9i 0aOzz6to/INpnK9z3BvCGG/nx+5rEPibzYNFCxhMGkJo+Kc1naJcipU810pue8WLtuYV p0QX8U9ILrlXFTNhnnSOVtqWpI43Wu68ejbo4uemu8nWscofBKJ3rowGjI0kRNmU+XwZ apJw== X-Gm-Message-State: AOJu0Yx5UYpP6jBWiOZR9tAMbgB+jf6upF1rZ+85dmYEXDqpsQmEmvL2 nRpx3gXR9BG4h+9dEmtbpHqTk6rKSE5eAXR7bws= X-Google-Smtp-Source: AGHT+IFXi8nrdPGw8l4TEpYp9iHTEAd763n93wjVWTEfdvyxLo6fsgiOxAplj7Q/blWpDVWfFK7MFQ== X-Received: by 2002:a5d:6a10:0:b0:31c:8880:5d0f with SMTP id m16-20020a5d6a10000000b0031c88805d0fmr762081wru.11.1695159982931; Tue, 19 Sep 2023 14:46:22 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id c13-20020a5d4ccd000000b0030fd03e3d25sm16657827wrt.75.2023.09.19.14.46.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Sep 2023 14:46:22 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/7] license/license_image: Fix license file layout to avoid overlapping files Date: Tue, 19 Sep 2023 22:46:15 +0100 Message-Id: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Sep 2023 21:46:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187878 Currently DEPLOY_DIR/licenses is added to SSTATE_ALLOW_OVERLAP_FILES. This leads to bugs since when one MACHINE_ARCH recipes is cleaned, it removes the files for another which then results in later build failures as license files disappear. The solution is to include SSTAGE_PKGARCH in the path names to the license files. That does mean a search has to be used to find the correct license files for a given PN but that can be done via SSTATE_ARCHS. The implication for other tools is the layout has changed so tools will need to adapt to the new paths. The benefit is no more strange build failures such as from patterns like: MACHINE=qemux86-64 bitbake core-image-minimal MACHINE=genericx86-64 bitbake core-image-minimal MACHINE=qemux86-64 bitbake linux-yocto -c clean MACHINE=genericx86-64 bitbake core-image-minimal -C rootfs [YOCTO #14123] For anyone finding this commit, I'd question how much people should be relying on this code for tooling and suggest the SPDX manifests should be the preferred data format going forward anyway. Signed-off-by: Richard Purdie --- meta/classes-global/license.bbclass | 2 +- meta/classes-global/sstate.bbclass | 2 -- meta/classes-recipe/license_image.bbclass | 41 ++++++++++++++++------- 3 files changed, 30 insertions(+), 15 deletions(-) diff --git a/meta/classes-global/license.bbclass b/meta/classes-global/license.bbclass index 23625f0104f..b2e0d3fabaf 100644 --- a/meta/classes-global/license.bbclass +++ b/meta/classes-global/license.bbclass @@ -29,7 +29,7 @@ python do_populate_lic() { lic_files_paths = find_license_files(d) # The base directory we wrangle licenses to - destdir = os.path.join(d.getVar('LICSSTATEDIR'), d.getVar('PN')) + destdir = os.path.join(d.getVar('LICSSTATEDIR'), d.getVar('SSTATE_PKGARCH'), d.getVar('PN')) copy_license_files(lic_files_paths, destdir) info = get_recipe_info(d) with open(os.path.join(destdir, "recipeinfo"), "w") as f: diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index c50198449c2..706c2ae9388 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -55,8 +55,6 @@ PV[vardepvalue] = "${PV}" SSTATE_EXTRAPATH[vardepvalue] = "" SSTATE_EXTRAPATHWILDCARD[vardepvalue] = "" -# For multilib rpm the allarch packagegroup files can overwrite (in theory they're identical) -SSTATE_ALLOW_OVERLAP_FILES = "${DEPLOY_DIR}/licenses/" # Avoid docbook/sgml catalog warnings for now SSTATE_ALLOW_OVERLAP_FILES += "${STAGING_ETCDIR_NATIVE}/sgml ${STAGING_DATADIR_NATIVE}/sgml" # sdk-provides-dummy-nativesdk and nativesdk-buildtools-perl-dummy overlap for different SDKMACHINE diff --git a/meta/classes-recipe/license_image.bbclass b/meta/classes-recipe/license_image.bbclass index fc859c7c659..b570a7d37c9 100644 --- a/meta/classes-recipe/license_image.bbclass +++ b/meta/classes-recipe/license_image.bbclass @@ -18,7 +18,7 @@ python() { python write_package_manifest() { # Get list of installed packages - license_image_dir = d.expand('${LICENSE_DIRECTORY}/${IMAGE_NAME}') + license_image_dir = d.expand('${LICENSE_DIRECTORY}/${SSTATE_PKGARCH}/${IMAGE_NAME}') bb.utils.mkdirhier(license_image_dir) from oe.rootfs import image_list_installed_packages from oe.utils import format_pkg_list @@ -49,7 +49,7 @@ python license_create_manifest() { pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name] rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY'), - d.getVar('IMAGE_NAME'), 'license.manifest') + d.getVar('SSTATE_PKGARCH'), d.getVar('IMAGE_NAME'), 'license.manifest') write_license_files(d, rootfs_license_manifest, pkg_dic, rootfs=True) } @@ -98,9 +98,13 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): license_file.write("FILES: %s\n\n" % pkg_dic[pkg]["FILES"]) for lic in pkg_dic[pkg]["LICENSES"]: - lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY'), - pkg_dic[pkg]["PN"], "generic_%s" % - re.sub(r'\+', '', lic)) + for pkgarch in d.getVar("SSTATE_ARCHS").split(): + lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY'), + pkgarch, + pkg_dic[pkg]["PN"], "generic_%s" % + re.sub(r'\+', '', lic)) + if os.path.exists(lic_file): + break # add explicity avoid of CLOSED license because isn't generic if lic == "CLOSED": continue @@ -130,8 +134,13 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): for pkg in sorted(pkg_dic): pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg) bb.utils.mkdirhier(pkg_rootfs_license_dir) - pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), - pkg_dic[pkg]["PN"]) + for pkgarch in d.getVar("SSTATE_ARCHS").split(): + pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), + pkgarch, pkg_dic[pkg]["PN"]) + if os.path.exists(pkg_license_dir): + break + if not os.path.exists(pkg_license_dir ): + bb.fatal("Couldn't find license information for dependency %s" % pkg) pkg_manifest_licenses = [canonical_license(d, lic) \ for lic in pkg_dic[pkg]["LICENSES"]] @@ -183,7 +192,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): os.lchown(p, 0, 0) os.chmod(p, stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH) - +write_license_files[vardepsexclude] = "SSTATE_ARCHS" def license_deployed_manifest(d): """ @@ -204,12 +213,18 @@ def license_deployed_manifest(d): man_dic[dep]["PN"] = dep man_dic[dep]["FILES"] = \ " ".join(get_deployed_files(dep_dic[dep])) - with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: + for pkgarch in d.getVar("SSTATE_ARCHS").split(): + licfile = os.path.join(lic_dir, pkgarch, dep, "recipeinfo") + if os.path.exists(licfile): + break + if not os.path.exists(licfile): + bb.fatal("Couldn't find license information for dependency %s" % dep) + with open(licfile, "r") as f: for line in f.readlines(): key,val = line.split(": ", 1) man_dic[dep][key] = val[:-1] - lic_manifest_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), + lic_manifest_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), d.getVar('SSTATE_PKGARCH'), d.getVar('IMAGE_NAME')) bb.utils.mkdirhier(lic_manifest_dir) image_license_manifest = os.path.join(lic_manifest_dir, 'image_license.manifest') @@ -217,7 +232,7 @@ def license_deployed_manifest(d): link_name = d.getVar('IMAGE_LINK_NAME') if link_name: - lic_manifest_symlink_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), + lic_manifest_symlink_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), d.getVar('SSTATE_PKGARCH'), link_name) # remove old symlink if os.path.islink(lic_manifest_symlink_dir): @@ -227,6 +242,8 @@ def license_deployed_manifest(d): if lic_manifest_dir != lic_manifest_symlink_dir: os.symlink(lic_manifest_dir, lic_manifest_symlink_dir) +license_deployed_manifest[vardepsexclude] = "SSTATE_ARCHS" + def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image @@ -255,7 +272,7 @@ def get_deployed_dependencies(d): break return deploy -get_deployed_dependencies[vardepsexclude] = "BB_TASKDEPDATA" +get_deployed_dependencies[vardepsexclude] = "BB_TASKDEPDATA SSTATE_ARCHS" def get_deployed_files(man_file): """ From patchwork Tue Sep 19 21:46:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4993ECE79AC for ; Tue, 19 Sep 2023 21:46:33 +0000 (UTC) Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by mx.groups.io with SMTP id smtpd.web11.22847.1695159985282874454 for ; Tue, 19 Sep 2023 14:46:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=FptA17u7; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.41, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-31c8321c48fso159546f8f.1 for ; Tue, 19 Sep 2023 14:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695159983; x=1695764783; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2CpFVOQnp8kMFEYPj5Ca02vYxdWnTUOGDBmRqRqbyP8=; b=FptA17u7l1H2UoGCa4s4n4ndDE3FFDf5gzWIBVzdhAqwP7V5b+WtHDq883oKc8fyRf pGJpxlmIXihD+HrTSiunAq3vCj3sRzVjnSw0WZ896dkzMiEyM2eDmtvicDdArbUHm9nZ L8R8TASNSSUdTR32BP1SrpWpw1n8xAL4aWRT8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695159983; x=1695764783; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2CpFVOQnp8kMFEYPj5Ca02vYxdWnTUOGDBmRqRqbyP8=; b=hf42D5RPWQp6oFcsUEG4sgug1kE+B+kW50gyMlzGA0nHFwCy0s8Y91X/fK7x5vjc3a 82VS9SmdslEke5cQM+3kDZftHyCupmkfp5hz4a1qWqoWJZmKBe8ljgc5g6ylg8zGMxbb 3N3aWH3PmLRRdzBB3pOYnWcJjl/taH6lFi7SgF4jHUOFtpC0ftnGJva6FE4svpxGZwA/ snm2Q1N4eXTTIH2AwOnD08dCptpEpHHxsIaUHLJopuNvnWO8NulXIVHBqk6viVelNX+C qaerJmAUcSClDot7r7jDv2k8PC4/dYYXxLl97CmUW+e8b1glUToazPDz7hHUfk/GqCDl 5zWg== X-Gm-Message-State: AOJu0YxK87MzJ02M1L4Fz4keu56fZ8w1fKERhj2OeS14+Zcgf/z0xNbc efGZJQ9OHSX5E7vFbP3YmOsxwk7gkzaxc/5ORHo= X-Google-Smtp-Source: AGHT+IEKd6eTX9MAlsjE/dz4N6XjUQ7X63+xDBmd8hQ76JkWDQXViNO3Ccm+s5LuLESCWDMGUlsvhQ== X-Received: by 2002:a5d:51d0:0:b0:320:6d6:315b with SMTP id n16-20020a5d51d0000000b0032006d6315bmr645324wrv.29.1695159983506; Tue, 19 Sep 2023 14:46:23 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id c13-20020a5d4ccd000000b0030fd03e3d25sm16657827wrt.75.2023.09.19.14.46.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Sep 2023 14:46:23 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/7] create-spdx/sbom: Ensure files don't overlap between machines Date: Tue, 19 Sep 2023 22:46:16 +0100 Message-Id: <20230919214621.903967-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> References: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Sep 2023 21:46:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187877 Currently the by-id and by-namespace SPDX files are created without reference to PACKAGE_ARCH. This means that for two machines using a common package architecture (e.g. genericx86-64 and qqemux86-64), there would be overlapping files. This means that the build of one can remove files from the other leading to build failures. An example would be: MACHINE=qemux86-64 bitbake core-image-minimal MACHINE=genericx86-64 bitbake core-image-minimal MACHINE=qemux86-64 bitbake linux-yocto -c clean MACHINE=genericx86-64 bitbake core-image-minimal -C rootfs To fix this, add PACKAGE_ARCH to the path used for the files and use a search path based upon PACKAGE_ARCHS to access them. Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 28 +++++++++++++++++----------- meta/lib/oe/sbom.py | 20 ++++++++++++++------ 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 9b28d124c78..655d63fcd38 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -349,6 +349,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) spdx_deps_file = Path(d.getVar("SPDXDEPS")) + package_archs = d.getVar("SSTATE_ARCHS").split() dep_recipes = [] @@ -356,7 +357,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): deps = json.load(f) for dep_pn, dep_hashfn in deps: - dep_recipe_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, "recipe-" + dep_pn, dep_hashfn) + dep_recipe_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, "recipe-" + dep_pn, dep_hashfn) spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_recipe_path) @@ -385,6 +386,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): return dep_recipes +collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS" def collect_dep_sources(d, dep_recipes): import oe.sbom @@ -533,6 +535,7 @@ python do_create_spdx() { include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" archive_sources = d.getVar("SPDX_ARCHIVE_SOURCES") == "1" archive_packaged = d.getVar("SPDX_ARCHIVE_PACKAGED") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") @@ -620,7 +623,7 @@ python do_create_spdx() { dep_recipes = collect_dep_recipes(d, doc, recipe) - doc_sha1 = oe.sbom.write_doc(d, doc, d.getVar("SSTATE_PKGARCH"), "recipes", indent=get_json_indent(d)) + doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d)) dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe)) recipe_ref = oe.spdx.SPDXExternalDocumentRef() @@ -685,7 +688,7 @@ python do_create_spdx() { add_package_sources_from_debug(d, package_doc, spdx_package, package, package_files, sources) - oe.sbom.write_doc(d, package_doc, d.getVar("SSTATE_PKGARCH"), "packages", indent=get_json_indent(d)) + oe.sbom.write_doc(d, package_doc, pkg_arch, "packages", indent=get_json_indent(d)) } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source @@ -756,6 +759,8 @@ python do_create_runtime_spdx() { creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") providers = collect_package_providers(d) + pkg_arch = d.getVar("SSTATE_PKGARCH") + package_archs = d.getVar("SSTATE_ARCHS").split() if not is_native: bb.build.exec_func("read_subpackage_metadata", d) @@ -772,7 +777,7 @@ python do_create_runtime_spdx() { if not oe.packagedata.packaged(package, localdata): continue - pkg_spdx_path = oe.sbom.doc_path(deploy_dir_spdx, pkg_name, d.getVar("SSTATE_PKGARCH"), "packages") + pkg_spdx_path = oe.sbom.doc_path(deploy_dir_spdx, pkg_name, pkg_arch, "packages") package_doc, package_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) @@ -827,7 +832,7 @@ python do_create_runtime_spdx() { if dep in dep_package_cache: (dep_spdx_package, dep_package_ref) = dep_package_cache[dep] else: - dep_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, dep_pkg, dep_hashfn) + dep_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, dep_pkg, dep_hashfn) spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_path) @@ -855,10 +860,10 @@ python do_create_runtime_spdx() { ) seen_deps.add(dep) - oe.sbom.write_doc(d, runtime_doc, d.getVar("SSTATE_PKGARCH"), "runtime", spdx_deploy, indent=get_json_indent(d)) + oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d)) } -do_create_runtime_spdx[vardepsexclude] += "OVERRIDES" +do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS" addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work SSTATETASKS += "do_create_runtime_spdx" @@ -993,6 +998,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx import bb.compress.zstd providers = collect_package_providers(d) + package_archs = d.getVar("SSTATE_ARCHS").split() creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -1022,7 +1028,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx pkg_name, pkg_hashfn = providers[name] - pkg_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, pkg_name, pkg_hashfn) + pkg_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn) pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) for p in pkg_doc.packages: @@ -1039,7 +1045,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx else: bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path)) - runtime_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, "runtime-" + name, pkg_hashfn) + runtime_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn) runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path) runtime_ref = oe.spdx.SPDXExternalDocumentRef() @@ -1111,7 +1117,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx }) for ref in doc.externalDocumentRefs: - ref_path = oe.sbom.doc_path_by_namespace(deploy_dir_spdx, ref.spdxDocument) + ref_path = oe.sbom.doc_path_by_namespace(deploy_dir_spdx, package_archs, ref.spdxDocument) collect_spdx_document(ref_path) collect_spdx_document(image_spdx_path) @@ -1134,4 +1140,4 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx tar.addfile(info, fileobj=index_str) -combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS" +combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS" diff --git a/meta/lib/oe/sbom.py b/meta/lib/oe/sbom.py index 1130fa668bd..cddbf3cc51c 100644 --- a/meta/lib/oe/sbom.py +++ b/meta/lib/oe/sbom.py @@ -38,12 +38,20 @@ def get_sdk_spdxid(sdk): return "SPDXRef-SDK-%s" % sdk -def doc_path_by_namespace(spdx_deploy, doc_namespace): - return spdx_deploy / "by-namespace" / doc_namespace.replace("/", "_") +def doc_path_by_namespace(spdx_deploy, archs, doc_namespace): + for pkgarch in archs: + filename = spdx_deploy / "by-namespace" / pkgarch / doc_namespace.replace("/", "_") + if os.path.exists(filename): + break + return filename -def doc_path_by_hashfn(spdx_deploy, doc_name, hashfn): - return spdx_deploy / "by-hash" / hashfn.split()[1] / (doc_name + ".spdx.json") +def doc_path_by_hashfn(spdx_deploy, archs, doc_name, hashfn): + for pkgarch in archs: + filename = spdx_deploy / "by-hash" / pkgarch / hashfn.split()[1] / (doc_name + ".spdx.json") + if os.path.exists(filename): + break + return filename def doc_path(spdx_deploy, doc_name, arch, subdir): @@ -61,11 +69,11 @@ def write_doc(d, spdx_doc, arch, subdir, spdx_deploy=None, indent=None): with dest.open("wb") as f: doc_sha1 = spdx_doc.to_json(f, sort_keys=True, indent=indent) - l = doc_path_by_namespace(spdx_deploy, spdx_doc.documentNamespace) + l = doc_path_by_namespace(spdx_deploy, [arch], spdx_doc.documentNamespace) l.parent.mkdir(exist_ok=True, parents=True) l.symlink_to(os.path.relpath(dest, l.parent)) - l = doc_path_by_hashfn(spdx_deploy, spdx_doc.name, d.getVar("BB_HASHFILENAME")) + l = doc_path_by_hashfn(spdx_deploy, [arch], spdx_doc.name, d.getVar("BB_HASHFILENAME")) l.parent.mkdir(exist_ok=True, parents=True) l.symlink_to(os.path.relpath(dest, l.parent)) From patchwork Tue Sep 19 21:46:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4837CCE79A9 for ; Tue, 19 Sep 2023 21:46:33 +0000 (UTC) Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by mx.groups.io with SMTP id smtpd.web10.22620.1695159986115079103 for ; Tue, 19 Sep 2023 14:46:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=L5tf2SBo; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.41, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-31c7912416bso5794511f8f.1 for ; Tue, 19 Sep 2023 14:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695159984; x=1695764784; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Ab6sRdNu8qmYmtz92b1qDrAzE9i87qhPKZ7mkT/6QU4=; b=L5tf2SBooXLi01Ni4kNjHwETxP32VmnXOe34KLs2s315i+Kpx9+NBP8hnLDfsNZwgo cuO31WkVq7UwOtqh2O69irSVrtD2BMqV/mg0gLaNjdywQCJL7N96GTHnZQWG53brus7L VWCvw9sPTp9gV4/Le7kWN68b9GtW9yiQWU67c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695159984; x=1695764784; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ab6sRdNu8qmYmtz92b1qDrAzE9i87qhPKZ7mkT/6QU4=; b=C24vnh/0qr2tmBCZ3LNf6KqOYN7dtijkjUa6o/sG+RSf+ss4JO83PJ+EH8LaBnw+cj 7ULyGwJfWnzdSJ02nX1nb/RpvuiCFvn+6aFlOWmvTQQchSHCg/0F67XpTj8fFL8HMWsF KrYBeI0wr5Upp6qHOIICIPvyiqf4z78XdBFrgvi3ef4R6dDxYhQkeCIAf3y0nf/vwEv0 YqQcu+KIINt8N0ePNRWH93K2fG35RP4vFmG1Mgavxig0eyzvno09ERxTnvvXqcDcHS2S fpOJslEmPfGTzzxwrUK/Pvt1QJOMdc15V7anQk4W92FRnY+IFfJWg/3NnGXBVWY+iJ2o ulsQ== X-Gm-Message-State: AOJu0YypNJdzgDJt7e1jAygZcM4vQdG7H0e37DGZ/Mt/1uCRDoJWn2+7 9Y2PN1MB0JlXZ2xJPLKLm3j6BdisO+j1DWvz2ok= X-Google-Smtp-Source: AGHT+IGUGNeBADJsDzOTOlilaf0N+5efi3rVtJhLUtb38MfkGkH3+KIl9wWJwJHhFg1+OVnqfoBReg== X-Received: by 2002:adf:d0cc:0:b0:317:61af:d64a with SMTP id z12-20020adfd0cc000000b0031761afd64amr667305wrh.3.1695159984250; Tue, 19 Sep 2023 14:46:24 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id c13-20020a5d4ccd000000b0030fd03e3d25sm16657827wrt.75.2023.09.19.14.46.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Sep 2023 14:46:23 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/7] sstate: Stop allowing overlapping symlinks from sstate Date: Tue, 19 Sep 2023 22:46:17 +0100 Message-Id: <20230919214621.903967-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> References: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Sep 2023 21:46:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187879 When originally implemented, overlapping symlinks in DEPLOY_DIR were common. That is no longer the case and these overlapping links are causing bugs in other areas (e.g. bug 14123). Therefore start showing errors for overlapping symlinks in shared areas. Whilst here, fix a broken file reference in the grep command to match current file layouts and update the message shown to users to match current times. Most of the message content is obsolete now due to other advances and changes in the way the staging code now works. Signed-off-by: Richard Purdie --- meta/classes-global/sstate.bbclass | 30 ++++++------------------------ 1 file changed, 6 insertions(+), 24 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 706c2ae9388..afcda2980b3 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -266,7 +266,7 @@ def sstate_install(ss, d): overlap_allowed = (d.getVar("SSTATE_ALLOW_OVERLAP_FILES") or "").split() match = [] for f in sharedfiles: - if os.path.exists(f) and not os.path.islink(f): + if os.path.exists(f): f = os.path.normpath(f) realmatch = True for w in overlap_allowed: @@ -276,36 +276,18 @@ def sstate_install(ss, d): break if realmatch: match.append(f) - sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}")) + sstate_search_cmd = "grep -rlF '%s' %s --exclude=index-* | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}")) search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0] if search_output: match.append(" (matched in %s)" % search_output.decode('utf-8').rstrip()) else: match.append(" (not matched to any task)") if match: - bb.error("The recipe %s is trying to install files into a shared " \ - "area when those files already exist. Those files and their manifest " \ - "location are:\n %s\nPlease verify which recipe should provide the " \ - "above files.\n\nThe build has stopped, as continuing in this scenario WILL " \ - "break things - if not now, possibly in the future (we've seen builds fail " \ - "several months later). If the system knew how to recover from this " \ - "automatically it would, however there are several different scenarios " \ - "which can result in this and we don't know which one this is. It may be " \ - "you have switched providers of something like virtual/kernel (e.g. from " \ - "linux-yocto to linux-yocto-dev), in that case you need to execute the " \ - "clean task for both recipes and it will resolve this error. It may be " \ - "you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning " \ - "those recipes should again resolve this error, however switching " \ - "DISTRO_FEATURES on an existing build directory is not supported - you " \ - "should really clean out tmp and rebuild (reusing sstate should be safe). " \ - "It could be the overlapping files detected are harmless in which case " \ - "adding them to SSTATE_ALLOW_OVERLAP_FILES may be the correct solution. It could " \ - "also be your build is including two different conflicting versions of " \ - "things (e.g. bluez 4 and bluez 5 and the correct solution for that would " \ - "be to resolve the conflict. If in doubt, please ask on the mailing list, " \ - "sharing the error and filelist above." % \ + bb.fatal("Recipe %s is trying to install files into a shared " \ + "area when those files already exist. The files and the manifests listing " \ + "them are:\n %s\n" + "Please adjust the recipes so only one recipe provides a given file. " % \ (d.getVar('PN'), "\n ".join(match))) - bb.fatal("If the above message is too much, the simpler version is you're advised to wipe out tmp and rebuild (reusing sstate is fine). That will likely fix things in most (but not all) cases.") if ss['fixmedir'] and os.path.exists(ss['fixmedir'] + "/fixmepath.cmd"): sharedfiles.append(ss['fixmedir'] + "/fixmepath.cmd") From patchwork Tue Sep 19 21:46:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5606ACE79AD for ; Tue, 19 Sep 2023 21:46:33 +0000 (UTC) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mx.groups.io with SMTP id smtpd.web11.22848.1695159986868874936 for ; Tue, 19 Sep 2023 14:46:27 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=crRNWwQB; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.49, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-40413ccdd4cso1719555e9.0 for ; Tue, 19 Sep 2023 14:46:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695159985; x=1695764785; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=S8Q3XVSQIF+0e8TJN32t2OX8fgNt8M8XIqe3MlRZdN4=; b=crRNWwQBqN4Vqc/UxRFy9rrwjWEu2eKHVnOc1rcULR6e00VKkMhaw/HIyf7zf5f46K 5eo+tmu/I8105ik1MnEIrXThRTd1YITNeevYSir5TtZF5qLks6r3EVQkP4mBM8hWXBI8 IoVTSOuk/PggDX8RMeIYPvPs7M8+r0Ztl3zoI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695159985; x=1695764785; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S8Q3XVSQIF+0e8TJN32t2OX8fgNt8M8XIqe3MlRZdN4=; b=mnaIf3k6YABcyTJpsKcHqyuM2OYfJEcyKikeVsgxEwRaTyVYojH6C0OSKygRRIe/QB jCx6R2VW1zGSQv2J2KoKDauKLC4e19RgVrWMzUikc3ISI/2oN4ok9KGIt6Pjdltye0Ad fmUHIPpGQdMifwqwtvj4TuIyEuQrffcxCj0b2LJRBLNEwXt23hEZOSO9JG4yCVcHDI41 C6Fu3kbnsMOj/D/gRXrqdKoUia4v/r4VLeP0BuPt7ydsar8eW7txD8OR0+hrSKauSIsA f1cXXJGQ03d2Y1nrRmKxi7X5uHI+N0M/xEV7PPI6varaWTAN6CSjzdTGg9PP3r8p198X 2EDw== X-Gm-Message-State: AOJu0YxoPXb/47bQb/rAuoCqqMK1dP+vumGoz1vQiO2d+d7tU+Afc+Kg 9znqZgS4tLcx8LUXhlE993GowGV4Z551Gc+2pao= X-Google-Smtp-Source: AGHT+IHX4S45TOK6fwgT5hmT3nFUUxwnAkuNeqdxuUfP8zFop57FcTY8GVZFe44tO7IFXnc/gRKUWg== X-Received: by 2002:a5d:5267:0:b0:31f:eb45:462e with SMTP id l7-20020a5d5267000000b0031feb45462emr588785wrc.28.1695159985252; Tue, 19 Sep 2023 14:46:25 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id c13-20020a5d4ccd000000b0030fd03e3d25sm16657827wrt.75.2023.09.19.14.46.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Sep 2023 14:46:24 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 4/7] sstate: Fix nativesdk entry in SSTATE_ARCHS Date: Tue, 19 Sep 2023 22:46:18 +0100 Message-Id: <20230919214621.903967-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> References: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Sep 2023 21:46:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187880 PACKAGE_ARCH gets refined by each target so this value isn't valid in all contexts. Tweak to use underlying variables to build it so that it remains valid in wider contexts and works with new usage in sdpx and license classes. Signed-off-by: Richard Purdie --- meta/classes-global/sstate.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index afcda2980b3..67070440a6c 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -88,7 +88,7 @@ SSTATE_ARCHS = " \ ${BUILD_ARCH}_${ORIGNATIVELSBSTRING} \ ${BUILD_ARCH}_${SDK_ARCH}_${SDK_OS} \ ${SDK_ARCH}_${SDK_OS} \ - ${SDK_ARCH}_${PACKAGE_ARCH} \ + ${SDK_ARCH}_${SDK_ARCH}-${SDKPKGSUFFIX} \ allarch \ ${PACKAGE_ARCH} \ ${PACKAGE_EXTRA_ARCHS} \ From patchwork Tue Sep 19 21:46:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56204CE79AE for ; Tue, 19 Sep 2023 21:46:33 +0000 (UTC) Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) by mx.groups.io with SMTP id smtpd.web10.22622.1695159987703922771 for ; Tue, 19 Sep 2023 14:46:28 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=RGsnzQWc; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f45.google.com with SMTP id ffacd0b85a97d-32008e339adso3122692f8f.2 for ; Tue, 19 Sep 2023 14:46:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695159986; x=1695764786; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=3u2/ZLsTJAU69eSBUEVc18IotitlL1CRKAYmXm9t6rw=; b=RGsnzQWc9WGz5islKwG2f/UI2MMRm7fBdzPMA7ymWxcPajZ4DsutghENszIfj3V4Ur 1KunrcqXp0YiPZzlT/IBachCoVM0kbnQrvtRlO7b9MFgFXAblA68zkKyZpCghZsGC9dM NzazasijQnuQLJHfTWcsRtn0hqs6dAp5wxj/I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695159986; x=1695764786; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3u2/ZLsTJAU69eSBUEVc18IotitlL1CRKAYmXm9t6rw=; b=u8lO31UmbdvtSrbYVA4DyuxEEc70Y3uwP+5mwba7DirqUonOa5BnccwNLlgOlus8Rn 0xjfbFlAwKXHk+xvVRk10xqgRi7FfBrq3/KHE2HzUyEw1I0uTN2nNRcoqjAEvvFoAPVk Ot+W4QcWHgVZ77t/iUtSyHShn3uhCwPRejgP4TNKJbpjqepZFqy2X1Le3hy5po8MrGR5 P+i3/fwH0a3yc6bLVNDzwtEz8LAuC1aYzBWYpPwQ2wp8b7kus4bsz5iXeIwvxQZYxPlY pZNplUk8SduhISCirqFv4uOzH/w87rGrppeIzgimzqLjbEQOD2H9UOnv49HZK342wcNz Hhpw== X-Gm-Message-State: AOJu0YzeSQEXjNh18jORp33dlHLTRGD15E6clAFQOXmz7cXkLZOPMSE6 KusjzldhOuqyLTiU4SLqmAB7hlQ8hLs1euA8a6M= X-Google-Smtp-Source: AGHT+IHZs/YxGDFdsAZI4qn05RyWya3bm5QRK1UMcuMZq/A9pZVNPr8+CxrMbDv0eIa6/1VZRM3seQ== X-Received: by 2002:a5d:6711:0:b0:31f:fa6a:936e with SMTP id o17-20020a5d6711000000b0031ffa6a936emr797586wru.17.1695159985911; Tue, 19 Sep 2023 14:46:25 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id c13-20020a5d4ccd000000b0030fd03e3d25sm16657827wrt.75.2023.09.19.14.46.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Sep 2023 14:46:25 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 5/7] elfutils: Fix reproducibility issue with bunzip2 Date: Tue, 19 Sep 2023 22:46:19 +0100 Message-Id: <20230919214621.903967-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> References: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Sep 2023 21:46:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187881 bunzip2 is only detected/needed for "make check" tests however if left floating, the generated makefiles used in the ptests are not deterministic. Force a value to avoid this. [YOCTO #15209] Signed-off-by: Richard Purdie --- meta/recipes-devtools/elfutils/elfutils_0.189.bb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta/recipes-devtools/elfutils/elfutils_0.189.bb b/meta/recipes-devtools/elfutils/elfutils_0.189.bb index 236f8cef92d..d6352de5cf5 100644 --- a/meta/recipes-devtools/elfutils/elfutils_0.189.bb +++ b/meta/recipes-devtools/elfutils/elfutils_0.189.bb @@ -32,6 +32,9 @@ inherit autotools gettext ptest pkgconfig EXTRA_OECONF = "--program-prefix=eu-" +# Only used at runtime for make check but we want deterministic makefiles for ptest so hardcode +CACHED_CONFIGUREVARS += "ac_cv_prog_HAVE_BUNZIP2=yes" + BUILD_CFLAGS += "-Wno-error=stringop-overflow" DEPENDS_BZIP2 = "bzip2-replacement-native" From patchwork Tue Sep 19 21:46:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64739CE79AA for ; Tue, 19 Sep 2023 21:46:33 +0000 (UTC) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mx.groups.io with SMTP id smtpd.web10.22623.1695159988233651732 for ; Tue, 19 Sep 2023 14:46:28 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=IU4grjvw; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.49, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-40473f1fe9fso64136315e9.2 for ; Tue, 19 Sep 2023 14:46:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695159986; x=1695764786; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=wrdueq8QVs+aWd6xbZq7emjNWkeQn5bpu9ZkC28y3EA=; b=IU4grjvwDYFyWKLbe/Ao9RI2gORUvg8Q1FFoj7GIKgjubaeqXQK4tb69VypARL8TTM KgEjM+sasIylZj37Cj5l9HI+tkoXirBIaHO73+qp4xheBGzy6P/nJG57MHMRtRSeukCB MbjSyr6W3Zs90RYA4i6HY5oe/3ZJTdroDqveI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695159986; x=1695764786; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wrdueq8QVs+aWd6xbZq7emjNWkeQn5bpu9ZkC28y3EA=; b=JitfLvgWATNnVyNsXrbYiM8Jpz8TetfgiVRHF/XeO4GJhXvu6SFcCmzG166OCDBOOC juX4JASn6+bGkOnI0F5vukyP/9ZUTxuqOci7aVQQT1qv2NBTpUsB/rlCvum0MVN8odHI F9RSKHhzfUmvsatcJmlUhud9NKL6G1MYjPy0mK/i19kKcaxrniPTHYLdkSfCWxfbLUj2 ox3ZWEghy5ymjU3jx85DU860aaKQ+nAgmXvS3Gl4Dw4DEHbHWkbYbgaYcNOC+oomFUFt VatAhsNrU87TjnHptgHFt6GM8qYy7NDnw1MUyUqhJVWbcoUoAh05tXE0HTqj8QeUyToV wDtw== X-Gm-Message-State: AOJu0YyA2C75l1yHrreqOyC3O+18w8bYJhMpYm8bt7YvQzrmqNi/FJ6q dvr1CoeAMzG2ixgczmZ1JkudjEVJ7ADhC5LJ43Y= X-Google-Smtp-Source: AGHT+IFHIQiL7rNFMRQ84bPh5XhaSFm04xf74ub3N3bDc/d42tVt+HPYpFs79nCVybC+I0Kz4p0K1w== X-Received: by 2002:a5d:448e:0:b0:320:976:f940 with SMTP id j14-20020a5d448e000000b003200976f940mr671485wrq.13.1695159986560; Tue, 19 Sep 2023 14:46:26 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id c13-20020a5d4ccd000000b0030fd03e3d25sm16657827wrt.75.2023.09.19.14.46.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Sep 2023 14:46:26 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 6/7] multilib: fix SSTATE_ARCHS for multilib usage Date: Tue, 19 Sep 2023 22:46:20 +0100 Message-Id: <20230919214621.903967-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> References: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Sep 2023 21:46:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187882 When building multilibs, we need to inject the multilib sstate pkgarch into SSTATE_ARCHS so the list forms a complete search path. Add a tweak to do this. Signed-off-by: Richard Purdie --- meta/classes/multilib.bbclass | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass index 8a1a51aabad..673ae98fec1 100644 --- a/meta/classes/multilib.bbclass +++ b/meta/classes/multilib.bbclass @@ -13,6 +13,7 @@ python multilib_virtclass_handler () { localdata = bb.data.createCopy(e.data) localdata.delVar('TMPDIR') e.data.setVar('STAGING_KERNEL_DIR', localdata.getVar('STAGING_KERNEL_DIR')) + e.data.appendVar('SSTATE_ARCHS', " " + localdata.getVar('TUNE_PKGARCH')) # There should only be one kernel in multilib configs # We also skip multilib setup for module packages. From patchwork Tue Sep 19 21:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64784CE79B0 for ; Tue, 19 Sep 2023 21:46:33 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web10.22624.1695159988950585924 for ; Tue, 19 Sep 2023 14:46:29 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=aP+qEv7B; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.53, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-31fa15f4cc6so5787094f8f.2 for ; Tue, 19 Sep 2023 14:46:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695159987; x=1695764787; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=n42VeLZx3OUuo7cIFQ0rW03T/RAdnGMA8htNqiINBsw=; b=aP+qEv7BMdcnUUzpQ5BddcQOqt2KRFU/BoYtupVdr1yEhMnUw8hPhkHvfvShhTHwxV bz1PiBPJ8789cvVdhe0NGlxx23CcW8dLdnArUIoFm3NwJx2VGZ2X7AdE/AaQD9WBxKgx AARGqlLEEfV2RSiobMC3LTxTMqAZTmCSM9JMg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695159987; x=1695764787; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n42VeLZx3OUuo7cIFQ0rW03T/RAdnGMA8htNqiINBsw=; b=v1zdwjzP+RdvY0+8cFx7O0wJgI9o0ky7FYvnLQ0W7GIjQGu8lp0hGceq2ZmtaaquNh IF3Ys6aqyc6oUoQ4a7Fzvb2fQTgQO6P56xTwu5pBPwT5UNUZ/9z9nUrKzkT9V0iYPmYM OkKcj2hu1P0wgqOZj2191Ba8Lj8tuHxuoRZtGmXefuUGXgcKeEPRl43BBRdjnW4zZ4Gu 2VoOSRoCsU2sKwbFeuJOtp26CX1rxNLm2KU+RaYneqQPjTGdbAOt3YxFdvR+sOLX/0pk wOw1b66gWhcxidGuGkKtXd2Afo2WY68g7ejvzEkXVc070UU2y3gKmWTx6rj5c9RxtwPJ 6fXw== X-Gm-Message-State: AOJu0YwRJ5B+Lee4MDmP509Z44/dsOQC5togKCvyKEImRgmEtOJtAyfT UUD75eb//k1sCKAqSNr8ynyYSLfvO1PDiUhYdq8= X-Google-Smtp-Source: AGHT+IEUnRN1H9udndERKcSca3o152GbOPGMvCgP6zwq1WTKDOKMD5MD2TdEhtNGw11OkftvZHJrDA== X-Received: by 2002:adf:dc8f:0:b0:321:5d9f:2d9f with SMTP id r15-20020adfdc8f000000b003215d9f2d9fmr701272wrj.47.1695159987258; Tue, 19 Sep 2023 14:46:27 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id c13-20020a5d4ccd000000b0030fd03e3d25sm16657827wrt.75.2023.09.19.14.46.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Sep 2023 14:46:26 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 7/7] oeqa/selftest/bbtests: Improve and update test_non_gplv3 Date: Tue, 19 Sep 2023 22:46:21 +0100 Message-Id: <20230919214621.903967-7-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> References: <20230919214621.903967-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Sep 2023 21:46:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187883 This test no longer worked with fixes to avoid overlapping license files in deploy. Fix the test but also improve the logging messages so we don't get "False is not True" messages upon failure. Signed-off-by: Richard Purdie --- meta/lib/oeqa/selftest/cases/bbtests.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/bbtests.py b/meta/lib/oeqa/selftest/cases/bbtests.py index 31aa5680f01..d242352ea2d 100644 --- a/meta/lib/oeqa/selftest/cases/bbtests.py +++ b/meta/lib/oeqa/selftest/cases/bbtests.py @@ -236,8 +236,11 @@ INHERIT:remove = \"report-error\" result = bitbake('selftest-ed', ignore_status=True) self.assertEqual(result.status, 0, "Bitbake failed, exit code %s, output %s" % (result.status, result.output)) lic_dir = get_bb_var('LICENSE_DIRECTORY') - self.assertFalse(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPL-3.0-or-later'))) - self.assertTrue(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPL-2.0-or-later'))) + arch = get_bb_var('SSTATE_PKGARCH') + filename = os.path.join(lic_dir, arch, 'selftest-ed', 'generic_GPL-3.0-or-later') + self.assertFalse(os.path.isfile(filename), msg="License file %s exists and shouldn't" % filename) + filename = os.path.join(lic_dir, arch, 'selftest-ed', 'generic_GPL-2.0-or-later') + self.assertTrue(os.path.isfile(filename), msg="License file %s doesn't exist" % filename) def test_setscene_only(self): """ Bitbake option to restore from sstate only within a build (i.e. execute no real tasks, only setscene)"""