From patchwork Wed Sep 20 10:57:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B136CE79CA for ; Wed, 20 Sep 2023 10:58:07 +0000 (UTC) Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by mx.groups.io with SMTP id smtpd.web10.35158.1695207485601880291 for ; Wed, 20 Sep 2023 03:58:05 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=ZkdpiPXQ; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.41, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-31427ddd3fbso5978868f8f.0 for ; Wed, 20 Sep 2023 03:58:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695207484; x=1695812284; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=apgn/GOOO0fzJtzOTS+6hQroxIEDPV6LTI8uRRCOqf8=; b=ZkdpiPXQf2KHJsbBxWdHIOxscW3MjPVB3uekJtHIcWKyFHWth0ibrTU3ViX86HDm4a djWiSDtU9rdqFyDC3ipeFvmSavmtwsprWKwnnW/VD5IZgNwao6xsU9iyGF0IScWMLiM3 h1fwn4G8obyKlTDtXV269beBzh+tl9sYTbywQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695207484; x=1695812284; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=apgn/GOOO0fzJtzOTS+6hQroxIEDPV6LTI8uRRCOqf8=; b=B3aYhNDby9RtsD52vzOzYUkXSKihnEBs17Zrx/fa8AqEfoYPajmBsCK2qyyD/BFtIp GPK6HIg0m41/bQ9f/47Ot3gaGYZp6dIXCZEOzIlscFe9/Huic5v7tnfGY+k2alNMJT7g LS9GVPWewjKOZULZI/VlpEl9QmWix455mGUhbbPfLxhJIzEyUsbCQ+NIEMmcR6hnJTuO O97tVmo1k/sEc+4QQN7wyoW7gHUMmgh8o0+9Ys1dVeHB5i7EUFotmBgnMYq+QMglfggO 0ki1mg0A0AJPlRGEVhkOHAl1DHMM5dCv4L62ZNPy20fvKEZaQJGxA+IQYhhu7UYfMrYT eBxg== X-Gm-Message-State: AOJu0YwYjlJJRXPwNQxfsXoLF8mH70sKMe1YOV7ORyP2jsPLN9Ug9YUl UzGAC9/HY0/FQXC/rlJkWy+jz7LaThbB+GdqDsM= X-Google-Smtp-Source: AGHT+IGi9GE7tNCwo2sziteOMxLw0hZxZMB66ghan8pZ8YONCzp1ot/vhXeo47KpFdnrgzUUnCqZaA== X-Received: by 2002:a5d:68c6:0:b0:31c:4c68:e73a with SMTP id p6-20020a5d68c6000000b0031c4c68e73amr1916169wrw.66.1695207483253; Wed, 20 Sep 2023 03:58:03 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id h1-20020adfe981000000b0031f300a4c26sm426636wrm.93.2023.09.20.03.58.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 03:58:02 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/6 v2] license/license_image: Fix license file layout to avoid overlapping files Date: Wed, 20 Sep 2023 11:57:57 +0100 Message-Id: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 20 Sep 2023 10:58:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187917 Currently DEPLOY_DIR/licenses is added to SSTATE_ALLOW_OVERLAP_FILES. This leads to bugs since when one MACHINE_ARCH recipes is cleaned, it removes the files for another which then results in later build failures as license files disappear. The solution is to include SSTAGE_PKGARCH in the path names to the license files. That does mean a search has to be used to find the correct license files for a given PN but that can be done via SSTATE_ARCHS. The implication for other tools is the layout has changed so tools will need to adapt to the new paths. The benefit is no more strange build failures such as from patterns like: MACHINE=qemux86-64 bitbake core-image-minimal MACHINE=genericx86-64 bitbake core-image-minimal MACHINE=qemux86-64 bitbake linux-yocto -c clean MACHINE=genericx86-64 bitbake core-image-minimal -C rootfs [YOCTO #14123] For anyone finding this commit, I'd question how much people should be relying on this code for tooling and suggest the SPDX manifests should be the preferred data format going forward anyway. Signed-off-by: Richard Purdie --- meta/classes-global/license.bbclass | 2 +- meta/classes-global/sstate.bbclass | 2 - meta/classes-recipe/license_image.bbclass | 46 +++++++++++++++++------ 3 files changed, 35 insertions(+), 15 deletions(-) v2 - use reversed order of SSTATE_ARCHS diff --git a/meta/classes-global/license.bbclass b/meta/classes-global/license.bbclass index 23625f0104f..b2e0d3fabaf 100644 --- a/meta/classes-global/license.bbclass +++ b/meta/classes-global/license.bbclass @@ -29,7 +29,7 @@ python do_populate_lic() { lic_files_paths = find_license_files(d) # The base directory we wrangle licenses to - destdir = os.path.join(d.getVar('LICSSTATEDIR'), d.getVar('PN')) + destdir = os.path.join(d.getVar('LICSSTATEDIR'), d.getVar('SSTATE_PKGARCH'), d.getVar('PN')) copy_license_files(lic_files_paths, destdir) info = get_recipe_info(d) with open(os.path.join(destdir, "recipeinfo"), "w") as f: diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index c50198449c2..706c2ae9388 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -55,8 +55,6 @@ PV[vardepvalue] = "${PV}" SSTATE_EXTRAPATH[vardepvalue] = "" SSTATE_EXTRAPATHWILDCARD[vardepvalue] = "" -# For multilib rpm the allarch packagegroup files can overwrite (in theory they're identical) -SSTATE_ALLOW_OVERLAP_FILES = "${DEPLOY_DIR}/licenses/" # Avoid docbook/sgml catalog warnings for now SSTATE_ALLOW_OVERLAP_FILES += "${STAGING_ETCDIR_NATIVE}/sgml ${STAGING_DATADIR_NATIVE}/sgml" # sdk-provides-dummy-nativesdk and nativesdk-buildtools-perl-dummy overlap for different SDKMACHINE diff --git a/meta/classes-recipe/license_image.bbclass b/meta/classes-recipe/license_image.bbclass index fc859c7c659..19b3dc55ba2 100644 --- a/meta/classes-recipe/license_image.bbclass +++ b/meta/classes-recipe/license_image.bbclass @@ -18,7 +18,7 @@ python() { python write_package_manifest() { # Get list of installed packages - license_image_dir = d.expand('${LICENSE_DIRECTORY}/${IMAGE_NAME}') + license_image_dir = d.expand('${LICENSE_DIRECTORY}/${SSTATE_PKGARCH}/${IMAGE_NAME}') bb.utils.mkdirhier(license_image_dir) from oe.rootfs import image_list_installed_packages from oe.utils import format_pkg_list @@ -49,7 +49,7 @@ python license_create_manifest() { pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name] rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY'), - d.getVar('IMAGE_NAME'), 'license.manifest') + d.getVar('SSTATE_PKGARCH'), d.getVar('IMAGE_NAME'), 'license.manifest') write_license_files(d, rootfs_license_manifest, pkg_dic, rootfs=True) } @@ -59,6 +59,8 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split() bad_licenses = expand_wildcard_licenses(d, bad_licenses) + pkgarchs = d.getVar("SSTATE_ARCHS").split() + pkgarchs.reverse() exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split() with open(license_manifest, "w") as license_file: @@ -98,9 +100,13 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): license_file.write("FILES: %s\n\n" % pkg_dic[pkg]["FILES"]) for lic in pkg_dic[pkg]["LICENSES"]: - lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY'), - pkg_dic[pkg]["PN"], "generic_%s" % - re.sub(r'\+', '', lic)) + for pkgarch in pkgarchs: + lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY'), + pkgarch, + pkg_dic[pkg]["PN"], "generic_%s" % + re.sub(r'\+', '', lic)) + if os.path.exists(lic_file): + break # add explicity avoid of CLOSED license because isn't generic if lic == "CLOSED": continue @@ -130,8 +136,13 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): for pkg in sorted(pkg_dic): pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg) bb.utils.mkdirhier(pkg_rootfs_license_dir) - pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), - pkg_dic[pkg]["PN"]) + for pkgarch in pkgarchs: + pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), + pkgarch, pkg_dic[pkg]["PN"]) + if os.path.exists(pkg_license_dir): + break + if not os.path.exists(pkg_license_dir ): + bb.fatal("Couldn't find license information for dependency %s" % pkg) pkg_manifest_licenses = [canonical_license(d, lic) \ for lic in pkg_dic[pkg]["LICENSES"]] @@ -183,7 +194,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True): os.lchown(p, 0, 0) os.chmod(p, stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH) - +write_license_files[vardepsexclude] = "SSTATE_ARCHS" def license_deployed_manifest(d): """ @@ -195,6 +206,8 @@ def license_deployed_manifest(d): dep_dic = {} man_dic = {} lic_dir = d.getVar("LICENSE_DIRECTORY") + pkgarchs = d.getVar("SSTATE_ARCHS").split() + pkgarchs.reverse() dep_dic = get_deployed_dependencies(d) for dep in dep_dic.keys(): @@ -204,12 +217,19 @@ def license_deployed_manifest(d): man_dic[dep]["PN"] = dep man_dic[dep]["FILES"] = \ " ".join(get_deployed_files(dep_dic[dep])) - with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: + + for pkgarch in pkgarchs: + licfile = os.path.join(lic_dir, pkgarch, dep, "recipeinfo") + if os.path.exists(licfile): + break + if not os.path.exists(licfile): + bb.fatal("Couldn't find license information for dependency %s" % dep) + with open(licfile, "r") as f: for line in f.readlines(): key,val = line.split(": ", 1) man_dic[dep][key] = val[:-1] - lic_manifest_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), + lic_manifest_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), d.getVar('SSTATE_PKGARCH'), d.getVar('IMAGE_NAME')) bb.utils.mkdirhier(lic_manifest_dir) image_license_manifest = os.path.join(lic_manifest_dir, 'image_license.manifest') @@ -217,7 +237,7 @@ def license_deployed_manifest(d): link_name = d.getVar('IMAGE_LINK_NAME') if link_name: - lic_manifest_symlink_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), + lic_manifest_symlink_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'), d.getVar('SSTATE_PKGARCH'), link_name) # remove old symlink if os.path.islink(lic_manifest_symlink_dir): @@ -227,6 +247,8 @@ def license_deployed_manifest(d): if lic_manifest_dir != lic_manifest_symlink_dir: os.symlink(lic_manifest_dir, lic_manifest_symlink_dir) +license_deployed_manifest[vardepsexclude] = "SSTATE_ARCHS" + def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image @@ -255,7 +277,7 @@ def get_deployed_dependencies(d): break return deploy -get_deployed_dependencies[vardepsexclude] = "BB_TASKDEPDATA" +get_deployed_dependencies[vardepsexclude] = "BB_TASKDEPDATA SSTATE_ARCHS" def get_deployed_files(man_file): """ From patchwork Wed Sep 20 10:57:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 809C8CE79C3 for ; Wed, 20 Sep 2023 10:58:07 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web10.35159.1695207485702094593 for ; Wed, 20 Sep 2023 03:58:06 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=AifguHyH; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-31dd10c2b8bso6299154f8f.3 for ; Wed, 20 Sep 2023 03:58:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695207484; x=1695812284; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=B3lpTK2br2JyGSjk4tnVougSWsXWsPbToRJySQ7zOQQ=; b=AifguHyHUjVI3r8WHr5e4Hwn0trlyQFRWZ0vDb4FIKnCiA+pejqTpYFtvoUJSJTaLd aWDy7nmKSERCyLmkMgkLBHid+Y4AFYLkRavb0IHDmpbEW0dc2Ymr15pf8SpMdKbjQ+4m 4vXAux3Q1g/P9VKDchhEfUazOoycc3CB+hZv8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695207484; x=1695812284; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B3lpTK2br2JyGSjk4tnVougSWsXWsPbToRJySQ7zOQQ=; b=ImuI8Ecpp6O6BYqnQK7ZVMto4ZbrySo5hmZwIBW1j4mnS0jZrV5wHsb8Ee4acNMyU0 C6GOCOWISkCS/kKdM0ceMHJ3ZusKho9W26XzQGVsQcBdQSruyWIo3LwAzSShRKWpkEZk RyYkpTbkznnW5LS3kbEqzdyeQXT+AH7RONyBw8adxoWFk2f2enqzgVvUfNYnrc2vfqQr rmPpzwlO6J/Y8YqE9DBCaz8MC5zHY/NxKbxhtNSjGtDC1KKMG2negNyn8hSNXt06A1o4 koX1iI2Jznbo2cIrSLGxGExJm0PGFZ+0d1M2ZBD/FlW8IVi8cdfkDdzigr2EugQOKTL4 Ek6w== X-Gm-Message-State: AOJu0YwFjBOTn+QUZ1uP19VfHR8KF9DnItIHb7vbQiTg5OhiBzzt6vTv Nz2SA3feDDok3RHDBxLOkIpHeZItLGqm9idg4EM= X-Google-Smtp-Source: AGHT+IHArPECI/+tlug9y/+TgeOuYg9izR4eYEhH7i3S4Z96s1TB8Ei+hFmllBdXInYXKkXx8LDwvA== X-Received: by 2002:adf:e712:0:b0:31a:d551:c2c0 with SMTP id c18-20020adfe712000000b0031ad551c2c0mr2105136wrm.57.1695207483873; Wed, 20 Sep 2023 03:58:03 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id h1-20020adfe981000000b0031f300a4c26sm426636wrm.93.2023.09.20.03.58.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 03:58:03 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/6 v2] create-spdx/sbom: Ensure files don't overlap between machines Date: Wed, 20 Sep 2023 11:57:58 +0100 Message-Id: <20230920105802.1008778-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> References: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 20 Sep 2023 10:58:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187918 Currently the by-id and by-namespace SPDX files are created without reference to PACKAGE_ARCH. This means that for two machines using a common package architecture (e.g. genericx86-64 and qqemux86-64), there would be overlapping files. This means that the build of one can remove files from the other leading to build failures. An example would be: MACHINE=qemux86-64 bitbake core-image-minimal MACHINE=genericx86-64 bitbake core-image-minimal MACHINE=qemux86-64 bitbake linux-yocto -c clean MACHINE=genericx86-64 bitbake core-image-minimal -C rootfs To fix this, add PACKAGE_ARCH to the path used for the files and use a search path based upon PACKAGE_ARCHS to access them. Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 31 ++++++++++++++++++---------- meta/lib/oe/sbom.py | 20 ++++++++++++------ 2 files changed, 34 insertions(+), 17 deletions(-) v2 - Use reversed order of SSTATE_ARCHS diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 9b28d124c78..39110823c0c 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -349,6 +349,8 @@ def collect_dep_recipes(d, doc, spdx_recipe): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) spdx_deps_file = Path(d.getVar("SPDXDEPS")) + package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs.reverse() dep_recipes = [] @@ -356,7 +358,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): deps = json.load(f) for dep_pn, dep_hashfn in deps: - dep_recipe_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, "recipe-" + dep_pn, dep_hashfn) + dep_recipe_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, "recipe-" + dep_pn, dep_hashfn) spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_recipe_path) @@ -385,6 +387,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): return dep_recipes +collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS" def collect_dep_sources(d, dep_recipes): import oe.sbom @@ -533,6 +536,7 @@ python do_create_spdx() { include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" archive_sources = d.getVar("SPDX_ARCHIVE_SOURCES") == "1" archive_packaged = d.getVar("SPDX_ARCHIVE_PACKAGED") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") @@ -620,7 +624,7 @@ python do_create_spdx() { dep_recipes = collect_dep_recipes(d, doc, recipe) - doc_sha1 = oe.sbom.write_doc(d, doc, d.getVar("SSTATE_PKGARCH"), "recipes", indent=get_json_indent(d)) + doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d)) dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe)) recipe_ref = oe.spdx.SPDXExternalDocumentRef() @@ -685,7 +689,7 @@ python do_create_spdx() { add_package_sources_from_debug(d, package_doc, spdx_package, package, package_files, sources) - oe.sbom.write_doc(d, package_doc, d.getVar("SSTATE_PKGARCH"), "packages", indent=get_json_indent(d)) + oe.sbom.write_doc(d, package_doc, pkg_arch, "packages", indent=get_json_indent(d)) } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source @@ -756,6 +760,9 @@ python do_create_runtime_spdx() { creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") providers = collect_package_providers(d) + pkg_arch = d.getVar("SSTATE_PKGARCH") + package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs.reverse() if not is_native: bb.build.exec_func("read_subpackage_metadata", d) @@ -772,7 +779,7 @@ python do_create_runtime_spdx() { if not oe.packagedata.packaged(package, localdata): continue - pkg_spdx_path = oe.sbom.doc_path(deploy_dir_spdx, pkg_name, d.getVar("SSTATE_PKGARCH"), "packages") + pkg_spdx_path = oe.sbom.doc_path(deploy_dir_spdx, pkg_name, pkg_arch, "packages") package_doc, package_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) @@ -827,7 +834,7 @@ python do_create_runtime_spdx() { if dep in dep_package_cache: (dep_spdx_package, dep_package_ref) = dep_package_cache[dep] else: - dep_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, dep_pkg, dep_hashfn) + dep_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, dep_pkg, dep_hashfn) spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_path) @@ -855,10 +862,10 @@ python do_create_runtime_spdx() { ) seen_deps.add(dep) - oe.sbom.write_doc(d, runtime_doc, d.getVar("SSTATE_PKGARCH"), "runtime", spdx_deploy, indent=get_json_indent(d)) + oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d)) } -do_create_runtime_spdx[vardepsexclude] += "OVERRIDES" +do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS" addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work SSTATETASKS += "do_create_runtime_spdx" @@ -993,6 +1000,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx import bb.compress.zstd providers = collect_package_providers(d) + package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs.reverse() creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -1022,7 +1031,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx pkg_name, pkg_hashfn = providers[name] - pkg_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, pkg_name, pkg_hashfn) + pkg_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn) pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) for p in pkg_doc.packages: @@ -1039,7 +1048,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx else: bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path)) - runtime_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, "runtime-" + name, pkg_hashfn) + runtime_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn) runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path) runtime_ref = oe.spdx.SPDXExternalDocumentRef() @@ -1111,7 +1120,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx }) for ref in doc.externalDocumentRefs: - ref_path = oe.sbom.doc_path_by_namespace(deploy_dir_spdx, ref.spdxDocument) + ref_path = oe.sbom.doc_path_by_namespace(deploy_dir_spdx, package_archs, ref.spdxDocument) collect_spdx_document(ref_path) collect_spdx_document(image_spdx_path) @@ -1134,4 +1143,4 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx tar.addfile(info, fileobj=index_str) -combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS" +combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS" diff --git a/meta/lib/oe/sbom.py b/meta/lib/oe/sbom.py index 1130fa668bd..cddbf3cc51c 100644 --- a/meta/lib/oe/sbom.py +++ b/meta/lib/oe/sbom.py @@ -38,12 +38,20 @@ def get_sdk_spdxid(sdk): return "SPDXRef-SDK-%s" % sdk -def doc_path_by_namespace(spdx_deploy, doc_namespace): - return spdx_deploy / "by-namespace" / doc_namespace.replace("/", "_") +def doc_path_by_namespace(spdx_deploy, archs, doc_namespace): + for pkgarch in archs: + filename = spdx_deploy / "by-namespace" / pkgarch / doc_namespace.replace("/", "_") + if os.path.exists(filename): + break + return filename -def doc_path_by_hashfn(spdx_deploy, doc_name, hashfn): - return spdx_deploy / "by-hash" / hashfn.split()[1] / (doc_name + ".spdx.json") +def doc_path_by_hashfn(spdx_deploy, archs, doc_name, hashfn): + for pkgarch in archs: + filename = spdx_deploy / "by-hash" / pkgarch / hashfn.split()[1] / (doc_name + ".spdx.json") + if os.path.exists(filename): + break + return filename def doc_path(spdx_deploy, doc_name, arch, subdir): @@ -61,11 +69,11 @@ def write_doc(d, spdx_doc, arch, subdir, spdx_deploy=None, indent=None): with dest.open("wb") as f: doc_sha1 = spdx_doc.to_json(f, sort_keys=True, indent=indent) - l = doc_path_by_namespace(spdx_deploy, spdx_doc.documentNamespace) + l = doc_path_by_namespace(spdx_deploy, [arch], spdx_doc.documentNamespace) l.parent.mkdir(exist_ok=True, parents=True) l.symlink_to(os.path.relpath(dest, l.parent)) - l = doc_path_by_hashfn(spdx_deploy, spdx_doc.name, d.getVar("BB_HASHFILENAME")) + l = doc_path_by_hashfn(spdx_deploy, [arch], spdx_doc.name, d.getVar("BB_HASHFILENAME")) l.parent.mkdir(exist_ok=True, parents=True) l.symlink_to(os.path.relpath(dest, l.parent)) From patchwork Wed Sep 20 10:57:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A346CE79C8 for ; Wed, 20 Sep 2023 10:58:07 +0000 (UTC) Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by mx.groups.io with SMTP id smtpd.web11.35323.1695207486569572711 for ; Wed, 20 Sep 2023 03:58:06 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=cKGfMyz4; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-32157c8e4c7so2840459f8f.1 for ; Wed, 20 Sep 2023 03:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695207485; x=1695812285; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Ab6sRdNu8qmYmtz92b1qDrAzE9i87qhPKZ7mkT/6QU4=; b=cKGfMyz4pEV8uCt7MyZgGkMiA3uPw3kzxSHVKFhqllQVzpa7aelwmOu7Qzby1UH/MX /AToFBFLI9xs4Ac2cTJX/tQoq76lVS6PieaFKzMRBZKKIvyOcveBMHSPsL/Yu+amN916 ySj5LRpnLD1k+05oEJ2RH5bNHf6S2SnWxRLDc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695207485; x=1695812285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ab6sRdNu8qmYmtz92b1qDrAzE9i87qhPKZ7mkT/6QU4=; b=oLBwvH87NmEwdSMfV8/hJycYjOjz+DC3yUd358KW7QlcYTVudfM1f2n23Qh2TudvVY +H18qK2xrYlOBGYYxHIyYfbznMzvnhHjzxzSUYbBMpqRkKigdye85bR77Gfpybf7joq/ +6GYzkD6o5MaGIaT6/7T1IZg2FKvWvx77JExCYrnYSugOlVKdcAOInwAvsElJjfYQo2t 5lxVlpCaaMXom3u9RQIzieS9sdJ8TEpjG41GHs0M7yq4n+5FnOgMJZhT4suEid5B3N// nF5LHj8DWOGNJBkDxC2nwMD2em4VcpRaPD0wZInlSYsCMq4/yFHTm2ZTJIfEYqNKAsp8 zz9Q== X-Gm-Message-State: AOJu0Yy1jMnN1bDyfs4CkB1/TtNabSU5KKk9HLi98Glu2n7MPQ7M9/Ep uSQ/f9M/KM7eU8IGfEVYNO1svheLU6XSaAZc8f8= X-Google-Smtp-Source: AGHT+IFYIHu8LViZPZEc1L/XINczX/v5PL2ylEwC/J6DspyMIpmqAfrkrazbDQRBgJ2WoCQ7WrSBgg== X-Received: by 2002:a5d:6947:0:b0:31f:f72c:df95 with SMTP id r7-20020a5d6947000000b0031ff72cdf95mr1658243wrw.21.1695207484772; Wed, 20 Sep 2023 03:58:04 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id h1-20020adfe981000000b0031f300a4c26sm426636wrm.93.2023.09.20.03.58.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 03:58:04 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/6] sstate: Stop allowing overlapping symlinks from sstate Date: Wed, 20 Sep 2023 11:57:59 +0100 Message-Id: <20230920105802.1008778-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> References: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 20 Sep 2023 10:58:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187919 When originally implemented, overlapping symlinks in DEPLOY_DIR were common. That is no longer the case and these overlapping links are causing bugs in other areas (e.g. bug 14123). Therefore start showing errors for overlapping symlinks in shared areas. Whilst here, fix a broken file reference in the grep command to match current file layouts and update the message shown to users to match current times. Most of the message content is obsolete now due to other advances and changes in the way the staging code now works. Signed-off-by: Richard Purdie --- meta/classes-global/sstate.bbclass | 30 ++++++------------------------ 1 file changed, 6 insertions(+), 24 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 706c2ae9388..afcda2980b3 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -266,7 +266,7 @@ def sstate_install(ss, d): overlap_allowed = (d.getVar("SSTATE_ALLOW_OVERLAP_FILES") or "").split() match = [] for f in sharedfiles: - if os.path.exists(f) and not os.path.islink(f): + if os.path.exists(f): f = os.path.normpath(f) realmatch = True for w in overlap_allowed: @@ -276,36 +276,18 @@ def sstate_install(ss, d): break if realmatch: match.append(f) - sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}")) + sstate_search_cmd = "grep -rlF '%s' %s --exclude=index-* | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}")) search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0] if search_output: match.append(" (matched in %s)" % search_output.decode('utf-8').rstrip()) else: match.append(" (not matched to any task)") if match: - bb.error("The recipe %s is trying to install files into a shared " \ - "area when those files already exist. Those files and their manifest " \ - "location are:\n %s\nPlease verify which recipe should provide the " \ - "above files.\n\nThe build has stopped, as continuing in this scenario WILL " \ - "break things - if not now, possibly in the future (we've seen builds fail " \ - "several months later). If the system knew how to recover from this " \ - "automatically it would, however there are several different scenarios " \ - "which can result in this and we don't know which one this is. It may be " \ - "you have switched providers of something like virtual/kernel (e.g. from " \ - "linux-yocto to linux-yocto-dev), in that case you need to execute the " \ - "clean task for both recipes and it will resolve this error. It may be " \ - "you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning " \ - "those recipes should again resolve this error, however switching " \ - "DISTRO_FEATURES on an existing build directory is not supported - you " \ - "should really clean out tmp and rebuild (reusing sstate should be safe). " \ - "It could be the overlapping files detected are harmless in which case " \ - "adding them to SSTATE_ALLOW_OVERLAP_FILES may be the correct solution. It could " \ - "also be your build is including two different conflicting versions of " \ - "things (e.g. bluez 4 and bluez 5 and the correct solution for that would " \ - "be to resolve the conflict. If in doubt, please ask on the mailing list, " \ - "sharing the error and filelist above." % \ + bb.fatal("Recipe %s is trying to install files into a shared " \ + "area when those files already exist. The files and the manifests listing " \ + "them are:\n %s\n" + "Please adjust the recipes so only one recipe provides a given file. " % \ (d.getVar('PN'), "\n ".join(match))) - bb.fatal("If the above message is too much, the simpler version is you're advised to wipe out tmp and rebuild (reusing sstate is fine). That will likely fix things in most (but not all) cases.") if ss['fixmedir'] and os.path.exists(ss['fixmedir'] + "/fixmepath.cmd"): sharedfiles.append(ss['fixmedir'] + "/fixmepath.cmd") From patchwork Wed Sep 20 10:58:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70F85CE79CA for ; Wed, 20 Sep 2023 10:58:17 +0000 (UTC) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by mx.groups.io with SMTP id smtpd.web10.35160.1695207487254364535 for ; Wed, 20 Sep 2023 03:58:07 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Krc2zcig; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.42, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-3216ba1b01eso1348478f8f.2 for ; Wed, 20 Sep 2023 03:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695207485; x=1695812285; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=S8Q3XVSQIF+0e8TJN32t2OX8fgNt8M8XIqe3MlRZdN4=; b=Krc2zcigJVM9c9LB2pHvkzuSS78AORWa/lo5nPsM8jISZXYQZYKCehWqVs1sYJgVX5 WtTqLst+c1Z24UFMusyO/bzU0hT5vodm+LAj5hPrKaV1QWfIgPIWIM80W1HbV1d11Oxr 2LmyBHB6PD4Cugn+9UZNwVHJfjLDD48Jh4JPM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695207485; x=1695812285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S8Q3XVSQIF+0e8TJN32t2OX8fgNt8M8XIqe3MlRZdN4=; b=MjlJWYcNVeY2O1T/3tNjelqmv//8imUreqVpuSRg3Du7NwYzUsoCtHpFlBd5G6B1Fx k+HJlp/i0CHfV3RoMcxpTlSpPsjLGa/aLXXbhX+qVDDKcEnDTVzbfXDb4sx9s4KoT/6X aORymczwEPqLaPihWdIRyEYsljETZY/pX5lZmx530j1e6LYVs5p/N1Sf8whISk4kpZoO upHNoAYXBZxR/CzPe2j9LUdFHkqcZmGKsyG427GjtXE59MQMimPcviCo8jD9mpHHdtk1 zPYuw8RwDyej3kMSKo8Hub5fNuK5Uf06PCgciK00qJF4o2tSMFe3Bqq/QjVwwVKvCRIR ntNw== X-Gm-Message-State: AOJu0YwO2X6Ve+JkRLstb35j4oGmFXm5jcBIpcaep57Y3TqT7AZwILhQ fdbR/Y5FlfRmf1y7H1S6tM7B75cXlW2CBnL5NEA= X-Google-Smtp-Source: AGHT+IGriTLzxQcsLnyDb28WSxSC0lHQzAsm3zJFd6wtyKPowQCiJ0SQEDz3Cf7Pkh+zaHJwTBmQ8g== X-Received: by 2002:a5d:47c6:0:b0:31f:d52a:82af with SMTP id o6-20020a5d47c6000000b0031fd52a82afmr3278357wrc.49.1695207485633; Wed, 20 Sep 2023 03:58:05 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id h1-20020adfe981000000b0031f300a4c26sm426636wrm.93.2023.09.20.03.58.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 03:58:05 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 4/6] sstate: Fix nativesdk entry in SSTATE_ARCHS Date: Wed, 20 Sep 2023 11:58:00 +0100 Message-Id: <20230920105802.1008778-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> References: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 20 Sep 2023 10:58:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187920 PACKAGE_ARCH gets refined by each target so this value isn't valid in all contexts. Tweak to use underlying variables to build it so that it remains valid in wider contexts and works with new usage in sdpx and license classes. Signed-off-by: Richard Purdie --- meta/classes-global/sstate.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index afcda2980b3..67070440a6c 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -88,7 +88,7 @@ SSTATE_ARCHS = " \ ${BUILD_ARCH}_${ORIGNATIVELSBSTRING} \ ${BUILD_ARCH}_${SDK_ARCH}_${SDK_OS} \ ${SDK_ARCH}_${SDK_OS} \ - ${SDK_ARCH}_${PACKAGE_ARCH} \ + ${SDK_ARCH}_${SDK_ARCH}-${SDKPKGSUFFIX} \ allarch \ ${PACKAGE_ARCH} \ ${PACKAGE_EXTRA_ARCHS} \ From patchwork Wed Sep 20 10:58:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70FD4CE79CB for ; Wed, 20 Sep 2023 10:58:17 +0000 (UTC) Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by mx.groups.io with SMTP id smtpd.web11.35326.1695207488203105516 for ; Wed, 20 Sep 2023 03:58:08 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=fSAl7C28; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-31c7912416bso6263883f8f.1 for ; Wed, 20 Sep 2023 03:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695207486; x=1695812286; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Cmx+gzqeyRnH2qIDgW0UQlU11PSGx0Kn/CBhxX9sEKA=; b=fSAl7C28Y8Ro/1yEZcXulwHd7Wgh3/PsVdXpSC7f6/dY/rVAyzRlDOo0Su+0DxdEXI auStAAt9dsyY0ryVFU/9sAr+jRo3CVAd2nzXZe44gF/Mlfm6mszMAjBwjUqfZwf/2WJl PpK6ke2so5i7dE/cJ9THq1CY9Nv4wbld0mGwg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695207486; x=1695812286; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cmx+gzqeyRnH2qIDgW0UQlU11PSGx0Kn/CBhxX9sEKA=; b=HDxcdPuu7LSYSxLWOAJmqj7ptvJDSV31BqpbyqVUkAa+eyNwXo1YH+R2t9POK/kNXi TRDGcJaXV0yU4RcB9rkvlO0tDu0Jq2l1XwlR+PO+s4FFVuosNCt37w49XyECM/Sd8aoT tT9MONBXZm1wUeMLUTJ+F6xUU6y2v3JS5OuKpVw+milVCv7mnZjAihpRiDWWyezAUxrP Y7yKmmPBHPABSmMarZAovyjbU+jzxjG2A1Vt53muWu0RtUd92e203j19yHTZ1uu2QTtr BY/D1Ii5lq1Jy3zwVczwpZtf4b4WWGkq2nGiTpHZZJsSVQ67RbEGVQjwTzpgJjniTaeO Tufw== X-Gm-Message-State: AOJu0Yw1e8N/Q9D43CdB9THECYzI9fM06LjiIspfFS2/3nQXZ5N8gKcn Pb7qK4lD5X8aEHwdxrzgTuO8sdaDTB+UmuNJ+AI= X-Google-Smtp-Source: AGHT+IH4+pHExUlGuroMgcJGdGdcKMkHySIR4Mnjrxup0iZ5Qjo+t0F1yG0jr6InQkOCJkj3q1Y/Dg== X-Received: by 2002:adf:cd0f:0:b0:314:1416:3be3 with SMTP id w15-20020adfcd0f000000b0031414163be3mr1674883wrm.70.1695207486373; Wed, 20 Sep 2023 03:58:06 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id h1-20020adfe981000000b0031f300a4c26sm426636wrm.93.2023.09.20.03.58.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 03:58:06 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 5/6 v2] multilib: fix SSTATE_ARCHS for multilib usage Date: Wed, 20 Sep 2023 11:58:01 +0100 Message-Id: <20230920105802.1008778-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> References: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 20 Sep 2023 10:58:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187921 When building multilibs, we need to inject the multilib sstate pkgarch into SSTATE_ARCHS so the list forms a complete search path. Add a tweak to do this. PACKAGE_ARCH defaults to TUNE_PKGARCH so this is equivalent and just guards against recipes changing the value which may have other unwanted side effects. Signed-off-by: Richard Purdie --- meta/classes-global/sstate.bbclass | 3 ++- meta/classes-recipe/image.bbclass | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) v2 - rework patch as we need multilib arches in 'normal' builds diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 67070440a6c..2676f18e0a2 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -83,6 +83,7 @@ SSTATE_HASHEQUIV_FILEMAP ?= " \ BB_HASHFILENAME = "False ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}" +SSTATE_ARCHS_TUNEPKG ??= "${TUNE_PKGARCH}" SSTATE_ARCHS = " \ ${BUILD_ARCH} \ ${BUILD_ARCH}_${ORIGNATIVELSBSTRING} \ @@ -90,7 +91,7 @@ SSTATE_ARCHS = " \ ${SDK_ARCH}_${SDK_OS} \ ${SDK_ARCH}_${SDK_ARCH}-${SDKPKGSUFFIX} \ allarch \ - ${PACKAGE_ARCH} \ + ${SSTATE_ARCHS_TUNEPKG} \ ${PACKAGE_EXTRA_ARCHS} \ ${MACHINE_ARCH}" SSTATE_ARCHS[vardepsexclude] = "ORIGNATIVELSBSTRING" diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass index 4f00162e789..7231fad940d 100644 --- a/meta/classes-recipe/image.bbclass +++ b/meta/classes-recipe/image.bbclass @@ -96,6 +96,7 @@ USE_DEPMOD ?= "1" PID = "${@os.getpid()}" PACKAGE_ARCH = "${MACHINE_ARCH}" +SSTATE_ARCHS_TUNEPKG = "${@all_multilib_tune_values(d, 'TUNE_PKGARCH')}" LDCONFIGDEPEND ?= "ldconfig-native:do_populate_sysroot" LDCONFIGDEPEND:libc-musl = "" From patchwork Wed Sep 20 10:58:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 30811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 791B3CE79CD for ; Wed, 20 Sep 2023 10:58:17 +0000 (UTC) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by mx.groups.io with SMTP id smtpd.web10.35161.1695207488918015823 for ; Wed, 20 Sep 2023 03:58:09 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Nm4YohWc; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.42, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-31c7912416bso6263895f8f.1 for ; Wed, 20 Sep 2023 03:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1695207487; x=1695812287; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=n42VeLZx3OUuo7cIFQ0rW03T/RAdnGMA8htNqiINBsw=; b=Nm4YohWccfztfJvrWuxwLayba/etvt0ShpnnSfEOewR3p8rPjKxsLt9T3ZYJaE2IFe CPwmq4cAPOfcUNUSiJc/7SZM3FUErQH0c8oKSuY/Qgnh3pVrvSGcBD0TwM5pBreGep50 QNZEK3CMxry/Hj57IY5+IakuCJ+OTLKsfOX0I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695207487; x=1695812287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n42VeLZx3OUuo7cIFQ0rW03T/RAdnGMA8htNqiINBsw=; b=eCfk2VLMxRJWKemuIaneMOhr/p03Gd4xlQvWhu2E/opL2UDqXVLhhtEwAZ9mvIbidr CQt6lS+MVfnE2YzBhGG0gSJhB9SEt8oyZ1oIBoMuqgP/SKVxXpu0qE99Vz/0w8Q//2in SUq6ZPh5HhAV0pdXz95SsIUz0ZRvRHmo4os0E3qp3Nsbojpe0+2j7ejQ3bSty4jWx6B9 pHSuSELlyX+EIIYNVAQTClhzpbPepsi6hZjq0vtDR/gofIHgArt5c9heFhLhTPP1h8vq RKX6RocxjIdAGt3VJBLUMiTWUZ3yCk+i5J4Dc02UShSa8fkJ90OyLhm+wfyiuCXGGWcZ 0S2Q== X-Gm-Message-State: AOJu0Yw3bG0tGh+0myV65MYyOHBGFzgjV1tTKVDiqrY98uJx12yHCYLe 1yHtV+sMcWYklCIqG/IEWSs1lf5zsz3gmorTcww= X-Google-Smtp-Source: AGHT+IHkNhlWDd6OAIs8xXAbDx1PnJEiCNug0Rn79vKX/xfDjat+fASluCBovThHdaryih4Ln9tSNg== X-Received: by 2002:a5d:6044:0:b0:316:efb9:101d with SMTP id j4-20020a5d6044000000b00316efb9101dmr1864091wrt.25.1695207487278; Wed, 20 Sep 2023 03:58:07 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:34a6:a1c2:97a:851d]) by smtp.gmail.com with ESMTPSA id h1-20020adfe981000000b0031f300a4c26sm426636wrm.93.2023.09.20.03.58.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 03:58:06 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 6/6] oeqa/selftest/bbtests: Improve and update test_non_gplv3 Date: Wed, 20 Sep 2023 11:58:02 +0100 Message-Id: <20230920105802.1008778-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> References: <20230920105802.1008778-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 20 Sep 2023 10:58:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187922 This test no longer worked with fixes to avoid overlapping license files in deploy. Fix the test but also improve the logging messages so we don't get "False is not True" messages upon failure. Signed-off-by: Richard Purdie --- meta/lib/oeqa/selftest/cases/bbtests.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/bbtests.py b/meta/lib/oeqa/selftest/cases/bbtests.py index 31aa5680f01..d242352ea2d 100644 --- a/meta/lib/oeqa/selftest/cases/bbtests.py +++ b/meta/lib/oeqa/selftest/cases/bbtests.py @@ -236,8 +236,11 @@ INHERIT:remove = \"report-error\" result = bitbake('selftest-ed', ignore_status=True) self.assertEqual(result.status, 0, "Bitbake failed, exit code %s, output %s" % (result.status, result.output)) lic_dir = get_bb_var('LICENSE_DIRECTORY') - self.assertFalse(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPL-3.0-or-later'))) - self.assertTrue(os.path.isfile(os.path.join(lic_dir, 'selftest-ed/generic_GPL-2.0-or-later'))) + arch = get_bb_var('SSTATE_PKGARCH') + filename = os.path.join(lic_dir, arch, 'selftest-ed', 'generic_GPL-3.0-or-later') + self.assertFalse(os.path.isfile(filename), msg="License file %s exists and shouldn't" % filename) + filename = os.path.join(lic_dir, arch, 'selftest-ed', 'generic_GPL-2.0-or-later') + self.assertTrue(os.path.isfile(filename), msg="License file %s doesn't exist" % filename) def test_setscene_only(self): """ Bitbake option to restore from sstate only within a build (i.e. execute no real tasks, only setscene)"""