From patchwork Tue Oct 31 22:47:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Louis Rannou X-Patchwork-Id: 33233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 394E0C4708E for ; Tue, 31 Oct 2023 22:48:15 +0000 (UTC) Received: from 16.mo583.mail-out.ovh.net (16.mo583.mail-out.ovh.net [87.98.174.144]) by mx.groups.io with SMTP id smtpd.web10.9389.1698792489638308483 for ; Tue, 31 Oct 2023 15:48:10 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=softfail (domain: syslinbit.com, ip: 87.98.174.144, mailfrom: louis.rannou@syslinbit.com) Received: from director9.ghost.mail-out.ovh.net (unknown [10.108.4.44]) by mo583.mail-out.ovh.net (Postfix) with ESMTP id 166FF287A5 for ; Tue, 31 Oct 2023 22:48:08 +0000 (UTC) Received: from ghost-submission-6684bf9d7b-nzml2 (unknown [10.110.208.94]) by director9.ghost.mail-out.ovh.net (Postfix) with ESMTPS id 688241FDBB; Tue, 31 Oct 2023 22:48:07 +0000 (UTC) Received: from syslinbit.com ([37.59.142.108]) by ghost-submission-6684bf9d7b-nzml2 with ESMTPSA id 4GDTEyeEQWUYnR8APQBgzw (envelope-from ); Tue, 31 Oct 2023 22:48:07 +0000 Authentication-Results: garm.ovh; auth=pass (GARM-108S002bf031180-bbc7-4a01-9298-57cced29fc53, E382B8EC8DEDBA5F41C2577A0B4F295D8A9180D4) smtp.auth=louis.rannou@syslinbit.com X-OVh-ClientIp: 45.81.62.9 From: Louis Rannou To: openembedded-core@lists.openembedded.org Cc: richard.purdie@linuxfoundation.org, jpewhacker@gmail.com, Samantha Jalabert , Louis Rannou Subject: [OE-core][RFC v2 11/12] create-spdx-3.0: support for License profile Date: Tue, 31 Oct 2023 23:47:32 +0100 Message-ID: <20231031224733.367227-12-louis.rannou@syslinbit.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231031224733.367227-1-louis.rannou@syslinbit.com> References: <20231031224733.367227-1-louis.rannou@syslinbit.com> MIME-Version: 1.0 X-Ovh-Tracer-Id: 9522861412599324125 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: -100 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvkedruddtfedgtddvucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepnfhouhhishcutfgrnhhnohhuuceolhhouhhishdrrhgrnhhnohhusehshihslhhinhgsihhtrdgtohhmqeenucggtffrrghtthgvrhhnpeegjefgfeeiveeifeekveefueelkeegueeitdettdevgfehheevlefhveevhedvudenucfkphepuddvjedrtddrtddruddpgeehrdekuddriedvrdelpdefjedrheelrddugedvrddutdeknecuvehluhhsthgvrhfuihiivgepheenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpeeolhhouhhishdrrhgrnhhnohhusehshihslhhinhgsihhtrdgtohhmqedpnhgspghrtghpthhtohepuddprhgtphhtthhopehophgvnhgvmhgsvgguuggvugdqtghorhgvsehlihhsthhsrdhophgvnhgvmhgsvgguuggvugdrohhrghdpoffvtefjohhsthepmhhoheekfedpmhhouggvpehsmhhtphhouhht List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 31 Oct 2023 22:48:15 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189887 From: Samantha Jalabert Add classes AnyLicenseInfo, LicenseExpression and SimpleLicensingText. Suppose inheritance of AnyLicenseInfo in LicenseExpression and SimpleLicensingText Add the option to enable Licensing Profile: SPDX_ENABLE_LICENSING = "1" Add methods to SPDX3SpdxDocument to return the list of existing SPDX3LicenseExpression and SPDX3SimpleLicensingText Split function convert_license_to_spdx into three separate functions and adapt them to match spdx3.0 classes Signed-off-by: Samantha Jalabert Signed-off-by: Louis Rannou --- meta/classes/create-spdx-3.0.bbclass | 197 +++++++++++++++++---------- meta/lib/oe/spdx3.py | 22 +++ 2 files changed, 148 insertions(+), 71 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 3ef01783a7..270d812abc 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -200,77 +200,102 @@ python() { d.setVar("SPDX_LICENSE_DATA", data) } -def convert_license_to_spdx(lic, document, d, existing={}): +def add_extracted_license(d, document, ident, name): from pathlib import Path - import oe.spdx + import oe.spdx3 - license_data = d.getVar("SPDX_LICENSE_DATA") - extracted = {} + extracted_info = oe.spdx3.SPDX3SimpleLicensingText() + extracted_info.name = name + extracted_info.licenseText = None - def add_extracted_license(ident, name): - nonlocal document + if name == "PD": + # Special-case this. + extracted_info.licenseText = "Software released to the public domain" + else: + # Seach for the license in COMMON_LICENSE_DIR and LICENSE_PATH + for directory in [d.getVar('COMMON_LICENSE_DIR')] + (d.getVar('LICENSE_PATH') or '').split(): + try: + with (Path(directory) / name).open(errors="replace") as f: + extracted_info.licenseText = f.read() + break + except FileNotFoundError: + pass + if extracted_info.licenseText is None: + # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set + filename = d.getVarFlag('NO_GENERIC_LICENSE', name) + if filename: + filename = d.expand("${S}/" + filename) + with open(filename, errors="replace") as f: + extracted_info.licenseText = f.read() + else: + bb.fatal("Cannot find any text for license %s" % name) - if name in extracted: - return + return extracted_info - extracted_info = oe.spdx.SPDX3ExtractedLicensingInfo() - extracted_info.name = name - extracted_info.licenseId = ident - extracted_info.extractedText = None +def convert(d, l, document): + import oe.spdx3 - if name == "PD": - # Special-case this. - extracted_info.extractedText = "Software released to the public domain" - else: - # Seach for the license in COMMON_LICENSE_DIR and LICENSE_PATH - for directory in [d.getVar('COMMON_LICENSE_DIR')] + (d.getVar('LICENSE_PATH') or '').split(): - try: - with (Path(directory) / name).open(errors="replace") as f: - extracted_info.extractedText = f.read() - break - except FileNotFoundError: - pass - if extracted_info.extractedText is None: - # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set - filename = d.getVarFlag('NO_GENERIC_LICENSE', name) - if filename: - filename = d.expand("${S}/" + filename) - with open(filename, errors="replace") as f: - extracted_info.extractedText = f.read() - else: - bb.fatal("Cannot find any text for license %s" % name) + license_data = d.getVar("SPDX_LICENSE_DATA") - extracted[name] = extracted_info - document.hasExtractedLicensingInfos.append(extracted_info) + if l == "(" or l == ")": + return l - def convert(l): - if l == "(" or l == ")": - return l + if l == "&": + return "AND" - if l == "&": - return "AND" + if l == "|": + return "OR" - if l == "|": - return "OR" + if l == "CLOSED": + return "NONE" - if l == "CLOSED": - return "NONE" + spdx_license = d.getVarFlag("SPDXLICENSEMAP", l) or l - spdx_license = d.getVarFlag("SPDXLICENSEMAP", l) or l - if spdx_license in license_data["licenses"]: - return spdx_license + if spdx_license in license_data["licenses"]: + lic = oe.spdx3.SPDX3LicenseExpression() + lic.licenseExpression = spdx_license + lic.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"] + return lic + else: + spdx_license = "LicenseText-" + l + return add_extracted_license(d, document, spdx_license, l) - try: - spdx_license = existing[l] - except KeyError: - spdx_license = "LicenseRef-" + l - add_extracted_license(spdx_license, l) - return spdx_license +def convert_license_to_spdx(lic, document, d, existing={}): + import oe.spdx3 + licenses_found = [] + licenses_id = [] + lic_split = lic.replace("(", " ( ").replace(")", " ) ").replace("|", " | ").replace("&", " & ").split() + for l in lic_split: + licenses_found.append(convert(d, l, document)) + + for element in licenses_found: + + existing_licenses = document.get_licenses() + + if isinstance(element, oe.spdx3.SPDX3LicenseExpression): + lic_type = "LicenseExpression" + else: + lic_type = "SimpleLicenseText" + + if isinstance(element, oe.spdx3.SPDX3AnyLicenseInfo) and not existing_licenses: + element.spdxId = new_spdxid(d, document, lic_type, "1") + licenses_id.append(element.spdxId) + document.element.append(element) + elif isinstance(element, oe.spdx3.SPDX3AnyLicenseInfo): + for existinglic in existing_licenses: + if ("licenseExpression" in element.properties() and "licenseExpression" in existinglic.properties() and element.licenseExpression == existinglic.licenseExpression) or \ + ("licenseText" in element.properties() and "licenseText" in existinglic.properties() and element.licenseText == existinglic.licenseText): + licenses_id.append(existinglic.spdxId) + break + + element.spdxId = new_spdxid(d, document, lic_type, str(len(existing_licenses) + 1)) + licenses_id.append(element.spdxId) + document.element.append(element) - return ' '.join(convert(l) for l in lic_split) + return licenses_id def process_sources(d): pn = d.getVar('PN') @@ -362,11 +387,31 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv hashSha256.hashValue = bb.utils.sha256_file(filepath) spdx_file.verifiedUsing.append(hashSha256) - # TODO: Rework when License Profile implemented - #if "SOURCE" in spdx_file.fileTypes: - # extracted_lics = extract_licenses(filepath) - # if extracted_lics: - # spdx_file.licenseInfoInFiles = extracted_lics + if d.getVar("SPDX_ENABLE_LICENSING") == "1" and \ + "source" in get_types(filepath): + extracted_lics = extract_licenses(filepath) + if extracted_lics: + for lic in extracted_lics: + current_licenses = doc.get_licenses_exp() + if current_licenses: + for c_lic in current_licenses: + if lic == c_lic.licenseExpression: + create_relationship(d, doc, spdx_file, "declaredLicense", c_lic) + break + + l = oe.spdx3.SPDX3LicenseExpression() + l.licenseExpression = lic + l.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"] + l.spdxId = new_spdxid(d, doc, "LicenseExpression", str(len(current_licenses) +1)) + doc.element.append(l) + create_relationship(d, doc, spdx_file, "declaredLicense", l) + else: + l = oe.spdx3.SPDX3LicenseExpression() + l.licenseExpression = lic + l.spdxId = new_spdxid(d, doc, "LicenseExpression", "1") + l.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"] + doc.element.append(l) + create_relationship(d, doc, spdx_file, "declaredLicense", l) doc.element.append(spdx_file) @@ -574,10 +619,13 @@ python do_create_spdx() { homepage = d.getVar("HOMEPAGE") if homepage: recipe.homePage = homepage -# TODO: Rework when License Profile implemented -# license = d.getVar("LICENSE") -# if license: -# recipe.licenseDeclared = convert_license_to_spdx(license, doc, d) + + if d.getVar("SPDX_ENABLE_LICENSING") == "1": + _license = d.getVar("LICENSE") + if _license: + licenseDeclared = convert_license_to_spdx(_license, doc, d) + for l in licenseDeclared: + create_relationship(d, doc, recipe, "declaredLicense", l) summary = d.getVar("SUMMARY") if summary: @@ -623,9 +671,14 @@ python do_create_spdx() { doc_sha1 = oe.sbom.write_doc(d, doc, doc, d.getVar("SSTATE_PKGARCH"), "recipes", indent=get_json_indent(d)) - #TODO: references - -# found_licenses = {license.name:recipe_ref.externalDocumentId + ":" + license.licenseId for license in doc.hasExtractedLicensingInfos} + # TODO: Recipe_ref not working with image_spdx_archive + #recipe_ref = oe.spdx3.SPDX3ExternalMap() + #recipe_ref.externalId = recipe.spdxId + #recipe_hash = oe.spdx3.SPDX3Hash() + #recipe_hash.algorithm = "sha1" + #recipe_hash.hashValue = doc_sha1 + #recipe_ref.verifiedUsing.append(recipe_hash) + #recipe_ref.definingDocument = get_doc_namespace(d, doc) if not recipe_spdx_is_native(doc, recipe): bb.build.exec_func("read_subpackage_metadata", d) @@ -643,8 +696,7 @@ python do_create_spdx() { generate_creationInfo(d, doc) # TODO: Rework when License Profile implemented - # package_doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"] - # package_doc.externalDocumentRefs.append(recipe_ref) + #doc.imports.append(recipe_ref) package_license = d.getVar("LICENSE:%s" % package) or d.getVar("LICENSE") @@ -653,9 +705,12 @@ python do_create_spdx() { spdx_package.spdxId = new_spdxid(d, doc, "package", pkg_name) spdx_package.name = pkg_name spdx_package.packageVersion = d.getVar("PV") - # TODO: Rework when License Profile implemented - #spdx_package.licenseDeclared = convert_license_to_spdx(package_license, package_doc, d, found_licenses) - spdx_package.suppliedBy = [ d.getVar("SPDX_SUPPLIER") ] + spdx_package.suppliedBy.append(get_supplier(d, doc)) + + if d.getVar("SPDX_ENABLE_LICENSING") == "1": + licenseDeclared = convert_license_to_spdx(package_license, doc, d) + for l in licenseDeclared: + create_relationship(d, doc, spdx_package, "declaredLicense", l) doc.element.append(spdx_package) diff --git a/meta/lib/oe/spdx3.py b/meta/lib/oe/spdx3.py index 9ab57ac015..0e00c7854e 100644 --- a/meta/lib/oe/spdx3.py +++ b/meta/lib/oe/spdx3.py @@ -286,6 +286,14 @@ class SPDX3SpdxDocument(SPDX3Bundle): return json.load(f, cls=Decoder) + def get_licenses(self): + licenses = [] + for el in self.element: + if isinstance(el, SPDX3AnyLicenseInfo): + licenses.append(el) + + return licenses + # # Profile: Software - Datatypes # @@ -323,6 +331,20 @@ class SPDX3Package(SPDX3SoftwareArtifact): class SPDX3File(SPDX3SoftwareArtifact): pass +# +# Profile: Simple Licensing +# + +class SPDX3AnyLicenseInfo(SPDX3Element): + pass + +class SPDX3LicenseExpression(SPDX3AnyLicenseInfo): + licenseExpression = _String() + licenseListVersion = _String() + +class SPDX3SimpleLicensingText(SPDX3AnyLicenseInfo): + licenseText = _String() + # # OpenEmbedded base class #