From patchwork Thu Nov 2 15:53:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 33469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E9C1C41535 for ; Thu, 2 Nov 2023 15:53:29 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.59]) by mx.groups.io with SMTP id smtpd.web10.34981.1698940405841993678 for ; Thu, 02 Nov 2023 08:53:28 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=JbpvjkvN; spf=pass (domain: weidmueller.com, ip: 40.107.22.59, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g2g2okfomYMD/n4k07Kr6v7O57cV8WIu0hgJolyll2ADSGX6kwKgU5TENYMJFvduEqpn3d4mNOGxMimQIygGBDGx/zNUdV7nC/ZHl2Xadf+e5I9+408aOLGeP4bvOKLUo1+WfBX/9KShrVlS8LbDm5mk6s+Pv+jfrrXAHzisx3nA6n6vHGTVhe9KmQ62tkV3Qa94IJHBO7xPRyd65cctZevGjAuTm0TVLhNJJ651VjHLd/knO9tpcPwyfBYLNKTn7KGh4V2UrOPPMAlgyjOK5mxIBksBGMJOedFXxQEgwFCMMXGpvmg+NHpyCWP6pfISaKtLIFxVivPzigrY03foLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9vGfgXRRyJOyeOB8xAqrx6Q6krN355Gi73g2EqyQ5gE=; b=DlCSMIHYIfuPMzEYofAKPu+f/WAKUAeeNxcc6F8QEQFpkJNCm+zS85quy0Ji/M41YdQ4JYnZcPOP3TwJym1n3CZzkro1qT3rep/G3FRln6girTvSNoKnp0QHGNn2Q3mdam0bCLiCRCozqhUPFAIrYIzM235HufpMvH0SgXM035JYURbQOkXOAfhuPze+KdP2VuL3i1vKGFKornNh8b7Uq0wKZgK77EG0F14+yArGww/W/9vBPkdeB/XO9maTrudW67+TE+E6uxuaAxbXGlHoa82JcowQBBUqtsxDtOJuDv+2OF5gP34nmSqP/UQZzJ0ENXiwQTKEeFLUywEDpv3C3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9vGfgXRRyJOyeOB8xAqrx6Q6krN355Gi73g2EqyQ5gE=; b=JbpvjkvNDGVQvs82YxsoFg66ernqGpvNgirV2lmr4sZrbmAdhUokuFi5mglkorU4vzRJZT0myPX8lR8GP8CpAUu3iwredrRNShzb19O0DUjm1FOprbld2qxwA/TfCBoV8Np95GFn2M2YogPrTFoEOCm5Yp6aiTRmovQKaJEoIYo= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PAWPR08MB10059.eurprd08.prod.outlook.com (2603:10a6:102:35d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.21; Thu, 2 Nov 2023 15:53:22 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c%7]) with mapi id 15.20.6954.019; Thu, 2 Nov 2023 15:53:22 +0000 From: lukas.funke-oss@weidmueller.com To: openembedded-core@lists.openembedded.org CC: Bruce Ashfield , Vyacheslav Yurkov , Martin Jansa , Peter Kjellerstedt , u.oelmann@pengutronix.de, Lukas Funke Subject: [OE-Core][PATCH v3 1/4] classes: go-vendor: Add go-vendor class Date: Thu, 2 Nov 2023 16:53:10 +0100 Message-ID: <20231102155313.309534-2-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> References: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0189.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ab::20) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PAWPR08MB10059:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d20a678-5a36-4ae4-6109-08dbdbbbd6ff X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ucv+wORl2mV6pw1wCF/X96ciwnz7J4uHIeuIIxEk/U52PruANDH1Z7ZT2qy3KbzJGLcbot9n87Xox3wYq4Gk5Zb5MF1e7QJA4pLMb1fEM7yHxRfgjmG/s5428LrUoYguigMbXE+OiQ7BdyVkKyc7yxLvPuSqtc4Mj78au9owXw8mNOF/bx76Bjcit65FeHwx3qCbLmqw0M+YNGYhy42UNkU5GkgPTQTqNIsqcb3dQFJVm1/t0Yc2oXdGf4MdHuuBNLTUCCiLt7kfvcpPIoenUndgfm+Da87STIlguuxvDHDkHP1foL40uJna3f3fmOMjD2DNBdCjPePxsJlD1iLzhoP4JF42BDRVjLaFUZh9MPq/ZRH8+oZDutH2DjHmP5IDdoLsEA842glfeDc2tsWNgriPN6dbP2bco410tHLY5cpFvcNKOzS5pHgD6pACYmpaFWpDi0j9jQT6zTs/Yz2UiWItTtLPFzeuiihyjOItNll8X4OpyaVPszXE4LgfgqmUimsfm8+/xwoORFlraEzmHWG3LyrdEmky1TRLo3oJrJ+p9Eqq1OHWMqi6SkrFZb0algFZzVcwsLF1arV7cWdqP07oN2zWb7lxcmGUA3eDD7NWjtzEiaBfEVy98ziIYvUz9T7J9TdiXu0WUZrHinOFtg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(346002)(39860400002)(366004)(396003)(376002)(230922051799003)(1800799009)(186009)(451199024)(64100799003)(84970400001)(478600001)(6506007)(52116002)(83380400001)(38100700002)(1076003)(107886003)(2616005)(6512007)(9686003)(41300700001)(2906002)(86362001)(36756003)(5660300002)(4326008)(8936002)(8676002)(6666004)(38350700005)(66556008)(66476007)(66946007)(6916009)(54906003)(316002)(26005)(966005)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1XvL9PGOzc1J+uCEa1i53gIkzTSmgXvSOAxpY8SEhd/HM7+UbeuTOd12A40x905skw0zcpwlU2GFw+OksHByjmgW8LfAt3cz91rPMX2R1h3KqjmaL7u7kB+xP4HKAa6bRbRz0V8J0PmADlsrQd2vRu1k1di2ms4Y/ergweEYsihpvoIIZg930kh7zabdaTq255ieWipALMf/y7Dft/KEte8f45sZRUEdk6/ZAIRzxPMNmnE81U/fq4dWWpNljknAtOx/f6KTV30RNYya1TbUmBfnTTqawNZcPtpKg/fRaEGJDynmoO5VQdaVlZNBZQebCybD0UvFpwG3ddietGAwWvYyBDWNhOM7mIb3nJ3yUejoxH8Zj9VWTsEMYVANAlMRY44/vG+XNIEu4CcOQ3F4+9lGDRZVQZfCF0LeNX5tqlYRwjZ6hZtd4cOLRViKKcQELHnSd/CEsR/xR6r38umImvfMVc5pdX33ZnDkew+90zdlAj9Pvph2wGuK5mX2yUZWPxTK+IFOFbnCHAZPg+XFPX8D1YTIZcUZbuG9utI3pTDcBElMD84Z1KXyMNdOC5X+Aby1aCvhKoQ3h7jw2Oa4pRV7ICjW4cxZaX8ebDpfrgno/wsKqXgoxXzD1St+1UaLvScqAHrYBq/CRjRXT5mll7ZcfdMOi+QJFNySgvT82WM4otQ1E5Kt8GZ6tpw6zD/P6zvxmYbkJA5MFl4z0IPAxZwFItMcLQjD1EpFsuwnT4vkmLRSpIEK2Nr5ZQfZDIN705UKbC9bbyJ4dnCWRBoDGvZRHjG9jC0ZcevZEe2r5pvwE/W1FLlhG53z2zAEscaWXZHuE7lgK0KLMy/U13nFRVBYHMWbjZUrJRpX76h1GrPsmbTfs898/moTOsqY9ftfYej1y/Pv4EV4loP734n7F82MHC5tyYkVRNUAPB6q78cv7rx96y2WVtaWlATrtK6x0iipcXbazuPksi/3BDcgrWZkd1QvOR/lhuqRUTLk6D4aaSUFFSsvhVcURcEcMaf+KkJJaalnc4hKJe9D9LJHPbYVG15tiBsejYwsq5z9l4BT6ojv8Q6/WcVVu/CagsPoqbmoBJmSzg49dhtfY4j3i1sKY6xt8zMDkUCc2Z3b0XPmCvX2B/4n3Lcss53SzNqj7YYATXGRw6lFvTRFdbD76ZeVNmkPCc6obQLJ/bzJ1bmYwv9xFO0x0fgQUp8nHf2bNGVUh8x0KL4dohay5P8MZwF9+8xunl1eBDE+VMyJgwtq6fmPxDFbH2mLpk0HZ8yoj/57XAUT03Qc5f3NSz2Mlvrm61Y3b0j+v60Kwbevnfmsq2pYIwe2WLfudniFKR+9RT127BB6dZ78CNs5HT0z9gbhT7D72sfz0QjSyxQLG0G++vaKfhkJe/2o0Ke2uEnXv3PGtNB6PfhzxHc8RI+Tl3nfxHKrgK0pQKClNEbss9l1NYog1G8yfL9u8E8tw2Lo/C93rbqN82vAsviei1f36duZdqAbhFcO7Or0Lzzx21TzSxGeYltCmzbWWKKtJKxunbX7IGNiJrkUik7Mm6MB0+4Uh+qpjl5tJrIZoU5Qddr3m7qbgijP6tjJ4LfdCHnrFsQopTPlxeC9QhoPdcumig== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d20a678-5a36-4ae4-6109-08dbdbbbd6ff X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 15:53:21.6932 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vqEcFdrScn3P39gktoOr1J+dNqAOgkXmUpbp7bDZYDD9xxN6Cajs6SM6UsIX+eyNxI+q+XbIfe0jZlerYTtKCg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10059 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 02 Nov 2023 15:53:29 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190084 From: Lukas Funke Signed-off-by: Lukas Funke --- meta/classes/go-vendor.bbclass | 200 +++++++++++++++++++++++++++++++++ 1 file changed, 200 insertions(+) create mode 100644 meta/classes/go-vendor.bbclass diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.bbclass new file mode 100644 index 0000000000..5b017b0b9d --- /dev/null +++ b/meta/classes/go-vendor.bbclass @@ -0,0 +1,200 @@ +# +# Copyright 2023 (C) Weidmueller GmbH & Co KG +# Author: Lukas Funke +# +# Handle Go vendor support for offline builds +# +# When importing Go modules, Go downloads the imported modules using +# a network (proxy) connection ahead of the compile stage. This contradicts +# the yocto build concept of fetching every source ahead of build-time +# and supporting offline builds. +# +# To support offline builds, we use Go 'vendoring': module dependencies are +# downloaded during the fetch-phase and unpacked into the modules 'vendor' +# folder. Additionally a manifest file is generated for the 'vendor' folder +# + +inherit go-mod + +def go_src_uri(repo, version, path=None, subdir=None, \ + vcs='git', replaces=None, pathmajor=None): + + destsuffix = "git/src/import/vendor.fetch" + module_path = repo if not path else path + + src_uri = "{}://{};name={}".format(vcs, repo, module_path.replace('/', '.')) + src_uri += ";destsuffix={}/{}@{}".format(destsuffix, repo, version) + + if vcs == "git": + src_uri += ";nobranch=1;protocol=https" + + src_uri += ";go_module_path={}".format(module_path) + + if replaces: + src_uri += ";go_module_replacement={}".format(replaces) + if subdir: + src_uri += ";go_subdir={}".format(subdir) + if pathmajor: + src_uri += ";go_pathmajor={}".format(pathmajor) + src_uri += ";is_go_dependency=1" + + return src_uri + +python do_vendor_unlink() { + + # We unlink + + go_import = d.getVar('GO_IMPORT') + source_dir = d.getVar('S') + linkname = os.path.join(source_dir, *['src', go_import, 'vendor']) + + os.unlink(linkname) +} + +addtask vendor_unlink before do_install after do_compile + +python do_go_vendor() { + import shutil + + src_uri = (d.getVar('SRC_URI') or "").split() + + if len(src_uri) == 0: + bb.error("SRC_URI is empty") + return + + default_destsuffix = "git/src/import/vendor.fetch" + fetcher = bb.fetch2.Fetch(src_uri, d) + go_import = d.getVar('GO_IMPORT') + source_dir = d.getVar('S') + + linkname = os.path.join(source_dir, *['src', go_import, 'vendor']) + vendor_dir = os.path.join(source_dir, *['src', 'import', 'vendor']) + import_dir = os.path.join(source_dir, *['src', 'import', 'vendor.fetch']) + + if os.path.exists(vendor_dir): + # Nothing to do except re-establish link to actual vendor folder + if not os.path.exists(linkname): + os.symlink(vendor_dir, linkname) + return + + bb.utils.mkdirhier(vendor_dir) + + modules = {} + + for url in fetcher.urls: + srcuri = fetcher.ud[url].host + fetcher.ud[url].path + + # Skip non Go module src uris + if not fetcher.ud[url].parm.get('is_go_dependency'): + continue + + destsuffix = fetcher.ud[url].parm.get('destsuffix') + # We derive the module repo / version in the following manner (exmaple): + # + # destsuffix = git/src/import/vendor.fetch/github.com/foo/bar@v1.2.3 + # p = github.com/foo/bar@v1.2.3 + # repo = github.com/foo/bar + # version = v1.2.3 + + p = destsuffix[len(default_destsuffix)+1:] + repo, version = p.split('@') + + module_path = fetcher.ud[url].parm.get('go_module_path') + + subdir = fetcher.ud[url].parm.get('go_subdir') + subdir = None if not subdir else subdir + + pathMajor = fetcher.ud[url].parm.get('go_pathmajor') + pathMajor = None if not pathMajor else pathMajor.strip('/') + + if not repo in modules: + modules[repo] = { "version": version, + "repo_path": os.path.join(import_dir, p), + "module_path": module_path, + "subdir": subdir, + "pathMajor": pathMajor } + + for module_key in sorted(modules): + + # only take the version which is explicitly listed + # as a dependency in the go.mod + module = modules[module_key] + module_path = module['module_path'] + rootdir = module['repo_path'] + subdir = module['subdir'] + pathMajor = module['pathMajor'] + + src = rootdir + + if subdir: + src = os.path.join(rootdir, subdir) + + # If the module is released at major version 2 or higher, the module + # path must end with a major version suffix like /v2. + # This may or may not be part of the subdirectory name + # + # https://go.dev/ref/mod#modules-overview + if pathMajor: + tmp = os.path.join(src, pathMajor) + # source directory including major version path may or may not exist + if os.path.exists(tmp): + src = tmp + + dst = os.path.join(vendor_dir, module_path) + + bb.debug(1, "cp %s --> %s" % (src, dst)) + shutil.copytree(src, dst, symlinks=True, \ + ignore=shutil.ignore_patterns(".git", \ + "vendor", \ + "*._test.go")) + + # If the root directory has a LICENSE file but not the subdir + # we copy the root license to the sub module since the license + # applies to all modules in the repository + # see https://go.dev/ref/mod#vcs-license + if subdir: + rootdirLicese = os.path.join(rootdir, "LICENSE") + subdirLicense = os.path.join(src, "LICENSE") + + if not os.path.exists(subdir) and \ + os.path.exists(rootdirLicese): + shutil.copy2(rootdirLicese, subdirLicense) + + # Copy vendor manifest + modules_txt_src = os.path.join(d.getVar('WORKDIR'), "modules.txt") + bb.debug(1, "cp %s --> %s" % (modules_txt_src, vendor_dir)) + shutil.copy2(modules_txt_src, vendor_dir) + + # Clean up vendor dir + # We only require the modules in the modules_txt file + fetched_paths = set([os.path.relpath(x[0], vendor_dir) for x in os.walk(vendor_dir)]) + + # Remove toplevel dir + fetched_paths.remove('.') + + vendored_paths = set() + with open(modules_txt_src) as f: + for line in f: + if not line.startswith("#"): + line = line.strip() + vendored_paths.add(line) + + # Add toplevel dirs into vendored dir, as we want to keep them + topdir = os.path.dirname(line) + while len(topdir): + if not topdir in vendored_paths: + vendored_paths.add(topdir) + + topdir = os.path.dirname(topdir) + + for path in fetched_paths: + if path not in vendored_paths: + realpath = os.path.join(vendor_dir, path) + if os.path.exists(realpath): + shutil.rmtree(realpath) + + # Create a symlink the the actual directory + os.symlink(vendor_dir, linkname) +} + +addtask go_vendor before do_patch after do_unpack From patchwork Thu Nov 2 15:53:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 33470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13AFEC4332F for ; Thu, 2 Nov 2023 15:53:39 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.59]) by mx.groups.io with SMTP id smtpd.web10.34981.1698940405841993678 for ; Thu, 02 Nov 2023 08:53:29 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=dQQKMPKP; spf=pass (domain: weidmueller.com, ip: 40.107.22.59, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qh6p2T04irVKJueIsPOEB31wUljHUK7bsIGHsWdwePaaOVC8YsZzIq1QMrClzSrf97KeygNAvcrtJ0laRiQGjW76RRpplkbgvfynkxE4DuQFcpBHOdtd43Ylz/Hu6NWWcPC4puJQVtB/qRYIeOoboK3MdfN3hjOQeRO1lqyJZ94ir55G0sIECfQfrS+sB+tNMfcXolWgZhfbxkHsBPpGAxWvQelI0VnALWs5sBoas5aOlCH3wob+M/pZh4rTMWOgz6OjSGmmWw+J+R5No4ZqCVicwoCpZB+ArX9sJ4VFE1Dit5yuAZ24vXCo70Ae15m/DbFhjUDp+rneXaXOXSXQWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5+C266ydI9GNGv1NWuP4ArggEAvXF0p8ahHlHV9Vxb8=; b=Hl/qF6c3ZbUlVDnFpXM3qkj7uGjZYwfZA4/k6RRbOJ7s8liz3VfOOsb9kxkutxLzEBQ10AyhZsZqyREcxRs7w2hw1FvWQE3yDAfhy0gXKuCp2MbaNW91IOQ4+jicUtSMKIHTEREQm692QnP08B4iCj1KRtrC+RD/fWecQb1Y8JsC/uViBfocp3NNp5cpa4a7vvtpkgCFB2yesAVD1Z83c89KPpXPhlF5b+e6hbxrTQ/O7mKgdPGfPzRkjN7fZ/oVMvGXoJh7vWrcoW9sGttsTd+JdagtQ26tJ4BPUDFSvTGpGWzEQSMe25NaBDVu2CnQg1LQemAHEVauX2IK8MS6NA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5+C266ydI9GNGv1NWuP4ArggEAvXF0p8ahHlHV9Vxb8=; b=dQQKMPKPlS7kcZ88ffZIhAQ74tYyUKKq5xEU/+kujsTr0oru0ngSrx2+EbpxWs0DDds/x/JpjCGUWMO4+KKtPYoGobvx3ACTIwNfzAM+cnDwlYcj7ddIIVyEePU46WJA6QYQkH6wnEZPB1Z+xXrAUKTiNdEvpCGwpxNDJcErp8Y= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PAWPR08MB10059.eurprd08.prod.outlook.com (2603:10a6:102:35d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.21; Thu, 2 Nov 2023 15:53:22 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c%7]) with mapi id 15.20.6954.019; Thu, 2 Nov 2023 15:53:22 +0000 From: lukas.funke-oss@weidmueller.com To: openembedded-core@lists.openembedded.org CC: Bruce Ashfield , Vyacheslav Yurkov , Martin Jansa , Peter Kjellerstedt , u.oelmann@pengutronix.de, Lukas Funke Subject: [OE-Core][PATCH v3 2/4] selftest: recipetool: Add test for go recipe handler Date: Thu, 2 Nov 2023 16:53:11 +0100 Message-ID: <20231102155313.309534-3-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> References: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0189.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ab::20) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PAWPR08MB10059:EE_ X-MS-Office365-Filtering-Correlation-Id: 3e2359f9-bdad-444a-7147-08dbdbbbd750 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sTCPR6NitOYRnrc+25aBgUXc3SSywu+0+iFipHqGMdCXFviu0M/F8xZCmQlq+CVb3sbRdXFoKLRgUrSEKmAu7XpatP4YrHMK9dT0tiTujzxfqRRLYWpjw37DzTx+PdEXtP0q7Ib7KboLXoB+h1Mv/EPgDixMXTwsaNe0PzCaEl46abo+0YTCjhpnSrtsmZhXk4RAb+2/liMraEfJn5K8dmmXp/XCWOTl9FIdRUY+U9ntoCx90DVIuM9SLO9SnyDP4ZiQ761rNAwFppA9G7NnbU0hYPo3WsTABM2itp6IaZFO4zrbY9w5MOJIQ7Ov5wFzou6tIpZw9fvH50CubVMmcc2lJRUMjbKzOxI+wIcZY4M0RjFOG4NPH0+U9wXRVPudGFqN2RxZJaXlfXFHdhL+XWQ7jmhoEBkWe2dWejFj0IuUBC7HW37TKXFq508LijbB/f89ox793/5cBvZg1oszRpu3D4Ir/f2ohTbKzu1ZacOZmvn4k7OWCB5HNcVD9EqDzHI2eVilCRp1bSCmCNCoo3O0Tq1vJsyBU4LpALmMTZcIen2/JiJXpGow8ID3GMdYO9UqSAlqAazlV4VTs2gDtZcPY5P4iNLBCH/C/NyLbg4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(346002)(39860400002)(366004)(396003)(376002)(230922051799003)(1800799009)(186009)(451199024)(64100799003)(478600001)(6506007)(52116002)(45080400002)(83380400001)(38100700002)(1076003)(107886003)(2616005)(6512007)(9686003)(41300700001)(30864003)(2906002)(86362001)(36756003)(5660300002)(4326008)(8936002)(8676002)(6666004)(38350700005)(66556008)(66476007)(66946007)(6916009)(54906003)(316002)(26005)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xU1szaeD+vtlFQ8+NEAaKN7WEuw2dwKuXxnJdexacJlgHlIamK6OIKK3Xy9ILDuUuEV1wzDsRQvNjBlFExOH2KZaMvvYJeoxjLmLbiYcZbYcs5WwkFDDbvAULV7enLtmVEjjh0/R0cEVyTd15lcw1w0l3rraDKTectlYPv7W4uJnos85wJNiHzHy0Ht60LSiu142gt5i7WwqT0t9DYnDwpJtMa/RAr173qw9y6a79w/soCcee57bK4ulv/a21u6mej+V8HVX/U8Dc5SJwMsQJFjK6ZBsDB/PO2qqDCw4FhTfJIzISSI9ZdAQLO+PuzlldAHreAtcHd5EarPQ5H71V8HucWDjF0fZZkZEkhfYJtvDF8yrrirEZlZXalEM9TgZfyICaHumq0b000axGQ90IfnMMdDekP7U+fFNS9pLWGF8QuUKd5d7YAYbppvm/uGX5l/QXtd0hH9aolB+PXzVwzN3ZcsxjpfBU8WhO7vg9K77QPYuKdnpD+zsqSw2xsA9wK/H57S20PEQq2cqM445WoVtGYrgCSXwch2660KFuVK7GoK/UVFmZce8qcJD3Z7Ydx5F9tXXzij57oPPzkZrBF6cbTFruxNh0BRin02PmsPPIz1CgyI4IC2u+GzlvFLC1lAmnkPTxItK9HRCT63axVkrtKZBTPHH6b4U38AUE8teZbLlJHbcb50k84i27BxazkpjuDNYLaKNAmZuX7pODlF614uvXLnbISrfx5D16Yqpw7O8hNkezGf38fSpF3wAilJchNXfTpGxlKW2eJI92k4gU/EVSVjbIlnSKfm758rf7l2JdrvnUBfvZao7TAAhwnobTdvfwzHgRT9Y+P+dZFZbegoCquedk75Iyxc2NN3Pf0gfn5xEG3VORZgvgMLdTRhtBHAU2ilpt+n6TifchuM8T1dZbndT+BOKseVOxPObD4CTGPtchvgmt9aB+7EAHNqmwxNOlC4lrApUr6hC3Yy26aRJINlcZUxGnkVnQDKFZreFZk/nHODtf8YtH7E7g07iUMSygFeRAP25JKJQLTI9w7admhxkSE1B9gGqotWCwaGyjnbrBI3T8kSmqoA2EzWyRHE5JLpbZDFnUq61ipaxK5puxJjJcmjdnN/lIlyZmyLo9oNVSyDM8m8CvzkgWPYA47HkUTO9JqTO8Pclz2EbEplYZhavvxuIMfzLg2UZMhbWDMlk+0EvkDAvPqHWiw5XDzlAsbSLzzxKcI6cFixjEPXI+NXbT6cm0KxoKv+1hQ7oow5zUY30toCW2U1DVdYKfu6l2ikpOV3Tt+OBdQQWEYyN+KwmQfeBXBpHyRxTj2OsUcfnPMd6A4QtXcFW2NigEdSJJFoXzRsxFSRsDWj8GGACOkvojgxCxjY1vSoOjDLdu6mPDxjzxb4JdGpc2A2oMNLAy2r3tf24XwpiJDiyL+RPVpz3dIV/s3mSboAp4bipy6VgC2Msu7z2KoQ1fkogUOorCo73niNksFSd8glZwZc4nksZHnQtMOlbE0rIoAITgWIxHNc39dIwFa/6oDKL9YubXYLK1IRO2iBuuaOPA+UFed4jq81RkIA6y8U1S+ylHzaLGOkscsd+SxSM9/whKHZ7IkZM9QhyRxb7jQ== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3e2359f9-bdad-444a-7147-08dbdbbbd750 X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 15:53:22.1826 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: M12CASAtOpLZVCg8TOx+Q7JLG8lApalJeFhjlxW9mREL0A5fu/+zpICXeLMnEz76A+mV026o8ZazSjvRAP0jlw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10059 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 02 Nov 2023 15:53:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190085 From: Lukas Funke This commit adds a test for the go recipetool handler. The choosen go project to test the created recipe was picked randomly. The SRC_URIs and the LIC_FILES_CHKSUMs are checked against there reference values. Signed-off-by: Lukas Funke --- meta/lib/oeqa/selftest/cases/recipetool.py | 185 +++++++++++++++++++++ 1 file changed, 185 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 48661bee6f..73b5ce1d83 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -532,6 +532,191 @@ class RecipetoolTests(RecipetoolBase): libpath = os.path.join(get_bb_var('COREBASE'), 'scripts', 'lib', 'recipetool') sys.path.insert(0, libpath) + def test_recipetool_create_go(self): + # Basic test to check go recipe generation + def urifiy(url, version, modulepath = None, pathmajor = None, subdir = None): + modulepath = ",path='%s'" % modulepath if len(modulepath) else '' + pathmajor = ",pathmajor='%s'" % pathmajor if len(pathmajor) else '' + subdir = ",subdir='%s'" % subdir if len(subdir) else '' + return "${@go_src_uri('%s','%s'%s%s%s)}" % (url, version, modulepath, pathmajor, subdir) + + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + + recipefile = os.path.join(temprecipe, 'edgex-go_git.bb') + deps_require_file = os.path.join(temprecipe, 'edgex-go', 'edgex-go-modules.inc') + lics_require_file = os.path.join(temprecipe, 'edgex-go', 'edgex-go-licenses.inc') + modules_txt_file = os.path.join(temprecipe, 'edgex-go', 'modules.txt') + + srcuri = 'https://github.com/edgexfoundry/edgex-go.git' + srcrev = "v3.0.0" + srcbranch = "main" + + result = runCmd('recipetool create -o %s %s -S %s -B %s' % (temprecipe, srcuri, srcrev, srcbranch)) + + self.maxDiff = None + inherits = ['go-vendor'] + + checkvars = {} + checkvars['GO_IMPORT'] = "github.com/edgexfoundry/edgex-go" + checkvars['SRC_URI'] = {'git://${GO_IMPORT};destsuffix=git/src/${GO_IMPORT};nobranch=1;name=${BPN};protocol=https', + 'file://modules.txt'} + checkvars['LIC_FILES_CHKSUM'] = {'file://src/${GO_IMPORT}/LICENSE;md5=8f8bc924cf73f6a32381e5fd4c58d603'} + + self.assertTrue(os.path.isfile(recipefile)) + self._test_recipe_contents(recipefile, checkvars, inherits) + + checkvars = {} + checkvars['VENDORED_LIC_FILES_CHKSUM'] = set( + ['file://src/${GO_IMPORT}/vendor/github.com/Microsoft/go-winio/LICENSE;md5=69205ff73858f2c22b2ca135b557e8ef', + 'file://src/${GO_IMPORT}/vendor/github.com/armon/go-metrics/LICENSE;md5=d2d77030c0183e3d1e66d26dc1f243be', + 'file://src/${GO_IMPORT}/vendor/github.com/cenkalti/backoff/LICENSE;md5=1571d94433e3f3aa05267efd4dbea68b', + 'file://src/${GO_IMPORT}/vendor/github.com/davecgh/go-spew/LICENSE;md5=c06795ed54b2a35ebeeb543cd3a73e56', + 'file://src/${GO_IMPORT}/vendor/github.com/eclipse/paho.mqtt.golang/LICENSE;md5=dcdb33474b60c38efd27356d8f2edec7', + 'file://src/${GO_IMPORT}/vendor/github.com/eclipse/paho.mqtt.golang/edl-v10;md5=3adfcc70f5aeb7a44f3f9b495aa1fbf3', + 'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-bootstrap/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff', + 'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-configuration/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff', + 'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-core-contracts/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff', + 'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-messaging/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff', + 'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-registry/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff', + 'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-secrets/v3/LICENSE;md5=f9fa2f4f8e0ef8cc7b5dd150963eb457', + 'file://src/${GO_IMPORT}/vendor/github.com/fatih/color/LICENSE.md;md5=316e6d590bdcde7993fb175662c0dd5a', + 'file://src/${GO_IMPORT}/vendor/github.com/fxamacker/cbor/v2/LICENSE;md5=827f5a2fa861382d35a3943adf9ebb86', + 'file://src/${GO_IMPORT}/vendor/github.com/go-jose/go-jose/v3/LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57', + 'file://src/${GO_IMPORT}/vendor/github.com/go-jose/go-jose/v3/json/LICENSE;md5=591778525c869cdde0ab5a1bf283cd81', + 'file://src/${GO_IMPORT}/vendor/github.com/go-kit/log/LICENSE;md5=5b7c15ad5fffe2ff6e9d58a6c161f082', + 'file://src/${GO_IMPORT}/vendor/github.com/go-logfmt/logfmt/LICENSE;md5=98e39517c38127f969de33057067091e', + 'file://src/${GO_IMPORT}/vendor/github.com/go-playground/locales/LICENSE;md5=3ccbda375ee345400ad1da85ba522301', + 'file://src/${GO_IMPORT}/vendor/github.com/go-playground/universal-translator/LICENSE;md5=2e2b21ef8f61057977d27c727c84bef1', + 'file://src/${GO_IMPORT}/vendor/github.com/go-playground/validator/v10/LICENSE;md5=a718a0f318d76f7c5d510cbae84f0b60', + 'file://src/${GO_IMPORT}/vendor/github.com/go-redis/redis/v7/LICENSE;md5=58103aa5ea1ee9b7a369c9c4a95ef9b5', + 'file://src/${GO_IMPORT}/vendor/github.com/golang/protobuf/LICENSE;md5=939cce1ec101726fa754e698ac871622', + 'file://src/${GO_IMPORT}/vendor/github.com/gomodule/redigo/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93', + 'file://src/${GO_IMPORT}/vendor/github.com/google/uuid/LICENSE;md5=88073b6dd8ec00fe09da59e0b6dfded1', + 'file://src/${GO_IMPORT}/vendor/github.com/gorilla/mux/LICENSE;md5=33fa1116c45f9e8de714033f99edde13', + 'file://src/${GO_IMPORT}/vendor/github.com/gorilla/websocket/LICENSE;md5=c007b54a1743d596f46b2748d9f8c044', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/consul/api/LICENSE;md5=b8a277a612171b7526e9be072f405ef4', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/errwrap/LICENSE;md5=b278a92d2c1509760384428817710378', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/go-cleanhttp/LICENSE;md5=65d26fcc2f35ea6a181ac777e42db1ea', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/go-hclog/LICENSE;md5=ec7f605b74b9ad03347d0a93a5cc7eb8', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/go-immutable-radix/LICENSE;md5=65d26fcc2f35ea6a181ac777e42db1ea', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/go-multierror/LICENSE;md5=d44fdeb607e2d2614db9464dbedd4094', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/go-rootcerts/LICENSE;md5=65d26fcc2f35ea6a181ac777e42db1ea', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/golang-lru/LICENSE;md5=f27a50d2e878867827842f2c60e30bfc', + 'file://src/${GO_IMPORT}/vendor/github.com/hashicorp/serf/LICENSE;md5=b278a92d2c1509760384428817710378', + 'file://src/${GO_IMPORT}/vendor/github.com/leodido/go-urn/LICENSE;md5=8f50db5538ec1148a9b3d14ed96c3418', + 'file://src/${GO_IMPORT}/vendor/github.com/mattn/go-colorable/LICENSE;md5=24ce168f90aec2456a73de1839037245', + 'file://src/${GO_IMPORT}/vendor/github.com/mattn/go-isatty/LICENSE;md5=f509beadd5a11227c27b5d2ad6c9f2c6', + 'file://src/${GO_IMPORT}/vendor/github.com/mitchellh/consulstructure/LICENSE;md5=96ada10a9e51c98c4656f2cede08c673', + 'file://src/${GO_IMPORT}/vendor/github.com/mitchellh/copystructure/LICENSE;md5=56da355a12d4821cda57b8f23ec34bc4', + 'file://src/${GO_IMPORT}/vendor/github.com/mitchellh/go-homedir/LICENSE;md5=3f7765c3d4f58e1f84c4313cecf0f5bd', + 'file://src/${GO_IMPORT}/vendor/github.com/mitchellh/mapstructure/LICENSE;md5=3f7765c3d4f58e1f84c4313cecf0f5bd', + 'file://src/${GO_IMPORT}/vendor/github.com/mitchellh/reflectwalk/LICENSE;md5=3f7765c3d4f58e1f84c4313cecf0f5bd', + 'file://src/${GO_IMPORT}/vendor/github.com/nats-io/nats.go/LICENSE;md5=86d3f3a95c324c9479bd8986968f4327', + 'file://src/${GO_IMPORT}/vendor/github.com/nats-io/nkeys/LICENSE;md5=86d3f3a95c324c9479bd8986968f4327', + 'file://src/${GO_IMPORT}/vendor/github.com/nats-io/nuid/LICENSE;md5=86d3f3a95c324c9479bd8986968f4327', + 'file://src/${GO_IMPORT}/vendor/github.com/pmezard/go-difflib/LICENSE;md5=e9a2ebb8de779a07500ddecca806145e', + 'file://src/${GO_IMPORT}/vendor/github.com/rcrowley/go-metrics/LICENSE;md5=1bdf5d819f50f141366dabce3be1460f', + 'file://src/${GO_IMPORT}/vendor/github.com/spiffe/go-spiffe/v2/LICENSE;md5=86d3f3a95c324c9479bd8986968f4327', + 'file://src/${GO_IMPORT}/vendor/github.com/stretchr/objx/LICENSE;md5=d023fd31d3ca39ec61eec65a91732735', + 'file://src/${GO_IMPORT}/vendor/github.com/stretchr/testify/LICENSE;md5=188f01994659f3c0d310612333d2a26f', + 'file://src/${GO_IMPORT}/vendor/github.com/x448/float16/LICENSE;md5=de8f8e025d57fe7ee0b67f30d571323b', + 'file://src/${GO_IMPORT}/vendor/github.com/zeebo/errs/LICENSE;md5=84914ab36fc0eb48edbaa53e66e8d326', + 'file://src/${GO_IMPORT}/vendor/golang.org/x/crypto/LICENSE;md5=5d4950ecb7b26d2c5e4e7b4e0dd74707', + 'file://src/${GO_IMPORT}/vendor/golang.org/x/mod/LICENSE;md5=5d4950ecb7b26d2c5e4e7b4e0dd74707', + 'file://src/${GO_IMPORT}/vendor/golang.org/x/net/LICENSE;md5=5d4950ecb7b26d2c5e4e7b4e0dd74707', + 'file://src/${GO_IMPORT}/vendor/golang.org/x/sync/LICENSE;md5=5d4950ecb7b26d2c5e4e7b4e0dd74707', + 'file://src/${GO_IMPORT}/vendor/golang.org/x/sys/LICENSE;md5=5d4950ecb7b26d2c5e4e7b4e0dd74707', + 'file://src/${GO_IMPORT}/vendor/golang.org/x/text/LICENSE;md5=5d4950ecb7b26d2c5e4e7b4e0dd74707', + 'file://src/${GO_IMPORT}/vendor/golang.org/x/tools/LICENSE;md5=5d4950ecb7b26d2c5e4e7b4e0dd74707', + 'file://src/${GO_IMPORT}/vendor/google.golang.org/genproto/LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57', + 'file://src/${GO_IMPORT}/vendor/google.golang.org/grpc/LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57', + 'file://src/${GO_IMPORT}/vendor/google.golang.org/protobuf/LICENSE;md5=02d4002e9171d41a8fad93aa7faf3956', + 'file://src/${GO_IMPORT}/vendor/gopkg.in/eapache/queue.v1/LICENSE;md5=1bfd4408d3de090ef6b908b0cc45a316', + 'file://src/${GO_IMPORT}/vendor/gopkg.in/yaml.v3/LICENSE;md5=3c91c17266710e16afdbb2b6d15c761c']) + + self.assertTrue(os.path.isfile(lics_require_file)) + self._test_recipe_contents(lics_require_file, checkvars, []) + + dependencies = \ + [ ('github.com/eclipse/paho.mqtt.golang','v1.4.2', '', '', ''), + ('github.com/edgexfoundry/go-mod-bootstrap','v3.0.1','github.com/edgexfoundry/go-mod-bootstrap/v3','/v3', ''), + ('github.com/edgexfoundry/go-mod-configuration','v3.0.0','github.com/edgexfoundry/go-mod-configuration/v3','/v3', ''), + ('github.com/edgexfoundry/go-mod-core-contracts','v3.0.0','github.com/edgexfoundry/go-mod-core-contracts/v3','/v3', ''), + ('github.com/edgexfoundry/go-mod-messaging','v3.0.0','github.com/edgexfoundry/go-mod-messaging/v3','/v3', ''), + ('github.com/edgexfoundry/go-mod-secrets','v3.0.1','github.com/edgexfoundry/go-mod-secrets/v3','/v3', ''), + ('github.com/fxamacker/cbor','v2.4.0','github.com/fxamacker/cbor/v2','/v2', ''), + ('github.com/gomodule/redigo','v1.8.9', '', '', ''), + ('github.com/google/uuid','v1.3.0', '', '', ''), + ('github.com/gorilla/mux','v1.8.0', '', '', ''), + ('github.com/rcrowley/go-metrics','v0.0.0-20201227073835-cf1acfcdf475', '', '', ''), + ('github.com/spiffe/go-spiffe','v2.1.4','github.com/spiffe/go-spiffe/v2','/v2', ''), + ('github.com/stretchr/testify','v1.8.2', '', '', ''), + ('go.googlesource.com/crypto','v0.8.0','golang.org/x/crypto', '', ''), + ('gopkg.in/eapache/queue.v1','v1.1.0', '', '', ''), + ('gopkg.in/yaml.v3','v3.0.1', '', '', ''), + ('github.com/microsoft/go-winio','v0.6.0','github.com/Microsoft/go-winio', '', ''), + ('github.com/hashicorp/go-metrics','v0.3.10','github.com/armon/go-metrics', '', ''), + ('github.com/cenkalti/backoff','v2.2.1+incompatible', '', '', ''), + ('github.com/davecgh/go-spew','v1.1.1', '', '', ''), + ('github.com/edgexfoundry/go-mod-registry','v3.0.0','github.com/edgexfoundry/go-mod-registry/v3','/v3', ''), + ('github.com/fatih/color','v1.9.0', '', '', ''), + ('github.com/go-jose/go-jose','v3.0.0','github.com/go-jose/go-jose/v3','/v3', ''), + ('github.com/go-kit/log','v0.2.1', '', '', ''), + ('github.com/go-logfmt/logfmt','v0.5.1', '', '', ''), + ('github.com/go-playground/locales','v0.14.1', '', '', ''), + ('github.com/go-playground/universal-translator','v0.18.1', '', '', ''), + ('github.com/go-playground/validator','v10.13.0','github.com/go-playground/validator/v10','/v10', ''), + ('github.com/go-redis/redis','v7.3.0','github.com/go-redis/redis/v7','/v7', ''), + ('github.com/golang/protobuf','v1.5.2', '', '', ''), + ('github.com/gorilla/websocket','v1.4.2', '', '', ''), + ('github.com/hashicorp/consul','v1.20.0','github.com/hashicorp/consul/api', '', 'api'), + ('github.com/hashicorp/errwrap','v1.0.0', '', '', ''), + ('github.com/hashicorp/go-cleanhttp','v0.5.1', '', '', ''), + ('github.com/hashicorp/go-hclog','v0.14.1', '', '', ''), + ('github.com/hashicorp/go-immutable-radix','v1.3.0', '', '', ''), + ('github.com/hashicorp/go-multierror','v1.1.1', '', '', ''), + ('github.com/hashicorp/go-rootcerts','v1.0.2', '', '', ''), + ('github.com/hashicorp/golang-lru','v0.5.4', '', '', ''), + ('github.com/hashicorp/serf','v0.10.1', '', '', ''), + ('github.com/leodido/go-urn','v1.2.3', '', '', ''), + ('github.com/mattn/go-colorable','v0.1.12', '', '', ''), + ('github.com/mattn/go-isatty','v0.0.14', '', '', ''), + ('github.com/mitchellh/consulstructure','v0.0.0-20190329231841-56fdc4d2da54', '', '', ''), + ('github.com/mitchellh/copystructure','v1.2.0', '', '', ''), + ('github.com/mitchellh/go-homedir','v1.1.0', '', '', ''), + ('github.com/mitchellh/mapstructure','v1.5.0', '', '', ''), + ('github.com/mitchellh/reflectwalk','v1.0.2', '', '', ''), + ('github.com/nats-io/nats.go','v1.25.0', '', '', ''), + ('github.com/nats-io/nkeys','v0.4.4', '', '', ''), + ('github.com/nats-io/nuid','v1.0.1', '', '', ''), + ('github.com/pmezard/go-difflib','v1.0.0', '', '', ''), + ('github.com/stretchr/objx','v0.5.0', '', '', ''), + ('github.com/x448/float16','v0.8.4', '', '', ''), + ('github.com/zeebo/errs','v1.3.0', '', '', ''), + ('go.googlesource.com/mod','v0.8.0','golang.org/x/mod', '', ''), + ('go.googlesource.com/net','v0.9.0','golang.org/x/net', '', ''), + ('go.googlesource.com/sync','v0.1.0','golang.org/x/sync', '', ''), + ('go.googlesource.com/sys','v0.7.0','golang.org/x/sys', '', ''), + ('go.googlesource.com/text','v0.9.0','golang.org/x/text', '', ''), + ('go.googlesource.com/tools','v0.6.0','golang.org/x/tools', '', ''), + ('github.com/googleapis/go-genproto','v0.0.0-20230223222841-637eb2293923','google.golang.org/genproto', '', ''), + ('github.com/grpc/grpc-go','v1.53.0','google.golang.org/grpc', '', ''), + ('go.googlesource.com/protobuf','v1.28.1','google.golang.org/protobuf', '', ''), + ] + + src_uri = set() + for d in dependencies: + src_uri.add(urifiy(*d)) + + checkvars = {} + checkvars['GO_DEPENDENCIES_SRC_URI'] = src_uri + + self.assertTrue(os.path.isfile(deps_require_file)) + self._test_recipe_contents(deps_require_file, checkvars, []) + + + def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths): dstdir = basedstdir self.assertTrue(os.path.exists(dstdir)) From patchwork Thu Nov 2 15:53:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 33468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30720C4332F for ; Thu, 2 Nov 2023 15:53:29 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.59]) by mx.groups.io with SMTP id smtpd.web10.34981.1698940405841993678 for ; Thu, 02 Nov 2023 08:53:27 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=xn4bx9py; spf=pass (domain: weidmueller.com, ip: 40.107.22.59, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WfNhhQcAdlrXvxzY2oZH2bpnRPmrj2i0qGAfPP8yUw4m/UUXyBT6lAwu+T6JMaTZRxVONGMJqhGB8uTzZd7czXKy5fsyesLDVyLGmqZPauguMJhMpEftRG5ZQUvBvsQmUzl5mGElL7myCJVyEduwvMAxC5JzoGDREh7JkuKHEumB1Yx/VxdL+3ZYifdURfKaRMFfN32v1gemuRBBsvVT3kU7X+ktMmdkQEEnlkro+o5GYOqKI7an191V04JJycOLDcEv1yGoE05TMJommVU2ob2c8FaUyCq1U4GLKxqICE7EOTnaSQkG1qlpwjtFfM9emMlbHTH3brySvpLPkaGACA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=t7LTMVL3y5KvcqM1eit0/rcSMWgre56MO2DD0PPlSPY=; b=LUAVOLYDxHNN3b+A3xSyb7DkRDOpsx7P6vO8w3Qz0tszOrJVXXlE0JNF68X7F1yzdMnmANAQk9c966ycjIdzeCC8+iCBOBJFPcOeLoacPYKyS0PAIXDSAhCuSOEWK+Sapn7gGLmHo0i1kLMCmp0unaBnvhXNEiFhoR2nbU5YXo7gLhvxvSvqUkfJyZ4aAsIHz68HPgZxhBnBpg0jcYYSM+B/CcyBZGyZEb+DQOv4rlCmJG7nIrTZnp2dxuKXhhyPEVyYlotLBc4l0jnitP0+Iyaep52FXSXY/cU+kCgLsUXcEgZKc27NGzcolfHNcfS3/uBYJ2tqGLuzzKCyX2i+sQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t7LTMVL3y5KvcqM1eit0/rcSMWgre56MO2DD0PPlSPY=; b=xn4bx9py/DZEKxm1H/LKceQ5KcqvOIHOJP5f3dPSUorq6EOajKzKlhyE1CmgngdP+FRSCuVXxnrmEvCrq9iirbnPq5OmAYVw6OlpPTSVQQLyu1Ek8cvY/VrKv4G2MWhcY17+OKVSr5DC4aeEibCSyTFlqIIE5DVvByvHBfWzkVI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PAWPR08MB10059.eurprd08.prod.outlook.com (2603:10a6:102:35d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.21; Thu, 2 Nov 2023 15:53:22 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c%7]) with mapi id 15.20.6954.019; Thu, 2 Nov 2023 15:53:22 +0000 From: lukas.funke-oss@weidmueller.com To: openembedded-core@lists.openembedded.org CC: Bruce Ashfield , Vyacheslav Yurkov , Martin Jansa , Peter Kjellerstedt , u.oelmann@pengutronix.de, Lukas Funke Subject: [OE-Core][PATCH v3 3/4] recipetool: Ignore *.go files while scanning for licenses Date: Thu, 2 Nov 2023 16:53:12 +0100 Message-ID: <20231102155313.309534-4-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> References: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0189.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ab::20) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PAWPR08MB10059:EE_ X-MS-Office365-Filtering-Correlation-Id: ad45e700-f353-4915-338a-08dbdbbbd793 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8RlO/rOiH3yNHWKojqra6xHrf9z5YStUuZubgtRZ6HMS38CWa6U/2XSjq7hRhktykjQs9pDnDWgJiyY2C/FYbiWyaH0oxEzpU514Vo+pQBtV+TsITf1a6zScJqUSviLgC7dNDLEza8kjfNzTjfRZ+fpjYHCcx7yaytEyKZ4D1R6FkS5JS+SrlVybgQHh8yu0AxO2McAFCouebkof352vJvEdXNLY87vVdd/spydR32ODKzPeDAB3lrzBe338RlPf9yBIVEqPD1aUoU+qC35yhrgUVcU7TtfIqpo6U4vQhHH20Zn3uieMScw++aZVtCqdKjS9IfM0i2I5a3wxYzYvEareOh1S33L9XlzxgN0Jh9rtFbpYTSsdbk6M6B4IfvlF+K0ML2GzVlAnI3B83eZDPEF9IO0sNjgum5c082pVbyE6mJDIwgWRiuhc1zgUsesTcaOLRvH1UKdURGXFSWkquq7mqfS3XGBHept/O0KLGRFvany4f//Qfxo0QgtUBmkPYCbOrtOhTTRVjOfTYuKmGS5fHNdopSPMBO24Q+onZb/T/yAEfiLQsVDF00Zo0vnHAWSlJ1xMThYIXIMLVbNBqqkcJwEX6RuGR3uojYCMc1vsLcwU4tglxrovgXS+Dt8S X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(346002)(39860400002)(366004)(396003)(376002)(230922051799003)(1800799009)(186009)(451199024)(64100799003)(478600001)(6506007)(52116002)(83380400001)(38100700002)(1076003)(107886003)(2616005)(6512007)(9686003)(41300700001)(2906002)(86362001)(4744005)(36756003)(5660300002)(4326008)(8936002)(8676002)(6666004)(38350700005)(66556008)(66476007)(66946007)(6916009)(54906003)(316002)(26005)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: MKat/Y8NqYBdqtFUWNkD0d47iFprpgOcLMmZufVl7+B03DYSPrmbFha30hzXOy0ZElbMZ+hO/dlnhOHeKMFKxceDHZ3vrUahxLsdwsoavgDvLhO7slPCCVpAkN+0II93xqAluscmwDe6wBzV9Q0F28FQ9GpnQZIi3gsfFqRX/kyyUpUw3MW7y6lTm5cDsqRQrNrWXEMleVc4Z8038+ehC7981OeKkDz0mzUl28kDbKzJvdTTR39dHf+H8AiKytjxc+BKKAGqMiYcLc7Ak6/M7IWafieELv3XcV5QtFM+hTr11qOSGQhS1Ybu6g3BCh+G9LAeqZuffMXZtCqcwxLgJ9k9s/7T3ayXxETEJSYF8u71gjx5+YTxTnMGeMAugvQMvUpfyEvM9vFKGhRWSy3tj03tLncG1iBqXQbkVe4gifrKiB4UuwqXawswUKteDi4A/K+nnv6hPZUVbBk4RnwWAjxTdncedkx4KlKFaA1rG8rzGjwR2LL+NFezYdqq3FWSkKbZQVsJoop3ztzsHDyaG12nxNfxssjR0c5sJXZaIxftjKPdZPPlf+UegQxAtf3+ikWc3FekTX54Xwl3JKy94ub66psx79buJs+ZrbGkyGASH92DcdAkY5Gw/EmYcQrb1pYrn/7Uo3QxSfv4jrdA/tKJDZB4d5+0hL6vAtVnubeTiaMUxAbwBUdw8C1otRBA+/5zihBzQquRXr+/jxOPgbhASN9uakaYLzEt0BHzZtnQ7LUvs/7VMpa9l0h7lEp2KPOeh6Mte0Nn6LvjDq6jcQYKUyOH/GqGIvJTRJ8ZkotzVVXKiPhLqTV4qlb6OmmmbqMQ8NJc9Nk2u67jn6FT2T1uI05dW+DfOEqf0uR01ztER7vVMnXZeMKkM80xRav9p3Nf5KejGg2g3Wiewt8N1qdpwpWEPD/hIcekIcbwm+bi8dTc+bKj+jL4PT8gjKHeh9bEJF6rQEQt7cXZCZuorfoAJTSn9MK5+0zOmDvNYEinMMRobf+t2cR89sz6Hg6s6ZKk0JkOSCKJ3Bd4E6EIrH8XS76d1iowB6ji7LOIxwbrXXfU+Ov9i+FPBtlwwc9zODtAj7XoOl7kWDMNlftLQdW7q7+YiHEXcJqqFyQ1uo6qidby+HdipcmpZfibQmeqjEjCtFdTvEYOaPGaxxRjN0SUxu3Q9djFPFh1IPtimu26fsORc8g5agmAQxBUNtiwniDaboxlRFjK/n+DQPEUnfCcuNNEacRuR/3Kpfn940T0dH/iyWJH/GSVJzs4oXabvCdj7oJrpPNevPuyF/Z+Zo6pk6t/BewVLG60+a3KhHvjkS51RwKuYjwU7BvBQeHXxEYLD4fscrlhQCIMwvUlWJFTHOTBfcTAMkLYe0s1Hk4lavLt/tByNPhyP07QZJ7a7ttFIXmj72wdXQdKNpKhk7MFUs1FuL8jIjCkRTqxVT+T9Nk3wlwwUNQA/lJTiQMKtPp+A7pu8CkcO+IvQU3JsANA5ccEcZOVpjCe5VAbPLg1amDlJtJHscukPdgYFLNNEugElXuyGeroxwd/qACDSVU/Jj0HeMWbKQcXg85c7AEB3Dmcfi3RXMEksfpRC/IhVra1Iu+Yv1IN9xhZDsCoRg== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad45e700-f353-4915-338a-08dbdbbbd793 X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 15:53:22.6293 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4gtdYf/CqhaWLgYgkKtk5N/eNjg83hrjdJ10gVfC3qhC0/bj37sMfD9WCW2za+q9LHKGdg4SkD4e02s7dD/BLA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10059 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 02 Nov 2023 15:53:29 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190083 From: Lukas Funke Signed-off-by: Lukas Funke --- scripts/lib/recipetool/create.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 143bc63e9d..293198d1c8 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -1212,7 +1212,7 @@ def guess_license(srctree, d): licenses = [] licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*', 'e[dp]l-v10'] - skip_extensions = (".html", ".js", ".json", ".svg", ".ts") + skip_extensions = (".html", ".js", ".json", ".svg", ".ts", ".go") licfiles = [] for root, dirs, files in os.walk(srctree): for fn in files: From patchwork Thu Nov 2 15:53:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 33471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A58DC4167D for ; Thu, 2 Nov 2023 15:53:39 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.59]) by mx.groups.io with SMTP id smtpd.web10.34981.1698940405841993678 for ; Thu, 02 Nov 2023 08:53:30 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=qeCUnJiJ; spf=pass (domain: weidmueller.com, ip: 40.107.22.59, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e/KuFl1WIPDfm8DbB6qljfiOPfmjbeTwcfGz5BjzSAwBK+/wOVBTYDNXKiHicqeULbQsC3gKn5N0Keggr6MsbXaKI0oikgDS90dsFTfuhE1F6d/U/sa0tioiIlb28yF30ZqM3pNfOvvwEzC9p5XhdBFfkev7ascc4nfVLlVf7mWwygGJrbeQiQ112Cti5xJQXCjep45GZbpg5ICQrP4kxOBUQYztkiPVD+8rz0J0be/WAL1ZIKEE1Nec/4zK/SdTjI5VGpSNj8ctG+0u9mDknDNLPWhIdvRbCPJuA6CIi6vT8IIbrDKj9VvKGqgykdqXNSXSbKzAgPhw/AQHyHnpag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LbkdDdSzs2Ylif4g/p1W8lxupFboseWINiC3VMjDjAs=; b=WBiJ5KUQLPspDgFjYnrA5Y+99TE3ZMUR3LpOzTH+9QNzrPl2L9i/AKtv40z2CMYY1BjsrgF9gh4rY5GPUPfLNemDRqRt2AGJULqVky4cVnwmdnhORkdCOXBc7UJvIuF5l2dTS1xBU+sq2/Tbu0bX0Y0ov9zU57msKPXzP2jRAgZDbumroAu1+oBZezziYrqBSuvO8DwLwXVR+sh4GThnr8wCpNz2j0wjzsg9+/70HlfzYCgbBNq/7Rw9696buNVbTBKis6WZYzG1IdUh7sUtNQocztAjIEpktYmLt0zg3Pl0gzHHeS0Qjam4Jz/lT5FJxHJhjG9ZirYeKEl3cieJJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LbkdDdSzs2Ylif4g/p1W8lxupFboseWINiC3VMjDjAs=; b=qeCUnJiJ+lzHaYAJ/c46kv2WDbShnODCwbTgOCpa2d6mioIRgm8l+yA+jkf4CDw9o7t+NoSAuhpS4yIMv9V5A0wyXL/VzSbTFh/5SoZwhKcY576WY1tnhsr1YPFpZvCGr5iEIqW319SVgqK92pfloXKRw5r9pAHooRegqWw4Jzs= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PAWPR08MB10059.eurprd08.prod.outlook.com (2603:10a6:102:35d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.21; Thu, 2 Nov 2023 15:53:23 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::c337:aad3:ff97:b20c%7]) with mapi id 15.20.6954.019; Thu, 2 Nov 2023 15:53:23 +0000 From: lukas.funke-oss@weidmueller.com To: openembedded-core@lists.openembedded.org CC: Bruce Ashfield , Vyacheslav Yurkov , Martin Jansa , Peter Kjellerstedt , u.oelmann@pengutronix.de, Lukas Funke Subject: [OE-Core][PATCH v3 4/4] recipetool: Add handler to create go recipes Date: Thu, 2 Nov 2023 16:53:13 +0100 Message-ID: <20231102155313.309534-5-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> References: <20231102155313.309534-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0189.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ab::20) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PAWPR08MB10059:EE_ X-MS-Office365-Filtering-Correlation-Id: 49572dc2-d6a2-4a04-57f2-08dbdbbbd7d8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hCjyAQKOCWYZIA3MIIeOTGvZHi1xC9I4w8gWfXrL5COe9wK0iRvHPFa5BUMSI7C6hKDHZ9vLgFznxgIwD17EdS8QqLhKCsR+sbsjiJbIg36sbJiMuF+BeaqlEi7dNmH1gbe3VUSlzXWVFYv25+l8rrUSGa3mEfWcqrsvYktRzqnucS/9mRhehVj74gFRmEByFypDxXH1ezF+MNI4yuMoLJxNlVbIVs2PS9ToHBdCXW3v/xBXgJjCEOjwuPPtQNXvnYbh2w0fkgoXeE+cWYdOna7fHX8/9GhaZZuv+Vmfqrf41BzNjms+wVUjS358kgxzI90o1Waafne9UFhf6hV0N6yMmYwckx8a9foJzLz742golO4sw7NxQFaRWnl43EStTOYTVleVT5hXm+Ytc7wREB6dhtFU9xnBuA0NfCkwsuS+YhTT8+lvrkIawn77jZiJcZvWJqKX23KxKPEsWNnDwS4NwEKwSVQot4kVAX8oHrRXzmuxfePDSAvlQCtaKzgXPuVDIdMUoe4dH5/mjkqriz2unmdcb+alGyftAUuwFrO/TeFDl/GvotlGRuOlNWeoM7PWlvHLO9jk+5N1lE41ggudlwLuOptAPaJXuW/0+24uvzyzMPrK3p29eELIHosuef7D6E9R0MI2f0pgo2ypUw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(346002)(39860400002)(366004)(396003)(376002)(230173577357003)(230273577357003)(230922051799003)(1800799009)(186009)(451199024)(64100799003)(478600001)(6506007)(52116002)(83380400001)(38100700002)(1076003)(107886003)(2616005)(6512007)(9686003)(41300700001)(30864003)(2906002)(86362001)(36756003)(5660300002)(4326008)(8936002)(8676002)(6666004)(38350700005)(66556008)(66476007)(66946007)(6916009)(54906003)(316002)(26005)(966005)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xUxUwnpsCS7tFFkPwWmpWS5VUobpEhbiOinnqyV7R1UbxwoYROBlH2nqcjxOsUdz43DEFmaOu5QXopu46WmQjXd+o+JV++CRbzzPBSCSSLMTBoSeSFnfYe8buwBy9VCCsd1L87L6xDEdN5yiAKMtsV8pa0gJcCV0Wi20A19eP89RfT3A8y0Tex6tL2sBmaZEMs1G0xO7JnjaOPZ5GPRyzIewni+M/zCkfIag6cEl0yJlHCMmLVALlcU71vyrY1zn+XuqQ3beoewZbN4UpgTOqyRzoed9NDzMYqCE5Fp64vLtwRhe+2uc45S8KWA1m5jzn54LmSbnzCKcgU5gEwarrZtVBh1gFW2coVtJiqinM3Faya6J7NWe/JPG4yY6yZnvEp41ADaJ9wp/KEIIHbSq/EPQhgQZnKdM3IJikxEVElCNCI/as2M2XyG9gqf5QkNfxUSQ+K5FRgsWRfdF+7QSDQ3zA+r6ZHA5blZXLfnwNi45ydXVgM7whDcG0uWht3jGV1WxxZU4wvund1fEYIB4hUCCw8pLhtT8fZxnoHfJ0b/gGh0si/V1xNhbW/qWzYHwf7EpnO+9pSzOiVTEhthzTwqOl57zh1fqG/PSWq+6XKPozUb2FVG2rT6S6fA6a5KQZUoS89N4V76XbEXrrXjbkO/3XO3gi/5hlQ/QPYgSE0RgeOi6Sf5ycvYZuKycTST+3/1+e/I4AvAvXU52OeEYWDxUfuk++0q9q6o0zsmoOHAqgV6SPr7NwUJxI4t/zIai5qw2Qz3csnTOHJNKRoIAPHzgHqEWRaq7TcWglOUL/AA11+AzGRIR2E0I6ft8UZ8tSm+DX/0ZGVrQySWb9HH9P6LYpUsP6xOLn1piGBCodQ0YuT5hYl4OWK8pzN6wgYoyq9k65xhVadUhBAfloW40/X/n9cEu+3plrjTbQ4arcu99+AxlMvCRuYCHvjL8Gb/CQoW8jLGzd7UV1JtJlAiU5wkz/6ZJaNZQlq9T4ORudSZSgL8QeOKpc8Zk6JEYJYA/p9icYqTs+CUs1wTRm5BfuFSKLDqtiSc8O+tXP3bjl1xzAwyo1Hd6WUP2sQ+M6Ry/NegsYCnKErWTtmWYPaDo0RtaimVXsSNY5aiQArSJ621LunMiKqavj8A4ogfaFIPzk4ixfLDWKtK7O2ygttL/0Q4eOMyWpJQCRLTHpZ/m96BSq55IhK8o9c+XY75triQzYM6igAUtYZXwJwZooqytAcTXe9xnKQUIxPOzm6RhWfEWLOhN69ph85Uu/48ztIyGbf8wjozzMJtgLC+zdXa3FQDzgiISVmFNA7RGaNL+oQenFR4EQ9NWih6EgiyCOVyVsAN3SosGERx/fYVsw9v2dL9hsxF04F766QlE0oNxR5kFVMLsOGMQDk228mtevkJszP1uZPazdSXuekSGdWu3gEappN90Xl2oBvyCVUf5q8LOI5e2gkmau/unhQbJzT8Gznb97n6/tl7CkCbMqActrge+nPUC7gA0BxqF015Y3dP3El2tsWkQL9+rk90ysdQmMjCGoqktVAm1ZoHXAizfbcraV1ZWd00Fe8KwrtF9YZoIqfN9PIw8v0S6OPR7TR6GIPOaHrtP1rAYlKtR1/Ggdw== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 49572dc2-d6a2-4a04-57f2-08dbdbbbd7d8 X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 15:53:23.1028 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: u+UdD20V7UVU0Zais0jIC+OS7xDzRwdE+BU7/OdqYbTZWI65l8/3uNxUOAXst04BgjxMN61ypYgFbLpTEFAnRQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10059 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 02 Nov 2023 15:53:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190086 From: Lukas Funke Signed-off-by: Lukas Funke --- scripts/lib/recipetool/create_go.py | 751 ++++++++++++++++++++++++++++ 1 file changed, 751 insertions(+) create mode 100644 scripts/lib/recipetool/create_go.py diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py new file mode 100644 index 0000000000..c7b2381a33 --- /dev/null +++ b/scripts/lib/recipetool/create_go.py @@ -0,0 +1,751 @@ +# Recipe creation tool - go support plugin +# +# The code is based on golang internals. See the afftected +# methods for further reference and information. +# +# Copyright (C) 2023 Weidmueller GmbH & Co KG +# Author: Lukas Funke +# +# SPDX-License-Identifier: GPL-2.0-only +# + + +from collections import namedtuple +from enum import Enum +from html.parser import HTMLParser +from recipetool.create import RecipeHandler, handle_license_vars +from recipetool.create import guess_license, tidy_licenses, fixup_license +from recipetool.create import determine_from_url +from urllib.error import URLError + +import bb.utils +import json +import logging +import os +import re +import subprocess +import sys +import shutil +import tempfile +import urllib.parse +import urllib.request + + +GoImport = namedtuple('GoImport', 'root vcs url suffix') +logger = logging.getLogger('recipetool') +CodeRepo = namedtuple( + 'CodeRepo', 'path codeRoot codeDir pathMajor pathPrefix pseudoMajor') + +tinfoil = None + +# Regular expression to parse pseudo semantic version +# see https://go.dev/ref/mod#pseudo-versions +re_pseudo_semver = re.compile( + r"^v[0-9]+\.(0\.0-|\d+\.\d+-([^+]*\.)?0\.)(?P\d{14})-(?P[A-Za-z0-9]+)(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$") +# Regular expression to parse semantic version +re_semver = re.compile( + r"^v(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$") + + +def tinfoil_init(instance): + global tinfoil + tinfoil = instance + + +class GoRecipeHandler(RecipeHandler): + """Class to handle the go recipe creation""" + + @staticmethod + def __ensure_go(): + """Check if the 'go' command is available in the recipes""" + recipe = "go-native" + if not tinfoil.recipes_parsed: + tinfoil.parse_recipes() + try: + rd = tinfoil.parse_recipe(recipe) + except bb.providers.NoProvider: + bb.error( + "Nothing provides '%s' which is required for the build" % (recipe)) + bb.note( + "You will likely need to add a layer that provides '%s'" % (recipe)) + return None + + bindir = rd.getVar('STAGING_BINDIR_NATIVE') + gopath = os.path.join(bindir, 'go') + + if not os.path.exists(gopath): + tinfoil.build_targets(recipe, 'addto_recipe_sysroot') + + if not os.path.exists(gopath): + logger.error( + '%s required to process specified source, but %s did not seem to populate it' % 'go', recipe) + return None + + return bindir + + def __resolve_repository_static(self, modulepath): + """Resolve the repository in a static manner + + The method is based on the go implementation of + `repoRootFromVCSPaths` in + https://github.com/golang/go/blob/master/src/cmd/go/internal/vcs/vcs.go + """ + + url = urllib.parse.urlparse("https://" + modulepath) + req = urllib.request.Request(url.geturl()) + + try: + resp = urllib.request.urlopen(req) + # Some modulepath are just redirects to github (or some other vcs + # hoster). Therefore, we check if this modulepath redirects to + # somewhere else + if resp.geturl() != url.geturl(): + bb.debug(1, "%s is redirectred to %s" % + (url.geturl(), resp.geturl())) + url = urllib.parse.urlparse(resp.geturl()) + modulepath = url.netloc + url.path + + except URLError as url_err: + # This is probably because the module path + # contains the subdir and major path. Thus, + # we ignore this error for now + logger.debug( + 1, "Failed to fetch page from [%s]: %s" % (url, str(url_err))) + + host, _, _ = modulepath.partition('/') + + class vcs(Enum): + pathprefix = "pathprefix" + regexp = "regexp" + type = "type" + repo = "repo" + check = "check" + schemelessRepo = "schemelessRepo" + + # GitHub + vcsGitHub = {} + vcsGitHub[vcs.pathprefix] = "github.com" + vcsGitHub[vcs.regexp] = re.compile( + r'^(?Pgithub\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$') + vcsGitHub[vcs.type] = "git" + vcsGitHub[vcs.repo] = "https://\\g" + + # Bitbucket + vcsBitbucket = {} + vcsBitbucket[vcs.pathprefix] = "bitbucket.org" + vcsBitbucket[vcs.regexp] = re.compile( + r'^(?Pbitbucket\.org/(?P[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+))(/(?P[A-Za-z0-9_.\-]+))*$') + vcsBitbucket[vcs.type] = "git" + vcsBitbucket[vcs.repo] = "https://\\g" + + # IBM DevOps Services (JazzHub) + vcsIBMDevOps = {} + vcsIBMDevOps[vcs.pathprefix] = "hub.jazz.net/git" + vcsIBMDevOps[vcs.regexp] = re.compile( + r'^(?Phub\.jazz\.net/git/[a-z0-9]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$') + vcsIBMDevOps[vcs.type] = "git" + vcsIBMDevOps[vcs.repo] = "https://\\g" + + # Git at Apache + vcsApacheGit = {} + vcsApacheGit[vcs.pathprefix] = "git.apache.org" + vcsApacheGit[vcs.regexp] = re.compile( + r'^(?Pgit\.apache\.org/[a-z0-9_.\-]+\.git)(/(?P[A-Za-z0-9_.\-]+))*$') + vcsApacheGit[vcs.type] = "git" + vcsApacheGit[vcs.repo] = "https://\\g" + + # Git at OpenStack + vcsOpenStackGit = {} + vcsOpenStackGit[vcs.pathprefix] = "git.openstack.org" + vcsOpenStackGit[vcs.regexp] = re.compile( + r'^(?Pgit\.openstack\.org/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(\.git)?(/(?P[A-Za-z0-9_.\-]+))*$') + vcsOpenStackGit[vcs.type] = "git" + vcsOpenStackGit[vcs.repo] = "https://\\g" + + # chiselapp.com for fossil + vcsChiselapp = {} + vcsChiselapp[vcs.pathprefix] = "chiselapp.com" + vcsChiselapp[vcs.regexp] = re.compile( + r'^(?Pchiselapp\.com/user/[A-Za-z0-9]+/repository/[A-Za-z0-9_.\-]+)$') + vcsChiselapp[vcs.type] = "fossil" + vcsChiselapp[vcs.repo] = "https://\\g" + + # General syntax for any server. + # Must be last. + vcsGeneralServer = {} + vcsGeneralServer[vcs.regexp] = re.compile( + "(?P(?P([a-z0-9.\\-]+\\.)+[a-z0-9.\\-]+(:[0-9]+)?(/~?[A-Za-z0-9_.\\-]+)+?)\\.(?Pbzr|fossil|git|hg|svn))(/~?(?P[A-Za-z0-9_.\\-]+))*$") + vcsGeneralServer[vcs.schemelessRepo] = True + + vcsPaths = [vcsGitHub, vcsBitbucket, vcsIBMDevOps, + vcsApacheGit, vcsOpenStackGit, vcsChiselapp, + vcsGeneralServer] + + if modulepath.startswith("example.net") or modulepath == "rsc.io": + logger.warning("Suspicious module path %s" % modulepath) + return None + if modulepath.startswith("http:") or modulepath.startswith("https:"): + logger.warning("Import path should not start with %s %s" % + ("http", "https")) + return None + + rootpath = None + vcstype = None + repourl = None + suffix = None + + for srv in vcsPaths: + m = srv[vcs.regexp].match(modulepath) + if vcs.pathprefix in srv: + if host == srv[vcs.pathprefix]: + rootpath = m.group('root') + vcstype = srv[vcs.type] + repourl = m.expand(srv[vcs.repo]) + suffix = m.group('suffix') + break + elif m and srv[vcs.schemelessRepo]: + rootpath = m.group('root') + vcstype = m[vcs.type] + repourl = m[vcs.repo] + suffix = m.group('suffix') + break + + return GoImport(rootpath, vcstype, repourl, suffix) + + def __resolve_repository_dynamic(self, modulepath): + """Resolve the repository root in a dynamic manner. + + The method is based on the go implementation of + `repoRootForImportDynamic` in + https://github.com/golang/go/blob/master/src/cmd/go/internal/vcs/vcs.go + """ + url = urllib.parse.urlparse("https://" + modulepath) + + class GoImportHTMLParser(HTMLParser): + + def __init__(self): + super().__init__() + self.__srv = [] + + def handle_starttag(self, tag, attrs): + if tag == 'meta' and list( + filter(lambda a: (a[0] == 'name' and a[1] == 'go-import'), attrs)): + content = list( + filter(lambda a: (a[0] == 'content'), attrs)) + if content: + self.__srv = content[0][1].split() + + @property + def import_prefix(self): + return self.__srv[0] if len(self.__srv) else None + + @property + def vcs(self): + return self.__srv[1] if len(self.__srv) else None + + @property + def repourl(self): + return self.__srv[2] if len(self.__srv) else None + + url = url.geturl() + "?go-get=1" + req = urllib.request.Request(url) + + try: + resp = urllib.request.urlopen(req) + + except URLError as url_err: + logger.warning( + "Failed to fetch page from [%s]: %s", url, str(url_err)) + return None + + parser = GoImportHTMLParser() + parser.feed(resp.read().decode('utf-8')) + parser.close() + + return GoImport(parser.import_prefix, parser.vcs, parser.repourl, None) + + def __resolve_from_golang_proxy(self, modulepath, version): + """ + Resolves repository data from golang proxy + """ + url = urllib.parse.urlparse("https://proxy.golang.org/" + + modulepath + + "/@v/" + + version + + ".info") + + # Transform url to lower case, golang proxy doesn't like mixed case + req = urllib.request.Request(url.geturl().lower()) + + try: + resp = urllib.request.urlopen(req) + except URLError as url_err: + logger.warning( + "Failed to fetch page from [%s]: %s", url, str(url_err)) + return None + + golang_proxy_res = resp.read().decode('utf-8') + modinfo = json.loads(golang_proxy_res) + + if modinfo and 'Origin' in modinfo: + origin = modinfo['Origin'] + _root_url = urllib.parse.urlparse(origin['URL']) + + # We normalize the repo URL since we don't want the scheme in it + _subdir = origin['Subdir'] if 'Subdir' in origin else None + _root, _, _ = self.__split_path_version(modulepath) + if _subdir: + _root = _root[:-len(_subdir)].strip('/') + + _commit = origin['Hash'] + _vcs = origin['VCS'] + return (GoImport(_root, _vcs, _root_url.geturl(), None), _commit) + + return None + + def __resolve_repository(self, modulepath): + """ + Resolves src uri from go module-path + """ + repodata = self.__resolve_repository_static(modulepath) + if not repodata or not repodata.url: + repodata = self.__resolve_repository_dynamic(modulepath) + if not repodata or not repodata.url: + logger.error( + "Could not resolve repository for module path '%s'" % modulepath) + # There is no way to recover from this + sys.exit(14) + if repodata: + logger.debug(1, "Resolved download path for import '%s' => %s" % ( + modulepath, repodata.url)) + return repodata + + def __split_path_version(self, path): + i = len(path) + dot = False + for j in range(i, 0, -1): + if path[j - 1] < '0' or path[j - 1] > '9': + break + if path[j - 1] == '.': + dot = True + break + i = j - 1 + + if i <= 1 or i == len( + path) or path[i - 1] != 'v' or path[i - 2] != '/': + return path, "", True + + prefix, pathMajor = path[:i - 2], path[i - 2:] + if dot or len( + pathMajor) <= 2 or pathMajor[2] == '0' or pathMajor == "/v1": + return path, "", False + + return prefix, pathMajor, True + + def __get_path_major(self, pathMajor): + if not pathMajor: + return "" + + if pathMajor[0] != '/' and pathMajor[0] != '.': + logger.error( + "pathMajor suffix %s passed to PathMajorPrefix lacks separator", pathMajor) + + if pathMajor.startswith(".v") and pathMajor.endswith("-unstable"): + pathMajor = pathMajor[:len("-unstable") - 2] + + return pathMajor[1:] + + def __build_coderepo(self, repo, path): + codedir = "" + pathprefix, pathMajor, _ = self.__split_path_version(path) + if repo.root == path: + pathprefix = path + elif path.startswith(repo.root): + codedir = pathprefix[len(repo.root):].strip('/') + + pseudoMajor = self.__get_path_major(pathMajor) + + logger.debug("root='%s', codedir='%s', prefix='%s', pathMajor='%s', pseudoMajor='%s'", + repo.root, codedir, pathprefix, pathMajor, pseudoMajor) + + return CodeRepo(path, repo.root, codedir, + pathMajor, pathprefix, pseudoMajor) + + def __resolve_version(self, repo, path, version): + hash = None + coderoot = self.__build_coderepo(repo, path) + + def vcs_fetch_all(): + tmpdir = tempfile.mkdtemp() + clone_cmd = "%s clone --bare %s %s" % ('git', repo.url, tmpdir) + bb.process.run(clone_cmd) + log_cmd = "git log --all --pretty='%H %d' --decorate=short" + output, _ = bb.process.run( + log_cmd, shell=True, stderr=subprocess.PIPE, cwd=tmpdir) + bb.utils.prunedir(tmpdir) + return output.strip().split('\n') + + def vcs_fetch_remote(tag): + # add * to grab ^{} + refs = {} + ls_remote_cmd = "git ls-remote -q --tags {} {}*".format( + repo.url, tag) + output, _ = bb.process.run(ls_remote_cmd) + output = output.strip().split('\n') + for line in output: + f = line.split(maxsplit=1) + if len(f) != 2: + continue + + for prefix in ["HEAD", "refs/heads/", "refs/tags/"]: + if f[1].startswith(prefix): + refs[f[1].removeprefix(prefix)] = f[0] + + for key, hash in refs.items(): + if key.endswith(r"^{}"): + refs[key.strip(r"^{}")] = hash + + return refs[tag] + + m_pseudo_semver = re_pseudo_semver.match(version) + + if m_pseudo_semver: + remote_refs = vcs_fetch_all() + short_commit = m_pseudo_semver.group('commithash') + for l in remote_refs: + r = l.split(maxsplit=1) + sha1 = r[0] if len(r) else None + if not sha1: + logger.error( + "Ups: could not resolve abbref commit for %s" % short_commit) + + elif sha1.startswith(short_commit): + hash = sha1 + break + else: + m_semver = re_semver.match(version) + if m_semver: + + def get_sha1_remote(re): + rsha1 = None + for line in remote_refs: + # Split lines of the following format: + # 22e90d9b964610628c10f673ca5f85b8c2a2ca9a (tag: sometag) + lineparts = line.split(maxsplit=1) + sha1 = lineparts[0] if len(lineparts) else None + refstring = lineparts[1] if len( + lineparts) == 2 else None + if refstring: + # Normalize tag string and split in case of multiple + # regs e.g. (tag: speech/v1.10.0, tag: orchestration/v1.5.0 ...) + refs = refstring.strip('(), ').split(',') + for ref in refs: + if re.match(ref.strip()): + rsha1 = sha1 + return rsha1 + + semver = "v" + m_semver.group('major') + "."\ + + m_semver.group('minor') + "."\ + + m_semver.group('patch') \ + + (("-" + m_semver.group('prerelease')) + if m_semver.group('prerelease') else "") + + tag = os.path.join( + coderoot.codeDir, semver) if coderoot.codeDir else semver + + # probe tag using 'ls-remote', which is faster than fetching + # complete history + hash = vcs_fetch_remote(tag) + if not hash: + # backup: fetch complete history + remote_refs = vcs_fetch_all() + hash = get_sha1_remote( + re.compile(fr"(tag:|HEAD ->) ({tag})")) + + logger.debug( + "Resolving commit for tag '%s' -> '%s'", tag, hash) + return hash + + def __generate_srcuri_inline_fcn(self, path, version, replaces=None): + """Generate SRC_URI functions for go imports""" + + logger.info("Resolving repository for module %s", path) + # First try to resolve repo and commit from golang proxy + # Most info is already there and we don't have to go through the + # repository or even perform the version resolve magic + golang_proxy_info = self.__resolve_from_golang_proxy(path, version) + if golang_proxy_info: + repo = golang_proxy_info[0] + commit = golang_proxy_info[1] + else: + # Fallback + # Resolve repository by 'hand' + repo = self.__resolve_repository(path) + commit = self.__resolve_version(repo, path, version) + + url = urllib.parse.urlparse(repo.url) + repo_url = url.netloc + url.path + + coderoot = self.__build_coderepo(repo, path) + + inline_fcn = "${@go_src_uri(" + inline_fcn += f"'{repo_url}','{version}'" + if repo_url != path: + inline_fcn += f",path='{path}'" + if coderoot.codeDir: + inline_fcn += f",subdir='{coderoot.codeDir}'" + if repo.vcs != 'git': + inline_fcn += f",vcs='{repo.vcs}'" + if replaces: + inline_fcn += f",replaces='{replaces}'" + if coderoot.pathMajor: + inline_fcn += f",pathmajor='{coderoot.pathMajor}'" + inline_fcn += ")}" + + return inline_fcn, commit + + def __go_handle_dependencies(self, go_mod, localfilesdir, extravalues, d): + + src_uris = [] + src_revs = [] + + def generate_src_rev(path, version, commithash): + src_rev = f"# {path}@{version} => {commithash}\n" + # Ups...maybe someone manipulated the source repository and the + # version or commit could not be resolved. This is a sign of + # a) the supply chain was manipulated (bad) + # b) the implementation for the version resolving didn't work + # anymore (less bad) + if not commithash: + src_rev += f"#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" + src_rev += f"#!!! Could not resolve version !!!\n" + src_rev += f"#!!! Possible supply chain attack !!!\n" + src_rev += f"#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" + src_rev += f"SRCREV_{path.replace('/', '.')} = \"{commithash}\"" + + return src_rev + + for require in go_mod['Require']: + path = require['Path'] + version = require['Version'] + + inline_fcn, commithash = self.__generate_srcuri_inline_fcn( + path, version) + src_uris.append(inline_fcn) + src_revs.append(generate_src_rev(path, version, commithash)) + + if go_mod['Replace']: + for replacement in go_mod['Replace']: + oldpath = replacement['Old']['Path'] + path = replacement['New']['Path'] + version = replacement['New']['Version'] + + inline_fcn, commithash = self.__generate_srcuri_inline_fcn( + path, version, oldpath) + src_uris.append(inline_fcn) + src_revs.append(generate_src_rev(path, version, commithash)) + + pn, _ = determine_from_url(go_mod['Module']['Path']) + go_mods_basename = "%s-modules.inc" % pn + + go_mods_filename = os.path.join(localfilesdir, go_mods_basename) + with open(go_mods_filename, "w") as f: + # We introduce this indirection to make the tests a little easier + f.write("SRC_URI += \"${GO_DEPENDENCIES_SRC_URI}\"\n") + f.write("GO_DEPENDENCIES_SRC_URI = \"\\\n") + for uri in src_uris: + f.write(" " + uri + " \\\n") + f.write("\"\n\n") + for rev in src_revs: + f.write(rev + "\n") + + extravalues['extrafiles'][go_mods_basename] = go_mods_filename + + def __go_run_cmd(self, cmd, cwd, d): + return bb.process.run(cmd, env=dict(os.environ, PATH=d.getVar('PATH')), + shell=True, cwd=cwd) + + def __go_native_version(self, d): + stdout, _ = self.__go_run_cmd("go version", None, d) + m = re.match(r".*\sgo((\d+).(\d+).(\d+))\s([\w\/]*)", stdout) + major = int(m.group(2)) + minor = int(m.group(3)) + patch = int(m.group(4)) + + return major, minor, patch + + def __go_mod_patch(self, srctree, localfilesdir, extravalues, d): + + patchfilename = "go.mod.patch" + go_native_version_major, go_native_version_minor, _ = self.__go_native_version( + d) + self.__go_run_cmd("go mod tidy -go=%d.%d" % + (go_native_version_major, go_native_version_minor), srctree, d) + stdout, _ = self.__go_run_cmd("go mod edit -json", srctree, d) + + # Create patch in order to upgrade go version + self.__go_run_cmd("git diff go.mod > %s" % (patchfilename), srctree, d) + # Restore original state + self.__go_run_cmd("git checkout HEAD go.mod go.sum", srctree, d) + + go_mod = json.loads(stdout) + tmpfile = os.path.join(localfilesdir, patchfilename) + shutil.move(os.path.join(srctree, patchfilename), tmpfile) + + extravalues['extrafiles'][patchfilename] = tmpfile + + return go_mod, patchfilename + + def __go_mod_vendor(self, go_mod, srctree, localfilesdir, extravalues, d): + # Perform vendoring to retrieve the correct modules.txt + tmp_vendor_dir = tempfile.mkdtemp() + + # -v causes to go to print modules.txt to stderr + _, stderr = self.__go_run_cmd( + "go mod vendor -v -o %s" % (tmp_vendor_dir), srctree, d) + + modules_txt_basename = "modules.txt" + modules_txt_filename = os.path.join(localfilesdir, modules_txt_basename) + with open(modules_txt_filename, "w") as f: + f.write(stderr) + + extravalues['extrafiles'][modules_txt_basename] = modules_txt_filename + + licenses = [] + lic_files_chksum = [] + licvalues = guess_license(tmp_vendor_dir, d) + shutil.rmtree(tmp_vendor_dir) + + if licvalues: + for licvalue in licvalues: + license = licvalue[0] + lics = tidy_licenses(fixup_license(license)) + lics = [lic for lic in lics if lic not in licenses] + if len(lics): + licenses.extend(lics) + lic_files_chksum.append( + 'file://src/${GO_IMPORT}/vendor/%s;md5=%s' % (licvalue[1], licvalue[2])) + + pn, _ = determine_from_url(go_mod['Module']['Path']) + licenses_basename = "%s-licenses.inc" % pn + + licenses_filename = os.path.join(localfilesdir, licenses_basename) + with open(licenses_filename, "w") as f: + f.write("GO_MOD_LICENSES = \"%s\"\n\n" % + ' & '.join(sorted(licenses, key=str.casefold))) + # We introduce this indirection to make the tests a little easier + f.write("LIC_FILES_CHKSUM += \"${VENDORED_LIC_FILES_CHKSUM}\"\n") + f.write("VENDORED_LIC_FILES_CHKSUM = \"\\\n") + for lic in lic_files_chksum: + f.write(" " + lic + " \\\n") + f.write("\"\n") + + extravalues['extrafiles'][licenses_basename] = licenses_filename + + def process(self, srctree, classes, lines_before, + lines_after, handled, extravalues): + + if 'buildsystem' in handled: + return False + + files = RecipeHandler.checkfiles(srctree, ['go.mod']) + if not files: + return False + + d = bb.data.createCopy(tinfoil.config_data) + go_bindir = self.__ensure_go() + if not go_bindir: + sys.exit(14) + + d.prependVar('PATH', '%s:' % go_bindir) + handled.append('buildsystem') + classes.append("go-vendor") + + stdout, _ = self.__go_run_cmd("go mod edit -json", srctree, d) + + go_mod = json.loads(stdout) + go_import = go_mod['Module']['Path'] + go_version_match = re.match("([0-9]+).([0-9]+)", go_mod['Go']) + go_version_major = int(go_version_match.group(1)) + go_version_minor = int(go_version_match.group(2)) + src_uris = [] + + localfilesdir = tempfile.mkdtemp(prefix='recipetool-go-') + extravalues.setdefault('extrafiles', {}) + # go.mod files with version < 1.17 may not include all indirect + # dependencies. Thus, we have to upgrade the go version. + if go_version_major == 1 and go_version_minor < 17: + logger.warning( + "go.mod files generated by Go < 1.17 might have incomplete indirect dependencies.") + go_mod, patchfilename = self.__go_mod_patch(srctree, localfilesdir, + extravalues, d) + src_uris.append( + "file://%s;patchdir=src/${GO_IMPORT}" % (patchfilename)) + + # Check whether the module is vendored. If so, we have nothing to do. + # Otherwise we gather all dependencies and add them to the recipe + if not os.path.exists(os.path.join(srctree, "vendor")): + + # Write additional $BPN-modules.inc file + self.__go_mod_vendor(go_mod, srctree, localfilesdir, extravalues, d) + lines_before.append("LICENSE += \" & ${GO_MOD_LICENSES}\"") + lines_before.append("require ${BPN}-licenses.inc") + + self.__rewrite_src_uri(lines_before, ["file://modules.txt"]) + + self.__go_handle_dependencies(go_mod, localfilesdir, extravalues, d) + lines_before.append("require ${BPN}-modules.inc") + + # Do generic license handling + handle_license_vars(srctree, lines_before, handled, extravalues, d) + self.__rewrite_lic_uri(lines_before) + + lines_before.append("GO_IMPORT = \"{}\"".format(go_import)) + lines_before.append("SRCREV_FORMAT = \"${BPN}\"") + + def __update_lines_before(self, updated, newlines, lines_before): + if updated: + del lines_before[:] + for line in newlines: + # Hack to avoid newlines that edit_metadata inserts + if line.endswith('\n'): + line = line[:-1] + lines_before.append(line) + return updated + + def __rewrite_lic_uri(self, lines_before): + + def varfunc(varname, origvalue, op, newlines): + if varname == 'LIC_FILES_CHKSUM': + new_licenses = [] + licenses = origvalue.split('\\') + for license in licenses: + license = license.strip() + uri, chksum = license.split(';', 1) + url = urllib.parse.urlparse(uri) + new_uri = os.path.join( + url.scheme + "://", "src", "${GO_IMPORT}", url.netloc + url.path) + ";" + chksum + new_licenses.append(new_uri) + + return new_licenses, None, -1, True + return origvalue, None, 0, True + + updated, newlines = bb.utils.edit_metadata( + lines_before, ['LIC_FILES_CHKSUM'], varfunc) + return self.__update_lines_before(updated, newlines, lines_before) + + def __rewrite_src_uri(self, lines_before, additional_uris = []): + + def varfunc(varname, origvalue, op, newlines): + if varname == 'SRC_URI': + src_uri = ["git://${GO_IMPORT};destsuffix=git/src/${GO_IMPORT};nobranch=1;name=${BPN};protocol=https"] + src_uri.extend(additional_uris) + return src_uri, None, -1, True + return origvalue, None, 0, True + + updated, newlines = bb.utils.edit_metadata(lines_before, ['SRC_URI'], varfunc) + return self.__update_lines_before(updated, newlines, lines_before) + + +def register_recipe_handlers(handlers): + handlers.append((GoRecipeHandler(), 60))