From patchwork Mon Feb 26 14:40:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 40090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88271C54E55 for ; Mon, 26 Feb 2024 14:41:00 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.138]) by mx.groups.io with SMTP id smtpd.web11.21659.1708958455668094328 for ; Mon, 26 Feb 2024 06:40:57 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=Kd5GUI0W; spf=pass (domain: weidmueller.com, ip: 40.107.22.138, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GRpzAqhCI4C/DAy4cHM4cjc+5CJ9p8R9xN8/ojv/gghptI0N0AUvFnBTd8bv5YyFHHFr4ndh3ziw7x3zowkbhC/UGacBg+8Gx9mS1FEP+A21hbepbXLATBFVpD1V1hoJ7Whq8CFhW2p0k4QLQJLgJQjVJVRjMW/EtaSfKU3R7gOKhAl2KsEt5aiamqJ4Xwy17zXJzE1AbyQgtHgmo7o9g+1wmGbP1NJJOx4F82oFc7r7pfU11qQtsZLpX/ZgjxnCDNYdQyq4zlTN3FXOK+R66vnkUC9dlYJY+HMeownC3H7l6J+Hxoi2A0V0Qg7BrkUWZPjHO1nqkpZkSwAuzKrm9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dMHkpJ66BNDh4vc2GgcBGq+n00YreQ/6i/pSAkSbl0w=; b=Z8jiiXDOJgx5gLINlFPW/I+1cVglj38dcvzB+Q0D4e0WcJcxUR4EY9vRWbOYZUipHraMGEORvyns1eFZEcbY/e5f64LyY9p+IZlBDnLddGVF5Q44bYBNzceNb6oVRkopjh8snOuWBFjstq1sn0mNaoOmcR757MFU5ytcPtUqopa9IaHkfwFIzEtNNlRBU2QCOifphDR5JYTkEFQATRrZZUj6Ya00srUhFy8wm3iFJnJDviXBp2KQRwvVxbJq1KCBYGJycnzdcpOiN7eDDoEvD9WvbWq9Gv/AvUc/H908pTwp9HdU74H+cYX/tBu7hE4Sph3e8dLh0mrUx/SgTFw+Kw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dMHkpJ66BNDh4vc2GgcBGq+n00YreQ/6i/pSAkSbl0w=; b=Kd5GUI0WAR7bmxQeamfiq4aG91mFJprxdXmTGUTHi7w04KIJrMJRGcSFljZl2mKPbM6fvFNjviOvfUHgjAzjrV49XFqhFRCbL76RxZK0iXNC+RK0TOgUnmMRpA873p+MS/YlgxBOVCYDOTkzOVLfYB9ufhxhIVuytjwsHFnatbU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PR3PR08MB5626.eurprd08.prod.outlook.com (2603:10a6:102:81::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.34; Mon, 26 Feb 2024 14:40:49 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5%6]) with mapi id 15.20.7316.034; Mon, 26 Feb 2024 14:40:49 +0000 From: Lukas Funke To: openembedded-core@lists.openembedded.org CC: Vyacheslav Yurkov , Lukas Funke Subject: [PATCH 1/4] classes: go-vendor: improve handling of go vendoring Date: Mon, 26 Feb 2024 15:40:37 +0100 Message-ID: <20240226144040.2007482-2-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> References: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR4P281CA0017.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:c9::13) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PR3PR08MB5626:EE_ X-MS-Office365-Filtering-Correlation-Id: 5c2c9e2f-3706-4d7b-717d-08dc36d8ec90 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +5caBHupJKI7NkrcA/Hp7WX2yaxT0yiEUP7Rq8E/vY04m2x55XL7UH8iZzCUmZik5k7wv97Eegi+cwglr4qa0SNIse98Mu2z4gDEN2uv66keFmlQIjRpDPqpo9xRTnhT2x8SjydnRL0Fzem55ilEV/LZV8n96a4lTA9p56GV75Xi52OWgmGMkpOwkRN2a0SFX6UbvLt7Kv4MOlAfhUQxiMBACKt3azjqWAlJuiCKEcAEnkr1mSjcxJ6ZykvHw11UjPdguoCxNWOec+PHgAlgN06ibcKTOhDkcbK0xHizWL+xZasDSdfJhr4YJm7NZr1JzYdBEYDfIJmSTh4nksrk6BRechfFY3aln8NaLe3kSwYfpEsHilO9pan55JNU/gZn6N48cJI6bCMhJsmIEHCwIRnKYmHrkKjnUKEtqgjExeCI6hmf3GX8xzgo1RLfIYXN7Cc+3ZCGWXR3D2nQhWl/eeWmujcgZkypb0yrIgoTdJM1dv+A6rJdqVUSHeZVf4nYirjH/a73+mnj1YjiJONGVfWzaDvOwbNvwxQUcY8J9pnTw1ucC4kE+LQlIFh1G4+Ko5Y3q6fdWNg1KN+kByyxG+vBX9uNi3pcvxtHx4KGgyu1SDpdhv5upbjpUlm1I5vHeMWJxzdUq1hM19mOukgCqghkBE3bcqVq7WBXsr8pJNUZPZ/MYHOGbdh+Kuo6YCDCF5mi9Nd+jol+4ZYVtFxDaA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1MrU7ypN/Z+iWmJMMXMhVzFoHYuippx2l/JcxIOFKKUN7tsIi3F/JLmYhTxgXHHrq30J2LNJWgblkXEQ9cPaOCcaJFwraT5N6lL5nuf+kZdxaNZlMsE6hJWhLLJJ6D87TXO+eCmmpHHjqoMH1H0AGln95AkytCjFXg4RxpkZHfjDTKh9RoYG4geyIQrJN/A+2C0NFVLbZtXA0sFMFIvxwBlCWox39MrV8Bokq21M9AOR7Sb23wMWlVO2RKrVKMP+RpkbD9xvbRa1ppcVOiS7GGVN4MoOxzSCCjAp5yUmK5LqwdC3qPVMFwa4ilbui7v1OaW7JKiXOzJR9NtLr+ePgtwedQRn5+PbDV/SR171aTKYPQsUd2pQuMFplCGaVS2eaQyGYGkeNU2LfNDZHUDKMTb2HS6eozuMEUzdaGgBtS9dABzCeKJ85w/3BgrQJOqBbTuXJxOJ2aFrCAAo274G35ds7yU9N3leiQQNAAo8u0rhxafK0cnxr1abQpiDf4DuSId119BUVa1PhUufS4Et5QrIwXIOFoImpBSH+Wql2Gk0fQXoxIZ459iZWdYkO90BYhWEqnIASWk/uBuju8PAYrs5LVNzAQcvH4F3UUqCvYVCVvT+O1CODNAWxEeAYT1CZWAwGP09nW3499Y8Tvoa95y+v6MDsHRqo68pOKLrQMoAv0qBWf0CmOTzPJv5y98t50buBxxmNxNwaxTE5hugj0OP7xinXh41cOXs4usup+4AhlYaHaFyC+wpn4IkJ+Akqds8b0orfpPd7SSKzptIPlz1MR7XbuxA6dR+l43z81mYkzS7f+vrNCmYrUxmiNe7OE6ACJjfbjK2fcZlf9weJTd0u6DHjHerYlNaNFGdc9HKv8aHiMZVwic5HYTOq5j2OYPkbGCOMObNslDyi0Wk7icnOqA6yZ/RxCAkxGnp5NT1u74IE6yqF1m+iAyFj2ll7GGJTpOBSr8m7yg0O7OgB3pjx4QN1UxDOY+2EcHvO5wGT1iXszFr03jWNlplzFaL7ndyFRP+P0ciDXERCVj6tAvzUuJ1vd8BuiQVKpevWbdDBimJQIP/k3Eh2fxQtTXCw/HqqEMylrDRYVH/kwedbo3S33sfsKG0GUnoCuEaFLadgClNOvqChZ+Nx5bldYihGmrlSsmKQkBUqjU375f80r0FYsqIwZB9Btt+eL9I+b9vpFmBiHlBNSJbFDE0pb+UaHguh4pghwCLDGDII4MmxTLv928qtC6umIJAC1m7FDvfLwXJ6s5WUDPs0bprK4MaBqEY1fl0yW/MMOZeLTHrNfkxT5fpMdDvrwJJye2po4IEXlRRo9Q1NfGKmLdaOn5d4iKnpm5Qzfx0mP+74paCK6WDUkLjf82fmtuN/G8VV1WfKow9dYUBVNGOldUmU8yFgUXjW21cVdlLcwh2hXaK8KG4svDmY+ce1WqcL06ZKGkQWXiXOHlYNofgckHZ0E5XJ+EN5KOGul/ohB8lmsKQSpNUGyE3feFL6Kbpicyy7Vm4GAZBR9PPc8HZRoC3ubFIo5B8MbEOQg4FLxa6+Y7KuXLkOO6eAM61H+WGcb89W4aeBWHNlvpKXGw+FPb8IktdTuqTcLRXG9ZjTO4ntlwmzw== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5c2c9e2f-3706-4d7b-717d-08dc36d8ec90 X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2024 14:40:49.0216 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: s+pQ4fP54zTwWWw2nfyMLpkS1KpM0Fc54w8TCcGrbOMVcAXBRtPmtFohPsQwXUL3qEdTdfdNhnJXP8YmTTvlKQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3PR08MB5626 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 26 Feb 2024 14:41:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196208 From: Lukas Funke This commit deals with some specialties around go: - Enable the use of //go:embed directives: Using this directive it is possible to include additional sournce files into your go source-code. Since these are no listed in the vendor manifest, we have to check for the pattern in the actual source code - Do not vendor //go:build ignore files Do not vendor source files which should be ignored during the build In addition to the changes above the behaviour of the vendoring folder is changed: the vendor folder is not linked any longer into the source directory, the files are rather copied directly into the projects vendor folder. Because the link was removed before packaging, yocto was unable to copie the listed licenses Signed-off-by: Lukas Funke --- meta/classes/go-vendor.bbclass | 324 ++++++++++++++++++++++++--------- 1 file changed, 240 insertions(+), 84 deletions(-) diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.bbclass index 1bbb99ac79..d4a7d7c224 100644 --- a/meta/classes/go-vendor.bbclass +++ b/meta/classes/go-vendor.bbclass @@ -40,18 +40,159 @@ def go_src_uri(repo, version, path=None, subdir=None, \ return src_uri -python do_vendor_unlink() { - go_import = d.getVar('GO_IMPORT') - source_dir = d.getVar('S') - linkname = os.path.join(source_dir, *['src', go_import, 'vendor']) - os.unlink(linkname) -} +def read_vendor_manifest(fname): + vendoredModulePaths = dict() + replacedPaths = dict() + + with open(fname) as _file: + content = _file.readlines() + modPath = "" + version = "" + for line in content: + # Modules starts with a single hash tag + # followed by the modules path + if line.startswith("# ", 0, 2): + t = line[2:].strip() + if "=>" in t: + lhs, rhs = t.split("=>") + + # This module has been replaced, use a local path + # we parse the line that has a pattern "# module-name [module-version] => local-path + lhs = lhs.strip().split() + rhs = rhs.strip().split() + + # Check for version replacement + # "# module versionABC => module versionXZY" + if len(lhs) == 2 and len(rhs) == 2: + lhsModPath = lhs[0] + rhsModPath = rhs[0] + if lhsModPath == rhsModPath: + modPath = lhsModPath + version = rhs[1] + + elif (len(lhs) == 1 or len(lhs) == 2) \ + and len(rhs) == 1: + replacedPaths[modPath] = rhs[0] + + else: + modPath, version = t.split() + if modPath not in vendoredModulePaths: + vendoredModulePaths[modPath] = {'version': version, + 'pkgs': set()} + + if not line.startswith("#"): + pkg = line.strip() + bb.debug(2, "manifest: module %s: add pkg %s" % (modPath, pkg)) + vendoredModulePaths[modPath]['pkgs'].add(pkg) + + return vendoredModulePaths, replacedPaths + + +def should_build(fname): + + with open(fname) as _file: + goBuildDirectivePos = -1 + endpos = -1 + content = _file.readlines() + + for i, line in enumerate(content): + if len(line.strip()) == 0 \ + or line.startswith("//"): + continue + endpos = i + break + + for i, line in enumerate(content): + if i == endpos: + break + if line.startswith("//go:build"): + goBuildDirectivePos = i + continue + if goBuildDirectivePos >= 0 and len(line.strip()) == 0: + directive = content[goBuildDirectivePos].strip().split() + if len(directive) > 1 and directive[1] == "ignore": + return False + return True + + +def match_potential_source_file(fname): + + basename = os.path.basename(fname) + + if basename.endswith("_test.go"): + return False + + # We assume that go version >= 1.17 + # (See https://golang.org/issue/42970.) + if basename == "go.mod" or basename == "go.sum": + return False + + if basename.endswith(".go"): + if not should_build(fname): + return False + + return True + + +def resolve_embed(fname, dst): + import glob + import re + import shutil + + go_embed_re = re.compile("//go:embed (.*)") + + basedir = os.path.dirname(fname) + + with open(fname) as _file: + for i, line in enumerate(_file.readlines()): + m = go_embed_re.search(line) + if not m: + continue + embeddedpaths = m.group(1).split() + for embeddedpath in embeddedpaths: + + p = os.path.join(basedir, embeddedpath) + + for f in glob.glob(p, recursive=True): + + relpath_embed_dst = os.path.relpath(f, basedir) + embed_dst = os.path.join(dst, relpath_embed_dst) + + embed_dst_dir = os.path.dirname(embed_dst) + + if not os.path.exists(embed_dst_dir): + bb.utils.mkdirhier(embed_dst_dir) + + bb.debug(1, "cp embedded file '%s' for source '%s' to '%s'" % + (f, fname, embed_dst)) + shutil.copy2(f, embed_dst) + + +def match_metadata_prefix(fname): + metaPrefixes = { + "AUTHORS", + "CONTRIBUTORS", + "COPYLEFT", + "COPYING", + "COPYRIGHT", + "LEGAL", + "LICENSE", + "NOTICE", + "PATENTS" + } + + for p in metaPrefixes: + if os.path.basename(fname).startswith(p): + return True + + return False -addtask vendor_unlink before do_package after do_install python do_go_vendor() { import shutil + import re + import glob src_uri = (d.getVar('SRC_URI') or "").split() @@ -63,15 +204,12 @@ python do_go_vendor() { go_import = d.getVar('GO_IMPORT') source_dir = d.getVar('S') - linkname = os.path.join(source_dir, *['src', go_import, 'vendor']) - vendor_dir = os.path.join(source_dir, *['src', 'import', 'vendor']) + vendor_dir = os.path.join(source_dir, *['src', go_import, 'vendor']) import_dir = os.path.join(source_dir, *['src', 'import', 'vendor.fetch']) if os.path.exists(vendor_dir): - # Nothing to do except re-establish link to actual vendor folder if not os.path.exists(linkname): - os.symlink(vendor_dir, linkname) - return + return bb.utils.mkdirhier(vendor_dir) @@ -86,7 +224,7 @@ python do_go_vendor() { destsuffix = fetcher.ud[url].parm.get('destsuffix') # We derive the module repo / version in the following manner (exmaple): - # + # # destsuffix = git/src/import/vendor.fetch/github.com/foo/bar@v1.2.3 # p = github.com/foo/bar@v1.2.3 # repo = github.com/foo/bar @@ -103,14 +241,25 @@ python do_go_vendor() { pathMajor = fetcher.ud[url].parm.get('go_pathmajor') pathMajor = None if not pathMajor else pathMajor.strip('/') - if not (repo, version) in modules: - modules[(repo, version)] = { - "repo_path": os.path.join(import_dir, p), - "module_path": module_path, - "subdir": subdir, - "pathMajor": pathMajor } + if not (repo, version, subdir) in modules: + modules[(repo, version, subdir)] = { + "repo_path": os.path.join(import_dir, p), + "module_path": module_path, + "subdir": subdir, + "pathMajor": pathMajor} - for module_key, module in modules.items(): + # Copy vendor manifest + modules_txt_src = os.path.join(d.getVar('WORKDIR'), "modules.txt") + if not os.path.exists(modules_txt_src): + bb.fatal("No vendor manifest present") + + manifest, replaced_paths = read_vendor_manifest(modules_txt_src) + + bb.debug(1, "copy vendoring manifest %s -> %s" % + (modules_txt_src, vendor_dir)) + shutil.copy2(modules_txt_src, vendor_dir) + + for _, module in modules.items(): # only take the version which is explicitly listed # as a dependency in the go.mod @@ -119,10 +268,12 @@ python do_go_vendor() { subdir = module['subdir'] pathMajor = module['pathMajor'] - src = rootdir + bb.debug(2, "processing module: %s" % module_path) + + modSrcDir = rootdir if subdir: - src = os.path.join(rootdir, subdir) + modSrcDir = os.path.join(rootdir, subdir) # If the module is released at major version 2 or higher, the module # path must end with a major version suffix like /v2. @@ -130,82 +281,87 @@ python do_go_vendor() { # # https://go.dev/ref/mod#modules-overview if pathMajor: - tmp = os.path.join(src, pathMajor) + tmp = os.path.join(modSrcDir, pathMajor) # source directory including major version path may or may not exist if os.path.exists(tmp): - src = tmp + modSrcDir = tmp - dst = os.path.join(vendor_dir, module_path) + modDstDir = os.path.join(vendor_dir, module_path) + modDstLicense = os.path.join(modDstDir, "LICENSE") - bb.debug(1, "cp %s --> %s" % (src, dst)) - shutil.copytree(src, dst, symlinks=True, dirs_exist_ok=True, \ - ignore=shutil.ignore_patterns(".git", \ - "vendor", \ - "*._test.go")) + bb.utils.mkdirhier(modDstDir) - # If the root directory has a LICENSE file but not the subdir - # we copy the root license to the sub module since the license - # applies to all modules in the repository - # see https://go.dev/ref/mod#vcs-license - if subdir: - rootdirLicese = os.path.join(rootdir, "LICENSE") - subdirLicense = os.path.join(src, "LICENSE") + bb.debug(2, "module source dir: %s" % modSrcDir) + bb.debug(2, "module dest dir: %s" % modDstDir) - if not os.path.exists(subdir) and \ - os.path.exists(rootdirLicese): - shutil.copy2(rootdirLicese, subdirLicense) + # Copy main license to package if it does not exist yet + licenseSearchPaths = [modSrcDir, rootdir] + if not os.path.exists(modDstLicense): + for p in licenseSearchPaths: + lic = os.path.join(p, "LICENSE") + if os.path.exists(lic): + shutil.copyfile(lic, modDstLicense) + break - # Copy vendor manifest - modules_txt_src = os.path.join(d.getVar('WORKDIR'), "modules.txt") - bb.debug(1, "cp %s --> %s" % (modules_txt_src, vendor_dir)) - shutil.copy2(modules_txt_src, vendor_dir) + # Copy source files + exclude = ["vendor", ".git", ".github"] + for root, dirs, files in os.walk(modSrcDir, topdown=True): - # Clean up vendor dir - # We only require the modules in the modules_txt file - fetched_paths = set([os.path.relpath(x[0], vendor_dir) for x in os.walk(vendor_dir)]) + dirs[:] = [d for d in dirs if d not in exclude] - # Remove toplevel dir - fetched_paths.remove('.') + pkgRelpath = os.path.relpath(root, modSrcDir) + pkg = os.path.join(module_path, pkgRelpath) - vendored_paths = set() - replaced_paths = dict() - with open(modules_txt_src) as f: - for line in f: - if not line.startswith("#"): - line = line.strip() - vendored_paths.add(line) - - # Add toplevel dirs into vendored dir, as we want to keep them - topdir = os.path.dirname(line) - while len(topdir): - if not topdir in vendored_paths: - vendored_paths.add(topdir) - - topdir = os.path.dirname(topdir) - else: - replaced_module = line.split("=>") - if len(replaced_module) > 1: - # This module has been replaced, use a local path - # we parse the line that has a pattern "# module-name [module-version] => local-path - actual_path = replaced_module[1].strip() - vendored_name = replaced_module[0].split()[1] - bb.debug(1, "added vendored name %s for actual path %s" % (vendored_name, actual_path)) - replaced_paths[vendored_name] = actual_path + # normalize the path, otherwise we get a "x/y/z/." + # for the main module path itself + pkg = os.path.normpath(pkg) + bb.debug(2, " processing pkg %s" % pkg) + + if not module_path in manifest: + bb.fatal("Module is not listed in manifest: %s" % module_path) + + # if the folder is not a package, we can skip it + if not pkg in manifest[module_path]['pkgs']: + bb.debug(2, " skipping pkg %s: not in manifest" % pkg) + continue - for path in fetched_paths: - if path not in vendored_paths: - realpath = os.path.join(vendor_dir, path) - if os.path.exists(realpath): - shutil.rmtree(realpath) + dst = os.path.join(vendor_dir, pkg) + bb.utils.mkdirhier(dst) + + for f in files: + srcfile = os.path.join(root, f) + dstfile = os.path.join(dst, f) + + bb.debug(2, "cp %s -> %s" % (srcfile, dstfile)) + + if match_potential_source_file(srcfile): + shutil.copyfile(srcfile, dstfile) + + if srcfile.endswith(".go"): + resolve_embed(srcfile, os.path.dirname(dstfile)) + + # copy metadata files + src = root + while pkg != module_path: + pkg = os.path.dirname(pkg) + dst = os.path.dirname(dst) + src = os.path.dirname(root) + + for f in os.listdir(src): + srcfile = os.path.join(src, f) + if match_metadata_prefix(srcfile): + dstfile = os.path.join(dst, f) + if not os.path.exists(dstfile): + shutil.copyfile(srcfile, dstfile) for vendored_name, replaced_path in replaced_paths.items(): - symlink_target = os.path.join(source_dir, *['src', go_import, replaced_path]) + symlink_target = os.path.join( + source_dir, *['src', go_import, replaced_path]) symlink_name = os.path.join(vendor_dir, vendored_name) - bb.debug(1, "vendored name %s, symlink name %s" % (vendored_name, symlink_name)) - os.symlink(symlink_target, symlink_name) - - # Create a symlink to the actual directory - os.symlink(vendor_dir, linkname) + bb.debug(1, "vendored name %s, symlink name %s, symlink target %s" + % (vendored_name, symlink_name, symlink_target)) + if not os.path.exists(symlink_name): + os.symlink(symlink_target, symlink_name) } addtask go_vendor before do_patch after do_unpack From patchwork Mon Feb 26 14:40:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 40088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 678FCC54E56 for ; Mon, 26 Feb 2024 14:41:00 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.138]) by mx.groups.io with SMTP id smtpd.web11.21659.1708958455668094328 for ; Mon, 26 Feb 2024 06:40:58 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=Y4CAJFHb; spf=pass (domain: weidmueller.com, ip: 40.107.22.138, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VLe2kU1e1XvDG6fvCY/C/tYfOif78mmxGHFnyDwUMzjrFmQpE15zxmkQfwbrUkbiveyuPTePfZG3RRRAAtbhLfb27+e3G604kgpjJO2NUgAJV4EvYkI3Ub5ePedm3EiLO4BmkW4I/WOB7xkOLzd5qTdh33L/FdCSzmWjYMhaHk8UJRoTB341J/YkfCVGyT/5goRRTrsPiNrpHWjp6PzdmtPEIhZ6gISbc6hyORGCrPGf9414z1GvVk87S4JBscSegAIUp7ja7pK4N/0gKwKgUhxcxBCb1oEaZ86kyPgfk4cxLvdkr4D8obKcvYyNlptJXU/bZeAzo5+pjLhd8rK/XA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lqH54TYEFAZaieTHobgEyxLwZtbKM4Xxde1XaFi9Y+E=; b=I7Tum1qjUXjzZ2yX/37gTKi9i4c7myZ6ssu2GSmS+ShTzoOw96lW/wxRcI5l3A5/c+lxz8hfIMIY4DUGfuGAfuudJOLjocIn3gNYphsMWLheN3tb/TVgaAaH6QLb6y/NXLpr2/EB6pJWXGyR0CXXwzoNKPdE0beowQsbRSMpFlc+FMv2BOIqQYd5O3f6J6pEQlh1C4HNVf9lTNX6Mw4lwDZ4N+5AqDdaqZHGShZN61mcOsOs7cUdgyqiGzti5uJ6m8MI9LrHSR3A32ubwWuQpjgtEy/WL1jcMdKPnq7mw0bq2U7tFFnVNc3DIoygfWoOlQoUhmyyVhXf54h1xOcSZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lqH54TYEFAZaieTHobgEyxLwZtbKM4Xxde1XaFi9Y+E=; b=Y4CAJFHbLPv1Matob8sYjl8tV55GCczaklPRlIOzpvdmradyMOWrMccYaiyuTGeFrF6A5VMX73d4t15KTcJo5en1I4XATbw3VoIlBONuO8GtjCUdUAqwjum4yaBNvZywCUun3zm3RXYgk8uFC7lUfkin4dM3+mnpZ7ItkjwXwUE= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PR3PR08MB5626.eurprd08.prod.outlook.com (2603:10a6:102:81::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.34; Mon, 26 Feb 2024 14:40:50 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5%6]) with mapi id 15.20.7316.034; Mon, 26 Feb 2024 14:40:50 +0000 From: Lukas Funke To: openembedded-core@lists.openembedded.org CC: Vyacheslav Yurkov , Lukas Funke Subject: [PATCH 2/4] recipetool: go: do not restore go.mod/sum file Date: Mon, 26 Feb 2024 15:40:38 +0100 Message-ID: <20240226144040.2007482-3-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> References: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR4P281CA0017.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:c9::13) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PR3PR08MB5626:EE_ X-MS-Office365-Filtering-Correlation-Id: 104b791a-2287-4667-67eb-08dc36d8ecb1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DdrPEdMkOCiDjQWpHMX4l23zXuq8ztGCgtQdRDpdMOxIMOq1vgQ/qX/5uk0dIL8VrsF9Vse8oqarJY16ds/2OjuuU/7d9YkvNpV6CK5k6BngY1AMqS2EiI7OcVK74aUEzTz2vFNysvpJG2oTD10+ogTYKwOWYT1CAgCR6RuoBzKv9PLtmruaOjSj6xkBd0APoOnX1oT/pm3Zu1sAHPKjeiwEndtivBvlfUG/o/psQ0UxwJ93Io/SN0dtqzy6jXWWv5KT7ZlQGrhPP08q9lkmhrKmn5q5SuOyOucPCp2K65p/4sisVOZ67MBTmHlJ0okvjnn5w2sgU5twdwnYhFpw43GlXNANdJM/yMGc0G0ZLZr+LHNW0oN+au2oXwEG/mDvWrGEW/JkZdIpjbI5hQzVyTmzTjP+NjQyYpONulLnhagPpbxAmGuhB9IXSwYN7U5TOHmD7So7ZuN9/47vy0gdimUhUVIwhZllBdqYV1ITuXfhjEn49WUI2tTj3nTYGvX8ax0ZHoB0povcAeBSZzYjN0Dn33i+u2/iUXAvvgOHgA5PPctIBXcNuJKC/vkhaTZs7suDcpOCZa40SulcnXCJC5iafRsTGhC0lIDX3yxFCvjoNYw6h/H0OlGTfQVp5+NS5qQgJAhwwkmajndpz//Y4IxDeqSde3mro5jO1WdXr/eaqPCqqPy0IRLMVCDMVMDXrheE9/8zlAN17bj/bF7DeQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Xo6xUSW1nhQ8tJqgfhxFaVQlMyI5FFeCCsGAT9NjXWC5weo78ocEeY2Fs+POcdowMduPdgoh9q+5mQnrTkLycpQmQWXdgYyuna9ehNAtvc4aBGpBZGm/Nls3nD4U3Jwq7CByB4bLMOQ0WX5JcFdhEKQmspueVSzY1Q6akQocnhsWmmjIEGrvwPMMJSIP9wBkl+AomJ4TSgLXHbZQKugmifi7eoY899XvtpLzitJLDZoSyLCgO+/7Pu3sgJvmUf1Dhi+6xUgCJ+96U/zBMnDynFfrsrDaOFWAS9q+LEDVAkl+NaH629NTQ3kC56LuFRSZWFyz5sOUvYliHXFzteSs6SRHd+FW9V/HdLmRBryxiMUeNceV2JVNp70xVhCBYa9WcYFE+NYjmdnNtu/Tt1Tq23bjcZfJu9v83JuS/P8+jPbHGNhtNiQGG2QoXz9hL8Nlzw3SYhpRBvNt+LE3pccfsNYLc+sX+qVrVzmIGh6sSS9ouC/c8y0j2x/qJgfgQNx/fg9AJf2BzaWBLdNL/yskxiBbf0q5baoI71qmOPlwp2EbkZSnFp2BPI79VK5/o1PnFvm5itAZ1ZHBf8lPWPpOQBwbKB/HSqiQGd5CFTgErQPSdL5KKHYTzAH79x3CsTwHS1bYaEBHREuAmcuIfcJuZsik2nIJaAiYyzpPNvcFkLwzr2sJgTNy2qWJE2UgMngqY4VOrKi6xwwRr+mVt275g84PFSGkc3uOztIZHl4xUNEMZpOf9yh03kvyJtvmkcsqB66Zr7k5aIV2+0V5J+rL9GnvZy8SBvVW2YyZ/v53hs9YHVdzHnFyr1sfJfkQGMseFxgrH3jonP4IGHShhLHt9GQ1RWESv9FzxlkYIjP+W2fKVBCEnXOV3UIgw8O2FOBpOp+20r6y4wQOru6/oDhL6UrVk9KTsuS4VPE/jgWN40rmX49EhDHeHWGqty7e9YsGl0eDAo2RUr9Z3S2EK3ae9Bcu2alpkJfYZC4os2IdjX//fBFbeIB8fsWx3OyT9ZW0XOAwBn0Y+WxAqR4XJOJWgz15xXM4K+pBKOT/sTC70Hi83MAEtYioL/I2oaAmsaCp0Vwd+p+KWd/qEiBZR0H9+zi4KKHo2N0EKxCaFWMujLv9AeonpbCn1WtXovoLDHutX703HRoE/iPjyJMwlMXG1wQZrJEPs3h5m2Yz2REwBIjL+6BAW8fAFyqS2D8On3891GRCU8O+BSNy1LPPcq9182JBN/fnMCgnNh/gi3xInqyN/HkwZkXuldGF1RpP52hu6tMNiLNJKq/98VGxvQXSbGPsoFE2wujuR11wPk0QHJs1IKumH2oU3lpBdE4E2DJuiJPZgPtivyVBLNX6Z+sNgtzdq+IA0QPvG/RQlh5Pt5dQlDcZPN3XCQFgAeeb4e29gelKFr68pp+qyXS2Urn2X6aMMYi2JmcCjtwqlwv9JtHdkPnB19mZYENAsdZ7lWHkMw1geVmZrINnfxoN3hcTdfs5xpberY8zcQkKjb6UkVi6ISQ9Z1l6kkJLGswtIcyhqoowxyFqB7S+eSkrdU44Fapc2otEWhCeT0XkxbyqvzJ+G4fGQYoBftjCrXUNxQElRfR5I5x+aOq8SNHRiZVMeA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 104b791a-2287-4667-67eb-08dc36d8ecb1 X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2024 14:40:49.2409 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 601o2yBbF41kVaxRmSKmjDcEcclpHqQ0YW0NzvGkCU0sqvwsJiL4GdNdMyWHqUQYvOPaCXv1J6hlXSJxoE73hQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3PR08MB5626 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 26 Feb 2024 14:41:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196209 From: Lukas Funke Do not restore 'go.mod' and 'go.sum' file after the projects go version is updated to >= 1.17. When we leave the go.mod file as-is then vendor manifest is created using the < 1.17 version. This results in an inconsistency between the patched (newer) 'go.mod' file and the 'older' modules.txt. Signed-off-by: Lukas Funke --- scripts/lib/recipetool/create_go.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py index c560831442..abe1f6e0da 100644 --- a/scripts/lib/recipetool/create_go.py +++ b/scripts/lib/recipetool/create_go.py @@ -598,8 +598,6 @@ class GoRecipeHandler(RecipeHandler): # Create patch in order to upgrade go version self.__go_run_cmd("git diff go.mod > %s" % (patchfilename), srctree, d) - # Restore original state - self.__go_run_cmd("git checkout HEAD go.mod go.sum", srctree, d) go_mod = json.loads(stdout) tmpfile = os.path.join(localfilesdir, patchfilename) From patchwork Mon Feb 26 14:40:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 40087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 663F8C54E51 for ; Mon, 26 Feb 2024 14:41:00 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.138]) by mx.groups.io with SMTP id smtpd.web11.21659.1708958455668094328 for ; Mon, 26 Feb 2024 06:40:59 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=HNhzrRDz; spf=pass (domain: weidmueller.com, ip: 40.107.22.138, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Un/rF0TRbaKfFp2whd2KbtaZR0blnzRVZ+Nacq3AQ9vh3mvQmnIX3s85YT6rk70dVBbCa045JOfj4DfbccDT2wOpXxCrkWL32RIDqBV5ozE3by4+tFk/mdrZk473uk6zC79RH0AtqMQlBJH1yVmkkzgjICndAu4Cmls4YhJ0BVX6qraV4feW1rezBukfaD1G1q/jytyMbIk5PyWz8NuSkCsP/DqmRtKgG2q0P4w5j60RjAuuDsLZbQ48U6MBUAzVzdbmPgxYIXGZd0OcYn225/w5jx/gaYSqsoYG/pt5Afz9zF9+g1spvx32NFdu7y6A9e8tw2MY6J6St0s+QHct/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7sttJbBjFdvWrjriC8zZIgcbkPzFRPv1M78vCqlnMKs=; b=jHjPvXrUmxtm1OMt9Jrw/cZtVfkqF71fYm2Paw/jQhu7U6z2Qij3MAstpesxx/DFL9IVFaoN5vL+e+OBQ9f9xucwuKaFT9+qpgsrmp0lcFi88znvw0ErppgvVfUXmYAIKUMZf1+/c0iJKvAsdv+vH69ZWwwGtK9WgLO+LHU/sHuIypSf7g+3SudLy+MBXgS35zneuKFDHasS5OlgC6MW4/g+50DdssqqpmPPNTsdbdqL9R7yGQ71AE7Wl6BaJU3xO1JMY4hxF0TTCX1OH0iyTB0AnmuLfz4uKGQyUxIcFX2fdY10cpYyPHLPSFCxEzjbvUbrLKKvdkg6njAUSdnh8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7sttJbBjFdvWrjriC8zZIgcbkPzFRPv1M78vCqlnMKs=; b=HNhzrRDzwQtYIBQfa8QdWoke/5eC32OV/mLSdgxAhzx86rskQ81gY1D3USeMCKOqzzS7I1uWRwFBIJUZlFtstJr7zJF0qn3+IsaTm+RPARXzXPEifVNqhGnJJZGwLUdeWVu9a1d+C4nFmdd+ePQUw8FLxTQxrX+mxNNnfDDxgps= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PR3PR08MB5626.eurprd08.prod.outlook.com (2603:10a6:102:81::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.34; Mon, 26 Feb 2024 14:40:51 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5%6]) with mapi id 15.20.7316.034; Mon, 26 Feb 2024 14:40:51 +0000 From: Lukas Funke To: openembedded-core@lists.openembedded.org CC: Vyacheslav Yurkov , Lukas Funke Subject: [PATCH 3/4] recipetool: go: ignore 'go: *' log messages Date: Mon, 26 Feb 2024 15:40:39 +0100 Message-ID: <20240226144040.2007482-4-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> References: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR4P281CA0017.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:c9::13) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PR3PR08MB5626:EE_ X-MS-Office365-Filtering-Correlation-Id: 26a4e539-5b62-4dc7-2aa8-08dc36d8ecd4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: I9ozTteDahKcjQpuhO5BQOspG0e3SFGjw9o8k8LvuyoelPD25JErDwzJfg737Xy5xazYgFtZbVcBbm1Fzbc1lh0p6mgnZIz84d+gCIA2lfbpyzEkOHSEeIPXDD5E7wI8RSQ7Op8JLhuogutW5tjD1MzEcSDrRB6hDyxdhdMvUsheww/1q7X6k2hsZQd5lw4jMc8tlWu/S6KriRL3Nej9YXi4JKrUIVPU/3ongqjItde9CKiZNBWFmKfsmFQ0LC0ZNDdNV710hAyZv87ClPHszfH9z+j3YNKHeowsjJbB5ItuDoQ3LT2TixOEUGDBRQijFU5I/hDZ9xqFqXMCutvndOUbhOe8XEKtzf6NUycRhmpmXhR67sAG+aTtg/7fUKILnRSSaADpOy8+Wnk4LfX4mo1JeXrKRW7CPvimG/o06WAgWDiZF/ql5UF9K5W25GL6q88gouWnEH93/7IEz2ZsMwaQFb5BuR37WwCsFX8d5vBRQM4nvqY8S84zvBlQmx8yJ5enz/8me3B2ojsVduEqHEq9V+8HvRfhYtQNHHw8SwD3inY36PqyMsi5+llJSEG6Mmj1BY87wcxrA69/IsZlEj4Rzh+aBw9812LRVZzK3lE7SMloFMadiOIpvWNPf/k0gTtGWYzZPIAphIA6/+86dzfoV1Rt204N7o74Mp9hk2g5EBWpv9X12GzvbUr5NjW1GCuRzPOfUSZLGlz6+Ckd8jruO69sYd2sXpT0w2/mrbA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: u0ZwCbvRn8LGH5pw5IbNSJjTpjfp4EnEHj0m+b+P272Q5o50lvJBLBS2E+9a+DE+5oZMpKx0Byh1ayIoAUQyKsGtVOyhpHVV3rX+KJx/mk5I2JhwHXmSlcbtxbAs1952dGSL4fub9pgNXf+TW2eZKjaQ8reoV8PLAwYOVldCAzs2Ua/PbwT9glGBx+M8mlZcnb66YqXiu/6VHoWtr5WSApbK8UookwU1+8dlhTJoL4w1FaeYe9URv0n8slJCKXvFxX5BeAwSoE2RYrJ29WSBdM/N0iC8w5PFH+Stof8EGBTrLrcSSIS5QTFlnIwWZdgt/h21u6rG6fpQ3YEeyc1SHi553yvNfsuNPE5texl9ubv2exaA3N8xTyxc75W+DGq3ubYD/BvPydX2AhPAv4HBDkOzJgZQY5Z5qXVvGmHcUKaLhU/MnNAyQbd6MpJaQLMqgdvdSsr7HqfzlKZ4bgYsJQZa1WRcv0wJnX26Gce0BB+uMamWBL78S2zhG5sx04DPFFpjYj+8i6rKBiH8CoI8W9qjwl+YvbvpEsU2oVrMstqtpGUIqQLUcsLcyz1cGa4AxG5w7wxCwp3LP1yHDJJ4AXaL3JiI9K/pgJrzBPVQfmZpZ2LP3daEE7x2ugwYq28R8g9sBNCFgs7xleZqzB7svN9BOMjOx8DP669KFHgg3e9DQUK2GNulaXdqFPQOHIIz7JbigXw4y3npDzn4VcxVPrPAXxstSP/Ibxs0wsFdb2Q4pWqGaQlpJwRngrjNsdgl76nnzZC0wq7tMLcHuGKuZfIWD4nxkxBNn//D6g3EAJxMyG1j8w2lFohDPpaNiXNJjoauLvDLLA5Z0TXE8Jc8DQ2xO3Qm5l/CSNOnZNIlRnJYBGZx11DIboy/suKGFoLg6EKr0DrslxWmZdK7F0euNfUwA+7S2PMemzQTck4lA+4SWSIAPWKeC9vP01KTzL0Q5BfCT6QPahpPYkNPMENb6D+MJahDeJaZQBRGCK0GNp1P9nbhPfgrhexkRo2ksiAKNF/atwK+nQxHPeaAdTJw++dBuSQJzi2xlPspz4bOTWMy/DkIV4lissdsC6WM4/Pxp6h5RfYuOWUpdbya20w/VPisEaWVRz4B3m+TLcy13ziSG/xubt1JGijDBDPKxGE/SVRuhFvldv6wYMdyUfUVJzPgBMoF/UVmWKbS8tmuAB/uNbM7OKNvU4rom560jSL67R5yHruph02voQxrEMfi4+kBwXbc7+wrfQhflqXkfWv4RSf+vqXpJHJGMI1pJz43lG34provt0d/ngjvoSX5XVdARLkPzRyS2hA2onayKIV0gPB8Bg8MIeLy7al3fW9u6fhhL82q/CEmPfoNK7Kz09AAFzSXFBfKz2BRFJPbO80LR6ifr7/cgzLXFjLYwHpQaa2C52g5OFerkuQUPPUxgAk7bnmyOdk3cAGX77ax0JCuOXdfk+V+WqCsJZmNeWcv1jdQZjFxAHSA0JVqOB7GhMswRD1Uw4qqNvyqOMa16DGp5BkvMNjcQD46gzPqLJUY/EOAXFx7l/YT7duxF3OVZV7SpgvY1ZOVySAdrqe41w1RiHSMbXhxqaM4MrL/f/fd93Vw+jhDK0Vp1k+kaBqnLQ== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 26a4e539-5b62-4dc7-2aa8-08dc36d8ecd4 X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2024 14:40:49.4612 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ZbGHhu/HuTHAZ4ENWPMlQT9MLnW5/Wi+WH99900smgeVp/+/iHb6FOBmIRiRTAfyb60sq4skMJnwN0pSRaUA0g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3PR08MB5626 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 26 Feb 2024 14:41:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196210 From: Lukas Funke Go will download verious modules during vendoring and tell us about it, i.e. "go: downloading foo/bar vX.Y.Z". These lines are mixed with the relevant modules.txt lines. Thus, ignore log messages starting with "go: .*". Signed-off-by: Lukas Funke --- scripts/lib/recipetool/create_go.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py index abe1f6e0da..cae7175a33 100644 --- a/scripts/lib/recipetool/create_go.py +++ b/scripts/lib/recipetool/create_go.py @@ -615,10 +615,15 @@ class GoRecipeHandler(RecipeHandler): _, stderr = self.__go_run_cmd( "go mod vendor -v -o %s" % (tmp_vendor_dir), srctree, d) + # ignore miscellaneous go log messages + vedoring_log = [x for x in stderr.splitlines() \ + if not x.startswith("go: ")] + modules_txt_basename = "modules.txt" modules_txt_filename = os.path.join(localfilesdir, modules_txt_basename) with open(modules_txt_filename, "w") as f: - f.write(stderr) + f.write('\n'.join(vedoring_log)) + f.write('\n') # don't forget the trailing newline extravalues['extrafiles'][modules_txt_basename] = modules_txt_filename From patchwork Mon Feb 26 14:40:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Funke X-Patchwork-Id: 40089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F3D7C54E41 for ; Mon, 26 Feb 2024 14:41:00 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.138]) by mx.groups.io with SMTP id smtpd.web11.21659.1708958455668094328 for ; Mon, 26 Feb 2024 06:40:59 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.onmicrosoft.com header.s=selector1-weidmueller-onmicrosoft-com header.b=oAbGiIco; spf=pass (domain: weidmueller.com, ip: 40.107.22.138, mailfrom: lukas.funke-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QEix+jLT67wDvojtF8z4qyjbYCmkcpZh4e23c4Xh5Y4KTnPYTVPQfUlOt1dVoUGcmG7lXQb7wkkMdv38pUjXFKpr4DAJC26EjtK+nsse1T4TuQQCjOOKVh9erc1EdjeWQnck16zHX09sOhy7Yt3YpfzTkTnpzhpRwnZuNqblPMRGYbPYOM7xK3Zr3p9BpPurMLbQZ26mOhK2OUCZ/p+mDrogtYzE8BxDhcMz06caUBZyH4MxVXqcMibZTStkN5jvNIUbdh/sAcAH+S6Ry16xQX2GFkVg0sxTIIFDqorgpLYHSUnCAS5WFgoEnxCyPgvqeEJ4SiB5t6ssXZrkp8XuBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ni9Z40ZvhkEBS2xcNmIbKTnT4g6YBuZxtuIP+NUZYNs=; b=XNgGMmA6iOLWvPrRvyJ9nPTYVOilGZxs+NkM317iWI/PJP4j24hO5cDjvkMv0BgPZEB0Z3PYGQcOg+R7kSEJapnmq1ZhOXj92uHbCpwnbzcMrhxyaO3S2cQA9ac1y4zPZ41ZPBJ3UhGdvk/0X9vuPxG2N11DfkNDE2CWz+jjXX+nlWmOPPaV7X8tZfF2yUL1UwX9PLgs2b7928MphZ9MerkVgUK5MAd1y2X5xT2cLN0JSEhy1IzyEjQ/HmDnkgvYWNIwyGEwghBtFFUf3wgvyXYA2TCZv8qQu6amIoiCthtcZjheDxFiQ2cxbt0a1jYjNtjlY8jn6S4s7LFPnJQuzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ni9Z40ZvhkEBS2xcNmIbKTnT4g6YBuZxtuIP+NUZYNs=; b=oAbGiIco1XpI4WJxEPwY+EZlDCN6nCIgi0DZ/I6ynI9s3kBWv8RRYhHvVa5p8ztXgoiuuF5umViDLWF3jkOoF/QCsdGiGs1rZYfQ5J3Yqe0Yi5CdVxSdD/rpJee4GquVt23WdDkOZ32CfwUHwS57MxHhPxROyp1qEWathR5HMLc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) by PR3PR08MB5626.eurprd08.prod.outlook.com (2603:10a6:102:81::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.34; Mon, 26 Feb 2024 14:40:52 +0000 Received: from AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5]) by AS2PR08MB8431.eurprd08.prod.outlook.com ([fe80::d003:60c0:6918:8a5%6]) with mapi id 15.20.7316.034; Mon, 26 Feb 2024 14:40:52 +0000 From: Lukas Funke To: openembedded-core@lists.openembedded.org CC: Vyacheslav Yurkov , Lukas Funke Subject: [PATCH 4/4] recipetool: go: apply pylint recommendations Date: Mon, 26 Feb 2024 15:40:40 +0100 Message-ID: <20240226144040.2007482-5-lukas.funke-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> References: <20240226144040.2007482-1-lukas.funke-oss@weidmueller.com> X-ClientProxiedBy: FR4P281CA0017.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:c9::13) To AS2PR08MB8431.eurprd08.prod.outlook.com (2603:10a6:20b:55a::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS2PR08MB8431:EE_|PR3PR08MB5626:EE_ X-MS-Office365-Filtering-Correlation-Id: 69460d8f-5993-40ec-7087-08dc36d8ecf5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Z1y+sMjYP2ezNWbrNpRIWIhI7LPWwBRQvVP556yhlXUaDQfXlSmcXAFA2HMhopXAhoJu3oqjv+6UzxqtvcBL9KDEteTMG9XCNO2TrGLpy3Xa+OAdv0W3LtF2MtL7SGyfRmz7J9cClspPTxGndE/ifFiB4kuow9S9EhxWOtmQ9AU64nMrHW2Mv3777/VucluyWNT7bAADyD+y9CLV/oPADBfhLRkagp4N3+sxvycwqWaK0hVl14K5VtBW3DGUbAtj+UOKfD24GPOYrOLH/93EaXdcMGKG9IA8IXyiV41/WJ8SnC0MiEIm8U6KILz1w7GQJ9H7YOOGzCVL3wEXZuEmqfRgq33zer4zd4NsIzdFNNEO3I5rsSNRZYIaRDzABHLKB+w7zUPBuYVNstuVnbOqxV9tDXM3MmUheayfhMLpde5u1rNHdSbDgDJfIGoLPRB8HaiRF+wy4RHJ7RwagspV/Sx5LaqNFAI3dXBuzPYDcDcK/6kichqbItj8bIuOCH+MKVeZHeX9ewMblL5SrpGZhwNNOsiPyF1joewXEnORsQQJjf7uF1sBJfHr4Qghm/yAWc6gOuKn6iRKFgGppBpC1w43tzaxx2TKlDEy8S15xbXkt/oLcWvLUZWZMnIt4Jy5jsSdpZQ8PCeCZPsAWqLo3rm35EHSO6A428M99AisiiKjyRoIS8u8xDTfsi9Byf+kpW+wbxKZ5YYR64+s8K2osQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS2PR08MB8431.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(230273577357003)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PoCKs74IlWI35Bi8U8MSY89A0SyMkFsPgQAF1vd5rLUXttofPcvwIbL/oL4VuYIYNa2TdcBrrjz3GXVnGYV8CwEGiuBnRjGVUySCZS2PDKGJ+hAco/xi4p8fL7sqPeYAaMW2DdIUgKfHAx5J6LyEGLgptQDek3mkevpJvbZvWb+EldT+6fQCanM6QZl7JUMG8EHeIzQMnn9rDVkfnELhN4MG7k1tNGaMYD94tz0Ki+gWgQwRMvAbsgJSaaFbHpQOcfKq76Q7J2HUwRXLVFJvYjdUGrXnIl2NYvUGSVvVdIj9LrxHzpqhXcpZR89ANIxVew0nwAZAzKU6ZYnG6N19uD4/4WoCxWqVD66dp38h+YS4MM+8EfMQiKKRv8qXnSDS44+EqDc3B3hqz9Xr8mnGeiP7NTDJIzcFIl0tclQy6xKNo7hAw9uySO5pLV5LW0WLK51z07nq4dH6EXepb4NDpx2Q9Z96UNljWAVVDrCyN8gQzA8tUeu9VL/HYa7QF1xjfgRrIa5QFLdBjSmVqIt1ec1PyR+dGy+EeACJ1KBVSuQAxfAudI9nwMDxP1GJO2/WeRGdhEEgn1jkzZrqmb4Ux+N80uAls1Nk7Qt+at2V8EVS+NF+fffjflETwZG8RU3jt8lTgToisWk7FTXxtzs7UoytLiVOicJoDqKRdzHbS8bHHIGJmE1HZhmUdqPZOqfnWRGtAf9Mtk98Ziq9EDM2bH1flaqRfXcAa0R+UC1H6l7ulO8pCnMYSEZJpPbRgNmf1AyxbE17RgaNiW+QXxtrNrlcB6ugdyyoomTHaxyuO4I7X0GRFUqesQB60xXzqHzhdcazQ/Tb5+yhXd+OxO5zqvg4rYI7mx6ZfGJrT3FP2qQ4m/nQ2CJCahN1Ww7mANPxjWf5Vl+GoxPJt1ePRE8PXVqmbVMfsoPg4omvlvB59Xr5CPRU85uDez/zgt8oZDPIozLIPxRlKqup3AQUJBJVMGk+egFB5/Lar8skYUiTRW9MFyIhy8hvFSaoYR3YzbVXsjIOnghWfXCvUl3Fl20U9VrcxR1hHkzyectWMPw9HmTpW0F5YJ5Kd64xMmRracTd8I4zc1c1t5ubLz1qVrW8jRLNZApq2eRpu6glmQcoViQHW+YJgTtZeX2h1bn/NhiHEeYJTX1euxgvBZ9F+ukUbkaREc1y8X/PUp5bQiObY59xk+EXt36Mi3O6Vb+5kpfmB0GJK68zr0y/JuVjkkjzs5mfueB5sxYAe8PfpabIwGaw3JbO/TKtskG/ehBumdKM0rxcHMZ5uFmhGNHF7/HCwmMvLkUEGl9GslHqzZLT1OC761Ec1m9LynK49TWonPx0Q/mVCYUiPwRemhASipydYchWJfnYVJMwCUTe5XXaTV3L8MwQ1kpy2vOx/x2u/giEqx3Z863byocioDhFkTfU3Ubyo6DEKFFq2h3riuo+bf7pK9G1ckoJgNtXFYQpLEzYcsRtYDlXhDiInQiEWpXSjRgdzzHl8t2NKKHz1eMs9o0Gi76cvLD86toE3QDrmFsE/7HQCq71dfD2CKj6AxI/D0PB2obsxbfMV3zxxMyUm/8rxID1zwD4G+FilYtWAbOa/84F37RWY/7G3KTKBT041g== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 69460d8f-5993-40ec-7087-08dc36d8ecf5 X-MS-Exchange-CrossTenant-AuthSource: AS2PR08MB8431.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2024 14:40:49.7036 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3h6aeAYZsjD1ziGjYCr7IEOI9bpQL4H/L54xj21OEZiodcZfYT2KUm+ppVxI9m8Copt9VGS7+6Gto7eZdfPQ/A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3PR08MB5626 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 26 Feb 2024 14:41:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196211 From: Lukas Funke Apply pylint recommendations where reasanable. Not all recommendations were applied: - long regex expressions were not split for a better readability - not all (short) variables were renamed in order to be consistent with the rest of the framework (e.g. 'd') Signed-off-by: Lukas Funke --- scripts/lib/recipetool/create_go.py | 357 ++++++++++++++++------------ 1 file changed, 205 insertions(+), 152 deletions(-) diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py index cae7175a33..41161b27f5 100644 --- a/scripts/lib/recipetool/create_go.py +++ b/scripts/lib/recipetool/create_go.py @@ -9,16 +9,6 @@ # SPDX-License-Identifier: GPL-2.0-only # - -from collections import namedtuple -from enum import Enum -from html.parser import HTMLParser -from recipetool.create import RecipeHandler, handle_license_vars -from recipetool.create import guess_license, tidy_licenses, fixup_license -from recipetool.create import determine_from_url -from urllib.error import URLError - -import bb.utils import json import logging import os @@ -30,6 +20,17 @@ import tempfile import urllib.parse import urllib.request +from collections import namedtuple +from enum import Enum +from html.parser import HTMLParser +from urllib.error import URLError + +import bb.utils + +from recipetool.create import RecipeHandler, handle_license_vars +from recipetool.create import guess_license, tidy_licenses, fixup_license +from recipetool.create import determine_from_url + GoImport = namedtuple('GoImport', 'root vcs url suffix') logger = logging.getLogger('recipetool') @@ -78,12 +79,14 @@ class GoRecipeHandler(RecipeHandler): if not os.path.exists(gopath): logger.error( - '%s required to process specified source, but %s did not seem to populate it' % 'go', recipe) + '%s required to process specified source, ' + 'but %s did not seem to populate it', 'go', recipe) return None return bindir - def __resolve_repository_static(self, modulepath): + @staticmethod + def __resolve_repository_static(modulepath): """Resolve the repository in a static manner The method is based on the go implementation of @@ -110,83 +113,84 @@ class GoRecipeHandler(RecipeHandler): # contains the subdir and major path. Thus, # we ignore this error for now logger.debug( - 1, "Failed to fetch page from [%s]: %s" % (url, str(url_err))) + "Failed to fetch page from [%s]: %s", url, str(url_err)) host, _, _ = modulepath.partition('/') - class vcs(Enum): - pathprefix = "pathprefix" - regexp = "regexp" - type = "type" - repo = "repo" - check = "check" - schemelessRepo = "schemelessRepo" + class Vcs(Enum): + """Version control system enum""" + PATHPREFIX = "pathprefix" + REGEXP = "regexp" + TYPE = "type" + REPO = "repo" + CHECK = "check" + SCHEMELESS_REPO = "schemelessRepo" # GitHub vcsGitHub = {} - vcsGitHub[vcs.pathprefix] = "github.com" - vcsGitHub[vcs.regexp] = re.compile( + vcsGitHub[Vcs.PATHPREFIX] = "github.com" + vcsGitHub[Vcs.REGEXP] = re.compile( r'^(?Pgithub\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$') - vcsGitHub[vcs.type] = "git" - vcsGitHub[vcs.repo] = "https://\\g" + vcsGitHub[Vcs.TYPE] = "git" + vcsGitHub[Vcs.REPO] = "https://\\g" # Bitbucket vcsBitbucket = {} - vcsBitbucket[vcs.pathprefix] = "bitbucket.org" - vcsBitbucket[vcs.regexp] = re.compile( + vcsBitbucket[Vcs.PATHPREFIX] = "bitbucket.org" + vcsBitbucket[Vcs.REGEXP] = re.compile( r'^(?Pbitbucket\.org/(?P[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+))(/(?P[A-Za-z0-9_.\-]+))*$') - vcsBitbucket[vcs.type] = "git" - vcsBitbucket[vcs.repo] = "https://\\g" + vcsBitbucket[Vcs.TYPE] = "git" + vcsBitbucket[Vcs.REPO] = "https://\\g" # IBM DevOps Services (JazzHub) vcsIBMDevOps = {} - vcsIBMDevOps[vcs.pathprefix] = "hub.jazz.net/git" - vcsIBMDevOps[vcs.regexp] = re.compile( + vcsIBMDevOps[Vcs.PATHPREFIX] = "hub.jazz.net/git" + vcsIBMDevOps[Vcs.REGEXP] = re.compile( r'^(?Phub\.jazz\.net/git/[a-z0-9]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$') - vcsIBMDevOps[vcs.type] = "git" - vcsIBMDevOps[vcs.repo] = "https://\\g" + vcsIBMDevOps[Vcs.TYPE] = "git" + vcsIBMDevOps[Vcs.REPO] = "https://\\g" # Git at Apache vcsApacheGit = {} - vcsApacheGit[vcs.pathprefix] = "git.apache.org" - vcsApacheGit[vcs.regexp] = re.compile( + vcsApacheGit[Vcs.PATHPREFIX] = "git.apache.org" + vcsApacheGit[Vcs.REGEXP] = re.compile( r'^(?Pgit\.apache\.org/[a-z0-9_.\-]+\.git)(/(?P[A-Za-z0-9_.\-]+))*$') - vcsApacheGit[vcs.type] = "git" - vcsApacheGit[vcs.repo] = "https://\\g" + vcsApacheGit[Vcs.TYPE] = "git" + vcsApacheGit[Vcs.REPO] = "https://\\g" # Git at OpenStack vcsOpenStackGit = {} - vcsOpenStackGit[vcs.pathprefix] = "git.openstack.org" - vcsOpenStackGit[vcs.regexp] = re.compile( + vcsOpenStackGit[Vcs.PATHPREFIX] = "git.openstack.org" + vcsOpenStackGit[Vcs.REGEXP] = re.compile( r'^(?Pgit\.openstack\.org/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(\.git)?(/(?P[A-Za-z0-9_.\-]+))*$') - vcsOpenStackGit[vcs.type] = "git" - vcsOpenStackGit[vcs.repo] = "https://\\g" + vcsOpenStackGit[Vcs.TYPE] = "git" + vcsOpenStackGit[Vcs.REPO] = "https://\\g" # chiselapp.com for fossil vcsChiselapp = {} - vcsChiselapp[vcs.pathprefix] = "chiselapp.com" - vcsChiselapp[vcs.regexp] = re.compile( + vcsChiselapp[Vcs.PATHPREFIX] = "chiselapp.com" + vcsChiselapp[Vcs.REGEXP] = re.compile( r'^(?Pchiselapp\.com/user/[A-Za-z0-9]+/repository/[A-Za-z0-9_.\-]+)$') - vcsChiselapp[vcs.type] = "fossil" - vcsChiselapp[vcs.repo] = "https://\\g" + vcsChiselapp[Vcs.TYPE] = "fossil" + vcsChiselapp[Vcs.REPO] = "https://\\g" # General syntax for any server. # Must be last. vcsGeneralServer = {} - vcsGeneralServer[vcs.regexp] = re.compile( + vcsGeneralServer[Vcs.REGEXP] = re.compile( "(?P(?P([a-z0-9.\\-]+\\.)+[a-z0-9.\\-]+(:[0-9]+)?(/~?[A-Za-z0-9_.\\-]+)+?)\\.(?Pbzr|fossil|git|hg|svn))(/~?(?P[A-Za-z0-9_.\\-]+))*$") - vcsGeneralServer[vcs.schemelessRepo] = True + vcsGeneralServer[Vcs.SCHEMELESS_REPO] = True vcsPaths = [vcsGitHub, vcsBitbucket, vcsIBMDevOps, vcsApacheGit, vcsOpenStackGit, vcsChiselapp, vcsGeneralServer] if modulepath.startswith("example.net") or modulepath == "rsc.io": - logger.warning("Suspicious module path %s" % modulepath) + logger.warning("Suspicious module path %s", modulepath) return None if modulepath.startswith("http:") or modulepath.startswith("https:"): - logger.warning("Import path should not start with %s %s" % - ("http", "https")) + logger.warning("Import path should not start with %s %s", + "http", "https") return None rootpath = None @@ -195,24 +199,25 @@ class GoRecipeHandler(RecipeHandler): suffix = None for srv in vcsPaths: - m = srv[vcs.regexp].match(modulepath) - if vcs.pathprefix in srv: - if host == srv[vcs.pathprefix]: + m = srv[Vcs.REGEXP].match(modulepath) + if Vcs.PATHPREFIX in srv: + if host == srv[Vcs.PATHPREFIX]: rootpath = m.group('root') - vcstype = srv[vcs.type] - repourl = m.expand(srv[vcs.repo]) + vcstype = srv[Vcs.TYPE] + repourl = m.expand(srv[Vcs.REPO]) suffix = m.group('suffix') break - elif m and srv[vcs.schemelessRepo]: + elif m and srv[Vcs.SCHEMELESS_REPO]: rootpath = m.group('root') - vcstype = m[vcs.type] - repourl = m[vcs.repo] + vcstype = m[Vcs.TYPE] + repourl = m[Vcs.REPO] suffix = m.group('suffix') break return GoImport(rootpath, vcstype, repourl, suffix) - def __resolve_repository_dynamic(self, modulepath): + @staticmethod + def __resolve_repository_dynamic(modulepath): """Resolve the repository root in a dynamic manner. The method is based on the go implementation of @@ -222,6 +227,7 @@ class GoRecipeHandler(RecipeHandler): url = urllib.parse.urlparse("https://" + modulepath) class GoImportHTMLParser(HTMLParser): + """HTML parser to handle go-import attribute in meta tags""" def __init__(self): super().__init__() @@ -237,15 +243,30 @@ class GoRecipeHandler(RecipeHandler): @property def import_prefix(self): - return self.__srv[0] if len(self.__srv) else None + """Returns the import prefix + + Returns: + string: The import prefix + """ + return self.__srv[0] if len(self.__srv) > 0 else None @property def vcs(self): - return self.__srv[1] if len(self.__srv) else None + """Returns the version control system + + Returns: + string: the repositories version control system + """ + return self.__srv[1] if len(self.__srv) > 1 else None @property def repourl(self): - return self.__srv[2] if len(self.__srv) else None + """Returns the repository URL + + Returns: + string: the repository URL + """ + return self.__srv[2] if len(self.__srv) > 2 else None url = url.geturl() + "?go-get=1" req = urllib.request.Request(url) @@ -303,24 +324,27 @@ class GoRecipeHandler(RecipeHandler): return None - def __resolve_repository(self, modulepath): + @staticmethod + def __resolve_repository(modulepath): """ Resolves src uri from go module-path """ - repodata = self.__resolve_repository_static(modulepath) + repodata = GoRecipeHandler.__resolve_repository_static(modulepath) if not repodata or not repodata.url: - repodata = self.__resolve_repository_dynamic(modulepath) + repodata = GoRecipeHandler.__resolve_repository_dynamic(modulepath) if not repodata or not repodata.url: logger.error( - "Could not resolve repository for module path '%s'" % modulepath) + "Could not resolve repository for module path '%s'", + modulepath) # There is no way to recover from this sys.exit(14) if repodata: - logger.debug(1, "Resolved download path for import '%s' => %s" % ( - modulepath, repodata.url)) + logger.debug("Resolved download path for import '%s' => %s", + modulepath, repodata.url) return repodata - def __split_path_version(self, path): + @staticmethod + def __split_path_version(path): i = len(path) dot = False for j in range(i, 0, -1): @@ -335,45 +359,49 @@ class GoRecipeHandler(RecipeHandler): path) or path[i - 1] != 'v' or path[i - 2] != '/': return path, "", True - prefix, pathMajor = path[:i - 2], path[i - 2:] + prefix, path_major = path[:i - 2], path[i - 2:] if dot or len( - pathMajor) <= 2 or pathMajor[2] == '0' or pathMajor == "/v1": + path_major) <= 2 or path_major[2] == '0' or path_major == "/v1": return path, "", False - return prefix, pathMajor, True + return prefix, path_major, True - def __get_path_major(self, pathMajor): - if not pathMajor: + @staticmethod + def __get_path_major(path_major): + if not path_major: return "" - if pathMajor[0] != '/' and pathMajor[0] != '.': - logger.error( - "pathMajor suffix %s passed to PathMajorPrefix lacks separator", pathMajor) + if path_major[0] != '/' and path_major[0] != '.': + logger.error("path_major suffix %s passed to PathMajorPrefix" + "lacks separator", path_major) - if pathMajor.startswith(".v") and pathMajor.endswith("-unstable"): - pathMajor = pathMajor[:len("-unstable") - 2] + if path_major.startswith(".v") and path_major.endswith("-unstable"): + path_major = path_major[:len("-unstable") - 2] - return pathMajor[1:] + return path_major[1:] - def __build_coderepo(self, repo, path): + @staticmethod + def __build_coderepo(repo, path): codedir = "" - pathprefix, pathMajor, _ = self.__split_path_version(path) + pathprefix, path_major, _ = GoRecipeHandler.__split_path_version(path) if repo.root == path: pathprefix = path elif path.startswith(repo.root): codedir = pathprefix[len(repo.root):].strip('/') - pseudoMajor = self.__get_path_major(pathMajor) + pseudo_major = GoRecipeHandler.__get_path_major(path_major) - logger.debug("root='%s', codedir='%s', prefix='%s', pathMajor='%s', pseudoMajor='%s'", - repo.root, codedir, pathprefix, pathMajor, pseudoMajor) + logger.debug("root='%s', codedir='%s', prefix='%s', " + "pathMajor='%s', pseudoMajor='%s'", + repo.root, codedir, pathprefix, path_major, pseudo_major) return CodeRepo(path, repo.root, codedir, - pathMajor, pathprefix, pseudoMajor) + path_major, pathprefix, pseudo_major) - def __resolve_version(self, repo, path, version): - hash = None - coderoot = self.__build_coderepo(repo, path) + @staticmethod + def __resolve_version(repo, path, version): + _hash = None + coderoot = GoRecipeHandler.__build_coderepo(repo, path) def vcs_fetch_all(): tmpdir = tempfile.mkdtemp() @@ -401,9 +429,9 @@ class GoRecipeHandler(RecipeHandler): if f[1].startswith(prefix): refs[f[1][len(prefix):]] = f[0] - for key, hash in refs.items(): + for key, commithash in refs.items(): if key.endswith(r"^{}"): - refs[key.strip(r"^{}")] = hash + refs[key.strip(r"^{}")] = commithash return refs[tag] @@ -416,31 +444,32 @@ class GoRecipeHandler(RecipeHandler): r = l.split(maxsplit=1) sha1 = r[0] if len(r) else None if not sha1: - logger.error( - "Ups: could not resolve abbref commit for %s" % short_commit) + logger.error("Ups: could not resolve abbref commit for %s", + short_commit) elif sha1.startswith(short_commit): - hash = sha1 + _hash = sha1 break else: m_semver = re_semver.match(version) if m_semver: - def get_sha1_remote(re): + def get_sha1_remote(regex): rsha1 = None for line in remote_refs: # Split lines of the following format: - # 22e90d9b964610628c10f673ca5f85b8c2a2ca9a (tag: sometag) + # 22e90d9b964610628c10f673ca5f85b8c2a2ca9a (tag: xyz) lineparts = line.split(maxsplit=1) sha1 = lineparts[0] if len(lineparts) else None refstring = lineparts[1] if len( lineparts) == 2 else None if refstring: # Normalize tag string and split in case of multiple - # regs e.g. (tag: speech/v1.10.0, tag: orchestration/v1.5.0 ...) + # regs. Example: + # tag: speech/v1.10.0, tag: orchestration/v1.5.0 ..) refs = refstring.strip('(), ').split(',') for ref in refs: - if re.match(ref.strip()): + if regex.match(ref.strip()): rsha1 = sha1 return rsha1 @@ -455,16 +484,16 @@ class GoRecipeHandler(RecipeHandler): # probe tag using 'ls-remote', which is faster than fetching # complete history - hash = vcs_fetch_remote(tag) - if not hash: + _hash = vcs_fetch_remote(tag) + if not _hash: # backup: fetch complete history remote_refs = vcs_fetch_all() - hash = get_sha1_remote( + _hash = get_sha1_remote( re.compile(fr"(tag:|HEAD ->) ({tag})")) logger.debug( - "Resolving commit for tag '%s' -> '%s'", tag, hash) - return hash + "Resolving commit for tag '%s' -> '%s'", tag, _hash) + return _hash def __generate_srcuri_inline_fcn(self, path, version, replaces=None): """Generate SRC_URI functions for go imports""" @@ -505,8 +534,8 @@ class GoRecipeHandler(RecipeHandler): return inline_fcn, commit def __go_handle_dependencies(self, go_mod, srctree, localfilesdir, extravalues, d): + del d - import re src_uris = [] src_revs = [] @@ -518,10 +547,10 @@ class GoRecipeHandler(RecipeHandler): # b) the implementation for the version resolving didn't work # anymore (less bad) if not commithash: - src_rev += f"#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" - src_rev += f"#!!! Could not resolve version !!!\n" - src_rev += f"#!!! Possible supply chain attack !!!\n" - src_rev += f"#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" + src_rev += "#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" + src_rev += "#!!! Could not resolve version !!!\n" + src_rev += "#!!! Possible supply chain attack !!!\n" + src_rev += "#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" src_rev += f"SRCREV_{path.replace('/', '.')} = \"{commithash}\"" return src_rev @@ -537,11 +566,13 @@ class GoRecipeHandler(RecipeHandler): version = replacement['New']['Version'] if os.path.exists(os.path.join(srctree, path)): - # the module refers to the local path, remove it from requirement list - # because it's a local module - go_mod['Require'][:] = [v for v in go_mod['Require'] if v.get('Path') != oldpath] + # the module refers to the local path, remove it + # from requirement list because it's a local module + go_mod['Require'][:] = [v for v in go_mod['Require'] \ + if v.get('Path') != oldpath] else: - # Replace the path and the version, so we don't iterate replacement list anymore + # Replace the path and the version, so we don't iterate + # replacement list anymore for require in go_mod['Require']: if require['Path'] == oldpath: require.update({'Path': path, 'Version': version}) @@ -574,30 +605,37 @@ class GoRecipeHandler(RecipeHandler): extravalues['extrafiles'][go_mods_basename] = go_mods_filename - def __go_run_cmd(self, cmd, cwd, d): + @staticmethod + def __go_run_cmd(cmd, cwd, d): return bb.process.run(cmd, env=dict(os.environ, PATH=d.getVar('PATH')), shell=True, cwd=cwd) - def __go_native_version(self, d): - stdout, _ = self.__go_run_cmd("go version", None, d) - m = re.match(r".*\sgo((\d+).(\d+).(\d+))\s([\w\/]*)", stdout) - major = int(m.group(2)) - minor = int(m.group(3)) - patch = int(m.group(4)) + @staticmethod + def __go_native_version(d): + stdout, _ = GoRecipeHandler.__go_run_cmd("go version", None, d) + match = re.match(r".*\sgo((\d+).(\d+).(\d+))\s([\w\/]*)", stdout) + major = int(match.group(2)) + minor = int(match.group(3)) + patch = int(match.group(4)) return major, minor, patch - def __go_mod_patch(self, srctree, localfilesdir, extravalues, d): + @staticmethod + def __go_mod_patch(srctree, localfilesdir, extravalues, d): patchfilename = "go.mod.patch" - go_native_version_major, go_native_version_minor, _ = self.__go_native_version( - d) - self.__go_run_cmd("go mod tidy -go=%d.%d" % - (go_native_version_major, go_native_version_minor), srctree, d) - stdout, _ = self.__go_run_cmd("go mod edit -json", srctree, d) + go_native_version_major, go_native_version_minor, _ = \ + GoRecipeHandler.__go_native_version(d) + + GoRecipeHandler.__go_run_cmd("go mod tidy -go=" \ + f"{go_native_version_major}." \ + f"{go_native_version_minor}", srctree, d) + + stdout, _ = GoRecipeHandler.__go_run_cmd("go mod edit -json", srctree, d) # Create patch in order to upgrade go version - self.__go_run_cmd("git diff go.mod > %s" % (patchfilename), srctree, d) + GoRecipeHandler.__go_run_cmd(f"git diff go.mod > {patchfilename}", + srctree, d) go_mod = json.loads(stdout) tmpfile = os.path.join(localfilesdir, patchfilename) @@ -607,12 +645,13 @@ class GoRecipeHandler(RecipeHandler): return go_mod, patchfilename - def __go_mod_vendor(self, go_mod, srctree, localfilesdir, extravalues, d): + @staticmethod + def __go_mod_vendor(go_mod, srctree, localfilesdir, extravalues, d): # Perform vendoring to retrieve the correct modules.txt tmp_vendor_dir = tempfile.mkdtemp() # -v causes to go to print modules.txt to stderr - _, stderr = self.__go_run_cmd( + _, stderr = GoRecipeHandler.__go_run_cmd( "go mod vendor -v -o %s" % (tmp_vendor_dir), srctree, d) # ignore miscellaneous go log messages @@ -621,9 +660,9 @@ class GoRecipeHandler(RecipeHandler): modules_txt_basename = "modules.txt" modules_txt_filename = os.path.join(localfilesdir, modules_txt_basename) - with open(modules_txt_filename, "w") as f: - f.write('\n'.join(vedoring_log)) - f.write('\n') # don't forget the trailing newline + with open(modules_txt_filename, "w") as file: + file.write('\n'.join(vedoring_log)) + file.write('\n') # don't forget the trailing newline extravalues['extrafiles'][modules_txt_basename] = modules_txt_filename @@ -634,13 +673,13 @@ class GoRecipeHandler(RecipeHandler): if licvalues: for licvalue in licvalues: - license = licvalue[0] - lics = tidy_licenses(fixup_license(license)) + lics = tidy_licenses(fixup_license(licvalue[0])) lics = [lic for lic in lics if lic not in licenses] - if len(lics): + if len(lics) > 0: licenses.extend(lics) lic_files_chksum.append( - 'file://src/${GO_IMPORT}/vendor/%s;md5=%s' % (licvalue[1], licvalue[2])) + 'file://src/${GO_IMPORT}/vendor/%s;md5=%s' + % (licvalue[1], licvalue[2])) # strip version part from module URL /vXX baseurl = re.sub(r'/v(\d+)$', '', go_mod['Module']['Path']) @@ -648,15 +687,15 @@ class GoRecipeHandler(RecipeHandler): licenses_basename = "%s-licenses.inc" % pn licenses_filename = os.path.join(localfilesdir, licenses_basename) - with open(licenses_filename, "w") as f: - f.write("GO_MOD_LICENSES = \"%s\"\n\n" % + with open(licenses_filename, "w") as file: + file.write("GO_MOD_LICENSES = \"%s\"\n\n" % ' & '.join(sorted(licenses, key=str.casefold))) # We introduce this indirection to make the tests a little easier - f.write("LIC_FILES_CHKSUM += \"${VENDORED_LIC_FILES_CHKSUM}\"\n") - f.write("VENDORED_LIC_FILES_CHKSUM = \"\\\n") + file.write("LIC_FILES_CHKSUM += \"${VENDORED_LIC_FILES_CHKSUM}\"\n") + file.write("VENDORED_LIC_FILES_CHKSUM = \"\\\n") for lic in lic_files_chksum: - f.write(" " + lic + " \\\n") - f.write("\"\n") + file.write(" " + lic + " \\\n") + file.write("\"\n") extravalues['extrafiles'][licenses_basename] = licenses_filename @@ -682,7 +721,6 @@ class GoRecipeHandler(RecipeHandler): stdout, _ = self.__go_run_cmd("go mod edit -json", srctree, d) go_mod = json.loads(stdout) - go_import = go_mod['Module']['Path'] go_version_match = re.match("([0-9]+).([0-9]+)", go_mod['Go']) go_version_major = int(go_version_match.group(1)) go_version_minor = int(go_version_match.group(2)) @@ -702,8 +740,10 @@ class GoRecipeHandler(RecipeHandler): if go_version_major == 1 and go_version_minor < 17: logger.warning( "go.mod files generated by Go < 1.17 might have incomplete indirect dependencies.") - go_mod, patchfilename = self.__go_mod_patch(srctree, localfilesdir, - extravalues, d) + go_mod, patchfilename = GoRecipeHandler.__go_mod_patch(srctree, + localfilesdir, + extravalues, + d) src_uris.append( "file://%s;patchdir=src/${GO_IMPORT}" % (patchfilename)) @@ -712,7 +752,10 @@ class GoRecipeHandler(RecipeHandler): if not os.path.exists(os.path.join(srctree, "vendor")): # Write additional $BPN-modules.inc file - self.__go_mod_vendor(go_mod, srctree, localfilesdir, extravalues, d) + GoRecipeHandler.__go_mod_vendor(go_mod, srctree, + localfilesdir, + extravalues, d) + lines_before.append("LICENSE += \" & ${GO_MOD_LICENSES}\"") lines_before.append("require %s-licenses.inc" % (pn)) @@ -728,6 +771,8 @@ class GoRecipeHandler(RecipeHandler): lines_before.append("GO_IMPORT = \"{}\"".format(baseurl)) lines_before.append("SRCREV_FORMAT = \"${BPN}\"") + return True + def __update_lines_before(self, updated, newlines, lines_before): if updated: del lines_before[:] @@ -741,21 +786,24 @@ class GoRecipeHandler(RecipeHandler): def __rewrite_lic_uri(self, lines_before): def varfunc(varname, origvalue, op, newlines): + del op, newlines if varname == 'LIC_FILES_CHKSUM': new_licenses = [] licenses = origvalue.split('\\') - for license in licenses: - if not license: - logger.warning("No license file was detected for the main module!") + for _license in licenses: + if not _license: + logger.warning("No license file was detected for" + "the main module!") # the license list of the main recipe must be empty # this can happen for example in case of CLOSED license # Fall through to complete recipe generation continue - license = license.strip() - uri, chksum = license.split(';', 1) + uri, chksum = _license.strip().split(';', 1) url = urllib.parse.urlparse(uri) - new_uri = os.path.join( - url.scheme + "://", "src", "${GO_IMPORT}", url.netloc + url.path) + ";" + chksum + new_uri = os.path.join(url.scheme + "://", + "src", "${GO_IMPORT}", + url.netloc + url.path) + new_uri += ";" + chksum new_licenses.append(new_uri) return new_licenses, None, -1, True @@ -768,8 +816,13 @@ class GoRecipeHandler(RecipeHandler): def __rewrite_src_uri(self, lines_before, additional_uris = []): def varfunc(varname, origvalue, op, newlines): + del op, newlines if varname == 'SRC_URI': - src_uri = ["git://${GO_IMPORT};destsuffix=git/src/${GO_IMPORT};nobranch=1;name=${BPN};protocol=https"] + src_uri = [ "git://${GO_IMPORT};" + "destsuffix=git/src/${GO_IMPORT};" + "nobranch=1;name=${BPN};" + "protocol=https"] + src_uri.extend(additional_uris) return src_uri, None, -1, True return origvalue, None, 0, True