From patchwork Sun Jun 4 01:39:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: belouargamohamed@gmail.com X-Patchwork-Id: 25089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ECFFC7EE2E for ; Sun, 4 Jun 2023 01:39:39 +0000 (UTC) Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by mx.groups.io with SMTP id smtpd.web11.10338.1685842771902408491 for ; Sat, 03 Jun 2023 18:39:32 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@gmail.com header.s=20221208 header.b=clWwhLi/; spf=pass (domain: gmail.com, ip: 209.85.218.46, mailfrom: belouargamohamed@gmail.com) Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-974f4897d87so277668066b.0 for ; Sat, 03 Jun 2023 18:39:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685842770; x=1688434770; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Hdigl3VfZJzfh8aru7Ft70O9S13l+5mgEuv3jx4031A=; b=clWwhLi/egiChJyC516DYzbztsvdQMd2qs4U92qoghCQT+ICOaUJidnG3OOnVRK83E KmAxklwZWg5cJdGC6hJuROzV77dKfhc4A0I/P9MzvdykS2AU0dlrwcVN68JGYmiQcolz jMPTK034EYwO0a6nBZrhvE4N8JGf17OplMYhmrbtQflxj0LlyUaPeOGWQOrPTMAmVPlv 8Xn97FSxNTQX8kM02zujeYx8jW4v4ig4ZH0Mg/wCiy4bVqQgx+oqDXhtQPeBsvxrQmgX 5rRvgVH6fkORfdIrD9B6M2HTb62I+LcFPMArEiLZNLcq/ZtcEuu2nmh7+81ZUkvDmoYM DdgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685842770; x=1688434770; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Hdigl3VfZJzfh8aru7Ft70O9S13l+5mgEuv3jx4031A=; b=kKIpEsWD7QZ/WHLKiDavI5z6uxkGHbgZOwflVqeKoEFOLDmothRRBUJHesAsjSPp88 ObWThXEIN4bpVxKcL8cxmmgZSZ85tekMBKS+uzNDS7eMMAcn5hcMZTMR37WQqI7xGUFI 91N0jmaHWZUcNgW5/8Zx1ma61CnXGJa4QJ3SkqxYWfNzAey//vfDa2/bEbgdGjf5owTl QLxwURjoRKbk2oD4JKS73GI1GY0M8CtiydAZeFz4VSbQX6IP0FD0znu4IWHoEZPPFaL0 TIDKo3jY4XsVoIAkxanzOdXYyXcLU6XajmYCrDE5QZnvwCD0kQLxL041KNIm64x9F2Yu pGOw== X-Gm-Message-State: AC+VfDwy7PEDEyF4mkmmQ1uwPxbeFPTp7mptVsu6uHCx2PSScaomYTAO Mb3gKvbwLYX5nCdc5PpEhC77zgpbLwWsYQ== X-Google-Smtp-Source: ACHHUZ7L8R7wqvXhgoUx5pjKjcJW9XO/OOo0CL8jv5pi5psQwgqtfuC64hkXm0VIXg+gl6gmO68QhQ== X-Received: by 2002:a17:907:97cf:b0:94b:4a4:2836 with SMTP id js15-20020a17090797cf00b0094b04a42836mr2553684ejc.69.1685842769695; Sat, 03 Jun 2023 18:39:29 -0700 (PDT) Received: from localhost.localdomain (2a02-842a-002e-7001-0180-174c-8c1b-346b.rev.sfr.net. [2a02:842a:2e:7001:180:174c:8c1b:346b]) by smtp.gmail.com with ESMTPSA id i8-20020adfdec8000000b0030796e103a1sm5825474wrn.5.2023.06.03.18.39.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 18:39:29 -0700 (PDT) From: belouargamohamed@gmail.com X-Google-Original-From: m.belouarga@technologyandstrategy.com To: bitbake-devel@lists.openembedded.org Cc: BELOUARGA Mohamed Subject: [master][PATCH v2 1/3] bitbake: fetch2: npmsw: Add support for the new format of the shrinkwrap file Date: Sun, 4 Jun 2023 03:39:17 +0200 Message-Id: <20230604013919.13904-1-m.belouarga@technologyandstrategy.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 04 Jun 2023 01:39:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14827 From: BELOUARGA Mohamed Npm is a package manager that has its own manner to handle installation of packages. But it is not yocto friendly, for instance NPM fetch dependencies in the middle of compilation. The shrinkwrap file changed its format over npm versions, but npm does not version this file, so we can use it properly. The actual changes make NPM depencies work with the actual shrinkwrap format. Signed-off-by: BELOUARGA Mohamed --- lib/bb/fetch2/npmsw.py | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/lib/bb/fetch2/npmsw.py b/lib/bb/fetch2/npmsw.py index cc81100b3a..359192a29e 100644 --- a/lib/bb/fetch2/npmsw.py +++ b/lib/bb/fetch2/npmsw.py @@ -41,20 +41,15 @@ def foreach_dependencies(shrinkwrap, callback=None, dev=False): with: name = the package name (string) params = the package parameters (dictionary) - deptree = the package dependency tree (array of strings) + destdir = the destination of the package (string) """ - def _walk_deps(deps, deptree): - for name in deps: - subtree = [*deptree, name] - _walk_deps(deps[name].get("dependencies", {}), subtree) - if callback is not None: - if deps[name].get("dev", False) and not dev: - continue - elif deps[name].get("bundled", False): - continue - callback(name, deps[name], subtree) - - _walk_deps(shrinkwrap.get("dependencies", {}), []) + packages = shrinkwrap.get("packages", {}) + + for package in packages: + if package != "": + name = package.split('node_modules/')[-1] + package_infos = packages.get(package, {}) + callback(name, package_infos, package) class NpmShrinkWrap(FetchMethod): """Class to fetch all package from a shrinkwrap file""" @@ -75,12 +70,10 @@ class NpmShrinkWrap(FetchMethod): # Resolve the dependencies ud.deps = [] - def _resolve_dependency(name, params, deptree): + def _resolve_dependency(name, params, destsuffix): url = None localpath = None extrapaths = [] - destsubdirs = [os.path.join("node_modules", dep) for dep in deptree] - destsuffix = os.path.join(*destsubdirs) unpack = True integrity = params.get("integrity", None) From patchwork Sun Jun 4 01:39:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: belouargamohamed@gmail.com X-Patchwork-Id: 25088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 060B5C77B73 for ; Sun, 4 Jun 2023 01:39:39 +0000 (UTC) Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) by mx.groups.io with SMTP id smtpd.web11.10339.1685842772670879614 for ; Sat, 03 Jun 2023 18:39:32 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@gmail.com header.s=20221208 header.b=bdsBpTV5; spf=pass (domain: gmail.com, ip: 209.85.208.54, mailfrom: belouargamohamed@gmail.com) Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-5149c76f4dbso5352355a12.1 for ; Sat, 03 Jun 2023 18:39:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685842770; x=1688434770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uu8hQ3tPPSIqAhxFvWtThhH8HzKW1gPvkTlj0R0WR1A=; b=bdsBpTV5z9Hp26lK5F7we5rix1bM50GlJpEyQcoLU/FpHKSaHrNtbaIfyx1NY/eX+P AIG0vOfiZ3XYSv7ox8Rd4gm+alRZB39YZDa2+4AkMXq9xPGrG4fGLUQFZg7zUvfxMRRT K0+fmMA5bop+rEnQNb5UYAI1rlavUgd95Cgifw9cFsOGI9HllVC3O+gZrL/4MIHplW2I +Sv7/Jh+3eRVA87qDLhUKZtQOYfju8MEYf0WFeMWs3LhNch2G4okVNm2QM8OfV4CGjsT QaQFuMG1ILvtKTJ+jsvu4jcqq7P+eAzuFECUetR41f7M0y5hHFyF3uIRl+6jsvnYNHNp sqHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685842770; x=1688434770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uu8hQ3tPPSIqAhxFvWtThhH8HzKW1gPvkTlj0R0WR1A=; b=EE5OPdYqFGTeLvaDw6kDJMe4aaxFxOv5igLD8MINl4HQ7ULdmnItnZD93cRqhfQI05 okdyZ89NvfKw0DoCDRi3mq0HK9xnvNnmo8Pt2PliYtbW9xWhk/XIWLLCftocjmJwKqRH qiOQlN1s5MkLEw0pUhXIEx2xpNGwkymMX97BeBAgks0q3uVMYngm02fa1aD3+vdY/8mG gQV9O4pzipVX/oPOFqY8RRSnEJYVfXf5cUGWrzR5QjWGOARyzym3n2xZy3C8zexMHlJG 8m0JveFScfeiLlSMbwWAW8KrwiiHLosWFoT6Ru6fvEkBt7zpc6yByvRpQa1Hr49pSfMN 5Uzw== X-Gm-Message-State: AC+VfDzHlHuszZefB5XC37Hc+Ry9q2cV1kb7QeYoLWShxNt/tvj0eCEy 3yNKObN9oUOxYtpI5B1HDLUVhi2S+fA7YA== X-Google-Smtp-Source: ACHHUZ7udpTa3l5ZGjaUW9VBjgtmV3lzHbnu5qMlRwvYmTLgN5umCLSF0ApEMlb5ma6PCve4gx2bnw== X-Received: by 2002:a17:907:3e1c:b0:94f:8aff:c8b3 with SMTP id hp28-20020a1709073e1c00b0094f8affc8b3mr2949783ejc.28.1685842770328; Sat, 03 Jun 2023 18:39:30 -0700 (PDT) Received: from localhost.localdomain (2a02-842a-002e-7001-0180-174c-8c1b-346b.rev.sfr.net. [2a02:842a:2e:7001:180:174c:8c1b:346b]) by smtp.gmail.com with ESMTPSA id i8-20020adfdec8000000b0030796e103a1sm5825474wrn.5.2023.06.03.18.39.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 18:39:29 -0700 (PDT) From: belouargamohamed@gmail.com X-Google-Original-From: m.belouarga@technologyandstrategy.com To: bitbake-devel@lists.openembedded.org Cc: BELOUARGA Mohamed Subject: [master][PATCH v2 2/3] fetch2: npmsw: Don't fetch dev dependencies when they are not demanded Date: Sun, 4 Jun 2023 03:39:18 +0200 Message-Id: <20230604013919.13904-2-m.belouarga@technologyandstrategy.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230604013919.13904-1-m.belouarga@technologyandstrategy.com> References: <20230604013919.13904-1-m.belouarga@technologyandstrategy.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 04 Jun 2023 01:39:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14828 From: BELOUARGA Mohamed Dev dependencies should not be fetched only if it is specified in the recipe. Signed-off-by: BELOUARGA Mohamed --- lib/bb/fetch2/npmsw.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/bb/fetch2/npmsw.py b/lib/bb/fetch2/npmsw.py index 359192a29e..d89be10caf 100644 --- a/lib/bb/fetch2/npmsw.py +++ b/lib/bb/fetch2/npmsw.py @@ -49,6 +49,8 @@ def foreach_dependencies(shrinkwrap, callback=None, dev=False): if package != "": name = package.split('node_modules/')[-1] package_infos = packages.get(package, {}) + if dev == False and package_infos.get("dev", False): + continue callback(name, package_infos, package) class NpmShrinkWrap(FetchMethod): From patchwork Sun Jun 4 01:39:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: belouargamohamed@gmail.com X-Patchwork-Id: 25090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D3EBC7EE24 for ; Sun, 4 Jun 2023 01:39:39 +0000 (UTC) Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by mx.groups.io with SMTP id smtpd.web10.10226.1685842773693235627 for ; Sat, 03 Jun 2023 18:39:34 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@gmail.com header.s=20221208 header.b=bIThevTr; spf=pass (domain: gmail.com, ip: 209.85.218.42, mailfrom: belouargamohamed@gmail.com) Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-973f78329e3so551627166b.3 for ; Sat, 03 Jun 2023 18:39:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685842772; x=1688434772; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1hC6HqyLjuwyvhLu0LLpZSDlo2wvGtic/KhURRPHCg0=; b=bIThevTrVJaiwVxoqm5XMMVPIIW/KH9VVTJ7yiQHFdN4novtClAifnf+Ketvys0WtQ 4ksSkI9r8uyCJsJL0YzAp7/Zz1fcDbWQS6vlx1vpNScEcGoeBjPuUg1KRw6Oqgk1IGCy NMZZCvOljqJQlWMpIa4PrroL8SR3iYgACQL0h2No1d//iAgFcgk41t+DyQv3L/K74hrr pwZwlIAXBvBaHsyP5SodoUXpcU1c0rufUAfrJO4Z8zC+4BeQRKo/klNwJc/1PKRXA5wV 8iusjJOBAiq7pZInFqMuMM1WWFoGgxxTYzzFXkcgm0h/9GJurCStVE6Sd2+LOaE3OY/A 34zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685842772; x=1688434772; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1hC6HqyLjuwyvhLu0LLpZSDlo2wvGtic/KhURRPHCg0=; b=e66380Gq0WnU60xQqZ5GsCB/t1xkmB3xxb2MuVnjClHlj0p1fGmY/nKEh8tRvlR8Ia sbHrFRb55jG5ETw5gaBs9qXMttYyG0Uj/KgbWg387jpBH6YNEofXcwbd40ZasPPslb2j mbiMhdCVCbf4SJ2vH73bPfqH/xjYvLRIRoYoEIbEKfBQbe7BFLoUkyrPtSHo4pfpr82o LPBi1yeTre68K8ig4PAEY/aXTF9of7SJICEw99jEz2GbMcVVr1m3eOrtAfa6zTC8Kn60 PfwH5mcMvXKk9Z92JjbKTGnLGqgAp8JfRACrTEavCry1UF5/JnWsm/3AmxRFGpcgwjVq BvaA== X-Gm-Message-State: AC+VfDzy8dPsYXZXkItpKsmwj3w9Iaybb3ex2h2zYc8U0LOGyNZZDflQ UbXrEdQsUyc7MliJfxae9Mg9TGfcboKn3g== X-Google-Smtp-Source: ACHHUZ724LjPFyIwOs24r3/RMOxqmYFypaCVsUYHu5As735EZrjs4SwxSYVBV741QxvLGZ+mnNlylw== X-Received: by 2002:a17:907:72c3:b0:96a:928c:d382 with SMTP id du3-20020a17090772c300b0096a928cd382mr2701680ejc.48.1685842771437; Sat, 03 Jun 2023 18:39:31 -0700 (PDT) Received: from localhost.localdomain (2a02-842a-002e-7001-0180-174c-8c1b-346b.rev.sfr.net. [2a02:842a:2e:7001:180:174c:8c1b:346b]) by smtp.gmail.com with ESMTPSA id i8-20020adfdec8000000b0030796e103a1sm5825474wrn.5.2023.06.03.18.39.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 18:39:30 -0700 (PDT) From: belouargamohamed@gmail.com X-Google-Original-From: m.belouarga@technologyandstrategy.com To: bitbake-devel@lists.openembedded.org Cc: BELOUARGA Mohamed Subject: [master][PATCH v2 3/3] fetch2: npm: Remove special caracters that causes recipe tool to fail Date: Sun, 4 Jun 2023 03:39:19 +0200 Message-Id: <20230604013919.13904-3-m.belouarga@technologyandstrategy.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230604013919.13904-1-m.belouarga@technologyandstrategy.com> References: <20230604013919.13904-1-m.belouarga@technologyandstrategy.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 04 Jun 2023 01:39:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14829 From: BELOUARGA Mohamed Packages like @(._.)/execute causes problems because they generate names that are not supported by yocto Signed-off-by: BELOUARGA Mohamed --- lib/bb/fetch2/npm.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py index e6d0598f5d..f83485ad85 100644 --- a/lib/bb/fetch2/npm.py +++ b/lib/bb/fetch2/npm.py @@ -44,9 +44,12 @@ def npm_package(package): """Convert the npm package name to remove unsupported character""" # Scoped package names (with the @) use the same naming convention # as the 'npm pack' command. - if package.startswith("@"): - return re.sub("/", "-", package[1:]) - return package + name = re.sub("/", "-", package) + name = name.lower() + name = re.sub(r"[^\-a-z0-9]", "", name) + name = name.strip("-") + return name + def npm_filename(package, version): """Get the filename of a npm package"""