From patchwork Mon Jul 31 09:44:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?RnLDqWTDqXJpYyBNYXJ0aW5zb25z?= X-Patchwork-Id: 28159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16041C41513 for ; Mon, 31 Jul 2023 09:44:26 +0000 (UTC) Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) by mx.groups.io with SMTP id smtpd.web11.3834.1690796656114806146 for ; Mon, 31 Jul 2023 02:44:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20221208 header.b=Z3BaUR+H; spf=pass (domain: gmail.com, ip: 209.85.167.50, mailfrom: frederic.martinsons@gmail.com) Received: by mail-lf1-f50.google.com with SMTP id 2adb3069b0e04-4fe2d152f62so2090876e87.0 for ; Mon, 31 Jul 2023 02:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690796654; x=1691401454; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8pZ2WX68aDWAS3Kz6YW/Ssi3m9JbUNikX7H6OcXIGKA=; b=Z3BaUR+Hk4Z/xPiZVkAW2Wp0h3QYV8gIWGcz5HoycfHvnlNMxAUja6bqCwv1F1C1CO NMKDfTi/AWHxsa/DDD97vawJlvLevNYSwE+8J0cBPcJA1p/hiKgeCkxbGu9PVF44SBGU BVSYBWlKKAHjJ1qHuiFT3v8kQPfcXSkP2eS4paq8w89l+PYMmKntvcvyxFpZm2kxUJBo lZnshsCKZkR7er0ThXNvmpvG8tCS1qovDYdQ/p1qP9RHAg36O/mXL82HJ+TqE93TyxjN 8KF6C3ZStMWiJYI6NCP0RMn60idtfhlvm6ic167KNhH2HlCWzSkiuZEB1yG4TOZFvcZ+ 6viw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690796654; x=1691401454; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8pZ2WX68aDWAS3Kz6YW/Ssi3m9JbUNikX7H6OcXIGKA=; b=gfhSouWTew4jqxtS40Xhc63FeSk2LC2jX84f+msA/N0rLaBuKbdDRfh8PNBLB93966 GVgyViZhA8iPEQAIQ+/NfD7h6pWtuMxNosolFuFbvDPn6tnagkTTq9CBey64ubTxuHfW I6mGKuK6GCjPH3O8yPIVOWSQCDZXz7U3wL9HNxOuER4AmZlhuKjVEifY1JaUUm0xpWow pXN2jhnWoG/mzirc1OxiD5b8Zc6qaeJBKPN1dHPHsjBdGqV7BNKZEfhkzHmnnimJYkMK nL98YaQ6HdnoFvND3hAfP1IqDd4Jh5Lx9NqdVXT2u1ReNZTyDVGQtcqKKd3NuFtKi1MJ AAHg== X-Gm-Message-State: ABy/qLZBCqM52ehrZ81mjjEeFGpTBs2k6U6lBtFTCk6w1ZfGWDdRqBbm aw+F4YvkrQ5BmlGA/wXLmL9MnVzIXhE= X-Google-Smtp-Source: APBJJlFyjRbQ+MvgnpFM/OOHOMsuSlVHflA9EYpUi9PgBEKugZv4qM/4/DERE0Syh+JYPVmIdoq50g== X-Received: by 2002:a05:6512:3e0a:b0:4f8:5ede:d452 with SMTP id i10-20020a0565123e0a00b004f85eded452mr6614640lfv.55.1690796653693; Mon, 31 Jul 2023 02:44:13 -0700 (PDT) Received: from work-pc.core.sigfox.net ([2a01:e0a:8d5:c6c0:c4e8:ab56:aff:118a]) by smtp.gmail.com with ESMTPSA id i17-20020a5d55d1000000b003143be36d99sm12526610wrw.58.2023.07.31.02.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Jul 2023 02:44:13 -0700 (PDT) From: frederic.martinsons@gmail.com To: openembedded-core@lists.openembedded.org Cc: Randy.MacLeod@windriver.com Subject: [PATCH 1/3] cargo.bbclass: Use --frozen flag for cargo operations Date: Mon, 31 Jul 2023 11:44:08 +0200 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 31 Jul 2023 09:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185129 From: Frederic Martinsons It supersed the --offline flag and guarantee that Cargo.lock file will not be modified during the build. Signed-off-by: Frederic Martinsons --- meta/classes-recipe/cargo.bbclass | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/cargo.bbclass b/meta/classes-recipe/cargo.bbclass index 3ef0bbbb44..8c0b92df8d 100644 --- a/meta/classes-recipe/cargo.bbclass +++ b/meta/classes-recipe/cargo.bbclass @@ -39,7 +39,12 @@ MANIFEST_PATH ??= "${S}/${CARGO_SRC_DIR}/Cargo.toml" RUSTFLAGS ??= "" BUILD_MODE = "${@['--release', ''][d.getVar('DEBUG_BUILD') == '1']}" -CARGO_BUILD_FLAGS = "-v --offline --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}" +# --frozen flag will prevent network access (which is required since only +# the do_fetch step is authorized to access network) +# and will require an up to date Cargo.lock file. +# This force the package being built to already ship a Cargo.lock, in the end +# this is what we want, at least, for reproducibility of the build. +CARGO_BUILD_FLAGS = "-v --frozen --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}" # This is based on the content of CARGO_BUILD_FLAGS and generally will need to # change if CARGO_BUILD_FLAGS changes. From patchwork Mon Jul 31 09:44:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?RnLDqWTDqXJpYyBNYXJ0aW5zb25z?= X-Patchwork-Id: 28158 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18B86C001E0 for ; Mon, 31 Jul 2023 09:44:26 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web11.3836.1690796657025769096 for ; Mon, 31 Jul 2023 02:44:17 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20221208 header.b=ezb1V9ju; spf=pass (domain: gmail.com, ip: 209.85.221.53, mailfrom: frederic.martinsons@gmail.com) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-3090d3e9c92so4484968f8f.2 for ; Mon, 31 Jul 2023 02:44:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690796655; x=1691401455; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Cy3Smm1Q5Af/CEbfiEybD/9tyOLy7rcgJyqCPuqDFZA=; b=ezb1V9ju6cteWSFIt7L5OXHlWF4QIh4wsuzx81QyC5RYJ85VQImQx9nj5lPkrBtZDC sjuL/ANHtURJiLjs6JPuEzkMQG24BFYkFE8hSEng4Ha5azMFhvZons0U3Ol969jmY7h7 68j+Ze8pqRsPDcnfKUFVBW4cxrBfI6jjrC4jrHpM6pZGUFnHkg3EC2LgvPQCFmPv1OAR imaYO9Wq/XbhsmQ0ZQCHcr1wIoptZoVQvRNjueOqgVYSobUntFd0iGwuhGmEEcrDKVd0 +eGO2qFQ7Pf1z6BTEEZXGubTISWELMOd+yrNPSU47W0eg4iN8X51sYPdY+kr20CgZt8R Ocaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690796655; x=1691401455; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cy3Smm1Q5Af/CEbfiEybD/9tyOLy7rcgJyqCPuqDFZA=; b=W1qugEURWHt/ir8WuWJq4f0XMp8IO1VMisR73p7F86QxJhxv5praQTLa5+j/UYFXkB woiLYla5gQv/Bk/jMOgzN8UyjywBn2f7MtarWC6D1JV2CA4EVk6YGC8M/K9PP0rooZob K4GtPo+WvLGDlt8Wddc9ThzCv1xLpvZGpEjRd7S+wGisPr3+ds0ZKsR5EQ1P4yktZKvU TqUfUGnRem/m2jT4maoA5C0Ydp9Lyf40EfSMPfoU5mmS3J0ThUSfs/HQTersAnn0Ezf0 qsiCPNUKby6pDRDTct4kk/ZUieo7byh0TRrnDB3dpOmmQoA6l7ERKL8ApmGgh9fVfWwW HY6A== X-Gm-Message-State: ABy/qLZvhe5qbSBmhGz/Zt+wWqudEo2PIx/h4mWuzD+y5JCcVxUghZcJ z0JudltBaAQguOP83OeD70wiQB7vpdM= X-Google-Smtp-Source: APBJJlFKP4l3rl3RLzc7HuCnprkXkdN5/bn0m7oL9rinWXhlmG1S8E97qPo+XdK3aYme2McUllOjbA== X-Received: by 2002:adf:ff8e:0:b0:317:61af:d64a with SMTP id j14-20020adfff8e000000b0031761afd64amr7688360wrr.3.1690796654834; Mon, 31 Jul 2023 02:44:14 -0700 (PDT) Received: from work-pc.core.sigfox.net ([2a01:e0a:8d5:c6c0:c4e8:ab56:aff:118a]) by smtp.gmail.com with ESMTPSA id i17-20020a5d55d1000000b003143be36d99sm12526610wrw.58.2023.07.31.02.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Jul 2023 02:44:14 -0700 (PDT) From: frederic.martinsons@gmail.com To: openembedded-core@lists.openembedded.org Cc: Randy.MacLeod@windriver.com Subject: [PATCH 2/3] cargo_common.bbclass: Handle Cargo.lock modifications for git dependencies Date: Mon, 31 Jul 2023 11:44:09 +0200 Message-Id: <3553fa6951145c0504e272b0c57eecbf109e7488.1690795930.git.frederic.martinsons@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 31 Jul 2023 09:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185130 From: Frederic Martinsons Now we use --frozen, Cargo.lock cannot be modified by cargo build. These patched git dependencies requires that the git url is removed from Cargo.lock. Fixes #15104 Signed-off-by: Frederic Martinsons --- meta/classes-recipe/cargo_common.bbclass | 40 ++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass index db54826ddb..01afb74640 100644 --- a/meta/classes-recipe/cargo_common.bbclass +++ b/meta/classes-recipe/cargo_common.bbclass @@ -117,6 +117,8 @@ cargo_common_do_configure () { } python cargo_common_do_patch_paths() { + import shutil + cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config") if not os.path.exists(cargo_config): return @@ -146,6 +148,44 @@ python cargo_common_do_patch_paths() { print('\n[patch."%s"]' % k, file=config) for name in v: print(name, file=config) + + if not patches: + return + + # Cargo.lock file is needed for to be sure that artifacts + # downloaded by the fetch steps are those expected by the + # project and that the possible patches are correctly applied. + # Moreover since we do not want any modification + # of this file (for reproducibility purpose), we prevent it by + # using --frozen flag (in CARGO_BUILD_FLAGS) and raise a clear error + # here is better than letting cargo tell (in case the file is missing) + # "Cargo.lock should be modified but --frozen was given" + + manifest_path = d.getVar("MANIFEST_PATH", True) + lockfile = os.path.join(os.path.dirname(manifest_path), "Cargo.lock") + if not os.path.exists(lockfile): + bb.fatal(f"{lockfile} file doesn't exist") + + # There are patched files and so Cargo.lock should be modified but we use + # --frozen so let's handle that modifications here. + # + # Note that a "better" (more elegant ?) would have been to use cargo update for + # patched packages: + # cargo update --offline -p package_1 -p package_2 + # But this is not possible since it requires that cargo local git db + # to be populated and this is not the case as we fetch git repo ourself. + + newlines = [] + with open(lockfile, "r") as f: + for line in f.readlines(): + if not line.startswith("source = \"git"): + newlines.append(line) + + lockfile_mod = lockfile + ".new" + with open(lockfile_mod, "w") as f: + f.writelines(newlines) + + shutil.move(lockfile_mod, lockfile) } do_configure[postfuncs] += "cargo_common_do_patch_paths" From patchwork Mon Jul 31 09:44:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?RnLDqWTDqXJpYyBNYXJ0aW5zb25z?= X-Patchwork-Id: 28160 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15FFEC001DF for ; Mon, 31 Jul 2023 09:44:26 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web10.3767.1690796658331620281 for ; Mon, 31 Jul 2023 02:44:18 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20221208 header.b=oYke7YBs; spf=pass (domain: gmail.com, ip: 209.85.128.42, mailfrom: frederic.martinsons@gmail.com) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-3fc0aecf15bso48330445e9.1 for ; Mon, 31 Jul 2023 02:44:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690796656; x=1691401456; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p+Je3rq+Bg2IulUMK4l4pCVlQTT7qpTbAubHcbfzdwY=; b=oYke7YBso2Olym25K1EdxVW8rlGiJP0bQXNY8Xl4tlCheXj8zzdSgcHfqi+RvMl83t H000QytMFEpTTJDSz+ATe1aaLTDKJZmMhHaFySRoY6lfViAMM2/uJDwohCVO15+gcy9v Y8Lavu9Z8q+A/wAF40ZaohVanDLcGlJrQx+r6/qMtVzc7yfUOuak/ngoMBa80S6q3cwV YCwWxzogZQgldIHNgJWH0Fv4FyJtY7tCoWOrf1Dger+ZIRGeGpEvN17OSR+7DPC+Pq3I VGrcm4rNVK7gg7pHPhKTF28MZalEaYZddZlKdHC7d9RWy9ztmeYrJ9e2Xs7abwbd1c1T mC/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690796656; x=1691401456; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p+Je3rq+Bg2IulUMK4l4pCVlQTT7qpTbAubHcbfzdwY=; b=iCbHs/ppEblIJ/8fLHudYQvv5fPQ2AHkLT6+Td33x3tGMmHpq+PUeFx8ao0tWILP1R beybm9k2cqMWBU/flQSgDys4Qf7WTxeFUCKlbBKESP17NVvZSMZ5o86i7aqpnexZxyxK QL+6caH8zvExm54c5pFARfHW+KMsOccthLhdJdLYnWZ75MrWFsHojHj3T0ZakXM3O4HX ChfbPUP7/0Th6i94ZWAUEIRK58XC820sAY/LDijbdF6cTPZh2OnxA6qqorswJqo3SBYX ANjad2EuTxC06sgH3QcND2GhMSlLnI3W4BPzMUfzkQgg7082D8MHRySlRnMPaqPmozES ryDQ== X-Gm-Message-State: ABy/qLYsOPi7ObJRggOIyApafa8Wm0fhOLwzZ+sEB3XczFn2SHQuGjEx xD/sM9pa6eG+k6Tgl4BhPC6SxUnYgok= X-Google-Smtp-Source: APBJJlEAU5NDA+1yqfY6Cat23D6j/pOTL0x5+9ftIEB0kSH/Y6eeLUtVPtZdLtC7k97KHsjxeAxKpQ== X-Received: by 2002:adf:f644:0:b0:314:362d:6d7b with SMTP id x4-20020adff644000000b00314362d6d7bmr7872306wrp.19.1690796655985; Mon, 31 Jul 2023 02:44:15 -0700 (PDT) Received: from work-pc.core.sigfox.net ([2a01:e0a:8d5:c6c0:c4e8:ab56:aff:118a]) by smtp.gmail.com with ESMTPSA id i17-20020a5d55d1000000b003143be36d99sm12526610wrw.58.2023.07.31.02.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Jul 2023 02:44:15 -0700 (PDT) From: frederic.martinsons@gmail.com To: openembedded-core@lists.openembedded.org Cc: Randy.MacLeod@windriver.com Subject: [PATCH 3/3] cargo.bbclass: Offer a way to use --offline instead of --frozen Date: Mon, 31 Jul 2023 11:44:10 +0200 Message-Id: <7a3e6d5250d6d0cc526dbede582815007a3b3e56.1690795930.git.frederic.martinsons@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 31 Jul 2023 09:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185131 From: Frederic Martinsons And use that for rust-hello-world recipe that did not ship a Cargo.lock file. Signed-off-by: Frederic Martinsons --- meta/classes-recipe/cargo.bbclass | 4 +++- meta/classes-recipe/cargo_common.bbclass | 3 +++ meta/recipes-extended/rust-example/rust-hello-world_git.bb | 2 ++ 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/cargo.bbclass b/meta/classes-recipe/cargo.bbclass index 8c0b92df8d..d8ab94f2b4 100644 --- a/meta/classes-recipe/cargo.bbclass +++ b/meta/classes-recipe/cargo.bbclass @@ -39,12 +39,14 @@ MANIFEST_PATH ??= "${S}/${CARGO_SRC_DIR}/Cargo.toml" RUSTFLAGS ??= "" BUILD_MODE = "${@['--release', ''][d.getVar('DEBUG_BUILD') == '1']}" + # --frozen flag will prevent network access (which is required since only # the do_fetch step is authorized to access network) # and will require an up to date Cargo.lock file. # This force the package being built to already ship a Cargo.lock, in the end # this is what we want, at least, for reproducibility of the build. -CARGO_BUILD_FLAGS = "-v --frozen --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}" +CARGO_EXTRA_FLAGS = "${@['--frozen', '--offline'][d.getVar('CARGO_NO_FROZEN') == '1']}" +CARGO_BUILD_FLAGS = "-v ${CARGO_EXTRA_FLAGS} --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}" # This is based on the content of CARGO_BUILD_FLAGS and generally will need to # change if CARGO_BUILD_FLAGS changes. diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass index 01afb74640..d17501182f 100644 --- a/meta/classes-recipe/cargo_common.bbclass +++ b/meta/classes-recipe/cargo_common.bbclass @@ -152,6 +152,9 @@ python cargo_common_do_patch_paths() { if not patches: return + if d.getVar("CARGO_NO_FROZEN") == 1: + return + # Cargo.lock file is needed for to be sure that artifacts # downloaded by the fetch steps are those expected by the # project and that the possible patches are correctly applied. diff --git a/meta/recipes-extended/rust-example/rust-hello-world_git.bb b/meta/recipes-extended/rust-example/rust-hello-world_git.bb index 1d91109b51..cad184837f 100644 --- a/meta/recipes-extended/rust-example/rust-hello-world_git.bb +++ b/meta/recipes-extended/rust-example/rust-hello-world_git.bb @@ -14,6 +14,8 @@ SUMMARY = "Hello World by Cargo for Rust" HOMEPAGE = "https://github.com/meta-rust/rust-hello-world" LICENSE = "MIT | Apache-2.0" +CARGO_NO_FROZEN = "1" + S = "${WORKDIR}/git" BBCLASSEXTEND = "native"