From patchwork Mon May 23 14:40:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 8404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40D7FC433F5 for ; Mon, 23 May 2022 14:40:25 +0000 (UTC) Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by mx.groups.io with SMTP id smtpd.web11.28419.1653316820760955472 for ; Mon, 23 May 2022 07:40:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=135unWeg; spf=softfail (domain: sakoman.com, ip: 209.85.216.49, mailfrom: steve@sakoman.com) Received: by mail-pj1-f49.google.com with SMTP id o13-20020a17090a9f8d00b001df3fc52ea7so17706167pjp.3 for ; Mon, 23 May 2022 07:40:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=0ybdaUkp+zU8rlXGmm8RdWiAh1u+E0AmcPmkbC30oQk=; b=135unWegUoTJqGzmrYDtFYWa28gBjS2F5D9+gtgp46OfH38SlLyaJD0Nu1hK70igfz tBMSUeFImdwwop4bNB8++XKsMLcUE4akNUF3IAoyC8SNqDzxOraixwYckVTlqsRraXXW ZxQhJjE2FJq0WhA3Jxgn+MjM0pykJd0gWwrqg/LRtHb2okkPwGn3H7OA+S3SBWI/yKQ4 67aRB/Upv8d5Gc8cKradYy5+ejR5VcGCRKDHS5q9Ytsu600FXmq2IiHJubv8UVOFK3hg MUaL6hGDRY+izPVaoh3Vr8V9c5YxsLSHdVV3OOIm6NVUoXLv7vA6g0PIWcJemXcUSv+6 xH9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0ybdaUkp+zU8rlXGmm8RdWiAh1u+E0AmcPmkbC30oQk=; b=29qf9UM8FCY/+PIuSapmDRXdsQ8JlfNoWhEUH//LO/6VxS0BHeMLeIBBog30cwq9pj CgnxZfHP9VFeoiEnPUMKvlbfInKpPEy4020XqpAhyubew+3qSz4vPoJMhIBsiNJii8iq K0rBRRLReGLX/FVISUOPPpa6YiZtlsZkslL0XUcEH0oQHru2GZKH6/S9sJ5ZUk3R2zN5 cRLiMbjqwSHHuHygGEPTqdXRNsKTdpCYRKOkG/tQTBJRTsqwpD9UB4o1dgRVAcFemgsj Rh/I+TPoK7GunjbUVB8DwMhahFUuLkkMlh/cuJ1uuUuKcc5dum4kBPnc2gnHDifaf0Zs UQCQ== X-Gm-Message-State: AOAM530FX0fonIL1b8HB6dC/VF8oWT0DNuMQREYmOx0ztHAw2KNIDPzt DRNSRxSEwhK46groap8wwF89PaurWdk74hNJ X-Google-Smtp-Source: ABdhPJymFO2zh3/6A0fPP5lrdSOQmfOLBFU+tMroE1dUYqUFqNzyCh8Je1lRLjBTQr+tk9CawBmOsw== X-Received: by 2002:a17:90b:1bc6:b0:1df:cb4b:836e with SMTP id oa6-20020a17090b1bc600b001dfcb4b836emr26172563pjb.117.1653316819529; Mon, 23 May 2022 07:40:19 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id u4-20020a655c04000000b003c14af50623sm4865576pgr.59.2022.05.23.07.40.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 07:40:18 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 1/3] fetch2/osc: Small fixes for osc fetcher Date: Mon, 23 May 2022 04:40:03 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 May 2022 14:40:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13708 From: Gunjan Gupta The current fetcher seemed to have some issues that made it difficult when trying to use the same. This patch fixes the following * Make consistent use of the path that needs to be used as oscdir * The path mentioned in os.access in download function was not same as ud.moddir which would result into invoking of fetch command instead of update command even if directory already existed * Before creating oscrc, make sure oscdir exists and create it if it does not exist * Updated the configuration to use apiurl and added a new parameter to control whether http or https needs to be used to connect to apiurl Signed-off-by: Gunjan Gupta Signed-off-by: Richard Purdie (cherry picked from commit 3ec78686f3c0ea2304097b86a965f9be4b0cb879) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/osc.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/bb/fetch2/osc.py b/lib/bb/fetch2/osc.py index 99a529e5..eb0f82c8 100644 --- a/lib/bb/fetch2/osc.py +++ b/lib/bb/fetch2/osc.py @@ -36,6 +36,7 @@ class Osc(FetchMethod): # Create paths to osc checkouts oscdir = d.getVar("OSCDIR") or (d.getVar("DL_DIR") + "/osc") relpath = self._strip_leading_slashes(ud.path) + ud.oscdir = oscdir ud.pkgdir = os.path.join(oscdir, ud.host) ud.moddir = os.path.join(ud.pkgdir, relpath, ud.module) @@ -49,7 +50,7 @@ class Osc(FetchMethod): else: ud.revision = "" - ud.localfile = d.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.path.replace('/', '.'), ud.revision)) + ud.localfile = d.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), relpath.replace('/', '.'), ud.revision)) def _buildosccommand(self, ud, d, command): """ @@ -86,7 +87,7 @@ class Osc(FetchMethod): logger.debug2("Fetch: checking for module directory '" + ud.moddir + "'") - if os.access(os.path.join(d.getVar('OSCDIR'), ud.path, ud.module), os.R_OK): + if os.access(ud.moddir, os.R_OK): oscupdatecmd = self._buildosccommand(ud, d, "update") logger.info("Update "+ ud.url) # update sources there @@ -114,20 +115,23 @@ class Osc(FetchMethod): Generate a .oscrc to be used for this run. """ - config_path = os.path.join(d.getVar('OSCDIR'), "oscrc") + config_path = os.path.join(ud.oscdir, "oscrc") + if not os.path.exists(ud.oscdir): + bb.utils.mkdirhier(ud.oscdir) + if (os.path.exists(config_path)): os.remove(config_path) f = open(config_path, 'w') + proto = ud.parm.get('proto', 'https') f.write("[general]\n") - f.write("apisrv = %s\n" % ud.host) - f.write("scheme = http\n") + f.write("apiurl = %s://%s\n" % (proto, ud.host)) f.write("su-wrapper = su -c\n") f.write("build-root = %s\n" % d.getVar('WORKDIR')) f.write("urllist = %s\n" % d.getVar("OSCURLLIST")) f.write("extra-pkgs = gzip\n") f.write("\n") - f.write("[%s]\n" % ud.host) + f.write("[%s://%s]\n" % (proto, ud.host)) f.write("user = %s\n" % ud.parm["user"]) f.write("pass = %s\n" % ud.parm["pswd"]) f.close() From patchwork Mon May 23 14:40:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 8402 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 465F0C433EF for ; Mon, 23 May 2022 14:40:25 +0000 (UTC) Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by mx.groups.io with SMTP id smtpd.web08.28371.1653316823615447568 for ; Mon, 23 May 2022 07:40:23 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=oSs1oxnh; spf=softfail (domain: sakoman.com, ip: 209.85.215.169, mailfrom: steve@sakoman.com) Received: by mail-pg1-f169.google.com with SMTP id a38so10994971pgl.9 for ; Mon, 23 May 2022 07:40:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=LosbZIoUSiLbaJOrMmsankthwkLeaud88Ahev/n0YhM=; b=oSs1oxnh1POZ4OPycWQSzInbIW+auaTLpdTab7352hX3ELKVD/2zKKzT7mK5FwHk59 qmdn/fyBjTVwtGIqo2xdcThgcQDQR4cDzaQWSwlDGu/PCfkX3Vvv8Bd0yFf2GbLQaSWt E5dzZcf8P9hISw+JvTzLaAFzia2Gt+T8x+tWU+YhnBgXFm1x6u+eqLYxFgof2K02iXQD 7X5MjOSdme0VZhkcnT/jgVtY3l7Ot2KEuBLJvswxnpAc0KOG2ZqqQkpwT+J+XU7DYErA rqCkPu+gUkniKXTPDyAULbutWE/3uWxVqZppLZctgn7QkECgDOKK/l5DMhUnNr46YHv6 oV0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LosbZIoUSiLbaJOrMmsankthwkLeaud88Ahev/n0YhM=; b=GEHsTynbegdImATREWBdlZgYSS/vFxFDrQx2meSPpXwUWo/KCiQwgFa7i7H8C2YcjL sPISi7AL82MegSQvFYwU95B4ifBoCbSPMU69smNjo8gf1rQ7GeyAfMTQfmscUifLFRpp Z8EOEM3e32TQU4Nr/F9xWYg11t75dSx9v0IacvKQ54z3nJuunOOS55a3DQ+WMWDyms1J XNge2NwJRwnwQrTPqTesoSpzlHUJAf89ZAfd3NMc9Ow7WJIkueM7eyGE5O5/Y7seao05 JJJtv7Eq5R1GfDlH4e60tEVEGKm8KqrvLOdUpWHCjXQkYRv1C0fXnz4BgrJeT8a0avjK KZ4w== X-Gm-Message-State: AOAM532WKqfY8zrFuA0+Xx0fSkO8GII/a61zioKkxl6rhIAPzAtj3+Tv cmR4QjjWbdLPZWYbFYXQqq1aLEzd2oebug50 X-Google-Smtp-Source: ABdhPJwtR4i/JqhSSfW5jkU+jS1ZmaW37O42KQrkwZJsFZmq/4EzT32sk7CpxuauHf2BHY2V3+f0SQ== X-Received: by 2002:a65:52cd:0:b0:3f5:f3fb:6780 with SMTP id z13-20020a6552cd000000b003f5f3fb6780mr21083298pgp.150.1653316822296; Mon, 23 May 2022 07:40:22 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id u4-20020a655c04000000b003c14af50623sm4865576pgr.59.2022.05.23.07.40.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 07:40:21 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 2/3] data: Do not depend on vardepvalueexclude flag Date: Mon, 23 May 2022 04:40:04 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 May 2022 14:40:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13709 From: Tomasz Dziendzielski If SRC_URI contains python function that extends vardepvalueexclude its value is being tracked by sstate-cache, which can lead to rebuilds if value is set dynamically (for example gerrit replicas). Return empty string if vardepvalueexclude is checked to fix this behaviour. Signed-off-by: Tomasz Dziendzielski Signed-off-by: Richard Purdie (cherry picked from commit f5f9a7b89a7d8321f03184e61ad6d5ed8d0f840e) Signed-off-by: Steve Sakoman --- lib/bb/data.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/bb/data.py b/lib/bb/data.py index 62ec2147..c09d9b04 100644 --- a/lib/bb/data.py +++ b/lib/bb/data.py @@ -277,6 +277,8 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, ignored_vars, d): try: if key[-1] == ']': vf = key[:-1].split('[') + if vf[1] == "vardepvalueexclude": + return deps, "" value, parser = d.getVarFlag(vf[0], vf[1], False, retparser=True) deps |= parser.references deps = deps | (keys & parser.execs) From patchwork Mon May 23 14:40:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 8405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EF10C433F5 for ; Mon, 23 May 2022 14:40:35 +0000 (UTC) Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by mx.groups.io with SMTP id smtpd.web11.28421.1653316826454011794 for ; Mon, 23 May 2022 07:40:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=gzt6tq6D; spf=softfail (domain: sakoman.com, ip: 209.85.216.54, mailfrom: steve@sakoman.com) Received: by mail-pj1-f54.google.com with SMTP id ev18so14258119pjb.4 for ; Mon, 23 May 2022 07:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=s8GBFl5L+MwDIirSZdicdf5LuYOTzU5QrXxXTpov5sk=; b=gzt6tq6DIKj9BHquJyvrsse9FkR1gJb9pdEO7P6waULxvOVKcnYv+j+Sem7MwUMdzp CPP+T4iCzUcudsPeOlz7zely8+UmjlBt4HMVznOkZKA35Er5gS9pVXU6ypflV73A299c WEeXuLG331IJkuga4Fj5frJ53XuxwvN12QUjAQU7TMEPog3shfZ5YjXYJ8j/AfUnXY2J +86nQCayknFYbR+xDDVAbm8lcaHuGC8Naw91R9qojw+CDxV+8v1Bmbg7EpTLzfhYW4m5 SoYYfTf/6LEbwLFBJTFeLUBrbdIUDGJJRhjYpU3Oqyc3X0cpx0Yt56IC9webC9Fv20cM XR2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s8GBFl5L+MwDIirSZdicdf5LuYOTzU5QrXxXTpov5sk=; b=CEMOdvXCdfE1S8CiZA/5cCnrbroDsKhu43pHQH55XpFOiaX/bq+lQo4W7Qm9xBZ4jb +r58f2oDooutNT/gzelfpK9gMtSoNyaxmyIrZOf8VRKLyiiqjVAN0nLYK/VtNtD6jlYR HmA1ce2skUu6PBezgqrjvKxZpZtAwPXyFWu5WjXBgVRIpu6PYqOLq9BkfvkiQFPqLCL/ 8bXyo29CAJI5rUP63ag2+0iZxJPmSX4maxe0tZb9bdfV2hVLNCZYWqKQNuEIWqET5QVE 7QUVSGajQAlz0ZulUFG7Hcc6NGsTDOMIZOUY+Q6MaOhDIj51hlj9kg93WEmeDFuEkboJ U1dQ== X-Gm-Message-State: AOAM531m6VlUZJlrqTXNElsh5vvzugXVXHWnuT5+3B4biE/6OyhePLyF j68TiftAJXQtPO1X/lPZ5JcUAYR4J4ZnATI6 X-Google-Smtp-Source: ABdhPJzqrCNaaVOce2JyLzk8eFtg0K4SeygEu17Koyr0Gq3JZMc1LLTL3Y2YI6QlYTo3HG5gIArXoA== X-Received: by 2002:a17:90a:bb10:b0:1e0:383b:de80 with SMTP id u16-20020a17090abb1000b001e0383bde80mr9041796pjr.67.1653316825262; Mon, 23 May 2022 07:40:25 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id u4-20020a655c04000000b003c14af50623sm4865576pgr.59.2022.05.23.07.40.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 07:40:24 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 3/3] build: Add clean_stamp API function to allow removal of task stamps Date: Mon, 23 May 2022 04:40:05 -1000 Message-Id: <4a7e44faaf2c4060ea54491f4dd20c83c538648f.1653314792.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 May 2022 14:40:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13710 From: Richard Purdie We currently have no API to be able to remove all the potential stamps of a task. It is unusual to need to do this, particularly as you could race against other things happening in the system but we do have a use case for this in cleaning up sysroots in OE-Core. The alternative is to mess with CLEANMASK in OE-Core but that is just going to add potential for errors. We need the first part of the make_stamp() function so separate that out so it can be called seperately. Signed-off-by: Richard Purdie (cherry picked from commit 4d671504a25863018ac51c21c005cef0a4d8f05c) Signed-off-by: Steve Sakoman --- lib/bb/build.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/lib/bb/build.py b/lib/bb/build.py index af60c3d8..55f68b98 100644 --- a/lib/bb/build.py +++ b/lib/bb/build.py @@ -835,11 +835,7 @@ def stamp_cleanmask_internal(taskname, d, file_name): return [cleanmask, cleanmask.replace(taskflagname, taskflagname + "_setscene")] -def make_stamp(task, d, file_name = None): - """ - Creates/updates a stamp for a given task - (d can be a data dict or dataCache) - """ +def clean_stamp(task, d, file_name = None): cleanmask = stamp_cleanmask_internal(task, d, file_name) for mask in cleanmask: for name in glob.glob(mask): @@ -850,6 +846,14 @@ def make_stamp(task, d, file_name = None): if name.endswith('.taint'): continue os.unlink(name) + return + +def make_stamp(task, d, file_name = None): + """ + Creates/updates a stamp for a given task + (d can be a data dict or dataCache) + """ + clean_stamp(task, d, file_name) stamp = stamp_internal(task, d, file_name) # Remove the file and recreate to force timestamp