From patchwork Fri Jul 15 11:21:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 10231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E545C43334 for ; Fri, 15 Jul 2022 11:21:35 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web12.5149.1657884085880378143 for ; Fri, 15 Jul 2022 04:21:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=C0j+O0zb; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f54.google.com with SMTP id b26so6335604wrc.2 for ; Fri, 15 Jul 2022 04:21:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=5sh4ArDDLniMpG1XSLpPmVwuscewd49TpH6Ys3gn3b0=; b=C0j+O0zbY6DzQLlr5fepxVJrz2+QVAQm6Wm9g7c2d+Z816OzLWePmBPxTRMDAGN3SI 0OuynkTiJc57PSScuayKJyRy+N7XX7G569LLutMXapLUdPty8acp9tmUnc6WlDDCgZIw 1AN7SP3z4Ix+z//ONyR8QxKaALDAKS9CRBdLI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=5sh4ArDDLniMpG1XSLpPmVwuscewd49TpH6Ys3gn3b0=; b=PZBPxccPlvWC7lPo0VF7hUAzQE6i9ZEZzYKvDw3V0VcvM0d+RMxqnyS593eoZc2Qo2 9wP10BtMuBSRq21h9SMeEIKiznEZNwrG+cfpljrRdq4z7uuEVt5lNb7rmnGxjACPUl1v pURei7/pzUp+VJ1FEoXCumrCWtr6VU4/pY/e9T+dRRI6tXjtLTR//Rr3tC1pg8ixr9lV jD+UxU0y4Unh5zA/UcWAYZDmPOJJXh3bmxiFQ9+T2eR+EqceLGtpDam7CLDmFbjDjysY mVZIHTEVCf+irPWkuu9hcIFvpqxA9ATy185BoV5MHo3nm/t8aoTknqgCK87ScyxYa/vI q7Nw== X-Gm-Message-State: AJIora/awHa5M66ZxDP5SN+tgmPjN9wLn14/y9VheBDm3bNrz4vfjTYZ 1Cy/LDM42gu6Mb30xu+9LdSXHIKIuenGQg== X-Google-Smtp-Source: AGRyM1s00SZIMbeVKTUZAmm1k1RrK9e84y2d4X5clcSsPAx3rZJcq1Y/wpb0jcpWuxn5nJoRjsUe+A== X-Received: by 2002:adf:e40a:0:b0:21d:aa00:a48 with SMTP id g10-20020adfe40a000000b0021daa000a48mr11687561wrm.158.1657884083702; Fri, 15 Jul 2022 04:21:23 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:ef6a:f831:81f1:d696]) by smtp.gmail.com with ESMTPSA id d15-20020adffbcf000000b0020fff0ea0a3sm3628229wrs.116.2022.07.15.04.21.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Jul 2022 04:21:23 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/2] fetch2: Drop DL_DIR fallback for local file fetcher Date: Fri, 15 Jul 2022 12:21:21 +0100 Message-Id: <20220715112122.634528-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 15 Jul 2022 11:21:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13829 A long time ago, we made DL_DIR a final fallback for the local fetcher. Since then we added checksum support and task hashes and the world has changed. There were warnings added some time ago if this fallback triggers and it is now time to drop it entirely. The original use case was for sstate however the sstate code now sets FILESPATH correctly so DL_DIR is no longer needed. There have been a few small bugs exposed by this change, missing mkdir calls and some minor test issues that needed tweaks. In general this simplifies and improves the fetcher code flow though. This completes a cleanup that ensures local files are correctly covered at parse time which ensures rebuilds and reparses happen at the right times. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 16 ++++++---------- lib/bb/fetch2/local.py | 8 -------- lib/bb/tests/fetch.py | 2 ++ 3 files changed, 8 insertions(+), 18 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index cd7dc4f132..3eda9bbe70 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1220,25 +1220,21 @@ def get_checksum_file_list(d): SRC_URI as a space-separated string """ fetch = Fetch([], d, cache = False, localonly = True) - - dl_dir = d.getVar('DL_DIR') filelist = [] for u in fetch.urls: ud = fetch.ud[u] - if ud and isinstance(ud.method, local.Local): + found = False paths = ud.method.localpaths(ud, d) for f in paths: pth = ud.decodedurl - if f.startswith(dl_dir): - # The local fetcher's behaviour is to return a path under DL_DIR if it couldn't find the file anywhere else - if os.path.exists(f): - bb.warn("Getting checksum for %s SRC_URI entry %s: file not found except in DL_DIR" % (d.getVar('PN'), os.path.basename(f))) - else: - bb.fatal(("Unable to get checksum for %s SRC_URI entry %s: file could not be found" + if os.path.exists(f): + found = True + filelist.append(f + ":" + str(os.path.exists(f))) + if not found: + bb.fatal(("Unable to get checksum for %s SRC_URI entry %s: file could not be found" "\nThe following paths were searched:" "\n%s") % (d.getVar('PN'), os.path.basename(f), '\n'.join(paths))) - filelist.append(f + ":" + str(os.path.exists(f))) return " ".join(filelist) diff --git a/lib/bb/fetch2/local.py b/lib/bb/fetch2/local.py index e7d1c8c58f..0bb987c644 100644 --- a/lib/bb/fetch2/local.py +++ b/lib/bb/fetch2/local.py @@ -57,12 +57,6 @@ class Local(FetchMethod): logger.debug2("Searching for %s in paths:\n %s" % (path, "\n ".join(filespath.split(":")))) newpath, hist = bb.utils.which(filespath, path, history=True) searched.extend(hist) - if not os.path.exists(newpath): - dldirfile = os.path.join(d.getVar("DL_DIR"), path) - logger.debug2("Defaulting to %s for %s" % (dldirfile, path)) - bb.utils.mkdirhier(os.path.dirname(dldirfile)) - searched.append(dldirfile) - return searched return searched def need_update(self, ud, d): @@ -78,8 +72,6 @@ class Local(FetchMethod): filespath = d.getVar('FILESPATH') if filespath: locations = filespath.split(":") - locations.append(d.getVar("DL_DIR")) - msg = "Unable to find file " + urldata.url + " anywhere. The paths that were searched were:\n " + "\n ".join(locations) raise FetchError(msg) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 20d7953f35..7fcf57e7ea 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -468,6 +468,7 @@ class MirrorUriTest(FetcherTest): "http://.*/.* file:///someotherpath/downloads/" def test_urireplace(self): + self.d.setVar("FILESPATH", ".") for k, v in self.replaceuris.items(): ud = bb.fetch.FetchData(k[0], self.d) ud.setup_localpath(self.d) @@ -925,6 +926,7 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_fetch_file_mirror_of_mirror(self): + self.d.setVar("FILESPATH", ".") self.d.setVar("MIRRORS", "http://.*/.* file:///some1where/ file:///some1where/.* file://some2where/ file://some2where/.* https://downloads.yoctoproject.org/releases/bitbake") fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz"], self.d) os.mkdir(self.dldir + "/some2where") From patchwork Fri Jul 15 11:21:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 10230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F2F1CCA47C for ; Fri, 15 Jul 2022 11:21:35 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web08.5232.1657884086350652840 for ; Fri, 15 Jul 2022 04:21:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Ll+bJKEs; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.52, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f52.google.com with SMTP id e15so1072561wro.5 for ; Fri, 15 Jul 2022 04:21:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ZhL6h3Z5+w5gQ/KEqCmTAYJtyzaisSZbUi6tSfQEF6M=; b=Ll+bJKEsSB/jIu1RkMHLHrZ/ZeAsRCQ+UJ4t7K9qewr+bt9tvQevuWVBuCsG8EQtoW x/i/3MgE+KgsoX6Fqum3CaTOsBCOwm15c8xSn3VLaeLEB+GKL7EYfy4onD0k+MlsTiUI VWzg3Np0f9IOxhUvTaMI1EW0gnSqEeDzTikFg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZhL6h3Z5+w5gQ/KEqCmTAYJtyzaisSZbUi6tSfQEF6M=; b=ilfWcdeggWH5n0o0X4+rePEzm22l5SwWR9gEE3JEW6Yzr1V69ljvf3b4vnvz1+7klU plxXrGP0rkPMNNnJGBd/QZAyRjxKLvGbscJbe1m3djXyvtaH7buHQT+YUktNUELqXCxi q1Ntg8rjGA7IceqgEWEZWNWWVcc6Cai+ciSwwi8WVbXNbRACMo5KJpulcR0CGQZMqyS/ sN3eqJLokQs3dTJh6KPBfB/ZxfKnKCdW/ICNIk4w3BmDYEYxQqHMY6g2wQN6TTeLfu5u rO05Q3pFFde/OuUEexAXpTxnLLm9nJGvWJR2cOUwE92AfE+uU/N5VBjZtgCb+NI+jMhX W/DA== X-Gm-Message-State: AJIora/pzz62IkmKSW4BuV3eNRzhef3jEVAR7kcQY7pfVNbl8W7FD5i1 NqjPp9MulxGD4BCx8fyor+n9/nmH0p9wfw== X-Google-Smtp-Source: AGRyM1u/IA5Wy5ZftfRftOeVE6vxkEv8Ww/gfefK38A+gMOd7PI4Qlmy52CjbnbrDBEf3pLaC9AsiQ== X-Received: by 2002:a05:6000:249:b0:21d:a952:31f1 with SMTP id m9-20020a056000024900b0021da95231f1mr12581069wrz.411.1657884084478; Fri, 15 Jul 2022 04:21:24 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:ef6a:f831:81f1:d696]) by smtp.gmail.com with ESMTPSA id d15-20020adffbcf000000b0020fff0ea0a3sm3628229wrs.116.2022.07.15.04.21.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Jul 2022 04:21:23 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/2] fetch2: Ensure directory exists before creating symlink Date: Fri, 15 Jul 2022 12:21:22 +0100 Message-Id: <20220715112122.634528-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220715112122.634528-1-richard.purdie@linuxfoundation.org> References: <20220715112122.634528-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 15 Jul 2022 11:21:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13830 If the mirrors code is trying to create a symlink and the parent directory doesn't exist, as might be the case for sstate mirrors where the fetch is into a subdir, it can silently fail. Ensure the directory exists in this case to avoid issues. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 3eda9bbe70..0fb718b23e 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1097,6 +1097,8 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): def ensure_symlink(target, link_name): if not os.path.exists(link_name): + dirname = os.path.dirname(link_name) + bb.utils.mkdirhier(dirname) if os.path.islink(link_name): # Broken symbolic link os.unlink(link_name)