From patchwork Fri Dec 9 08:58:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 16573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A182C4167B for ; Fri, 9 Dec 2022 08:58:48 +0000 (UTC) Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by mx.groups.io with SMTP id smtpd.web10.8843.1670576326049650113 for ; Fri, 09 Dec 2022 00:58:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=daj3zFwt; spf=pass (domain: gmail.com, ip: 209.85.221.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f51.google.com with SMTP id o5so4526791wrm.1 for ; Fri, 09 Dec 2022 00:58:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=QAurBWsNYHUiDRIfZ6C3ncqydGSa+pcSK8qSMj+wcXQ=; b=daj3zFwtrJ5IjRxDHMEj9xhr2F8ZAm3NB53EZkMdi/wN8b+xTAe1W3mFP9842h+X4D fVDvFiiyeK38dFOMfQu8F5ZDHN/61KRye79h40T9NrM3GopQaEtYRHzCeT/An1hL46Ut OFsZFR7T+XbWQRFiAPHS1iwmtNU0uiZ1lCQ0JfGX0fJItvzO7qiNq1kpntyWgZUtszVo sRTryD/pFdY08NnpQaGW/6zxpmEv3wmZot1A3itUPChiRUlSodXTXqmhhJhOOqzWG4Ae iyQNrc+5a65qc6bZL0T10b99lzbMQQG0mdX2pGIHWn9HqVdJhper1j78KOnVGTUy4CtI y6AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QAurBWsNYHUiDRIfZ6C3ncqydGSa+pcSK8qSMj+wcXQ=; b=jRiJpZeiXVOUXmKeY9TZWiGXyjFdDgkbbbcA4iOThaAnbIahmoy8Pfo3WZqSk6xDuG wDP5QK8ye0H//fejBmBRn06PgxQFoREaB9cCh5HcSK1Wwn4cBOKOMZc/J8r+rIRH6yoV LZcn6NCG2HvwuKrD/nqnh2SjQzcYJWLRLh6b7V6bOK/5urJeOSxuQTCaceMNVs/DlvBC 4YecUP3QwqXe7xXK9ZmVLUrQL6HTFpQ+zQYZlj6Bbjy7hHnhjSrPEEm+eT9rSHsiQbJK rH2Ordqsf5ZH6xF150NeQCESm7xxgnBArMK1GGkvNv7xvfLisO08tlcwz3ZaAAv7Mn4k 0iIg== X-Gm-Message-State: ANoB5pnwiNqXzR07bwrCjh3ooDfA4mFQYxX81f/6gye/mUOtx1lU831h Fk6UCtPrky0+2ibXPttku06AHEoPJL0= X-Google-Smtp-Source: AA0mqf7m+QRQgbVFHQgSu7o2PJfSV862epHlcHjJFK4moRiUtaJ6lElFDpoL1/PSzgrNRR/MrKcUmQ== X-Received: by 2002:a05:6000:1c1d:b0:236:8cca:d3c9 with SMTP id ba29-20020a0560001c1d00b002368ccad3c9mr3640138wrb.53.1670576324226; Fri, 09 Dec 2022 00:58:44 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id n6-20020a5d4006000000b0022cc0a2cbecsm835991wrp.15.2022.12.09.00.58.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Dec 2022 00:58:43 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH v3] fetch2/wget.py: correctly match versioned directories Date: Fri, 9 Dec 2022 09:58:40 +0100 Message-Id: <20221209085840.3733809-1-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 09 Dec 2022 08:58:48 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14158 When obtaining latest upstream versions, the code needs to check if the existing tarball is in a versioned directory (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and if it is, it needs to first obtain the list of all such versioned directories and then check all of them by going one step up in the directory hierarchy. Existing code was returning a correct match when the component name did not have numbers, e.g. a check on 'source/epiphany/43/' would return 43, but was stopping too soon when the component name itself had numbers ('source/libxml2/2.10/' would return libxml2). This change ensures the last match is taken instead of the first. Also, adjust the fetcher tests to check that versioned directories are correctly traversed in this case (e.g. the step to go one level up is taken and a new tarball is discovered in a different versioned directory). Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/fetch2/wget.py | 4 +- .../software/libxml2/2.10/index.html | 20 ++++++++++ .../software/libxml2/2.9/index.html | 40 +++++++++++++++++++ .../software/libxml2/index.html | 19 +++++++++ bitbake/lib/bb/tests/fetch.py | 3 ++ 5 files changed, 84 insertions(+), 2 deletions(-) create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py index 821afa5b58..7ccea3ad89 100644 --- a/bitbake/lib/bb/fetch2/wget.py +++ b/bitbake/lib/bb/fetch2/wget.py @@ -644,10 +644,10 @@ class Wget(FetchMethod): # search for version matches on folders inside the path, like: # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz dirver_regex = re.compile(r"(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") - m = dirver_regex.search(path) + m = dirver_regex.findall(path) if m: pn = d.getVar('PN') - dirver = m.group('dirver') + dirver = m[-1][0] dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn))) if not dirver_pn_regex.search(dirver): diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html new file mode 100644 index 0000000000..4e41af6d6a --- /dev/null +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html @@ -0,0 +1,20 @@ + + +Index of /sources/libxml2/2.10/ +

Index of /sources/libxml2/2.10/

+ + + + + + + + + + + + + + + +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.10.32.5 MiB2022-Oct-14 12:55
libxml2-2.10.0.news7.1 KiB2022-Aug-17 11:55
libxml2-2.10.0.sha256sum174 B2022-Aug-17 11:55
libxml2-2.10.0.tar.xz2.6 MiB2022-Aug-17 11:55
libxml2-2.10.1.news455 B2022-Aug-25 11:33
libxml2-2.10.1.sha256sum174 B2022-Aug-25 11:33
libxml2-2.10.1.tar.xz2.6 MiB2022-Aug-25 11:33
libxml2-2.10.2.news309 B2022-Aug-29 14:56
libxml2-2.10.2.sha256sum174 B2022-Aug-29 14:56
libxml2-2.10.2.tar.xz2.5 MiB2022-Aug-29 14:56
libxml2-2.10.3.news294 B2022-Oct-14 12:55
libxml2-2.10.3.sha256sum174 B2022-Oct-14 12:55
libxml2-2.10.3.tar.xz2.5 MiB2022-Oct-14 12:55
diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html new file mode 100644 index 0000000000..abdfdd0fa2 --- /dev/null +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html @@ -0,0 +1,40 @@ + + +Index of /sources/libxml2/2.9/ +

Index of /sources/libxml2/2.9/

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.9.143.0 MiB2022-May-02 12:03
libxml2-2.9.0.sha256sum87 B2022-Feb-14 18:27
libxml2-2.9.0.tar.xz3.0 MiB2022-Feb-14 18:27
libxml2-2.9.1.sha256sum87 B2022-Feb-14 18:28
libxml2-2.9.1.tar.xz3.0 MiB2022-Feb-14 18:28
libxml2-2.9.10.sha256sum88 B2022-Feb-14 18:42
libxml2-2.9.10.tar.xz3.2 MiB2022-Feb-14 18:42
libxml2-2.9.11.sha256sum88 B2022-Feb-14 18:43
libxml2-2.9.11.tar.xz3.2 MiB2022-Feb-14 18:43
libxml2-2.9.12.sha256sum88 B2022-Feb-14 18:45
libxml2-2.9.12.tar.xz3.2 MiB2022-Feb-14 18:45
libxml2-2.9.13.news26.6 KiB2022-Feb-20 12:42
libxml2-2.9.13.sha256sum174 B2022-Feb-20 12:42
libxml2-2.9.13.tar.xz3.1 MiB2022-Feb-20 12:42
libxml2-2.9.14.news1.0 KiB2022-May-02 12:03
libxml2-2.9.14.sha256sum174 B2022-May-02 12:03
libxml2-2.9.14.tar.xz3.0 MiB2022-May-02 12:03
libxml2-2.9.2.sha256sum87 B2022-Feb-14 18:30
libxml2-2.9.2.tar.xz3.2 MiB2022-Feb-14 18:30
libxml2-2.9.3.sha256sum87 B2022-Feb-14 18:31
libxml2-2.9.3.tar.xz3.2 MiB2022-Feb-14 18:31
libxml2-2.9.4.sha256sum87 B2022-Feb-14 18:33
libxml2-2.9.4.tar.xz2.9 MiB2022-Feb-14 18:33
libxml2-2.9.5.sha256sum87 B2022-Feb-14 18:35
libxml2-2.9.5.tar.xz3.0 MiB2022-Feb-14 18:35
libxml2-2.9.6.sha256sum87 B2022-Feb-14 18:36
libxml2-2.9.6.tar.xz3.0 MiB2022-Feb-14 18:36
libxml2-2.9.7.sha256sum87 B2022-Feb-14 18:37
libxml2-2.9.7.tar.xz3.0 MiB2022-Feb-14 18:37
libxml2-2.9.8.sha256sum87 B2022-Feb-14 18:39
libxml2-2.9.8.tar.xz3.0 MiB2022-Feb-14 18:39
libxml2-2.9.9.sha256sum87 B2022-Feb-14 18:40
libxml2-2.9.9.tar.xz3.0 MiB2022-Feb-14 18:40
diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html new file mode 100644 index 0000000000..c183e06a55 --- /dev/null +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html @@ -0,0 +1,19 @@ + + +Index of /sources/libxml2/ +

Index of /sources/libxml2/

+ + + + + + + + + + + + + + +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
2.0/-2009-Jul-14 13:04
2.1/-2009-Jul-14 13:04
2.10/-2022-Oct-14 12:55
2.2/-2009-Jul-14 13:04
2.3/-2009-Jul-14 13:05
2.4/-2009-Jul-14 13:05
2.5/-2009-Jul-14 13:05
2.6/-2009-Jul-14 13:05
2.7/-2022-Feb-14 18:24
2.8/-2022-Feb-14 18:26
2.9/-2022-May-02 12:04
cache.json22.8 KiB2022-Oct-14 12:55
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index 0af06e46e5..ad3d4dea7d 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -1401,6 +1401,9 @@ class FetchLatestVersionTest(FetcherTest): # http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz ("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz", "", "") : "2.8.12.1", + # https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz + ("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz", "", "") + : "2.10.3", # # packages with versions only in current directory #