From patchwork Tue Feb 27 22:04:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 40188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70E64C54798 for ; Tue, 27 Feb 2024 22:04:41 +0000 (UTC) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mx.groups.io with SMTP id smtpd.web11.2221.1709071478678393391 for ; Tue, 27 Feb 2024 14:04:38 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=N5henHbg; spf=softfail (domain: sakoman.com, ip: 209.85.210.182, mailfrom: steve@sakoman.com) Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6e4d869b019so2463465b3a.0 for ; Tue, 27 Feb 2024 14:04:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1709071478; x=1709676278; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=QVxlT+L7ovL/yzsh4GRyeN/puMEGuFuZPt8K2Fzba80=; b=N5henHbgcdFR7Yzoh4NmQsYLxRch1DS4mNJzODMDLxsX9hHp6gmrdnri/Q1ov3fSqi wXebxdw5W4BXKY01zqdkfiPtvbG4pYzRu1Jow6tW3/RrbWgrYPqX+5FWHZpEQaSVHqjR BchyYPVuGJDBXBvlhY9QaEH0uWue624iYrn5BNUz3J07JXLd1b3YTMyKJbS1uwRY+ng5 VgVbx6BRAUplgy3Datmo1rLhCttAqvusWT6M7e58FnH+gMNt4bq5zMM1z5+QahUl/dTa /wJQYn3I90bVEwpU2joAjxAmLx3X1MCMoha0sUrH3wRdjAFBoz6ZMe+IQLyP54EJqZ6p gJHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709071478; x=1709676278; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QVxlT+L7ovL/yzsh4GRyeN/puMEGuFuZPt8K2Fzba80=; b=YrYkBauSDvqm9CXl9odfZBvW5X1FbMxpm01CUiNhJMc94wQt851CprkqoxlIvrCBt1 dD8GfoXIDdp8dwdX8EkjylhwKCyUNin1MFnrzMLHY2Aj36ZznE4F7J1pHFRciAFyXq8A qzIZed2TGOZW1i8pMI2pAMkL8Ewje3u2KwJrsQBeMTE82AOW8S0Et/Al+E4HVFhvb93B dv8tQkdK+sRvi6+dwwTETj6sK1o7PuIHKh/mM60JQ5zWKytYpJtV0JTjOfDWQDaqIQIn CRK6sgvx9gwnvMOWtI8J5ryKkqFZhX3et32amLmlhiV1NxypWZ0sE9olbltF8QB0FWd4 H2ew== X-Gm-Message-State: AOJu0Yx0unF2rWkBIBv1cwL3MPkZX33EhHLTDR4MhkJbk+8Wd5Vi7DjR QnNL3FvFyD2CiQz/dLMcn5aiAk37AnNn4zHITrBiJ9bUO8AGWDN1XfumKPjd55BYitZJI9XQh61 I6mo= X-Google-Smtp-Source: AGHT+IEE/5lqijlcH5hz9e0OV2OjR/m/idd6K5Wza1R/7dn9ZrMkUDS7r3RV/NDCgcCqLi3pM8eU3A== X-Received: by 2002:a62:ce0b:0:b0:6e5:58f1:cb35 with SMTP id y11-20020a62ce0b000000b006e558f1cb35mr1122449pfg.9.1709071478003; Tue, 27 Feb 2024 14:04:38 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-234-108-41.hawaiiantel.net. [72.234.108.41]) by smtp.gmail.com with ESMTPSA id t13-20020a056a0021cd00b006e50c0d6421sm5059772pfj.11.2024.02.27.14.04.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 14:04:37 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 1/6] codeparser: replace deprecated ast.Str and 's' Date: Tue, 27 Feb 2024 12:04:20 -1000 Message-Id: <4591011449212c8e494ea42348acb2d27a82a51b.1709071317.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 27 Feb 2024 22:04:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15967 From: Chris Laplante These have been deprecated since 3.8 Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman --- lib/bb/codeparser.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py index 9d66d3ae4..bb890dddd 100644 --- a/lib/bb/codeparser.py +++ b/lib/bb/codeparser.py @@ -225,8 +225,8 @@ class PythonParser(): def visit_Call(self, node): name = self.called_node_name(node.func) if name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs): - if isinstance(node.args[0], ast.Str): - varname = node.args[0].s + if isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str): + varname = node.args[0].value if name in self.containsfuncs and isinstance(node.args[1], ast.Str): if varname not in self.contains: self.contains[varname] = set() From patchwork Tue Feb 27 22:04:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 40190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D9E9C5478C for ; Tue, 27 Feb 2024 22:04:41 +0000 (UTC) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mx.groups.io with SMTP id smtpd.web10.2140.1709071480453462217 for ; Tue, 27 Feb 2024 14:04:40 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=FFrAUP50; spf=softfail (domain: sakoman.com, ip: 209.85.210.182, mailfrom: steve@sakoman.com) Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6e4e7e2594cso2719790b3a.2 for ; Tue, 27 Feb 2024 14:04:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1709071480; x=1709676280; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=rKIvVPMI1PYCZdh2qsONOH7vYiuZFtGOzCSLzCBz7ws=; b=FFrAUP50JPvPgBGJVJ87nVrOMGml8yyjwR3hrK45kKnPO0Bm7jKrx2OS34cOBJcnzy 5X+rGoL88kaWSP43bU8s02nCGrFO/pjfCOzyhO8aEnFeT+pomnQ5h7MmIGtuxCG1FGYR oVtM7ABRn6vHEmdt3lusVBSr6cQSWC1YdxcPLWzCi7H1MU9uB3rCTlkvH12DJmI9EYdp Nc26zG8jO63U//YSHceI6W9oMa5DUiifIPSoytVBkgriJIi5ApNb6dJ//5zRA2JkzWmH Qid+ktru2epI5C569WMWgylylZRYy6lbG01WdqSQ2saFYb0/9Cbx6CikRXnSt85I1jjI WDbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709071480; x=1709676280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rKIvVPMI1PYCZdh2qsONOH7vYiuZFtGOzCSLzCBz7ws=; b=VKBrxItUsQ3f00bD/0jWHpd4R8chFQ8LqaH83PPfdCyc2Mvutpqbl3TczKX1NDzdo2 +4kJrMA6lO7xhE2YyHH4v0Cbt7UmDwy+gveGO1TSw36d5lY2vVM00V3VpuAAMUpnZyzI HQ3yzTFn23q08XTTsLbMLavdun3tO+A0fq3d37mJHDZ45y823KeJFpNm6ZtVs5n0mo4J CX7lsn1AP1vxsnFYCSwt5KdoY0e5iMbv+mMFcnJwpGeLmJ0FVqqkyJGuwSaYMueQDTES 9ux7WPKok0jXzT4KpQKb4W108PS+DPf3VwWua9dEMeZoPhWfiv74Pny37CptKtVhL2aW WKlA== X-Gm-Message-State: AOJu0Yxa91UEF4H5vz0TKxGXFXnSFToFpC2NTqE0I7VGEp8+hB1lTuy1 vzGQZpzqro3IWuoNbAeJ1TN9ttotVvegHqBx6dkUpbneD5jHL7PWpwGNiQw20dr29W27X4hW3+D BHUk= X-Google-Smtp-Source: AGHT+IEUvheXQnX/mqpgEGQ7FPodfc4t3eRB4cjIxSgcPKP+qLL2o3yf7Ew3s6T3jWQIYXecLS9DRw== X-Received: by 2002:aa7:98c2:0:b0:6e3:6c9f:599e with SMTP id e2-20020aa798c2000000b006e36c9f599emr10746300pfm.22.1709071479803; Tue, 27 Feb 2024 14:04:39 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-234-108-41.hawaiiantel.net. [72.234.108.41]) by smtp.gmail.com with ESMTPSA id t13-20020a056a0021cd00b006e50c0d6421sm5059772pfj.11.2024.02.27.14.04.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 14:04:39 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 2/6] bitbake/codeparser.py: address ast module deprecations in py 3.12 Date: Tue, 27 Feb 2024 12:04:21 -1000 Message-Id: <6c19b6cf105ac321ec89da1a876a317020c45ab7.1709071317.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 27 Feb 2024 22:04:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15968 From: Alexander Kanavin Specifically: /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead if isinstance(node.args[0], ast.Str): /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead value = node.args[0].s Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman --- lib/bb/codeparser.py | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py index bb890dddd..6ce0c5182 100644 --- a/lib/bb/codeparser.py +++ b/lib/bb/codeparser.py @@ -227,17 +227,17 @@ class PythonParser(): if name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs): if isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str): varname = node.args[0].value - if name in self.containsfuncs and isinstance(node.args[1], ast.Str): + if name in self.containsfuncs and isinstance(node.args[1], ast.Constant): if varname not in self.contains: self.contains[varname] = set() - self.contains[varname].add(node.args[1].s) - elif name in self.containsanyfuncs and isinstance(node.args[1], ast.Str): + self.contains[varname].add(node.args[1].value) + elif name in self.containsanyfuncs and isinstance(node.args[1], ast.Constant): if varname not in self.contains: self.contains[varname] = set() - self.contains[varname].update(node.args[1].s.split()) + self.contains[varname].update(node.args[1].value.split()) elif name.endswith(self.getvarflags): - if isinstance(node.args[1], ast.Str): - self.references.add('%s[%s]' % (varname, node.args[1].s)) + if isinstance(node.args[1], ast.Constant): + self.references.add('%s[%s]' % (varname, node.args[1].value)) else: self.warn(node.func, node.args[1]) else: @@ -245,8 +245,8 @@ class PythonParser(): else: self.warn(node.func, node.args[0]) elif name and name.endswith(".expand"): - if isinstance(node.args[0], ast.Str): - value = node.args[0].s + if isinstance(node.args[0], ast.Constant): + value = node.args[0].value d = bb.data.init() parser = d.expandWithRefs(value, self.name) self.references |= parser.references @@ -256,8 +256,8 @@ class PythonParser(): self.contains[varname] = set() self.contains[varname] |= parser.contains[varname] elif name in self.execfuncs: - if isinstance(node.args[0], ast.Str): - self.var_execs.add(node.args[0].s) + if isinstance(node.args[0], ast.Constant): + self.var_execs.add(node.args[0].value) else: self.warn(node.func, node.args[0]) elif name and isinstance(node.func, (ast.Name, ast.Attribute)): From patchwork Tue Feb 27 22:04:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 40191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BCE5C54E4A for ; Tue, 27 Feb 2024 22:04:51 +0000 (UTC) Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by mx.groups.io with SMTP id smtpd.web10.2141.1709071482195877941 for ; Tue, 27 Feb 2024 14:04:42 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=KK2r1IjN; spf=softfail (domain: sakoman.com, ip: 209.85.210.169, mailfrom: steve@sakoman.com) Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6e558a67f70so458474b3a.0 for ; Tue, 27 Feb 2024 14:04:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1709071481; x=1709676281; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=xTtnClzude1w+mRbVlZD25E2otjtUn7s20Uas2oq4pk=; b=KK2r1IjNv4DSHnML+n67qOLoJtE4Y0rIeLXizIOw6O0aolwIhQ4Xc6I+kWkNRkFn1e qqRJVcCoRri/UtC52h6QwJ9PPUcZZeIWMa7vdMnwWkFtIptyAbP6ulkaxtWAa/+hligf +5zY7uSN4peAjy1qTlwLR0wHd+HMmWndVQGA4EU1yQC9rgpxpJviVLfqVKRXJSpgO4p0 XEiXE6ZjuI1gKsZYo2dDawJcLMgtij1r+kmpB44lZ2zPndJiCxsM6fkiuHZO1fEjcFZo h4krxq+CmJOfaYYkmbuZGu6F2uslZPe/UxoT3VHPVsQhWPdy3ItO9Ck+SqrEfn0Y/D4i LNuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709071481; x=1709676281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xTtnClzude1w+mRbVlZD25E2otjtUn7s20Uas2oq4pk=; b=K6LPjGqUB0P0O5+MeIP21LYnUl+S6oexbBHCdzT9ehwAgeczCAQhCCm8cuvNxYsNKr jY66FP4sV8yUH1V8qViP6Q8VouuT8DEh6Aid1mWADvp3qAdpYS//JwgU8wL+y2IauEPP 0Kq+RWby8tpZCOQ9lYgdV2+Kr7WTtZXItmSnRrJhYYWTrLyGlLzwYg4OPsdCQRu3HQmr I2fiEcCLRGuvV8JhCVKvb5DLtBDZkjhiy0MgFC82S9aBZKaPeBWiGUSHOXALo2ioOPPu 7eDmIHXj/mTiIoNjCwdsfK1z6s53oYlrV4XUnj4T/C1dRYus73eYT4cZK1M0OaMvL/HG jrBQ== X-Gm-Message-State: AOJu0Yz2ENSPAnmI5lz3QmI5tswqOFNZF+OvutUL6PDJvHxEr1GzzRXm ZK1xa3/p25e7/Kx12/CCxO4vHxXk+FeFlrqa/DzwMSvMz6FGN1gnuDndQyqEy+ntDgVyNc6CWbo wfqg= X-Google-Smtp-Source: AGHT+IHMXB4pwDaU/CtEFtUiraMmnkSzHf5B6HZxc0EBeDsBKDLuzWIkB9V/7b4O73YQan3wCOh70A== X-Received: by 2002:a62:5ec7:0:b0:6e4:aca:ce0d with SMTP id s190-20020a625ec7000000b006e40acace0dmr10060260pfb.31.1709071481472; Tue, 27 Feb 2024 14:04:41 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-234-108-41.hawaiiantel.net. [72.234.108.41]) by smtp.gmail.com with ESMTPSA id t13-20020a056a0021cd00b006e50c0d6421sm5059772pfj.11.2024.02.27.14.04.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 14:04:41 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 3/6] bitbake/lib/bs4/tests/test_tree.py: python 3.12 regex Date: Tue, 27 Feb 2024 12:04:22 -1000 Message-Id: <7efed7bc09c56e41d3074a26388581a62f145625.1709071317.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 27 Feb 2024 22:04:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15969 From: Adrian Freihofer Python 3 interprets string literals as Unicode strings, and therefore \s is treated as an escaped Unicode character which is not correct. Declaring the RegEx pattern as a raw string instead of unicode is required for Python 3. Signed-off-by: Adrian Freihofer Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman --- lib/bs4/tests/test_tree.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bs4/tests/test_tree.py b/lib/bs4/tests/test_tree.py index 8e5c66426..cf0f1abe0 100644 --- a/lib/bs4/tests/test_tree.py +++ b/lib/bs4/tests/test_tree.py @@ -585,7 +585,7 @@ class SiblingTest(TreeTest): ''' # All that whitespace looks good but makes the tests more # difficult. Get rid of it. - markup = re.compile("\n\s*").sub("", markup) + markup = re.compile(r"\n\s*").sub("", markup) self.tree = self.soup(markup) From patchwork Tue Feb 27 22:04:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 40194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DF4AC5478C for ; Tue, 27 Feb 2024 22:04:51 +0000 (UTC) Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) by mx.groups.io with SMTP id smtpd.web10.2143.1709071485648114572 for ; Tue, 27 Feb 2024 14:04:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=lt2RoKVz; spf=softfail (domain: sakoman.com, ip: 209.85.167.173, mailfrom: steve@sakoman.com) Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3bbc649c275so2438080b6e.0 for ; Tue, 27 Feb 2024 14:04:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1709071485; x=1709676285; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=skqpA8dwO4nqyqGt21HDGobZuOo8BOSXp1xQG2u+Xkc=; b=lt2RoKVzsje+h1zqcB++LNo9mIty0nxXKwctJmm2sudig+/7V1D4vvR0sZqmZBDTnN acDm2mr4ecgyOWyqNe+QiD5EbfE5431eWLGQ4nyocJ/ezC23B2/iaiAHToxoH7fWjySl nLvztmclePa/Kb4EhMp+XHwJFRjHCG6GmNJYxnMSt6eYqhG/rMQlo/oAtI8dNRxrQ578 M5S+2Vly58rsxWRK8uKUkdQIq+o7EjoHbkAJ4/AMOAxnhA4e/HC9DyRCa5RJzjqUDffM tfs/JpWH364QdTfh1ug0QyzQ1IRBg3bxuarJyfPfYpSWjTs2Pim60KBGlhV7lj8pdstD VF1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709071485; x=1709676285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=skqpA8dwO4nqyqGt21HDGobZuOo8BOSXp1xQG2u+Xkc=; b=E31JS0T7FzWQ8HTI9v0xbxAayuh8wttLQRo0ya0lmuwxU1q10M2kmCMduVqtyq+SEl oj3afCSt8fG5DX5IEZ1fmJKuDbWC2ehCi6xpcX7X/J8XmJy4zPsTiFmjVWfyU6KZfbYk 2Ode//YUxQc1Y24xi6FPfoZEPhDKvCA5NOYGFqc3NPVYhqfyEln8QkTyd0xckHoPzR9J NkAYqaa2tABBdyfF95YQ6ajKcZHN0xE/vZ0esTHqEVOuiJAqO20HVfckguWzOOj4Slo/ yTKj4bl4DPdlAwAxuZxydVN/4n669vPWrECBjyGmIJ42A7WqSZrTHc9rDWoUZ0P+Rr7m hrcw== X-Gm-Message-State: AOJu0YxtICezay71zshn1+l8kAR2olsgUYf9BoSbzNu89lCMpXzE22qD PkWA0ALzj9CaQNNleeH3yMjOjxJ/V6NehsVF2hBJqm916jQ50kLc88+gNmQsn19PiFmsfQcqtZC +m3E= X-Google-Smtp-Source: AGHT+IFChllsgextrgxBmXDijMNwgLYimcsNfp6RiITY5AOtchRRHoFK98K1F8/pKVwVCyVjqrATKQ== X-Received: by 2002:a05:6808:14d1:b0:3c1:9333:9bf4 with SMTP id f17-20020a05680814d100b003c193339bf4mr3965038oiw.6.1709071483242; Tue, 27 Feb 2024 14:04:43 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-234-108-41.hawaiiantel.net. [72.234.108.41]) by smtp.gmail.com with ESMTPSA id t13-20020a056a0021cd00b006e50c0d6421sm5059772pfj.11.2024.02.27.14.04.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 14:04:42 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 4/6] tests/fetch: git-lfs restore _find_git_lfs Date: Tue, 27 Feb 2024 12:04:23 -1000 Message-Id: <1d5d4a46c772f44ab652f95b37f508b890828e67.1709071317.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 27 Feb 2024 22:04:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15970 From: Paulo Neves Not restoring the mocked _find_git_lfs leads to other tests failing. (cherry picked from commit 70f848631450bd723c223227c21c60e815ee033d) Signed-off-by: Paulo Neves Signed-off-by: Richard Purdie Signed-off-by: Philip Lorenz Signed-off-by: Steve Sakoman --- lib/bb/tests/fetch.py | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 7bace415d..743d3e3ff 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2210,12 +2210,16 @@ class GitLfsTest(FetcherTest): shutil.rmtree(self.gitdir, ignore_errors=True) fetcher.unpack(self.d.getVar('WORKDIR')) - # If git-lfs cannot be found, the unpack should throw an error - with self.assertRaises(bb.fetch2.FetchError): - fetcher.download() - ud.method._find_git_lfs = lambda d: False - shutil.rmtree(self.gitdir, ignore_errors=True) - fetcher.unpack(self.d.getVar('WORKDIR')) + old_find_git_lfs = ud.method._find_git_lfs + try: + # If git-lfs cannot be found, the unpack should throw an error + with self.assertRaises(bb.fetch2.FetchError): + fetcher.download() + ud.method._find_git_lfs = lambda d: False + shutil.rmtree(self.gitdir, ignore_errors=True) + fetcher.unpack(self.d.getVar('WORKDIR')) + finally: + ud.method._find_git_lfs = old_find_git_lfs def test_lfs_disabled(self): import shutil @@ -2230,17 +2234,21 @@ class GitLfsTest(FetcherTest): fetcher, ud = self.fetch() self.assertIsNotNone(ud.method._find_git_lfs) - # If git-lfs can be found, the unpack should be successful. A - # live copy of git-lfs is not required for this case, so - # unconditionally forge its presence. - ud.method._find_git_lfs = lambda d: True - shutil.rmtree(self.gitdir, ignore_errors=True) - fetcher.unpack(self.d.getVar('WORKDIR')) + old_find_git_lfs = ud.method._find_git_lfs + try: + # If git-lfs can be found, the unpack should be successful. A + # live copy of git-lfs is not required for this case, so + # unconditionally forge its presence. + ud.method._find_git_lfs = lambda d: True + shutil.rmtree(self.gitdir, ignore_errors=True) + fetcher.unpack(self.d.getVar('WORKDIR')) + # If git-lfs cannot be found, the unpack should be successful - # If git-lfs cannot be found, the unpack should be successful - ud.method._find_git_lfs = lambda d: False - shutil.rmtree(self.gitdir, ignore_errors=True) - fetcher.unpack(self.d.getVar('WORKDIR')) + ud.method._find_git_lfs = lambda d: False + shutil.rmtree(self.gitdir, ignore_errors=True) + fetcher.unpack(self.d.getVar('WORKDIR')) + finally: + ud.method._find_git_lfs = old_find_git_lfs class GitURLWithSpacesTest(FetcherTest): test_git_urls = { From patchwork Tue Feb 27 22:04:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 40193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B7FDC54E51 for ; Tue, 27 Feb 2024 22:04:51 +0000 (UTC) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mx.groups.io with SMTP id smtpd.web11.2224.1709071485817156863 for ; Tue, 27 Feb 2024 14:04:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=IPfSAv8m; spf=softfail (domain: sakoman.com, ip: 209.85.210.173, mailfrom: steve@sakoman.com) Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6e4c359e48aso3100632b3a.1 for ; Tue, 27 Feb 2024 14:04:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1709071485; x=1709676285; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=RqKOw1uhQzbEcZlLZczArygnCwYvkY6zV5GdIcqo6do=; b=IPfSAv8mlsDz5T5Go7Grr8ViQql1SXQcEspJ5rrGSWX7B1H5HjWeYX8OUxZdc1sWBg EiBYLaF2qLtJMJQzv1Ar01s9KPylYO5Uo5bGPbI3X+S44MbkHw4dCCAei+OCB1ZA8l/8 YalxopZLZCqrpUeLt2VlxQSZNiAkc3YP1R5DzTb2IslrYmMP+C7HegYUzj2C+pCP1zXx R78B1+tHeOvHgFQaHSGvpTpRJ94ZngIb0ps1Z760kHV54JG2Ef+HCoHxAEbOQalE/LqF rXv/pP37qDLPmA0HbDzV5xwCL/cUrK9C+eARk0y4JlL+6ptH/3mIvowQWX6/tMmVnkvo QhWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709071485; x=1709676285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RqKOw1uhQzbEcZlLZczArygnCwYvkY6zV5GdIcqo6do=; b=SjMxxC1RxmQRUaOsshi9izeQZ2nJTFUFFxEQHp+S4lBwDcKroDNJdlPTdRXNsqz2iq f/jsib8Sb5yQNOcQeS7+KG1J/KFsjgTjEhYncHkVSPb0oBnPPVT1nxs2ePc31gJlfwCF hQT2WyaixbGvAOZuVBXOsRsrIBy4siqiIPc56uHODi9cLx4XwvbCdZwTUpNQcD/iM6uR 67itvr8slBcLgCa7x7Z0hT5Fg2OwHUR+Cdk1BhmWawDAUn+JaviZII2cBB4X8/xgYNgj JJh98pfyG2naz0TGoKjVF2U9sq7MRsU9miNFVAvxhbZxBMvnmA4jqJGHPSgLlMOOsIZK Kzgw== X-Gm-Message-State: AOJu0Yw0QurOG30kYIMC8MnXnmUYjzkXnYwOdbG1smrtHvO8LxjjnvAE dHlZ9osCybAAiGYBpworXzK55h+XjF5RCW2o/l8eqJ+V0XSwrQFX/poFu/rp5splPPm1BfGyeOl qst4= X-Google-Smtp-Source: AGHT+IGaqIn+///cUz7zVB1GanxFLmqYZdBfu7k8lpwWFmBK7KujJomkknW8vwZgw2ozk/brgPrDpg== X-Received: by 2002:a05:6a00:13a1:b0:6e5:55b9:b64 with SMTP id t33-20020a056a0013a100b006e555b90b64mr1906298pfg.1.1709071485081; Tue, 27 Feb 2024 14:04:45 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-234-108-41.hawaiiantel.net. [72.234.108.41]) by smtp.gmail.com with ESMTPSA id t13-20020a056a0021cd00b006e50c0d6421sm5059772pfj.11.2024.02.27.14.04.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 14:04:44 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 5/6] tests/fetch: Add real git lfs tests and decorator Date: Tue, 27 Feb 2024 12:04:24 -1000 Message-Id: <2a6fd774405a58f3ef2953a0dc3eca063e0cf890.1709071317.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 27 Feb 2024 22:04:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15971 From: Paulo Neves Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. (cherry picked from commit d2be7f7f652360f13cd66d0850f3e19ffe2afb0a) Signed-off-by: Paulo Neves Signed-off-by: Richard Purdie Signed-off-by: Philip Lorenz Signed-off-by: Steve Sakoman --- lib/bb/tests/fetch.py | 44 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 743d3e3ff..847a35602 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2159,6 +2159,12 @@ class GitShallowTest(FetcherTest): self.assertIn("fstests.doap", dir) class GitLfsTest(FetcherTest): + def skipIfNoGitLFS(): + import shutil + if not shutil.which('git-lfs'): + return unittest.skip('git-lfs not installed') + return lambda f: f + def setUp(self): FetcherTest.setUp(self) @@ -2192,6 +2198,44 @@ class GitLfsTest(FetcherTest): ud = fetcher.ud[uri] return fetcher, ud + def get_real_git_lfs_file(self): + self.d.setVar('PATH', os.environ.get('PATH')) + fetcher, ud = self.fetch() + fetcher.unpack(self.d.getVar('WORKDIR')) + unpacked_lfs_file = os.path.join(self.d.getVar('WORKDIR'), 'git', "Cat_poster_1.jpg") + return unpacked_lfs_file + + @skipIfNoGitLFS() + @skipIfNoNetwork() + def test_real_git_lfs_repo_succeeds_without_lfs_param(self): + self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master") + f = self.get_real_git_lfs_file() + self.assertTrue(os.path.exists(f)) + self.assertEqual("c0baab607a97839c9a328b4310713307", bb.utils.md5_file(f)) + + @skipIfNoGitLFS() + @skipIfNoNetwork() + def test_real_git_lfs_repo_succeeds(self): + self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master;lfs=1") + f = self.get_real_git_lfs_file() + self.assertTrue(os.path.exists(f)) + self.assertEqual("c0baab607a97839c9a328b4310713307", bb.utils.md5_file(f)) + + @skipIfNoGitLFS() + @skipIfNoNetwork() + def test_real_git_lfs_repo_succeeds(self): + self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master;lfs=0") + f = self.get_real_git_lfs_file() + # This is the actual non-smudged placeholder file on the repo if git-lfs does not run + lfs_file = ( + 'version https://git-lfs.github.com/spec/v1\n' + 'oid sha256:34be66b1a39a1955b46a12588df9d5f6fc1da790e05cf01f3c7422f4bbbdc26b\n' + 'size 11423554\n' + ) + + with open(f) as fh: + self.assertEqual(lfs_file, fh.read()) + def test_lfs_enabled(self): import shutil From patchwork Tue Feb 27 22:04:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 40192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BCA0C54798 for ; Tue, 27 Feb 2024 22:04:51 +0000 (UTC) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by mx.groups.io with SMTP id smtpd.web10.2144.1709071487605250160 for ; Tue, 27 Feb 2024 14:04:47 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=M1pwqRNx; spf=softfail (domain: sakoman.com, ip: 209.85.210.177, mailfrom: steve@sakoman.com) Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6e48eef8be5so3419324b3a.1 for ; Tue, 27 Feb 2024 14:04:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1709071487; x=1709676287; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=OXIFWexID0xGYsCI0zG/1ifVCF3tzpnI8nza+FoI6xs=; b=M1pwqRNxa1aebUPvprRrlz9dLEqCtiFeRUH0xavZMnxsj1kU8HZhXfIXpZp+ZoqBo6 W5/kYpbPwsrEAcUVPIKiu0/LRlf8An+MQe5kSrbvEqzJNzd4xLbZZwI6bX2lHXCEGrrt aKtKWKrUmgvtsRVDGaBJjVA86xDju/9rg9yWfzIiuQcXM4KNuWyvYHs5vhJTUeHiS2vb Rx3Sr59TJMYs/RsMlu6HXKNChsuB3Bj2HatjgWXKUenJMIc74x0xGM6PrjAfhxJA74nY WfsHxeO3UUGI0wXnjsHxFoc2r3qqBbp9cd0JHvBdSDYw9g8T381wRYAVjLxkYW4Zb4yZ I9IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709071487; x=1709676287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OXIFWexID0xGYsCI0zG/1ifVCF3tzpnI8nza+FoI6xs=; b=o4qDO3mYDtPisVBZFiY1PbW45kwRC/lOKDMVjwHPoN9B3pMeSYFqjHERzVGUtDy42h 4luknIp6r5lA6z9ZaFpcXG/z5Pia+WLrSUl2tjjRI+O25xglSrIh//kqIn1CpTuDdSWP EGfPB1XcNLtJ3tO/O1Gv8ItrlnSnjsjBammZtT/nGdQvsdrIwMlFXG2oYpS8/zT008wt g9Q5wVQs7UjSKzxNve8XAcJkHJuphfGRa+vuXWbecsymZSdWppsrMTG7zVKhZhAv+cbb FmAR/I1VXw1tNJ369UlTsWzQpWAFPyZc8c9I49nmBltzIda1mnApEFxdnSoBBi5lLhP/ jEdQ== X-Gm-Message-State: AOJu0YyeS8Lxh0yW2PLskGLQznZHDgT2WtsxoHlWOwgRsxXwYNDIt+Gq tKbEviOCd9k5dvCZVADJcrmiixboqtJZUiYlcoGpmrBzUhdG/0CPMsTd1gO3SAznrcfR37YqUGs 366Y= X-Google-Smtp-Source: AGHT+IFZfqt143H1YaL9lNx6jAoWgiIRQ7W8IOo1eL/L/uBkaQ6rzqLBGZENQLCq7cdJtGjQAehYnQ== X-Received: by 2002:a05:6a00:2d1a:b0:6e5:5daf:f397 with SMTP id fa26-20020a056a002d1a00b006e55daff397mr408612pfb.13.1709071486804; Tue, 27 Feb 2024 14:04:46 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-234-108-41.hawaiiantel.net. [72.234.108.41]) by smtp.gmail.com with ESMTPSA id t13-20020a056a0021cd00b006e50c0d6421sm5059772pfj.11.2024.02.27.14.04.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 14:04:46 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 6/6] fetch2: Ensure that git LFS objects are available Date: Tue, 27 Feb 2024 12:04:25 -1000 Message-Id: <40fd5f4eef7460ca67f32cfce8e229e67e1ff607.1709071317.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 27 Feb 2024 22:04:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15972 From: Philip Lorenz The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (cherry picked from commit cfae1556bf671acec119a6c8bbc4b667a856b9ae) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie Signed-off-by: Philip Lorenz Signed-off-by: Steve Sakoman --- lib/bb/fetch2/git.py | 45 ++++++++++++++++++++++++++++++++++++-- lib/bb/tests/fetch.py | 51 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 90 insertions(+), 6 deletions(-) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 4d6e57ade..9ecc855af 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -307,7 +307,10 @@ class Git(FetchMethod): return ud.clonedir def need_update(self, ud, d): - return self.clonedir_need_update(ud, d) or self.shallow_tarball_need_update(ud) or self.tarball_need_update(ud) + return self.clonedir_need_update(ud, d) \ + or self.shallow_tarball_need_update(ud) \ + or self.tarball_need_update(ud) \ + or self.lfs_need_update(ud, d) def clonedir_need_update(self, ud, d): if not os.path.exists(ud.clonedir): @@ -319,6 +322,15 @@ class Git(FetchMethod): return True return False + def lfs_need_update(self, ud, d): + if self.clonedir_need_update(ud, d): + return True + + for name in ud.names: + if not self._lfs_objects_downloaded(ud, d, name, ud.clonedir): + return True + return False + def clonedir_need_shallow_revs(self, ud, d): for rev in ud.shallow_revs: try: @@ -406,7 +418,7 @@ class Git(FetchMethod): if missing_rev: raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev) - if self._contains_lfs(ud, d, ud.clonedir) and self._need_lfs(ud): + if self.lfs_need_update(ud, d): # Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching # of all LFS blobs needed at the srcrev. # @@ -649,6 +661,35 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output)) return output.split()[0] != "0" + def _lfs_objects_downloaded(self, ud, d, name, wd): + """ + Verifies whether the LFS objects for requested revisions have already been downloaded + """ + # Bail out early if this repository doesn't use LFS + if not self._need_lfs(ud) or not self._contains_lfs(ud, d, wd): + return True + + # The Git LFS specification specifies ([1]) the LFS folder layout so it should be safe to check for file + # existence. + # [1] https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md#intercepting-git + cmd = "%s lfs ls-files -l %s" \ + % (ud.basecmd, ud.revisions[name]) + output = runfetchcmd(cmd, d, quiet=True, workdir=wd).rstrip() + # Do not do any further matching if no objects are managed by LFS + if not output: + return True + + # Match all lines beginning with the hexadecimal OID + oid_regex = re.compile("^(([a-fA-F0-9]{2})([a-fA-F0-9]{2})[A-Fa-f0-9]+)") + for line in output.split("\n"): + oid = re.search(oid_regex, line) + if not oid: + bb.warn("git lfs ls-files output '%s' did not match expected format." % line) + if not os.path.exists(os.path.join(wd, "lfs", "objects", oid.group(2), oid.group(3), oid.group(1))): + return False + + return True + def _need_lfs(self, ud): return ud.parm.get("lfs", "1") == "1" diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 847a35602..5aa3e464d 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -6,6 +6,7 @@ # SPDX-License-Identifier: GPL-2.0-only # +import contextlib import unittest import hashlib import tempfile @@ -2182,10 +2183,14 @@ class GitLfsTest(FetcherTest): bb.utils.mkdirhier(self.srcdir) self.git_init(cwd=self.srcdir) - with open(os.path.join(self.srcdir, '.gitattributes'), 'wt') as attrs: - attrs.write('*.mp3 filter=lfs -text') - self.git(['add', '.gitattributes'], cwd=self.srcdir) - self.git(['commit', '-m', "attributes", '.gitattributes'], cwd=self.srcdir) + self.commit_file('.gitattributes', '*.mp3 filter=lfs -text') + + def commit_file(self, filename, content): + with open(os.path.join(self.srcdir, filename), "w") as f: + f.write(content) + self.git(["add", filename], cwd=self.srcdir) + self.git(["commit", "-m", "Change"], cwd=self.srcdir) + return self.git(["rev-parse", "HEAD"], cwd=self.srcdir).strip() def fetch(self, uri=None, download=True): uris = self.d.getVar('SRC_URI').split() @@ -2205,6 +2210,44 @@ class GitLfsTest(FetcherTest): unpacked_lfs_file = os.path.join(self.d.getVar('WORKDIR'), 'git', "Cat_poster_1.jpg") return unpacked_lfs_file + @skipIfNoGitLFS() + def test_fetch_lfs_on_srcrev_change(self): + """Test if fetch downloads missing LFS objects when a different revision within an existing repository is requested""" + self.git(["lfs", "install", "--local"], cwd=self.srcdir) + + @contextlib.contextmanager + def hide_upstream_repository(): + """Hide the upstream repository to make sure that git lfs cannot pull from it""" + temp_name = self.srcdir + ".bak" + os.rename(self.srcdir, temp_name) + try: + yield + finally: + os.rename(temp_name, self.srcdir) + + def fetch_and_verify(revision, filename, content): + self.d.setVar('SRCREV', revision) + fetcher, ud = self.fetch() + + with hide_upstream_repository(): + workdir = self.d.getVar('WORKDIR') + fetcher.unpack(workdir) + + with open(os.path.join(workdir, "git", filename)) as f: + self.assertEqual(f.read(), content) + + commit_1 = self.commit_file("a.mp3", "version 1") + commit_2 = self.commit_file("a.mp3", "version 2") + + self.d.setVar('SRC_URI', "git://%s;protocol=file;lfs=1;branch=master" % self.srcdir) + + # Seed the local download folder by fetching the latest commit and verifying that the LFS contents are + # available even when the upstream repository disappears. + fetch_and_verify(commit_2, "a.mp3", "version 2") + # Verify that even when an older revision is fetched, the needed LFS objects are fetched into the download + # folder. + fetch_and_verify(commit_1, "a.mp3", "version 1") + @skipIfNoGitLFS() @skipIfNoNetwork() def test_real_git_lfs_repo_succeeds_without_lfs_param(self):