From patchwork Fri Aug 26 08:40:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Zhukov X-Patchwork-Id: 11927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDC23ECAAD7 for ; Fri, 26 Aug 2022 08:41:14 +0000 (UTC) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mx.groups.io with SMTP id smtpd.web11.34816.1661503266503976885 for ; Fri, 26 Aug 2022 01:41:07 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@zhukoff.net header.s=fm1 header.b=Z5DWjmAJ; spf=pass (domain: zhukoff.net, ip: 66.111.4.25, mailfrom: pavel@zhukoff.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id B0DF35C0134; Fri, 26 Aug 2022 04:41:05 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 26 Aug 2022 04:41:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zhukoff.net; h= cc:cc:content-transfer-encoding:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to; s=fm1; t=1661503265; x=1661589665; bh=vKZl7wlpof4Bl/RybLmnOPx8i j5HpqGDpuhTkb3I4oA=; b=Z5DWjmAJEYVrNdePznGpAOUrAfDo2h1zQSElLRHR5 EN0c8it5fpmQtsQyKPxup0JeESTgZEHb6Ts4HOBxHCLSkpoHV404/DYseAABLjEj etG4OUF+TkZzBuxNmtqWEC/nXkN4Xp1mAacxrveGlDemv5CiJ2GRgD+RA1Wjl4oz hCwRUsafgsgRhZ/mp7fE7Z6ncAVCN07C0BAGB91jYmBtEnR9DAMJ9AuIQhxLqVxX Q/XZW2dLZRSNSgFDte3v25PN+ki9Miv8V8VI79koT9eIeeENrfDcSFekfuz+oKXV xNK82yreiaQCxHNLC5uXx7fTl4kq0f1Jn2Ad7tAUZ1wtg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1661503265; x=1661589665; bh=vKZl7wlpof4Bl/RybLmnOPx8ij5HpqGDpuh Tkb3I4oA=; b=GjgE3HzJHgbRpILf022BPoGx4sSdaXatyOl+v3lMI+mqh+NRfMC hQExb9BGKOzpzf+p7moooPrgK+ClJS5stQDpXJfDTyFtQqs+NtAQtxsgNdeMnAgO w512e5I6diDjVZBGrgEeOKl0xIGqLhydzeJ7diCvOuftIbw2LkjnFMPeiIfSFnqT NgVFPY1TmtpgxOmkxNBNicT+6kRfY84tfVFQylPHUP2mvcw1o6UPx/D6iATxB5WG LKJxAnlzfdfFwYVi/OyjZZBqc6bRT0G0mX2L8s7RuFP8jbN2DrY9kMxY/ba+dp0c fcJfwlvjWQ3mLjJo93sIgQy/i4vDpJzEzcg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdejhedgtdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvfevufffkffoggfgsedtkeertd ertddtnecuhfhrohhmpefrrghvvghlucgkhhhukhhovhcuoehprghvvghlseiihhhukhho fhhfrdhnvghtqeenucggtffrrghtthgvrhhnpeevfeejkeejudegledtkeejhfettdefje euffehieekvddugfdtffffledvvdefhfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehprghvvghlseiihhhukhhofhhfrdhnvght X-ME-Proxy: Feedback-ID: ib94946c9:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 26 Aug 2022 04:41:04 -0400 (EDT) From: Pavel Zhukov To: bitbake-devel@lists.openembedded.org Cc: Pavel Zhukov Subject: [PATCH 1/3] gitsm: Error out if submodule refers to parent repo Date: Fri, 26 Aug 2022 10:40:29 +0200 Message-Id: <20220826084031.11919-1-pavel@zhukoff.net> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Aug 2022 08:41:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13934 If submodule refers to specific revision of the parent repository it causes deadlock in bitbake locking mechanism (lock is acquired to fetch the parent and cannot be released before all submodules are fetched). raise FetchError in such situation to prevent deadlocking. [Yocto 14045] Signed-off-by: Pavel Zhukov --- lib/bb/fetch2/gitsm.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py index c1950e48..25d5db0e 100644 --- a/lib/bb/fetch2/gitsm.py +++ b/lib/bb/fetch2/gitsm.py @@ -115,6 +115,9 @@ class GitSM(Git): # This has to be a file reference proto = "file" url = "gitsm://" + uris[module] + if "{}{}".format(ud.host, ud.path) in url: + raise bb.fetch2.FetchError("Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake." \ + "Consider using git fetcher instead.") url += ';protocol=%s' % proto url += ";name=%s" % module From patchwork Fri Aug 26 08:40:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Zhukov X-Patchwork-Id: 11928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED09BECAAA3 for ; Fri, 26 Aug 2022 08:41:14 +0000 (UTC) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mx.groups.io with SMTP id smtpd.web08.35011.1661503267417239478 for ; Fri, 26 Aug 2022 01:41:08 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@zhukoff.net header.s=fm1 header.b=VmC1Pp12; spf=pass (domain: zhukoff.net, ip: 66.111.4.25, mailfrom: pavel@zhukoff.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 9C4515C004D; Fri, 26 Aug 2022 04:41:06 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 26 Aug 2022 04:41:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zhukoff.net; h= cc:cc:content-transfer-encoding:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm1; t=1661503266; x=1661589666; bh=uK FuBt5pj0faFRF07vm7DDfy4Iinw4RQjHbVbB+Lb2U=; b=VmC1Pp12+I5fC/ZOuw O9PMxfJoiam5loDhXYZyrBMN2T8lRQSnvqpsPugiXmfQtzZpFkZpw2Oj3nmjYZDf JU0WmVUKynsQ6q6DkI+NXeNcsv/WRN6Ff3YwVFP9IjNDNKGHyMAv9DXP3lz7X6XG UZljU8itsz/HNaiesNmfcLvymYb3I0pD73KMNAmTD9I4rGhjIUJf5L5WLYwaPnuJ 54/YnUkbe817/hBr1iXOOVdlWMzmBmb2mMQkefo+yrLrAmIQyaiIHHAB3gb6E/PY qlFO9ZHUeVFNR/fiDS3uCFppHteqJQ+Z0Bj8pD1gHgsjc2VxJrL+vE44g2Bxpw3E qQDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1661503266; x=1661589666; bh=uKFuBt5pj0faF RF07vm7DDfy4Iinw4RQjHbVbB+Lb2U=; b=mQ1q1JCjATfm2kn+cFYIZvVW8lldR egmkjqbRuKs1TKOm762j8b1tjPOXyU3ly8UTbpBByDaQQAxVXzRD77FFe6eJzPOb 0VaQ1O0QbFNBjHVohJKxojMSt8wES3v++IXxs28lqkfnGjIJSvrtwMYt88mJF3G3 aavj7DLU52j0knmzl3BNlYy97C+USjxYzc+JxaYXD/FQlDOcOd1s1uKsk7HKmE7F qJc5Z3oIdvCosGCoWiNRU9LDULWNyGVtsteH2KqGnxYgPPcuaD+TE/scKoNt7/eH q8PpPJMr4xvY2sQG/f3me9eWwn3oji9gf7JenlZLJvQxogWdpXqgGAbrQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdejhedgtdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvfevufffkffojghfggfgsedtke ertdertddtnecuhfhrohhmpefrrghvvghlucgkhhhukhhovhcuoehprghvvghlseiihhhu khhofhhfrdhnvghtqeenucggtffrrghtthgvrhhnpeehfeevgfdtleekledvudeuuedutd dukeetudfhffejtddvtdegfffgjeeifeduteenucffohhmrghinhepghhoohhglhgvrdgt ohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepph grvhgvlhesiihhuhhkohhffhdrnhgvth X-ME-Proxy: Feedback-ID: ib94946c9:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 26 Aug 2022 04:41:05 -0400 (EDT) From: Pavel Zhukov To: bitbake-devel@lists.openembedded.org Cc: Pavel Zhukov Subject: [PATCH 2/3] tests: Add Timeout class Date: Fri, 26 Aug 2022 10:40:30 +0200 Message-Id: <20220826084031.11919-2-pavel@zhukoff.net> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220826084031.11919-1-pavel@zhukoff.net> References: <20220826084031.11919-1-pavel@zhukoff.net> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Aug 2022 08:41:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13935 The class and exception aim to test rare cases there deadlocks are possible. Can be used in context managers: with Timeout(): do_deadlock() Signed-off-by: Pavel Zhukov --- lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 7fcf57e7..e69b4b05 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -11,6 +11,7 @@ import hashlib import tempfile import collections import os +import signal import tarfile from bb.fetch2 import URI from bb.fetch2 import FetchMethod @@ -22,6 +23,24 @@ def skipIfNoNetwork(): return unittest.skip("network test") return lambda f: f +class TestTimeout(Exception): + pass + +class Timeout(): + + def __init__(self, seconds): + self.seconds = seconds + + def handle_timeout(self, signum, frame): + raise TestTimeout("Test failed: timeout reached") + + def __enter__(self): + signal.signal(signal.SIGALRM, self.handle_timeout) + signal.alarm(self.seconds) + + def __exit__(self, exc_type, exc_val, exc_tb): + signal.alarm(0) + class URITest(unittest.TestCase): test_uris = { "http://www.google.com/index.html" : { From patchwork Fri Aug 26 08:40:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Zhukov X-Patchwork-Id: 11926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC7C1ECAAD2 for ; Fri, 26 Aug 2022 08:41:14 +0000 (UTC) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mx.groups.io with SMTP id smtpd.web09.34865.1661503268174007091 for ; Fri, 26 Aug 2022 01:41:08 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@zhukoff.net header.s=fm1 header.b=QLnW6IYz; spf=pass (domain: zhukoff.net, ip: 66.111.4.25, mailfrom: pavel@zhukoff.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 80E1B5C002F; Fri, 26 Aug 2022 04:41:07 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Fri, 26 Aug 2022 04:41:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zhukoff.net; h= cc:cc:content-transfer-encoding:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm1; t=1661503267; x=1661589667; bh=+g eNOVo9YzUMavYaVzHhCZJ7kHdHCdJ6+aPCGr5mexg=; b=QLnW6IYzvtzYdYF9ib +KVrlycEcrg2+rwFtZ8DcQq3oufpoUq2TNSThZeTwc6kOXAnDcD/WxRpkL5equbM Ad9GA1mYtG96ElfPeQcznZXxzUvzNko8anWy0BCvF3Y2NWzNMX2tgXDUhtZUv7dF whe5iBSJCHtK9zA/kGQpgEIOtyAhma3I+Q4Oh4YtD3ESsO405HSoSZppFQDSezMe ytO8/fWTG2i9nCtaegkZqmIOHaQZvCXwjoaQkzKBPRFgxTTb19cJlRSaiZ05fdLX YdpdrLiPvP84xvPaoxT4Bg59ugdQzcb328K+QGWNWm+HdWGnkIOQeF1wBj5tfpI5 CajQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1661503267; x=1661589667; bh=+geNOVo9YzUMa vYaVzHhCZJ7kHdHCdJ6+aPCGr5mexg=; b=jvCrIEyYh6bdVjgobExlXkYwNF9U/ wZGa+q3ypIdQYHoPLXHONENLJgkiNmd193rQWnoU1zN5d4BocXvKm8xzEXxacgU9 Xdt/8DWVUjrynMZKvDlMiJn+VueRBuhUYBNUTEANyLA2hVbZ3PKcNaTKm7s0tBuW 92AmhYQm5KXELJczCUUtsLoX4Gs6jgwEMJ1um+v+79TCQ11s3UTASSjrSu7O2IZd HMesN4IkbFCjctuOuR5M/jym4aaWM4tacMd6FxV+vJ9yC9ALNzLPS8zJ8eW5Kjl5 t/vi1kjT3wdqT3g8XVUaDq/ha1ckhg3txdJVsMulYv1d7ojcn+WyhvVoQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdejhedgtdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvfevufffkffojghfggfgsedtke ertdertddtnecuhfhrohhmpefrrghvvghlucgkhhhukhhovhcuoehprghvvghlseiihhhu khhofhhfrdhnvghtqeenucggtffrrghtthgvrhhnpeegkeelueeggfekiefgtdfgledvvd ettdffffeuheetgeevjeeutedvtddtkeekjeenucffohhmrghinhepghhithhhuhgsrdgt ohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepph grvhgvlhesiihhuhhkohhffhdrnhgvth X-ME-Proxy: Feedback-ID: ib94946c9:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 26 Aug 2022 04:41:06 -0400 (EDT) From: Pavel Zhukov To: bitbake-devel@lists.openembedded.org Cc: Pavel Zhukov Subject: [PATCH 3/3] tests: Add test for possible gitsm deadlock Date: Fri, 26 Aug 2022 10:40:31 +0200 Message-Id: <20220826084031.11919-3-pavel@zhukoff.net> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220826084031.11919-1-pavel@zhukoff.net> References: <20220826084031.11919-1-pavel@zhukoff.net> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Aug 2022 08:41:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13936 Signed-off-by: Pavel Zhukov --- lib/bb/tests/fetch.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index e69b4b05..b4ed691f 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1191,6 +1191,15 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/ctest/README.md')), msg='Missing submodule checkout') self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/testrunner/readme.md')), msg='Missing submodule checkout') + @skipIfNoNetwork() + def test_git_submodule_reference_to_parent(self): + self.recipe_url = "gitsm://github.com/gflags/gflags.git;protocol=https;branch=master" + self.d.setVar("SRCREV", "14e1138441bbbb584160cb1c0a0426ec1bac35f1") + with Timeout(60): + fetcher = bb.fetch.Fetch([self.recipe_url], self.d) + with self.assertRaises(bb.fetch2.FetchError): + fetcher.download() + class SVNTest(FetcherTest): def skipIfNoSvn(): import shutil