From patchwork Sun Apr 3 10:21:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 6246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EAA3C35276 for ; Mon, 4 Apr 2022 18:46:41 +0000 (UTC) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mx.groups.io with SMTP id smtpd.web12.21492.1648981308558317896 for ; Sun, 03 Apr 2022 03:21:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=TRFVEjBH; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.49, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f49.google.com with SMTP id r7so4201434wmq.2 for ; Sun, 03 Apr 2022 03:21:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=362KOpTFDMmRiPsNMCGrH25RehV/7miKSBWP3lo1b4s=; b=TRFVEjBHcGIjb5+oxMMzYew+nPWE4G1c7kBm3NFtuxE7hsVkcbxi51eAbSs4b1+wnG pBnBXU8+JX+mpIL2J1+c3SaJa+xR9hvuHeLL5c7Xc2+lgVXuAu4zWMdoWaeILLXksrD9 WibKmXLSiVV4OF1IY+6RVSGXJ9IcXn6CpJmTY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=362KOpTFDMmRiPsNMCGrH25RehV/7miKSBWP3lo1b4s=; b=12ElWlYsDQdpHSSnQRxnVTNZw1HC9WC06TkROFHJR5w+VNQKmyyW2hm+sKmw9ZJboq EH47Sypes1Wip8HQp6fMHiPdI4PnMjhRlBUs+1beqz6e3il3o7i9xrI+EOoHiE4TxyyU mW3ccLUOFxabI9z/DbDZ+wXfR8WHU6sLhpVFFtofYInuO9hapBpQjLazA8BuBXs3ywZm IZhaYp0wXbAJlkIbn1eOE2z4wsFhTKZqEoPEuvfZ3mYR+zRny9QIS0CFwD1fIBpAq5Z0 Q7n6knPxM9XQEpj/+N5moRBDbjHh9Quv1TPXK8dUj91m09FIwr81Tun+i0H2YLSzyuwm 5Q7w== X-Gm-Message-State: AOAM531cdzSbtnEl5XI0BNb29S7P2DxFn6tCs2n33NLQ5MBXY6eLgwZV zLDHcZ0xlDOBB3t7g6gl8porKADwLC8IrgQn X-Google-Smtp-Source: ABdhPJzFMb8hTiPxKYfgBX8m6DR9moGkPEtw8OyAn/yNPn81m+e2/9ndEpY5AAdH7FvFn+JJ8MfOkQ== X-Received: by 2002:a05:600c:213:b0:38e:6379:af70 with SMTP id 19-20020a05600c021300b0038e6379af70mr5078278wmi.157.1648981306633; Sun, 03 Apr 2022 03:21:46 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:819:f853:3436:28c4]) by smtp.gmail.com with ESMTPSA id z3-20020a1cf403000000b0037d1f4a2201sm6327981wma.21.2022.04.03.03.21.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Apr 2022 03:21:45 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/5] parse: Ensure any existing siggen is closed down first Date: Sun, 3 Apr 2022 11:21:40 +0100 Message-Id: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Apr 2022 18:46:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13571 We're still seeing issues with unclosed asyncio event loops. At the init site, make sure any existing one is closed first to try and avoid this. Signed-off-by: Richard Purdie --- lib/bb/parse/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/bb/parse/__init__.py b/lib/bb/parse/__init__.py index c01807ba87..347609513b 100644 --- a/lib/bb/parse/__init__.py +++ b/lib/bb/parse/__init__.py @@ -113,6 +113,8 @@ def init(fn, data): return h['init'](data) def init_parser(d): + if hasattr(bb.parse, "siggen"): + bb.parse.siggen.exit() bb.parse.siggen = bb.siggen.init(d) def resolve_file(fn, d): From patchwork Sun Apr 3 10:21:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 6248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74B42C3527E for ; Mon, 4 Apr 2022 18:46:41 +0000 (UTC) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx.groups.io with SMTP id smtpd.web10.21740.1648981309462523314 for ; Sun, 03 Apr 2022 03:21:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=GN0BVbi9; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.51, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f51.google.com with SMTP id f6-20020a1c3806000000b0038e4a0fc5easo3944011wma.3 for ; Sun, 03 Apr 2022 03:21:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ShS6ahVR1DSBAP71BnNT7RVSKwzYR387v0pRmkonUSw=; b=GN0BVbi9xvPdgg6x9xNJr8ThgCG458C5/QbkTHY6rJ8fqOn4LyRgLkGArBXTMiIK33 jtl+5DW6cGkRF+e6tQHyF5+KYd4Ti71+XwCRU31kBwDEMLcvsg/XYkTZrNA+1Xe8G01P TosKrJmdoV0IPLK84+wv50JaESWz4yzseuofI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ShS6ahVR1DSBAP71BnNT7RVSKwzYR387v0pRmkonUSw=; b=Q1NsSKQAZ39BWpTZCHsFuHjU4mMDOAO487xlNh6k9bwbruCc3t6DmcFaYbhWiCRB6R Dj4Y13jPBNcritIlvQB6wOep84Hs52yo2UIY1WTtT285aznzR9wnxU7sg02It6UoJvLo srKS3agtw/rsMYGhwspHYnQmP6aubgcDpMoIzpzjvCkZppItfFwCQ0KcmNgR+n9ge9gQ E/AERna5UTrfB79W9o4n0HAM41TNBLN+5ySEzV40csU+qJ42QdrOANYWQIKrI/MfxJIR WJUQktm1KFyceaiSvG4W6UYM2/q1OM5tNs2dYlgmxknG9jyVjTU4K1hLn42CeEG+Pf6N Ra8A== X-Gm-Message-State: AOAM532iPQ9e8jT3LhiDDD4EkKx29I0bBXeDX72UuEf79UTrc/p1LlvY XOMR+2VUIxCtLKe07nJ7tO1+JAU29e88Mxa0 X-Google-Smtp-Source: ABdhPJy0284Qa15wwCqxSFIeFh+9GFf9QDGkagZ1k11lF6X8HkLr6EPkhM/V47NmVDFS82omYes4hA== X-Received: by 2002:a05:600c:ca:b0:38c:b426:3124 with SMTP id u10-20020a05600c00ca00b0038cb4263124mr15457826wmm.59.1648981307716; Sun, 03 Apr 2022 03:21:47 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:819:f853:3436:28c4]) by smtp.gmail.com with ESMTPSA id z3-20020a1cf403000000b0037d1f4a2201sm6327981wma.21.2022.04.03.03.21.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Apr 2022 03:21:46 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/5] data: Ensure vardepsexclude or BB_BASEHASH_IGNORE_VARS covers contains items Date: Sun, 3 Apr 2022 11:21:41 +0100 Message-Id: <20220403102144.1679700-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> References: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Apr 2022 18:46:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13572 Adding bb.utils.filter('WARN_QA', 'patch-fuzz', d) when WARN_QA is in BB_BASEHASH_IGNORE_VARS or in vardepsexclude should not add a dependency on WARN_QA. Fix it and add some tests. Signed-off-by: Richard Purdie --- lib/bb/data.py | 21 ++++++++++++--------- lib/bb/tests/codeparser.py | 28 +++++++++++++++++++++++----- 2 files changed, 35 insertions(+), 14 deletions(-) diff --git a/lib/bb/data.py b/lib/bb/data.py index 5d02bab99b..363901b5d3 100644 --- a/lib/bb/data.py +++ b/lib/bb/data.py @@ -272,7 +272,7 @@ def update_data(d): """Performs final steps upon the datastore, including application of overrides""" d.finalize(parent = True) -def build_dependencies(key, keys, shelldeps, varflagsexcl, d): +def build_dependencies(key, keys, shelldeps, varflagsexcl, ignored_vars, d): deps = set() try: if key[-1] == ']': @@ -283,12 +283,15 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): return deps, value varflags = d.getVarFlags(key, ["vardeps", "vardepvalue", "vardepsexclude", "exports", "postfuncs", "prefuncs", "lineno", "filename"]) or {} vardeps = varflags.get("vardeps") + exclusions = varflags.get("vardepsexclude", "").split() - def handle_contains(value, contains, d): + def handle_contains(value, contains, exclusions, d): newvalue = [] if value: newvalue.append(str(value)) for k in sorted(contains): + if k in exclusions or k in ignored_vars: + continue l = (d.getVar(k) or "").split() for item in sorted(contains[k]): for word in item.split(): @@ -316,7 +319,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): parser.parse_python(value, filename=varflags.get("filename"), lineno=varflags.get("lineno")) deps = deps | parser.references deps = deps | (keys & parser.execs) - value = handle_contains(value, parser.contains, d) + value = handle_contains(value, parser.contains, exclusions, d) else: value, parsedvar = d.getVarFlag(key, "_content", False, retparser=True) parser = bb.codeparser.ShellParser(key, logger) @@ -324,9 +327,9 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): deps = deps | shelldeps deps = deps | parsedvar.references deps = deps | (keys & parser.execs) | (keys & parsedvar.execs) - value = handle_contains(value, parsedvar.contains, d) + value = handle_contains(value, parsedvar.contains, exclusions, d) if hasattr(parsedvar, "removes"): - value = handle_remove(value, deps, parsedvar.removes, d) + value = handle_remove(value, deps, parsedvar.removes, varflags, d) if vardeps is None: parser.log.flush() if "prefuncs" in varflags: @@ -339,7 +342,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): value, parser = d.getVarFlag(key, "_content", False, retparser=True) deps |= parser.references deps = deps | (keys & parser.execs) - value = handle_contains(value, parser.contains, d) + value = handle_contains(value, parser.contains, exclusions, d) if hasattr(parser, "removes"): value = handle_remove(value, deps, parser.removes, d) @@ -359,7 +362,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): deps |= set(varfdeps) deps |= set((vardeps or "").split()) - deps -= set(varflags.get("vardepsexclude", "").split()) + deps -= set(exclusions) except bb.parse.SkipRecipe: raise except Exception as e: @@ -380,7 +383,7 @@ def generate_dependencies(d, ignored_vars): tasklist = d.getVar('__BBTASKS', False) or [] for task in tasklist: - deps[task], values[task] = build_dependencies(task, keys, shelldeps, varflagsexcl, d) + deps[task], values[task] = build_dependencies(task, keys, shelldeps, varflagsexcl, ignored_vars, d) newdeps = deps[task] seen = set() while newdeps: @@ -389,7 +392,7 @@ def generate_dependencies(d, ignored_vars): newdeps = set() for dep in nextdeps: if dep not in deps: - deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, varflagsexcl, d) + deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, varflagsexcl, ignored_vars, d) newdeps |= deps[dep] newdeps -= seen #print "For %s: %s" % (task, str(deps[task])) diff --git a/lib/bb/tests/codeparser.py b/lib/bb/tests/codeparser.py index f485204791..71ed382ab8 100644 --- a/lib/bb/tests/codeparser.py +++ b/lib/bb/tests/codeparser.py @@ -318,7 +318,7 @@ d.getVar(a(), False) "filename": "example.bb", }) - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), self.d) self.assertEqual(deps, set(["somevar", "bar", "something", "inexpand", "test", "test2", "a"])) @@ -365,7 +365,7 @@ esac self.d.setVarFlags("FOO", {"func": True}) self.setEmptyVars(execs) - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), self.d) self.assertEqual(deps, set(["somevar", "inverted"] + execs)) @@ -375,7 +375,7 @@ esac self.d.setVar("FOO", "foo=oe_libinstall; eval $foo") self.d.setVarFlag("FOO", "vardeps", "oe_libinstall") - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), self.d) self.assertEqual(deps, set(["oe_libinstall"])) @@ -384,7 +384,7 @@ esac self.d.setVar("FOO", "foo=oe_libinstall; eval $foo") self.d.setVarFlag("FOO", "vardeps", "${@'oe_libinstall'}") - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), self.d) self.assertEqual(deps, set(["oe_libinstall"])) @@ -399,7 +399,7 @@ esac # Check dependencies self.d.setVar('ANOTHERVAR', expr) self.d.setVar('TESTVAR', 'anothervalue testval testval2') - deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), self.d) + deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), self.d) self.assertEqual(sorted(values.splitlines()), sorted([expr, 'TESTVAR{anothervalue} = Set', @@ -412,6 +412,24 @@ esac # Check final value self.assertEqual(self.d.getVar('ANOTHERVAR').split(), ['anothervalue', 'yetanothervalue', 'lastone']) + def test_contains_vardeps_excluded(self): + # Check the ignored_vars option to build_dependencies is handled by contains functionality + varval = '${TESTVAR2} ${@bb.utils.filter("TESTVAR", "somevalue anothervalue", d)}' + self.d.setVar('ANOTHERVAR', varval) + self.d.setVar('TESTVAR', 'anothervalue testval testval2') + self.d.setVar('TESTVAR2', 'testval3') + deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(["TESTVAR"]), self.d) + self.assertEqual(sorted(values.splitlines()), sorted([varval])) + self.assertEqual(deps, set(["TESTVAR2"])) + self.assertEqual(self.d.getVar('ANOTHERVAR').split(), ['testval3', 'anothervalue']) + + # Check the vardepsexclude flag is handled by contains functionality + self.d.setVarFlag('ANOTHERVAR', 'vardepsexclude', 'TESTVAR') + deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), self.d) + self.assertEqual(sorted(values.splitlines()), sorted([varval])) + self.assertEqual(deps, set(["TESTVAR2"])) + self.assertEqual(self.d.getVar('ANOTHERVAR').split(), ['testval3', 'anothervalue']) + #Currently no wildcard support #def test_vardeps_wildcards(self): # self.d.setVar("oe_libinstall", "echo test") From patchwork Sun Apr 3 10:21:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 6250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC4F8C38161 for ; Mon, 4 Apr 2022 18:46:41 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web12.21494.1648981310211985793 for ; Sun, 03 Apr 2022 03:21:50 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=d9X8sg+y; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f44.google.com with SMTP id c190-20020a1c35c7000000b0038e37907b5bso6029525wma.0 for ; Sun, 03 Apr 2022 03:21:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=1LdTSFwbDv/1zD8thlAkrEczrnjyegh5tLs5/tTtVvU=; b=d9X8sg+yZpaTfiERxzeVeCGZoqRmG0weJpuwzuS2y7VwDkN7unkbB7sgl+KYRqLxSZ l79UU3GIcoNAfCFP6GRHocNdQTN005bcRX5tKHv+TIM9kpMPoAYkSJf0qY51QjQ72que aS+BYL7fXteqcY4gxHyP/QCnB9+PLqkvTAnCQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1LdTSFwbDv/1zD8thlAkrEczrnjyegh5tLs5/tTtVvU=; b=6IpN/YwwRieCiouQj9Uq4Q6ZMYafvXb+q1YNAW1B7/HwrhoAVZYKyUTVd394RDQQmk GYby4pyRXgaEz5aK5agRM379ewnZvdfBkdxhPQ1qqFXid2uUPYrydMzbItegBKVD7rXZ +sGx+WibsEdP3DN4VSFTKSemSkFHAt1pLmHYRM5rp5FAVTIZ6kda7MqC0XjrkCt2VYSS lZIAL4KA5g+/3aQHhpuYntnMcNr9McxkHoHutmWbqtnbxwwebljdVsrZqZpXvxzx5o31 6aEookLSP6i24eIeK4h7fU8k09VyP4pf0u1GSOUma6tPc+tlCHNrt+oUNCkfI3mDqM4I xQkQ== X-Gm-Message-State: AOAM531iOy5EWK/shqTKnlNOpB/kdsOxUaqJMB5R2x2jY10Fdp5veyD2 wk6dtE4cVh9AtZwCpm3i9miSbqtiAiyQ/YX4 X-Google-Smtp-Source: ABdhPJxc9NwC+97bob7vAB9uzOfu+uXtBuspQqSnAMbAMLyGuP0jTnx0s4OOShVtqYBCjNYWDv77zA== X-Received: by 2002:a05:600c:3016:b0:38c:8786:d3b6 with SMTP id j22-20020a05600c301600b0038c8786d3b6mr15276043wmh.135.1648981308561; Sun, 03 Apr 2022 03:21:48 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:819:f853:3436:28c4]) by smtp.gmail.com with ESMTPSA id z3-20020a1cf403000000b0037d1f4a2201sm6327981wma.21.2022.04.03.03.21.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Apr 2022 03:21:48 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/5] server/process: Disable gc around critical section Date: Sun, 3 Apr 2022 11:21:42 +0100 Message-Id: <20220403102144.1679700-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> References: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Apr 2022 18:46:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13573 The python gc can trigger whilst we're holding the event stream lock and when cleaning up objects, they can trigger warnings. This translates into a new event which would then need the lock and we can deadlock. Disable gc whilst we hold that lock to avoid this unfortunate and problematic situation. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index ce53fdc678..19ef83980f 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -28,6 +28,7 @@ import re import datetime import pickle import traceback +import gc import bb.server.xmlrpcserver from bb import daemonize from multiprocessing import queues @@ -739,8 +740,10 @@ class ConnectionWriter(object): self.event = self def _send(self, obj): + gc.disable() with self.wlock: self.writer.send_bytes(obj) + gc.enable() def send(self, obj): obj = multiprocessing.reduction.ForkingPickler.dumps(obj) From patchwork Sun Apr 3 10:21:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 6247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D810C3527B for ; Mon, 4 Apr 2022 18:46:41 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web12.21496.1648981311270855312 for ; Sun, 03 Apr 2022 03:21:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=fXaMu5GW; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id 123-20020a1c1981000000b0038b3616a71aso3939189wmz.4 for ; Sun, 03 Apr 2022 03:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=rVbiRmr/NbumgmhRLGfvurdQY1O0euEclrse/J2c8S4=; b=fXaMu5GWSbmOyXlFWMGyuIEVxIziw2a3T1tnynJXKn0gL0bXj+AzISAHWo+U+p7V4y la4FFyqik/KWyQRsB5MC2A47ngDs8DvXTvnteKBJZxi/JgCOy93vDOGlIqfLEXBPy9Ci aLI9ntVDiYjGodSv+VCCZkDwSyXUw4Wso7FIs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rVbiRmr/NbumgmhRLGfvurdQY1O0euEclrse/J2c8S4=; b=0qdpfmRXxVLzyhWnzEDS0EXVMXsmxr5G5G1aYRMI3yr1M9p49OmXT8/hLI6rJzmGez 9Zi40ZnmEQ9qjsnb+Spw1NEK8402VC27rTzlq3G2g26mIzqkU989zb8LYGhVl0Lufjre xdLxJT2CmmNCcoch8dlADnfm7GaEoSu0jhyWEFQDpaL/R/tbJMIqpH4RuKTlbmrBwbTZ ajFPUwDlpuantzErCcxEMjRtR0sy4IqYqn63dkcxNo3zO2YK63xhXzb5TyC1rbqpqQ1r oFA6AMpL+LCShlF4SRVaiF+5QpuMfpzcqJIji1tV5mnFqiipPn+qgKHTElfPNFheXdnu 4WaQ== X-Gm-Message-State: AOAM533TzerdFP81VailpAyW25eIUyF4TsJLgxLUN5LmwOikbKaszDOn 40hB19yEc8CG2HnjI4KdpkppK1Z4HtSA9Wto X-Google-Smtp-Source: ABdhPJyeckJp9hKD7U87UjdhqJ0eZiDgvLXdPQA1TOOJp1j/FDibFNoq4T6I/0VDlftHoHlgKSCkSw== X-Received: by 2002:a05:600c:154d:b0:38c:e9b8:d13f with SMTP id f13-20020a05600c154d00b0038ce9b8d13fmr15285354wmg.183.1648981309506; Sun, 03 Apr 2022 03:21:49 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:819:f853:3436:28c4]) by smtp.gmail.com with ESMTPSA id z3-20020a1cf403000000b0037d1f4a2201sm6327981wma.21.2022.04.03.03.21.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Apr 2022 03:21:48 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/5] cooker: Reset and rebuild inotify watches Date: Sun, 3 Apr 2022 11:21:43 +0100 Message-Id: <20220403102144.1679700-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> References: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Apr 2022 18:46:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13574 The recent inotify changes can cause entire build trees to be monitored which is suboptimal for performance. Rather than trying increasingly convoluted tricks to try and handle add/removed directories, rebuild the inotify watch when we reparse the configuration or metadata. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 54 ++++++++++++++++++++++++++---------------------- 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 5a941f879f..7685db11f3 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -188,27 +188,15 @@ class BBCooker: bb.debug(1, "BBCooker starting %s" % time.time()) sys.stdout.flush() - self.configwatcher = pyinotify.WatchManager() - bb.debug(1, "BBCooker pyinotify1 %s" % time.time()) - sys.stdout.flush() + self.configwatcher = None + self.confignotifier = None - self.configwatcher.bbseen = set() - self.configwatcher.bbwatchedfiles = set() - self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications) - bb.debug(1, "BBCooker pyinotify2 %s" % time.time()) - sys.stdout.flush() self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \ pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO - self.watcher = pyinotify.WatchManager() - bb.debug(1, "BBCooker pyinotify3 %s" % time.time()) - sys.stdout.flush() - self.watcher.bbseen = set() - self.watcher.bbwatchedfiles = set() - self.notifier = pyinotify.Notifier(self.watcher, self.notifications) - bb.debug(1, "BBCooker pyinotify complete %s" % time.time()) - sys.stdout.flush() + self.watcher = None + self.notifier = None # If being called by something like tinfoil, we need to clean cached data # which may now be invalid @@ -259,9 +247,29 @@ class BBCooker: sys.stdout.flush() self.handlePRServ() + def setupConfigWatcher(self): + if self.configwatcher: + self.configwatcher.close() + self.confignotifier = None + self.configwatcher = None + self.configwatcher = pyinotify.WatchManager() + self.configwatcher.bbseen = set() + self.configwatcher.bbwatchedfiles = set() + self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications) + + def setupParserWatcher(self): + if self.watcher: + self.watcher.close() + self.notifier = None + self.watcher = None + self.watcher = pyinotify.WatchManager() + self.watcher.bbseen = set() + self.watcher.bbwatchedfiles = set() + self.notifier = pyinotify.Notifier(self.watcher, self.notifications) + def process_inotify_updates(self): for n in [self.confignotifier, self.notifier]: - if n.check_events(timeout=0): + if n and n.check_events(timeout=0): # read notified events and enqeue them n.read_events() n.process_events() @@ -281,10 +289,6 @@ class BBCooker: self.configwatcher.bbseen.remove(event.pathname) # Could remove all entries starting with the directory but for now... bb.parse.clear_cache() - if "IN_CREATE" in event.maskname: - self.add_filewatch([[event.pathname]], watcher=self.configwatcher, dirs=True) - elif "IN_DELETE" in event.maskname and event.pathname in self.configwatcher.bbseen: - self.configwatcher.bbseen.remove(event.pathname) if not event.pathname in self.inotify_modified_files: self.inotify_modified_files.append(event.pathname) self.baseconfig_valid = False @@ -304,10 +308,6 @@ class BBCooker: self.watcher.bbseen.remove(event.pathname) # Could remove all entries starting with the directory but for now... bb.parse.clear_cache() - if "IN_CREATE" in event.maskname: - self.add_filewatch([[event.pathname]], dirs=True) - elif "IN_DELETE" in event.maskname and event.pathname in self.watcher.bbseen: - self.watcher.bbseen.remove(event.pathname) if not event.pathname in self.inotify_modified_files: self.inotify_modified_files.append(event.pathname) self.parsecache_valid = False @@ -377,6 +377,8 @@ class BBCooker: if mod not in self.orig_sysmodules: del sys.modules[mod] + self.setupConfigWatcher() + # Need to preserve BB_CONSOLELOG over resets consolelog = None if hasattr(self, "data"): @@ -1658,6 +1660,8 @@ class BBCooker: self.updateCacheSync() if self.state != state.parsing and not self.parsecache_valid: + self.setupParserWatcher() + bb.parse.siggen.reset(self.data) self.parseConfiguration () if CookerFeatures.SEND_SANITYEVENTS in self.featureset: From patchwork Sun Apr 3 10:21:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 6251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 442D6C4708B for ; Mon, 4 Apr 2022 18:46:42 +0000 (UTC) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mx.groups.io with SMTP id smtpd.web09.21403.1648981312070901157 for ; Sun, 03 Apr 2022 03:21:52 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=U3W7ZdEb; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f44.google.com with SMTP id h4so10249855wrc.13 for ; Sun, 03 Apr 2022 03:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=OITC/9oS4D13Zcqn+s0//rTxraO4VvFWSDtzjU5SXp4=; b=U3W7ZdEbdQbgwbQWDl2wrFicSn4HwWvTXEnbCvNLBRAAFZk0ahqL/p2BOLfAGb8zJZ HCyVW4bTr07XXv4Tj13cDv7KZRmBVyqgq00VtUPKsN01GQlN8koqxxxVWwqPo0v/lpKo 4AL/F0Ah+dw/FydQNEUBvLoin/uYBaVhr0iZ8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OITC/9oS4D13Zcqn+s0//rTxraO4VvFWSDtzjU5SXp4=; b=PuDEmMTBSmZKRkn8CVTvlpDG+8OlhwLxTMdPqh8u/TtKcyijO4xYOlhbnUuh/0FwB0 D+a2LVHnZrxVegOF5DAM2dzRwQZH71B8o5l9/OuZuTLZHcLblR9N0OYbmsq3raJroOd0 MatrxbbeIq8LiZEMRoTk3Mf+OIMHSBJ/CNdsQGzMFvRYnfPZgYmA6A7yeOQI9jl1gI2Z NelEofJP+M2dsPlBADatTIcqaXdXH+ey6M9Fkae3qU4UiNV1dq7NLbdiGb1ImsYvZPEr g7t+6LbjUmRrCztlbGh9sytqP+QDpfbIgJGdh4N3SoJr2dqWRAPodsOJhuwTOQL7fHrm zodQ== X-Gm-Message-State: AOAM532KytpfPa1SsivR6P/hkBZl7aWvS9FE+8j3xFGg8vkjkHUI7c2s SOK6qLkxal37EpdtgBgi9naBaJOyGSOra0Xa X-Google-Smtp-Source: ABdhPJxyRKw/SO9BfWUepQ48NA1mDqnFTlCd6IobADij4ci//LzvLB8/JyfP02cAKRN96EINSfY2FQ== X-Received: by 2002:adf:d1e3:0:b0:204:7e76:8a70 with SMTP id g3-20020adfd1e3000000b002047e768a70mr13964679wrd.580.1648981310375; Sun, 03 Apr 2022 03:21:50 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:819:f853:3436:28c4]) by smtp.gmail.com with ESMTPSA id z3-20020a1cf403000000b0037d1f4a2201sm6327981wma.21.2022.04.03.03.21.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Apr 2022 03:21:49 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 5/5] pyinotify: Handle potential latent bug Date: Sun, 3 Apr 2022 11:21:44 +0100 Message-Id: <20220403102144.1679700-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> References: <20220403102144.1679700-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Apr 2022 18:46:42 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13575 The kernel inotify code can set more than one of the bits in the mask, fsnotify_change() in linux/fsnotify.h is quite clear that IN_ATTRIB, IN_MODIFY and IN_ACCESS can arrive together. We don't care about two of these from a bitbake perspective but it probably explains why in real world builds, we've seen: pyinotify.ProcessEventError: Unknown mask 0x00000006 This module code assumes only one mask bit can be present. Since we don't care about two of these events, just mask them out for now. The "upstream" code is unmainained since 2015. Signed-off-by: Richard Purdie --- lib/pyinotify.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/pyinotify.py b/lib/pyinotify.py index 8c94b3e334..5c9b6d0fe2 100644 --- a/lib/pyinotify.py +++ b/lib/pyinotify.py @@ -603,6 +603,17 @@ class _ProcessEvent: unknown event. """ stripped_mask = event.mask - (event.mask & IN_ISDIR) + # Bitbake hack - we see event masks of 0x6, IN_MODIFY & IN_ATTRIB + # The kernel inotify code can set more than one of the bits in the mask, + # fsnotify_change() in linux/fsnotify.h is quite clear that IN_ATTRIB, + # IN_MODIFY and IN_ACCESS can arrive together. + # This breaks the code below which assume only one mask bit is ever + # set in an event. We don't care about attrib or access in bitbake so drop those + if (stripped_mask & IN_MODIFY) and (stripped_mask & IN_ATTRIB): + stripped_mask = stripped_mask - (stripped_mask & IN_ATTRIB) + if (stripped_mask & IN_MODIFY) and (stripped_mask & IN_ACCESS): + stripped_mask = stripped_mask - (stripped_mask & IN_ACCESS) + maskname = EventsCodes.ALL_VALUES.get(stripped_mask) if maskname is None: raise ProcessEventError("Unknown mask 0x%08x" % stripped_mask)