From patchwork Fri Mar 25 09:29:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97C50C433EF for ; Fri, 25 Mar 2022 09:29:16 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web12.3213.1648200555613764768 for ; Fri, 25 Mar 2022 02:29:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=ZaGCtTF/; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.52, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f52.google.com with SMTP id u3so10046923wrg.3 for ; Fri, 25 Mar 2022 02:29:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=8z4Pgx4CxP+WkFQ95Yj12nPdIbmDN+ajTvF2ytd+krY=; b=ZaGCtTF/mKKSI1G+uArhJ/WAzC6qr9O4kQxuZxdQ4W37KfvSocvHx9Q/1/bwVVuzl6 x0zO5q2+G0TajzcF1/VZ5PQUalMMwBCf4F4g6M2q5r91qinKeXpURS2JjEn7o73fyS7i N4ueU2iPmBTcZXG8ee30DGQjz87fUg/3g54Gs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=8z4Pgx4CxP+WkFQ95Yj12nPdIbmDN+ajTvF2ytd+krY=; b=aI5sDt6E5vUKlIauX9KEVgslG103K/YOvyyvm3rxhkVZKRuG0mZhLDH9+ra6o5a8nI 3ZGc5ichMcjwVAQ8kkhlru/xBv4HCscpPKIvCUHK1r3enBb/P/zUX7CexO+CGu0AUR87 gMWHV4Gbn3ojdaMWgte/ckzgvE79/kmxpOfBVxRwrMnQv2NLNZ/boJL9t29qkfgmjSvh GQmJKZAVsmZmMviC1p2PBFATBa09HQCbkY04y8qc5+hKgPYl58NiXyIxeMfaQOC2MBWn XYy033EH5Cmnh51agS/6jry+Q9HnSk+3gwAKS19IDeIvxzEYeL2FBjK0bovpEjSfdyRx TRCg== X-Gm-Message-State: AOAM531hlh5bhler77Jq85JGECXtm2dCBzkBNtMlLveq+W4XzV5FqMtL BrVQqXPceJvle8KiubQhtKsrT2mPvCGZMizO X-Google-Smtp-Source: ABdhPJw0wLNzY8/FOufBhN0OGS4aSlJDvEYtQcGF5WQauYCfZ9jRFo++VKJOM7jGZcoBkB8m1JAQow== X-Received: by 2002:adf:e44b:0:b0:1f0:250a:d3ef with SMTP id t11-20020adfe44b000000b001f0250ad3efmr8263703wrm.402.1648200553566; Fri, 25 Mar 2022 02:29:13 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:f167:7010:a0cc:679f]) by smtp.gmail.com with ESMTPSA id p8-20020a5d4e08000000b002054b5437f2sm4478904wrt.115.2022.03.25.02.29.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 02:29:13 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/3] cooker/siggen: Support exit calls and use for hashserv client Date: Fri, 25 Mar 2022 09:29:10 +0000 Message-Id: <20220325092912.1350321-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 25 Mar 2022 09:29:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13503 We have shutdown functions within the async client code but the siggen doesn't currently call them. We notice on python 3.10 (such as on fedora35) that at exit, there is a stray asyncio process left behind. Usually this doesn't cause problems but it could potentially be a cause of a hang. For general cleanliness and completness, add in hooks to call the exit handler. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 1 + lib/bb/siggen.py | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index f79dc2770d..c0a7a2fd79 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1717,6 +1717,7 @@ class BBCooker: def post_serve(self): self.shutdown(force=True) prserv.serv.auto_shutdown() + bb.parse.siggen.exit() if self.hashserv: self.hashserv.process.terminate() self.hashserv.process.join() diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 96e16a45f8..ff6b24ccda 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -156,6 +156,9 @@ class SignatureGenerator(object): return DataCacheProxy() + def exit(self): + return + class SignatureGeneratorBasic(SignatureGenerator): """ """ @@ -489,6 +492,12 @@ class SignatureGeneratorUniHashMixIn(object): self._client = hashserv.create_client(self.server) return self._client + def exit(self): + if getattr(self, '_client', None) is not None: + self._client.close() + self._client = None + return super().exit() + def get_stampfile_hash(self, tid): if tid in self.taskhash: # If a unique hash is reported, use it as the stampfile hash. This From patchwork Fri Mar 25 09:29:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5832 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 598EDC433FE for ; Fri, 25 Mar 2022 09:29:17 +0000 (UTC) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mx.groups.io with SMTP id smtpd.web08.3042.1648200556341582805 for ; Fri, 25 Mar 2022 02:29:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=eAZPJ1Iy; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f44.google.com with SMTP id d7so10017876wrb.7 for ; Fri, 25 Mar 2022 02:29:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=OkscGc+l1fxuX6R+HqF+eyvA5WuFdfrjftyHNI25bmw=; b=eAZPJ1Iy72rRF/elyJw+VegzTfcy/Cn5a5MGxWrpbPLV1S4z2QicV+XO6D4goP4c7v NsmM+86xd6UNjbJWwkw0VPP567w+WBe7OEjE3MfehayQQfYwEuVBJsM92BG1Y8g/Sgaj 93Q28xuzfxaJGbtzI47FQomfnB7zvKgON8Mac= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OkscGc+l1fxuX6R+HqF+eyvA5WuFdfrjftyHNI25bmw=; b=NS6bgXgvjTaI552237Ev/eGvQiSHVMxoeSsSMTvleYCuoysLpNkYlQdA8u111jZPmT 6NOFzl/a0e8RRTZiHv18XcUmu+90kEjxd5ptdvyVzWMLQPYtZVc0IUzpDC29076YiSca qDCG2hKjhrdLALput2TWZiMrYcuz/wQP5ZWIJdjsVZVclFJoSxY3iF9c7JzglmJYQPBX KoiiEXz/6pjwQojdsMzJy6J+xCC0R2D5fe9bn7v+rky2c041DiRrhZfROodnMBf5zeI0 gbEuSnNSnJaKt9XJIHTFUZRDUwbJI8WBB6C6R9tLS1yUwesP67cxN/rA9/pvMgdmKA8e IsZA== X-Gm-Message-State: AOAM533xJxEOZmgdgRQJi9DeOZk9xuFw5+VqDeg/J4IjhfrsNtvghpz4 bJJED9XIkZsXkY3Ax99t7ItNvm57yxELDVEi X-Google-Smtp-Source: ABdhPJwO5crCGKb7RMpT4Anpq0GKd004/q8trtPxTPWZ9r4VvNlNhuv2EpI4m9H8kHR5NObw4GNhkg== X-Received: by 2002:adf:f604:0:b0:203:d9ca:bccf with SMTP id t4-20020adff604000000b00203d9cabccfmr8226780wrp.331.1648200554258; Fri, 25 Mar 2022 02:29:14 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:f167:7010:a0cc:679f]) by smtp.gmail.com with ESMTPSA id p8-20020a5d4e08000000b002054b5437f2sm4478904wrt.115.2022.03.25.02.29.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 02:29:13 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/3] siggen: Add missing reset handler to hashserv signature generator Date: Fri, 25 Mar 2022 09:29:11 +0000 Message-Id: <20220325092912.1350321-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220325092912.1350321-1-richard.purdie@linuxfoundation.org> References: <20220325092912.1350321-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 25 Mar 2022 09:29:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13504 When we have a client connection, we should close that connection when reset() is called on the siggen. Add the missing function. Signed-off-by: Richard Purdie --- lib/bb/siggen.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index ff6b24ccda..39a5ebe6f5 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -492,6 +492,12 @@ class SignatureGeneratorUniHashMixIn(object): self._client = hashserv.create_client(self.server) return self._client + def reset(self): + if getattr(self, '_client', None) is not None: + self._client.close() + self._client = None + return super().reset() + def exit(self): if getattr(self, '_client', None) is not None: self._client.close() From patchwork Fri Mar 25 09:29:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AEC7C433EF for ; Fri, 25 Mar 2022 09:29:18 +0000 (UTC) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mx.groups.io with SMTP id smtpd.web10.3103.1648200556822153533 for ; Fri, 25 Mar 2022 02:29:17 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=caVKy5nS; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f44.google.com with SMTP id t11so10025130wrm.5 for ; Fri, 25 Mar 2022 02:29:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2i1V81xIb+SLb5DJbFAyAjfhSiQZMK7KmDRHGM2gIYk=; b=caVKy5nS2GlIrH4uDgZnjysAbPYpLzq3elGBdWQ+cpeFSGQ15bWYtENoRQhg13ntYd haiPsrJwM54XTGzcWAggarTl/AaHk+n7mBhjjxdmza6HlLRM6vdd7+BZzfKdNLdYZI5r Wiz51NrCl5HSLWFqXaD3Gonnvd/BZP8lED3gU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2i1V81xIb+SLb5DJbFAyAjfhSiQZMK7KmDRHGM2gIYk=; b=1AvNXk8ogD4L6mE1OwYvj/6TeMBLWDrVc8CIy1/weswgoNqMQtB2D2jq6kucgM4U9L yPYyrDESfhoz1L7IQjdvnYBp829H3AyhOt9ASPnGOFwV1IaoEVa+xhX/7ct8JgRUQzWy fTEAyH63EUruBNPE8Pf6A/RfoPzuGj90iHg+RmCEKaVXwzNPQSnUUBKb+samWj3UPCXH X1/sIq+nuJcdF/jR6I1E2poUhrGOV1fWTJKXdv5lBkYwa13UDsR5InIvsr8nVLh695gp BU4S6qS1cyF2xcJ3rFh38OwHDxhmuKuTvJvsJHGfqPQvt0BseYHDtJElMqWhy42Doz8A d2Kg== X-Gm-Message-State: AOAM5331pBmCHNw7s3VZ1IYJ6Ni+SLO2yk6BVnK1/XEN+BKk8nZJaB+5 p/kaZUnlCiMBGmGFTZ7kWOoG4kJwUa028t2E X-Google-Smtp-Source: ABdhPJxm2d3UBqyeaHG5z0T1YEaSmJt6YPzGenEf2EaShmZFXg63byMHtefYN/gaOZGRDtHIq8Szfw== X-Received: by 2002:adf:ec04:0:b0:205:8537:af5c with SMTP id x4-20020adfec04000000b002058537af5cmr8066017wrn.357.1648200555052; Fri, 25 Mar 2022 02:29:15 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:f167:7010:a0cc:679f]) by smtp.gmail.com with ESMTPSA id p8-20020a5d4e08000000b002054b5437f2sm4478904wrt.115.2022.03.25.02.29.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 02:29:14 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/3] server/process: Move threads left debug to after cooker shutdown Date: Fri, 25 Mar 2022 09:29:12 +0000 Message-Id: <20220325092912.1350321-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220325092912.1350321-1-richard.purdie@linuxfoundation.org> References: <20220325092912.1350321-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 25 Mar 2022 09:29:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13505 This debug is useful but the cooker shutdown or post_serve() may have cleanup left so run after those. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 1636616660..efc3f04b4c 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -243,9 +243,6 @@ class ProcessServer(): ready = self.idle_commands(.1, fds) - if len(threading.enumerate()) != 1: - serverlog("More than one thread left?: " + str(threading.enumerate())) - serverlog("Exiting") # Remove the socket file so we don't get any more connections to avoid races try: @@ -263,6 +260,9 @@ class ProcessServer(): self.cooker.post_serve() + if len(threading.enumerate()) != 1: + serverlog("More than one thread left?: " + str(threading.enumerate())) + # Flush logs before we release the lock sys.stdout.flush() sys.stderr.flush()