From patchwork Tue Dec 20 15:15:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 16999 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 253CBC3DA7A for ; Tue, 20 Dec 2022 15:16:14 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web11.52655.1671549367456309891 for ; Tue, 20 Dec 2022 07:16:07 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=LB0Q0SSZ; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.52, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f52.google.com with SMTP id h11so11998441wrw.13 for ; Tue, 20 Dec 2022 07:16:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=rjUR8wW045Dmbgzbvu2jqdBO9AgPEXWTs4vUQPlmhSM=; b=LB0Q0SSZIxOTtCOF8k/6VGrReA0FQTCD4zzClpIMAKHrmjVTnMO8PeVS9x35y7dbP3 ziZHL6Lt/OnEgY2SZahfKTwSesMcXJoBLdSmrU8dW++t5oWKQEL9Fbv0C6cp6P7H+ch8 SOCzSddXGu+6AQiVneO6xZge/hy2lgWL4+CqU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rjUR8wW045Dmbgzbvu2jqdBO9AgPEXWTs4vUQPlmhSM=; b=CML3i7vbwyotnv0YmLllf9Ws9O+mnr0LcHEo1Eh4VDze0RpX7/FLgiheL7u9G17eDe ZkvG0B9jkpCr/FTRVV7jgZYp4xnW6wdROUKYRofzPBn+z4TDILXKSZCDJLOPoH6JwlAd E1U83rjJNor/ePPvXVLlKAOPF/m9QMF841UqyshpflCh1Hrib/AK+8h63mOG8VbzXGGO sTN4sLaXa3lWQCiB6xLPoZ+tuAtbga1a+tCxu5MfZzedHOqEJ7vMiMnY3D5rI17WoJCI 8eAq0QH+6NU2PCR82n37ehW2/0ApOQr0oLuSnUt4sYbpXBJfDUZroIMpXzRsPQ2Mb9VR pTKQ== X-Gm-Message-State: ANoB5pmVln8clCj4OYdokj3jtQv8DDMcKAVg4dXic5C0VKvoLHblrfWn x6J1x4MCgDHLxAvC54q9femMUqc3Ww63IVxi X-Google-Smtp-Source: AA0mqf62pvfH5XVoRTVA72H4lmmp43BqeNdRWQpiHEBdVvsB26uuTi2QlOOqGZox5pYGidXO2PFsdA== X-Received: by 2002:a5d:4e04:0:b0:242:4f41:2efc with SMTP id p4-20020a5d4e04000000b002424f412efcmr44880009wrt.35.1671549365173; Tue, 20 Dec 2022 07:16:05 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1273:dbb7:632c:a9de]) by smtp.gmail.com with ESMTPSA id k12-20020adff28c000000b0022e57e66824sm14896439wro.99.2022.12.20.07.16.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 07:16:04 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/6] knotty: Ping the server/cooker periodically Date: Tue, 20 Dec 2022 15:15:59 +0000 Message-Id: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 20 Dec 2022 15:16:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14210 We've seeing failures where the UI hangs if the server disappears. Ping the cooker/server if we've not had any events in the last minute so we can check if it is still alive. Signed-off-by: Richard Purdie --- lib/bb/ui/knotty.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index 61cf0a37f4..ab1a367be0 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -625,7 +625,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): printintervaldelta = 10 * 60 # 10 minutes printinterval = printintervaldelta - lastprint = time.time() + pinginterval = 1 * 60 # 1 minutes + lastevent = lastprint = time.time() termfilter = tf(main, helper, console_handlers, params.options.quiet) atexit.register(termfilter.finish) @@ -637,6 +638,14 @@ def main(server, eventHandler, params, tf = TerminalFilter): printinterval += printintervaldelta event = eventHandler.waitEvent(0) if event is None: + if (lastevent + pinginterval) <= time.time(): + ret, error = server.runCommand(["ping"]) + if error or not ret: + termfilter.clearFooter() + print("No reply after pinging server (%s, %s), exiting." % (str(error), str(ret))) + return_value = 3 + main.shutdown = 2 + lastevent = time.time() if main.shutdown > 1: break if not parseprogress: @@ -644,6 +653,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): event = eventHandler.waitEvent(0.25) if event is None: continue + lastevent = time.time() helper.eventHandler(event) if isinstance(event, bb.runqueue.runQueueExitWait): if not main.shutdown: From patchwork Tue Dec 20 15:16:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 16994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E56A8C10F1E for ; Tue, 20 Dec 2022 15:16:13 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web11.52654.1671549367414948382 for ; Tue, 20 Dec 2022 07:16:07 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=YKpvi/QX; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f43.google.com with SMTP id ja17so8978396wmb.3 for ; Tue, 20 Dec 2022 07:16:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=JLt2c4IDRxacKvW3U0bB7ZZwge/xqtMUqoLIHNZIJRQ=; b=YKpvi/QXCm89Lavr90imme0+xcco9isbUnzOLSki+N7JmP+NH7xIZhcvc2GMDskymY NIbQ3SFa3HZ8JUIs3uRKw1p/kcLguQNRa9P8J2jXpISbHxv3/eeNAzatxnkoooPRMbc5 WqS4kkIirwMULrhdCz1JpvhrHenENLPGcdB0w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JLt2c4IDRxacKvW3U0bB7ZZwge/xqtMUqoLIHNZIJRQ=; b=GAsPtULq2o1l1RfsVV0EBeOW8U1tuJGL9+BhkQhrhaFLG5UZhWn3TGhxKy481oC29Q eGDBXN/RY3DIfAFkEB42yV1KYDDJFrL1tT0B+HeV1lKL3P0++Sg5X8GNUqJeOXDa8eUn qKNIUSQcwGrYXYfr+/jytJxVhGNHoIY6zCBfazapMrpBwGo0wvLS/n86z6DbfLyzm1mO +0hAgKoN/gqmdq4OnaCV6+y/fPcu+2YX9tx12T6nmZd61DeDfLruuguJCQ6VElDtY9sl wDpk3RJqYJ5Ha6jEXhzvEpoMr3FYbn/NR5fkkS6Urjzkvx+K8H13nnx2lyfd8CbW/APK hCUg== X-Gm-Message-State: ANoB5pm0ut+1k55s4euLB2Ughy+sSzjATh/YXznpQDkdLI2MDio1/gOJ hCs/skZCooxi4osqHmkSChNTXy4klpspJ/o1 X-Google-Smtp-Source: AA0mqf5X4yyryzY7OsjJviMdXz127UCMOq7y9g+azzzYItkZWQ5kyC3Tx2OkxiA13VBCZql1SAUquw== X-Received: by 2002:a05:600c:26cb:b0:3d2:1d51:246e with SMTP id 11-20020a05600c26cb00b003d21d51246emr29475268wmv.9.1671549365719; Tue, 20 Dec 2022 07:16:05 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1273:dbb7:632c:a9de]) by smtp.gmail.com with ESMTPSA id k12-20020adff28c000000b0022e57e66824sm14896439wro.99.2022.12.20.07.16.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 07:16:05 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/6] siggen: Minor code improvement Date: Tue, 20 Dec 2022 15:16:00 +0000 Message-Id: <20221220151604.415637-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> References: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 20 Dec 2022 15:16:13 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14209 Tweak the code to remove duplication and only set if the attribute isn't already there to avoid overwriting. Signed-off-by: Richard Purdie --- lib/bb/siggen.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index f551c74890..0e79404f76 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -273,11 +273,11 @@ class SignatureGeneratorBasic(SignatureGenerator): super().setup_datacache_from_datastore(mcfn, d) mc = bb.runqueue.mc_from_tid(mcfn) - setattr(self.datacaches[mc], "siggen_varvals", {}) + for attr in ["siggen_varvals", "siggen_taskdeps", "siggen_gendeps"]: + if not hasattr(self.datacaches[mc], attr): + setattr(self.datacaches[mc], attr, {}) self.datacaches[mc].siggen_varvals[mcfn] = d.getVar("__siggen_varvals") - setattr(self.datacaches[mc], "siggen_taskdeps", {}) self.datacaches[mc].siggen_taskdeps[mcfn] = d.getVar("__siggen_taskdeps") - setattr(self.datacaches[mc], "siggen_gendeps", {}) self.datacaches[mc].siggen_gendeps[mcfn] = d.getVar("__siggen_gendeps") def rundep_check(self, fn, recipename, task, dep, depname, dataCaches): From patchwork Tue Dec 20 15:16:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 16996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03ACEC46467 for ; Tue, 20 Dec 2022 15:16:14 +0000 (UTC) Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by mx.groups.io with SMTP id smtpd.web11.52656.1671549368092364893 for ; Tue, 20 Dec 2022 07:16:08 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=BlqxxWZR; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.49, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f49.google.com with SMTP id w15so12025917wrl.9 for ; Tue, 20 Dec 2022 07:16:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Xx4BDvpCKiU4eAx3ImV9AdAbqu4ntwxpUH0i1NpX9no=; b=BlqxxWZRf6BglZBupIZWTqZdsCBVJqp9SJwP9HSXwQzL8ggoYKTjNSHSrvbdN4beey IYwtuykdmBpWBiaRRKxfkzIwxCgovT32p83WrbGzQE4cfcwoaV4+KIXAt0VTxIrdSGq2 eIOZ2vuR4WTfCKkuj0u9cyzj0RYLDtVJKs2SE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xx4BDvpCKiU4eAx3ImV9AdAbqu4ntwxpUH0i1NpX9no=; b=GdxIvbFzE0aeAYCZRhjf7M2ubHJrhujbeg9c2ICRq9t6jdXE9i6QKRBsVnGO5xCtrj V7kayLasiuUgRDcP9ws+ytwWdywOgAf7byTdKXOIjGz123VMONOOJnzdKnfMu5KNvgSP d8kSdYRB+ID8ArGRSQGqgj1dVEm51wLeLsSyHMWwpoYEM9mPoktx6d7bL9GaHOTBxoFi 9fwZqCPLhRIRV/UYasYze1a/RJ2A+fqx9zNQpyG61FyKvVE3RzmF/TvkpgNTsa2vkA16 bLP8xYy9ZpE5nKNgq/k7hwrjr7oMzgaVhwKYVR3nf0sPKusa4UmvRG3Mo2MVbfpLKKS1 Syzg== X-Gm-Message-State: AFqh2kpIQ3V8k/myXDu7iS345Y9PwAE2jxOtOdoBTXLxyxw5UrNaIxPU RoFWJbRtRItbXNTfue5kftM4l9uZ7mI2I3jx X-Google-Smtp-Source: AMrXdXu2sn6JnFORh6PvkAmq2CR5I0EBvUa0i8BT6woUaMP3pcQ+X7s+jmoJi1bgapkQV0RGyvvcQw== X-Received: by 2002:a5d:5c0d:0:b0:242:1809:7e13 with SMTP id cc13-20020a5d5c0d000000b0024218097e13mr1896225wrb.11.1671549366271; Tue, 20 Dec 2022 07:16:06 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1273:dbb7:632c:a9de]) by smtp.gmail.com with ESMTPSA id k12-20020adff28c000000b0022e57e66824sm14896439wro.99.2022.12.20.07.16.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 07:16:05 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/6] event: Add enable/disable heartbeat code Date: Tue, 20 Dec 2022 15:16:01 +0000 Message-Id: <20221220151604.415637-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> References: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 20 Dec 2022 15:16:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14211 Currently heartbeat events are always generated by the server whilst it is active. Change this so they only appear when builds are running, which is when most code would expect to be executed. This removes a number of races around changes in the datastore which can happen outside of builds. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 4 ++++ lib/bb/event.py | 9 +++++++++ lib/bb/server/process.py | 4 ++-- 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 48c3002ce3..815610ff82 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1467,6 +1467,7 @@ class BBCooker: buildname = self.databuilder.mcdata[mc].getVar("BUILDNAME") if fireevents: bb.event.fire(bb.event.BuildStarted(buildname, [item]), self.databuilder.mcdata[mc]) + bb.event.enable_heartbeat() # Execute the runqueue runlist = [[mc, item, task, fn]] @@ -1500,6 +1501,7 @@ class BBCooker: if not retval: if fireevents: bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, item, failures, interrupted), self.databuilder.mcdata[mc]) + bb.event.disable_heartbeat() self.command.finishAsyncCommand(msg) # We trashed self.recipecaches above self.parsecache_valid = False @@ -1545,6 +1547,7 @@ class BBCooker: for mc in self.multiconfigs: bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, targets, failures, interrupted), self.databuilder.mcdata[mc]) finally: + bb.event.disable_heartbeat() self.command.finishAsyncCommand(msg) return False if retval is True: @@ -1578,6 +1581,7 @@ class BBCooker: for mc in self.multiconfigs: bb.event.fire(bb.event.BuildStarted(buildname, ntargets), self.databuilder.mcdata[mc]) + bb.event.enable_heartbeat() rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) if 'universe' in targets: diff --git a/lib/bb/event.py b/lib/bb/event.py index 97668601a1..21e9a1b025 100644 --- a/lib/bb/event.py +++ b/lib/bb/event.py @@ -69,6 +69,7 @@ _eventfilter = None _uiready = False _thread_lock = threading.Lock() _thread_lock_enabled = False +_heartbeat_enabled = False if hasattr(__builtins__, '__setitem__'): builtins = __builtins__ @@ -83,6 +84,14 @@ def disable_threadlock(): global _thread_lock_enabled _thread_lock_enabled = False +def enable_heartbeat(): + global _heartbeat_enabled + _heartbeat_enabled = True + +def disable_heartbeat(): + global _heartbeat_enabled + _heartbeat_enabled = False + def execute_handler(name, handler, event, d): event.data = d addedd = False diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 12498a6c47..826f6f52cc 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -372,7 +372,7 @@ class ProcessServer(): # Create new heartbeat event? now = time.time() - if now >= self.next_heartbeat: + if bb.event._heartbeat_enabled and now >= self.next_heartbeat: # We might have missed heartbeats. Just trigger once in # that case and continue after the usual delay. self.next_heartbeat += self.heartbeat_seconds @@ -386,7 +386,7 @@ class ProcessServer(): if not isinstance(exc, bb.BBHandledException): logger.exception('Running heartbeat function') self.quit = True - if nextsleep and now + nextsleep > self.next_heartbeat: + if nextsleep and bb.event._heartbeat_enabled and now + nextsleep > self.next_heartbeat: # Shorten timeout so that we we wake up in time for # the heartbeat. nextsleep = self.next_heartbeat - now From patchwork Tue Dec 20 15:16:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 16997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05B9BC3DA6E for ; Tue, 20 Dec 2022 15:16:14 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web10.52462.1671549368572209284 for ; Tue, 20 Dec 2022 07:16:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=AJ9XqqzJ; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f54.google.com with SMTP id m14so12030556wrh.7 for ; Tue, 20 Dec 2022 07:16:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=LjHs9eTMVzPr08ed8b/HNdgt6n8GV7WJj79epKR6no8=; b=AJ9XqqzJsqrWRmuR8YAlBxanu6C3clRuZQ2GcCECZmi9NYg7JrKBAWrX8ak3DY0weD bo58L0u9rMV+FuSk8zBGfm/lgyhc0xm9KzIzpGW21BPHzXg7FoAvwXsRcrWQpGW2aZVD TVejJbquWGwKMQhmjUY+3dL8W1t4MFWl6bBw0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjHs9eTMVzPr08ed8b/HNdgt6n8GV7WJj79epKR6no8=; b=TWTp9b+LOcekI/BmZn/KPQf3corgu0FUw/TZ775nOaRQZR6kMbXD3SokhmhDVgUCoE Ne3sicrAWN6P59BwzAHdtHtLLUoFdIIY3L3zbEaAALnwPq9WDpbQG+FBgzuagCGQR7wp 4Mow9rEqBHYIdBv9we61Z8l+TanH88bWvJUfcm0aH+QFgTzvueg6e93BFRasnqfTL3X6 yNQtn+WK4/seg8TJ9c2ANILRqnJXRPnA/ncqmcCwpnv1/XEn5zB8wxUOU0LbxKoaYWgS zK81Vgyy1JkgNJJ3+JYOdo9yKAYoYV+iRuBK1qcEnS0sqJVsELbKtV8dTQmCLPNp1wTP FsyA== X-Gm-Message-State: AFqh2kq0CE477X520WCGvb+I/JgRjT1T7KUVjvwgdEvOwroGyNqWkYar k5ABey4jBOAml+xM6pz3CF6a5QpKk5pK+Kec X-Google-Smtp-Source: AMrXdXvUugRA1viWpjj7xTQYLhFLGGIxZ7c4Rfrd25nzoo+W+qgirAOBhNaFfarpUZuBjya5w0sVSw== X-Received: by 2002:a5d:58f8:0:b0:241:ee8d:f818 with SMTP id f24-20020a5d58f8000000b00241ee8df818mr1927715wrd.46.1671549366867; Tue, 20 Dec 2022 07:16:06 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1273:dbb7:632c:a9de]) by smtp.gmail.com with ESMTPSA id k12-20020adff28c000000b0022e57e66824sm14896439wro.99.2022.12.20.07.16.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 07:16:06 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/6] event: builtins fix for 'd' deletion Date: Tue, 20 Dec 2022 15:16:02 +0000 Message-Id: <20221220151604.415637-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> References: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 20 Dec 2022 15:16:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14212 I've been seeing event handlers where 'd' seems to disappear half way through event handler execution. This is problematic when multiple threads are active since this code assumes single threading. Add in a recursive lock to avoid d being deleted from a context it shouldn't be. Signed-off-by: Richard Purdie --- lib/bb/event.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/bb/event.py b/lib/bb/event.py index 21e9a1b025..5df5b3a3ee 100644 --- a/lib/bb/event.py +++ b/lib/bb/event.py @@ -68,6 +68,7 @@ _catchall_handlers = {} _eventfilter = None _uiready = False _thread_lock = threading.Lock() +_builtins_lock = threading.RLock() _thread_lock_enabled = False _heartbeat_enabled = False @@ -93,6 +94,9 @@ def disable_heartbeat(): _heartbeat_enabled = False def execute_handler(name, handler, event, d): + if _thread_lock_enabled: + _builtins_lock.acquire() + event.data = d addedd = False if 'd' not in builtins: @@ -115,6 +119,9 @@ def execute_handler(name, handler, event, d): del event.data if addedd: del builtins['d'] + if _thread_lock_enabled: + _builtins_lock.release() + def fire_class_handlers(event, d): if isinstance(event, logging.LogRecord): From patchwork Tue Dec 20 15:16:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 16995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E51EBC4332F for ; Tue, 20 Dec 2022 15:16:13 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web10.52463.1671549369341410535 for ; Tue, 20 Dec 2022 07:16:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=hOquWpNu; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f54.google.com with SMTP id b24-20020a05600c4a9800b003d21efdd61dso9057807wmp.3 for ; Tue, 20 Dec 2022 07:16:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=5/QkP+R7JzhaP4NO32Ri8AK0ASIbzYTO8+Qrkj6UJp8=; b=hOquWpNuhozB36VlZN/rYtBmoJ7UGa4YzLQW7jLtEFbUPcakAVGRrDLOhZXbToTKH8 wcVkD0VVqtbgJ9wU4QELGo4w0RUlpASOK/St3VF5uohUqYFL+z822SGcSDQULuiwMgik 1A7jrRebhsvtGLf5aQb5+bleqgyWuGo0IIzwQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5/QkP+R7JzhaP4NO32Ri8AK0ASIbzYTO8+Qrkj6UJp8=; b=FOdTEd1HXSxnUhzgpuAkFc4xmgUJNJR345WfSHlinwHfBKlTSEetCnRJIcKv2xMrL/ 6PuXmCxFZwE3kbxyhl+W7J8sSpZ6jFX1s5SpxX3qNziGdLuJZuikFKM1aj5Q9qOcMZS7 z6vgKiFhvvroZUgA9MPetYnUcwBSLWac7I4StjhPH1ZB3A+BdJkh94OK7chfv92Ec8p/ MnP78ubMtJzg2jw6lzwcej5XDz0Zpli7TxNareGXO4nbUDWq6VbcXDwDUu/3iqYdYW3p NhkFOZv7brOqZ7sGs1W43iHNv0fM+4riT4sdzPyUu6cSGcTaOLAhRKbGfV6JwADlF4iz z/gA== X-Gm-Message-State: ANoB5pke9zhE0njMaTqBhWkq0t2kwyfvJjP26bRTkn9cffsxGlx9TCUz hVq9bg+Ox7Isj17aP30vtOcwZYTBrubPQYJM X-Google-Smtp-Source: AA0mqf4UReaiqq8g6dUY4ZVG1hl2HPu4qEBPRpU2MYPkwDLgmKp9HArplMbXZeGkAgGomYpHtO5jsg== X-Received: by 2002:a05:600c:3b1b:b0:3d1:fe12:fe34 with SMTP id m27-20020a05600c3b1b00b003d1fe12fe34mr35443199wms.39.1671549367518; Tue, 20 Dec 2022 07:16:07 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1273:dbb7:632c:a9de]) by smtp.gmail.com with ESMTPSA id k12-20020adff28c000000b0022e57e66824sm14896439wro.99.2022.12.20.07.16.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 07:16:07 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 5/6] server/process: Run idle commands in a separate idle thread Date: Tue, 20 Dec 2022 15:16:03 +0000 Message-Id: <20221220151604.415637-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> References: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 20 Dec 2022 15:16:13 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14213 When bitbake is off running heavier "idle" commands, it doesn't service it's command socket which means stopping/interrupting it is hard. It also means we can't "ping" from the UI to know if it is still alive. For those reasons, split idle command execution into it's own thread. The commands are generally already self containted so this is easier than expected. We do have to be careful to only handle inotify poll() from a single thread at a time. It also means we always have to use a thread lock when sending events since both the idle thread and the command thread may generate log messages (and hence events). The patch does depend on a couple of previous fixes to the builtins locking in event.py and the heartbeat enable/disable changes. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 13 +++++---- lib/bb/server/process.py | 60 +++++++++++++++++++++++++--------------- 2 files changed, 46 insertions(+), 27 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 815610ff82..f578d2ceea 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -220,6 +220,8 @@ class BBCooker: bb.debug(1, "BBCooker startup complete %s" % time.time()) sys.stdout.flush() + self.inotify_threadlock = threading.Lock() + def init_configdata(self): if not hasattr(self, "data"): self.initConfigurationData() @@ -248,11 +250,12 @@ class BBCooker: self.notifier = pyinotify.Notifier(self.watcher, self.notifications) def process_inotify_updates(self): - for n in [self.confignotifier, self.notifier]: - if n and n.check_events(timeout=0): - # read notified events and enqueue them - n.read_events() - n.process_events() + with self.inotify_threadlock: + for n in [self.confignotifier, self.notifier]: + if n and n.check_events(timeout=0): + # read notified events and enqueue them + n.read_events() + n.process_events() def config_notifications(self, event): if event.maskname == "IN_Q_OVERFLOW": diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 826f6f52cc..95e2968742 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -87,6 +87,7 @@ class ProcessServer(): self.maxuiwait = 30 self.xmlrpc = False + self.idle = None self._idlefuns = {} self.bitbake_lock = lock @@ -144,6 +145,7 @@ class ProcessServer(): self.cooker.pre_serve() bb.utils.set_process_name("Cooker") + bb.event.enable_threadlock() ready = [] newconnections = [] @@ -274,6 +276,9 @@ class ProcessServer(): ready = self.idle_commands(.1, fds) + if self.idle: + self.idle.join() + serverlog("Exiting (socket: %s)" % os.path.exists(self.sockname)) # Remove the socket file so we don't get any more connections to avoid races try: @@ -342,33 +347,44 @@ class ProcessServer(): msg.append(":\n%s" % procs) serverlog("".join(msg)) + def idle_thread(self): + while not self.quit: + nextsleep = 0.1 + fds = [] + for function, data in list(self._idlefuns.items()): + try: + retval = function(self, data, False) + if retval is False: + del self._idlefuns[function] + nextsleep = None + elif retval is True: + nextsleep = None + elif isinstance(retval, float) and nextsleep: + if (retval < nextsleep): + nextsleep = retval + elif nextsleep is None: + continue + else: + fds = fds + retval + except SystemExit: + raise + except Exception as exc: + if not isinstance(exc, bb.BBHandledException): + logger.exception('Running idle function') + del self._idlefuns[function] + self.quit = True + + if nextsleep is not None: + select.select(fds,[],[],nextsleep)[0] + def idle_commands(self, delay, fds=None): nextsleep = delay if not fds: fds = [] - for function, data in list(self._idlefuns.items()): - try: - retval = function(self, data, False) - if retval is False: - del self._idlefuns[function] - nextsleep = None - elif retval is True: - nextsleep = None - elif isinstance(retval, float) and nextsleep: - if (retval < nextsleep): - nextsleep = retval - elif nextsleep is None: - continue - else: - fds = fds + retval - except SystemExit: - raise - except Exception as exc: - if not isinstance(exc, bb.BBHandledException): - logger.exception('Running idle function') - del self._idlefuns[function] - self.quit = True + if not self.idle: + self.idle = threading.Thread(target=self.idle_thread) + self.idle.start() # Create new heartbeat event? now = time.time() From patchwork Tue Dec 20 15:16:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 16998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01DC4C4167B for ; Tue, 20 Dec 2022 15:16:14 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web11.52658.1671549369839708108 for ; Tue, 20 Dec 2022 07:16:10 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=hYOElbxk; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f54.google.com with SMTP id f18so12053178wrj.5 for ; Tue, 20 Dec 2022 07:16:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=OJX9Z3oUFK8oiNNWZ5g2cehZXEB0ktG3pm9Lqncclew=; b=hYOElbxkF0B/NG7sCKNutJEM1KIhGXyZ2cQ6nZmcrtjsKLOFJs2ZarzNRFENswcgqF ueXGgu8ZL8QRmvO6+K/NdJYKLk8Thd98eDsga1cbhHL7ELda0MfeceVoFc4diMN/q4EY VG1+kD0kBVv8Sy2P7rIZ/uf08UlT1BlJxWBUU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OJX9Z3oUFK8oiNNWZ5g2cehZXEB0ktG3pm9Lqncclew=; b=oiblem9ZkLSTvY7e4dpyxu3ilX9WMBBRVs6PCWj8dUYjdmHD3RFUIUaSv2QG0CjwZy QwBepxraApQUQTfaGIv2Wd3M722LMsRjA8PvGKCwLY76NKpBOnfvsJyHb+S8zh68LuDB juWeBLBk5jHRZscaqFaPtY+fOpjZQDXdsRMcTiKstaaGnSIIPATmO3XjZ+evk3r6h8jz wsBWEz1aD2LEI1gyiwfshLx9KhqI/DybrYiZhHnzu/3h3Lsb+4MfzKxt6/eeRIHMH7mJ zkKrot//ULe7i93SZp4aK9upe75w2nScDHfQG2tL59ePlnbwPkQLBlSk50xzDFlnir+0 tEJQ== X-Gm-Message-State: ANoB5pnMnl762NNGJBuLH8vJ7gDPqEp8nqU/CLFTVKkPQ9HkqmR/H4GP odwdagSzOzgaDChXaYCay1qEwlfbPKBy/b3t X-Google-Smtp-Source: AA0mqf6A1U5Yp9w8nR8yVIeflV++ALRGENyL0c30XbWZq6n0VW/QP+oP9QT3aVkOPo7CJ67cwpBqbw== X-Received: by 2002:a5d:526a:0:b0:242:877d:ca63 with SMTP id l10-20020a5d526a000000b00242877dca63mr29886983wrc.44.1671549368156; Tue, 20 Dec 2022 07:16:08 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1273:dbb7:632c:a9de]) by smtp.gmail.com with ESMTPSA id k12-20020adff28c000000b0022e57e66824sm14896439wro.99.2022.12.20.07.16.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 07:16:07 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 6/6] server/process: Add bitbake.sock race handling Date: Tue, 20 Dec 2022 15:16:04 +0000 Message-Id: <20221220151604.415637-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> References: <20221220151604.415637-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 20 Dec 2022 15:16:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14214 We've seen cases where the bitbake.sock file appears to disappear but the server continues to hold bitbake.lock. The most likely explaination is that some previous build directory was moved out the way, a server there kept running, eventually exited and removed the sock file from the wrong directory. To guard against this, save the inode information for the sock file and check it before deleting the file. The new code isn't entirely race free but should guard against what is a rare but annoying potential issue. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 95e2968742..6f43330fae 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -28,6 +28,7 @@ import datetime import pickle import traceback import gc +import stat import bb.server.xmlrpcserver from bb import daemonize from multiprocessing import queues @@ -94,6 +95,9 @@ class ProcessServer(): self.bitbake_lock_name = lockname self.sock = sock self.sockname = sockname + # It is possible the directory may be renamed. Cache the inode of the socket file + # so we can tell if things changed. + self.sockinode = os.stat(self.sockname)[stat.ST_INO] self.server_timeout = server_timeout self.timeout = self.server_timeout @@ -281,8 +285,14 @@ class ProcessServer(): serverlog("Exiting (socket: %s)" % os.path.exists(self.sockname)) # Remove the socket file so we don't get any more connections to avoid races + # The build directory could have been renamed so if the file isn't the one we created + # we shouldn't delete it. try: - os.unlink(self.sockname) + sockinode = os.stat(self.sockname)[stat.ST_INO] + if sockinode == self.sockinode: + os.unlink(self.sockname) + else: + serverlog("bitbake.sock inode mismatch (%s vs %s), not deleting." % (sockinode, self.sockinode)) except Exception as err: serverlog("Removing socket file '%s' failed (%s)" % (self.sockname, err)) self.sock.close()