From patchwork Sat Dec 31 16:58:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9F28C54EBE for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx.groups.io with SMTP id smtpd.web11.45611.1672505933112061040 for ; Sat, 31 Dec 2022 08:58:53 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=TyjU3+vx; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.51, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f51.google.com with SMTP id m8-20020a05600c3b0800b003d96f801c48so15480797wms.0 for ; Sat, 31 Dec 2022 08:58:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=I8OzwOY1eyomQtXLHt7a5zwI4hGjNsSkPlz0lOUSUKk=; b=TyjU3+vx7KUrawu9UEH5/jl36sZw1NQt/7t3c8cAzawRsmT5KWp64nOhtxxbLX7SxL wiYPOvPPSKjKm8uJftYea5vyrgIfOxNWbUEM8J9WeCdgQRlXJqAcDq4BQgECIKBHqrE8 qiiMvKAtbv2/MPFaphba7zvuU/d17IUxf74Aw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I8OzwOY1eyomQtXLHt7a5zwI4hGjNsSkPlz0lOUSUKk=; b=c2t+IBNjRk+ckYBVcFTLh/7zBeStPb+wuKM4dCeQTW6dfo+p1AeTWsYdB9vkXdEnFK 9Skb83on6eAbvCqPwrYRNcc9cbilicHipYLvIRWtkfpIA2GmQfsujSZWKGb5ZHUZvmj1 TJotCk+SwVIt+2Un0ajG+f8Up966IB48RMh5rbH60NlnS2Q11YUS2D+B9kuch1WMxXj1 cF5TW5LzX9zZX0HyeWucIGfvzFsgWfl7a52II/MPfSkYtUcSdmm6/4+rxTMVPEIlHdNU RDS0DW3FHxMv/9cBB5TRm0zJWZ5bIIwPexWYeI0InbkKKPEOf5AbJjpxjf6DEchUcZPR J9dg== X-Gm-Message-State: AFqh2kqIs/xPNW+u/hxMTc8AzbGtQotDxOWSXKf4b2NpcHhk4liWb328 dN2IJYJ0ui7P9aeVIA2DZz+CjXptypTOJwsw X-Google-Smtp-Source: AMrXdXu74B8arCwBzbxJS8dunIfnTdkFi3p7curPPzl9bpdV6wo9+EK5R44UO3/Xf1JiAp73C6MNeQ== X-Received: by 2002:a05:600c:5c8:b0:3d1:4145:b3b with SMTP id p8-20020a05600c05c800b003d141450b3bmr25892342wmd.9.1672505931434; Sat, 31 Dec 2022 08:58:51 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:51 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 01/11] knotty: Avoid looping with tracebacks Date: Sat, 31 Dec 2022 16:58:39 +0000 Message-Id: <20221231165849.964169-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14262 If there are events queued and there is an exception in the main loop of the UI code, it will print tracebacks on the console indefinitely. Avoid that by improving the loop exit conditions. Signed-off-by: Richard Purdie --- lib/bb/ui/knotty.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index 61cf0a37f4..761c2dddaf 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -630,15 +630,13 @@ def main(server, eventHandler, params, tf = TerminalFilter): termfilter = tf(main, helper, console_handlers, params.options.quiet) atexit.register(termfilter.finish) - while True: + while main.shutdown < 2: try: if (lastprint + printinterval) <= time.time(): termfilter.keepAlive(printinterval) printinterval += printintervaldelta event = eventHandler.waitEvent(0) if event is None: - if main.shutdown > 1: - break if not parseprogress: termfilter.updateFooter() event = eventHandler.waitEvent(0.25) From patchwork Sat Dec 31 16:58:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAEF1C54EBF for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.45612.1672505933951647862 for ; Sat, 31 Dec 2022 08:58:54 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=H9/VtWMn; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f48.google.com with SMTP id m26-20020a05600c3b1a00b003d9811fcaafso11121358wms.5 for ; Sat, 31 Dec 2022 08:58:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=GWMUJ/cgsztgOARPqY8t9XdFraxGdowgfoqwVsnfCmY=; b=H9/VtWMn31dMjcq6qk7s1p5NmyakmyiX+I4kpK5tfbMXx8dzygL5Sfgu/dIC0xOAmB 4gIPOI1KrFYB9izZC+lF0SvjsohARVZee/A3X89rKLNeRfLq8YI2T9EvOAoQVw4BX+5q Ol/A64EmGSX4XfxYYr5sSKqAd40OP27FQ4kV0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GWMUJ/cgsztgOARPqY8t9XdFraxGdowgfoqwVsnfCmY=; b=CBTkI/UTwcFmBEZKoM1iaeVc+3N+ZV4QxMxJ0AFLsbZ3SdAfJL0Tz7s0/5Tw9Nt8E4 I7iWnYNSKtK0h9EqynO7IFX+Wzm5kvBDbcXG/8xy0k9LxyLg/VAxIUoVlVKVUkSQLl1+ +AHPgpKDKwz0I8HtkwmKi8mTIoGwuc8XXD6cKbadM3K98knBmaYL7xHao5azt1S72T3W i3bC2ZdsPGBKuy+zXUXYlo8q9HPsjvyffOMb6/Lg47zTSXLv0fMliMMKuANzLn3lv50p /z86Ub+SrebsHGbPnN6dnxnduqy0Ja9Au+2rc+wFBfSbnFApJ2VDaO6t48NkxOuIJQ67 IVCg== X-Gm-Message-State: AFqh2kpMx9HxsqMDnkwPBvRtiUVfcUOyFWhh0m94pjEYdoiWo/K2T2Hk BdoAlhFNpxKpmotaPdOydRnkOWU7csw77yQ5 X-Google-Smtp-Source: AMrXdXt758yaELVzWrQGQgEH/m+CNmFO4M6GwkgEuZwH9+eWLD0f8fzCOQ9LA29gn1saQ9XHDt2QGA== X-Received: by 2002:a05:600c:a4d:b0:3cf:6e78:e2ca with SMTP id c13-20020a05600c0a4d00b003cf6e78e2camr28373349wmq.5.1672505932088; Sat, 31 Dec 2022 08:58:52 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:51 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 02/11] event: Add enable/disable heartbeat code Date: Sat, 31 Dec 2022 16:58:40 +0000 Message-Id: <20221231165849.964169-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14263 Currently heartbeat events are always generated by the server whilst it is active. Change this so they only appear when builds are running, which is when most code would expect to be executed. This removes a number of races around changes in the datastore which can happen outside of builds. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 4 ++++ lib/bb/event.py | 9 +++++++++ lib/bb/server/process.py | 4 ++-- 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 4e49b0e90b..c712744382 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1467,6 +1467,7 @@ class BBCooker: buildname = self.databuilder.mcdata[mc].getVar("BUILDNAME") if fireevents: bb.event.fire(bb.event.BuildStarted(buildname, [item]), self.databuilder.mcdata[mc]) + bb.event.enable_heartbeat() # Execute the runqueue runlist = [[mc, item, task, fn]] @@ -1500,6 +1501,7 @@ class BBCooker: if not retval: if fireevents: bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, item, failures, interrupted), self.databuilder.mcdata[mc]) + bb.event.disable_heartbeat() self.command.finishAsyncCommand(msg) # We trashed self.recipecaches above self.parsecache_valid = False @@ -1545,6 +1547,7 @@ class BBCooker: for mc in self.multiconfigs: bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, targets, failures, interrupted), self.databuilder.mcdata[mc]) finally: + bb.event.disable_heartbeat() self.command.finishAsyncCommand(msg) return False if retval is True: @@ -1578,6 +1581,7 @@ class BBCooker: for mc in self.multiconfigs: bb.event.fire(bb.event.BuildStarted(buildname, ntargets), self.databuilder.mcdata[mc]) + bb.event.enable_heartbeat() rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) if 'universe' in targets: diff --git a/lib/bb/event.py b/lib/bb/event.py index 303b7a943f..db90724444 100644 --- a/lib/bb/event.py +++ b/lib/bb/event.py @@ -69,6 +69,7 @@ _eventfilter = None _uiready = False _thread_lock = threading.Lock() _thread_lock_enabled = False +_heartbeat_enabled = False def enable_threadlock(): global _thread_lock_enabled @@ -78,6 +79,14 @@ def disable_threadlock(): global _thread_lock_enabled _thread_lock_enabled = False +def enable_heartbeat(): + global _heartbeat_enabled + _heartbeat_enabled = True + +def disable_heartbeat(): + global _heartbeat_enabled + _heartbeat_enabled = False + def execute_handler(name, handler, event, d): event.data = d try: diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 586d46af88..91eb6e0ad9 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -382,7 +382,7 @@ class ProcessServer(): # Create new heartbeat event? now = time.time() - if now >= self.next_heartbeat: + if bb.event._heartbeat_enabled and now >= self.next_heartbeat: # We might have missed heartbeats. Just trigger once in # that case and continue after the usual delay. self.next_heartbeat += self.heartbeat_seconds @@ -396,7 +396,7 @@ class ProcessServer(): if not isinstance(exc, bb.BBHandledException): logger.exception('Running heartbeat function') self.quit = True - if nextsleep and now + nextsleep > self.next_heartbeat: + if nextsleep and bb.event._heartbeat_enabled and now + nextsleep > self.next_heartbeat: # Shorten timeout so that we we wake up in time for # the heartbeat. nextsleep = self.next_heartbeat - now From patchwork Sat Dec 31 16:58:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABA72C54EF1 for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web11.45613.1672505934355516511 for ; Sat, 31 Dec 2022 08:58:54 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=JM3xObN/; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.42, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f42.google.com with SMTP id b24-20020a05600c4a9800b003d21efdd61dso17659071wmp.3 for ; Sat, 31 Dec 2022 08:58:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=jvrUha3NQoEF2ZHe8pOGBDv3YGKG2quYwV269VUJITA=; b=JM3xObN/oSuWF6R+9sO36Ba5QdMh1eWh+KeiOcLRW28ZTJzw+qiv56c7k+g+PzVOG2 wxmEE+9c+UEEWsHPCnGoqJPJeoLfpd+KbXJmS5BcBmF2vbXPRqs++h2frbnenaCNg0nE 7tBi2VN/c/mlGGL1LS7pYwFWaIZ8o0aKy9dHY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jvrUha3NQoEF2ZHe8pOGBDv3YGKG2quYwV269VUJITA=; b=1MYY80vUPiY2Qr8uGTJn+ayReW8BkrboF/PKKnEGCia8SUa4E0Ia/Oq4XE2TViglFM G/Vkf2ROR/OccEX8U6wvAbzOIMzEpge5qRJ6zeetB2csxGHv08SYFEF6rom5OXSOLWrw xjfpJbOPzwkpqwUA2/K6P/c8CrrrmL/mIFWVE++cJZIrT6lm7Q/oCMvmFv93FPN8okyr EnbDkb/YXm5v/hCNjoVLUlSbcFRc5m0S0jxWQXejBeLRU220G5jRIAT+Iy+1V2nDe1XV 2A54d6zeBTm4MJxKjxJ0DV1vUPqxX5KXCkN60ezZhArnrsF/YnXMiqlJeiYO8aAHg8pL cb/A== X-Gm-Message-State: AFqh2kqZb2samhHBKD6KtIQ6Zeu+Wsd3l7Hcm2M6rJaHCPs0mqR5rfXA zdmUhjvpUl1/K5W/eajpJ9uRG8Gu2TSkBn7N X-Google-Smtp-Source: AMrXdXswfeAYZxiXgYDtNs9sUvd9PWOrw3mha17EnuJ80rIvtSYYVK2OpSTnb4Y6LWpkjJnrkUxxpQ== X-Received: by 2002:a05:600c:1819:b0:3cf:8d51:fae1 with SMTP id n25-20020a05600c181900b003cf8d51fae1mr25483852wmp.0.1672505932603; Sat, 31 Dec 2022 08:58:52 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:52 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 03/11] cooker/cookerdata: Rework the way the datastores are reset Date: Sat, 31 Dec 2022 16:58:41 +0000 Message-Id: <20221231165849.964169-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14264 As far as I could tell, the current code could result in some strange situations where some data was set back to the original data store copy but the multiconfig data was not restored. There are also some changes made to the datastore which did not persist. The data store was also being reset at every client reset, which seems a little excessive if we can reset it to the original condition properly. Move the __depends -> __base_depends rename into databuilder along with the __bbclasstype change so these are saved in the original data. Tweak the databuilder code to be clearer and easier to follow on which copies are where, then save copies of all the mc datastores. Finally, drop the cache invalidation upon reset for the base config as we shouldn't need that now (oe-selftest -r tinfoil works with memory resident bitbake which was the original reproducer requiring that change). Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 9 +++------ lib/bb/cookerdata.py | 31 +++++++++++++++++++------------ 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index c712744382..7e0d6b47bf 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -400,9 +400,7 @@ class BBCooker: self.disableDataTracking() for mc in self.databuilder.mcdata.values(): - mc.renameVar("__depends", "__base_depends") self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher) - mc.setVar("__bbclasstype", "recipe") self.baseconfig_valid = True self.parsecache_valid = False @@ -436,10 +434,8 @@ class BBCooker: upstream=upstream, ) self.hashserv.serve_as_process() - self.data.setVar("BB_HASHSERVE", self.hashservaddr) - self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservaddr) - self.databuilder.data.setVar("BB_HASHSERVE", self.hashservaddr) for mc in self.databuilder.mcdata: + self.databuilder.mcorigdata[mc].setVar("BB_HASHSERVE", self.hashservaddr) self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservaddr) bb.parse.init_parser(self.data) @@ -1788,8 +1784,9 @@ class BBCooker: if hasattr(self, "data"): self.databuilder.reset() self.data = self.databuilder.data + # In theory tinfoil could have modified the base data before parsing, + # ideally need to track if anything did modify the datastore self.parsecache_valid = False - self.baseconfig_valid = False class CookerExit(bb.event.Event): diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py index 650ae05ec9..b4e0c4216b 100644 --- a/lib/bb/cookerdata.py +++ b/lib/bb/cookerdata.py @@ -263,6 +263,7 @@ class CookerDataBuilder(object): self.mcdata = {} def parseBaseConfiguration(self, worker=False): + mcdata = {} data_hash = hashlib.sha256() try: self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles) @@ -288,18 +289,18 @@ class CookerDataBuilder(object): bb.parse.init_parser(self.data) data_hash.update(self.data.get_hash().encode('utf-8')) - self.mcdata[''] = self.data + mcdata[''] = self.data multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split() for config in multiconfig: if config[0].isdigit(): bb.fatal("Multiconfig name '%s' is invalid as multiconfigs cannot start with a digit" % config) - mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config) - bb.event.fire(bb.event.ConfigParsed(), mcdata) - self.mcdata[config] = mcdata - data_hash.update(mcdata.get_hash().encode('utf-8')) + parsed_mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config) + bb.event.fire(bb.event.ConfigParsed(), parsed_mcdata) + mcdata[config] = parsed_mcdata + data_hash.update(parsed_mcdata.get_hash().encode('utf-8')) if multiconfig: - bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data) + bb.event.fire(bb.event.MultiConfigParsed(mcdata), self.data) self.data_hash = data_hash.hexdigest() except (SyntaxError, bb.BBHandledException): @@ -332,17 +333,23 @@ class CookerDataBuilder(object): if issues: raise bb.BBHandledException() + for mc in mcdata: + mcdata[mc].renameVar("__depends", "__base_depends") + mcdata[mc].setVar("__bbclasstype", "recipe") + # Create a copy so we can reset at a later date when UIs disconnect - self.origdata = self.data - self.data = bb.data.createCopy(self.origdata) - self.mcdata[''] = self.data + self.mcorigdata = mcdata + for mc in mcdata: + self.mcdata[mc] = bb.data.createCopy(mcdata[mc]) + self.data = self.mcdata[''] def reset(self): # We may not have run parseBaseConfiguration() yet - if not hasattr(self, 'origdata'): + if not hasattr(self, 'mcorigdata'): return - self.data = bb.data.createCopy(self.origdata) - self.mcdata[''] = self.data + for mc in self.mcorigdata: + self.mcdata[mc] = bb.data.createCopy(self.mcorigdata[mc]) + self.data = self.mcdata[''] def _findLayerConf(self, data): return findConfigFile("bblayers.conf", data) From patchwork Sat Dec 31 16:58:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99BBAC5479D for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web11.45614.1672505934919659975 for ; Sat, 31 Dec 2022 08:58:55 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=TcE9+FxT; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f54.google.com with SMTP id m26-20020a05600c3b1a00b003d9811fcaafso11121378wms.5 for ; Sat, 31 Dec 2022 08:58:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=xzEkTuDCEqi8EJUfsRwlIRpYqBNbkQJ5BumzLYhXEoU=; b=TcE9+FxTlaZENudwsVDuAda6JHm2/1Ctg5n4LulUJG++HcH9wGOXxnZmoIpXZTb8/k EhknZOMV+91xyTF0iYaqcbpRFnaLsBryWnrA8q446Ol97qEg6xVr4YKScfzZKeEksRFk d4N7iu3VMQzyv4FntX6kxK2Mkn1fRByVHCTFg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xzEkTuDCEqi8EJUfsRwlIRpYqBNbkQJ5BumzLYhXEoU=; b=FAc4+lvMlJuUwIjA8yU8Q9LN2r2cDYtqgxBcf0BffXvyZUnNlzkKuVG0G9FS4ueVU/ r5r0b+CCR7oXiKL5bA+5/9JAnZtnicI+uwB2tZd0IeO83h3ykClHOPWS2mWW9isfrA2S zEtVzLXK/KKmCLn0XIHumTnmEvAvyY7yzLnvMOljpruKepKlQ/WOfKoXwZ9MTRL8SIU6 RwnCO0ZVCQ0l4EO+N/HMYecbuNKm5B+H3LHXWNFocoRYIhk2DHTrm6x+S7usXCVXhmOV QzFX5nv0JKC9Dw4U3ols+D7qnalqU065mIHXdcIw6sJkMNUPj7wctzG5y0Pyg/dKMFmM Zdjg== X-Gm-Message-State: AFqh2koQ5CkYqoD9x3BxNFP47NKAkwEiK4dEu95x5TM8MT2vYUF8sh6K MVs8lr6hfdrlarMHYtIYga+LLACn9T9pxOBk X-Google-Smtp-Source: AMrXdXtWaCdtTunV3dzpcFqRp/BtuWTpjylw5S0ZwdAc7b1vibVnNK5fq96d6IJbgNG96a6dSIpkwA== X-Received: by 2002:a05:600c:34ce:b0:3cf:614e:b587 with SMTP id d14-20020a05600c34ce00b003cf614eb587mr27215418wmq.26.1672505933238; Sat, 31 Dec 2022 08:58:53 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:52 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 04/11] server/process: Improve exception and idle function logging Date: Sat, 31 Dec 2022 16:58:42 +0000 Message-Id: <20221231165849.964169-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14265 Currently if the idle functions loop suffers a traceback, it is silently dropped and there is no log message to say what happened. This change at least means the traceback is in the cooker log, making some debugging possible. Add some logging to show when handlers are added/removed to allow a better idea of what the server code is doing from the server log file. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 91eb6e0ad9..eba28ad983 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -106,6 +106,7 @@ class ProcessServer(): """Register a function to be called while the server is idle""" assert hasattr(function, '__call__') self._idlefuns[function] = data + serverlog("Registering idle function %s" % str(function)) def run(self): @@ -361,6 +362,7 @@ class ProcessServer(): try: retval = function(self, data, False) if retval is False: + serverlog("Removing idle function %s" % str(function)) del self._idlefuns[function] nextsleep = None elif retval is True: @@ -378,6 +380,7 @@ class ProcessServer(): if not isinstance(exc, bb.BBHandledException): logger.exception('Running idle function') del self._idlefuns[function] + serverlog("Exception %s broke the idle_thread, exiting" % traceback.format_exc()) self.quit = True # Create new heartbeat event? @@ -395,6 +398,7 @@ class ProcessServer(): except Exception as exc: if not isinstance(exc, bb.BBHandledException): logger.exception('Running heartbeat function') + serverlog("Exception %s broke in idle_commands, exiting" % traceback.format_exc()) self.quit = True if nextsleep and bb.event._heartbeat_enabled and now + nextsleep > self.next_heartbeat: # Shorten timeout so that we we wake up in time for From patchwork Sat Dec 31 16:58:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C463C54EBC for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web10.45723.1672505935437585692 for ; Sat, 31 Dec 2022 08:58:55 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Y48qjxxs; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f48.google.com with SMTP id ay2-20020a05600c1e0200b003d22e3e796dso17682753wmb.0 for ; Sat, 31 Dec 2022 08:58:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Touq52SotAOn0qhLlACHQaacjfMd05+VLjI7h38+LSw=; b=Y48qjxxsqiz13HSaCZGYzbJ40dqb79oH03CZFbHsa1Uc2KNg573NMXozSMshlrBeY5 J6G8AVcfaU7kDSiK1qAWR7Uq9ZbFPgXI3Z67szS2dj+Xn/rlFGvQiaufwRzhKLaW4MXi f70zbBK0bcyI7O18S6GydXTTYjoxSvSIdmZqs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Touq52SotAOn0qhLlACHQaacjfMd05+VLjI7h38+LSw=; b=JTD1Ddf1ix0sTPusCsK9orWrWliwoP3MLYEEIF1FpTGeG0XH8Ok2Qb4aLC0dVLc9vz iGK38cQA5RFjwtdoPla1ACz51HVfXWyPW7m+idquWZV0ehKcOqt0kWTyIweyUFr6AUP2 YI1XlVDINZLr0lzlYTtk7TH5fHd6Ql1TBRc8DS3RKuCGMD1k+JxTHozIRv9h0X4Fgaty Q8AS5KYqZL9oI/oyE0ts4JhUJQ6MmI7Td9GjceR1R/cGKRgX+GmBx0xvRiwPG/Ey+mXq L06G3TlVBWo06hmEKUHNAGYQAodSRwpruBETU4NL3Gk/yQAAzWvWa9JpTij++073tIKZ EGYQ== X-Gm-Message-State: AFqh2koRyRCYoMGwZj/kw1SpmiARzC+cgNMrfoPNrlwU5yQkm0M6pWjS PSXnw1kViCyOC8hm4YFRpHXJyhDsh9/B9dkW X-Google-Smtp-Source: AMrXdXvlZ82VRwX4h9+zeMynx/mmyX3CHFDb7j3Sxnf2+A7qbJRStBOO2hj+1KFpheMR6kw5eszeaw== X-Received: by 2002:a05:600c:3ba4:b0:3cf:d428:21d6 with SMTP id n36-20020a05600c3ba400b003cfd42821d6mr24604956wms.3.1672505933819; Sat, 31 Dec 2022 08:58:53 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:53 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 05/11] command: Tweak finishAsyncCommand ordering to avoid races Date: Sat, 31 Dec 2022 16:58:43 +0000 Message-Id: <20221231165849.964169-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14266 Signed-off-by: Richard Purdie --- lib/bb/command.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/command.py b/lib/bb/command.py index 0208e30ec4..cbac07f516 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -152,8 +152,8 @@ class Command: bb.event.fire(CommandExit(code), self.cooker.data) else: bb.event.fire(CommandCompleted(), self.cooker.data) - self.currentAsyncCommand = None self.cooker.finishcommand() + self.currentAsyncCommand = None def reset(self): if self.remotedatastores: From patchwork Sat Dec 31 16:58:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C2A2C3DA7A for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx.groups.io with SMTP id smtpd.web10.45725.1672505936049422241 for ; Sat, 31 Dec 2022 08:58:56 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=PCAV0j/0; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f45.google.com with SMTP id b24-20020a05600c4a9800b003d21efdd61dso17659096wmp.3 for ; Sat, 31 Dec 2022 08:58:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=tkaLMV/TXg2nYYXIFNhxmxwLEBZPUn1o58dOjIwSSU4=; b=PCAV0j/06SDucy5QeOlMI/eJhQcEX/AtKqOrOahIe0QEUurKNUFoRviH1NY2YPlQOs d6FQ6ZVLaA35sGlxVF+Xbgt+huc+ZW/l1f/SKDaBRqzS0AoNBbDCIL8dmA3s+/hgoIY2 70N1drdUJ/buzq4HRaErvAbvyEKSnT68xuAlY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tkaLMV/TXg2nYYXIFNhxmxwLEBZPUn1o58dOjIwSSU4=; b=IRmPDUT6pgfiST4ukE9TtSv3fxMry+DOGQyBnUnDT3zbGtPxLuYgNWXMEewZv+eEWK DuFcdARTqcJ+9bnqJQD4jj2th05hBxNAU+QnVXtRUWw+Dbf56teLW/AnTEsYpYZwUEhO fv50NrBxiLbg2drNDS7LGrCGKBOXEEnVlYgHAnWulVUmiVx/5/jRJ5rED8+ZcnRWb/9o +0lgsafo6g2yzOvAIvoM9vcqO2gsJ51RwU58s/xpbDLVWyq28oxwSwcB7o+pZfJHqbUq 83doOOqxx84tBss5uHj8LEKzxHU1VRtrXpOUrL3bxZFvtnkUrnfTuxYSJaU4BiFUJ4gI lJjA== X-Gm-Message-State: AFqh2koI2U47LHq3Ic6WkAhprzaCebb+ZSG4tChEK0+ovPrLzixj866N jSTjPNTDZcrzFAKjH2JWcdpxI9Cc7NAV/GKG X-Google-Smtp-Source: AMrXdXuy1J1Pem9QiLPonBEiFAZG0ShU4bbFZi6Un15vbz1AEaSt8XW39FJfmwTQsQscF2/3tsLq7g== X-Received: by 2002:a05:600c:42d5:b0:3d6:e790:c9a0 with SMTP id j21-20020a05600c42d500b003d6e790c9a0mr30904283wme.10.1672505934436; Sat, 31 Dec 2022 08:58:54 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:54 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 06/11] cooker: Ensure commands clean up any parser processes Date: Sat, 31 Dec 2022 16:58:44 +0000 Message-Id: <20221231165849.964169-7-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14268 When finishing a command, we need to ensure any parsing processes that may have been started are cleaned up before we reset the cooker state. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 7e0d6b47bf..adc232e288 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1768,6 +1768,9 @@ class BBCooker: self.parser.final_cleanup() def finishcommand(self): + if hasattr(self.parser, 'shutdown'): + self.parser.shutdown(clean=False) + self.parser.final_cleanup() self.state = state.initial def reset(self): From patchwork Sat Dec 31 16:58:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A700C54EBD for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx.groups.io with SMTP id smtpd.web11.45611.1672505933112061040 for ; Sat, 31 Dec 2022 08:58:56 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=GEc8awTL; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.51, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f51.google.com with SMTP id m8-20020a05600c3b0800b003d96f801c48so15480867wms.0 for ; Sat, 31 Dec 2022 08:58:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=bQVwYqKnKRdBBVAcqZ6blcxUxTziaVQKFtM3IobJZdQ=; b=GEc8awTL4TJcPYLlhcd8Ij4SsN3qnh+KIM0soh/eQomlUpAwwyPwp167WEnh1LkOri izzWF1T26bb52SDNDtgfVpW5AuvikP6jlYm/ongd0O1QKgZNSR7gSRBTvo/wR0aSuiBP MRfY4GPV0MuSPOTM8/czyoSXqztCpw1l7u5PM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bQVwYqKnKRdBBVAcqZ6blcxUxTziaVQKFtM3IobJZdQ=; b=hY/ozPrrJ3IrmKt8X0TKF9j5+GcaogbyJETrOw6/2oN5VjqXk0A1GAxzeDd0pAOXDl O1A62iDwaJ7pfEBnI7JClwqdQpMg+Ta1rofTyRYVUeF0Yxdcdwr54qnAuMlVbmlUE8dP OSISCs8OcNaKfECrAeGIE7ax/zYYxX1DCLPL+yTHLcEHh99lgsOf3IjVQ7KbBHvA+PWs g08M+pHp1vsD62FPJlM/911XuHOqzbc9uLa0Wxkp8Swa0YaU+3St8hnXCmmdeCo4Wylo zYfi4wxAQnHxZfze5wW++xDTd/AOzTky2UMYy3LI5ONFCt2mj93cCWiiuEP37N6FR4UP pcqQ== X-Gm-Message-State: AFqh2kolhPQOHMvlcAY3cZWACIYrsH/42Ti1n8ymAMDy5WYChE85aBNe xLfm54LhwpQhSksD3PrQ631QyJ0LNcsdYQgM X-Google-Smtp-Source: AMrXdXt5ymB0Oj2sYb0/Q3T8sVzfbYnmvFiHDNFuDqz3pLBSTCXt4qqIT+fc2ibJ+Ii39py4VQ7EFQ== X-Received: by 2002:a05:600c:2112:b0:3d3:396e:5e36 with SMTP id u18-20020a05600c211200b003d3396e5e36mr24813471wml.0.1672505935004; Sat, 31 Dec 2022 08:58:55 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:54 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 07/11] server/process: Improve idle loop exit code Date: Sat, 31 Dec 2022 16:58:45 +0000 Message-Id: <20221231165849.964169-8-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14267 When idle handlers want to exit, returning "False" isn't very clear and also causes challenges with the ordering of the removing the idle handler and marking that no async command is running. Use a specific class to signal the exit condition allowing clearer code and allowing the async command to be cleared after the handler has been removed, reducing any opportunity for races. Signed-off-by: Richard Purdie --- lib/bb/command.py | 13 +++++-------- lib/bb/cooker.py | 13 +++++-------- lib/bb/server/process.py | 11 ++++++++++- 3 files changed, 20 insertions(+), 17 deletions(-) diff --git a/lib/bb/command.py b/lib/bb/command.py index cbac07f516..732327d84d 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -128,22 +128,19 @@ class Command: else: return False except KeyboardInterrupt as exc: - self.finishAsyncCommand("Interrupted") - return False + return bb.server.process.idleFinish("Interrupted") except SystemExit as exc: arg = exc.args[0] if isinstance(arg, str): - self.finishAsyncCommand(arg) + return bb.server.process.idleFinish(arg) else: - self.finishAsyncCommand("Exited with %s" % arg) - return False + return bb.server.process.idleFinish("Exited with %s" % arg) except Exception as exc: import traceback if isinstance(exc, bb.BBHandledException): - self.finishAsyncCommand("") + return bb.server.process.idleFinish("") else: - self.finishAsyncCommand(traceback.format_exc()) - return False + return bb.server.process.idleFinish(traceback.format_exc()) def finishAsyncCommand(self, msg=None, code=None): if msg or msg == "": diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index adc232e288..ded9369787 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1489,23 +1489,21 @@ class BBCooker: failures += len(exc.args) retval = False except SystemExit as exc: - self.command.finishAsyncCommand(str(exc)) if quietlog: bb.runqueue.logger.setLevel(rqloglevel) - return False + return bb.server.process.idleFinish(str(exc)) if not retval: if fireevents: bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, item, failures, interrupted), self.databuilder.mcdata[mc]) bb.event.disable_heartbeat() - self.command.finishAsyncCommand(msg) # We trashed self.recipecaches above self.parsecache_valid = False self.configuration.limited_deps = False bb.parse.siggen.reset(self.data) if quietlog: bb.runqueue.logger.setLevel(rqloglevel) - return False + return bb.server.process.idleFinish(msg) if retval is True: return True return retval @@ -1535,8 +1533,7 @@ class BBCooker: failures += len(exc.args) retval = False except SystemExit as exc: - self.command.finishAsyncCommand(str(exc)) - return False + return bb.server.process.idleFinish(str(exc)) if not retval: try: @@ -1544,8 +1541,8 @@ class BBCooker: bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, targets, failures, interrupted), self.databuilder.mcdata[mc]) finally: bb.event.disable_heartbeat() - self.command.finishAsyncCommand(msg) - return False + return bb.server.process.idleFinish(msg) + if retval is True: return True return retval diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index eba28ad983..5acc105e08 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -71,6 +71,10 @@ def get_lockfile_process_msg(lockfile): return procs.decode("utf-8") return None +class idleFinish(): + def __init__(self, msg): + self.msg = msg + class ProcessServer(): profile_filename = "profile.log" profile_processed_filename = "profile.log.processed" @@ -361,7 +365,12 @@ class ProcessServer(): for function, data in list(self._idlefuns.items()): try: retval = function(self, data, False) - if retval is False: + if isinstance(retval, idleFinish): + serverlog("Removing idle function %s at idleFinish" % str(function)) + del self._idlefuns[function] + self.cooker.command.finishAsyncCommand(retval.msg) + nextsleep = None + elif retval is False: serverlog("Removing idle function %s" % str(function)) del self._idlefuns[function] nextsleep = None From patchwork Sat Dec 31 16:58:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F4CCC54E76 for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mx.groups.io with SMTP id smtpd.web11.45615.1672505937283650852 for ; Sat, 31 Dec 2022 08:58:57 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=P7nz+ZRE; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f47.google.com with SMTP id o15so17507728wmr.4 for ; Sat, 31 Dec 2022 08:58:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=DsTD4xHKFfe8PPl0pYTf+UUQ2w5qyHP00COIBsaItA8=; b=P7nz+ZRE1UXAVj3fP9RpulUR1UuY+Pr9iqdUeOoP80z+Y/xUmfuVxTg6DYJCux6Q26 6I8z3b5EWjjawZPAznTT0b9C3ylT+B+yaDTDLjVKAPMLxkEU/OCCFUMORpGqoVjv6i6l ymLfpeoMQ0P2WsCRd9HvrfQVexvIs62XFKUUM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DsTD4xHKFfe8PPl0pYTf+UUQ2w5qyHP00COIBsaItA8=; b=JjlFcdkQXuRJR/dYzK1rfb+j6rNwATZ+4HwNuqEUHM5DTFZl023TgMbleBEM+j8fCM R/LXMxo0clbv2WprvGI3iNwHAygmrTX/Btt8cNYJv/SbFbPvgPffytktKebFhGMZXiSF /GMsLnA6axJga9q3K9fSydMxy5w1KmJ0qhFZZl8mpgRCBaLi7o8SnmY/yL4d9bIlgjMl x6L3cV1HrnzmURrRvd+wbmm/aY8gNmuc66B5xNMYvyh6QV96ndGeNQxf5G8qXlonL/Ru Wn2w0TJnjV3jHCFwvv/pOpK2uALLuEtV+v4DT9y2rHzqrZfX/fWnPiozpEb1WjnK9ukC CZVA== X-Gm-Message-State: AFqh2kqq4TxKMFOTsYYTynDOJyXETFDHA0iYsXbe8DmdWBvHwBQAiw5N lRO+UCvrPp1F4+y6kwLRWg3ccwMh0enXIpqk X-Google-Smtp-Source: AMrXdXvFenwAnk687RWfIuvhBkHoCcf3WWZXbLQ/rE93x/wmQ3gA9nSRpVslo/Xr5y+UbmQETZSLZQ== X-Received: by 2002:a05:600c:35ca:b0:3d1:f2de:195a with SMTP id r10-20020a05600c35ca00b003d1f2de195amr28281707wmq.32.1672505935552; Sat, 31 Dec 2022 08:58:55 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:55 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 08/11] event: Always use threadlock Date: Sat, 31 Dec 2022 16:58:46 +0000 Message-Id: <20221231165849.964169-9-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14269 With the move to a server idle thread, we always need threading. The existing accessor functions could end up turning this off! I was going to hold the lock whilst changing it, check if the value was already set, cache the result and also fix the event code to always release the lock with a try/finally. Instead, disable the existing functions and use a with: block to handle the lock, keeping things much simpler. Signed-off-by: Richard Purdie --- lib/bb/event.py | 73 ++++++++++++++++++++----------------------- lib/bb/tests/event.py | 17 +--------- 2 files changed, 35 insertions(+), 55 deletions(-) diff --git a/lib/bb/event.py b/lib/bb/event.py index db90724444..7826541a64 100644 --- a/lib/bb/event.py +++ b/lib/bb/event.py @@ -68,16 +68,15 @@ _catchall_handlers = {} _eventfilter = None _uiready = False _thread_lock = threading.Lock() -_thread_lock_enabled = False _heartbeat_enabled = False def enable_threadlock(): - global _thread_lock_enabled - _thread_lock_enabled = True + # Always needed now + return def disable_threadlock(): - global _thread_lock_enabled - _thread_lock_enabled = False + # Always needed now + return def enable_heartbeat(): global _heartbeat_enabled @@ -179,36 +178,30 @@ def print_ui_queue(): def fire_ui_handlers(event, d): global _thread_lock - global _thread_lock_enabled if not _uiready: # No UI handlers registered yet, queue up the messages ui_queue.append(event) return - if _thread_lock_enabled: - _thread_lock.acquire() - - errors = [] - for h in _ui_handlers: - #print "Sending event %s" % event - try: - if not _ui_logfilters[h].filter(event): - continue - # We use pickle here since it better handles object instances - # which xmlrpc's marshaller does not. Events *must* be serializable - # by pickle. - if hasattr(_ui_handlers[h].event, "sendpickle"): - _ui_handlers[h].event.sendpickle((pickle.dumps(event))) - else: - _ui_handlers[h].event.send(event) - except: - errors.append(h) - for h in errors: - del _ui_handlers[h] - - if _thread_lock_enabled: - _thread_lock.release() + with _thread_lock: + errors = [] + for h in _ui_handlers: + #print "Sending event %s" % event + try: + if not _ui_logfilters[h].filter(event): + continue + # We use pickle here since it better handles object instances + # which xmlrpc's marshaller does not. Events *must* be serializable + # by pickle. + if hasattr(_ui_handlers[h].event, "sendpickle"): + _ui_handlers[h].event.sendpickle((pickle.dumps(event))) + else: + _ui_handlers[h].event.send(event) + except: + errors.append(h) + for h in errors: + del _ui_handlers[h] def fire(event, d): """Fire off an Event""" @@ -322,21 +315,23 @@ def set_eventfilter(func): _eventfilter = func def register_UIHhandler(handler, mainui=False): - bb.event._ui_handler_seq = bb.event._ui_handler_seq + 1 - _ui_handlers[_ui_handler_seq] = handler - level, debug_domains = bb.msg.constructLogOptions() - _ui_logfilters[_ui_handler_seq] = UIEventFilter(level, debug_domains) - if mainui: - global _uiready - _uiready = _ui_handler_seq - return _ui_handler_seq + with _thread_lock: + bb.event._ui_handler_seq = bb.event._ui_handler_seq + 1 + _ui_handlers[_ui_handler_seq] = handler + level, debug_domains = bb.msg.constructLogOptions() + _ui_logfilters[_ui_handler_seq] = UIEventFilter(level, debug_domains) + if mainui: + global _uiready + _uiready = _ui_handler_seq + return _ui_handler_seq def unregister_UIHhandler(handlerNum, mainui=False): if mainui: global _uiready _uiready = False - if handlerNum in _ui_handlers: - del _ui_handlers[handlerNum] + with _thread_lock: + if handlerNum in _ui_handlers: + del _ui_handlers[handlerNum] return def get_uihandler(): diff --git a/lib/bb/tests/event.py b/lib/bb/tests/event.py index 4de4cced5e..d959f2d95d 100644 --- a/lib/bb/tests/event.py +++ b/lib/bb/tests/event.py @@ -451,10 +451,9 @@ class EventHandlingTest(unittest.TestCase): and disable threadlocks tests """ bb.event.fire(bb.event.OperationStarted(), None) - def test_enable_threadlock(self): + def test_event_threadlock(self): """ Test enable_threadlock method """ self._set_threadlock_test_mockups() - bb.event.enable_threadlock() self._set_and_run_threadlock_test_workers() # Calls to UI handlers should be in order as all the registered # handlers for the event coming from the first worker should be @@ -462,20 +461,6 @@ class EventHandlingTest(unittest.TestCase): self.assertEqual(self._threadlock_test_calls, ["w1_ui1", "w1_ui2", "w2_ui1", "w2_ui2"]) - - def test_disable_threadlock(self): - """ Test disable_threadlock method """ - self._set_threadlock_test_mockups() - bb.event.disable_threadlock() - self._set_and_run_threadlock_test_workers() - # Calls to UI handlers should be intertwined together. Thanks to the - # delay in the registered handlers for the event coming from the first - # worker, the event coming from the second worker starts being - # processed before finishing handling the first worker event. - self.assertEqual(self._threadlock_test_calls, - ["w1_ui1", "w2_ui1", "w1_ui2", "w2_ui2"]) - - class EventClassesTest(unittest.TestCase): """ Event classes test class """ From patchwork Sat Dec 31 16:58:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DD05C53210 for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web10.45727.1672505937825932764 for ; Sat, 31 Dec 2022 08:58:58 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=PlKLGLgA; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f54.google.com with SMTP id ay40so17541915wmb.2 for ; Sat, 31 Dec 2022 08:58:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=wuK2FZRia/8yhVlRsgd6JACcbt/mEBKwKZyw+Ww6Ibs=; b=PlKLGLgAkgtm0/xdnkG0D/zk8HhdnfZrn5t2x2OMl6Tki55GHlmZN+ff2E+UxjQ5vS zvV/eqYBom+t89bagJW16PkhnGZn31fVEsInl35/kdHT1u808zsBuygmh0UtN3AP/MQh LjiDiYLbHdXgWBNAY3rRD31s8VasMd5DUmvZ8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wuK2FZRia/8yhVlRsgd6JACcbt/mEBKwKZyw+Ww6Ibs=; b=u2352PR37ZZjF4gkKI1c0fYRDhKGq1tG6eXEVC4ZKbpFkpbg4n6rCK4H3mxra45bad lIx9ZxxrFg+YNX9XATQPLkLJFCUMq9yky7uiGJU2qr5i9Ks9UFBADbPLUp/sCFnWLfyL DgFyNmIdyzS3rUBsRp2MEaDAorn0cutboSI4SivLhiMQ15o5dFIQl6dRX5/IqaN202ND itLSzbriZYmWKNggaNiIuiyr5+3nbQVbzZAhkbGlwN50IKbkbfrcI9wPHr5PxZRErToO c/JmAh52BWSyhWwoaLPCSYiipjABCHQfQ2dD2ftvdd/9gDgbnPg/OWG9T6SDVCvgGSWJ MiVw== X-Gm-Message-State: AFqh2kpK/Zgc68VWM925jzF/KKwcnggFtAmRC/Mr2cRSMHnJc8OeTzd4 tb+siHP3Dk1kEpGuTYO6lzhTLij1Fhkv6p+9 X-Google-Smtp-Source: AMrXdXv9s8hzEB/lvQhDojGCwj1kGs6KdvcNDZOrzPVEe8PgfHDW1UrqRKNLwtmPFy3fTm2aP+Qtvw== X-Received: by 2002:a1c:6a14:0:b0:3d1:f302:1975 with SMTP id f20-20020a1c6a14000000b003d1f3021975mr25127110wmc.33.1672505936192; Sat, 31 Dec 2022 08:58:56 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:55 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 09/11] server/process: Add locking around idle functions accesses Date: Sat, 31 Dec 2022 16:58:47 +0000 Message-Id: <20221231165849.964169-10-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14270 In preparation for adding splitting bitbakes work into two threads, add locking around the idle functions list accesses. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 5acc105e08..2aee9ef051 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -93,6 +93,7 @@ class ProcessServer(): self.xmlrpc = False self._idlefuns = {} + self._idlefuncsLock = threading.Lock() self.bitbake_lock = lock self.bitbake_lock_name = lockname @@ -109,7 +110,8 @@ class ProcessServer(): def register_idle_function(self, function, data): """Register a function to be called while the server is idle""" assert hasattr(function, '__call__') - self._idlefuns[function] = data + with self._idlefuncsLock: + self._idlefuns[function] = data serverlog("Registering idle function %s" % str(function)) def run(self): @@ -358,21 +360,28 @@ class ProcessServer(): serverlog("".join(msg)) def idle_commands(self, delay, fds=None): + def remove_idle_func(function): + with self._idlefuncsLock: + del self._idlefuns[function] + nextsleep = delay if not fds: fds = [] - for function, data in list(self._idlefuns.items()): + with self._idlefuncsLock: + items = list(self._idlefuns.items()) + + for function, data in items: try: retval = function(self, data, False) if isinstance(retval, idleFinish): serverlog("Removing idle function %s at idleFinish" % str(function)) - del self._idlefuns[function] + remove_idle_func(function) self.cooker.command.finishAsyncCommand(retval.msg) nextsleep = None elif retval is False: serverlog("Removing idle function %s" % str(function)) - del self._idlefuns[function] + remove_idle_func(function) nextsleep = None elif retval is True: nextsleep = None @@ -388,7 +397,7 @@ class ProcessServer(): except Exception as exc: if not isinstance(exc, bb.BBHandledException): logger.exception('Running idle function') - del self._idlefuns[function] + remove_idle_func(function) serverlog("Exception %s broke the idle_thread, exiting" % traceback.format_exc()) self.quit = True From patchwork Sat Dec 31 16:58:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76B28C3DA7D for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by mx.groups.io with SMTP id smtpd.web10.45728.1672505938558996607 for ; Sat, 31 Dec 2022 08:58:58 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=YfB1DVTi; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.53, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f53.google.com with SMTP id p1-20020a05600c1d8100b003d8c9b191e0so17643295wms.4 for ; Sat, 31 Dec 2022 08:58:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=3p/MVBSdcbrUUrfQr9fHHbqC++SMamPwWXQSwdSbVUg=; b=YfB1DVTiUtFe6Mhmx3QheBtlQxUMK6+7lZsuAxAeoR7WtWhEC1eplk9lYbZmqkhOgD AZUL6LM1DxJ/uXU6lyC/KVlbMFZjEOiPugY3FaOWSA9SzChIOtNSRl/asc1hUM5rTyL2 idPw5uFbUJHMmzNfPnvUGlfoWXw3NZ54wPDcA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3p/MVBSdcbrUUrfQr9fHHbqC++SMamPwWXQSwdSbVUg=; b=EKCT+5/5+Nokx6WwdTzIBY+PvJNzs709VGcciUb/Z4ZUSwqZjFVvpzpt4wq+Kchbs3 8p7CHxexSRMvvrWBqZf8sVMpeP3UYVktZ7HHG7dYVv9P69tJ+aEafoSeKfU3YuK0A9TW Pp8UAAmYBgHPIGPV7ZbLFBh/pKiS+n6SAcwkwSSxMKbxPTpzWxEnFSo6Y/Eun60ZY1Xq R66tJ2o+d5ij3T39c6jG4xsLOcYE/VsId89ItCaL/V2uZ+Jrl29NE9wr14zaW8Vdk7rL 8IrNAhHkjnQm/ZT5rGgNog4U03gG3maA2L5zH5cvQiwmRRfDbMdji+GHN2TBzFBT3K7Z vtJA== X-Gm-Message-State: AFqh2krLSIw2wTndRHdQXbJKW4DWPwnMelUbBjJ8GHQdtQ8iwwh1CRwZ Jd9SEP7bW5Vl8evrk64TihIArsy5RI2rqjSk X-Google-Smtp-Source: AMrXdXuaEtPPp34xTgvbVxsktUbop1PfFTvZ5BLq0LzpwVFf2QKPRZ6BUhQbtYTbedWExGIsTZkq9A== X-Received: by 2002:a05:600c:3ca2:b0:3d9:a5a2:65fa with SMTP id bg34-20020a05600c3ca200b003d9a5a265famr3877782wmb.7.1672505936819; Sat, 31 Dec 2022 08:58:56 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:56 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 10/11] server/process: Run idle commands in a separate idle thread Date: Sat, 31 Dec 2022 16:58:48 +0000 Message-Id: <20221231165849.964169-11-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14272 When bitbake is off running heavier "idle" commands, it doesn't service it's command socket which means stopping/interrupting it is hard. It also means we can't "ping" from the UI to know if it is still alive. For those reasons, split idle command execution into it's own thread. The commands are generally already self containted so this is easier than expected. We do have to be careful to only handle inotify poll() from a single thread at a time. It also means we always have to use a thread lock when sending events since both the idle thread and the command thread may generate log messages (and hence events). The patch depends on previous fixes to the builtins locking in event.py and the heartbeat enable/disable changes as well as other locking additions. We use a condition to signal from the idle thread when other sections of code can continue, thanks to Joshua Watt for the review and tweaks squashed into this patch. We do have some sync points where we need to ensure any currently executing commands have finished before we can start a new async command for example. Signed-off-by: Richard Purdie --- lib/bb/command.py | 11 ++-- lib/bb/cooker.py | 29 +++++++--- lib/bb/server/process.py | 105 ++++++++++++++++++++++------------ lib/bb/server/xmlrpcserver.py | 2 +- 4 files changed, 97 insertions(+), 50 deletions(-) diff --git a/lib/bb/command.py b/lib/bb/command.py index 732327d84d..0706b89271 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -60,7 +60,7 @@ class Command: # FIXME Add lock for this self.currentAsyncCommand = None - def runCommand(self, commandline, ro_only = False): + def runCommand(self, commandline, process_server, ro_only=False): command = commandline.pop(0) # Ensure cooker is ready for commands @@ -84,7 +84,7 @@ class Command: if not hasattr(command_method, 'readonly') or not getattr(command_method, 'readonly'): return None, "Not able to execute not readonly commands in readonly mode" try: - self.cooker.process_inotify_updates() + self.cooker.process_inotify_updates_apply() if getattr(command_method, 'needconfig', True): self.cooker.updateCacheSync() result = command_method(self, commandline) @@ -100,7 +100,10 @@ class Command: else: return result, None if self.currentAsyncCommand is not None: - return None, "Busy (%s in progress)" % self.currentAsyncCommand[0] + # Wait for the idle loop to have cleared (30s max) + process_server.wait_for_idle(timeout=30) + if self.currentAsyncCommand is not None: + return None, "Busy (%s in progress)" % self.currentAsyncCommand[0] if command not in CommandsAsync.__dict__: return None, "No such command" self.currentAsyncCommand = (command, commandline) @@ -109,7 +112,7 @@ class Command: def runAsyncCommand(self): try: - self.cooker.process_inotify_updates() + self.cooker.process_inotify_updates_apply() if self.cooker.state in (bb.cooker.state.error, bb.cooker.state.shutdown, bb.cooker.state.forceshutdown): # updateCache will trigger a shutdown of the parser # and then raise BBHandledException triggering an exit diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index ded9369787..a5a635858c 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -149,7 +149,7 @@ class BBCooker: Manages one bitbake build run """ - def __init__(self, featureSet=None, idleCallBackRegister=None): + def __init__(self, featureSet=None, idleCallBackRegister=None, waitIdle=None): self.recipecaches = None self.eventlog = None self.skiplist = {} @@ -164,6 +164,7 @@ class BBCooker: self.configuration = bb.cookerdata.CookerConfiguration() self.idleCallBackRegister = idleCallBackRegister + self.waitIdle = waitIdle bb.debug(1, "BBCooker starting %s" % time.time()) sys.stdout.flush() @@ -220,6 +221,8 @@ class BBCooker: bb.debug(1, "BBCooker startup complete %s" % time.time()) sys.stdout.flush() + self.inotify_threadlock = threading.Lock() + def init_configdata(self): if not hasattr(self, "data"): self.initConfigurationData() @@ -248,11 +251,18 @@ class BBCooker: self.notifier = pyinotify.Notifier(self.watcher, self.notifications) def process_inotify_updates(self): - for n in [self.confignotifier, self.notifier]: - if n and n.check_events(timeout=0): - # read notified events and enqueue them - n.read_events() - n.process_events() + with self.inotify_threadlock: + for n in [self.confignotifier, self.notifier]: + if n and n.check_events(timeout=0): + # read notified events and enqueue them + n.read_events() + + def process_inotify_updates_apply(self): + with self.inotify_threadlock: + for n in [self.confignotifier, self.notifier]: + if n and n.check_events(timeout=0): + n.read_events() + n.process_events() def config_notifications(self, event): if event.maskname == "IN_Q_OVERFLOW": @@ -1744,7 +1754,7 @@ class BBCooker: return def post_serve(self): - self.shutdown(force=True) + self.shutdown(force=True, idle=False) prserv.serv.auto_shutdown() if hasattr(bb.parse, "siggen"): bb.parse.siggen.exit() @@ -1754,12 +1764,15 @@ class BBCooker: if hasattr(self, "data"): bb.event.fire(CookerExit(), self.data) - def shutdown(self, force = False): + def shutdown(self, force=False, idle=True): if force: self.state = state.forceshutdown else: self.state = state.shutdown + if idle: + self.waitIdle(30) + if self.parser: self.parser.shutdown(clean=not force) self.parser.final_cleanup() diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 2aee9ef051..ac7749d36c 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -92,8 +92,11 @@ class ProcessServer(): self.maxuiwait = 30 self.xmlrpc = False + self.idle = None + # Need a lock for _idlefuns changes self._idlefuns = {} self._idlefuncsLock = threading.Lock() + self.idle_cond = threading.Condition(self._idlefuncsLock) self.bitbake_lock = lock self.bitbake_lock_name = lockname @@ -151,6 +154,12 @@ class ProcessServer(): return ret + def wait_for_idle(self, timeout=30): + # Wait for the idle loop to have cleared + with self.idle_cond: + # FIXME - the 1 is the inotify processing in cooker which always runs + self.idle_cond.wait_for(lambda: len(self._idlefuns) <= 1, timeout) + def main(self): self.cooker.pre_serve() @@ -174,6 +183,12 @@ class ProcessServer(): self.controllersock.close() self.controllersock = False if self.haveui: + # Wait for the idle loop to have cleared (30s max) + self.wait_for_idle(30) + if self.cooker.command.currentAsyncCommand is not None: + serverlog("Idle loop didn't finish queued commands after 30s, exiting.") + self.quit = True + fds.remove(self.command_channel) bb.event.unregister_UIHhandler(self.event_handle, True) self.command_channel_reply.writer.close() @@ -185,7 +200,7 @@ class ProcessServer(): self.cooker.clientComplete() self.haveui = False ready = select.select(fds,[],[],0)[0] - if newconnections: + if newconnections and not self.quit: serverlog("Starting new client") conn = newconnections.pop(-1) fds.append(conn) @@ -257,7 +272,7 @@ class ProcessServer(): continue try: serverlog("Running command %s" % command) - self.command_channel_reply.send(self.cooker.command.runCommand(command)) + self.command_channel_reply.send(self.cooker.command.runCommand(command, self)) serverlog("Command Completed (socket: %s)" % os.path.exists(self.sockname)) except Exception as e: stack = traceback.format_exc() @@ -285,6 +300,9 @@ class ProcessServer(): ready = self.idle_commands(.1, fds) + if self.idle: + self.idle.join() + serverlog("Exiting (socket: %s)" % os.path.exists(self.sockname)) # Remove the socket file so we don't get any more connections to avoid races # The build directory could have been renamed so if the file isn't the one we created @@ -300,7 +318,7 @@ class ProcessServer(): self.sock.close() try: - self.cooker.shutdown(True) + self.cooker.shutdown(True, idle=False) self.cooker.notifier.stop() self.cooker.confignotifier.stop() except: @@ -359,47 +377,60 @@ class ProcessServer(): msg.append(":\n%s" % procs) serverlog("".join(msg)) - def idle_commands(self, delay, fds=None): + def idle_thread(self): def remove_idle_func(function): with self._idlefuncsLock: del self._idlefuns[function] + self.idle_cond.notify_all() - nextsleep = delay - if not fds: + while not self.quit: + nextsleep = 0.1 fds = [] - with self._idlefuncsLock: - items = list(self._idlefuns.items()) + with self._idlefuncsLock: + items = list(self._idlefuns.items()) - for function, data in items: - try: - retval = function(self, data, False) - if isinstance(retval, idleFinish): - serverlog("Removing idle function %s at idleFinish" % str(function)) - remove_idle_func(function) - self.cooker.command.finishAsyncCommand(retval.msg) - nextsleep = None - elif retval is False: - serverlog("Removing idle function %s" % str(function)) + for function, data in items: + try: + retval = function(self, data, False) + if isinstance(retval, idleFinish): + serverlog("Removing idle function %s at idleFinish" % str(function)) + remove_idle_func(function) + self.cooker.command.finishAsyncCommand(retval.msg) + nextsleep = None + elif retval is False: + serverlog("Removing idle function %s" % str(function)) + remove_idle_func(function) + nextsleep = None + elif retval is True: + nextsleep = None + elif isinstance(retval, float) and nextsleep: + if (retval < nextsleep): + nextsleep = retval + elif nextsleep is None: + continue + else: + fds = fds + retval + except SystemExit: + raise + except Exception as exc: + if not isinstance(exc, bb.BBHandledException): + logger.exception('Running idle function') remove_idle_func(function) - nextsleep = None - elif retval is True: - nextsleep = None - elif isinstance(retval, float) and nextsleep: - if (retval < nextsleep): - nextsleep = retval - elif nextsleep is None: - continue - else: - fds = fds + retval - except SystemExit: - raise - except Exception as exc: - if not isinstance(exc, bb.BBHandledException): - logger.exception('Running idle function') - remove_idle_func(function) - serverlog("Exception %s broke the idle_thread, exiting" % traceback.format_exc()) - self.quit = True + serverlog("Exception %s broke the idle_thread, exiting" % traceback.format_exc()) + self.quit = True + + if nextsleep is not None: + select.select(fds,[],[],nextsleep)[0] + + def idle_commands(self, delay, fds=None): + nextsleep = delay + if not fds: + fds = [] + + if not self.idle: + self.idle = threading.Thread(target=self.idle_thread) + self.idle.start() # Create new heartbeat event? now = time.time() @@ -592,7 +623,7 @@ def execServer(lockfd, readypipeinfd, lockname, sockname, server_timeout, xmlrpc writer = ConnectionWriter(readypipeinfd) try: featureset = [] - cooker = bb.cooker.BBCooker(featureset, server.register_idle_function) + cooker = bb.cooker.BBCooker(featureset, server.register_idle_function, server.wait_for_idle) cooker.configuration.profile = profile except bb.BBHandledException: return None diff --git a/lib/bb/server/xmlrpcserver.py b/lib/bb/server/xmlrpcserver.py index 01f55538ae..2e65dc34a9 100644 --- a/lib/bb/server/xmlrpcserver.py +++ b/lib/bb/server/xmlrpcserver.py @@ -118,7 +118,7 @@ class BitBakeXMLRPCServerCommands(): """ Run a cooker command on the server """ - return self.server.cooker.command.runCommand(command, self.server.readonly) + return self.server.cooker.command.runCommand(command, self.server, self.server.readonly) def getEventHandle(self): return self.event_handle From patchwork Sat Dec 31 16:58:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76172C4167B for ; Sat, 31 Dec 2022 16:59:02 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.45612.1672505933951647862 for ; Sat, 31 Dec 2022 08:58:58 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=LiVP7RuG; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f48.google.com with SMTP id m26-20020a05600c3b1a00b003d9811fcaafso11121491wms.5 for ; Sat, 31 Dec 2022 08:58:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=oEXcLKOHfSMHOKaafg/oqFGDbhEob0mfkv3eOhYT4aw=; b=LiVP7RuGQCGymamFzy3neIyDdbKqxFJWvlbiRVAOAx5akDfEtTB/9Wty+IAunYFABO QtAHvPYC1YwXBSFRbLiQCHa6D6AvTxTKgec5VhREBuqxfDiAZdSWU774WFW89/+nKYyu 95y7ecFVRXDLf/uADegjQrvx3fsmE+0l+z5gY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oEXcLKOHfSMHOKaafg/oqFGDbhEob0mfkv3eOhYT4aw=; b=otrYOkROUlv/fs+yvk1Vf8PX5cV/apjJ721GlwBXM+hVNGEY8yL2qhjdVRfZX1Gqt2 0SD51lamqqbn0uKi1klZYMHXalVylDOTAiyGdDvE5pJi1pb07iq65wA1u/aIhVhJzcyL IIjrKqXrYAAIYhoMxfIKi1fMdV0EnlWE9FxJ4mxdPW3lEpKoZkoMrbLdwrFm4gdMQb3a uScOqtWNv20oF1DoK4xkyMQmcC7sdCHP/TFXO11qeelHCqdV7i+1Ns71vwZfYE7V2xue 1oTMOkbGdbLklMSJ9OIRu9Q4tjTX41tkdaJ+iwdnA/vfGT7tdIsgUyPj+j7APRGFOLCF DDRA== X-Gm-Message-State: AFqh2kr0wBG5rSxRXr7YYWclymps1I71ULKVC7vgzSYIFR3J0YFST00a KfizdxKFQdyXBZZuE74e78X+uCvbJ6OhSyuO X-Google-Smtp-Source: AMrXdXsAqbeGjj/u6bfLGc3d9fSAmn900TFF54YaV8qZv6pLNOTnSSkoDsJiYbhGsuI6Kdwyd1VALg== X-Received: by 2002:a05:600c:3644:b0:3d2:39dc:4ab9 with SMTP id y4-20020a05600c364400b003d239dc4ab9mr28437408wmq.13.1672505937620; Sat, 31 Dec 2022 08:58:57 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1a1f:74ee:21ca:3bb]) by smtp.gmail.com with ESMTPSA id a1-20020a05600c348100b003b47b80cec3sm36582812wmq.42.2022.12.31.08.58.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Dec 2022 08:58:57 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 11/11] knotty: Ping the server/cooker periodically Date: Sat, 31 Dec 2022 16:58:49 +0000 Message-Id: <20221231165849.964169-12-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> References: <20221231165849.964169-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 31 Dec 2022 16:59:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14271 We're seeing failures where the UI hangs if the server disappears. Ping the cooker/server if we've not had any events in the last minute so we can check if it is still alive. Signed-off-by: Richard Purdie --- lib/bb/ui/knotty.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index 761c2dddaf..7370a1adfd 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -625,7 +625,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): printintervaldelta = 10 * 60 # 10 minutes printinterval = printintervaldelta - lastprint = time.time() + pinginterval = 1 * 60 # 1 minute + lastevent = lastprint = time.time() termfilter = tf(main, helper, console_handlers, params.options.quiet) atexit.register(termfilter.finish) @@ -637,11 +638,20 @@ def main(server, eventHandler, params, tf = TerminalFilter): printinterval += printintervaldelta event = eventHandler.waitEvent(0) if event is None: + if (lastevent + pinginterval) <= time.time(): + ret, error = server.runCommand(["ping"]) + if error or not ret: + termfilter.clearFooter() + print("No reply after pinging server (%s, %s), exiting." % (str(error), str(ret))) + return_value = 3 + main.shutdown = 2 + lastevent = time.time() if not parseprogress: termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None: continue + lastevent = time.time() helper.eventHandler(event) if isinstance(event, bb.runqueue.runQueueExitWait): if not main.shutdown: