From patchwork Tue Jun 21 18:20:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 9427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E593CC433EF for ; Tue, 21 Jun 2022 18:21:25 +0000 (UTC) Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by mx.groups.io with SMTP id smtpd.web09.47105.1655835675999463112 for ; Tue, 21 Jun 2022 11:21:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=bNj4R24p; spf=softfail (domain: sakoman.com, ip: 209.85.214.169, mailfrom: steve@sakoman.com) Received: by mail-pl1-f169.google.com with SMTP id o18so6194693plg.2 for ; Tue, 21 Jun 2022 11:21:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=k1x4vw8yHkic5ZcHlK8FUyQKYpvGk1aOXSLZ2F9R2IA=; b=bNj4R24pCDJyRAaP2hTaH1+A7MB2i2Dj3JiTJerkyIRnuN+iAMQlgM2lN7oiRCzNyM +tycHQC+nL8/hLVe1AuQdd3yb2CKZ+nT+rjey1wY4ih0pv8ArW0j6eGNcYe+dQmmHMXH bI9SZN2PmW5U+Tt1xMN6h4oVzejNu93ErLA3iBlWG/BRadyrKKbqz9W6scZXfji0MB0a v/lHmuWODryzTLdFKtH0xAnnpOPIffxSMTi0AP/8h99BISJ4K861nRDtckHN+pZKTvVF zqjth+84lwrCT5CIGAZvd4gEK4mCgyCCR325gfXmmC//zVdwJYBST52sRPqdIG0ohtDQ L3Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=k1x4vw8yHkic5ZcHlK8FUyQKYpvGk1aOXSLZ2F9R2IA=; b=tawkGdMBeJmM6ngA5Fay5W7TnH/QQxyn4P83ujL+n+QxvlQyfn9yrhjNzJh3uCKmcM uzEGbYWHIrtNq1RxIZOijnssROVlkfKpTT0o2kgEXCMDbrDXTgMq9Niq8aQAIZc68j93 EwZvDSbb2ItGqqCsZwBpTA1hIlGEfWNO0kbyq1J8arNwnv/9uXG8kArxJV2lxsRAYKnG QK+qkXdqci+eRlTbWdXbWOECSDrbT9lmDKbPvP5hzGL7HvhP9OQKFHKEl2efwWYnI8sT mdS0wMwikUGjZvhgXBTJ8pDS3cHsUam29AqaCXDMCFODXoI/24jCHaLRVN1xl/6+W/Wn iLlw== X-Gm-Message-State: AJIora881nH1r9bNrh/rUT2CZNA0DG1yc/0gGQmhJ5YyV5PGxgyq5G43 oULrd7bOVlOMZtMkn/brJfsvcwjkXPb7aWJA X-Google-Smtp-Source: AGRyM1sIcNd2El4s7CwxC9lKmPfViCgVd+bOE6GELUZzaL/ZCVDq2Xb5i2xduY7xkLxrUkowEInWjQ== X-Received: by 2002:a17:90b:18b:b0:1ec:b259:a6af with SMTP id t11-20020a17090b018b00b001ecb259a6afmr10878529pjs.237.1655835674853; Tue, 21 Jun 2022 11:21:14 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id t13-20020a1709027fcd00b0016392bd5060sm9755503plb.142.2022.06.21.11.21.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 11:21:14 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 1/5] server/process: Disable gc around critical section Date: Tue, 21 Jun 2022 08:20:57 -1000 Message-Id: <0784db7dd0fef6f0621ad8d74372f44e87fef950.1655835530.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 18:21:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13779 From: Richard Purdie The python gc can trigger whilst we're holding the event stream lock and when cleaning up objects, they can trigger warnings. This translates into a new event which would then need the lock and we can deadlock. Disable gc whilst we hold that lock to avoid this unfortunate and problematic situation. Signed-off-by: Richard Purdie (cherry picked from commit 96a6303949cefd469bcf5ed250ff512271354357) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 3c9ed706..4bdb84ae 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -25,6 +25,7 @@ import subprocess import errno import re import datetime +import gc import bb.server.xmlrpcserver from bb import daemonize from multiprocessing import queues @@ -671,8 +672,10 @@ class ConnectionWriter(object): def send(self, obj): obj = multiprocessing.reduction.ForkingPickler.dumps(obj) + gc.disable() with self.wlock: self.writer.send_bytes(obj) + gc.enable() def fileno(self): return self.writer.fileno() From patchwork Tue Jun 21 18:20:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 9429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CA99CCA480 for ; Tue, 21 Jun 2022 18:21:26 +0000 (UTC) Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by mx.groups.io with SMTP id smtpd.web11.36530.1655835678330381253 for ; Tue, 21 Jun 2022 11:21:18 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=Kr6x0+Tb; spf=softfail (domain: sakoman.com, ip: 209.85.216.48, mailfrom: steve@sakoman.com) Received: by mail-pj1-f48.google.com with SMTP id t3-20020a17090a510300b001ea87ef9a3dso14271787pjh.4 for ; Tue, 21 Jun 2022 11:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3m6+pa1arYVW3goM/30xOx1bZq6FNi7sU+pqa6YgwLY=; b=Kr6x0+TbEsyFix5GgJIJOsEeN3e7ZSyeKat7K30tnBaJXxYeLfHhnW8afr8bWmbALh SQGo+U2gNDmss5Ro/iVB1kDMe9Ew3fzAk1JFd9uLMEFdDv8DsNRGStNM+iERlVWw2AfZ dbIQf01q52r11auqK3fi8DxPhTqyIVyZ9u+yIJR2xRqQQmQVTeEN+dB15L6lIfCxG6Ry 38LiYuRwRvtvw2Fbg47xk4lmdkVI47SdrOGe4bntvTw2hNOINTLrqXuhm0d0YAWzbirr 4REO46xHuEKk8lcxS8Q3oBvwPC7cIpebyCPcM6m2XOU3lisOCXx7zcqltV3pz2n5veLj TXyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3m6+pa1arYVW3goM/30xOx1bZq6FNi7sU+pqa6YgwLY=; b=FAV1UbgFqeEdngDadc44/W679n/2QoL6vOcRQiRbfaDv66NfBwBBovZpxYbMnvYHV2 rNDmTlkK6qvtb92MqVEz42WSc1xMmohsvc8r334kvdRCcDPu5XlexTPUELYUGLsueKGd JemsXDIBsd6snnwqtiq/4t9zi/DiKQ8uvByJG7sRWq8cFMnYyXCbTJ0oy7als5aXl/bv 65i076LZdsIq58C846Qi6rFUl05srdnsJjNrNZoG7dI7eYqV+3jb9pslclMlNt+1UIag C6Z3yMKhRKC6ujUCFkSXcO5Ox0btayYzTWPcoXcWR2FXzal8AEiAWxmBliEC2eG+9cVH flKQ== X-Gm-Message-State: AJIora9KlR0eEpeYG8znfxqv9ok3w+IKbAc+nUU5T9kTGq2vIlmotyPQ vQGlgtF8UJE8Buv0q3rF8a3i+P2DsQcV9ly2 X-Google-Smtp-Source: AGRyM1vbH/RuC3rDyM5vxgUN7WqnVcU2EmML823N6eSSNsuE0njwJehUz72qP4/4ROJCyfUum1o1IQ== X-Received: by 2002:a17:90b:38c2:b0:1e8:7ad1:bec5 with SMTP id nn2-20020a17090b38c200b001e87ad1bec5mr44547123pjb.51.1655835677351; Tue, 21 Jun 2022 11:21:17 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id t13-20020a1709027fcd00b0016392bd5060sm9755503plb.142.2022.06.21.11.21.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 11:21:16 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 2/5] tinfoil/data_smart: Allow variable history emit() to function remotely Date: Tue, 21 Jun 2022 08:20:58 -1000 Message-Id: <144a1cfe8b60c677bb6ec66c242e064c7ba3ed88.1655835530.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 18:21:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13780 From: Richard Purdie We can't access the emit() function of varhistory currently as the datastore parameter isn't handled correctly, nor is the output stream. Add a custom wrapper for this function which handles the two details correctly. Signed-off-by: Richard Purdie (cherry picked from commit ba0fa084ccd2b1ade96425d158fd31e49e42f286) Signed-off-by: Steve Sakoman --- lib/bb/command.py | 12 ++++++++++++ lib/bb/tinfoil.py | 4 ++++ 2 files changed, 16 insertions(+) diff --git a/lib/bb/command.py b/lib/bb/command.py index 98c945ed..b8429b27 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -20,6 +20,7 @@ Commands are queued in a CommandQueue from collections import OrderedDict, defaultdict +import io import bb.event import bb.cooker import bb.remotedata @@ -478,6 +479,17 @@ class CommandsSync: d = command.remotedatastores[dsindex].varhistory return getattr(d, method)(*args, **kwargs) + def dataStoreConnectorVarHistCmdEmit(self, command, params): + dsindex = params[0] + var = params[1] + oval = params[2] + val = params[3] + d = command.remotedatastores[params[4]] + + o = io.StringIO() + command.remotedatastores[dsindex].varhistory.emit(var, oval, val, o, d) + return o.getvalue() + def dataStoreConnectorIncHistCmd(self, command, params): dsindex = params[0] method = params[1] diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py index 28f1e562..8bec8cba 100644 --- a/lib/bb/tinfoil.py +++ b/lib/bb/tinfoil.py @@ -53,6 +53,10 @@ class TinfoilDataStoreConnectorVarHistory: def remoteCommand(self, cmd, *args, **kwargs): return self.tinfoil.run_command('dataStoreConnectorVarHistCmd', self.dsindex, cmd, args, kwargs) + def emit(self, var, oval, val, o, d): + ret = self.tinfoil.run_command('dataStoreConnectorVarHistCmdEmit', self.dsindex, var, oval, val, d.dsindex) + o.write(ret) + def __getattr__(self, name): if not hasattr(bb.data_smart.VariableHistory, name): raise AttributeError("VariableHistory has no such method %s" % name) From patchwork Tue Jun 21 18:20:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 9430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3873CCA473 for ; Tue, 21 Jun 2022 18:21:25 +0000 (UTC) Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by mx.groups.io with SMTP id smtpd.web08.47032.1655835681073956879 for ; Tue, 21 Jun 2022 11:21:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=c8qYCDgl; spf=softfail (domain: sakoman.com, ip: 209.85.216.46, mailfrom: steve@sakoman.com) Received: by mail-pj1-f46.google.com with SMTP id p5so8001909pjt.2 for ; Tue, 21 Jun 2022 11:21:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=74ZOMBboF3dgbyvMndqG+/Xsl4OI/PyEevqBGequ69g=; b=c8qYCDglft20b8L0nyJqDIrekXzI4utzUS1iBrfd0fmz23qtNLONX3KVnHrF/gMl3A Ss8LaL90LKFglIqNq6QwOM0CrhHws0XoWUcSe5PkUJrw14sO0Bvl9rH1MOk2XC2JLXCC dNx3xsBhRzjdUFGyaXK5OlY/d4EbH6ik2sYXyyTq9Gdqxdb16sb1IGOyRDKvyZRFJlEC U2ZpqaHWuU+0ycypl/jd09WGQLs1yt0OGEvLbsWeouKHNwM8aDrHFTHOk0b0sUR/Kv9r 6z1UVzEAUE36DGu/Qmb28zBsNb6TnXU7/sQz5S1Xe3yeauIJrXa0JIGnhJi/4hp/I4Gr npbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=74ZOMBboF3dgbyvMndqG+/Xsl4OI/PyEevqBGequ69g=; b=ocmPyJrmsMMrTSytzMokOvGNRIgI+GZDsJvVTmvB5a0yUIJaAWFn9pOMHWUEcOCZcY D0Gd6VkEbH9RZvl+jm3YFy8SADmcMSOLi3UXAwH39sh8aGMMzFlDD6KiCGV2cTmdDWz5 s37yp9AeCzDZh3/40lRWCEhDJ5su19PWgAMWoltjijdutaGNkRcPPG6VaZpSaU3i1s3E h2vOhPNPVAM3aqOkoBbEpjJ3uYpZmRIhAjfGhvETDqERdcChh1wTSIzxmpFXRrtAXr+0 xrmSwCN9uRQIy+nuI5gblmiaLDNTv15W0tAMgeM/H6kwx7qxbAP9e6sEpJ/6ukFe6z2Y DaQQ== X-Gm-Message-State: AJIora82u02DrVLK0KbeCyopkaPa5keHhE7boDyfgA8qdiYsqv4BFj8X x4MPYxZk6FegMPWIsn3O2mZ0W96KrW86RdDT X-Google-Smtp-Source: AGRyM1tOEUobXwojl/k/rc1N6q/IX5lf12ubT1RCX7oBHOQEmJ1Llu/35PynIxZYH3IrgANiEw46CA== X-Received: by 2002:a17:90b:314a:b0:1e8:5362:5620 with SMTP id ip10-20020a17090b314a00b001e853625620mr34735019pjb.9.1655835680049; Tue, 21 Jun 2022 11:21:20 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id t13-20020a1709027fcd00b0016392bd5060sm9755503plb.142.2022.06.21.11.21.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 11:21:19 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 3/5] bin/bitbake-getvar: Add a new command to query a variable value (with history) Date: Tue, 21 Jun 2022 08:20:59 -1000 Message-Id: <47ed06d441152f8b6d374cacfac2c668c354423e.1655835530.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 18:21:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13781 From: Richard Purdie We've talked about having this for long enough. Add a command which queries a single variable value with history. This saves "bitbake -e | grep" and avoids the various pitfalls that has. It also provides a neat example of using tinfoil to make such a query. Parameters to limit the output to just the value, to limit to a variable flag and to not expand the output are provided. [YOCTO #10748] Signed-off-by: Richard Purdie (cherry picked from commit 4c1881b620e885f55d7772f8626b8a76c2828333) Signed-off-by: Steve Sakoman --- bin/bitbake-getvar | 48 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100755 bin/bitbake-getvar diff --git a/bin/bitbake-getvar b/bin/bitbake-getvar new file mode 100755 index 00000000..94232192 --- /dev/null +++ b/bin/bitbake-getvar @@ -0,0 +1,48 @@ +#! /usr/bin/env python3 +# +# Copyright (C) 2021 Richard Purdie +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import argparse +import io +import os +import sys + +bindir = os.path.dirname(__file__) +topdir = os.path.dirname(bindir) +sys.path[0:0] = [os.path.join(topdir, 'lib')] + +import bb.tinfoil + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Bitbake Query Variable") + parser.add_argument("variable", help="variable name to query") + parser.add_argument("-r", "--recipe", help="Recipe name to query", default=None, required=False) + parser.add_argument('-u', '--unexpand', help='Do not expand the value (with --value)', action="store_true") + parser.add_argument('-f', '--flag', help='Specify a variable flag to query (with --value)', default=None) + parser.add_argument('--value', help='Only report the value, no history and no variable name', action="store_true") + args = parser.parse_args() + + if args.unexpand and not args.value: + print("--unexpand only makes sense with --value") + sys.exit(1) + + if args.flag and not args.value: + print("--flag only makes sense with --value") + sys.exit(1) + + with bb.tinfoil.Tinfoil(tracking=True) as tinfoil: + if args.recipe: + tinfoil.prepare(quiet=2) + d = tinfoil.parse_recipe(args.recipe) + else: + tinfoil.prepare(quiet=2, config_only=True) + d = tinfoil.config_data + if args.flag: + print(str(d.getVarFlag(args.variable, args.flag, expand=(not args.unexpand)))) + elif args.value: + print(str(d.getVar(args.variable, expand=(not args.unexpand)))) + else: + bb.data.emit_var(args.variable, d=d, all=True) From patchwork Tue Jun 21 18:21:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 9428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E15D3C43334 for ; Tue, 21 Jun 2022 18:21:25 +0000 (UTC) Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by mx.groups.io with SMTP id smtpd.web10.46486.1655835683687731131 for ; Tue, 21 Jun 2022 11:21:23 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=WEylasN7; spf=softfail (domain: sakoman.com, ip: 209.85.215.171, mailfrom: steve@sakoman.com) Received: by mail-pg1-f171.google.com with SMTP id 68so8413748pgb.10 for ; Tue, 21 Jun 2022 11:21:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=RWreAhsu1Woco4xoJ+omiB1eNfLNCHaCA2c2QqR3pSE=; b=WEylasN7YnscbmmWqfNKP/x7OtqsZx9+SFmtkycr+fqwFTrWKXwLhLF976CPmtw1m9 zAQIMra+7oPMib2QcQyVW9lQJjTmD6HUsoVXUONrZ9Xcy9+QcJB0tlYbbFB+ZFwf8JHb FM+9LB6B4Tbj4xmU5Y7Nmhoor+d9wMNYpGhRuxEf5ZMoD8k2ur7MF5osw5Eusb1+1c1q 4iHE+c/bIRZ8IRSuwA9NdprHAN85ureBjutjrgfi/v9OI2mf4sbzVHJhg2khR0L96UbV PnovA/ZTBU8eNH6L7OluVtzOAD/2x40Jd0pu1OSdsLM6yw2w9HH09yrDkJg0V3ysLgS2 FKRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RWreAhsu1Woco4xoJ+omiB1eNfLNCHaCA2c2QqR3pSE=; b=E1WPL/YllBWoDXSpElAzhIwUSZxClYbHcmCScrdQI3NJLUVkO8RqGh1574hoMbdmpm eqmdo1wA4X89bopPsTsrnWDh4xu/P1p4Aw6Wrr9IIdRoVJ+18lHtpmbeJ2i/Pz3uMlg5 7tNtB9ePrq0ObrB/NRoFaVyyxtRbza//Jjyqn7ThFiz/ISpbJPRCbaURNd3LXyxchFv5 TeLo6GVKjloJFw7h1sRGFQrLCun9PfuW5aEllntlpvUv+gbAIcIcQklV9jkX/+/8uWjc NO7HQeDMdiPltL6pN6asdBrezLbzsE58Oxnl1xlQ2LJL9reURk66yvSuyQgP4A20zVbt 6dmQ== X-Gm-Message-State: AJIora/GmR9btsPufZKoKj4n/OktyajuuZ5AYc0IDC8KF9r9A7p9HpMw 2HO+zvUzHnJxiqfL5c3mGUik+/t4pqt8RglU X-Google-Smtp-Source: AGRyM1sBmvyMe9uLOH9KsKq9gHSApi4wRx8wxlXOrpLyOCfXLs/UPHMmL6XNXOkcMnfq7UU5JiyaOg== X-Received: by 2002:a63:7844:0:b0:40c:9792:5d6d with SMTP id t65-20020a637844000000b0040c97925d6dmr14208726pgc.360.1655835682722; Tue, 21 Jun 2022 11:21:22 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id t13-20020a1709027fcd00b0016392bd5060sm9755503plb.142.2022.06.21.11.21.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 11:21:22 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 4/5] knotty: display active tasks when printing keepAlive() message Date: Tue, 21 Jun 2022 08:21:00 -1000 Message-Id: <36fe8bae9fec61547ee0b13bcb721033afd3ac0e.1655835530.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 18:21:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13782 From: Ross Burton In interactive bitbake sessions it is obvious what tasks are running when one of them hangs or otherwise takes a long time. However, in non-interactive sessions (such as automated builds) bitbake just prints a message saying that it is "still alive" with no clues as to what tasks are active still. By simply listing the active tasks when printing the keep alive message, we don't need to parse the bitbake log to identify which of the tasks is still active and has presumably hung. Signed-off-by: Ross Burton Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie (cherry picked from commit 30f6c3f175617beea8e8bb75dcf255611e3fc2fd) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman --- lib/bb/ui/knotty.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index e70c2464..71c5a3e9 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -227,7 +227,9 @@ class TerminalFilter(object): def keepAlive(self, t): if not self.cuu: - print("Bitbake still alive (%ds)" % t) + print("Bitbake still alive (no events for %ds). Active tasks:" % t) + for t in self.helper.running_tasks: + print(t) sys.stdout.flush() def updateFooter(self): From patchwork Tue Jun 21 18:21:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 9431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06003C433EF for ; Tue, 21 Jun 2022 18:22:56 +0000 (UTC) Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by mx.groups.io with SMTP id smtpd.web11.36553.1655835768871663066 for ; Tue, 21 Jun 2022 11:22:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=qalKlHhd; spf=softfail (domain: sakoman.com, ip: 209.85.214.172, mailfrom: steve@sakoman.com) Received: by mail-pl1-f172.google.com with SMTP id k7so13259269plg.7 for ; Tue, 21 Jun 2022 11:22:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=PEyq7LNyz7JntBDQn9aUhPG4pYZkRAR/BiMR78ZNWQU=; b=qalKlHhdHV4rUmhb7fdH4uLZlN64h48U+WLF2YxrlZO8olCd8C6xlet0jWy8KAFv3m vTWlsH3BicY76uaUwd2UQ9/g0wOAxLoC3IY970sBU2Sw/9WfjzhBPyb23u8Lo2qGLuYd ZxbdXeZwGuzJT/uF5P/ziaw97q5T/nRFDBI1Oc6FDS1ik5rjln7YYQDn+leGuKBdgo2K CJvh7vmSHo3TUdsXvcJvMaSieNefqVqz/0F0d4eJW4+bMsx0FHaIbJ3HLbVkb4BdBDwq WcMnrrki73OTaE9Qji40HIG2KPRXSMF0zXZ3KIZGN3fnOgCLoTZa9nizzongspcymjja /G0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PEyq7LNyz7JntBDQn9aUhPG4pYZkRAR/BiMR78ZNWQU=; b=LsdMpxa1mnq6jEu5Y8KRxSB0PRmGdBbD+8nhV3Uz50XhiNgiWv5cmxu5b0p9mO34t6 DcfkrNlHMIfA63tC9QLEZQvid8ieOEMFQNjndQFWtumeB/xTGbxnmM9WCApaSrTqecSJ LlW1NKAfDpCEwT7qTflXZrLj3Qu6ZvylZVAu49dtBwBO+HsZadJ/HHQX+7IUB7RVHEIV x9CRKCyFuNSseeg7rktM4A9fLoi2ECrtrQNjyxKTm53eQincZH7czf9MKOjo6WPkOjJH X/qVeR93frn6868MBIFuRzOfUBbVM1Sv9813RXyJnWZ0XPHW0ODsTp0azbq9YERTrkwB ka6A== X-Gm-Message-State: AJIora8kSK0CKwRmxEnd3pE7CJxVOMWpLst7tKr9Q1Avt+dyZ93Tcu2X 2foQspEwPg15jYz6Ke2rGWANWuUO2deQqYrL X-Google-Smtp-Source: AGRyM1uEWB/fSSY6upQU0eNmh2VwRSYO+7LThlk37thEBG9nk+wfVG35vhp/YtN4bq9CCEyh0zur8Q== X-Received: by 2002:a17:90a:e013:b0:1ec:7b12:e326 with SMTP id u19-20020a17090ae01300b001ec7b12e326mr25619585pjy.13.1655835767824; Tue, 21 Jun 2022 11:22:47 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id t13-20020a1709027fcd00b0016392bd5060sm9755503plb.142.2022.06.21.11.22.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 11:22:47 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 5/5] knotty: reduce keep-alive timeout from 5000s (83 minutes) to 10 minutes Date: Tue, 21 Jun 2022 08:21:01 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 18:22:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13783 From: Ross Burton The keep alive timeout is excessively long at 83 minutes (5000 seconds), reduce this to 10 minutes: this should be long enough that it rarely triggers in normal builds, but when it does it has useful information. Signed-off-by: Ross Burton Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie (cherry picked from commit dcf52157d3635925491783be656c6b76d1efe1a4) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman --- lib/bb/ui/knotty.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index 71c5a3e9..d1f74389 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -599,7 +599,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): warnings = 0 taskfailures = [] - printinterval = 5000 + printintervaldelta = 10 * 60 # 10 minutes + printinterval = printintervaldelta lastprint = time.time() termfilter = tf(main, helper, console_handlers, params.options.quiet) @@ -609,7 +610,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): try: if (lastprint + printinterval) <= time.time(): termfilter.keepAlive(printinterval) - printinterval += 5000 + printinterval += printintervaldelta event = eventHandler.waitEvent(0) if event is None: if main.shutdown > 1: @@ -640,7 +641,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, logging.LogRecord): lastprint = time.time() - printinterval = 5000 + printinterval = printintervaldelta if event.levelno >= bb.msg.BBLogFormatter.ERROR: errors = errors + 1 return_value = 1