From patchwork Tue Nov 16 16:26:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 52 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CF9CC433EF for ; Tue, 16 Nov 2021 16:27:48 +0000 (UTC) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mx.groups.io with SMTP id smtpd.web08.319.1637080066468883180 for ; Tue, 16 Nov 2021 08:27:47 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.20, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10169"; a="220936567" X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="220936567" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:11 -0800 X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="506500474" Received: from mmohdso1-mobl.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.237.91]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:07 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.52][PATCH 1/4] cooker: Handle parse threads disappearing to avoid hangs Date: Wed, 17 Nov 2021 00:26:57 +0800 Message-Id: <920111a330be59e5be2068a8f1a9edcbc6c14402.1637079892.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 16 Nov 2021 16:27:48 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13061 From: Richard Purdie If one of the parse threads disappears during parsing for some reason, bitbake currently hangs. Avoid this (and zombie threads hanging around) by joining() threads which have exited. Signed-off-by: Richard Purdie (cherry picked from commit dc86a533d951d13643ce446533370da804782afc) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 06b40c13..8ae8e4ec 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2040,6 +2040,7 @@ class Parser(multiprocessing.Process): result = pending.pop() else: try: + time.sleep(0.25) job = self.jobs.pop() except IndexError: self.results.close() @@ -2218,7 +2219,7 @@ class CookerParser(object): yield not cached, mc, infos def parse_generator(self): - while True: + while self.processes: if self.parsed >= self.toparse: break @@ -2232,6 +2233,14 @@ class CookerParser(object): raise value else: yield result + for process in self.processes.copy(): + if not process.is_alive(): + process.join() + self.processes.remove(process) + + if not (self.parsed >= self.toparse): + raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None) + def parse_next(self): result = [] From patchwork Tue Nov 16 16:26:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 50 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B502AC433EF for ; Tue, 16 Nov 2021 16:27:12 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mx.groups.io with SMTP id smtpd.web11.297.1637080031078195453 for ; Tue, 16 Nov 2021 08:27:11 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.126, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10169"; a="220613336" X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="220613336" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:10 -0800 X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="506500479" Received: from mmohdso1-mobl.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.237.91]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:08 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.52][PATCH 2/4] cooker: Remove debug code, oops :( Date: Wed, 17 Nov 2021 00:26:58 +0800 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 16 Nov 2021 16:27:12 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13059 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit 19291665fa8b6cc331290f2542af3e8e653203f1) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 8ae8e4ec..9be6603f 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2040,7 +2040,6 @@ class Parser(multiprocessing.Process): result = pending.pop() else: try: - time.sleep(0.25) job = self.jobs.pop() except IndexError: self.results.close() From patchwork Tue Nov 16 16:26:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 53 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A321EC433F5 for ; Tue, 16 Nov 2021 16:27:51 +0000 (UTC) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mx.groups.io with SMTP id smtpd.web08.319.1637080066468883180 for ; Tue, 16 Nov 2021 08:27:50 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.20, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10169"; a="220936599" X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="220936599" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:16 -0800 X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="506500485" Received: from mmohdso1-mobl.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.237.91]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:10 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.52][PATCH 3/4] cooker: Handle parsing results queue race Date: Wed, 17 Nov 2021 00:26:59 +0800 Message-Id: <8eaddb92a5fd14de6b5995aa92a6eed03b90a252.1637079892.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 16 Nov 2021 16:27:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13062 From: Richard Purdie The previous fix introduced a race where the queue might not be empty but all the parser processes have exited. Handle this correctly to avoid occasional errors. Signed-off-by: Richard Purdie (cherry picked from commit 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 9be6603f..c032762f 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2218,24 +2218,28 @@ class CookerParser(object): yield not cached, mc, infos def parse_generator(self): - while self.processes: + empty = False + while self.processes or not empty: + for process in self.processes.copy(): + if not process.is_alive(): + process.join() + self.processes.remove(process) + if self.parsed >= self.toparse: break try: result = self.result_queue.get(timeout=0.25) except queue.Empty: + empty = True pass else: + empty = False value = result[1] if isinstance(value, BaseException): raise value else: yield result - for process in self.processes.copy(): - if not process.is_alive(): - process.join() - self.processes.remove(process) if not (self.parsed >= self.toparse): raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None) From patchwork Tue Nov 16 16:27:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 54 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CAF6C433FE for ; Tue, 16 Nov 2021 16:27:51 +0000 (UTC) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mx.groups.io with SMTP id smtpd.web09.313.1637080070688926670 for ; Tue, 16 Nov 2021 08:27:50 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.20, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10169"; a="220936600" X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="220936600" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:16 -0800 X-IronPort-AV: E=Sophos;i="5.87,239,1631602800"; d="scan'208";a="506500497" Received: from mmohdso1-mobl.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.237.91]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Nov 2021 08:27:13 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.52][PATCH 4/4] cooker: Fix task-depends.dot for multiconfig targets Date: Wed, 17 Nov 2021 00:27:00 +0800 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 16 Nov 2021 16:27:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13063 From: Richard Purdie The right hand side of dependencies in the task dependency file generated by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix this. [YOCTO #14621] Signed-off-by: Richard Purdie (cherry picked from commit 1d5ca721040c5e39aefa11219f62710de6587701) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index c032762f..3688aa08 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -814,7 +814,9 @@ class BBCooker: for dep in rq.rqdata.runtaskentries[tid].depends: (depmc, depfn, _, deptaskfn) = bb.runqueue.split_tid_mcfn(dep) deppn = self.recipecaches[depmc].pkg_fn[deptaskfn] - depend_tree["tdepends"][dotname].append("%s.%s" % (deppn, bb.runqueue.taskname_from_tid(dep))) + if depmc: + depmc = "mc:" + depmc + ":" + depend_tree["tdepends"][dotname].append("%s%s.%s" % (depmc, deppn, bb.runqueue.taskname_from_tid(dep))) if taskfn not in seen_fns: seen_fns.append(taskfn) packages = []