From patchwork Fri Aug 5 10:32:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 10990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6D22C00140 for ; Fri, 5 Aug 2022 10:32:39 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web10.5519.1659695556928293962 for ; Fri, 05 Aug 2022 03:32:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=eNfhIbnQ; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.52, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f52.google.com with SMTP id l4so2718856wrm.13 for ; Fri, 05 Aug 2022 03:32:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc; bh=GYFe283MHiR69w8+jOB1hXP7ILkG9qoTUTErGESO098=; b=eNfhIbnQeQzr9XGWxC56sJzG+uC8agiQXZsTcWGB+q8FlPpwH+7tH9uZks5Iqewobl igsJiRWTVy86ssyq8RaVOSYLA65tZCKJZ1BgG4wrofVVyuvBJqetjoVo6PxoeU8VFf8T wF7DRxkzGE4v8E7AUAG3qB0/UjyCw6+R8BQRQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc; bh=GYFe283MHiR69w8+jOB1hXP7ILkG9qoTUTErGESO098=; b=yJMajLjpSr+95G0pi6oHfR/TdvuYqgMoJWpB2TInQELIMED5qyvQP30+QJG46om6pE 6sezi4OYj9T1uyeACv0uPszbebWydrmHMGXeoJouww8x2aAWzefmsPNQ+MuUzimBw4I6 JvSVzC9nypnrxRBQPl5J+fFljiof/U07N9b2EFlknUI1KKuXaOafFPJZc5ElcqSIoWhs OBSfX6Jt5cY7aich3BNC5ryeXfYRf2TGXnaZX5LCOVAbDkyxAxgqBsoJx2kaj0DalXLJ hd3wsea1iMOI6rdGOOyw8T3UPQabxQbkZsvR35/3QZnnoj2kHhSl2wcak8fZf4G2+deQ 3Lcw== X-Gm-Message-State: ACgBeo1PzRuy/LUvidMiqzkGolM671EDiisID5MVd8ZcbYUg5N6yQXZ/ LEyWTNgTHbJJC3e0HCUX+Z8hqvYhJRPWGw== X-Google-Smtp-Source: AA6agR6hLYDJxheu3hU8Lg5fWNxi9o9sKx89QFONlllKuhygqqChgxHPigPRTTVJZH8AkvxLX+OZnw== X-Received: by 2002:adf:fa4a:0:b0:221:61d0:c79c with SMTP id y10-20020adffa4a000000b0022161d0c79cmr2386234wrr.444.1659695554718; Fri, 05 Aug 2022 03:32:34 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:f483:fcb0:1452:5a5a]) by smtp.gmail.com with ESMTPSA id e12-20020a5d65cc000000b002207a8773b7sm4501401wrw.27.2022.08.05.03.32.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Aug 2022 03:32:34 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/3] runqueue: Ensure deferred tasks are sorted by multiconfig Date: Fri, 5 Aug 2022 11:32:30 +0100 Message-Id: <20220805103232.3674163-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 05 Aug 2022 10:32:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13873 We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 359b503297..4d4ac3b991 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2506,11 +2506,14 @@ class RunQueueExecute: if update_tasks: self.sqdone = False - for tid in [t[0] for t in update_tasks]: - h = pending_hash_index(tid, self.rqdata) - if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: - self.sq_deferred[tid] = self.sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) + for mc in sorted(sqdata.multiconfigs): + for tid in sorted([t[0] for t in update_tasks]): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: From patchwork Fri Aug 5 10:32:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 10992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA1BDC3F6B0 for ; Fri, 5 Aug 2022 10:32:59 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web09.5473.1659695557785809113 for ; Fri, 05 Aug 2022 03:32:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=MS5QIw0z; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.53, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f53.google.com with SMTP id j1so2798216wrw.1 for ; Fri, 05 Aug 2022 03:32:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc; bh=F2hCJeQbfoqBe21zqmSE+YDSfi4TlR5zW5ja4oN+C+A=; b=MS5QIw0zraE+kP10xQiwXrfMpAS9TRewaevt/tPkRLXSF0RP9KE26qZwVf0c0WHpeU vnTwBfSnzPmqBmzcwmoSWDeKXT9L68MQ1fifXb/eDKxAZlRoiLiRqLfio46u4W4qnzjL 1pOLKDAWZCYyT+Nuhrw3vT1c3LB+DzVxvPCfM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc; bh=F2hCJeQbfoqBe21zqmSE+YDSfi4TlR5zW5ja4oN+C+A=; b=74/O4N2V1cpvfLywgYmjJGmTeOXOSBPNekkWXNTK1c4+Rosz863nUovZr0cAu7o875 OGBRFEthy8Cz0n4XMDgLV4It0zs2ae+3SVwZw4OV9BdD+MZg4nfvDun+D8oO/tTAqKO+ kWHlmUywjaY1Yw0IKVuEjU/cjI2V0ASYROpbs7x72D0UCoFX1J5ulQYVr32oHEpxNtcH 57KFfNIwQKIhRSTpCD7/lw1bn/sInKAlWn3RI0nF6R2JFIsoPhuwzp/QZMOivs/8csHe D9oaShr4zL1sOzb8CaGAidhTQmc/bsKbeuYYTo6B+hPNLPMDUsnsRwVoqKfA8Mqg/Bb8 lqXQ== X-Gm-Message-State: ACgBeo2VHrYC5Cf7ifhQTY/A1zw4W5D1u/RY1iSfw+R0230jtLxYXfYb bXMS+l6nCxFAUums9MDyJuOlXvqLIuF5wA== X-Google-Smtp-Source: AA6agR5lQ4ITbGKayCr6UK0IzUD58GnxmxJOlSodlL0m1yI1faEKerLmYG0+ptPbLWaxiPBgKEWWew== X-Received: by 2002:a5d:584c:0:b0:220:590f:27bd with SMTP id i12-20020a5d584c000000b00220590f27bdmr3912590wrf.342.1659695555962; Fri, 05 Aug 2022 03:32:35 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:f483:fcb0:1452:5a5a]) by smtp.gmail.com with ESMTPSA id e12-20020a5d65cc000000b002207a8773b7sm4501401wrw.27.2022.08.05.03.32.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Aug 2022 03:32:35 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/3] runqueue: Improve deadlock warning messages Date: Fri, 5 Aug 2022 11:32:31 +0100 Message-Id: <20220805103232.3674163-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220805103232.3674163-1-richard.purdie@linuxfoundation.org> References: <20220805103232.3674163-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 05 Aug 2022 10:32:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13874 Tweak the deadlock breaking messages to be explict about which task is blocked on which other task. The messages currently imply it is "freeing" the blocking task which is confusing. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 4d4ac3b991..e06246f741 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2237,10 +2237,12 @@ class RunQueueExecute: # No more tasks can be run. If we have deferred setscene tasks we should run them. if self.sq_deferred: - tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0]) - logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid) - if tid not in self.runq_complete: - self.sq_task_failoutright(tid) + deferred_tid = list(self.sq_deferred.keys())[0] + blocking_tid = self.sq_deferred.pop(deferred_tid) + logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s blocked by %s" % (deferred_tid, blocking_tid)) + if blocking_tid not in self.runq_complete: + logger.warning("Failing blocking task %s" % (blocking_tid)) + self.sq_task_failoutright(blocking_tid) return True if self.failed_tids: From patchwork Fri Aug 5 10:32:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 10991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B89B8C00140 for ; Fri, 5 Aug 2022 10:32:59 +0000 (UTC) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by mx.groups.io with SMTP id smtpd.web11.5415.1659695558968168310 for ; Fri, 05 Aug 2022 03:32:39 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=IIjoopaR; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.42, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f42.google.com with SMTP id h13so2764265wrf.6 for ; Fri, 05 Aug 2022 03:32:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc; bh=xSuBCUQ5IyT8jvvSu5L3m+qUQyNgc+pMau4xWSucHo4=; b=IIjoopaRdkJ+3beGiv9q4gKGxpM6lNKb1iMWtXzGzms1tOPzAZ40IepKZgRDt9JyiO WoqjZZJm/s8YAFUtfqar9VcxldldNOJUCQg9E3LdWb3MxQCPVaPzdiFSqu0P4ca0eFkS m8Pyvbp46d6s6/bsDyHDNrMT0HxJYLvsekS8o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc; bh=xSuBCUQ5IyT8jvvSu5L3m+qUQyNgc+pMau4xWSucHo4=; b=21KQRu7sWS6Y/mhLkGhfjm0Sk680NMvWRFPL8gftRiIMPuG1g7eAXSgTQm5wJmTSom grIa9SGiMlZsdh7pFYWh6PqbvYguj3u/CBFN7AkaFUxmK45/xAKRaYEcReVFM51V4sO9 8fvIoxKu+lXuUxTbxEUagOfBBGMP9evWVt/wFPvMn1wMB2OdEQwgeW8lKb4OQhiG7yTy a1ppWvAp4/2OALKCxyv0FkKOD1mdJIh/LnwCGjZj8H+nr5GVYJshk8D9B9xbomDUDHgx FBB48ClYDMhhBnbXKyTva/VKHg8PHHRrO7YEulAm4cnmU4qNyXIiLY5PkJAZ8JpRoQ6j TC1g== X-Gm-Message-State: ACgBeo0uAnezZWfE77PFQZLAvsjKq17RgAo7nK/isugOWB0WJupYsGfg u2K0hcj9+Ul57OkSKHP6Ka7wLmCdH1XdCA== X-Google-Smtp-Source: AA6agR5llCBPD78COLFQMjCLRCPwKD9VRkUs1RWWbX9daQB8JRNUhHwNXVOpZhDd+B+K6Sc8Tp3j7g== X-Received: by 2002:a05:6000:118b:b0:21f:110d:99a9 with SMTP id g11-20020a056000118b00b0021f110d99a9mr3682404wrx.505.1659695557043; Fri, 05 Aug 2022 03:32:37 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:f483:fcb0:1452:5a5a]) by smtp.gmail.com with ESMTPSA id e12-20020a5d65cc000000b002207a8773b7sm4501401wrw.27.2022.08.05.03.32.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Aug 2022 03:32:36 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/3] runqueue: Drop deadlock breaking force fail Date: Fri, 5 Aug 2022 11:32:32 +0100 Message-Id: <20220805103232.3674163-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220805103232.3674163-1-richard.purdie@linuxfoundation.org> References: <20220805103232.3674163-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 05 Aug 2022 10:32:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13875 I'm 99% certain this failing of a scenequeue task corrupts runqueue and causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted and did weird things so drop this code. We've seen builds where the deadlock triggers and it then tries to run tasks where the SQ task already ran with very confusing failures. It is likely it is this code causing it. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index e06246f741..98b73223c6 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2240,9 +2240,6 @@ class RunQueueExecute: deferred_tid = list(self.sq_deferred.keys())[0] blocking_tid = self.sq_deferred.pop(deferred_tid) logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s blocked by %s" % (deferred_tid, blocking_tid)) - if blocking_tid not in self.runq_complete: - logger.warning("Failing blocking task %s" % (blocking_tid)) - self.sq_task_failoutright(blocking_tid) return True if self.failed_tids: