From patchwork Wed Feb 15 10:43:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabio Berton X-Patchwork-Id: 19570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ECE2C64EC7 for ; Wed, 15 Feb 2023 10:44:26 +0000 (UTC) Received: from EUR02-VI1-obe.outbound.protection.outlook.com (EUR02-VI1-obe.outbound.protection.outlook.com [40.107.241.85]) by mx.groups.io with SMTP id smtpd.web11.12143.1676457746724761121 for ; Wed, 15 Feb 2023 02:42:28 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@criticaltechworks.com header.s=selector2 header.b=zVQvCvbY; spf=pass (domain: criticaltechworks.com, ip: 40.107.241.85, mailfrom: fabio.berton@criticaltechworks.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ju75+1xwZ8a8LfhQpBA/vw1uBtdDgX8HedmnFRKUEwVNruV5HL/x1fkEFqDGOxtobaGs7gZ/Ib0lriCm1ZvDoEm9AevA/Wwtpzap8vamPURYgknpMXS83Y9P22WjzJYCzBSRyj9Fj7Sa/Ghn1sz74Nv8G1lltOj/3zuJzsx6bPA1fHCijiefIU9T+rxWpvJljaP8MPA0Z3HlZvYY3QXbmTWVSqtVSCzhOeRjrgjvK+oZcIdGwslnwIriAZguLRWGXGbU/9VUESSnciEiYFBYIv8GV1qjBsHiAYve4qctYZ5364tPk5Qj41Y7EXS1aVJVcQeJmUI4maOlSaKqCOnNLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uQr6OaCpDNdFsBvKZ9VcNiSYUb/L4iB5IH2ykF8IduQ=; b=LnOqRek6Kwf5fpNQRwBWXuF+UlFnoeAgYJ6zn833QryzEi2X+0HbdlmrCx3PcVMEv1WndabUVOdvknxfe6T85RJlnPAqe/04aZl4z0FWlyQ2Wo6W8qGbYEeOKQvucnPJnIi0kooKv597j/nwLMUY7Udpg3VsrBEpfEWK8bPwFiIRQipqaoshEpNscXSN41vGqzc8Z0ej9KyIxNHWihpw29+2N2ulh5anW+U7Z2ORsnzCS/gYhzLJAbubJGNcWwSF5PavFVwUJzPLOnFVTfRxB4qhGTBVx83OAe5HH6MmCTz5sJVvFjKiKSBMMH8lUYcKUWD0DkeCSOQp+sDGdJxDAg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=criticaltechworks.com; dmarc=pass action=none header.from=criticaltechworks.com; dkim=pass header.d=criticaltechworks.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=criticaltechworks.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uQr6OaCpDNdFsBvKZ9VcNiSYUb/L4iB5IH2ykF8IduQ=; b=zVQvCvbYOXm4jD7mBqYhZ1MgXGOi8B9puc3T0PCV2wM5sAWZppMKMcNh7r8yi6C7PPCvPCc8K7xTk7TIitr6Q82TAoko7Qj5jiWds12iNJNMYLbO6vruZ4rDEXuPZwaOVysU26hxvD42HcD7Mqe6zXxzNmPGx0f4k5OSSvjsL0H0vJXsQhGesJrCeSKG7/LwYUdKXE6t+vxKG0quVUZTG9EHSbdTPtNi1FW1I8bLZu7jTe7hwstllSCR2KOprgE5SNy+5MKz1XXXsCeuuSfZdrraJi8T4kBxsQtj0bFkhlZqWmtBn1jTbVuzfMfc6hbDXxCUeOzI5QrFRkwWn1uWcA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=criticaltechworks.com; Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) by DU2P193MB2034.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2ff::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26; Wed, 15 Feb 2023 10:42:25 +0000 Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8]) by PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8%7]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 10:42:24 +0000 From: "Fabio Berton" To: bitbake-devel@lists.openembedded.org Subject: [1.46][PATCH 1/6] runqueue: Fix multiconfig deferred task sstate validity caching issue Date: Wed, 15 Feb 2023 10:43:16 +0000 Message-ID: <20230215104321.1665505-2-fabio.berton@criticaltechworks.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> References: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> X-ClientProxiedBy: LO2P265CA0305.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::29) To PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXP193MB1519:EE_|DU2P193MB2034:EE_ X-MS-Office365-Filtering-Correlation-Id: a29019f0-f384-4ad6-9d4a-08db0f4152e2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GCQCzEhgrqLvzptr4t4aTbbj1x7TWDhKZvPkFqmdwy/+Ut9DNkCEcnDjOcv9EVdjLqDyGIm2+Vej+j4qYndXcV0Go1CMXsyP9ZA1uIHbOwb0Ko2KeNXblmNl93MwFZb3Fc6dI2863NCbV8Ef5sEDbrQKYDl//FKltdZvTDK5n+Ce/flj03rvlzlAROLrrywU0lqu2ik+rUyXRIhgxwPF8doC6riI7HHPMKXtCwf3FUhKpfIN5dTGatrchrqYCQQQt2BNUC3Rj0dNRbf4wFwWng+ExB1FMNpUZNRP+NV4l12dgnO+Ru+1hLk8QM5C71fcKJN1xbiX0n3BsmiZSw/6TFqfMoL7ykzFCIWAc4fkaciCV6cwu3o5hkNEuegzUCBDqjHp/AZaj56S/2/5KV6tg4a8e1459F+1iPupBPW6K6CUUQFV2zjoHDI1s3RcKutAApar24xq2DycQuixLTg5+DHqnFe3Ja6b/x7uogP0t6bS2rzR8Pt91xJacYsbR2SzyHcMJJJYM0V6utmG7TDsS8NQ7ZawOUcfnMeAPCDmYzQPP4wvx1SUpHb2zPUzYqVDCR1MaYvUT5XX9O09oIOpCqT/Ndu5Qb7D6xpZZkC9epqJ++lfSd2i1mtcfOaIwMZt2VLQSSMofZPzpnWKf9poHiDIW9RsRh8D4pk4fDcfmjtNOmiFLRNJII0Dio1F/5LheMtV1ldIt2JfVHc7zUfeEA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXP193MB1519.EURP193.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(136003)(39850400004)(346002)(376002)(396003)(366004)(451199018)(83380400001)(478600001)(2616005)(52116002)(6486002)(6506007)(1076003)(6666004)(36756003)(6512007)(26005)(186003)(44832011)(38100700002)(38350700002)(41300700001)(5660300002)(86362001)(8936002)(316002)(66476007)(6916009)(8676002)(66946007)(2906002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ioFawyqEkZaV1/8wzkDuTCJy1NSePsGrjDSMEOODacX4ibpiWD1XmLkALqHmec074//prhYRU0+5U0u5tmHO0HgKi+L8lncU6ravF8gsUg1W5TEGI4tuPA4pu9w4NkCEjEn31r9tXr66DDeZvVLv1hjTZmZxGKVlpM7lHGZhlwtZFiGAqzfEK9agfc3NP0kZrduYgcbttewNfpvTNaIV1ZClVzLRjCl+WesnXkwpDycqqjQj11SEOnH8cIoH+Hkp/+Dx5kzYmrESXMCHJpj4BzHj+fGsWQSEUd43/5A+ZVnVS2GLMLYcJyirRG77BuBvE9ziMDTvKTSS6ADwsNckiq2tdR4gzZGSebIUDxah7MD2qy9HkFZo8tUeZPdA0/VVENmo+znpxUq22N+oh9y0ZB12h2tpfnN5rI85WUPC+7WgABh7KmcdOd22dIakUxz+Wkj1EgSl5HmOYd020S6o+nRPsC46tgrhCNGrVFVMtuKL7SAKDmgtMuCqlx2/vDJn5FmHwjWA7ZXHUuyCOajYPeBmpMrn+fyLZl2ANJGG1IbZ6Re7WYMScyPa+QH74ZPqLq+IaAhxqR/C2yJkubNv8iWst8cULL4Rn2jCrAu/KRVm5KA95TX7S5d/StrkQ2qgNUDsxCF2nK99rfKgdKZU9r4a4dhL02pcRLpfMqt5573nRCCC+IAGzl+sjT+tHLOixeMC/RdaNpSLmHBW6d9BZB8D8MBKEavNNTyb9JD4VWWSdnKnBU3ySzjUwkgYDNJKHMjyDgbGVFy4cpd8+96CWZq4V7HS/Z3HO1+USJD1huC9/7NnxmCsTyVnabuyq9W3VRedD2fsIcJDW5jP9mPBfyaNbNUkEuYZ75lH6JzhKCN5C+Gs7hU/BKTS6nBauH5evZKwury0EFXLXoS8icHgJMEklZgDIl+WhnlfNFfRxxdD3jP1N2ybhXAyddhVoeQHmITifP1rvBYi72eL3dIg+1f3xQkBlm/xEoOO9oNDXmP5iNslmheMAZ69DO1Zrszz6o6bR0p50tH7NGRTPUCGVTdcA+/gfCPDWWH2ACj7Mfi3z2qmidrSW6ylJt0idOGf9RVJi/SmLoBuWklZc3+Tm2QQk1roPhKjWAdt3uVr0+91wxRkAGRRSsu4bSk3PMrh0NEFUiB2fwDNWDdDpZx5deX7JWxP8jYorYJvZYpJTPHthGTK26c2mhlq6j+LgERbCwqNZan0La/UfslrT6Nsb8oukMQ/z8hIV+8dTdzHGANUkJMo8X8g3D8BmZDVpV12TBInYn3Hhy0eY8xdVVPFnpktia0YN33wZ2JPWNaqKK7rRDg962sEk5w1SLoCq9+pSPIgauVOF4Gujj3YZYmH6nXQUU1A8XKOlFodqlnUFISJMASyIEM7oCk9NTgz8F2OoZjJXFPqPV/LHbjCrQpjpXtfgmtzv22U2t4hJrcimBmglfhuNNwSmhcMkaJ+RZqelcWZsViXlq6Whmj6uv/YLtK1OsJaQDKg1sBRWjdzbBUi97vtS1tQ7T0bwp4nqeBIFV49AjQz/3ySXuUyV2UzCOySI0M1GuK0PHjPpimRy8fqBkGQ6N/LtQB1TKVEvZU6J2AolAxJ1ktak7reyX0w15hUll6RDr+CQMAZQXLMoRM= X-OriginatorOrg: criticaltechworks.com X-MS-Exchange-CrossTenant-Network-Message-Id: a29019f0-f384-4ad6-9d4a-08db0f4152e2 X-MS-Exchange-CrossTenant-AuthSource: PAXP193MB1519.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 10:42:24.8759 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 61f30b8e-4f6b-44fe-9bc2-041e3a9f7346 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: f1TNlm+SuuStotpv+YxT+IUmqWOFc0tdTUVbVza95XdCPUcx5UxDuu8cbXN6mKe2clz+NRhudelv0JYvmyV0jRY+23JNW6hnRadBc6GODo0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2P193MB2034 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 15 Feb 2023 10:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14427 From: Richard Purdie We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. Signed-off-by: Richard Purdie (cherry picked from commit edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Fabio Berton --- lib/bb/runqueue.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 6cdc72a8..f82bc413 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2084,8 +2084,6 @@ class RunQueueExecute: logger.debug(1, "%s didn't become valid, skipping setscene" % nexttask) self.sq_task_failoutright(nexttask) return True - else: - self.sqdata.outrightfail.remove(nexttask) if nexttask in self.sqdata.outrightfail: logger.debug(2, 'No package found, so skipping setscene task %s', nexttask) self.sq_task_failoutright(nexttask) @@ -2843,6 +2841,8 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s sqdata.stamppresent.remove(tid) if tid in sqdata.valid: sqdata.valid.remove(tid) + if tid in sqdata.outrightfail: + sqdata.outrightfail.remove(tid) (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) @@ -2887,10 +2887,10 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s if tid in sqrq.scenequeue_covered: continue - sqdata.outrightfail.add(tid) - h = pending_hash_index(tid, rqdata) if h not in sqdata.hashes: + if tid in tids: + sqdata.outrightfail.add(tid) sqdata.hashes[h] = tid else: sqrq.sq_deferred[tid] = sqdata.hashes[h] From patchwork Wed Feb 15 10:43:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabio Berton X-Patchwork-Id: 19572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53D32C636D4 for ; Wed, 15 Feb 2023 10:44:26 +0000 (UTC) Received: from EUR02-VI1-obe.outbound.protection.outlook.com (EUR02-VI1-obe.outbound.protection.outlook.com [40.107.241.85]) by mx.groups.io with SMTP id smtpd.web11.12143.1676457746724761121 for ; Wed, 15 Feb 2023 02:42:29 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@criticaltechworks.com header.s=selector2 header.b=Y/Rn7MzL; spf=pass (domain: criticaltechworks.com, ip: 40.107.241.85, mailfrom: fabio.berton@criticaltechworks.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=huHMu1k6V9cGRg6q1hRp5I7UKxvY0x6rd0xCfuRluAx9lzNyb5Pq7UQmkkuHic/4f/UJQRjysO0RcZ7HolGkcFK68xiUO2CplnkEqaFz052BQv+0rM+jUc+/yL5JLyX+gkWdtu4pCbN9AE08L+xWHxjmNqfWC6nu1wKxs2AoAqO3N3RGu23zMWR6IZkw+5RjIxsMvMqv4fZzjm5EWefzB9hSJLgGLLQTkGACLGFq3IaHbT+NdZoM8lNEa9aLNhz9MIhkM7B07oGc/BXedijXkU8gt2tL8Zvnt8NfDNXQQFCz+2Qk9Jp3N5XqrLsHyGtZFp3f66Wt5IR2tjHwM0dHoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Mivc0Q2IimyZP1H6e4e/Q3m865Y//aHybwU23tKd39g=; b=hycBYYI/1ZfdQNMg8i+RbbRy1jPo9lisu+Wk645WmX6fe5h6BzxaAcnBAka+vaUf0A4VzFsB8M7mKbRZ1aarQy43yi3qUyWDbQ1HwT5t6Ojqeh1GXi4XrVyN2utlWwsV0G20LXsoIFH5mC+gxw4a3vJy3Y/srOtyy3vO6fAIkr59V21wRMMa95uZchfvnC0gI0iPWGOnIDXg28Ox14Vy0oMUcz3oxec+zUXyPWkwqBG6tZom/k4JPtiKsAHjakCmILF2ZP0tnr1vP9144YK7YRwRFKt5bcqEd/mt4F/dQlsl0K8PkeAL5ppfLIxXKtU32pQFARLmgStJQqRVyGPgRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=criticaltechworks.com; dmarc=pass action=none header.from=criticaltechworks.com; dkim=pass header.d=criticaltechworks.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=criticaltechworks.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mivc0Q2IimyZP1H6e4e/Q3m865Y//aHybwU23tKd39g=; b=Y/Rn7MzLXstsEx2iECnH2//QnWiJkP37kcPzdtV+nlLAfjrDT11sTkU6GNe7jcecgTjRlaVqpijr7nMIMbJdXwE/mLa40Ojj8hi7pfHpUFZUUnOP/vwPxdMVBrzmlOQjbbgPIkRzd4A+cBJgOk0Tj+e4rPlv/D6YL4AM9CmVGdwmn5ziYbW48gFl9YlI1b08tsPOJVwolGTfccFbmN7cbGRHvpQmhpUuxBRJEJMVIiyQVeyE+/fWDS3JSSfbBeWzp7HZgLpSe/osVQUBDoKaJthxoEkMLBeYr4MiU8pcWkri3qk8aMk80PKNd1fmkfctVdZp7lf5stRvXISW/3xB1Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=criticaltechworks.com; Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) by DU2P193MB2034.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2ff::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26; Wed, 15 Feb 2023 10:42:25 +0000 Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8]) by PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8%7]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 10:42:25 +0000 From: "Fabio Berton" To: bitbake-devel@lists.openembedded.org Subject: [1.46][PATCH 2/6] runqueue: Handle deferred task rehashing in multiconfig builds Date: Wed, 15 Feb 2023 10:43:17 +0000 Message-ID: <20230215104321.1665505-3-fabio.berton@criticaltechworks.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> References: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> X-ClientProxiedBy: LO2P265CA0305.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::29) To PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXP193MB1519:EE_|DU2P193MB2034:EE_ X-MS-Office365-Filtering-Correlation-Id: 45dee73d-3ea8-4f6f-f8d7-08db0f415384 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wEezKxKaOa4cXyXFj5RsOTlRK1X3QsBMmFhfuSZLHqKzPEu66nV/9JPfOlYx9fMN0ds0GfUyEhtkw0xPlvuth40zP4jlavtP9t4gc6LdQAHj/sh4aK+PPnQwc8RK+8cqXtOWzlTKX5C9o1x9fiuLcdNuGKz0WiolifaTsWVqJs6nTrvoaZIlj915GpBrFtTERUGJ5uWx46AyahQi9SrY21g+QguAr53llzVXYsWG7IECFD7xFD2+qycSHSRjXEz7WeuRXp4dxT1LzfXH+yiK33P5ctino63wdShIznKMUb71uE46/Bf8K5zb84Pb07NeIVNzPi1eMPlx24h+pFG7cLngf/76+L99DsJ6gbFXDyMMepA/npj1oKKiZomD7bZdXabzOVMj0kG1Mj9NNCFpuCM536Bajf7UrPXCJRUI2iFrKQ1ZOveUDd0QHnLegiV7h4k2OCahHDCQ/r+/yLlgnfNbtWZho3QhHklecNx0YVCjMejxrelDaOUAlDIt5ftjnXC/FKrhb+ofC/0Ujdbrya93udIszjK0IVEAvnV5jiIGQTU0s7aFleyd4y20tEoUrLYnCJMYm2PbE+wzgFQb2rs02FvFuGy0+kaRnXs9mpMRQFwL+v/rPdd+9ZSja3iBItSGR2dAGOH+P+mJHpUhbN5SAMy6UrVc1ZkTnE108r6Fy5ZP0p60MvoblIFGQA86pmdR7skoXN0XSEKKqVQvIQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXP193MB1519.EURP193.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(136003)(39850400004)(346002)(376002)(396003)(366004)(451199018)(83380400001)(478600001)(2616005)(52116002)(6486002)(6506007)(1076003)(6666004)(36756003)(6512007)(26005)(186003)(44832011)(38100700002)(38350700002)(41300700001)(5660300002)(86362001)(8936002)(316002)(66476007)(6916009)(8676002)(66946007)(2906002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fCJ0RLUj4sc9oiLuWewNLBQ80P+Un5SCCKkCGIAuwbipzOZhnDzHDxjCYgxp4K/hpGU+cU05g/PfPEeSuZvjZ3zmgEwM1mh3AFH5DGObKyzXpEuBP+/NzcdUNCBiqLuouHJtpIotrAiEOz/uHc22B4pIohqzFNIuZdle3VxVv4QRa6uRbdW/oJIn25V4BiIdjtMloH1tqC1pvx5+pXlVhzTyRldmemF5+cg7Yqj1Sray7pTR059hNEcFPNANzRSJEzP6SwsI9PVQ7h2V6JnhicrKVWxIRO2cSz04/UL8e08goHMXe+y0Ay1Aaod5BeqWieoi75G2jpM/90luJU8f+il39G4cZJdm3s6kK0xbSDCOtxZRJpVQt3yhIQKUmD2SO9D6FDZbFWLLAzQFjIFDMziSVJiovJfUKrtwJho2hsrHQE2cTLxz5Z95VDtD3XggR+qWMKGMvtLW+XdBXul+Y8t2SI72IC7jeCyLlK6dUT3dMaTWZRLENGpiOa/4tk49HebicCuno/cnyxyDGlVadOlE40yjwcDzJDQrYF+S/c9GcAcMFS/Y+aR3uwZRIjmiagvacVzgndsTsb5Hmo5KoAERd9MmdWvsk2nOFkQEYFOfbP2vg9n0b0bLZOYVUq9/Gy8XLFfrck+Uu4Y5prXJhW6zFj5yrJW+fJGyFxZ5UwqSFrA4uoxOvecYtI/ATzY0KBvDp/Xwq/wG5cyV8d/TZzcA5j6FBGNO/UtDPZw0z0kr7JVMjgOKsSkTa8uSsa9lFC0H7FxpEpbL7MPHgSMuJVu2oNW6u44zfTIgrSIJ2mvqRXUhpqBwxmZFGI93Jz/utJtF03GJ8rIYXi7RBCNHd2jWdpdLjFDutN0UWI5tB33xw8gf+iYcEQD0mnEzSu5PBvpm+h50Byz/EBYlu2hTVfw9eFqucaQRLaNzwrBJ/IXr8wzWOU3xsJSKSPIiGhEXaaJWr6FbSrdg/kmLapw7iPAcAvqeEUGXu3Br401mP8QM5Rzf2v57aGMKsWZfED5BUQPRNJOMbEeyKFxgoQJR2WiEEduSo3U2lco9hMoRiqQK4LEb/fOwbW9OfOR4LWYGgV4FhxDS8ymeVL6GJrIzUFDRvSXFJUn9o43hwpAIhvcrWRETJ60nEAht1ciuHH/MxNTVyTwSL5Fj6j4SyJSBjIJEIc8gttyVdoqTAWEy2tu1ck1afjMZkiAjgDD1RGKs2cjSHt1O7xbwcDuE90syB5ch150bQvQsYgfbgiK1u/0jIcSRtZwoc+/ODYzLnaESQm60/J1lD+0mCvixATGD5erMgy4VnXfIQRt3p4hR1qeU69nXJZ7vpsHj9/bw37uWw8o9GzzYKcBtGww1ZQFF6nv81Q75TqCz+8EVEEWT3QIz6bUx1v9I7mxupj6m5o07O7OeLpeA4diE0vB/CUzI76Jd94nmZujUMYzkAktTCSvUwd3J2LlHqxkrSy1y9uutrJlhnIDVuE8SSgyl3F1PCbX8UGh32n5Dfw0oNnFhdFKdL9drUxlwifKO4fR0b7PXAHlCUWnu/ymHhvFQturPkvpjLFozrxFX9j6MGbzH8+JS1oFQH36ETP3qFfc7v0TQSdRsPIBqTYyuOKIf2xgQLp0HutLoF65zmhmYp37nlIs= X-OriginatorOrg: criticaltechworks.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45dee73d-3ea8-4f6f-f8d7-08db0f415384 X-MS-Exchange-CrossTenant-AuthSource: PAXP193MB1519.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 10:42:25.3446 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 61f30b8e-4f6b-44fe-9bc2-041e3a9f7346 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wwmPun5FOuS1YyxlZbuoceL8Fx537MNFXnenTqsaCe/M3V9PgQ80OLwTfW9kEKkx80sTgMXtcZcTOvusi6JjKLtrWlqMv9OMLyOXqvlSVdo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2P193MB2034 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 15 Feb 2023 10:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14428 From: Richard Purdie If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. Signed-off-by: Richard Purdie (cherry picked from commit 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Fabio Berton --- lib/bb/runqueue.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index f82bc413..a437aeef 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2348,10 +2348,16 @@ class RunQueueExecute: self.updated_taskhash_queue.remove((tid, unihash)) if unihash != self.rqdata.runtaskentries[tid].unihash: - hashequiv_logger.verbose("Task %s unihash changed to %s" % (tid, unihash)) - self.rqdata.runtaskentries[tid].unihash = unihash - bb.parse.siggen.set_unihash(tid, unihash) - toprocess.add(tid) + # Make sure we rehash any other tasks with the same task hash that we're deferred against. + torehash = [tid] + for deftid in self.sq_deferred: + if self.sq_deferred[deftid] == tid: + torehash.append(deftid) + for hashtid in torehash: + hashequiv_logger.verbose("Task %s unihash changed to %s" % (hashtid, unihash)) + self.rqdata.runtaskentries[hashtid].unihash = unihash + bb.parse.siggen.set_unihash(hashtid, unihash) + toprocess.add(hashtid) # Work out all tasks which depend upon these total = set() From patchwork Wed Feb 15 10:43:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabio Berton X-Patchwork-Id: 19571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B082C64ED6 for ; Wed, 15 Feb 2023 10:44:26 +0000 (UTC) Received: from EUR02-VI1-obe.outbound.protection.outlook.com (EUR02-VI1-obe.outbound.protection.outlook.com [40.107.241.85]) by mx.groups.io with SMTP id smtpd.web11.12143.1676457746724761121 for ; Wed, 15 Feb 2023 02:42:30 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@criticaltechworks.com header.s=selector2 header.b=XtZotCNw; spf=pass (domain: criticaltechworks.com, ip: 40.107.241.85, mailfrom: fabio.berton@criticaltechworks.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZgwHCFxn7K7XoZYGuY85A07B1oXNwTizW0eT8c0xFRLbpri+IugkBT9OHGICiFUT+Cx14Ag6brjJtQeWF7CFZ749i002Qzy03zy4Y312xuJ2+dMv8Nyo++5CZGdsmynRbBh3vkJOA6et3m2XcnwlPEH353iVyXRfsgwXlyuxX3ni8gNFm3zvNBOx8T2NR4mzMYc0E4mJOToLCQGzzAF1tx5BurvgFNaxzUFutesD5bAp0bn7xTvuD5gMPAaABdHSJuDSYs2DA+tW/icVMkA7cNF5gXx510X1aqx85iK9FYOLVvZU/XswPvVJrtJpeGmHP61BMJ6VH7ANMsJa6yfmGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Px38mVwyaZJxkmhHu0w19XXqwBdo9bv48HlQ4IG5FB4=; b=Sg8XNZbD1h0onmpd3lkkxwzDtgfOotuDQgD6WyOGHCIU6W3PXFqWJme8emOpbyvj0wDJm2VClP6wfkiBGmkHeZK1jS5U3gtF7WpDsuIOaHSEX7KpdVlvWlXiX6uBBtRQTwsfADHNg1xbWDoehZoYwA2ZouIoP/55Jew83lieWw1812mypWEynW6iONayXJcU8dXLUZRI8gxTuFgDyiLycnE+XZ1Fm2XJGpQlF/Are0d/mH/pWBReFLF/uWBKm9heCrhQLUxEdP7YyphhowU/KeeKm+t5f0tqtktYsw0UKiADc9dygemp/0Aen0tbtX1JmjmuUvVLij5vtROQjn/HfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=criticaltechworks.com; dmarc=pass action=none header.from=criticaltechworks.com; dkim=pass header.d=criticaltechworks.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=criticaltechworks.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Px38mVwyaZJxkmhHu0w19XXqwBdo9bv48HlQ4IG5FB4=; b=XtZotCNwzoubUbK1XMYlf+SJ0Tf7WvXnuL3eGBaldL36xF3m0bCCM/yHW8deWDPc8G01sLStUql+DPPYpTRj9Pcl6zfXIpIGVB75QzrEs9e+HIHUkYKTmHBPnHHYbBlW4Erm2wzb7F08aOfHk7xqxAmNzr0WDZkytSZySKBpSX1FT33rIhdtsW151FmFwg6u2zuRj65OiDHynB9UHX+g+k7ZlxufsVKfmKOAnNdds1mYWICiAL9zAxFjeiyB1yqXFJ1qRHTCRjw0eASbQQjeCr6O6RqVct8X/xRY6rCjVp/24mY9yO6NSBD/VjLmVVkt15l/+RskpQgungTMW4vecA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=criticaltechworks.com; Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) by DU2P193MB2034.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2ff::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26; Wed, 15 Feb 2023 10:42:25 +0000 Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8]) by PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8%7]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 10:42:25 +0000 From: "Fabio Berton" To: bitbake-devel@lists.openembedded.org Subject: [1.46][PATCH 3/6] runqueue: Improve multiconfig deferred task issues Date: Wed, 15 Feb 2023 10:43:18 +0000 Message-ID: <20230215104321.1665505-4-fabio.berton@criticaltechworks.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> References: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> X-ClientProxiedBy: LO2P265CA0305.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::29) To PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXP193MB1519:EE_|DU2P193MB2034:EE_ X-MS-Office365-Filtering-Correlation-Id: b3c2a10d-f2dc-480d-b781-08db0f4153c7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KeG1Ll0qilMrwobLf08vMxw4u6Q2MOjCT0c5OeZU4thtb9Nf+BmtzVmmMJI5zyucoWde+oNANMLeBBLyC5QnBqDChK2gIxodAb4+P/FFplDZP9x1qEE/ans7PghSlrBpCIXgeSgFuiIN3TR90IE1HHKstJrakUWJy59eaUcBuJva+3AiL6F9wBaU6C5TiUYUeQl1KsFOcwvjCekGNJOuPlEKPQaf7OKDB67PZnz+Q/SkSW2gJsShNGYp3G2XxfKm8IvpF4neBDo7cg655h2Ac+YsDLEk2Ae8IhF4PdB3rgntE4pyUjpQP8tmrrVGR+jysVOw+x/ovZ8cSv1uoYBH21UuEXLbyGYrKTX8pIJRNtpjC8IPPtv5lWr644ycrO3KJeMY5/MFSYh4nLYBHI0moeosm0nEgiOYdW8aeJCGRcplGxpyLkBiB3+bneV+p/weNI5fH7VdcSAEcV1GNt2tUFRRxb3jPe4vMzADzBKk0C+jOUT/unz2mHpPb7WrIIQiX2REH7vXZJR8Z5JOSsx9xXxpgp14NYK59UFS8/wlSAfEQCJglH4UqLlW4RxqFf9dD3ZLwx61b/6zByi2PSdsrjTCBShDkOpbYqfY6K0FP5AKVELHGjB9ZrvaDh3t1KVsSdBtDO9YGFaWg8LV/6g/ilU5h6MjzFc7TzyB5XuDbPDoqc74S4du+S2D1Gl2EJziBw9qmAss8/vWLyu6FXEPhw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXP193MB1519.EURP193.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(136003)(39850400004)(346002)(376002)(396003)(366004)(451199018)(83380400001)(478600001)(2616005)(52116002)(6486002)(6506007)(1076003)(6666004)(36756003)(6512007)(26005)(186003)(44832011)(38100700002)(38350700002)(41300700001)(5660300002)(86362001)(8936002)(316002)(66476007)(6916009)(8676002)(66946007)(2906002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qPvqxWfdD6X5ghsqLRlNM1VBDnIs8g4IzoaGz6xStarkSbmP628TuCyJY6ojwMmGd+OyPX3GqzjkEnoTkR/ijH7M5tHScYQDld48MVOIVI47nFF57aRApDD3vA3NMH7US6WVP000nbnlQ4V1xDTSiNnFI+nkvioE3RDtbHEGY3jOBzQxuqsH+4uBDv4GV8RJHEyntJivEKcc+0418AcfQY/ILJ+kNsqUz8mLHMaBxZ97WslNJAwqOHVHLaQ9ea3lAZwWpo4MWyfxtNKrHH6apXEGwY70pUsEDAr5XO2RiMISvtAtbi5UzmZTeiTWfdhSCCvEfBdpFuLTF6W2Ty0UWVF3w7yarucdB3CKNSJQwjHSvzcpP9F7XRWHlkbPf9+VyKK64eStPc/ntcoS2YuKo+1H5B82sNzyjrvrUHI2otJ1Y5/BxT9Oq3XobuPqm4eE50jTxrSL4BF0N5zj9pBSor2un5Pzde+rpfCNf9T+A84X73o9n0EFqnVud2A7YSxfweDacAXlhovGyc7W94vtJSJNCYUOW3jj++M/1nYQei4PXpq0ppzyetTYngQEyFdFSa7TTkiR7NFLfnkGCXRB/jNvjln5OjvnrPxIq2dFxaL9GK9rWeDH58i96hMPF/wEc1IrdDlEWAC2KTaXnyQ8mBuQEX5vwtpACX8PxULl+TdaV1A/9T40RxRGMN93tOYN2/2yen9/bIku9kigotQ/rxnG0WONj9M4dFsmnTwuk2WJWnov3MXUw7nq5XGO3ieYL+bzyKbZnwm/1NqN9YPLkWKdxU/3grcfIwiRfqED+1ZrEUoVq8G9cfcupqxIRcwNfw6jkZmr2w49k0aZKHXVD83ewGaKcK7Qp9LDTbl1G5NReujirsvWN29PEjkeh7gAXGa2Ckz/X57Ohs+ibvJOd0JRSBn3xEDMUMuAjgLokfWQLY0VvFRWfZo7jirKKR3d5FxcAN24SxSDsUlgOlN4+Aw8AxDdIhYaJOWjp8tYGCNlSjwz1uD+YAUU9guLwc/Q3x75NTTsl+xn7IdNjUC5Hl2V7G2wy6SOGiMyr9wRBUsHymWS+hu4cN+9qb7+QJYAZ0A+RYDZvNWvNPj7E5Ea2hsNiMEhr+2T1pY1SXAjJ16l0f7RXaNHrlkTiAJsyzGZFlJpGj8BulpZgy/c4zgJM7S2K40pMrOSG1/0xyQNx4k5CMNhdRZmeQuXt1807yVovbsXJzo/KYOooAyP9207eApY9Pcf9f8XJAqvOypJ/dSc1Iwd5ssrbs2lX6L5/oK/7QbqOM3mC6FF37Fzln1BptYLO9FChH3dI7MYbBlzCaPlTSWd2mzZFa01jr8+tBHwqugpYeqCar0YP4ar+yf7ouuJDqNp6FNX/k53MLUJky6jSbid/Yqi+1oH7uyYjNGeACn0Cpkbh+TUHJ0dQ7ZIXYVTyQZeYT3tmxn5Rr5o4SiO6rXo7V5fBPc7F+33c5r71R7I2/buo8A15sEWY+6n7bs8WGDwUmaxs1xqRegFb4h9o4IBeWvWTIpLMDTrwIskl97CLOcxb4uU/7gmnTmejjElEICWEtTLZbUdxTnf0N47Ft2AYonKClHvy3R/47UhYUs3UKNaFd10G+oR2rzFJTE15oysJm6VJvUk4QpX9rQ= X-OriginatorOrg: criticaltechworks.com X-MS-Exchange-CrossTenant-Network-Message-Id: b3c2a10d-f2dc-480d-b781-08db0f4153c7 X-MS-Exchange-CrossTenant-AuthSource: PAXP193MB1519.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 10:42:25.7976 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 61f30b8e-4f6b-44fe-9bc2-041e3a9f7346 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Yc9SG0Hl3273xZ6IPxxfw//o2Qa5kE1SZMl5jEQhdMrPeQ6zIzOUvdtdDNd6EoZhqReRlglZSwmBXOtOW5dOd3oWK2vbKb4Jj8bDaKdcJpg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2P193MB2034 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 15 Feb 2023 10:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14429 From: Richard Purdie The previous patches have exposed new issues with this code path, the issues being around what should happen when the hash of a task changes and the task is or is not on the deferred task list. Rather than rebuilding the deferred task list during each rehash event, build it once at the start of a build. This avoids the problem of tasks being added back after they have run and also avoids problems of always ensuring the same task is deferred. It also allows the 'outrightfail' codepath to be handled separately as the conditions are subtly differnt. One significant win for the new approch is the build is not continually printing out lists of deferred tasks, that list remains fairly static from the start of the build. Logic is added in to ensure a rehashed task with a hash matching other deferred tasks is deferred along with them as a small optimization. An interesting test case for this code was reported by Mark Hatle with four multiconfigs, each the same apart from TMPDIR and running a build of: bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal which is interesting in that the build of buildtools partially overlaps core-image-minimal and the build has a rehash event for qemuwrapper-cross even without any external hash equivalence server or preexisting data. Signed-off-by: Richard Purdie (cherry picked from commit bb424e0a6d274d398f434f7df63951da9ce305b3) Signed-off-by: Fabio Berton --- lib/bb/runqueue.py | 58 +++++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index a437aeef..9efd1c86 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2496,6 +2496,11 @@ class RunQueueExecute: if update_tasks: self.sqdone = False + for tid in [t[0] for t in update_tasks]: + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: @@ -2836,6 +2841,19 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): sqdata.stamppresent = set() sqdata.valid = set() + sqdata.hashes = {} + sqrq.sq_deferred = {} + for mc in sorted(sqdata.multiconfigs): + for tid in sorted(sqdata.sq_revdeps): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, rqdata) + if h not in sqdata.hashes: + sqdata.hashes[h] = tid + else: + sqrq.sq_deferred[tid] = sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True) def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True): @@ -2876,32 +2894,20 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s sqdata.valid |= rq.validate_hashes(tocheck, cooker.data, len(sqdata.stamppresent), False, summary=summary) - sqdata.hashes = {} - sqrq.sq_deferred = {} - for mc in sorted(sqdata.multiconfigs): - for tid in sorted(sqdata.sq_revdeps): - if mc_from_tid(tid) != mc: - continue - if tid in sqdata.stamppresent: - continue - if tid in sqdata.valid: - continue - if tid in sqdata.noexec: - continue - if tid in sqrq.scenequeue_notcovered: - continue - if tid in sqrq.scenequeue_covered: - continue - - h = pending_hash_index(tid, rqdata) - if h not in sqdata.hashes: - if tid in tids: - sqdata.outrightfail.add(tid) - sqdata.hashes[h] = tid - else: - sqrq.sq_deferred[tid] = sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) - + for tid in tids: + if tid in sqdata.stamppresent: + continue + if tid in sqdata.valid: + continue + if tid in sqdata.noexec: + continue + if tid in sqrq.scenequeue_covered: + continue + if tid in sqrq.scenequeue_notcovered: + continue + if tid in sqrq.sq_deferred: + continue + sqdata.outrightfail.add(tid) class TaskFailure(Exception): """ From patchwork Wed Feb 15 10:43:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabio Berton X-Patchwork-Id: 19575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6835DC677F1 for ; Wed, 15 Feb 2023 10:44:26 +0000 (UTC) Received: from EUR02-VI1-obe.outbound.protection.outlook.com (EUR02-VI1-obe.outbound.protection.outlook.com [40.107.241.85]) by mx.groups.io with SMTP id smtpd.web11.12143.1676457746724761121 for ; Wed, 15 Feb 2023 02:42:31 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@criticaltechworks.com header.s=selector2 header.b=dW61MZjm; spf=pass (domain: criticaltechworks.com, ip: 40.107.241.85, mailfrom: fabio.berton@criticaltechworks.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Izv6eAqZUTH9Q8sQd5s9/0wHaFO7C9mHItrHO4aminUkqY0nPEOi9vnEm6vLzjTdTTr1udScz0KYW1Irwm79Z9rtMNFAn4ol6QnuezdQiUsU2pXXwFkAGd8nwilJtYS8kn1gkFD6RIwuK4Rywii0i3wzqygiW8jjoDEyNnISp4mkOQ4GjKE8KatOUm3hMJih/Ntnb0ACxOX8xCy2RDQl1ivMWuTnFa27nGlKtWJI8wgLurQ7jOceyVa+Yzqfik/kAJVHP1OtvB8rgnD9TrybjTnFjq4tJMSRdQ2nmELosJ+ZDkLJXVWITmjO+xvpvFxYIU0PjldNVNJ6cuPLkgBJdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=miHKIMl2vakxB+gQwYNNBV0XugD6hbxWn7nC9QC6mOk=; b=aOpdkrEvZYLH5L84inobT88L93XFYvl9r6n16bHXtt4HtfORByr8ZXUbnOqJT9iesR9EQFebP4tU3Nliq4Gic6WOVSOOyY2T4vdhgLvxxWAmpMDMp7GkycxFuYr4kTt99ofgK8l3LY+hIvqLLGrRLUmMBnYN55ZoPA/saT/ImmWE+ZX2bkQAyj8hQTS4sE09VIiObr7tHmOiW0COwAbGoTD+QV6Ca0Q044VzHtV5Z3P8R5vJ8sAQaBmXJELg4MhPuKNsosIj1l3WK2MqQFsvJ2cTZisJ64e85hcWZ4ME6VU8cA5E0N2TwUA1B5qly0Y55rLvBI1CH1Wtr5Mk2gpKKQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=criticaltechworks.com; dmarc=pass action=none header.from=criticaltechworks.com; dkim=pass header.d=criticaltechworks.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=criticaltechworks.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=miHKIMl2vakxB+gQwYNNBV0XugD6hbxWn7nC9QC6mOk=; b=dW61MZjmXiTCGyZ95zzdPhRzL/48MsLXW7jBbOeoCPAcdVpYvVrZ1QLnomGtI+gMzdiRVK7lB2Io/PF84Z4n2qOENQVa/1T7nQIeU+GYWhLLIFxiFUPuWHObOn6og6hkBqMRNpj9VnDHLUOJP3WHGM9aukBjEWq3Gzm4NM3UurlOG/cCirY1cJ7/Q+gi8uz/hmunsFaRmgjCP75we6toMiyLP8IGUobqhJ/NiW+ycnRp+vUxJ/iFVcStPo0vpDp+2+0nrVL74tUECFXUJGdI7OOIQ/bhtB+kVSi6/gyH1hnaigPMyfvQvl97gF9WrZ5vGqKWAEx6t4SRKDc38+VVLA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=criticaltechworks.com; Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) by DU2P193MB2034.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2ff::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26; Wed, 15 Feb 2023 10:42:26 +0000 Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8]) by PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8%7]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 10:42:26 +0000 From: "Fabio Berton" To: bitbake-devel@lists.openembedded.org Subject: [1.46][PATCH 4/6] runqueue: Avoid deadlock avoidance task graph corruption Date: Wed, 15 Feb 2023 10:43:19 +0000 Message-ID: <20230215104321.1665505-5-fabio.berton@criticaltechworks.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> References: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> X-ClientProxiedBy: LO2P265CA0305.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::29) To PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXP193MB1519:EE_|DU2P193MB2034:EE_ X-MS-Office365-Filtering-Correlation-Id: e56635dc-1ec8-4840-615e-08db0f415416 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YQV1lMwdLJk4UnSmo6ma9pG4LKBvNaO08IUyo1Ia/QLMrxon1a0vA7eA3ZTyguibit7fbUIxcs5Z4PPzKvXTz3HkQKEZxaI4sHLyCmpdExRAH3CfJKTOJylOsod5229hZqHuudonm77+u2TDUQZKfbjIP18BmtV2DNx1wyydtKf1519zOZtxiD+BBGz7JRr+uAWR7WQ2BVeqUxa648R5IHmjajNvRwa0FjiG7ChzRNfQgCaUr3TScoA0OAb/00+cORf6Fzg4tSRyke1G+oWoExMDVT0lcyh6GfLDtfSvmWz39gQ6BE0XlTIs5sHNbCRgu4CFX78Df3PRJKmmTtcI7RIFmrbLLNmMQs8gvHCZleg6Vn+mC7eV2QxXsEYedo9Rmhh9LHHfaZFpjYfw08Fy5yXWPEUn2oxqLrdiWUW8LoI9/ZrjRUVhiQpYs7wmXJDaDIv3/KMFUNF+CRyEAMGVp4TAVEK/5HTmAKyJND9WZ113+naBsqy1f5+kQvCOdtlOsiuUU+OvfqbuNw5d7fuBy6HknLxV75wXIR0F4my04hZifpE6MM4gECTMta7exUi/MQIm4nf5fMQS4cwJHy69B5iIHof2x8fBu/f0R0PCLv4Gb43vzbybI0YtwHEsJC1diUsBPwMA2plfdlLugGFjKLC8QsY7j+Npv/LrTdmTSQUQSjDx91H6m8EnRzIjMyItH2/SeDas+dz8aJxsve/xeg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXP193MB1519.EURP193.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(136003)(39850400004)(346002)(376002)(396003)(366004)(451199018)(83380400001)(478600001)(2616005)(52116002)(6486002)(6506007)(1076003)(6666004)(36756003)(6512007)(26005)(186003)(44832011)(38100700002)(38350700002)(41300700001)(5660300002)(86362001)(8936002)(316002)(66476007)(6916009)(8676002)(66946007)(2906002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CwNjA3iwso+Zce3Of2HqL7QLcyMqMKPMELcgp1tqW2yjeh0OXbZHl6URBPk2moMIyKZjikj66G8xGpZnX/1R7YcCZUPHoEJz/BM5cDBm/IvqDymztYRhSo0UFZ85PsjlLkJY9jUJjqO/GzzRXqUeMBe6LtI3iU1kV/gZ0rM33j3eVIqW0kkVQAehzSl9pm8FkNPfPq0cjnT+bXfcADIm2f0Fu7mcRWE798bN9BuYDIH5oAMv3Ey4bhYl5p203dqIpehC0g03mNtjmMpVBgK/wu5vzZhm5YjdyYHhgE1NpKlTdb3d6+2LpayhinM52I+NUJeXxutGPOULl35Trac2jDINrtcP/5ITaT3thvyUSVaNnin1Uj28ZvTr1qOGJ/by2+MS3EnIW4qO8mtgJVOuycxYpCQofl0+p3pRJl7nbKKe2AyjNXvk/FD6omn/hgXhIlJXejfmZuLiK21Q5AE1Y8RqJB0tBbOK+DCwtYwWqILWSooHOWes/+/fHkErIEtoWORUE/x9cOAJLjugr1r+ewZTQpMks5tTNwAoFERu/e3oT28x3dtuw03CJDBwNJvA9riIOniybhgphX+8cYMk1sZ/UyS6OOy+jZffGDiSD69PiU7CdCzB7hPiRH9QopAWapB343qjkK4hLeVW5R6uirBNRoXfFtr27+EcWBWfvFIizi8wgGn1eu5ICsLEnnsSPUTKes8u2XIJvO/t68wDc9ZQ3Vi2IiGpSTsJVQDFAN4baupDBQu4xM2JzQ5S24Ys/TNPRjQMUUs+h6VIghONwtXm1BfrJaq4k5IYskqga4j3N2bHQQislL/cD47eUgSyFGEdkXv2g0vJOkHiZGUknEhQWCTnuv+ji5ZZoUHsbgRyGxlJb5gLC1nvfEAFmTtkrN5TbZEgG8LvhEbhY6MIExzlhGwz+8owRcmI12WJX2p2SKWs/6rQyaXnNDSmAd8LQCWaJSXtWpyQdTGlMoOUdpmWtaelk/Kca+7+PHJEMpdVxpobbIDDgrwi7M23JW5oJeIMC+4heoh+xX4UJ8rR2h6G93G/i5cXrgXDyNywNlHIOXGC515wtZOV/V3Z3T3haJvQPZLUoFodT/oviorz6MPJHM0BAxjnGBdfPYnuX3beDBRY5+9De4l8HJOFsaGzASX3w69CNXqt+ZywQNwF50lVnEJpgmmDoOK3QavPCQJeoDkZTEV5Nr5hkY0YITSmVEoMIZBWEAWDUiNca85LNigBas+Mdv2DmIX+q0LVL2a59YQnVkLlSBTUqflDcswil4l8Vxxch1c0cOMs2/V2Ux3HFZ2Vz1eiBfTGT08l/9wDNkaf8RY6BQZWPv1d7HcVcR5HwpX5SEi6zGbTouR2ot7SZhkLZEeLkElZlUjMDQchNMlWr+wSzY2I7rShGjs/l/joUb8eb/S2YiW1qnVgotvtUu7l0EpwWOhm/e9hrNfZHPm+Bk/ARgHuoYw+g11N8KcfTStMn2p/IPIK47GrdGUUfpkEX/0UGNIWL3oFc9m2Ir84dfDOn54Ga09dft8MpVaBXiJncIBJrCA/JbSkiTQ4EcMS101CK5cgIKYqfM9pRdHsA7E/MftINv5f2/x5QYt0u9N+nx/JxmUoo7r+sVdM3DKlkvwo0nZ8k4vKQhU= X-OriginatorOrg: criticaltechworks.com X-MS-Exchange-CrossTenant-Network-Message-Id: e56635dc-1ec8-4840-615e-08db0f415416 X-MS-Exchange-CrossTenant-AuthSource: PAXP193MB1519.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 10:42:26.3287 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 61f30b8e-4f6b-44fe-9bc2-041e3a9f7346 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6QMP6ydamzN5Dua0l2ci/qlSmW7ow8aqMg1SyZBCA53GE9k2/TImGEMNx8y1ykYC3vcXc4WctLSqA5CI/bB2fhlSGODUCWA+ge5kTGhUBsY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2P193MB2034 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 15 Feb 2023 10:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14431 From: Richard Purdie If the deferred task deadlock avoidance code triggers, it could mark an executed task as failed which leads to "covered and not covered" error messages. Improve the logic so if the deadlock code is triggered, it doesn't cause the errors. Signed-off-by: Richard Purdie (cherry picked from commit 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75) Signed-off-by: Fabio Berton --- lib/bb/runqueue.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 9efd1c86..c7f50fde 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2234,7 +2234,8 @@ class RunQueueExecute: if self.sq_deferred: tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0]) logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid) - self.sq_task_failoutright(tid) + if tid not in self.runq_complete: + self.sq_task_failoutright(tid) return True if len(self.failed_tids) != 0: From patchwork Wed Feb 15 10:43:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabio Berton X-Patchwork-Id: 19573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4819EC636CC for ; Wed, 15 Feb 2023 10:44:26 +0000 (UTC) Received: from EUR02-VI1-obe.outbound.protection.outlook.com (EUR02-VI1-obe.outbound.protection.outlook.com [40.107.241.85]) by mx.groups.io with SMTP id smtpd.web11.12143.1676457746724761121 for ; Wed, 15 Feb 2023 02:42:32 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@criticaltechworks.com header.s=selector2 header.b=srT+ysbU; spf=pass (domain: criticaltechworks.com, ip: 40.107.241.85, mailfrom: fabio.berton@criticaltechworks.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=R1WsK/s0Bb31S9z9heVmuWL6m6jJrAWgedtpcvK4ESH1A8JkLJXlOFfh7UzMNI0bBHUv7VqeZUcsV311uUNcsdkmT9OyBQYFM9Ekvdkv9sOl7GGA1n5Tg9RCR3A3UdB8Nf42kkfbcAuHYDeqWsMaX08vM0a4XUpDmkkQ9lq86bea6MQ86QI8YHMfZZJGjtDZjaRMrCRUUZl9LhdHhxeTI1gZ/kQCz8p/DywRjWeoDKOCcfOvRjSn9jzpEF8MTguGBwnU1xQL2YQwBjt02KMVIj1CWwsAUFQ0WurvKNq3DNd0fnEjqJnFeaE9E02YY9WJZezfeoIuLC8Mj9T4z+UVeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LtEOaUsvmqUyNdKzWK/Gq7ckd77NWpiL8I6PNk6uRLQ=; b=Hv8+8EJBCgE9U38h3UFdTr60Q1+k0yvrouxMrG5eQELcnLSzpHaC2RDXNXxFPtPbnQW86ZMr8AHaRIWJ7rWXpALie2r4oL4U+24lXqfREEqC4ASBOlqsi1hx6Q1gb1onPHx59aSoAqVzExYbTHyDh+xfDqeo8Jz8l9gwadAq8lAuvytFLPwAiAi2C0zEyr5XtI9rHuPVhEWVlFIQiR2C0DhKG6IqzsEVkf5S1zavxvTCx/EVdg9+x9NKV7z6E9uW10K1NSpZl2rnqpSOt2VjYqX/qP4VgHpZBbLoOaNFNR7mB7lKsp5xdfu5bMmGrrw+QRAKUCcx5SNgYyEBvguvnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=criticaltechworks.com; dmarc=pass action=none header.from=criticaltechworks.com; dkim=pass header.d=criticaltechworks.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=criticaltechworks.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LtEOaUsvmqUyNdKzWK/Gq7ckd77NWpiL8I6PNk6uRLQ=; b=srT+ysbUyrnOXtr3Kbx+WB8J7l1Doe03pbBQXmcc0f1Sj6mdCTXle6K1pbMSCiLivvis5yntgjP+IIK098LdVKa8+hpKIuWwtl/Rj6jV5FdqEoLCNNuQvpFZHHdCcuSyjTt8V1upNL1pY2BPsCpTuLIsWOKNkEXGTW2//926ePFbicimb2xU1jZjGd6fl6craOcA7IeDJsk3tN+iq+6SNqey5WCsu1uaBPA1c7p238YMxDpqVF/KKW5YnSE8vZaAs0kuAexOWPTAWttxfejXVrOE1awjWvfdqOmhQa/2veZ+JCaGKsou+JmS70TYkL6rfQRKJL3tUKnhAItxlzdBbg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=criticaltechworks.com; Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) by DU2P193MB2034.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2ff::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26; Wed, 15 Feb 2023 10:42:27 +0000 Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8]) by PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8%7]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 10:42:27 +0000 From: "Fabio Berton" To: bitbake-devel@lists.openembedded.org Subject: [1.46][PATCH 5/6] runqueue: Fix issues with multiconfig deferred task deadlock messages Date: Wed, 15 Feb 2023 10:43:20 +0000 Message-ID: <20230215104321.1665505-6-fabio.berton@criticaltechworks.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> References: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> X-ClientProxiedBy: LO2P265CA0305.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::29) To PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXP193MB1519:EE_|DU2P193MB2034:EE_ X-MS-Office365-Filtering-Correlation-Id: ef37e3fa-18c7-40f2-8345-08db0f41545f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 13nr31d+DWwQaZGLsMp2u8SpeHlG+Jfm5fXfw+mO9hCX4JpNv4HG7MwSSRvKOyhYEHFoO4VgnrAQrpHC8Cdri5IaNxCpmmM4LfbHbcJEbZbbodTlXXBHQGAMklRrLJRxjGc371QXRsnDN8isn5CA5Z7KnuV/Spxd0N+1tiaNE9SEIGakT6mKLq5q1i/WJF5iVdDDc8IJ4KhPAmW9BkVXWU8D0towfNhh/Lj/ucxEOlaHIdMmbPl3ZkMC917byHNXtp74x/lAQBgdJPgL6HeVT8JQKa2yPLPphbiNueH1B6OcsouiFBFMNvsXYBFg3HfeQmnkW9nAze5tA6hcIz8ozaOedsHJVuHyXh340KgZPpcRbZ/PavHvV3K1byjLZi4ed6Q1YOH0FvsgT46q12sJBGUsdsUqbxJzLSqijREKeMpWuGEkrsa4OLkdyx1OmChE7PHcIMMtJR0xGnqdaZgl+ipXGZ0V4AG8cdMwIoZT0fX/NI+0/T769r4jLSthtrDO1qtt4mUuNRDNszj9+JjEaRFX2u1wIzhGGRqhQlLj26W0BpAn0JmCDLObLdejWx4R+sQOhnYbjxW6p5+KCeJx1uitrUKcB7Kmq/mJTrLgd6f6+4teuGDFyvdi0OpFjRTpRbnOPLje3+vOuvIwM9QXmvo1Xq2ZFhZ32/g3NNhQ5HXaBsVDT2TQgP8StEtGXCUCeknVKfVNEzoKKhjMEK93Yg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXP193MB1519.EURP193.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(136003)(39850400004)(346002)(376002)(396003)(366004)(451199018)(83380400001)(478600001)(2616005)(52116002)(6486002)(6506007)(1076003)(6666004)(36756003)(6512007)(26005)(186003)(44832011)(38100700002)(38350700002)(41300700001)(5660300002)(86362001)(8936002)(316002)(66476007)(6916009)(8676002)(66946007)(2906002)(66556008)(15650500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Wg/Exg3ho0QSXj7Hms+koo/IGu3CJQv3rMeRHLwVWlP59JDHzyEDPwPzhghXLhONWVvx745IxPDWZ8TnAQFfvwyuzxbygaiElvJFQAVjEucdUNyldqhUnlvCmj8CKS/+hV73Hq66H1eneTrfGIWLq0M/J93hT4IYfkbspgLifJEfOEUy4iBbfVWYvzAD1TgWzuaKVwxak3fixR/2qdl+xEDgrTuW4d3oDUlTpu0rPVFIoQ6Ym/XV0qiZND+oUcHeA7Txe/sIsVQEu40Iwna+YX1gJU5GL77y+PdQcXE34tx++jcySTyagqRsKRCmejEDDhVtd2sWgoCmesGUns0xIemA77HyQJVGUovpIKqWKNS1DTouU5CmAEAw95vHEYhjRU94uAfuOtxsGt13I0l4JgvANZcUfAli9Op53Z5KZxTuZvPDtmOOrmPIb5Nd8WqjL9fe+KhyhPS+y7qGYX/FHpGoBoUiI1OGYrsufp09ZeLba3mNh5QHcMBpCw3+mKPavfT5e5Pn5kMUIw7nSFtjKlqprIjEY5xbNRj2TKbzx0vH+8jcx5I2NBIiVDeAuZfBnqI05QMQJX5L6LFD1SiIrVcdgVZeHdsgvB9oACu1W11z+zFFNJ8WOORF4/Ws3kh0QMuaajG5Jies+nt0xPDPNc/DdKU9s2FNMocTVTJdJ1hU+FQQESyzu1/FLL1iannE2X11zWmf1TI4Z19xlCnXJoJ6Amc9u3k6UVcYaWVclMabkPBAb9s/30edPWJw0P0BFasuzf5h9+itr0xJ+IW+mGLpfv71fdv9msPvJUjlMw7etoEu5Ozurv6edlY9YV9m1DwlcdxyeWitEpRALsRLDG04ssc2e3CBYP+UUpttuLO19LVgRZXas8+kN8MSu1OmMMVxKerxSaQzUWtTEUW1WG96Ah9eXH3ASVOx01GH8PiYXCKOUqiPFEZbrgCodJuAWRkSBZVLsKH0Sr1eAYukMfbA4GbcQeN1Y10KaQV7KTu2fMBz16pVrj+zIhy2SBiax/fjaa3Bp+pUpuJgzXW3OPRzh3+M6BUitszF25Wc+nD9x+uD6igj4aMs2QVQ+TKQ8SaSVCNsfIvKKgWjIoIuYSeN4Ama45BXoQvqHp2ZVTKaW++wiKBp7EVejxe0PaVFkGATmaPQIxJFhJ6O9M3gFn527c3Bpr1/i7LkW+EyBeEkrBV88WhQgqRU2HPjERrBRxqyyYRMLY8WjpJAeYGtgrudokUDHAtVe54js0e8GNM25zRTCqgbmY04PMchRBVElgY16E5QqBCyIwUyatqKoxK/FDSeAXh2iSVlZrSPTaKPY1/acjk40XJosvoWab258xtNoZNtMJTD4I2vbTFMA4yp3lUqQDwN1X1ajOj+Yi0HzhzDABliDpk+bXeBBKCQ1x4nSP42vZRhK4U12B7nzAmx2r8pjh6AP32TxJ9ErWzK7UErDyhpaGAuusjkRFWSvF6flQ+6QPAL8zWRuW4KqzefCUURjFHW9kWWqG/hLx+7irxJYTTJVLbUav/NENnoJ9DAnpmwGxaLZLvjX1WOqkJpaiOiNZC5eKGrmkDTdI/hKPQ3bb3KzFGFCS04jdWkl9bcbTcxR+UVO1whJoFWa7p4VxWvR1eOVirCkeWarN8= X-OriginatorOrg: criticaltechworks.com X-MS-Exchange-CrossTenant-Network-Message-Id: ef37e3fa-18c7-40f2-8345-08db0f41545f X-MS-Exchange-CrossTenant-AuthSource: PAXP193MB1519.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 10:42:26.9693 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 61f30b8e-4f6b-44fe-9bc2-041e3a9f7346 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rFgPGOaq8XrdM8cFSKqEgkV6Zl5prDmLyl4pPhPaIMf0c4dw5sIgoojxi0NTkqXyZPJTanlMMCQT8OEMVNR0n7C94V+ktugGsDxwA+c+IMI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2P193MB2034 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 15 Feb 2023 10:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14432 From: Richard Purdie In multiconfig builds with large numbers of identical tasks, builds were deadlocking after recent runqueue changes upon rebuilds where there was heavy sstate usage (i.e. on second builds after a first completed). The issue was that deferred tasks were being left indefinitely on the deferred list. The deadlock handler was then "breaking" things by failing tasks that had already succeeded, leading to the task being on both covered and not covered lists, giving a further error. The fix is to clean up the deferred task list when each setscene task completes. I'd previously been hoping to avoid iterating that list but it appears unavoidable. [YOCTO #14342] Signed-off-by: Richard Purdie (cherry picked from commit ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea) Signed-off-by: Fabio Berton --- lib/bb/runqueue.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index c7f50fde..f63a2191 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1975,6 +1975,12 @@ class RunQueueExecute: self.setbuildable(revdep) logger.debug(1, "Marking task %s as buildable", revdep) + for t in self.sq_deferred.copy(): + if self.sq_deferred[t] == task: + logger.debug(2, "Deferred task %s now buildable" % t) + del self.sq_deferred[t] + update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) + def task_complete(self, task): self.stats.taskCompleted() bb.event.fire(runQueueTaskCompleted(task, self.stats, self.rq), self.cfgData) From patchwork Wed Feb 15 10:43:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabio Berton X-Patchwork-Id: 19574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BBD0C64ED8 for ; Wed, 15 Feb 2023 10:44:26 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.57]) by mx.groups.io with SMTP id smtpd.web11.12145.1676457750437552896 for ; Wed, 15 Feb 2023 02:42:30 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@criticaltechworks.com header.s=selector2 header.b=yojIQGml; spf=pass (domain: criticaltechworks.com, ip: 40.107.20.57, mailfrom: fabio.berton@criticaltechworks.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m1gNNr0bxN2ohIXGOIUjIiwDIGWSZFQjkLJ5nQvZzRFbKHHV4EHA2JLlLF4QoorE00kShPZPW5b1ejGtvTiiGMlNs0UkCYVpf2yTeTr0LvkpOlzfCzfBhyjtToU79fn2usg5diOj4pnm+ljgkOeVfFW2gmWgmeJAd2b78PFk6NBrqhi+ea/uRsVJcRLGAs+R1zN/S3c44OQEg/sbehWtgJ0w8L7ZwFvK8Zn/YZLPkxyEUK9s1FFKXwBLSJT6+IsFc/Ip8o55+lg10JX2oVqWPy3rmNfT9Bqk9y9cwxhW3hXTyA9xBCejisnu6gR2vsefjeONuOZinXiufBHaZAvnZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=35kSsyS+kM6LZmWpCoabv+RBeUK8Uhl2tvHuaDgvP2g=; b=nh98MqKsT3sXD6Ns87FPIrGz0oEn8xdv5Q0v1YpMzhMNUaXexJJPmLvlzpvV48TMWnEUOfwpfOcInamzKOIybqsVxs93Eb2J3u7c6KYZA9UBwuMKGs9W6ze0OANA6QbILmw4qA6EB40+wOAJCmwESDRJFqTmVQdSPcj/wnYKoE2veKMD56b8xuivXJd58oCbPCBs6VVmZonZMcL0Ld7xk9tEJpg5IB1MdBMK3mY4oxg1Zh8W81rFFP14W8OpBwwqyDEOyb+eEfZHmRGeCkOBHo4PqjOLY8AsX8IqmaGLjsFCeCeQwPBaQ+z9/RHguI5ObJJadOM0tvOmhr79kjYLIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=criticaltechworks.com; dmarc=pass action=none header.from=criticaltechworks.com; dkim=pass header.d=criticaltechworks.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=criticaltechworks.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=35kSsyS+kM6LZmWpCoabv+RBeUK8Uhl2tvHuaDgvP2g=; b=yojIQGmlWW7b6Z5VSdsouJ0GyKjWX5T+0OUV6zxY/sTfhUzj3xbXO4CoCVp1PQalOIuxBsJjqwglEINCUM21w+J4LILhy7So9e6suwrvv6htQPvFiLxeca2xByUnJjc1u4aS8rm713yPJPA1kADcyQt4EgF5Tnfn3HXUy1AG6oTqAjUckqGxDxKPmMOrvhtRAd9jvwHQfycxV7cTqZCDtwLSaNtPysgGCh8/G2TxRNII608YCm0C5yU8Bd5u/05CXFtkb8+hE6NcXsqAWzW4voX8sw/6XLe6HJ0G7WAXmOKj7ZwySv8lNj9o96C1RIGiNgFFhf8FlJesXhhStiZt4w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=criticaltechworks.com; Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) by DB8P193MB0712.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:152::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.24; Wed, 15 Feb 2023 10:42:27 +0000 Received: from PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8]) by PAXP193MB1519.EURP193.PROD.OUTLOOK.COM ([fe80::2d2f:75be:a1a1:2ed8%7]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 10:42:27 +0000 From: "Fabio Berton" To: bitbake-devel@lists.openembedded.org Subject: [1.46][PATCH 6/6] runqueue: Ensure deferred tasks are sorted by multiconfig Date: Wed, 15 Feb 2023 10:43:21 +0000 Message-ID: <20230215104321.1665505-7-fabio.berton@criticaltechworks.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> References: <20230215104321.1665505-1-fabio.berton@criticaltechworks.com> X-ClientProxiedBy: LO2P265CA0305.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a5::29) To PAXP193MB1519.EURP193.PROD.OUTLOOK.COM (2603:10a6:102:13b::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXP193MB1519:EE_|DB8P193MB0712:EE_ X-MS-Office365-Filtering-Correlation-Id: ba5ece0c-9bb4-42ea-6699-08db0f4154c4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5iMoAV/SsGHQ5I0wR3wFiziT2tgRK9Epd3aN04Yysji82jIoIERTLS2+6txA9EQNSKQ5+xQMbk3UD8OmNa/gqB3dWr89wvMcmvRzx5TbaU+fSsd6PuaJhnS2QXn05FE2dQUyq6JrWTQr1UvKeKMdRMwv9eK+kW/FYvHuYc+YqQbStowa+y/lpIkHYL0tLLU5zBRcx0uLZnof4NZ/eP3Nh/Ck7aCwxFvQzt0MyKIOtU3NySRimO/R6hbfw9n7BHBrxtl+HuYkIlO9v8fqkwjJ4MaCPVHkQa7uhLXynpKtb8X4rCZnUS8WU+87jLg/tiMQVJ97l5dzWDDVVOwsrOKGgJG0/jPyTvsOy8Eq0KzFMIc5ilUyNkmPEffZqq6KBjNfMr9FZ6xkHfnMMkdOWL5QErfZAYR1xQw2MR+piJDkiDhk+UAJbmIztuGKVqYtegH3AziNSli7KLdto1hp++c+yQaD2gqnhXAUvNhBVtRx0JrR8YIZG8ay+s7mG6BKHlFceGAXGqdubW6TlRUB6PbOyx06cNEqlq2ZLjCEi5CcgMWJSf+cqETFmNA0I3bloQLR4xLjxjcUkJKKX8owSMieLKrlVf3QawkpOiSPoet1N0tqQST5G+FBNNNknuOpNLPC3aHDgE/jdzF9phk0ou6rccK+m+zsZ438BLMebUILAi7pYpG2hBf2QUwjK7EEXYaszioJVO1POVy5buV4kwlp5w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXP193MB1519.EURP193.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(396003)(346002)(136003)(39850400004)(366004)(376002)(451199018)(186003)(6512007)(26005)(2616005)(6506007)(478600001)(6486002)(316002)(1076003)(6666004)(52116002)(66476007)(66946007)(8676002)(6916009)(66556008)(41300700001)(83380400001)(8936002)(5660300002)(2906002)(44832011)(38350700002)(38100700002)(36756003)(86362001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: g3EI70CZRquo4QllK9GsGD8YS1Rq2DQZtkt2x0nACnHGWvSh1fyvYc9aC4ARYIYqPzLeVRJmGhrmWsKVYUewc7F2ErH+zE0GHLXPWP15nros68p42b6HfV+4nG9coHunmPZSbKzX3wsfQX1Ldcvqh1lcfAmv/TfoJuEXzKczNm6l3EXKb7EWdGqZfjfQeosiQ2jPNxU+YFZ/n5NlSrlpI9mYpVS8lDepTMx/PZALs1EhfBxrGke/gADB7NPjEDLkpRhIL3ruDHCxfA7l+CYZAf+IJVfqT9twhZtQFPgZgQgb67ThElVmciMDOwJTVrlCepnNw5SoQVK5VYdqfkRlxOFeredlEcSb8JdhEHT3RjugEP4GG55YdIlJPXANixBdeqzWdRuoHjTBLn5pG0TkLzMPcRwdZYEWNxFMjzYR+o2K7V6ACamzZ7wokDCVcSQIMXMUNjVqmtxN46dFAOPr4x/cX3YWbTaOcdk00UqWkOTEo17mo0y2OFTGnzuw15b0GWkAr1vsqLxek6R2u5H7jYqOBj5Ik0XeLdQ7eW6p0QoR2/DG6ccYO3kIzSvfOewCiEuqT9N92LnilpGN+z/xelutcWK32u9cmCUEaOa7k6l7MTIk3QwrzsEdMxSSRALx6FvOvLDVgH2fTnoh4MyUs1+vGGdTBKnPXGsTbHD/CLEdDuQH7rjGmP8h2MODdOM4B2pm/idvsJ6seVlls1kmPUxyy+3whIf1d2vfoVf2hVzRAHlrS65I65cVdHa/oeDjFuXa66v9aWx6RrVsTxL4124CVuPEDEw1xRtC6k2y/naUDeImvVDpmL1xJyU5DVcX74fPdrQJPEGG6rM7PQh8lJm/S8eHnFcGuQLhvuZ2hxhUoSLQ1TR4WyJ/1bkqEiuh38rm/kfQ47ncjtfpt5pSsLC7hmcn2vtcdrj1WwTBXrvxrY2h+g7OejDxBbCN3sltg1k+7uDH8LZSV86ukgiSWtmdSu9ZZk6f50hwZzcsIoSxFdBOotxVtVewpHSUdj1LhO8wVagdtBEj0Bepmd5EiiAQL+0eeQKg1E6AF3st57XpepjVfcgJhNqiaT5uzJZ15Fw2pijrV3fytP/ido2i7D9LdEG+JsdVH8YIOwG6/xLO3VHjQf12xES0XbaCpchVh6l0ptM7AHINOHZ3bkt5jU6aQNtS9IiLc/6lfFBSP2yIqOLt0Dp6jH8cMHVbYzze426yDfoyPjH5eSyyTRe1QZrtIAkBgLyrf7rlqMZ7InDX1nTgkKqSgAcYpaZCjNfF39GA5P1zzupzpXHr80VH/Rqyjvcd5pdw4E6NQsIMuKBMsLm0TXnBj1GHcSP1Tlc1bu0bjALwmZZxTNy+z6YHN/LXAgEdi5T3nFWDtYEFE2SgLkAPYaa4pla9tbkX1CpG+JYeJfiBJ2H3MIV2hcbvmhpsUliez4DSlMy3aCmXCJPq0CuUajUNSDLpQXE694MTd0Deb+A2KGiGdIdl9VXrn+tNzLKb34bD6TvOUrGOTHuVa3UCnNoiaHbqtJ9Vei/iJKstYpI+EgAkzvulfkCJiwwRZwju0hi4i69SxnSxl917Ynuzlyvs6awKT9FX4vi48rYX/4rJdYETzwvZ/FKkLQ8pvxa9+IE1nnT2kkQ6IeA= X-OriginatorOrg: criticaltechworks.com X-MS-Exchange-CrossTenant-Network-Message-Id: ba5ece0c-9bb4-42ea-6699-08db0f4154c4 X-MS-Exchange-CrossTenant-AuthSource: PAXP193MB1519.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 10:42:27.3911 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 61f30b8e-4f6b-44fe-9bc2-041e3a9f7346 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bwQ0PwFVwPw3P77f768RdVsSl9Nj3X54FJytsDlrWCFtM0ULN7awKafxGadgWzspdafA1wJNXnK8/skLmdCGLmIubuwrrNVExtaHfEu2kSs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8P193MB0712 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 15 Feb 2023 10:44:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14430 From: Richard Purdie We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. Signed-off-by: Richard Purdie (cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Fabio Berton --- lib/bb/runqueue.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index f63a2191..2a1299db 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2503,11 +2503,14 @@ class RunQueueExecute: if update_tasks: self.sqdone = False - for tid in [t[0] for t in update_tasks]: - h = pending_hash_index(tid, self.rqdata) - if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: - self.sq_deferred[tid] = self.sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) + for mc in sorted(self.sqdata.multiconfigs): + for tid in sorted([t[0] for t in update_tasks]): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: