From patchwork Tue Nov 14 23:34:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 34489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49C03C4332F for ; Tue, 14 Nov 2023 23:34:55 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web11.872.1700004892827678584 for ; Tue, 14 Nov 2023 15:34:53 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=NIGp23j5; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-32dc918d454so3707961f8f.2 for ; Tue, 14 Nov 2023 15:34:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1700004891; x=1700609691; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=TtRcrBCPvcFLp6ZPuuluGCE2FwWn71aatab+VAcIZas=; b=NIGp23j5KwNQlX9WUt77RJHrL+L6YALLEUFLOHjuSaS1Ojc4PCY/jz6olHaPeBBgQf VSshmONIo4QlHY8KJwEmdfqfb+EIjNT25TyW1Z7MiDKLns+RDqQdK2xPY1906pCKhcSi V3v+ZjKc/TcZkmo8/HEsUIoddz3Oy8OEQsBUc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700004891; x=1700609691; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TtRcrBCPvcFLp6ZPuuluGCE2FwWn71aatab+VAcIZas=; b=w6wHCHUq1Ij/+fP1BfJgC1k9U46p9AaYIknlzKL5i+kE1xLUPtEcUeILOPfmJeEZUE huevDAd8DyjW3tpKqeOXIMEjNmH3rlHVfytCfMxoYUmkQ5N2JUzdhINt4S4L3gzUfkoD aSacOc4bizpTsq14i4UaJQjMAtD7y64zN5RlvPAfaVgZjZBCyqY51c7/SPnx5ofMA7ul uRHBlxoe4LiUM52YmaDIVdW9DvbEyQmbE3ahc116rgehUIv8xcLtJh+sX0EUUGRff+Ie Zg6hHyU7EaUNxVjQRI6hHPiPyTNCJewWD1N6BZ5T1SOJvn+KDm8MJtUlpcG2cnBvMf8z MquQ== X-Gm-Message-State: AOJu0YzYqqK051nzxsTzG/1TWHXe9pBwah10RiUNzzU2bqkrGThf4D6F 6Bvco6DDJk2HKvdwqWaQTpUOcWbOBgnOrgj796I= X-Google-Smtp-Source: AGHT+IEwUm1b0KXnLV+DREruiLV7gd7c7A6vPTaFvjNbUI9L8ZjmuE9v28/Dw9YBbOLWubTA10N4XQ== X-Received: by 2002:a5d:5888:0:b0:32f:7d50:62ef with SMTP id n8-20020a5d5888000000b0032f7d5062efmr8749505wrf.13.1700004890489; Tue, 14 Nov 2023 15:34:50 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:bbf9:c35a:2291:eb7c]) by smtp.gmail.com with ESMTPSA id f15-20020a5d50cf000000b0032f78feb826sm8930203wrt.104.2023.11.14.15.34.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Nov 2023 15:34:50 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/3] runqueue: Move 'cantskip' into sqdata Date: Tue, 14 Nov 2023 23:34:47 +0000 Message-Id: <20231114233449.2567899-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 Nov 2023 23:34:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15526 Conceptually, this belongs in sqdata so this is the next step in refactoring this code, continuing what was started a while ago. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 475c3c5052..2a8b89b264 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1862,11 +1862,6 @@ class RunQueueExecute: self.tasks_notcovered = set() self.scenequeue_notneeded = set() - # We can't skip specified target tasks which aren't setscene tasks - self.cantskip = set(self.rqdata.target_tids) - self.cantskip.difference_update(self.rqdata.runq_setscene_tids) - self.cantskip.intersection_update(self.rqdata.runtaskentries) - schedulers = self.get_schedulers() for scheduler in schedulers: if self.scheduler == scheduler.name: @@ -2399,7 +2394,7 @@ class RunQueueExecute: return notcovered = set(self.scenequeue_notcovered) - notcovered |= self.cantskip + notcovered |= self.sqdata.cantskip for tid in self.scenequeue_notcovered: notcovered |= self.sqdata.sq_covered_tasks[tid] notcovered |= self.sqdata.unskippable.difference(self.rqdata.runq_setscene_tids) @@ -2784,6 +2779,11 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): sq_revdeps_squash = {} sq_collated_deps = {} + # We can't skip specified target tasks which aren't setscene tasks + sqdata.cantskip = set(rqdata.target_tids) + sqdata.cantskip.difference_update(rqdata.runq_setscene_tids) + sqdata.cantskip.intersection_update(rqdata.runtaskentries) + # We need to construct a dependency graph for the setscene functions. Intermediate # dependencies between the setscene tasks only complicate the code. This code # therefore aims to collapse the huge runqueue dependency tree into a smaller one @@ -2852,7 +2852,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): for tid in rqdata.runtaskentries: if not rqdata.runtaskentries[tid].revdeps: sqdata.unskippable.add(tid) - sqdata.unskippable |= sqrq.cantskip + sqdata.unskippable |= sqdata.cantskip while new: new = False orig = sqdata.unskippable.copy() From patchwork Tue Nov 14 23:34:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 34488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F49FC4167D for ; Tue, 14 Nov 2023 23:34:55 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web10.809.1700004892973973421 for ; Tue, 14 Nov 2023 15:34:53 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Mo0uvND2; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-32d849cc152so4251484f8f.1 for ; Tue, 14 Nov 2023 15:34:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1700004891; x=1700609691; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=OrDDHmbx/TQl1ZR9zskYMPhmAVnLwNXB1qyb5nOvz+A=; b=Mo0uvND25JHcRpGgLFa4h/1KlUDl5Z4xlaRfy9QMQYgvwGmLelx/q5W0UVjkLuPxWF 5G8SCVQY4BpDQRXs5xCt/qm6e5hzwJc13Gx61gUQE0wJbNh4hT2izFRmV/IP/2xGSByP GsKDxF9VR1RGcAp94I2yai48xZKifihasVeJ0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700004891; x=1700609691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OrDDHmbx/TQl1ZR9zskYMPhmAVnLwNXB1qyb5nOvz+A=; b=APhpxzkDNjHg+pGa/SKefESNXexBmYhyzv0cFjgww4ksRr2zvAuTFp1xBJPQWaHouQ Mq2uesIFH7DNd/pzEhKQ1eHThskYfUmNR6hh0FJYzS4+wqBh/joVYJZvTmIjUKMpvvvp GFVxtj6/furNVpgHVGLD2S3PcjMzQbbupVuju6nCqLgC5fTi55xHssYY0XqeMURq4+Xi K/LIcM1NjnAkERpcIfWUcNOIgpXPWzCCUMN5XDUGW4tV8M7Qv8XoMfiQYJ2v7xARg0wF JoXi54C6kyzrIyRf2TIEU3MUL4KinYHSL3vGyTlATgEHoIEKm/Fpf5SAHirBkrMo6H0I rX9w== X-Gm-Message-State: AOJu0YwLLhcYaxmj7dhvc/7uNAHFfXz/scs3vn9hslcYzTo4ABaYsP6/ vVUvQmGgjH2jWkU8JAmT3UJf5wTngbps+CBApbM= X-Google-Smtp-Source: AGHT+IEhaS2/Y0oku2j2mlJqyt07x2+k6rllg+XD5iiPtI0TNo5gsXi1eYWgNcPZL8uuP5Po2oLk3w== X-Received: by 2002:a05:6000:1108:b0:32d:9a36:b518 with SMTP id z8-20020a056000110800b0032d9a36b518mr8447601wrw.69.1700004891130; Tue, 14 Nov 2023 15:34:51 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:bbf9:c35a:2291:eb7c]) by smtp.gmail.com with ESMTPSA id f15-20020a5d50cf000000b0032f78feb826sm8930203wrt.104.2023.11.14.15.34.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Nov 2023 15:34:50 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/3] runqueue: Refactor StaleSetSceneTasks event out of build_scenequeue_data Date: Tue, 14 Nov 2023 23:34:48 +0000 Message-Id: <20231114233449.2567899-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231114233449.2567899-1-richard.purdie@linuxfoundation.org> References: <20231114233449.2567899-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 Nov 2023 23:34:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15527 The event at the end of the function doesn't really belong there, move it out so the function becomes cleaner. It also avoids having to pass a number of parameters around, another sign this is the correct thing to do. This continues refactoring started a while ago to improve and modularise the code. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 2a8b89b264..02d7ff9768 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1874,7 +1874,23 @@ class RunQueueExecute: #if self.rqdata.runq_setscene_tids: self.sqdata = SQData() - build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self) + build_scenequeue_data(self.sqdata, self.rqdata, self) + + update_scenequeue_data(self.sqdata.sq_revdeps, self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=True) + + # Compute a list of 'stale' sstate tasks where the current hash does not match the one + # in any stamp files. Pass the list out to metadata as an event. + found = {} + for tid in self.rqdata.runq_setscene_tids: + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) + stamps = bb.build.find_stale_stamps(taskname, taskfn) + if stamps: + if mc not in found: + found[mc] = {} + found[mc][tid] = stamps + for mc in found: + event = bb.event.StaleSetSceneTasks(found[mc]) + bb.event.fire(event, self.cooker.databuilder.mcdata[mc]) def runqueue_process_waitpid(self, task, status, fakerootlog=None): @@ -2773,7 +2789,7 @@ class SQData(object): # A list of normal tasks a setscene task covers self.sq_covered_tasks = {} -def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): +def build_scenequeue_data(sqdata, rqdata, sqrq): sq_revdeps = {} sq_revdeps_squash = {} @@ -2962,22 +2978,6 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): sqrq.sq_deferred[tid] = sqdata.hashes[h] bb.debug(1, "Deferring %s after %s" % (tid, sqdata.hashes[h])) - update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True) - - # Compute a list of 'stale' sstate tasks where the current hash does not match the one - # in any stamp files. Pass the list out to metadata as an event. - found = {} - for tid in rqdata.runq_setscene_tids: - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) - stamps = bb.build.find_stale_stamps(taskname, taskfn) - if stamps: - if mc not in found: - found[mc] = {} - found[mc][tid] = stamps - for mc in found: - event = bb.event.StaleSetSceneTasks(found[mc]) - bb.event.fire(event, cooker.databuilder.mcdata[mc]) - def check_setscene_stamps(tid, rqdata, rq, stampcache, noexecstamp=False): (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) From patchwork Tue Nov 14 23:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 34490 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5802CC2BB86 for ; Tue, 14 Nov 2023 23:34:55 +0000 (UTC) Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by mx.groups.io with SMTP id smtpd.web11.874.1700004893700748916 for ; Tue, 14 Nov 2023 15:34:53 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=eUAjicvA; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-32df66c691dso3649760f8f.3 for ; Tue, 14 Nov 2023 15:34:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1700004892; x=1700609692; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=m1a4BKhXs8TWX9F9zm3EdncrTzCTKCR1yu0GguiRqKs=; b=eUAjicvA+rzX1Mm4y1p4T6vHf6eTKs1omVYkza6iNddW7YOLADUL1yjlu/nlDCJ1lE g28vVNDLYzPZrsD9Gjgh3yZZQXb59XNh4009E/FlfvVwdwmwyBrdneac7QQchvYnpQ4q K/ZPYgXXAO4J+q8lqp9pqtPhe1K9VSyCqY+WY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700004892; x=1700609692; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m1a4BKhXs8TWX9F9zm3EdncrTzCTKCR1yu0GguiRqKs=; b=o31EtGlEdnu6VEQ7/59KsUAMkWX2wJE9NMSBQGVsWApWnpaSZSynTj8K6wCt/X2zsT io7ShiBukz8sipYDxpwcr5eFqHUMn6Z8Ksfgds4RKhsv21sNNDUS4lW1l2HU6quXvdo2 z8JUGeeUvQBte/eUm7HGgJOegYQTGopAY76ArC8uWVzuoHMdcyakY0JhPhK5LuQ7Qm0V dpP12/DDtkU0m4Lc6uHImK9IgTK6XDeULjzVKSE8AgHfo+bBtGA0IexDdkH3TZslSN4h Ifh5d6dmG7lh+W1fCgz4ScU9PAdERGyO+HMy6bsrMnCmRRUSK0hz2HeU7bXZuywxKyqA jgfQ== X-Gm-Message-State: AOJu0Yx9eHAx8b2mrN3Ec9Rkdzmvme9HBZgvRuAOtSIJ2uKevfz8kpD1 GgHe7GcC4n72rrc26XkQHO5PkxJ45lB72gi0Y2o= X-Google-Smtp-Source: AGHT+IEc5+i965PDhDcSbMfVk2tz2CFmTjMpi6AEq4/MPofQ12Iu8lUcnb2s/4rb+qCV3ZnVc49cew== X-Received: by 2002:a5d:670a:0:b0:32f:7b99:2ed with SMTP id o10-20020a5d670a000000b0032f7b9902edmr8586796wru.22.1700004892138; Tue, 14 Nov 2023 15:34:52 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:bbf9:c35a:2291:eb7c]) by smtp.gmail.com with ESMTPSA id f15-20020a5d50cf000000b0032f78feb826sm8930203wrt.104.2023.11.14.15.34.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Nov 2023 15:34:51 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/3] tox.ini: Add py 3.11 and 3.12 Date: Tue, 14 Nov 2023 23:34:49 +0000 Message-Id: <20231114233449.2567899-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231114233449.2567899-1-richard.purdie@linuxfoundation.org> References: <20231114233449.2567899-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 Nov 2023 23:34:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15528 Some of the autobuilder workers have python 3.11 or in the future 3.12. Add these to allow toaster tests to work on those platforms. Signed-off-by: Richard Purdie --- lib/toaster/tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/toaster/tox.ini b/lib/toaster/tox.ini index 8c0fe17181..d058558b05 100644 --- a/lib/toaster/tox.ini +++ b/lib/toaster/tox.ini @@ -1,5 +1,5 @@ [tox] -envlist = py38, py39, py310 +envlist = py38, py39, py310, py311, py312 skipsdist = True toxworkdir = {env:TOX_WORKDIR:.tox} passenv = *