From patchwork Wed Aug 2 14:24:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 28316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60850C04FE1 for ; Wed, 2 Aug 2023 14:24:39 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web10.16618.1690986276812410336 for ; Wed, 02 Aug 2023 07:24:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20221208.gappssmtp.com header.s=20221208 header.b=MbyjAdyU; spf=pass (domain: baylibre.com, ip: 209.85.128.42, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-3fbc63c2e84so73632935e9.3 for ; Wed, 02 Aug 2023 07:24:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20221208.gappssmtp.com; s=20221208; t=1690986274; x=1691591074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EflKhG89q8EvP6TD/PB7h6vgK2ZX0lwrszQqIilaU+0=; b=MbyjAdyU9LDYo8mwYixr4DVisfy6rgjZREncenZliD3rSbRrQACqiOvBus+QPPbI6s cZufT1quzWq/M6vXgjldH+gF9g58MfxToV/GHWgnoxrEhHPAliR+oYVOvQ8elSFNDuk5 0IbC4nCN3tb0h5rRD9aOj3iWfvJmhWdWu31+37VqYYm+9GL6Y1aRCqTKunDdozx21Vm/ hKfmx6IdmMP/5hbsqKKU7qYhNiA0hLPxzttFmY3KIUG6OZQiqKpPu05D5XiLzmg36krU YKeodxR/9IvYnNWPdktCpnDwVAcO1SPHaTFXLuBpQzwL+zJ1SvtrhQBeTivACutRq4lg h6FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690986274; x=1691591074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EflKhG89q8EvP6TD/PB7h6vgK2ZX0lwrszQqIilaU+0=; b=k48G+sF4hJN+B09ykof5RlSncGR+60GTxSLd5bRxVm89NsjZUokhi3rYWTgOsmCUaF UTPQuKaoHTk/Szr6BI3dQGSA47I3++AvYUqCUUIU/TA7CazWJZOP1I6gNSDVosvUJEbf ysXODqUVWvT2VWgbKrzJ8IWMhBEo4DRDpYWdik3ZMeVvuQPnBzHaKPKyBXMFgVRSHGHV fNg78MO2hNQ3NnvOLegVNIjjvpyBghM3BmTK3iI77QNAfSzYku43VFfmzbnfr7MZlQJK fRmvuGQ/0LEhxCEmfo+TJn2aMUuhUrWlSh8dsnhwo2d9Wpwj41Y1IcHUSex1miLLcJXo 0kQw== X-Gm-Message-State: ABy/qLbdGwNvdCxnuczdj2WzAnfR2nn8uMPwSRCs7NGqNCei70YNQYDo K7Oh/pcTnNg95ofeTnmV5nhsPwysbrEvkKCJwTWh2g== X-Google-Smtp-Source: APBJJlEqsUcx85sNV4jhP5ZL4PMDloYxiH9esmRdl2fFeWcFvVLhUdQUnqq51+57fIYEpSNOJllBXA== X-Received: by 2002:a5d:5309:0:b0:317:60f2:c08b with SMTP id e9-20020a5d5309000000b0031760f2c08bmr4882513wrv.31.1690986274660; Wed, 02 Aug 2023 07:24:34 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id y17-20020adfd091000000b003178dc2371bsm15707079wrh.7.2023.08.02.07.24.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Aug 2023 07:24:34 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 1/5] bitbake.conf: include bblock.conf Date: Wed, 2 Aug 2023 16:24:28 +0200 Message-ID: <20230802142432.2296716-2-jstephan@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230802142432.2296716-1-jstephan@baylibre.com> References: <20230802142432.2296716-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 02 Aug 2023 14:24:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185404 include conf/bblock.conf. This file is generated by the bblock tool. It locks some package tasks by fixing their signatures. See bblock -h for more details Signed-off-by: Julien Stephan --- meta/conf/bitbake.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 475d6523bb8..e4a6bdef0ea 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -832,6 +832,7 @@ include conf/documentation.conf include conf/licenses.conf require conf/sanity.conf require conf/cve-check-map.conf +include conf/bblock.conf ################################################################## # Weak variables (usually to retain backwards compatibility) From patchwork Wed Aug 2 14:24:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 28315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B744C04E69 for ; Wed, 2 Aug 2023 14:24:39 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web11.16294.1690986276863056692 for ; Wed, 02 Aug 2023 07:24:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20221208.gappssmtp.com header.s=20221208 header.b=TfCkx+20; spf=pass (domain: baylibre.com, ip: 209.85.128.50, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742eso73769355e9.3 for ; Wed, 02 Aug 2023 07:24:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20221208.gappssmtp.com; s=20221208; t=1690986275; x=1691591075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=k2BdRDUVCjfe1FYRsLCR0rHlrUx6UUMYpLR8fWZVcfc=; b=TfCkx+20FpCtsNRID863vhZfc1ceLtVj+w2d0oUm9MBUhOG9UfLFcO9XqDm0+xhAxW NiY9p5Etn4XT5ZtVIb0KFSEGaxr+l236ekOrrRVHNWMgma6IJwPzAaVz+rDdU+9uofeI T7mVX2nXz6ztnIo0z8SQ56o/eZkRpylUtIiPO0dJz4UzB4nrQ3TPb5e2gFxDPBPKonav zixafVb8TUHiEzS2ddty4uLLi4NCQ3pdXLxR57T8QzYeYhyjrnCyNUcBZ+gecKw3Sb4g TaX/rnkcoflQeXtolN+EXc4dsQMQv9XJx3YkO5D6hYBCEPTNQmxtNcWiJQwFuAWWWTuU XPsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690986275; x=1691591075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k2BdRDUVCjfe1FYRsLCR0rHlrUx6UUMYpLR8fWZVcfc=; b=I7yakYXTf5obRB2K/KfBYl96E8y7Agb02moPKlQMdRFOyBhFpdo2l23/KLVqBTY7Bh WzwCzoVhicLKpTKyALVWjQNce+50bhUhdHTX10BAAGWAJK8Q4uOL6zlSoiF5twBxtY7W Rtp9gQfiOsuFuVwXShY9frpVzbdXx3U5iH0/UsDLHDmX1nabLWGmYBKk4oUlO8/GXssh nC146S40/ww+3NVdCl5NxAs7nDa/zKVrDw7KAlT9Hd4Yhv92e2O0sSw3IS/pKGu+ptnd c6u/LmYeE1EhdeqxP8MdaNP1Yrbxjhez0vD7ZxChyVPkM2NdaxzqeOTY9awcRN0Xz8xP A3/A== X-Gm-Message-State: ABy/qLbJc6DRmxcSWjO7Hp7hptTvl6f5NeAQ8AhXi/AAmhtuDnsTcAPZ wsMukgG5yBkzvopuw0wOMMnfMJwPJQXnHKiaxwXpFw== X-Google-Smtp-Source: APBJJlESNlpGRIGy805VvGmJ8lUnSgkKJmYKyU54QeE6E/cYgSTjo69vg4OJo+bGbMCF7NMcJVXtpQ== X-Received: by 2002:adf:edcf:0:b0:317:759a:8ca6 with SMTP id v15-20020adfedcf000000b00317759a8ca6mr4680732wro.62.1690986275092; Wed, 02 Aug 2023 07:24:35 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id y17-20020adfd091000000b003178dc2371bsm15707079wrh.7.2023.08.02.07.24.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Aug 2023 07:24:34 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures Date: Wed, 2 Aug 2023 16:24:29 +0200 Message-ID: <20230802142432.2296716-3-jstephan@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230802142432.2296716-1-jstephan@baylibre.com> References: <20230802142432.2296716-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 02 Aug 2023 14:24:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185405 adding a new command in cooker to compute and get task signatures this commit also add the associated command and event needed to get the signatures using tinfoil Signed-off-by: Julien Stephan --- bitbake/lib/bb/command.py | 6 ++++++ bitbake/lib/bb/cooker.py | 16 ++++++++++++++++ bitbake/lib/bb/event.py | 8 ++++++++ 3 files changed, 30 insertions(+) diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py index a355f56c60c..12202779ac0 100644 --- a/bitbake/lib/bb/command.py +++ b/bitbake/lib/bb/command.py @@ -776,3 +776,9 @@ class CommandsAsync: bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache = False + + def getTaskSignatures(self, command, params): + res = command.cooker.getTaskSignatures(params[0], params[1]) + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data) + command.finishAsyncCommand() + getTaskSignatures.needcache = True diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 11c9fa2c40d..687cdde5e6d 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -1542,6 +1542,22 @@ class BBCooker: self.idleCallBackRegister(buildFileIdle, rq) + def getTaskSignatures(self, target, task): + sig = [] + + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt) + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) + rq.rqdata.prepare() + + for key in rq.rqdata.runtaskentries: + pn = bb.parse.siggen.tidtopn[key] + taskname = bb.runqueue.taskname_from_tid(key) + if pn in target: + if (task and taskname in task) or (not task): + rq.rqdata.prepare_task_hash(key) + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)]) + return sig + def buildTargets(self, targets, task): """ Attempt to build the targets specified diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py index 0d0e0a68aac..f8acacd80d1 100644 --- a/bitbake/lib/bb/event.py +++ b/bitbake/lib/bb/event.py @@ -857,6 +857,14 @@ class FindSigInfoResult(Event): Event.__init__(self) self.result = result +class GetTaskSignatureResult(Event): + """ + Event to return results from GetTaskSignatures command + """ + def __init__(self, sig): + Event.__init__(self) + self.sig = sig + class ParseError(Event): """ Event to indicate parse failed From patchwork Wed Aug 2 14:24:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 28314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B79DC41513 for ; Wed, 2 Aug 2023 14:24:39 +0000 (UTC) Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by mx.groups.io with SMTP id smtpd.web11.16295.1690986277693423847 for ; Wed, 02 Aug 2023 07:24:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20221208.gappssmtp.com header.s=20221208 header.b=OAJ210kt; spf=pass (domain: baylibre.com, ip: 209.85.221.51, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-317716a4622so6349609f8f.1 for ; Wed, 02 Aug 2023 07:24:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20221208.gappssmtp.com; s=20221208; t=1690986276; x=1691591076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6uhR97aLN33Lpj5pk6L5LM7sy2JV3qJwmrAk+O9JL40=; b=OAJ210ktCfxUcptl04H73uRctSseXMUluRy3VWhbrkUHp+hYjk9hgEo1vLnUhN+nVk kdLb/TBRoeYrd3vCaaDxiA1ezKkc8nf1ARV0ie5hmoPPyFz1bDg9BKVGoaNT2ZKVWVU2 jEBk4+MqeHWtLQ24bYFAxey/hIi2MPvQgjmEludC+oVljWyCeFvTnkX4p5KVXgngr/I5 +Isie9ajMQPhThGLXBydq1fBJ6kSN8xuh4mV6DubZ7FwWBd8qMS1rDVOqjdSiBnrojgQ ZXZh8dJmDFJtZ4k7aHMD6MJb6p6DrxmVERAddu/JdDaqxstAm3O1jhBuLMDgcaKi/Ga3 HN6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690986276; x=1691591076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6uhR97aLN33Lpj5pk6L5LM7sy2JV3qJwmrAk+O9JL40=; b=im7FmsBLgO5q9lJvF5iw1crVD71zg2iXArxwIzY4xvFZQikgufVOoyAcfIFeR/vXDU 1MW12JMz6qJXoNUFbHhnfOQWlwQcMQ/jPg6aTG66/2xetOjlA1cVxiw9RUAh50QuO+gx DAIDue13qjit0+vLbsz8hX6aBW4PWSgi/KJAzcGUGQsd8FhJm89o6VQC6FNAD+GysnDm in8dry7FW6qQH2/uX5SXiNdNVb+VrEjJoyQFvLvGUUxM/ezBQGmN7OokvZx2WGKAz6iW nCHNusX49bgTKrYt3u6S+kQWHYcFEVittrVibCVpD9az7hDqOFnExl2n4RqrO+Djh/Fw yZHA== X-Gm-Message-State: ABy/qLYA6WTePOU4+NvHL8MAke1Q+SIxgQpnuFTwYUgfsLmgc97S9Wuf pcTAsUIQMpmcXtwdhEE0Gsv54NEfWdYXNjPiKlxfHQ== X-Google-Smtp-Source: APBJJlGYP5YGWEVm4CSHyzuR/8Zg+5XoZjyEXtMQZkGjMzOMivcall+v8x5aR8ZbiVEK6PyRMl1aIg== X-Received: by 2002:a05:6000:1371:b0:317:5e22:3ed4 with SMTP id q17-20020a056000137100b003175e223ed4mr4784546wrz.49.1690986275767; Wed, 02 Aug 2023 07:24:35 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id y17-20020adfd091000000b003178dc2371bsm15707079wrh.7.2023.08.02.07.24.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Aug 2023 07:24:35 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 3/5] sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK Date: Wed, 2 Aug 2023 16:24:30 +0200 Message-ID: <20230802142432.2296716-4-jstephan@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230802142432.2296716-1-jstephan@baylibre.com> References: <20230802142432.2296716-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 02 Aug 2023 14:24:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185406 as of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and "error", displaying respectively a warning or a fatal error message only when a task is locked and the task signature is different from the locked one. The "info" level is introduced to add a "note" message to remind the user that a recipe is locked even if the signature is equivalent to the locked one. The "warn" and "error" level display the warn/error message for each task having a mismatch of the signature. Doing this with the "info" level would result in very verbose output if there are several tasks locked, so the info level will only print once the list of recipes that have locked signature. Signed-off-by: Julien Stephan --- meta/lib/oe/sstatesig.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index f943df181e6..90002f67550 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -104,6 +104,7 @@ class SignatureGeneratorOEBasicHashMixIn(object): self.lockedhashfn = {} self.machine = data.getVar("MACHINE") self.mismatch_msgs = [] + self.lockedsigs_msgs = "" self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or "").split() self.unlockedrecipes = { k: "" for k in self.unlockedrecipes } @@ -264,6 +265,12 @@ class SignatureGeneratorOEBasicHashMixIn(object): warn_msgs = [] error_msgs = [] sstate_missing_msgs = [] + info_msgs = None + + if self.lockedsigs: + self.lockedsigs_msgs = "The following recipes have locked tasks:" + for pn in self.lockedsigs: + self.lockedsigs_msgs += " %s" % (pn) for tid in sq_data['hash']: if tid not in found: @@ -276,7 +283,9 @@ class SignatureGeneratorOEBasicHashMixIn(object): % (pn, taskname, sq_data['hash'][tid])) checklevel = d.getVar("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK") - if checklevel == 'warn': + if checklevel == 'info': + info_msgs = self.lockedsigs_msgs + if checklevel == 'warn' or checklevel == 'info': warn_msgs += self.mismatch_msgs elif checklevel == 'error': error_msgs += self.mismatch_msgs @@ -287,6 +296,8 @@ class SignatureGeneratorOEBasicHashMixIn(object): elif checklevel == 'error': error_msgs += sstate_missing_msgs + if info_msgs: + bb.note(info_msgs) if warn_msgs: bb.warn("\n".join(warn_msgs)) if error_msgs: From patchwork Wed Aug 2 14:24:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 28313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41D0DC04FDF for ; Wed, 2 Aug 2023 14:24:39 +0000 (UTC) Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by mx.groups.io with SMTP id smtpd.web10.16619.1690986278026522524 for ; Wed, 02 Aug 2023 07:24:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20221208.gappssmtp.com header.s=20221208 header.b=0FjNaAdH; spf=pass (domain: baylibre.com, ip: 209.85.128.52, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-3fe1d462762so33260705e9.0 for ; Wed, 02 Aug 2023 07:24:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20221208.gappssmtp.com; s=20221208; t=1690986276; x=1691591076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=009/UH2rLkaom+7AVxLOfVqIecs9/ZSxSKVujijLUTU=; b=0FjNaAdH8Lu+UnTDyRLrx6Ez/L7HOAObFTKmWysBX4O2AcrCNBgQ6B8+Xj0NP7a1zq Pkl0NLsg9Bcuzi3vpkR8AEIqP3UJQxo6TCx7gdhARRA/8gi1mOK8PiNoSmw/oWWAEvxi m8LM2DTayup16/zJ55otplhfFPfbKcKdGoPclRgFP5rMdcgDauJxqVMBLtawe2Cgls/m kNQyaPnBbe6N9397P+pHS38UHfAnsGGPVvFqJPUP7MoKEZSOqjFwbwC5LnikTIlPoQ53 i+/sHQkyNNfbBmLM/jty1s9Sr8l/UMHV4EoSHVfsz4YpHxF3e4OwcRQtub34+ry8S/WV A0mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690986276; x=1691591076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=009/UH2rLkaom+7AVxLOfVqIecs9/ZSxSKVujijLUTU=; b=Ar9kMFQsrcxSi7sz9wq6o/K/rEKYcUci/M6cm7WPbYhLb+zNu5h/FEse8b1yUfgs/n 3oduy7Y8z9tlJFNA2pj7DwlDoF8/7E+qh35Z88rEUkBa+tjLtJq0YISP39fvy3dvvm+9 K75AjTufj711tbEd+n1YI9NydSSRST9KQbtOgl7x1oKyBG5CKkA7tHxzpaCBuTNQwvv3 hAtNSp0LpSW3SVI65/ViBktjXouDRG+7usvDrnpGHTW+jWIDxC0X57e8jqN3HkAp707P z2HrQcH1WENliXiCT3Xt0kLoHFM2bJgQlE6uodrr7VfM2DJ9HXvxeg5lyaNOGXxBRlT8 o3zw== X-Gm-Message-State: ABy/qLbMxUwOxwr7cFf3n0rd7m4+LQSqMSVvmX96wwZRaPjlHWb8n8f3 nVU9XBDVwfp5moIXUDJOXl10oB5btNf38ioAdYW4OQ== X-Google-Smtp-Source: APBJJlHPYMDFJWDmEBjx55lqc7ylIjxV6P/dHXE8Ggzh+bVHHQ6cLkl2+jRzRYAMfIynx1WFW6TL0Q== X-Received: by 2002:a05:6000:11c5:b0:317:594a:dbde with SMTP id i5-20020a05600011c500b00317594adbdemr4669031wrx.20.1690986276198; Wed, 02 Aug 2023 07:24:36 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id y17-20020adfd091000000b003178dc2371bsm15707079wrh.7.2023.08.02.07.24.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Aug 2023 07:24:35 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 4/5] scripts/bblock: add a script to lock/unlock recipes Date: Wed, 2 Aug 2023 16:24:31 +0200 Message-ID: <20230802142432.2296716-5-jstephan@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230802142432.2296716-1-jstephan@baylibre.com> References: <20230802142432.2296716-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 02 Aug 2023 14:24:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185407 bblock script allows to lock/unlock recipes to latest task signatures. The idea is to prevent some recipes to be rebuilt during development. For example when working on rust recipe, one may not want rust-native to be rebuilt. This tool can be used, with proper environment set up, using the following command: bblock if a 's task signature change, this task will not be built again and sstate cache will be used. [YOCTO #13425] Signed-off-by: Julien Stephan --- scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100755 scripts/bblock diff --git a/scripts/bblock b/scripts/bblock new file mode 100755 index 00000000000..e5cf78722c7 --- /dev/null +++ b/scripts/bblock @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +# bblock +# lock/unlock task to latest signature +# +# Copyright (c) 2023 BayLibre, SAS +# Author: Julien Stepahn +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import os +import sys +import logging + +scripts_path = os.path.dirname(os.path.realpath(__file__)) +lib_path = scripts_path + "/lib" +sys.path = sys.path + [lib_path] + +import scriptpath + +scriptpath.add_bitbake_lib_path() + +import bb.tinfoil +import bb.msg + +import argparse_oe + +myname = os.path.basename(sys.argv[0]) +logger = bb.msg.logger_create(myname) + + +def getTaskSignatures(tinfoil, pn, tasks): + tinfoil.set_event_mask( + [ + "bb.event.GetTaskSignatureResult", + "logging.LogRecord", + "bb.command.CommandCompleted", + "bb.command.CommandFailed", + ] + ) + ret = tinfoil.run_command("getTaskSignatures", pn, tasks) + if ret: + while True: + event = tinfoil.wait_event(1) + if event: + if isinstance(event, bb.command.CommandCompleted): + break + elif isinstance(event, bb.command.CommandFailed): + logger.error(str(event)) + sys.exit(2) + elif isinstance(event, bb.event.GetTaskSignatureResult): + sig = event.sig + elif isinstance(event, logging.LogRecord): + logger.handle(event) + else: + logger.error("No result returned from getTaskSignatures command") + sys.exit(2) + return sig + + +def parseRecipe(tinfoil, recipe): + try: + tinfoil.parse_recipes() + d = tinfoil.parse_recipe(recipe) + except Exception: + logger.error("Failed to get recipe info for: %s" % recipe) + sys.exit(1) + return d + + +def bblockDump(lockfile): + try: + with open(lockfile, "r") as lockfile: + for line in lockfile: + print(line.strip()) + except IOError: + return 1 + return 0 + + +def bblockReset(lockfile, pns, package_archs, tasks): + if not pns: + logger.info("Unlocking all recipes") + try: + os.remove(lockfile) + except FileNotFoundError: + pass + else: + logger.info("Unlocking {pns}".format(pns=pns)) + tmp_lockfile = lockfile + ".tmp" + with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile: + for line in infile: + if not ( + any(element in line for element in pns) + and any(element in line for element in package_archs.split()) + ): + outfile.write(line) + else: + if tasks and not any(element in line for element in tasks): + outfile.write(line) + os.remove(lockfile) + os.rename(tmp_lockfile, lockfile) + + +def main(): + parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe") + parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock") + parser.add_argument( + "-t", + "--tasks", + help="Comma separated list of tasks", + type=lambda s: [task for task in s.split(",")], + ) + parser.add_argument( + "-r", + "--reset", + action="store_true", + help="Unlock pn recipes, or all recipes if pn is empty", + ) + parser.add_argument( + "-d", + "--dump", + action="store_true", + help="Dump generated bblock.conf file", + ) + + global_args, unparsed_args = parser.parse_known_args() + + with bb.tinfoil.Tinfoil() as tinfoil: + tinfoil.prepare(config_only=True) + + package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS") + builddir = tinfoil.config_data.getVar("TOPDIR") + lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir) + + if global_args.dump: + bblockDump(lockfile) + return 0 + + if global_args.reset: + bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks) + return 0 + + with open(lockfile, "a") as lockfile: + s = "" + if lockfile.tell() == 0: + s = "# Generated by bblock\n" + s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"\n' + s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n' + s += "\n" + + for pn in global_args.pn: + d = parseRecipe(tinfoil, pn) + package_arch = d.getVar("PACKAGE_ARCH") + siggen_locked_sigs_package_arch = d.getVar( + "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch) + ) + sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks) + for sig in sigs: + new_entry = "{pn}:{taskname}:{sig}".format( + pn=sig[0], taskname=sig[1], sig=sig[2] + ) + if ( + siggen_locked_sigs_package_arch + and not new_entry in siggen_locked_sigs_package_arch + ) or not siggen_locked_sigs_package_arch: + s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format( + package_arch=package_arch, new_entry=new_entry + ) + lockfile.write(s) + return 0 + + +if __name__ == "__main__": + try: + ret = main() + except Exception: + ret = 1 + import traceback + + traceback.print_exc() + sys.exit(ret) From patchwork Wed Aug 2 14:24:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 28317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C760C001DF for ; Wed, 2 Aug 2023 14:24:39 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web10.16620.1690986278288863561 for ; Wed, 02 Aug 2023 07:24:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20221208.gappssmtp.com header.s=20221208 header.b=n54M7ak6; spf=pass (domain: baylibre.com, ip: 209.85.221.48, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-31751d7d96eso5884269f8f.1 for ; Wed, 02 Aug 2023 07:24:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20221208.gappssmtp.com; s=20221208; t=1690986276; x=1691591076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+I6XDFBaTn8V7kyBaNTIJ4iu87wSZOjTRTUfXfYTh5c=; b=n54M7ak6LCsofMFAfVbxMp56omjybpcEEJvhTLAtFkS/MZnNgzGS4n4ZSqB73Mb7be nf7Fy0Vu3D0n2wZ0NucGeopysxk512eGiiuilKP3eADdA65TOessdXrZjQDEyuuIRZkB 9KEDxgm8RMzmJjRWSTH/vlIsV4NrJqdKeJsnl8U/tmIPpoThHzI3e7cA/RZvcwwnil53 Lz0k0O8Psv2TNWXmRX0LIuhmKwd0Z7uB66ERIo8aLPvEWlKAC9xc0XWa1MtzV4HAoAGp cxsexP9uhCRI/HxMpn++T0h6WdxEuvvA1/BtOcEtnwXIZEbQThd3ALlj+JyxGvCzcIvg 2AtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690986276; x=1691591076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+I6XDFBaTn8V7kyBaNTIJ4iu87wSZOjTRTUfXfYTh5c=; b=SEWxT6iGdGzJDMKIhOiBAudcn+dAxbNLqicxnFQTkXGtieWRvOWiAfMukJspqu9imN a4l8cirrIBwLBsOJQp/n8v9rlV++gkj1eP9irh9di8XogGwY2SYfAyerOsJ8zkV27cde pCgi3eAL+/BsL8hK3hHXPGyj0iBh7dQDbuJDUivoalPpuACherbveaCIYHcqezYWUQsg BXMHLQ6xXuFfdma4SiOj6xUUQlBXDbSDSWxyDv98R2w6DVSCf1UYMnhXGxmX2qW9Wqxq eAURWsL7BuXpDlZhmJekalSUVTJSR5xp0Pst7evFu1+3UQbMb9ZQcImkZ1ga7aLNsvQ8 yu/A== X-Gm-Message-State: ABy/qLZbxe2QduBahwVL0X1faFPEdDbiT9GHlTLW6wn/SRWC0+Wyoonr IjAs9bs83blghd5e1SoPxZxB+b9y++fhrvcjnzamVA== X-Google-Smtp-Source: APBJJlHzmOcRz/jF2wOtioAQSOwjF6ijyVpPSGNBu89cFHWxpN2Ia/zc2TkqnRdxCI+Ncwoj2IeBHA== X-Received: by 2002:adf:f0cf:0:b0:314:2e95:1ec9 with SMTP id x15-20020adff0cf000000b003142e951ec9mr5148841wro.10.1690986276621; Wed, 02 Aug 2023 07:24:36 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id y17-20020adfd091000000b003178dc2371bsm15707079wrh.7.2023.08.02.07.24.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Aug 2023 07:24:36 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 5/5] oeqa/selftest/bblock: add self test for bblock tool Date: Wed, 2 Aug 2023 16:24:32 +0200 Message-ID: <20230802142432.2296716-6-jstephan@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230802142432.2296716-1-jstephan@baylibre.com> References: <20230802142432.2296716-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 02 Aug 2023 14:24:39 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/185408 it implements various combination of locking single/multiple recipe(s)/task(s) it also tests that locked sig are architecture dependant Signed-off-by: Julien Stephan --- meta/lib/oeqa/selftest/cases/bblock.py | 146 +++++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py diff --git a/meta/lib/oeqa/selftest/cases/bblock.py b/meta/lib/oeqa/selftest/cases/bblock.py new file mode 100644 index 00000000000..803ea3d64c8 --- /dev/null +++ b/meta/lib/oeqa/selftest/cases/bblock.py @@ -0,0 +1,146 @@ +# +# Copyright (c) 2023 BayLibre, SAS +# Author: Julien Stepahn +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import os +import re + +import oeqa.utils.ftools as ftools +from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake + +from oeqa.selftest.case import OESelftestTestCase + +class BBLock(OESelftestTestCase): + + @classmethod + def setUpClass(cls): + super(BBLock, cls).setUpClass() + cls.lockfile = cls.builddir+"/conf/bblock.conf" + + def unlock_recipes(self, recipes=None, tasks=None): + cmd = 'bblock -r ' + if recipes: + cmd += ' '.join(recipes) + if tasks: + cmd += ' -t ' + ','.join(tasks) + result = runCmd(cmd) + + if recipes: + # ensure all signatures are removed from lockfile + contents = ftools.read_file(self.lockfile) + for recipe in recipes: + for task in tasks: + find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents) + self.assertFalse(find_in_contents, msg = "%s:%s should not be present into bblock.conf anymore" % (recipe, task)) + self.assertExists(self.lockfile) + else: + self.assertNotExists(self.lockfile) + + def lock_recipes(self, recipes, tasks=None, machine=None): + cmd = 'bblock ' + ' '.join(recipes) + if tasks: + cmd += ' -t ' + ','.join(tasks) + + if machine: + cmd = "MACHINE=" + machine + " " + cmd + with open("/tmp/tmp.txt", "w") as file: + file.write(cmd) + result = runCmd(cmd) + + self.assertExists(self.lockfile) + + # ensure all signatures are added to lockfile + contents = ftools.read_file(self.lockfile) + for recipe in recipes: + for task in tasks: + find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents) + self.assertTrue(find_in_contents, msg = "%s:%s was not added into bblock.conf. bblock output: %s" % (recipe, task, result.output)) + + def modify_tasks(self, recipes, tasks): + task_append = "" + for recipe in recipes: + bb_vars = get_bb_vars(['PV'], recipe) + recipe_pv = bb_vars['PV'] + recipe_append_file = recipe + '_' + recipe_pv + '.bbappend' + + os.mkdir(os.path.join(self.testlayer_path, 'recipes-test', recipe)) + recipe_append_path = os.path.join(self.testlayer_path, 'recipes-test', recipe, recipe_append_file) + + for task in tasks: + task_append += "%s:append() {\necho \"modify task hash\"\n}\n" % task + ftools.write_file(recipe_append_path, task_append) + self.add_command_to_tearDown('rm -rf %s' % os.path.join(self.testlayer_path, 'recipes-test', recipe)) + + def test_lock_single_recipe_single_task(self): + recipes = ["quilt"] + tasks = ["do_compile"] + self._run_test(recipes, tasks) + + def test_lock_single_recipe_multiple_tasks(self): + recipes = ["quilt"] + tasks = ["do_compile" ,"do_install"] + self._run_test(recipes, tasks) + + def test_lock_architecture_specific(self): + recipes = ["quilt"] + tasks = ["do_compile"] + + # lock quilt's do_compile task for qemuarm machine + self.lock_recipes(recipes, tasks, "qemuarm") + + # modify quilt's do_compile task + self.modify_tasks(recipes, tasks) + + # build quilt using by default qemux86-64. + # No Note/Warning should be emitted since sig is locked for qemuarm + # (quilt package is architecture dependant) + info_message = "NOTE: The following recipes have locked tasks: " + recipes[0] + warn_message = "The %s:%s sig is computed to be" % (recipes[0], tasks[0]) + result = bitbake(recipes[0] + " -n") + self.assertNotIn(info_message, result.output) + self.assertNotIn(warn_message, result.output) + + def test_lock_multiple_recipe_single_task(self): + recipes = ["quilt", "bc"] + tasks= ["do_compile"] + self._run_test(recipes, tasks) + + def _run_test(self, recipes, tasks=None): + # unlock all recipes and ensure no bblock.conf file exist + self.unlock_recipes() + + # lock tasks for recipes + result = self.lock_recipes(recipes, tasks) + + # build recipes. At this stage we should have a Note about recipes + # having locked task's sig, but no warning since sig still match + info_message = "NOTE: The following recipes have locked tasks: " + " ".join(recipes) + for recipe in recipes: + result = bitbake(recipe + " -n") + self.assertIn(info_message, result.output) + for task in tasks: + warn_message = "The %s:%s sig is computed to be" % (recipe, task) + self.assertNotIn(warn_message, result.output) + + ## modify all tasks that are locked to trigger a sig change then build the recipes + ## at this stage we should have a Note as before, but also a warning for all + ## locked stage indicating the sig mismatch + self.modify_tasks(recipes, tasks) + for recipe in recipes: + result = bitbake(recipe + " -n") + self.assertIn(info_message, result.output) + for task in tasks: + warn_message = "The %s:%s sig is computed to be" % (recipe, task) + self.assertIn(warn_message, result.output) + + # unlock all tasks and rebuild, no more Note/Warning should remain + self.unlock_recipes(recipes, tasks) + for recipe in recipes: + result = bitbake(recipe + " -n") + self.assertNotIn(info_message, result.output) + for task in tasks: + warn_message = "The %s:%s sig is computed to be" % (recipe, task) + self.assertNotIn(warn_message, result.output)