From patchwork Wed Jun 1 18:22:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 8711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45613C3F2D4 for ; Wed, 1 Jun 2022 18:22:45 +0000 (UTC) Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by mx.groups.io with SMTP id smtpd.web12.1527.1654107762284672385 for ; Wed, 01 Jun 2022 11:22:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=SngCs0Rm; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f46.google.com with SMTP id c5-20020a1c3505000000b0038e37907b5bso3547616wma.0 for ; Wed, 01 Jun 2022 11:22:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=aZfnGwcIoyQxVXjefpVBBTRy7VwHZqlVhsQdXh+G9TI=; b=SngCs0Rmh9KZ+xkzFIO+ql+/3OoLlmEwpqF+CO69dL6edPft9JGkzefVt3z8QI9iga Zrcqn24P3LLelL64NqFmtEFV+i8ZwPLNbVahKhrfrCQbuUfLgFnPYFxEJYXGNSMAahXs pIjngP8aNxUlUe4EH6cn7cIaft8yRH1slR1hs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=aZfnGwcIoyQxVXjefpVBBTRy7VwHZqlVhsQdXh+G9TI=; b=gamRB09iPizO5uHYGNrHL/A5wZESXuREhllNwZQgXJlkn0JYW6VsEGN39tjbcnIxE8 JJdjtuetj4Ox8yljnrjGP7mdQI83hV0hzm4DTJl6YMUZYjkS0lDNuPwM3/P/0FTQdKTr M4Yynv22SRsrO9pCBfwTyrN5THsGtBSkpmzQbheJ9rWb4jcUrP+YfaPbsCYl22rVILdq K7NHc9fxc2WBJjZX6QKNm4SK7WKA0DiqolNK34dFriZj8Z5d4fxxmfWdkOtHQK0kC5YN twwI15/HAUAvahd4yXdT1Vu4ol2a9bDgqVjdX7+Jvdn1lkC0EmwRVqbyCRHTfBF50vRV grqg== X-Gm-Message-State: AOAM5301fIhyiMjFCL8VUp5cPsJmukmFDboagjKtlWn2WL/+wQA0vzhg D9w3/CipGV1ULTHn/MjONgN24D/GTBoHYpWP X-Google-Smtp-Source: ABdhPJyfgz24sBiSYo65gJhaZAwXUguTPhfLSGXyrG+/bDCyz6fq+qjCvqbUwRwpRSPv+c1z5S2XfA== X-Received: by 2002:a05:600c:4f90:b0:397:89b1:539b with SMTP id n16-20020a05600c4f9000b0039789b1539bmr26120026wmq.149.1654107760168; Wed, 01 Jun 2022 11:22:40 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eab:cafb:b8d4:fe67]) by smtp.gmail.com with ESMTPSA id l14-20020a05600c2cce00b0039751bb8c62sm6665150wmc.24.2022.06.01.11.22.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 11:22:39 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/3] runqueue: Fix unihash cache mismatch issues Date: Wed, 1 Jun 2022 19:22:37 +0100 Message-Id: <20220601182239.103588-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jun 2022 18:22:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13725 Very occasionally we see errors in eSDK testing on the autobuilder where the task hashes in the eSDK don't match what was just built. I was able to inspect one of these build directories and noticed that the bb_unihashes.dat file in the eSDK was zero sized. Whilst inspecting the code to understand the cause, I noticed that updated hashes are not saved out in subsequent updates of the values in the rehash process. Add a missing sync call to ensure this happens. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index f34f1568e2..1e47fe70ef 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2299,6 +2299,9 @@ class RunQueueExecute: self.rqdata.runtaskentries[hashtid].unihash = unihash bb.parse.siggen.set_unihash(hashtid, unihash) toprocess.add(hashtid) + if torehash: + # Need to save after set_unihash above + bb.parse.siggen.save_unitaskhashes() # Work out all tasks which depend upon these total = set() From patchwork Wed Jun 1 18:22:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 8710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 426B1C43334 for ; Wed, 1 Jun 2022 18:22:45 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web12.1528.1654107762532212830 for ; Wed, 01 Jun 2022 11:22:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=ZJepdSBO; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.52, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f52.google.com with SMTP id k16so3473936wrg.7 for ; Wed, 01 Jun 2022 11:22:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=vuDqOGGBcJe8glsvpoBDAvoSKsBQ4DL1lfZh0bI+GSI=; b=ZJepdSBOKtK8jisuIBmY3sV1PsY6/ymicovi4NgAbf5u+wQzattwAI6TK8amO28C0o HMOe/8QT0oIC/s4EFan3+OOCdLiaAnxrY9sKmItWgr160CjWaUPAlWsqhkldJds37MXP U2VunPzCQAFHcbbkKD/p7IL4kaI3Gyvy0haY8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vuDqOGGBcJe8glsvpoBDAvoSKsBQ4DL1lfZh0bI+GSI=; b=ZgdfoWst4kwk3a8UlAuG0HqcqdG7cLV4A0UmIAmIGGS/VOq/nqYqEmO/5unTzkGICT 1WMgVLyARXOOLl74PoQTJOjfz7r7KaX9PHlvG+Lajao8EPQ5KePXMIrTCrd64sz0cz6I RpsQRi6e6STQIC0XmC6hBdeNUFBuPCtBcQ/dS0GDKdVj9BFVtOrSFazyfjfhAZFypHcd w0TVW3eOosLKs6i/UXTKUQaAmt++iTXAlVJaratP7RWP204zdjc3mRQQOnxlnLXcKaMC 39dDXdJyXlrpgl3cSWNct31t6WCFt2/IMqsjMboHQAWk7mdRarPWbrvN/5/qAUzgpi/F 4wBg== X-Gm-Message-State: AOAM532Jt62p+o6F6Qp3ajst/TPeeY8bGqt+NZJ1hvARxGSND8nLzCyP C89wROHYCAUqZgcTiIKH3K+UQC8zVQ1wqKPu X-Google-Smtp-Source: ABdhPJyCHyArgW/P3A6vRtsiEBFrejMQETB9WY2hcEKoYA0YyHNj1QyzdUeHazi/x3qTsLWvGsoi2A== X-Received: by 2002:adf:f1cb:0:b0:210:a2a:51ec with SMTP id z11-20020adff1cb000000b002100a2a51ecmr588451wro.294.1654107760815; Wed, 01 Jun 2022 11:22:40 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eab:cafb:b8d4:fe67]) by smtp.gmail.com with ESMTPSA id l14-20020a05600c2cce00b0039751bb8c62sm6665150wmc.24.2022.06.01.11.22.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 11:22:40 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/3] cache/siggen: Add unihash cache copy function Date: Wed, 1 Jun 2022 19:22:38 +0100 Message-Id: <20220601182239.103588-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220601182239.103588-1-richard.purdie@linuxfoundation.org> References: <20220601182239.103588-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jun 2022 18:22:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13726 We see rare failures in eSDK generation with zero sized unihash cache files. This is almost certainly due to races in the cache file being updated. Add a copy function where the cache file can be copied with the lock held to avoid this. Signed-off-by: Richard Purdie --- lib/bb/cache.py | 9 +++++++++ lib/bb/siggen.py | 6 ++++++ 2 files changed, 15 insertions(+) diff --git a/lib/bb/cache.py b/lib/bb/cache.py index 92e9a3ced7..988c596c39 100644 --- a/lib/bb/cache.py +++ b/lib/bb/cache.py @@ -24,6 +24,7 @@ from collections.abc import Mapping import bb.utils from bb import PrefixLoggerAdapter import re +import shutil logger = logging.getLogger("BitBake.Cache") @@ -998,3 +999,11 @@ class SimpleCache(object): p.dump([data, self.cacheversion]) bb.utils.unlockfile(glf) + + def copyfile(self, target): + if not self.cachefile: + return + + glf = bb.utils.lockfile(self.cachefile + ".lock") + shutil.copy(self.cachefile, target) + bb.utils.unlockfile(glf) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 08eca7860e..3f3d6df54d 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -120,6 +120,9 @@ class SignatureGenerator(object): def save_unitaskhashes(self): return + def copy_unitaskhashes(self, targetdir): + return + def set_setscene_tasks(self, setscene_tasks): return @@ -358,6 +361,9 @@ class SignatureGeneratorBasic(SignatureGenerator): def save_unitaskhashes(self): self.unihash_cache.save(self.unitaskhashes) + def copy_unitaskhashes(self, targetdir): + self.unihash_cache.copyfile(targetdir) + def dump_sigtask(self, fn, task, stampbase, runtime): tid = fn + ":" + task From patchwork Wed Jun 1 18:22:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 8709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43311CCA473 for ; Wed, 1 Jun 2022 18:22:45 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web11.1547.1654107763505817775 for ; Wed, 01 Jun 2022 11:22:43 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Fjp1Od9Y; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f44.google.com with SMTP id m32-20020a05600c3b2000b0039756bb41f2so1546825wms.3 for ; Wed, 01 Jun 2022 11:22:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2bPcbiP1IwQ4QoflwSKWRQTFz2EYFkKmVq6xeV0tKTY=; b=Fjp1Od9YvGesPRSpNmqtqYxXA23H5by2mVAbY+gBgKMPppuHJlwsfXCk6Vs18LlCFA 8ucgQnvzS+6uhLmwbVLsl9xCKo1TzJ6h+SfjjA2tLRNWySi5ASuAgs7q0sYuj/Sr6SkU BH1wUaW+AGDk+eeCxADPyFOrRcHymjEcilwF8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2bPcbiP1IwQ4QoflwSKWRQTFz2EYFkKmVq6xeV0tKTY=; b=GyVBlH2glh6Fw8Tmns1OqSEA3Y80ltQDHlYNDriHVguIQJIMP4K2owfmFgdDUbueO6 hnvyHjcgkq0TcbkQTHplOHbYqnwolSW/uSEdkr4006UjPZuzqhWx9INuWVcrQ/AgDTMD ESv9vvCq9x26nQpyg8VZJDPnjOCiF5x+qvt80Lj1mVHtp7xUjbiT1R5RqE8aOPDX6bzx pag3oXz1oBZvZz82i5z+jey8/48ywwg52DZE7eAVQQMfUVU24yFvftL2iEQTieHYOfhY De3L7RAiMnxcutITj6PlkYIC1CmelOul5+w7OHZroZHmSe6oSocyVCzKeKaGgvwuFJ3k pDnQ== X-Gm-Message-State: AOAM531cQyZ6wKYLJ5JAfHEQJG/tzNOpU3Ef/NsfQKmkAWohJchoZZDx IaQy/hXhTM2vKC81rAgf4kQJE6G2kiCYKZUD X-Google-Smtp-Source: ABdhPJw2cdRDuH+knyVURkiEY5U0OGxUktvRIy3t152j+Wo6SZz5ONLU6moPxYmsMgNkvg0WWaZ/bw== X-Received: by 2002:a7b:c841:0:b0:394:890e:9195 with SMTP id c1-20020a7bc841000000b00394890e9195mr29841407wml.59.1654107761838; Wed, 01 Jun 2022 11:22:41 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eab:cafb:b8d4:fe67]) by smtp.gmail.com with ESMTPSA id l14-20020a05600c2cce00b0039751bb8c62sm6665150wmc.24.2022.06.01.11.22.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 11:22:41 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/3] bitbake: Bump to version 2.0.1 Date: Wed, 1 Jun 2022 19:22:39 +0100 Message-Id: <20220601182239.103588-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220601182239.103588-1-richard.purdie@linuxfoundation.org> References: <20220601182239.103588-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jun 2022 18:22:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13727 This allows OE to depend on the unihash copy functionality. Signed-off-by: Richard Purdie --- bin/bitbake | 2 +- lib/bb/__init__.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/bitbake b/bin/bitbake index 042c91807d..b56f6207c6 100755 --- a/bin/bitbake +++ b/bin/bitbake @@ -28,7 +28,7 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException if sys.getfilesystemencoding() != "utf-8": sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.") -__version__ = "2.0.0" +__version__ = "2.0.1" if __name__ == "__main__": if __version__ != bb.__version__: diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py index b8333bdb81..6b470aa195 100644 --- a/lib/bb/__init__.py +++ b/lib/bb/__init__.py @@ -9,7 +9,7 @@ # SPDX-License-Identifier: GPL-2.0-only # -__version__ = "2.0.0" +__version__ = "2.0.1" import sys if sys.version_info < (3, 6, 0):