From patchwork Tue Oct 4 15:54:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 800C3C433FE for ; Tue, 4 Oct 2022 15:55:18 +0000 (UTC) Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by mx.groups.io with SMTP id smtpd.web10.12291.1664898915909779205 for ; Tue, 04 Oct 2022 08:55:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=0uOOvstT; spf=softfail (domain: sakoman.com, ip: 209.85.216.41, mailfrom: steve@sakoman.com) Received: by mail-pj1-f41.google.com with SMTP id l12so13051620pjh.2 for ; Tue, 04 Oct 2022 08:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=DLZEoSid4yGN20VnYLy6CeAOYQ19VXJJ/FawhZptvUk=; b=0uOOvstTnNDZC9Jy1dC/tnAyC+kbMwBg37tIQU42nCYdyRij+JQRAh6pqCl/UTy4I0 o1KtyGfCI4a4DLZ26FfX/YMPZ0+Df25Ka25ihStMus06hUhaToxaokBOoDU937IHTiUr BdoN3auWMR+2MeL7ZJ8QE8+GEBY+0KgWS08xC8uODBFxNnQ19GJyNbRluVjtFGQ9an9Y KYMjyysEChVYtpSIChLhCwEr0+lF9VaWjkgP47EoJ10zdfFhNFd/Q1xI5N1W6L4GCu9T JusTySuY1S1+HaO0+XM0kY9hSYlAZo2NOh9imZTQ9vJeKRshxSLVaA1OGesOSgQE41AD FqrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=DLZEoSid4yGN20VnYLy6CeAOYQ19VXJJ/FawhZptvUk=; b=flNuOHT0Ut2oaT8wUuaavfNqwQj4DmEdzwl8T1wxQ1ZfpB0Kihk//Roi8Z2tmRoIx8 4j473NNgXP8p5eKPx9BNpDSSswCrdw5Z4nzau9uDoU2jb3TcobaIVtmEEkUPwI4diucB 2JzIVShdQulFlrEGpKCx7ubDzS8g09P6vA20FI++YarWuSkNhWGib+qeBqw2cP3ng5k4 WupKEsk4m79ceZpFFDavMSmUo647jPxQkZXluQIkC6KKtvrsvzsYD46WOl6xePde4MCB 7JUgos/u8ka1xJJepq9N+rttQ2OTBCVf/RNu6RUJ6f+UIOukboJX25Cr2lN18P1tS7B1 JhDw== X-Gm-Message-State: ACrzQf2NF3zAQpnJoEQvVJT3Rr/9aBvd1rGT9QNm2T9xzCKO3+d6GpQP 0/XLnMIqMEhVil/jgwtQFzcbRLyxnptORXgI X-Google-Smtp-Source: AMsMyM75DWD02N/PYt8+YCNVfQrciQtEtI/g9LCkkRB5Vvrd1PkDKuAOAGBYx7n12AqNhUyozQ7E1Q== X-Received: by 2002:a17:902:e74e:b0:178:b4c4:5ac4 with SMTP id p14-20020a170902e74e00b00178b4c45ac4mr27760862plf.11.1664898914985; Tue, 04 Oct 2022 08:55:14 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:14 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 1/8] runqueue: Ensure deferred tasks are sorted by multiconfig Date: Tue, 4 Oct 2022 05:54:52 -1000 Message-Id: <513bfd771d9095fcb6a8bf93806673dbf988a4de.1664898736.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14016 From: Richard Purdie We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. Signed-off-by: Richard Purdie (cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 111dc0ee..a23f7a89 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2523,11 +2523,14 @@ class RunQueueExecute: if update_tasks: self.sqdone = False - for tid in [t[0] for t in update_tasks]: - h = pending_hash_index(tid, self.rqdata) - if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: - self.sq_deferred[tid] = self.sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) + for mc in sorted(self.sqdata.multiconfigs): + for tid in sorted([t[0] for t in update_tasks]): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: From patchwork Tue Oct 4 15:54:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B78DC433FE for ; Tue, 4 Oct 2022 15:55:28 +0000 (UTC) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mx.groups.io with SMTP id smtpd.web11.12406.1664898918135068556 for ; Tue, 04 Oct 2022 08:55:18 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=6zxo2lZP; spf=softfail (domain: sakoman.com, ip: 209.85.210.182, mailfrom: steve@sakoman.com) Received: by mail-pf1-f182.google.com with SMTP id f140so8228681pfa.1 for ; Tue, 04 Oct 2022 08:55:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=dJN/iQdUKnAO96vhpniYvoPZSJmLqx5bazJL5maOIxQ=; b=6zxo2lZPmHuT02awghdjGICNM+p6SjdP9Z/7GwVl5BWGIn7a5OYAcvlEbV54WA4+xS wX+0+wNb8JJddNUHCoYUrmUs0JsrdO5DhIkOio2s/c5hmXiYf7reW1Yfw9bZg0jxHB0o XFQ1iqlQImh1rPGLtZHafIDdv3Hem2yvDtHy6oNHKpFmaEsacwSIekSbWPKome3sAtLL JCcZTF5bc0tUDXyrKvpJ9Put7TtZBagiWWvUupdCc5auJiu5QQrIW84Ou19jPXV4X+RO KPV412ZMg5S1i8ZwgLqhvDPHM8i5C4YMGXBhwPRLnZFw00uBxsfSckoef507DAl1tvuw dRcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=dJN/iQdUKnAO96vhpniYvoPZSJmLqx5bazJL5maOIxQ=; b=2ianNp2LLCMztBBLECBlsQZGoo71rWa5aUQYQ9cVUovhLbT+W6f02/PqA4XaI9IgGu XY5X+IQODLf0uT7kcGh1BubxRtLnX8/3jRnuFdz1zaurJx4ffa4gjV5Zm32+lVl74SEN Dcm9BtApjmIYySPBl/9T4heRB1JlLlSkq7nhlFPagd3C0fpJlcm+hYJw+v2hR/6B5LY1 USvHi9z+TGrjh8aZdhQ1UjMB/MBbSQb6A0BEEFfiT1HswcMz7Qvs3qPmJlpegHamcjLy Wq6d+f4ymKpgrKkLTujRCPtCaNQiB5/2SDC6lKHJcXBdU/fIjXAG4OhwVIcE9tkJwsCz oUlw== X-Gm-Message-State: ACrzQf1z6ttI892gp+YgCtKfC2YX6frzYzXE6fZ/OQ4nSERDLRsiXWgf 6k0GanxKYtIsIlm9U2CbOxty5kx0ahJ6OhZU X-Google-Smtp-Source: AMsMyM5RSFrZKOCkRr7QoS2f3aEcsX72+NnfANFkrK+rRwl0Ekwc6GCQaFRZGLT5KQH2hCImg4vdsA== X-Received: by 2002:a63:cf48:0:b0:439:7d1c:fedc with SMTP id b8-20020a63cf48000000b004397d1cfedcmr23960461pgj.281.1664898917175; Tue, 04 Oct 2022 08:55:17 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:16 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 2/8] runqueue: Improve deadlock warning messages Date: Tue, 4 Oct 2022 05:54:53 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14017 From: Richard Purdie Tweak the deadlock breaking messages to be explict about which task is blocked on which other task. The messages currently imply it is "freeing" the blocking task which is confusing. Signed-off-by: Richard Purdie (cherry picked from commit cf7f60b83adaded180f6717cb4681edc1d65b66d) Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index a23f7a89..0d89740b 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2254,10 +2254,12 @@ class RunQueueExecute: # No more tasks can be run. If we have deferred setscene tasks we should run them. if self.sq_deferred: - tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0]) - logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid) - if tid not in self.runq_complete: - self.sq_task_failoutright(tid) + deferred_tid = list(self.sq_deferred.keys())[0] + blocking_tid = self.sq_deferred.pop(deferred_tid) + logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s blocked by %s" % (deferred_tid, blocking_tid)) + if blocking_tid not in self.runq_complete: + logger.warning("Failing blocking task %s" % (blocking_tid)) + self.sq_task_failoutright(blocking_tid) return True if self.failed_tids: From patchwork Tue Oct 4 15:54:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E4EEC43217 for ; Tue, 4 Oct 2022 15:55:28 +0000 (UTC) Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by mx.groups.io with SMTP id smtpd.web10.12291.1664898915909779205 for ; Tue, 04 Oct 2022 08:55:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=nZLsNp4i; spf=softfail (domain: sakoman.com, ip: 209.85.216.41, mailfrom: steve@sakoman.com) Received: by mail-pj1-f41.google.com with SMTP id l12so13051810pjh.2 for ; Tue, 04 Oct 2022 08:55:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=kdSPLUVms5XZm3DMauYn+86F+BW4pJfYPBI5O+7Kudw=; b=nZLsNp4iUYI0ia7DL+jYJncHhn9Tk98CcIiLgY8/ZO6cHEkpAOYoLoOlhyTCGuZstj Beaxv6zFWgU9NvyndRabGySywbmtJaY1oZ3188Q7U4LwazFV0+e7KqaJji753ITRGn9n mR5klzth74OePzQH65XXLPM1lzGoAE77oOQ8ZLn4MHF9Cr30ouHWyx/ng7FjpTP++elB J/onN7h7bjrQGflH+/QXZJcPGiaKS3IQvh+Iwqm5LF7dVkY+pHlOnf8/iXwPor7jpwMS nGukowF+1hYS+kfQEi99hhR+dgj/AvarY0dJZUh/ByvD1MVe337Dk34jOBJ4FZUYRtb+ dQmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=kdSPLUVms5XZm3DMauYn+86F+BW4pJfYPBI5O+7Kudw=; b=y0vRuuw6F5a01XBZY9JSBYkm1ZyYmx2WQRy5b/D/r/fDrA+0tp8zRRIuLX41+K2RPP NwlQItSj0Hf4nz0nZ0Ts54sQnxy8y94m/LrPMrDsp1oKmdKD9j5Qoa9Ei3vC5/FP5azt Lu9JHhe4n2drGEsbY+QHMBUrVffVe1njQ2rJf2xUqTW9hDWC4yVojeC16k8cPrBNchBl leKtIig81Jqs8omS0VeyL3kv6dNsJrt+ZUrE9Ynu7WyMTytb18IrDDV7UqrCJT6fGhX+ O4qKHYawruvgPWRlRWLaEoxgbVIFSBwJLj+80gMTnjyuvIiJfXrgPyE+yWfOT+wnbSHF ZdRQ== X-Gm-Message-State: ACrzQf1r9PtU2zapLcjAGmlmi7rJtBkCSx6/feFmLMGnfW3f2JkFfHQI TuKAU3IOCL4oo8xqjv9g/UqLpiraCxltbVvA X-Google-Smtp-Source: AMsMyM4nCxtKZyyVpE/s+Z2GhliIVDVvCzn8Cx72qS/E5VyEAyurnh2vb/ZL0D8PiAoHQFusYEPA3g== X-Received: by 2002:a17:90b:4f45:b0:203:6d82:205c with SMTP id pj5-20020a17090b4f4500b002036d82205cmr383204pjb.224.1664898920178; Tue, 04 Oct 2022 08:55:20 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:18 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 3/8] runqueue: Drop deadlock breaking force fail Date: Tue, 4 Oct 2022 05:54:54 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14018 From: Richard Purdie I'm 99% certain this failing of a scenequeue task corrupts runqueue and causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted and did weird things so drop this code. We've seen builds where the deadlock triggers and it then tries to run tasks where the SQ task already ran with very confusing failures. It is likely it is this code causing it. Signed-off-by: Richard Purdie (cherry picked from commit 8efced47fcb47851a370fd6786df6fb377f99963) Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 0d89740b..48e25401 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2257,9 +2257,6 @@ class RunQueueExecute: deferred_tid = list(self.sq_deferred.keys())[0] blocking_tid = self.sq_deferred.pop(deferred_tid) logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s blocked by %s" % (deferred_tid, blocking_tid)) - if blocking_tid not in self.runq_complete: - logger.warning("Failing blocking task %s" % (blocking_tid)) - self.sq_task_failoutright(blocking_tid) return True if self.failed_tids: From patchwork Tue Oct 4 15:54:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BA5EC433F5 for ; Tue, 4 Oct 2022 15:55:28 +0000 (UTC) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mx.groups.io with SMTP id smtpd.web08.12416.1664898923443777737 for ; Tue, 04 Oct 2022 08:55:23 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=zcbk0U2G; spf=softfail (domain: sakoman.com, ip: 209.85.210.182, mailfrom: steve@sakoman.com) Received: by mail-pf1-f182.google.com with SMTP id a26so4484789pfg.7 for ; Tue, 04 Oct 2022 08:55:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=Wu7o3tuilGEMQ9r85i4+JQBAVRqbhHSx9nM74Xja2fQ=; b=zcbk0U2G1FqLIvvCY3KTSFGSaVGaDKaGwvQWXSJPCeyK/ISJwabKpoyR+eN/g02T4z 4YBwtshyNG8KCEYq2r3cR4Tn1ymWRGiH1eB6/diB7IoDTmtZMYvlJ1K94wZcmXlstaMW /M3UuEZB+VkeunRqpkrWG2vL7dp84LjJeoOS24Lo34wF4pa8bZYpvIld3GzQQVfHvpMP 4Sb/GYyOUiLZVeo1WZBB57j3hhyLtZp+4kvZThSvaX6bMElKBUDaeyB3UOmAGRvFhJIg ixPP50fX2STMFTFCiUeO2V5prR2VzS+9MayQLDMkWpiLI8U3qiSCWd6hD5Dv3mU4tWsO ayYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Wu7o3tuilGEMQ9r85i4+JQBAVRqbhHSx9nM74Xja2fQ=; b=F7QkmC9K4OlvbFHgN9yBKvhAVwmm+jNYUqr9MRUihfvEBaylzHiJ1Q1H4fiwd/QYrr cKgWmJV6Q66TE4Lpyhfa/pBO2etv8CpqSPbj7EB1y20C3fRYN0MCZ8lXRMRrThO0wFkY 20nlQEjz9F8lnyEx20xc7WwuYvN705qW1ZTH1+HrNkRuYECRH2hzclqy7ToKgA5v6T30 JItmI18N23HZHd4RZ3uuJLAZguWSZSwpBBDgbtLd3uvhRVK/3xDeEraevfgTYkqJU7nQ AEKFAsF37ETNhKiyJ0NzceBC0rNo5op0zwd2ibWM+JjHwXVqbw52VSOGEhol0pK6l4jA aSQw== X-Gm-Message-State: ACrzQf2tGvhcIp0A/1RCOW40x2T1kKNkKHZWV2gxVqaBMaL3K+j3gNmz XI/emN/Kj6DTHQ3vsGt1+xNOu5gKh6aJOJ06 X-Google-Smtp-Source: AMsMyM5tiwaSOBo7fslP50HRcTzC54rphGQMmgk7JG1GcvqJogjBsWXTaeQHQ86cy6B5X6pnZkYKxg== X-Received: by 2002:a63:6a85:0:b0:43b:d845:f67d with SMTP id f127-20020a636a85000000b0043bd845f67dmr23169506pgc.349.1664898922483; Tue, 04 Oct 2022 08:55:22 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:21 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 4/8] siggen: Fix insufficent entropy in sigtask file names Date: Tue, 4 Oct 2022 05:54:55 -1000 Message-Id: <63bb5591e833de0e7b552963ad9bc4b39e56fda9.1664898736.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14019 From: Joshua Watt Signature generation uses mkstemp() to get a file descriptor to a unique file and then write the signature into it. However, the unique file name generation in glibc is based on the system timestamp, which means that with highly parallel builds it is more likely than one might expect expected that a conflict will occur between two different builder nodes. When operating over NFS (such as a shared sstate cache), this can cause race conditions and rare failures (particularly with NFS servers that may not correctly implement O_EXCL). The signature generation code is particularly susceptible to races since a single "sigtask." prefix used for all signatures from all tasks, which makes collision even more likely. To work around this, add an internal implementation of mkstemp() that adds additional truly random entropy to the file name to eliminate conflicts. Signed-off-by: Joshua Watt Signed-off-by: Luca Ceresoli (cherry picked from commit 97955f3c1c738aa4b4478a6ec10a08094ffc689d) Signed-off-by: Steve Sakoman --- lib/bb/siggen.py | 2 +- lib/bb/utils.py | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 9fa568f6..bd6fc204 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -419,7 +419,7 @@ class SignatureGeneratorBasic(SignatureGenerator): bb.error("Taskhash mismatch %s versus %s for %s" % (computed_taskhash, self.taskhash[tid], tid)) sigfile = sigfile.replace(self.taskhash[tid], computed_taskhash) - fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.") + fd, tmpfile = bb.utils.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.") try: with bb.compress.zstd.open(fd, "wt", encoding="utf-8", num_threads=1) as f: json.dump(data, f, sort_keys=True, separators=(",", ":"), cls=SetEncoder) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index 95b3c898..92d44c52 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -28,6 +28,8 @@ import signal import collections import copy import ctypes +import random +import tempfile from subprocess import getstatusoutput from contextlib import contextmanager from ctypes import cdll @@ -1756,3 +1758,22 @@ def is_local_uid(uid=''): if str(uid) == line_split[2]: return True return False + +def mkstemp(suffix=None, prefix=None, dir=None, text=False): + """ + Generates a unique filename, independent of time. + + mkstemp() in glibc (at least) generates unique file names based on the + current system time. When combined with highly parallel builds, and + operating over NFS (e.g. shared sstate/downloads) this can result in + conflicts and race conditions. + + This function adds additional entropy to the file name so that a collision + is independent of time and thus extremely unlikely. + """ + entropy = "".join(random.choices("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890", k=20)) + if prefix: + prefix = prefix + entropy + else: + prefix = tempfile.gettempprefix() + entropy + return tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=dir, text=text) From patchwork Tue Oct 4 15:54:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61828C4332F for ; Tue, 4 Oct 2022 15:55:28 +0000 (UTC) Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by mx.groups.io with SMTP id smtpd.web10.12292.1664898925982367504 for ; Tue, 04 Oct 2022 08:55:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=wl5VQwAP; spf=softfail (domain: sakoman.com, ip: 209.85.216.54, mailfrom: steve@sakoman.com) Received: by mail-pj1-f54.google.com with SMTP id o9-20020a17090a0a0900b0020ad4e758b3so2837314pjo.4 for ; Tue, 04 Oct 2022 08:55:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=VgtIAAt3fhXpdYUi0bA5aXUJZ92leAc0liU2SOYUgaA=; b=wl5VQwAPdIe1V4tMazQ5PRVFKW2/719+oopsWAQVZ49x43NkDqwOp8S3Or824+JUWG 1sJrx7FhefLxEfwuXOsXsbO9XL/M6Tyz4fPjLVxOtp9z1GeTQr0FvI+M/VKE6iomVV5Y T2WUjqvFpw790mTtR19V5IyfukQCQsVl0rQEJFvOJVkHlXJbiet31ZyfWUF2akGDwMK9 +jr7Z80Z1yDcrS4xXURWP9Joqd/q1PEfhqFFCTOeKZ+lUyx9vPjqRpPqXsa5MPeNIfzS i4eElIO6gES3nTIeI4lJLEZ0gHYrHLlrqVN8Q8LYQAUjZ061bH8MG6MRsnP9GMjp8AUV jMSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=VgtIAAt3fhXpdYUi0bA5aXUJZ92leAc0liU2SOYUgaA=; b=TahJaxF7vV72H+O35AP2o5cPlE5Ynf9wczLKtSoW7D3Wch9ziNERR4fXvOWGQJlOaW XkWFvFYODULqI2keNteiVMHmZfJiA64ZWeYxlRu7wlyNzkxayLHw4xesBgey84go7wYm W2Gh4ujDcwVrXATbaiKkor93RamFbrlPWViLbaSDPkpAuPdWjt5F11TRPXx6Ubjb07mt UeG1cEyAtxAq45SNe8GZiw7W0AfF1MOxDtoeo4ue9zBR9NxuhzDY+aTaiCTfAXa7igd1 WUPJvXkMbcIc0QoPp96IfG+N1mt81NZm75LQHfH2GfKx04NJfI4cKuupTQ6tJBigEgY0 wdcg== X-Gm-Message-State: ACrzQf0qFxQKROzsbhRY1faCWRteWIze7fdFSlCKJB7v79nQo8E3Omx/ aiJBAcCCaU6r2JKn00+5oqYT0hqjrmciEMMs X-Google-Smtp-Source: AMsMyM7/kLaYXTw2j89j8nmKv7Hfl3pZ1H/BaciH4hxx4MBqDQfPu9fuyURhhmO4PCH979u0CVW56A== X-Received: by 2002:a17:902:ec82:b0:178:1062:3031 with SMTP id x2-20020a170902ec8200b0017810623031mr27514776plg.89.1664898924859; Tue, 04 Oct 2022 08:55:24 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:24 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 5/8] bitbake: Add copyright headers where missing Date: Tue, 4 Oct 2022 05:54:56 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14020 From: Richard Purdie Where copyright headers were not present, add them to make things clear. Signed-off-by: Richard Purdie (cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Steve Sakoman --- bin/bitbake-prserv | 2 ++ bin/bitbake-worker | 2 ++ bin/git-make-shallow | 2 ++ lib/bb/COW.py | 2 ++ lib/bb/asyncrpc/__init__.py | 2 ++ lib/bb/asyncrpc/client.py | 2 ++ lib/bb/asyncrpc/serv.py | 2 ++ lib/bb/codeparser.py | 2 ++ lib/bb/compress/_pipecompress.py | 2 ++ lib/bb/compress/lz4.py | 2 ++ lib/bb/compress/zstd.py | 2 ++ lib/bb/daemonize.py | 2 ++ lib/bb/exceptions.py | 2 ++ lib/bb/fetch2/osc.py | 2 ++ lib/bb/process.py | 2 ++ lib/bb/siggen.py | 2 ++ lib/bb/tests/compression.py | 2 ++ lib/bb/tests/cooker.py | 2 ++ lib/bblayers/__init__.py | 2 ++ lib/bblayers/action.py | 2 ++ lib/bblayers/common.py | 2 ++ lib/bblayers/layerindex.py | 2 ++ lib/bblayers/query.py | 2 ++ lib/prserv/__init__.py | 2 ++ lib/prserv/client.py | 2 ++ lib/prserv/db.py | 2 ++ lib/prserv/serv.py | 2 ++ lib/toaster/manage.py | 2 ++ 28 files changed, 56 insertions(+) diff --git a/bin/bitbake-prserv b/bin/bitbake-prserv index 323df66d..5be42f3c 100755 --- a/bin/bitbake-prserv +++ b/bin/bitbake-prserv @@ -1,5 +1,7 @@ #!/usr/bin/env python3 # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/bin/bitbake-worker b/bin/bitbake-worker index 9d850ec7..2f3e9f72 100755 --- a/bin/bitbake-worker +++ b/bin/bitbake-worker @@ -1,5 +1,7 @@ #!/usr/bin/env python3 # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/bin/git-make-shallow b/bin/git-make-shallow index 1d00fbf1..d0532c5a 100755 --- a/bin/git-make-shallow +++ b/bin/git-make-shallow @@ -1,5 +1,7 @@ #!/usr/bin/env python3 # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/COW.py b/lib/bb/COW.py index 23c22b65..76bc08a3 100644 --- a/lib/bb/COW.py +++ b/lib/bb/COW.py @@ -3,6 +3,8 @@ # # Copyright (C) 2006 Tim Ansell # +# SPDX-License-Identifier: GPL-2.0-only +# # Please Note: # Be careful when using mutable types (ie Dict and Lists) - operations involving these are SLOW. # Assign a file to __warn__ to get warnings about slow operations. diff --git a/lib/bb/asyncrpc/__init__.py b/lib/bb/asyncrpc/__init__.py index c2f2b3c0..9a85e996 100644 --- a/lib/bb/asyncrpc/__init__.py +++ b/lib/bb/asyncrpc/__init__.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/asyncrpc/client.py b/lib/bb/asyncrpc/client.py index 34960197..881434d2 100644 --- a/lib/bb/asyncrpc/client.py +++ b/lib/bb/asyncrpc/client.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/asyncrpc/serv.py b/lib/bb/asyncrpc/serv.py index b4cffff2..e14df18e 100644 --- a/lib/bb/asyncrpc/serv.py +++ b/lib/bb/asyncrpc/serv.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py index 3b3c3b41..9d66d3ae 100644 --- a/lib/bb/codeparser.py +++ b/lib/bb/codeparser.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/compress/_pipecompress.py b/lib/bb/compress/_pipecompress.py index 5de17a82..4a403d62 100644 --- a/lib/bb/compress/_pipecompress.py +++ b/lib/bb/compress/_pipecompress.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # # Helper library to implement streaming compression and decompression using an diff --git a/lib/bb/compress/lz4.py b/lib/bb/compress/lz4.py index 0f6bc51a..88b09893 100644 --- a/lib/bb/compress/lz4.py +++ b/lib/bb/compress/lz4.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/compress/zstd.py b/lib/bb/compress/zstd.py index 50c42133..cdbbe9d6 100644 --- a/lib/bb/compress/zstd.py +++ b/lib/bb/compress/zstd.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/daemonize.py b/lib/bb/daemonize.py index 4957bfd4..76894044 100644 --- a/lib/bb/daemonize.py +++ b/lib/bb/daemonize.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/exceptions.py b/lib/bb/exceptions.py index ecbad599..801db9c8 100644 --- a/lib/bb/exceptions.py +++ b/lib/bb/exceptions.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/fetch2/osc.py b/lib/bb/fetch2/osc.py index eb0f82c8..bf4c2f05 100644 --- a/lib/bb/fetch2/osc.py +++ b/lib/bb/fetch2/osc.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # """ diff --git a/lib/bb/process.py b/lib/bb/process.py index be2c15a1..4c7b6d39 100644 --- a/lib/bb/process.py +++ b/lib/bb/process.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index bd6fc204..9a20fc8e 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/tests/compression.py b/lib/bb/tests/compression.py index d3ddf67f..95af3f96 100644 --- a/lib/bb/tests/compression.py +++ b/lib/bb/tests/compression.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bb/tests/cooker.py b/lib/bb/tests/cooker.py index c82d4b7b..9e524ae3 100644 --- a/lib/bb/tests/cooker.py +++ b/lib/bb/tests/cooker.py @@ -1,6 +1,8 @@ # # BitBake Tests for cooker.py # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bblayers/__init__.py b/lib/bblayers/__init__.py index 4e7c09da..78efd297 100644 --- a/lib/bblayers/__init__.py +++ b/lib/bblayers/__init__.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bblayers/action.py b/lib/bblayers/action.py index 6723e2c6..454c2514 100644 --- a/lib/bblayers/action.py +++ b/lib/bblayers/action.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bblayers/common.py b/lib/bblayers/common.py index 6c76ef35..f7b9cee3 100644 --- a/lib/bblayers/common.py +++ b/lib/bblayers/common.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py index 79365162..0ac8fd2e 100644 --- a/lib/bblayers/layerindex.py +++ b/lib/bblayers/layerindex.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/bblayers/query.py b/lib/bblayers/query.py index 525d4f0d..9142ec44 100644 --- a/lib/bblayers/query.py +++ b/lib/bblayers/query.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/prserv/__init__.py b/lib/prserv/__init__.py index 9961040b..38ced818 100644 --- a/lib/prserv/__init__.py +++ b/lib/prserv/__init__.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/prserv/client.py b/lib/prserv/client.py index a3f19dda..69ab7a4a 100644 --- a/lib/prserv/client.py +++ b/lib/prserv/client.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/prserv/db.py b/lib/prserv/db.py index 2710d4a2..b4bda707 100644 --- a/lib/prserv/db.py +++ b/lib/prserv/db.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py index 0a20b927..c686b206 100644 --- a/lib/prserv/serv.py +++ b/lib/prserv/serv.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/lib/toaster/manage.py b/lib/toaster/manage.py index ae32619d..f8de49c2 100755 --- a/lib/toaster/manage.py +++ b/lib/toaster/manage.py @@ -1,5 +1,7 @@ #!/usr/bin/env python3 # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # From patchwork Tue Oct 4 15:54:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BD21C43219 for ; Tue, 4 Oct 2022 15:55:28 +0000 (UTC) Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by mx.groups.io with SMTP id smtpd.web12.12504.1664898927981725831 for ; Tue, 04 Oct 2022 08:55:28 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=N24L3Of/; spf=softfail (domain: sakoman.com, ip: 209.85.216.45, mailfrom: steve@sakoman.com) Received: by mail-pj1-f45.google.com with SMTP id lx7so13253408pjb.0 for ; Tue, 04 Oct 2022 08:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=gE1JcnoXBI2sJQmRPjFIy2Bdfj0eBjSqgCpOzPe+AXg=; b=N24L3Of/z46vtiJLLNMtC97HLsViEQ04H5XwGtjUbh+WzMGqSL7Y38zgl8QtCY6+rN NUN0jtcPtRYtlRnBDKb9bAnmcQ7SvYRs5CPS5hwZ+vinxIbvBBreWKGkjkagKyUtv+X7 Z0ig8DyDTLGXEAoivLl8LhhwaUS+bfsjYAhJqaAlNFYAimqtioLmHUgbXhwuHBc83HDE syCpjtqibhnEWFolzQHeSFrvP9n7m7a104nbyE3nG1nWzrpSewn5ArJ3WjB2R6Yh/if9 8L5HkBrYrP4wXseBWZZcHHrpv7mypEXf72+/aGWNRSeNYsHM/ASUIe0qJV4AgTo/AkKn 9z5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=gE1JcnoXBI2sJQmRPjFIy2Bdfj0eBjSqgCpOzPe+AXg=; b=TSHlIy7nJUin/f20/PnN20e82EmSHeoUbp9cM255KRmoJm7cgXwzgwkJL3FwfoO1+Y gFTcKg0Y1CzbWdNvdcfoksCNE2b+TjhiRcb8tNYAoELNkvFt+B5bpMuMjYNkcatqM90v nDlbwBTfMo5aQvpiA4FP3w8uBc0BDerBdx0j42CNAZu77TY0UzbfYzsOSICuqqKkObD4 pxYaNWcJvOGwlmudSgtJlmLzd81v02VPvZnWCbgbV/T5X75438Hgmp7GEercf+ppIyeD zwkyUI9gRtcPcHSRPXmmIR3oFgBnQO2innJrBcNAsI8VkgFaJ9R7UMGYLUrWAgUMBaCa jiMw== X-Gm-Message-State: ACrzQf1H4rirNpzXLdyWK6EPl67aADQgBIpVTuhFQPtHFexJJqqS4wYd oyKLHpgihgP/wT8C3RPH/2R723uIy/734i2B X-Google-Smtp-Source: AMsMyM6WLzSN3URR3ib497Jd4T0zjlc7RO+UIwiq4J+NA1PyJeqw4Fon84cA5ioM6UWoEx0LMDq2vA== X-Received: by 2002:a17:90b:4b0d:b0:203:690d:9f7f with SMTP id lx13-20020a17090b4b0d00b00203690d9f7fmr358543pjb.145.1664898927041; Tue, 04 Oct 2022 08:55:27 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:26 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 6/8] Fix npm to use https rather than http Date: Tue, 4 Oct 2022 05:54:57 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14021 From: Neil Horman Hit this error while building nlf-native recently: { "error": { "summary": "URI malformed", "detail": "" } } Some poking about led me to discover that: 1) The npm.py tool replaces npm:// with http://, not https:// 2) Some versions of the npm tool don't handle 301 redirects properly, choosing to display the above error instead when using the default nodejs registry It would be good to go fix npm to handle the redirect properly, but it seems like it would also be good to assume secure http when contacting a registry, hence, this patch Signed-off-by: Neil Horman Signed-off-by: Richard Purdie (cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/npm.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py index 8f7c10ac..8a179a33 100644 --- a/lib/bb/fetch2/npm.py +++ b/lib/bb/fetch2/npm.py @@ -156,7 +156,7 @@ class Npm(FetchMethod): raise ParameterError("Invalid 'version' parameter", ud.url) # Extract the 'registry' part of the url - ud.registry = re.sub(r"^npm://", "http://", ud.url.split(";")[0]) + ud.registry = re.sub(r"^npm://", "https://", ud.url.split(";")[0]) # Using the 'downloadfilename' parameter as local filename # or the npm package name. From patchwork Tue Oct 4 15:54:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6071EC4332F for ; Tue, 4 Oct 2022 15:55:38 +0000 (UTC) Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by mx.groups.io with SMTP id smtpd.web10.12293.1664898930090141835 for ; Tue, 04 Oct 2022 08:55:30 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=DV5ajeYV; spf=softfail (domain: sakoman.com, ip: 209.85.216.53, mailfrom: steve@sakoman.com) Received: by mail-pj1-f53.google.com with SMTP id x1-20020a17090ab00100b001fda21bbc90so19005571pjq.3 for ; Tue, 04 Oct 2022 08:55:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=gebORtQXX9mXxvhJ88KIdWlFpm9g069u8KhIW60lErE=; b=DV5ajeYVG5IrVxTKxpscB7RrDXMrkeQH6VFysj9k4oJ+/09e16DqkpN9l1HE/sAckf V5m1dgHeB2jz74DzG79YR3F7yIR2rBSFaCHQkVHW/BFkvGK8tKQVkVOw9dtFjy81lNEb i65DhL4RsFEZgAsvF9odJ9m7YDfjr+GcGsVn7uob/IjLW4Kp+yyrqMjWS3jp597CB+qV 4Im5unsBviA/egsiQGDHfOsHTbIbWgFc9hHUYURdC5gZ/k7hUoPnqIhammq9R/Xzt4Cp D+ElROBJQ9v1/Gn7fctllheTvoFdJ77Vlcbvcvx+TYImQetgSBiPSwyJJHRSBri2RYeX 6olw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=gebORtQXX9mXxvhJ88KIdWlFpm9g069u8KhIW60lErE=; b=1r1Dd1ZsqJdZ0XqXyOFPKzHHMVFqW3/GDigyUflA563W14bF/r9O4R6YH1D5ldDXSN qXG++g2TP7IsuyeqCmM1GSkNut9cOg41Vu9Gn86GXw8rxyDnOj7dXMVW4a5/WtB65/fU 5P7Gvpp+jRNeyGKRPV2xy//uQavlaThuGf5iKinwplQ6RMRnmHZ0G9hrP2sTGk5DmXb8 D0OVbcpGBhskMubc53u15mOU+4igcyyUdf3sp0q5zAKjO6BcP70O/mS1KHRCOxk4fsnr fPKJEUlkFbPFUZJuBzTeXo/Z5IaiLtwrcA5cbHXkA+jqHhPR5CM2HtVYWRh1aEt2KPIX NOhQ== X-Gm-Message-State: ACrzQf3yBkVYsbW4nNqWpP5L7A1EQEfZ3V0iEBprHThmour3eVK+Ljdg pwzu23QFCcqbiKamHOTQpztv/rxNW0WxOYNG X-Google-Smtp-Source: AMsMyM7v2kWOUw3HJBaguJtXKIIBZf2DlJ59GAAdKEy3KwupxUSPs2IMcPTQim+MVivqln2ZP31dRQ== X-Received: by 2002:a17:90a:50:b0:20a:b146:e75e with SMTP id 16-20020a17090a005000b0020ab146e75emr384322pjb.216.1664898929170; Tue, 04 Oct 2022 08:55:29 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:28 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 7/8] gitsm: Error out if submodule refers to parent repo Date: Tue, 4 Oct 2022 05:54:58 -1000 Message-Id: <987712c4c8fefd86a1f5116c11ee86e296e852ee.1664898736.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:38 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14022 From: Pavel Zhukov If submodule refers to specific revision of the parent repository it causes deadlock in bitbake locking mechanism (lock is acquired to fetch the parent and cannot be released before all submodules are fetched). raise FetchError in such situation to prevent deadlocking. [Yocto 14045] Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie (cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/gitsm.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py index c1950e48..25d5db0e 100644 --- a/lib/bb/fetch2/gitsm.py +++ b/lib/bb/fetch2/gitsm.py @@ -115,6 +115,9 @@ class GitSM(Git): # This has to be a file reference proto = "file" url = "gitsm://" + uris[module] + if "{}{}".format(ud.host, ud.path) in url: + raise bb.fetch2.FetchError("Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake." \ + "Consider using git fetcher instead.") url += ';protocol=%s' % proto url += ";name=%s" % module From patchwork Tue Oct 4 15:54:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 13533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 606F4C433F5 for ; Tue, 4 Oct 2022 15:55:38 +0000 (UTC) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mx.groups.io with SMTP id smtpd.web08.12417.1664898932267689714 for ; Tue, 04 Oct 2022 08:55:32 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=KlI/OS+w; spf=softfail (domain: sakoman.com, ip: 209.85.214.180, mailfrom: steve@sakoman.com) Received: by mail-pl1-f180.google.com with SMTP id h10so10302172plb.2 for ; Tue, 04 Oct 2022 08:55:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=nHVvsCc4sF3f1OELLFL7N9/tPU0TcAzz54wittgRkMc=; b=KlI/OS+wIrzAhQ+fEA/478CvhmNT+uCcfiFljCJUroHxOgBDPst8SZZEcp4trVuFkP gkrk7l5btCKVLxLsdhg+E43ibjuiJ7pDD/cxKizzLgBCE0NDH9p8f7Jx5N+wNt1ISpAg CvGW33vvxSGJdMvkgOUTwXjf38Y40lrpab5xFqu2sFo7gP51tRbCQHMky+j3AFIkwezy P/TK6TZ64swkCVsx6oL+p13lex9DnIF0Es50WeTQCAWBP395L80Lq6T8h/E/HK6tkV5G p8l3uJCQ+uBg2M75+efop5EYQv0mrOotrNyiwHcCHxHuLjLsu9OhaGQK3uKc27XiPArU g8Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=nHVvsCc4sF3f1OELLFL7N9/tPU0TcAzz54wittgRkMc=; b=jUWW0dueWnEdcvCKvKUCaN1SUbZ1gG6D6k6fsVB8/XXj/lXVOpSd/rL0s/VURRJs4n IsoF2I9wWcJnHqclrj+MAcqKoimNEmDBlK3+jluCDlhEtB5YstSUrCuLwRyxoUa5HcBT mLQUgtLXMb+b/XXhxo69zITVEqTp+VqNSFX8eH14mQhOwhLekoJenr4zeVMOgssKzokZ uy5eapJ0xiQH5o/DZu53lEDlFOzJmMLEEjdu1JTz+10sQWPVkpvD2LL+aK+jjSiXb0T/ bFs13PCP+8JCtjpsoih35R76fQRGQ7SNuftxfmnBySq+AvZh8PShmrYnk/1wfG8mpbZR lQug== X-Gm-Message-State: ACrzQf0JTZi920sSP0Ac+yQ6dvG/FxmMQcQiqW2vdfgbgRddiqnOeR7M IVxMWXBlzzpGM9kjrc1tyv+uVpgm4aEiIjeP X-Google-Smtp-Source: AMsMyM4AdQhCiSG/9qzcscfKRgY/qa6PfqMd5MJjfDwRdCA59CIldjWaE1SekUBOVyffU+MaeENXIg== X-Received: by 2002:a17:90b:3b44:b0:205:d22a:e94 with SMTP id ot4-20020a17090b3b4400b00205d22a0e94mr371379pjb.15.1664898931343; Tue, 04 Oct 2022 08:55:31 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id k18-20020a170902c41200b0017e232b6724sm4716457plk.69.2022.10.04.08.55.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Oct 2022 08:55:30 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 8/8] asyncrpc/client: Fix unix domain socket chdir race issues Date: Tue, 4 Oct 2022 05:54:59 -1000 Message-Id: <6603c3e39f1cf746669ec6c9f0be8c6e6ece426e.1664898736.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 04 Oct 2022 15:55:38 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14023 From: Richard Purdie The connect_unix() call had a bug where if a relative path to a socket was passed (which the non-async client always does), and the current working directory was changed after the initial call, it would fail to reconnect if it became disconnected, since the socket couldn't be found relative to the new current working directory. To work around this, change the socket connection for UNIX domain sockets to be synchronous and change current working before connecting. This isn't ideal since the connection could block the entire event loop, but in practice this shouldn't happen since the socket are local files anyway. Help debugging and resolving from Joshua Watt. Signed-off-by: Richard Purdie (cherry picked from commit 5964bb67bb20df7f411ee0650cf189504a05cf25) Signed-off-by: Steve Sakoman --- lib/bb/asyncrpc/client.py | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/lib/bb/asyncrpc/client.py b/lib/bb/asyncrpc/client.py index 881434d2..fa042bbe 100644 --- a/lib/bb/asyncrpc/client.py +++ b/lib/bb/asyncrpc/client.py @@ -31,7 +31,17 @@ class AsyncClient(object): async def connect_unix(self, path): async def connect_sock(): - return await asyncio.open_unix_connection(path) + # AF_UNIX has path length issues so chdir here to workaround + cwd = os.getcwd() + try: + os.chdir(os.path.dirname(path)) + # The socket must be opened synchronously so that CWD doesn't get + # changed out from underneath us so we pass as a sock into asyncio + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0) + sock.connect(os.path.basename(path)) + finally: + os.chdir(cwd) + return await asyncio.open_unix_connection(sock=sock) self._connect_sock = connect_sock @@ -150,14 +160,8 @@ class Client(object): setattr(self, m, self._get_downcall_wrapper(downcall)) def connect_unix(self, path): - # AF_UNIX has path length issues so chdir here to workaround - cwd = os.getcwd() - try: - os.chdir(os.path.dirname(path)) - self.loop.run_until_complete(self.client.connect_unix(os.path.basename(path))) - self.loop.run_until_complete(self.client.connect()) - finally: - os.chdir(cwd) + self.loop.run_until_complete(self.client.connect_unix(path)) + self.loop.run_until_complete(self.client.connect()) @property def max_chunk(self):