From patchwork Tue Jun 7 14:17:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 8980 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93EEACCA47C for ; Tue, 7 Jun 2022 14:17:41 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web12.13089.1654611458822943738 for ; Tue, 07 Jun 2022 07:17:39 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=CKt2hYHs; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f46.google.com with SMTP id a15so15863943wrh.2 for ; Tue, 07 Jun 2022 07:17:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=h0Yj2Fzgr+IPkdpXXbXzfSW3oXM/mbk/wn1BlAHKi4o=; b=CKt2hYHsWUXtCDCDVxbQuvb4jqBdi+bAQYvfs5HaBWDqU/abUzCurki8iIX6IYigHR +9SlhGZkjOwjMX1bi9pMRC4JA2AgsP9NwLqFbUXAJmBprnh2N3BdP6fStIg4RN7U01gB 0g27Ar1XdEmdgh9XChB0DwwRMBZJgaQ8iGcxk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=h0Yj2Fzgr+IPkdpXXbXzfSW3oXM/mbk/wn1BlAHKi4o=; b=fYWQVs7akyx8PNWlc0Egr5IHsABVwdU89SBhdNJXY0CW7M87Ig7Le0SqUspRA2p/7G KdlWiqhBwHAOOYWEA5i3JUKqnXRzZKbn1X6dqRTFbzehO7raaFbvivx2CfsaE+to61N8 3fRzW5s2nBugepPX1YrTsz/nYFPmZjqcKW3UdXkKbdG/1azXyenjr753jB6dvIm1p6an AjcXyJWpXr2/qOzZZv+O/zKGRcdwD2tzbitqyRO75hpb9cGCmQFd3JOCi6Z2shoIMClq t+q9d7zK3gmjxgVfF0EdY5a7Nrr9YFx43cibzuch2T08dfNRmwyxQceNXAPFy5sAPhUL ZSRA== X-Gm-Message-State: AOAM530OsxvJf201B7kwt2jDX3e8jWv0iClgbN0AKBLKeI1UaVAPd+Lr tP8VETUrNrtCFk963M1UNI4RI9py4nVGPw== X-Google-Smtp-Source: ABdhPJzL8E7gEvipl4nhTtNxhbE+Fq3gjsUM3xiRVcFIrxH79q2k+51kcLYjLjo+T79WYbqr1bik3Q== X-Received: by 2002:a05:6000:147:b0:214:7d6e:cb1d with SMTP id r7-20020a056000014700b002147d6ecb1dmr21238054wrx.650.1654611456747; Tue, 07 Jun 2022 07:17:36 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:e835:fc94:6c5b:fc12]) by smtp.gmail.com with ESMTPSA id bd9-20020a05600c1f0900b0039c47767e23sm10483182wmb.33.2022.06.07.07.17.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jun 2022 07:17:36 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/3] fetch/wget: Move files into place atomically Date: Tue, 7 Jun 2022 15:17:33 +0100 Message-Id: <20220607141735.341158-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 Jun 2022 14:17:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13736 Signed-off-by: Richard Purdie --- lib/bb/fetch2/wget.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index b3a3de571a..b2b542e1dc 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -106,10 +106,9 @@ class Wget(FetchMethod): fetchcmd = self.basecmd - if 'downloadfilename' in ud.parm: - localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) - bb.utils.mkdirhier(os.path.dirname(localpath)) - fetchcmd += " -O %s" % shlex.quote(localpath) + localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp" + bb.utils.mkdirhier(os.path.dirname(localpath)) + fetchcmd += " -O %s" % shlex.quote(localpath) if ud.user and ud.pswd: fetchcmd += " --auth-no-challenge" @@ -133,6 +132,10 @@ class Wget(FetchMethod): self._runwget(ud, d, fetchcmd, False) + # Remove the ".tmp" and move the file into position atomically + # Our lock prevents multiple writers but mirroring code may grab incomplete files + os.rename(localpath, localpath[:-4]) + # Sanity check since wget can pretend it succeed when it didn't # Also, this used to happen if sourceforge sent us to the mirror page if not os.path.exists(ud.localpath): From patchwork Tue Jun 7 14:17:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 8981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95169C3F2D4 for ; Tue, 7 Jun 2022 14:17:41 +0000 (UTC) Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) by mx.groups.io with SMTP id smtpd.web08.13240.1654611459212812539 for ; Tue, 07 Jun 2022 07:17:39 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=A4w9ZDRV; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f45.google.com with SMTP id p10so24379768wrg.12 for ; Tue, 07 Jun 2022 07:17:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=NUid9dP0T4FX5pvY1hUyiPifnB0fg6D4dpOqDoj+O2M=; b=A4w9ZDRVIzwnM4A39i0GL603NdDirRUY0nz2sJ+BoNb/As3tf8oMtQZQDd4DmdOSbW sb8Wo5UNfpGkPXHVAkkHt+2C4UKOFnEWnc44IVXtpWvzdm/YPIsAfp56XO4wqTnxrmmu Ho4wpLMkfyp2aD9cjiZwYdZT1q9C0OqOEWCwU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NUid9dP0T4FX5pvY1hUyiPifnB0fg6D4dpOqDoj+O2M=; b=tod7++LQDxvwfNHyICN8IJMcgDAlbQH6DkiEdTy0CiozWvz78lsK1UrA/n1fGEONl1 cq7Ydfps1uvkqYGkm/uSHxspL2Ioep5DI6QbR0qaJwpGV5oWxAq71/o/MbWkdu0164R3 qpKNZb+CHZe2mBi1jPCNEGkRjE0BBKKr1CePWplIsoY1hixpno7PvOT/mOY1Ct6uQ2vJ MIM6NKN5MpEvYsr05ibV5/qgJJ4I+VNya4B4UEg/rYPb7yx2FUwW7xRTQyfus8z04odN cwp5RjxK9UdFULaqRs2H54/mKDVVO/6G1EONjMIZTvvqf411RNkqJFK+jQuZXDdHbgAW xaTQ== X-Gm-Message-State: AOAM5328VepN4hlEXorX26+WWs0OVQHHoapmOQSq8wf/c8jzOYYpyOK/ GlADWAKcMWCFlygXeZzoge0ib5iKKePsuQ== X-Google-Smtp-Source: ABdhPJw3eG59zNoBoc+Ahhx4uZF6/NPflAA0+qsBotcQUnqLtnWqs4v6rrla6hDKtCm93//0kDHegQ== X-Received: by 2002:adf:e8cd:0:b0:210:2b10:ab22 with SMTP id k13-20020adfe8cd000000b002102b10ab22mr26863755wrn.476.1654611457420; Tue, 07 Jun 2022 07:17:37 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:e835:fc94:6c5b:fc12]) by smtp.gmail.com with ESMTPSA id bd9-20020a05600c1f0900b0039c47767e23sm10483182wmb.33.2022.06.07.07.17.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jun 2022 07:17:36 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/3] server/process: Avoid risk of exception deadlocks Date: Tue, 7 Jun 2022 15:17:34 +0100 Message-Id: <20220607141735.341158-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220607141735.341158-1-richard.purdie@linuxfoundation.org> References: <20220607141735.341158-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 Jun 2022 14:17:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13737 The open coded lock acquire/release in the UI event handler doesn't cover the case an exception occurs and if one did, it could deadlock the code. Switch to use 'with' statements which would handle this possibility. We have seen deadlocks in the UI at exit this so this removes a possible cause. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 613956f30f..74b74dc39b 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -667,18 +667,14 @@ class BBUIEventQueue: self.t.start() def getEvent(self): - self.eventQueueLock.acquire() - - if len(self.eventQueue) == 0: - self.eventQueueLock.release() - return None - - item = self.eventQueue.pop(0) + with self.eventQueueLock: + if len(self.eventQueue) == 0: + return None - if len(self.eventQueue) == 0: - self.eventQueueNotify.clear() + item = self.eventQueue.pop(0) + if len(self.eventQueue) == 0: + self.eventQueueNotify.clear() - self.eventQueueLock.release() return item def waitEvent(self, delay): @@ -686,10 +682,9 @@ class BBUIEventQueue: return self.getEvent() def queue_event(self, event): - self.eventQueueLock.acquire() - self.eventQueue.append(event) - self.eventQueueNotify.set() - self.eventQueueLock.release() + with self.eventQueueLock: + self.eventQueue.append(event) + self.eventQueueNotify.set() def send_event(self, event): self.queue_event(pickle.loads(event)) From patchwork Tue Jun 7 14:17:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 8979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93293C43334 for ; Tue, 7 Jun 2022 14:17:41 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web11.12808.1654611460026640319 for ; Tue, 07 Jun 2022 07:17:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=U/rqBz7t; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id x17so24392711wrg.6 for ; Tue, 07 Jun 2022 07:17:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=hk1MOm/dn7Atwiib8i/3PyKJPqST7zE20Fp914FJBeo=; b=U/rqBz7tS5eoXwNP5OKahgEvSRUqb86VwJ1QveSG94kTEgG9hbi/h7/73O9kAWMNIA 3uNUfszJvVzLtikVZum7LKTa00PJlC/DfChtOS0Fv7mdV6v4iueGL/qcpQutI+kRIH+B GrvrcXeP38Sc3Uzr0tx/1YzYCFjBYxsq+NOuQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hk1MOm/dn7Atwiib8i/3PyKJPqST7zE20Fp914FJBeo=; b=GT54TfXEA57G2xNgiQOy3612yd+uB01yavY06IBZLCWWOQ7IH3B8LX8eZLC0CLFcmV 38I5Bisoxynp+HL2ZWNB5h54qXlVRfmf2KaCmgAzAwriF12jpat3CGRTjqjLrkWW8TzM QXTHPqD7pHD7YJceqIPPFa829f/sHqSF19lXkOLZ+PiYUB1pppCJibux5z5nwywE7dRx Isi38hPehAjwFODPiE1n/OPnniQ9QYD2PQcFyFKjV1XVNlB0u41YJ49uTMcT7hz7OYXc niNAKEMBP4byRhn76PkuY3y96RjYqeTqmndTvE0pka1uEs/9ppT1Leo2EGkDwVquvVUY K74A== X-Gm-Message-State: AOAM533eejSzO0yecE7n0H45dkkhv8jmm1FOpDCvOYRYiF3N4RrlhIYm e2DaN/Q4jd8Ov2RFbAPaMfCRsaoB+3utzQ== X-Google-Smtp-Source: ABdhPJw3EHYNGsOoOXklRuouyH36YM0lm2RSglcnKmb+7HoOvM154agOriVtasJwIiTuS02YCDApqg== X-Received: by 2002:a05:6000:257:b0:216:509c:2c4b with SMTP id m23-20020a056000025700b00216509c2c4bmr18147217wrz.196.1654611458217; Tue, 07 Jun 2022 07:17:38 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:e835:fc94:6c5b:fc12]) by smtp.gmail.com with ESMTPSA id bd9-20020a05600c1f0900b0039c47767e23sm10483182wmb.33.2022.06.07.07.17.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jun 2022 07:17:37 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/3] server/process: Remove daemonic thread usage Date: Tue, 7 Jun 2022 15:17:35 +0100 Message-Id: <20220607141735.341158-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220607141735.341158-1-richard.purdie@linuxfoundation.org> References: <20220607141735.341158-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 Jun 2022 14:17:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13738 We're seeing UI deadlocks occasionally and this is possibly due to the use of a daemonic thread in the UI event queue processing. This thread could terminate holding a threading Lock() which would cause issues for the process when exitting. Change the shutdown process to handle this more cleanly. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 74b74dc39b..b330520a98 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -437,6 +437,7 @@ class BitBakeProcessServerConnection(object): self.socket_connection = sock def terminate(self): + self.events.close() self.socket_connection.close() self.connection.connection.close() self.connection.recv.close() @@ -662,7 +663,6 @@ class BBUIEventQueue: self.reader = ConnectionReader(readfd) self.t = threading.Thread() - self.t.daemon = True self.t.run = self.startCallbackHandler self.t.start() @@ -693,13 +693,16 @@ class BBUIEventQueue: bb.utils.set_process_name("UIEventQueue") while True: try: - self.reader.wait() + self.reader.wait(0.25) event = self.reader.get() self.queue_event(event) - except EOFError: + except (EOFError, OSError): # Easiest way to exit is to close the file descriptor to cause an exit break + + def close(self): self.reader.close() + self.t.join() class ConnectionReader(object):