From patchwork Sat Apr 16 22:28:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jose Quaresma X-Patchwork-Id: 6756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88701C6370A for ; Mon, 18 Apr 2022 14:25:59 +0000 (UTC) Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by mx.groups.io with SMTP id smtpd.web10.23816.1650148135658857277 for ; Sat, 16 Apr 2022 15:28:56 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=JnBZMv4X; spf=pass (domain: gmail.com, ip: 209.85.221.47, mailfrom: quaresma.jose@gmail.com) Received: by mail-wr1-f47.google.com with SMTP id u3so14545412wrg.3 for ; Sat, 16 Apr 2022 15:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=biHgQI1cqvV99gVVyj6UQ5Fo82KTb/qOM2TMPveBy6k=; b=JnBZMv4XnhIg3UZLY5VpVj3wiibH6j/KpSFw/XdIA+vlT7b7NIT0anVB9gKFNHdyh6 TVT+d6cGWwcJhJbTxnfU2eZEkaYIeSdz/YlymXoO9qhZ2hMzIpxNFVleeJJhM3KSSLP4 LK5DV89GXNIlSScFHgf8eAKTCtc37IgbWRWzcBqapzNsjYFlrTWnR/fBhyKD6oD8S4Oi g/PsyTRJSPH2PekYvnaebXuohRvc0Chbh5POl1vGWYnZM+SycqgL452mvPMG27p4ZuUl GTWstN6cMI8uJ0xIUAO3rPY5++tGUZtFS7/1FnXfPzcdJhrKgfImmRl5WLVCk7HLi50C DRQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=biHgQI1cqvV99gVVyj6UQ5Fo82KTb/qOM2TMPveBy6k=; b=qSCGmrUIN2sjZrNulOrR7HCVW2nioCMj1zQt+6XXo6SLWTpogzaXWYo9klz/cMNibO Y6Bznpw+449kk5IppSLnwdGyA2PW+hcLHmchMborhGXT7zAX4qXsx+A821T7WEfdCYkY kzrg4aPJxzmmjVliljoYOFxd6cESW3XLv8a9BIk6wvqamjjiRb9zH+2eATavU+A7qo09 fJI7P/sN2DIhHAfO26zkU7b7RLsgIHV0e0anXoC6kgknHukE0vzxZsFGB1+SVLEE20Gj Xr8zFbh/AE074e7SKEZO5GKkZHdqpNcCH5xplk6M6jigMiUrJF1Ke0k5EXH62+bc+PQO 8SeA== X-Gm-Message-State: AOAM531tRFeWe8GwI3W7OAIGP7+ipcJ4PQ3mUwCcSnJGBMpvA/8DA3l/ 6AYMm0ZVz3IX+Hjg+aSBjs0mYLbUMu/Vsw== X-Google-Smtp-Source: ABdhPJzuZqdEqwKOVF/kCAQDMjjBmdKaF4nwHTT2XDWYC8Wj8qBlvzwzrjHcjd1drZ4AidqaNiey4A== X-Received: by 2002:a5d:44d0:0:b0:207:9ac8:2c3b with SMTP id z16-20020a5d44d0000000b002079ac82c3bmr3525460wrr.688.1650148133707; Sat, 16 Apr 2022 15:28:53 -0700 (PDT) Received: from CTW-01195.lan (176.57.115.89.rev.vodafone.pt. [89.115.57.176]) by smtp.gmail.com with ESMTPSA id b1-20020a5d40c1000000b00207b49d3023sm6838104wrq.44.2022.04.16.15.28.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 16 Apr 2022 15:28:53 -0700 (PDT) From: Jose Quaresma To: openembedded-core@lists.openembedded.org Cc: Jose Quaresma Subject: [RFC][PATCH v2 1/4] sstate: add a LockedSet class as python set() is not thread safe Date: Sat, 16 Apr 2022 23:28:43 +0100 Message-Id: <20220416222846.219151-1-quaresma.jose@gmail.com> X-Mailer: git-send-email 2.35.3 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Apr 2022 14:25:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164563 With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool. This piece of code is taen from the stackoverflow https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 Signed-off-by: Jose Quaresma --- meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 1c0cae4893..a3ba748a1e 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -918,8 +918,28 @@ sstate_unpack_package () { BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): - found = set() - missed = set() + # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe + import threading + class LockedSet(set): + """A set where add(), remove(), and 'in' operator are thread-safe""" + def __init__(self, *args, **kwargs): + self._lock = threading.Lock() + super(LockedSet, self).__init__(*args, **kwargs) + + def add(self, elem): + with self._lock: + super(LockedSet, self).add(elem) + + def remove(self, elem): + with self._lock: + super(LockedSet, self).remove(elem) + + def __contains__(self, elem): + with self._lock: + super(LockedSet, self).__contains__(elem) + + found = LockedSet() + missed = LockedSet() def gethash(task): return sq_data['unihash'][task] From patchwork Sat Apr 16 22:28:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jose Quaresma X-Patchwork-Id: 6759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AA4DC63713 for ; Mon, 18 Apr 2022 14:25:59 +0000 (UTC) Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by mx.groups.io with SMTP id smtpd.web11.23561.1650148136499138250 for ; Sat, 16 Apr 2022 15:28:56 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=R7HTC3YZ; spf=pass (domain: gmail.com, ip: 209.85.221.47, mailfrom: quaresma.jose@gmail.com) Received: by mail-wr1-f47.google.com with SMTP id e21so14512742wrc.8 for ; Sat, 16 Apr 2022 15:28:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LJ7euGVKUlvcWmL/y+a/4fVq+QW+BN14fPp+4uqLQJQ=; b=R7HTC3YZqgf/LKeOLAvZ8ll7RT9ZQi/EuFwM1Hy9TaeieYWITNcVCqdaQAGg86LMaO 48PYSl2B2o6eDw3IhwMVbJNIObEcQUmsNll9MJrmSRfS9HmBQFU+gJ2I/Me2Bi0Tdz3c Qq61byqwvtGUpiAcwElMA06ptIKHKfhbC89hzpXUzXjL01sS6eYvpepHOJyELpMwUkPM azyRccPRsKC7IjBdDJOcQdAbyRi7Nt390lrxWnL6CyFeF0UBZUjHOw1OJtUdlc2YO+KX 21bo4ZCLkyhzR7qvuyNN5MXH3VNNBi3bOb3JUNaNP/FrVN3lXHwzD7+Mz81+TS5GO0mJ 5DbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LJ7euGVKUlvcWmL/y+a/4fVq+QW+BN14fPp+4uqLQJQ=; b=HnU/c/Gk2KU6uMjx234piMLG08ejm37TAMXcEuZMDNKRfHzq6AO8lhbCvYDeK9ZMdS Sv1GJ/DlppzV0qfwZMJHqYUHCulmLGmZke/Qppe95Okmz4sOJWqHwNdtsQAQ8be7fz/2 vRf6CsncjlkoavZBkGV/IlJmlNih8UiULgxoTiE5LvT/Yp3msKM1foP+o2FiOttfSnnK J91AN2APZmzQpgfOXbZnU853W6ksVo2JkxfRpTgjg2sG/nbL9ImTZR2YmsjvXmRjIf5S e98TcrwQ5CcVJqBRo2SckKUHoivwpbRIQY/+h4nYr8TU12UCSC0Jwt5b7J0DDbKOaH5S yQ3Q== X-Gm-Message-State: AOAM5307Xamr7aoh3CB2FkrN6YZNh06Nx0NFzOA+SOYrUEVUkKeyXQKa Q1H9bsOAhvIvm1g2Li1uLqQBbWjGL2xXhQ== X-Google-Smtp-Source: ABdhPJzYx7YG5GXuEwirAzf9iZXP3U6fcOrPTpYYIo+KjhdVLr/jSP1PLGI8sOHZx2q8Viyqdorkmw== X-Received: by 2002:a5d:588a:0:b0:204:1f46:cf08 with SMTP id n10-20020a5d588a000000b002041f46cf08mr3734446wrf.133.1650148134805; Sat, 16 Apr 2022 15:28:54 -0700 (PDT) Received: from CTW-01195.lan (176.57.115.89.rev.vodafone.pt. [89.115.57.176]) by smtp.gmail.com with ESMTPSA id b1-20020a5d40c1000000b00207b49d3023sm6838104wrq.44.2022.04.16.15.28.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 16 Apr 2022 15:28:54 -0700 (PDT) From: Jose Quaresma To: openembedded-core@lists.openembedded.org Cc: Jose Quaresma Subject: [RFC][PATCH v2 2/4] sstate: store the tasklist in a LockedSet Date: Sat, 16 Apr 2022 23:28:44 +0100 Message-Id: <20220416222846.219151-2-quaresma.jose@gmail.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220416222846.219151-1-quaresma.jose@gmail.com> References: <20220416222846.219151-1-quaresma.jose@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Apr 2022 14:25:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164564 With this aproach the progress bar will be independent of the ThreadedPool. Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 Signed-off-by: Jose Quaresma --- meta/classes/sstate.bbclass | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index a3ba748a1e..b9e9ef8e26 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -1026,29 +1026,31 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, bb.error("SState: cannot test %s: %s\n%s" % (srcuri, repr(e), traceback.format_exc())) if progress: - bb.event.fire(bb.event.ProcessProgress(msg, len(tasklist) - thread_worker.tasks.qsize()), d) + tasks.remove(arg) + bb.event.fire(bb.event.ProcessProgress(msg, total_tasks - len(tasks)), d) - tasklist = [] + tasks = LockedSet() for tid in missed: sstatefile = d.expand(getsstatefile(tid, siginfo, d)) - tasklist.append((tid, sstatefile)) + tasks.add((tid, sstatefile)) - if tasklist: - nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist)) + total_tasks = len(tasks) + if total_tasks: + nproc = min(int(d.getVar("BB_NUMBER_THREADS")), total_tasks) - progress = len(tasklist) >= 100 + progress = total_tasks >= 100 if progress: msg = "Checking sstate mirror object availability" - bb.event.fire(bb.event.ProcessStarted(msg, len(tasklist)), d) + bb.event.fire(bb.event.ProcessStarted(msg, total_tasks), d) # Have to setup the fetcher environment here rather than in each thread as it would race fetcherenv = bb.fetch2.get_fetcher_environment(d) with bb.utils.environment(**fetcherenv): bb.event.enable_threadlock() - pool = oe.utils.ThreadedPool(nproc, len(tasklist), + pool = oe.utils.ThreadedPool(nproc, total_tasks, worker_init=checkstatus_init, worker_end=checkstatus_end, name="sstate_checkhashes-") - for t in tasklist: + for t in tasks: pool.add_task(checkstatus, t) pool.start() pool.wait_completion() From patchwork Sat Apr 16 22:28:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jose Quaresma X-Patchwork-Id: 6758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99F22C6370E for ; Mon, 18 Apr 2022 14:25:59 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web12.23463.1650148137302122126 for ; Sat, 16 Apr 2022 15:28:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ionIfrjY; spf=pass (domain: gmail.com, ip: 209.85.221.46, mailfrom: quaresma.jose@gmail.com) Received: by mail-wr1-f46.google.com with SMTP id c10so14556391wrb.1 for ; Sat, 16 Apr 2022 15:28:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZTI0HiRvnsvl0GxnZXzbb2yHO7cI7d4IaUoxqmlGLmk=; b=ionIfrjYMUQC9vxGhXzXAaHxV5a0WOCgUu+gYFYShQB0Nyrw4YI0W73JCnOZfybZxF cyBo5ES09ejKbcl2VtNnvTvfwFjkMcPGx8/xTbaXeSQcPv7P2Wz1jPGS7g9KA1l6wNVD zO0x0O9JciKIHb3guBjBAD3RwB60JNlIx/Efh1BKrW485S3etl1MP0TtxdmEXpzcWog0 5nWYpQAGgZn6h4htOgrntWDe824zL22KYCwvpfRx9KrPmdLnKBNPD6vZ/Tq3jrIgUBV6 L9+n8CxSyX1O+G5gVZllPSs3XGPnuPXRup/cnhgQvLSDPMk06Uz0zhETEhG/7lPtXR5j 70FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZTI0HiRvnsvl0GxnZXzbb2yHO7cI7d4IaUoxqmlGLmk=; b=dv3BxdX/WrJBEG8nBWO3Gl0XPc3UhHCIrHaeSYJSuNGSGYSTb11uRoImLBOs4b+fuP K8LhwS4StB6k50YfCPoXkCQMfhS/OBBLC5BMNJxAyRRKn1fPRJhkMR5n8TPPkEECvKgP C6LKSYul2jKBrLkBH2g1TE1u1uaYUf4m7ti8Ek/KuAWLTuQY2HxfdbfRJQAM7te3xMj7 U9OxShjyGEpfiXS1zQbkJ+vmuatT37W324brYD4ujcLmUNfTLY19azcoejmN/see1ylG +oGxzlcCb49xaqoKZeb4ViQurAEtN4rI6z0zPUygq6ASGkZKs3CHtHuYDdgQkAULrtI9 a6ew== X-Gm-Message-State: AOAM533NunNH/dfLA15InEzFgNodeQ29WdehF87n5UwtC57XCYAUhmLt wgjlJ1HTlsSMYxi1DoOL2uqE84ve9zU3hQ== X-Google-Smtp-Source: ABdhPJzuwNZyzZvIz23V2cftbSL+dBUJWQdi2GNG9az7brNy6yu/vze1UdYdFpYNw8hNw114PrspFg== X-Received: by 2002:a05:6000:1f11:b0:207:a62d:bb32 with SMTP id bv17-20020a0560001f1100b00207a62dbb32mr3474936wrb.386.1650148135697; Sat, 16 Apr 2022 15:28:55 -0700 (PDT) Received: from CTW-01195.lan (176.57.115.89.rev.vodafone.pt. [89.115.57.176]) by smtp.gmail.com with ESMTPSA id b1-20020a5d40c1000000b00207b49d3023sm6838104wrq.44.2022.04.16.15.28.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 16 Apr 2022 15:28:55 -0700 (PDT) From: Jose Quaresma To: openembedded-core@lists.openembedded.org Cc: Jose Quaresma Subject: [RFC][PATCH v2 3/4] sstate: use the python3 ThreadPoolExecutor instead of the OE ThreadedPool Date: Sat, 16 Apr 2022 23:28:45 +0100 Message-Id: <20220416222846.219151-3-quaresma.jose@gmail.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220416222846.219151-1-quaresma.jose@gmail.com> References: <20220416222846.219151-1-quaresma.jose@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Apr 2022 14:25:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164565 for the FetchConnectionCache use a queue where each thread can get an unsed connection_cache that is properly initialized before we fireup the ThreadPoolExecutor. for the progress bar we need an adictional task counter that is protected with thread lock as it runs inside the ThreadPoolExecutor. Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 Signed-off-by: Jose Quaresma --- meta/classes/sstate.bbclass | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index b9e9ef8e26..7a18d9d554 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -997,15 +997,19 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, localdata.delVar('BB_NO_NETWORK') from bb.fetch2 import FetchConnectionCache - def checkstatus_init(thread_worker): - thread_worker.connection_cache = FetchConnectionCache() + def checkstatus_init(): + while not connection_cache_pool.full(): + connection_cache_pool.put(FetchConnectionCache()) - def checkstatus_end(thread_worker): - thread_worker.connection_cache.close_connections() + def checkstatus_end(): + while not connection_cache_pool.empty(): + connection_cache = connection_cache_pool.get() + connection_cache.close_connections() - def checkstatus(thread_worker, arg): + def checkstatus(arg): (tid, sstatefile) = arg + connection_cache = connection_cache_pool.get() localdata2 = bb.data.createCopy(localdata) srcuri = "file://" + sstatefile localdata2.setVar('SRC_URI', srcuri) @@ -1015,7 +1019,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, try: fetcher = bb.fetch2.Fetch(srcuri.split(), localdata2, - connection_cache=thread_worker.connection_cache) + connection_cache=connection_cache) fetcher.checkstatus() bb.debug(2, "SState: Successful fetch test for %s" % srcuri) found.add(tid) @@ -1025,6 +1029,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, except Exception as e: bb.error("SState: cannot test %s: %s\n%s" % (srcuri, repr(e), traceback.format_exc())) + connection_cache_pool.put(connection_cache) + if progress: tasks.remove(arg) bb.event.fire(bb.event.ProcessProgress(msg, total_tasks - len(tasks)), d) @@ -1047,13 +1053,13 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, fetcherenv = bb.fetch2.get_fetcher_environment(d) with bb.utils.environment(**fetcherenv): bb.event.enable_threadlock() - pool = oe.utils.ThreadedPool(nproc, total_tasks, - worker_init=checkstatus_init, worker_end=checkstatus_end, - name="sstate_checkhashes-") - for t in tasks: - pool.add_task(checkstatus, t) - pool.start() - pool.wait_completion() + import concurrent.futures + from queue import Queue + connection_cache_pool = Queue(nproc) + checkstatus_init() + with concurrent.futures.ThreadPoolExecutor(max_workers=nproc) as executor: + executor.map(checkstatus, tasks) + checkstatus_end() bb.event.disable_threadlock() if progress: From patchwork Sat Apr 16 22:28:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jose Quaresma X-Patchwork-Id: 6761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BC0BC63714 for ; Mon, 18 Apr 2022 14:25:59 +0000 (UTC) Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by mx.groups.io with SMTP id smtpd.web09.23369.1650148138283937114 for ; Sat, 16 Apr 2022 15:28:58 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=h8N27Hlq; spf=pass (domain: gmail.com, ip: 209.85.221.49, mailfrom: quaresma.jose@gmail.com) Received: by mail-wr1-f49.google.com with SMTP id w4so14500377wrg.12 for ; Sat, 16 Apr 2022 15:28:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=odps9OMZoa+CuQMJ13hLPdm0u5Q0DfFHZMk3XMsI8rs=; b=h8N27HlqjvtbCij7k9QekLEyn6T3DEZ1ZbagXvk/Qqgsb0HhNvJZDn/hFMleamtiAu gR8yELXvt4+pZPYVQpFluUM5bGVJIhy0Jpu+cWxe25rw/gc5pbjBsRbpnTisI7Mqp6fL mZ1gy05JebVTKBV7uDVYDeYKyQfJ5qww8k7N9cKdWTXCEVzh+JyeZaBiAV610ocVLZOH qMNb69uepkb/sQH0nvGyfyufR22bR+a780j3q8bOC49psF6Eh3HnSZcVL8qRKEkHWzt4 ISNtF8mrJCNb5/P4RDX6llTZGpYfIUmef4BTz6TKjFkIbcvA31g0v46Pkfmwh7oXD8/e aocQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=odps9OMZoa+CuQMJ13hLPdm0u5Q0DfFHZMk3XMsI8rs=; b=ZcRTY3rg9xSZ83SHiOQPWtzfW2gsDpnCVdSLk4pbGv0cvSB/S7fXrAe9wc8TpD+/iI X0eXwUhhwdJHNXmasizdvIXRGrUjTzsu59rmNn8z7QMOo0IIdUrUoK7LVeiK3IqplHEa lUO44/IpFZTf2Zmw1szOtps8i4x9vTAxMMSb2r0gKaxTaWbS8fgdx2E4uXXJwinSYu4b DGrms6SnjoiWWOO19G57LmUesJCzdkL+vgm4NPEdM2iYqR5+uQPEd3TGn6u6vAJ4wjka M/cXd8R6FSTS+NIwipgt1VzuCqFZX3g02GuB0jPYN8gP+kCbDRI0KIWooMJbXD0mHKB9 43VQ== X-Gm-Message-State: AOAM532oHxMgDwb5qKLdaByMzBB0axR5gpvm+QgYM/pBveF+8T1eXWBb lXjChCBXWQakZjthBwced4kvlZJOD5rn+A== X-Google-Smtp-Source: ABdhPJyuZA8S7/KWP285rguq6Idct17O3HgDaWkOgHy8qCxstVM2zTRB9oPUk8r0NgATpKi8WdpRQQ== X-Received: by 2002:a5d:4483:0:b0:20a:92b2:ff81 with SMTP id j3-20020a5d4483000000b0020a92b2ff81mr145953wrq.549.1650148136658; Sat, 16 Apr 2022 15:28:56 -0700 (PDT) Received: from CTW-01195.lan (176.57.115.89.rev.vodafone.pt. [89.115.57.176]) by smtp.gmail.com with ESMTPSA id b1-20020a5d40c1000000b00207b49d3023sm6838104wrq.44.2022.04.16.15.28.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 16 Apr 2022 15:28:56 -0700 (PDT) From: Jose Quaresma To: openembedded-core@lists.openembedded.org Cc: Jose Quaresma Subject: [RFC][PATCH v2 4/4] oe/utils: remove the ThreadedPool Date: Sat, 16 Apr 2022 23:28:46 +0100 Message-Id: <20220416222846.219151-4-quaresma.jose@gmail.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220416222846.219151-1-quaresma.jose@gmail.com> References: <20220416222846.219151-1-quaresma.jose@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 18 Apr 2022 14:25:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164566 The ThreadedPool in OE-core is mainly because python2 doesn't have threaded pools but python2 is dead for some time now and python3 have a ThreadPoolExecutor. The only local in OE-core where this ThreadedPool is in use is on the sstate.bbclass that is ported to the python3 ThreadPoolExecutor. Signed-off-by: Jose Quaresma --- meta/lib/oe/utils.py | 64 -------------------------------------------- 1 file changed, 64 deletions(-) diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index 46fc76c261..1ee947d584 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -473,70 +473,6 @@ def get_multilib_datastore(variant, d): localdata.setVar("MLPREFIX", "") return localdata -# -# Python 2.7 doesn't have threaded pools (just multiprocessing) -# so implement a version here -# - -from queue import Queue -from threading import Thread - -class ThreadedWorker(Thread): - """Thread executing tasks from a given tasks queue""" - def __init__(self, tasks, worker_init, worker_end, name=None): - Thread.__init__(self, name=name) - self.tasks = tasks - self.daemon = True - - self.worker_init = worker_init - self.worker_end = worker_end - - def run(self): - from queue import Empty - - if self.worker_init is not None: - self.worker_init(self) - - while True: - try: - func, args, kargs = self.tasks.get(block=False) - except Empty: - if self.worker_end is not None: - self.worker_end(self) - break - - try: - func(self, *args, **kargs) - except Exception as e: - # Eat all exceptions - bb.mainlogger.debug("Worker task raised %s" % e, exc_info=e) - finally: - self.tasks.task_done() - -class ThreadedPool: - """Pool of threads consuming tasks from a queue""" - def __init__(self, num_workers, num_tasks, worker_init=None, worker_end=None, name="ThreadedPool-"): - self.tasks = Queue(num_tasks) - self.workers = [] - - for i in range(num_workers): - worker = ThreadedWorker(self.tasks, worker_init, worker_end, name=name + str(i)) - self.workers.append(worker) - - def start(self): - for worker in self.workers: - worker.start() - - def add_task(self, func, *args, **kargs): - """Add a task to the queue""" - self.tasks.put((func, args, kargs)) - - def wait_completion(self): - """Wait for completion of all the tasks in the queue""" - self.tasks.join() - for worker in self.workers: - worker.join() - class ImageQAFailed(Exception): def __init__(self, description, name=None, logfile=None): self.description = description