From patchwork Wed Nov 24 17:15:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ED2BC433EF for ; Wed, 24 Nov 2021 17:15:33 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web09.950.1637774131808146874 for ; Wed, 24 Nov 2021 09:15:32 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 753AE1042 for ; Wed, 24 Nov 2021 09:15:31 -0800 (PST) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1D2283F66F for ; Wed, 24 Nov 2021 09:15:31 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/3] oe/utils: allow naming threads in ThreadedPool Date: Wed, 24 Nov 2021 17:15:27 +0000 Message-Id: <20211124171529.4107434-1-ross.burton@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 24 Nov 2021 17:15:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/158729 When looking at logs involving thread pools it is useful if the threads can be named. Signed-off-by: Ross Burton --- meta/lib/oe/utils.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index cf65639647..7982b2b511 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -483,8 +483,8 @@ from threading import Thread class ThreadedWorker(Thread): """Thread executing tasks from a given tasks queue""" - def __init__(self, tasks, worker_init, worker_end): - Thread.__init__(self) + def __init__(self, tasks, worker_init, worker_end, name=None): + Thread.__init__(self, name=name) self.tasks = tasks self.daemon = True @@ -515,13 +515,12 @@ class ThreadedWorker(Thread): class ThreadedPool: """Pool of threads consuming tasks from a queue""" - def __init__(self, num_workers, num_tasks, worker_init=None, - worker_end=None): + def __init__(self, num_workers, num_tasks, worker_init=None, worker_end=None, name="ThreadedPool-"): self.tasks = Queue(num_tasks) self.workers = [] - for _ in range(num_workers): - worker = ThreadedWorker(self.tasks, worker_init, worker_end) + for i in range(num_workers): + worker = ThreadedWorker(self.tasks, worker_init, worker_end, name=name + str(i)) self.workers.append(worker) def start(self): From patchwork Wed Nov 24 17:15:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DD49C433F5 for ; Wed, 24 Nov 2021 17:15:33 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web12.928.1637774132431514926 for ; Wed, 24 Nov 2021 09:15:32 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0836611FB for ; Wed, 24 Nov 2021 09:15:32 -0800 (PST) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A693B3F66F for ; Wed, 24 Nov 2021 09:15:31 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/3] sstate: explicitly name the checkhashes thread pool Date: Wed, 24 Nov 2021 17:15:28 +0000 Message-Id: <20211124171529.4107434-2-ross.burton@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211124171529.4107434-1-ross.burton@arm.com> References: <20211124171529.4107434-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 24 Nov 2021 17:15:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/158730 Give this thread pool a name for clarity. Signed-off-by: Ross Burton --- meta/classes/sstate.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 0a78a2fb69..0326d27c74 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -1014,7 +1014,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, bb.event.enable_threadlock() pool = oe.utils.ThreadedPool(nproc, len(tasklist), - worker_init=checkstatus_init, worker_end=checkstatus_end) + worker_init=checkstatus_init, worker_end=checkstatus_end, + name="sstate_checkhashes-") for t in tasklist: pool.add_task(checkstatus, t) pool.start() From patchwork Wed Nov 24 17:15:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 394F7C433FE for ; Wed, 24 Nov 2021 17:15:34 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.941.1637774133189386143 for ; Wed, 24 Nov 2021 09:15:33 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9453F1FB for ; Wed, 24 Nov 2021 09:15:32 -0800 (PST) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3C05C3F66F for ; Wed, 24 Nov 2021 09:15:32 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/3] oe/utils: by default cap cpu_count() to 64 cores Date: Wed, 24 Nov 2021 17:15:29 +0000 Message-Id: <20211124171529.4107434-3-ross.burton@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211124171529.4107434-1-ross.burton@arm.com> References: <20211124171529.4107434-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 24 Nov 2021 17:15:34 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/158731 Larger systems may have large numbers of cores, but beyond a certain point they can't all be used for compiling: whilst purely compute-intensive jobs can be parallelised to hundreds of cores, operations such as compressing (needs lots of RAM) or compiling (lots of I/O) don't scale linearly. For example, the Marvel ThunderX2 has 32 cores, each capable of executing four threads, and can be configured with two sockets, making 256 CPUs according to Linux. Zstd using 256 threads has been seen to fail to allocate memory during even small recipes such as iso-codes. Add a default cap of 64 CPUs to the cpu_count() method so that extreme parallisation is limited. 64 is high enough that meaningful gains beyond it are unlikely, but high enough that most systems won't be effected. Signed-off-by: Ross Burton --- meta/lib/oe/utils.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index 7982b2b511..136650e6f7 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -248,9 +248,9 @@ def trim_version(version, num_parts=2): trimmed = ".".join(parts[:num_parts]) return trimmed -def cpu_count(at_least=1): +def cpu_count(at_least=1, at_most=64): cpus = len(os.sched_getaffinity(0)) - return max(cpus, at_least) + return max(min(cpus, at_most), at_least) def execute_pre_post_process(d, cmds): if cmds is None: