From patchwork Mon Aug 28 12:48:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Martin_Hundeb=C3=B8ll?= X-Patchwork-Id: 29611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39392C83F17 for ; Mon, 28 Aug 2023 12:48:59 +0000 (UTC) Received: from www530.your-server.de (www530.your-server.de [188.40.30.78]) by mx.groups.io with SMTP id smtpd.web11.12527.1693226929044283058 for ; Mon, 28 Aug 2023 05:48:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@geanix.com header.s=default2211 header.b=d5L23ERb; spf=pass (domain: geanix.com, ip: 188.40.30.78, mailfrom: martin@geanix.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=geanix.com; s=default2211; h=Content-Transfer-Encoding:Content-Type:MIME-Version: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References; bh=ZbuD7SqOLzdiVy4UZZsIZjpcepKOafwWTWSO7KBvU8E=; b=d5L23ERbknTc11LI792YcxgAhi SMpJGuTU4wy0ese/g7/FcCdE7fELoxRKo27OoDeWmthXYT1ZSh3/uC1P90sImJZUED9Q52cKuaWXs J3ueyxwzrc8TuS6MVcvk9pgvkudBvMKksgf1K+IhmtZWhg5uYkQPAyB0H3BUJegm37xmDsm4F2rCV IPoz6PeXNgLTecjX0MoinYcguY8oqlIZc3CVTWxAM3tqJDN0DehixL0f/T3WgJB8wAGjLnCgTGiGD 6kRb7hj5txGa01yzdXgtguCVZqMf7UooFg00Q16LKtoHFjBgyiBxaEdOQe0re1IqqqKdt9ojtK+h5 9lsAyONQ==; Received: from sslproxy03.your-server.de ([88.198.220.132]) by www530.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qabfi-0009GN-8v; Mon, 28 Aug 2023 14:48:46 +0200 Received: from [185.17.218.86] (helo=rap..) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qabfh-000BNd-So; Mon, 28 Aug 2023 14:48:45 +0200 From: =?utf-8?q?Martin_Hundeb=C3=B8ll?= To: openembedded-core@lists.openembedded.org Cc: =?utf-8?q?Martin_Hundeb=C3=B8ll?= Subject: [RFC PATCH 1/5] classes: jobserver: support gnu make fifo jobserver Date: Mon, 28 Aug 2023 14:48:25 +0200 Message-ID: <20230828124834.376779-1-martin@geanix.com> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Authenticated-Sender: martin@geanix.com X-Virus-Scanned: Clear (ClamAV 0.103.8/27014/Mon Aug 28 09:38:26 2023) List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Aug 2023 12:48:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/186824 Add a class to implement the gnu make fifo style jobserver. The class can be activated by symply adding an `INHERIT += "jobserver"` to the local configuration. Furthermore, one can configure an external jobserver (i.e. a server shared between multiple builds), by configured the `JOBSERVER_FIFO` variable to point at an existing jobserver fifo. The jobserver class uses the fifo style jobserver, which doesn't require passing open file descriptors around. It does, however, require make-4.4, which isn't available in common distro yet. To work around this, the class make all recipes (except make and its dependencies itself) depend on `virtual/make-native`. Signed-off-by: Martin Hundebøll --- meta/classes-global/jobserver.bbclass | 80 +++++++++++++++++++++++++++ meta/conf/bitbake.conf | 2 +- 2 files changed, 81 insertions(+), 1 deletion(-) create mode 100644 meta/classes-global/jobserver.bbclass diff --git a/meta/classes-global/jobserver.bbclass b/meta/classes-global/jobserver.bbclass new file mode 100644 index 0000000000..c76909fe50 --- /dev/null +++ b/meta/classes-global/jobserver.bbclass @@ -0,0 +1,80 @@ +JOBSERVER_FIFO ?= "" +JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of creating a per-build server." + +addhandler jobserver_setup_fifo +jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed" + +python jobserver_setup_fifo() { + # don't setup a per-build fifo, if an external one is configured + if d.getVar("JOBSERVER_FIFO"): + return + + # don't use a job-server if no parallelism is configured + jobs = oe.utils.parallel_make(d) + if jobs in (None, 1): + return + + # reduce jobs by one as a token has implicitly been handed to the + # process requesting tokens + jobs -= 1 + + fifo = d.getVar("TMPDIR") + "/jobserver_fifo" + + # and old fifo might be lingering; remove it + if os.path.exists(fifo): + os.remove(fifo) + + # create a new fifo to use for communicating tokens + os.mkfifo(fifo) + + # fill the fifo with the number of tokens to hand out + wfd = os.open(fifo, os.O_RDWR) + written = os.write(wfd, b"+" * jobs) + if written != (jobs): + bb.error("Failed to fil make fifo: {} != {}".format(written, jobs)) + + # configure the per-build fifo path to use + d.setVar("JOBSERVER_FIFO", fifo) +} + +python () { + # don't configure the fifo if none is defined + fifo = d.getVar("JOBSERVER_FIFO") + if not fifo: + return + + # avoid making make-native or its dependencies depend on make-native itself + if d.getVar("PN") in ( + "make-native", + "libtool-native", + "pkgconfig-native", + "automake-native", + "autoconf-native", + "m4-native", + "texinfo-dummy-native", + "gettext-minimal-native", + "quilt-native", + "gnu-config-native", + ): + return + + # don't make unwilling recipes depend on make-native + if d.getVar('INHIBIT_DEFAULT_DEPS', False): + return + + # make other recipes depend on make-native to make sure it is new enough to + # support the --jobserver-auth=fifo: syntax (from make-4.4 and onwards) + d.appendVar("DEPENDS", " virtual/make-native") + + # disable the "-j " flag, as that overrides the jobserver fifo tokens + d.setVar("PARALLEL_MAKE", "") + d.setVar("PARALLEL_MAKEINST", "") + + # set and export the jobserver in the environment + d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo) + d.setVarFlag("MAKEFLAGS", "export", "1") + + # ignore the joberserver argument part of MAKEFLAGS in the hash, as that + # shouldn't change the build output + d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "| --jobserver-auth=fifo:" + fifo) +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index cf7ff3328c..8cf188270b 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -946,7 +946,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ - OMP_NUM_THREADS BB_CURRENTTASK" + OMP_NUM_THREADS BB_CURRENTTASK JOBSERVER_FIFO" BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS" BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \ From patchwork Mon Aug 28 12:48:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Martin_Hundeb=C3=B8ll?= X-Patchwork-Id: 29609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FF16C71153 for ; Mon, 28 Aug 2023 12:48:59 +0000 (UTC) Received: from www530.your-server.de (www530.your-server.de [188.40.30.78]) by mx.groups.io with SMTP id smtpd.web11.12525.1693226928442329477 for ; Mon, 28 Aug 2023 05:48:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@geanix.com header.s=default2211 header.b=YYcIOznq; spf=pass (domain: geanix.com, ip: 188.40.30.78, mailfrom: martin@geanix.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=geanix.com; s=default2211; h=Content-Transfer-Encoding:Content-Type:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID; bh=WMKH/LeQFSNR+brCBRxwqemAIo7bBhkLmujW3OZ0N8Q=; b=YYcIOznqGVrMK37hmbCGJ/TPuZ 0J9kG7fKsJ0XUxY6obOCk1atd0/W0fvcNgd09EfPj8Co+FXduoZK7OttncX+OQe1nkJR2jPXLQ/pc 3VLPQDn4O1K7C+ufSq8c31wVye2onBEIT8Ka55PNtjHOcpcgGSru7Dj6r/jQi4kfp9TAFA/hv1paZ rIygp94FT6oxVwNTRfJllI0HgGJDl+b67fnMQVtBW1zWCjLxYGZ2+BhTugC1Bt0oswIljNW116SMH 5nJGxcUXm68lpiS/TxWMilf+258yw0klckNIhRkLlANDICGGRFKCbDQHiSTr03R4fMDwg4pxCFc2b 7tWi9clA==; Received: from sslproxy03.your-server.de ([88.198.220.132]) by www530.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qabfi-0009GP-Dv; Mon, 28 Aug 2023 14:48:46 +0200 Received: from [185.17.218.86] (helo=rap..) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qabfh-000BNd-WE; Mon, 28 Aug 2023 14:48:46 +0200 From: =?utf-8?q?Martin_Hundeb=C3=B8ll?= To: openembedded-core@lists.openembedded.org Cc: =?utf-8?q?Martin_Hundeb=C3=B8ll?= Subject: [RFC PATCH 2/5] scripts: build-env: allow passing JOBSERVER_FIFO from environment Date: Mon, 28 Aug 2023 14:48:26 +0200 Message-ID: <20230828124834.376779-2-martin@geanix.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230828124834.376779-1-martin@geanix.com> References: <20230828124834.376779-1-martin@geanix.com> MIME-Version: 1.0 X-Authenticated-Sender: martin@geanix.com X-Virus-Scanned: Clear (ClamAV 0.103.8/27014/Mon Aug 28 09:38:26 2023) List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Aug 2023 12:48:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/186823 Sharing a common jobserver fifo between multiple (containerized) builds is much easier, if an administrator can configure said jobserver fifo path in the environment. Append the JOBSERVER_FIFO variable name to the list of variables configurable through the environment. Signed-off-by: Martin Hundebøll --- scripts/oe-buildenv-internal | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal index 2fdb19565a..c8e67ffb8f 100755 --- a/scripts/oe-buildenv-internal +++ b/scripts/oe-buildenv-internal @@ -112,7 +112,7 @@ HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY \ all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \ SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND \ SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA BB_SETSCENE_ENFORCE \ -BB_LOGCONFIG" +BB_LOGCONFIG JOBSERVER_FIFO" BB_ENV_PASSTHROUGH_ADDITIONS="$(echo $BB_ENV_PASSTHROUGH_ADDITIONS $BB_ENV_PASSTHROUGH_ADDITIONS_OE | tr ' ' '\n' | LC_ALL=C sort --unique | tr '\n' ' ')" From patchwork Mon Aug 28 12:48:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Martin_Hundeb=C3=B8ll?= X-Patchwork-Id: 29610 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 396E1C83F18 for ; Mon, 28 Aug 2023 12:48:59 +0000 (UTC) Received: from www530.your-server.de (www530.your-server.de [188.40.30.78]) by mx.groups.io with SMTP id smtpd.web11.12526.1693226928445806681 for ; Mon, 28 Aug 2023 05:48:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@geanix.com header.s=default2211 header.b=N0H8+qAQ; spf=pass (domain: geanix.com, ip: 188.40.30.78, mailfrom: martin@geanix.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=geanix.com; s=default2211; h=Content-Transfer-Encoding:Content-Type:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID; bh=gS71PDMljRsaLXsXf+5y7YCOtc9jQJJnpMPaMlCx5os=; b=N0H8+qAQtL5L2ICs8eQVCJBaQT wLB7MzzRCg3zTHE/0sKTj7m8SSwPmg+VhBNHzeV9qxFrbQ/3EZ+b1UQNi2H+HQA8/TPQ4DFOKil2t jqeUFO8SNG9QdePJH/hEO708yGUvMbnu3DbbojY1Ra+E+HXcEZlgLs2NB68YYhDIXznAdMs97HZ5i 740CMrE6sQKuV/r7iENxV6mp+gy02ZYmo9JHlTAMuI18HOrWDkJVD9okb2C8S40j/8CWuxLkgJ5FL tt5Wi4NlltFRGqUoatq3V2PDiU2kGAJBQw/QMDya1oknrz07CPTHVREpsjFkU6CwrQdjSYou8ns+S Y6hMCaPQ==; Received: from sslproxy03.your-server.de ([88.198.220.132]) by www530.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qabfi-0009GS-Gs; Mon, 28 Aug 2023 14:48:46 +0200 Received: from [185.17.218.86] (helo=rap..) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qabfi-000BNd-3Q; Mon, 28 Aug 2023 14:48:46 +0200 From: =?utf-8?q?Martin_Hundeb=C3=B8ll?= To: openembedded-core@lists.openembedded.org Cc: =?utf-8?q?Martin_Hundeb=C3=B8ll?= Subject: [RFC PATCH 3/5] ninja: build modified version with GNU Make jobserver support Date: Mon, 28 Aug 2023 14:48:27 +0200 Message-ID: <20230828124834.376779-3-martin@geanix.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230828124834.376779-1-martin@geanix.com> References: <20230828124834.376779-1-martin@geanix.com> MIME-Version: 1.0 X-Authenticated-Sender: martin@geanix.com X-Virus-Scanned: Clear (ClamAV 0.103.8/27014/Mon Aug 28 09:38:26 2023) List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Aug 2023 12:48:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/186825 Ninja doesn't (yet) support the GNU Make jobserver out of the box, but there is a pull request adding that support[1]. Switch the SRC_URI and SRCREV to point at the source of that pull request, to make ninja play nicely together with the recently added jobserver class. Signed-off-by: Martin Hundebøll --- .../ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename meta/recipes-devtools/ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb} (84%) diff --git a/meta/recipes-devtools/ninja/ninja_1.11.1.bb b/meta/recipes-devtools/ninja/ninja_1.12.0.bb similarity index 84% rename from meta/recipes-devtools/ninja/ninja_1.11.1.bb rename to meta/recipes-devtools/ninja/ninja_1.12.0.bb index 8e297ec4d4..9abdd40a92 100644 --- a/meta/recipes-devtools/ninja/ninja_1.11.1.bb +++ b/meta/recipes-devtools/ninja/ninja_1.12.0.bb @@ -6,9 +6,9 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=a81586a64ad4e476c791cda7e2f2c52e" DEPENDS = "re2c-native ninja-native" -SRCREV = "a524bf3f6bacd1b4ad85d719eed2737d8562f27a" +SRCREV = "c9e21dbbc4c746ba397c0f9bec5f65c99f783c08" -SRC_URI = "git://github.com/ninja-build/ninja.git;branch=release;protocol=https" +SRC_URI = "git://github.com/stefanb2/ninja.git;branch=topic-issue-1139-part-3-jobserver-fifo;protocol=https" UPSTREAM_CHECK_GITTAGREGEX = "v(?P.*)" S = "${WORKDIR}/git" From patchwork Mon Aug 28 12:48:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Martin_Hundeb=C3=B8ll?= X-Patchwork-Id: 29608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E63DC83F15 for ; Mon, 28 Aug 2023 12:48:59 +0000 (UTC) Received: from www530.your-server.de (www530.your-server.de [188.40.30.78]) by mx.groups.io with SMTP id smtpd.web10.12634.1693226928781548746 for ; Mon, 28 Aug 2023 05:48:50 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@geanix.com header.s=default2211 header.b=GdfYuW9E; spf=pass (domain: geanix.com, ip: 188.40.30.78, mailfrom: martin@geanix.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=geanix.com; s=default2211; h=Content-Transfer-Encoding:Content-Type:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID; bh=uT3sgcTfsX+QhIqTGvwok3EOS0ZwcOUEPeBrH9OLFmU=; b=GdfYuW9E+Wz1RKWsOGyrgzNVCP 2luaFugjWnVLT6NVQ9xjLwk+OCQ2zKJ4zslRAsWSQt2saTCglIK5VvqgCMRtAnvvPCCO7ODhYRm8E 2+xogVsJ7QSdYoIaLUDOCRekSkBhWg3wBkNQYkNIkMbuu5ZUT5Yf2mGwZ/6L1GeyjeFMXaPQQmplM bh2opxi9NF4i2oHstkxTAsxRVVnBtv+UimllmOdxmOUkK1NsBN+bTwm5jO9KJflABwBd47DVrHcPq imS5AfJN9zAQukxOQIw7t1cjjNam5gzoViQa3pwuFFrqyaTlNlQBcDgcbGxBenmknD4xTiO7g5zBC GBloEYHQ==; Received: from sslproxy03.your-server.de ([88.198.220.132]) by www530.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qabfi-0009Gc-MD; Mon, 28 Aug 2023 14:48:46 +0200 Received: from [185.17.218.86] (helo=rap..) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qabfi-000BNd-6x; Mon, 28 Aug 2023 14:48:46 +0200 From: =?utf-8?q?Martin_Hundeb=C3=B8ll?= To: openembedded-core@lists.openembedded.org Cc: =?utf-8?q?Martin_Hundeb=C3=B8ll?= Subject: [RFC PATCH 4/5] qemu: enable parallel builds when using the jobserver class Date: Mon, 28 Aug 2023 14:48:28 +0200 Message-ID: <20230828124834.376779-4-martin@geanix.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230828124834.376779-1-martin@geanix.com> References: <20230828124834.376779-1-martin@geanix.com> MIME-Version: 1.0 X-Authenticated-Sender: martin@geanix.com X-Virus-Scanned: Clear (ClamAV 0.103.8/27014/Mon Aug 28 09:38:26 2023) List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Aug 2023 12:48:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/186827 If the jobserver class is enabled, the PARALLEL_MAKE variable is unset in favor of configuring a shared jobserver in the MAKEFLAGS variable. However, the qemu makefile translates the missing `-j` argument to `-j1` when calling into meson / ninja. Avoid this by setting `-j` without a value. For normal/GNU make, this can result in a fork bomb, but for ninja, it simply makes it use the jobserver fifa instead. Signed-off-by: Martin Hundebøll --- meta/recipes-devtools/qemu/qemu.inc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc index ea02bf0c73..7da05fcbf4 100644 --- a/meta/recipes-devtools/qemu/qemu.inc +++ b/meta/recipes-devtools/qemu/qemu.inc @@ -110,6 +110,11 @@ EXTRA_OECONF = " \ ${PACKAGECONFIG_CONFARGS} \ " +# Avoid the qemu makefile treating a missing `-j` argument as `-j1` when +# calling meson / ninja. This happens when the `jobserver` class is used, since +# it manages parallelism in the MAKEFLAGS variable instead of PARALLEL_MAKE. +EXTRA_OEMAKE:append = "${@' -j' if d.getVar('JOBSERVER_FIFO') else ''}" + B = "${WORKDIR}/build" #EXTRA_OECONF:append = " --python=${HOSTTOOLS_DIR}/python3" From patchwork Mon Aug 28 12:48:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Martin_Hundeb=C3=B8ll?= X-Patchwork-Id: 29607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D43BC83F11 for ; Mon, 28 Aug 2023 12:48:59 +0000 (UTC) Received: from www530.your-server.de (www530.your-server.de [188.40.30.78]) by mx.groups.io with SMTP id smtpd.web10.12633.1693226928748651427 for ; Mon, 28 Aug 2023 05:48:50 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@geanix.com header.s=default2211 header.b=0xBUnPjL; spf=pass (domain: geanix.com, ip: 188.40.30.78, mailfrom: martin@geanix.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=geanix.com; s=default2211; h=Content-Transfer-Encoding:Content-Type:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID; bh=SU1HkXsX2P60GTrXtwLB+S5QsRDvMWiIUrUziT7CGZ4=; b=0xBUnPjLFR+BjIU/don5QkrpHF vKPL/VvN2qLuzGdKCV18HM7Vjo+0C9SqsY+m/VHymEjRF3ET9RXbIBNdh9xcwczZFT1Wf5zH3frHh +lARNljpxS/kIXthifI8uxmqzpZxjrmignDuKTdY+Q3vcYAWkQCVDbJ0AesrnB1PxBDsGD1mv8ME1 IR9yAGlbFHJTomDP0CMOcwZy1e1f1Cb0U3DEHCn8xBv48NA8aTyUfWNW0GQ6+Yg1D8niwqpttmZdq oI9IP7TqeDzLYcsfrBMnpTowV4nZYyH/g8avfDRFTpsA6bcU9JnoqXNoyHyUnR+q+vnaCROjrv6ci inedf3tg==; Received: from sslproxy03.your-server.de ([88.198.220.132]) by www530.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qabfi-0009Go-Oz; Mon, 28 Aug 2023 14:48:46 +0200 Received: from [185.17.218.86] (helo=rap..) by sslproxy03.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qabfi-000BNd-AQ; Mon, 28 Aug 2023 14:48:46 +0200 From: =?utf-8?q?Martin_Hundeb=C3=B8ll?= To: openembedded-core@lists.openembedded.org Cc: =?utf-8?q?Martin_Hundeb=C3=B8ll?= Subject: [RFC PATCH 5/5] contrib: add python service and systemd unit to run shared jobserver Date: Mon, 28 Aug 2023 14:48:29 +0200 Message-ID: <20230828124834.376779-5-martin@geanix.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230828124834.376779-1-martin@geanix.com> References: <20230828124834.376779-1-martin@geanix.com> MIME-Version: 1.0 X-Authenticated-Sender: martin@geanix.com X-Virus-Scanned: Clear (ClamAV 0.103.8/27014/Mon Aug 28 09:38:26 2023) List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Aug 2023 12:48:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/186826 For CI setups that might end up building multiple yocto builds in parallel, a shared jobserver can reduce the total load of the system. Setting up such a jobserver is simple, but it does require a process hanging around to keep the jobserver fifo open (to avoid blocking token requests). Add a simple python script that creates such a jobserver fifo and waits forever. Also add a systemd unit file to start the python service at boot. The systemd unit can be installed in $HOME/.config/systemd/user/, but one might need to add a droplet config (i.e. `systemctl --user edit jobserver.service`) to setup the PYTHONPATH variable to make the python script loadable. Signed-off-by: Martin Hundebøll --- contrib/jobserver/jobserver.py | 78 +++++++++++++++++++++++++++++ contrib/jobserver/jobserver.service | 10 ++++ 2 files changed, 88 insertions(+) create mode 100644 contrib/jobserver/jobserver.py create mode 100644 contrib/jobserver/jobserver.service diff --git a/contrib/jobserver/jobserver.py b/contrib/jobserver/jobserver.py new file mode 100644 index 0000000000..41b085f47f --- /dev/null +++ b/contrib/jobserver/jobserver.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 + +from pathlib import Path +from threading import Event +import argparse +import os +import shutil +import signal + +resumed = Event() +runtime_dir = os.environ.get("XDG_RUNTIME_DIR", "/run") + +def signal_handler(signum, _frame): + """Wait for an external signal exit the process gracefully.""" + resumed.set() + + +def main(path, user, group, mode, jobs): + """Setup a fifo to used as jobserver shared between builds.""" + try: + path.unlink(missing_ok=True) + os.mkfifo(path) + shutil.chown(path, user, group) + os.chmod(path, mode) + except (FileNotFoundError, PermissionError) as exc: + raise SystemExit(f"failed to create fifo: {path}: {exc.strerror}") + + print(f"jobserver: {path}: {jobs} jobs") + fifo = os.open(path, os.O_RDWR) + os.write(fifo, b"+" * jobs) + + print("jobserver: ready; waiting indefinitely") + signal.signal(signal.SIGTERM, signal_handler) + signal.signal(signal.SIGINT, signal_handler) + resumed.wait() + + print("jobserver: exiting") + path.unlink() + os.close(fifo) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + prog='Make jobserver', + description='Simple application to instantiate a jobserver fifo and hang around', + ) + parser.add_argument( + "--mode", + help="Permission to apply to jobserver fifo", + type=lambda v: int(v, 8), + default=0o0666, + ) + parser.add_argument( + "--user", + help="Username or id to assign ownership of fifo to", + default=os.getuid(), + ) + parser.add_argument( + "--group", + help="Groupname of id to assign ownership of fifo to", + default=os.getgid(), + ) + parser.add_argument( + "path", + help="Path to jobserver fifo path", + type=Path, + nargs='?', + default=f"{runtime_dir}/jobserver", + ) + parser.add_argument( + "jobs", + help="Number of tokens to load jobserver with", + type=int, + nargs='?', + default=os.cpu_count(), + ) + args = parser.parse_args() + main(args.path, args.user, args.group, args.mode, args.jobs) diff --git a/contrib/jobserver/jobserver.service b/contrib/jobserver/jobserver.service new file mode 100644 index 0000000000..bbc7167ac0 --- /dev/null +++ b/contrib/jobserver/jobserver.service @@ -0,0 +1,10 @@ +[Unit] +Description=Shared jobserver fifo + +[Service] +Type=simple +Environment=PYTHONUNBUFFERED=1 +ExecStart=python jobserver.py + +[Install] +WantedBy=multi-user.target