[1/5] classes: jobserver: support gnu make fifo jobserver

Add a class to implement the gnu make fifo style jobserver. The class
can be activated by symply adding an `INHERIT += "jobserver"` to the
local configuration.

Furthermore, one can configure an external jobserver (i.e. a server
shared between multiple builds), by configured the `JOBSERVER_FIFO`
variable to point at an existing jobserver fifo.

The jobserver class uses the fifo style jobserver, which doesn't require
passing open file descriptors around. It does, however, require
make-4.4, which isn't available in common distro yet. To work around
this, the class make all recipes (except make and its dependencies
itself) depend on `virtual/make-native`.

Signed-off-by: Martin Hundebøll <martin@geanix.com>
---
 meta/classes-global/jobserver.bbclass | 80 +++++++++++++++++++++++++++
 meta/conf/bitbake.conf                |  2 +-
 2 files changed, 81 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes-global/jobserver.bbclass

Hi 

Very interesting patch series. I stumbled upon a couple of spelling
errors while looking through it, and have a question about the
hardcoded dependency list.

On Wed, 2024-04-03 at 09:02 +0200, Martin Hundeb?ll wrote:
> Add a class to implement the gnu make fifo style jobserver. The class
> can be activated by symply adding an `INHERIT += "jobserver"` to the

s/symply/simply/

> local configuration.
> 
> Furthermore, one can configure an external jobserver (i.e. a server
> shared between multiple builds), by configured the `JOBSERVER_FIFO`
> variable to point at an existing jobserver fifo.
> 
> The jobserver class uses the fifo style jobserver, which doesn't require
> passing open file descriptors around. It does, however, require
> make-4.4, which isn't available in common distro yet. To work around
> this, the class make all recipes (except make and its dependencies
> itself) depend on `virtual/make-native`.
> 
> Signed-off-by: Martin Hundebøll <martin@geanix.com>
> ---
>  meta/classes-global/jobserver.bbclass | 80 +++++++++++++++++++++++++++
>  meta/conf/bitbake.conf                |  2 +-
>  2 files changed, 81 insertions(+), 1 deletion(-)
>  create mode 100644 meta/classes-global/jobserver.bbclass
> 
> diff --git a/meta/classes-global/jobserver.bbclass b/meta/classes-global/jobserver.bbclass
> new file mode 100644
> index 0000000000..c76909fe50
> --- /dev/null
> +++ b/meta/classes-global/jobserver.bbclass
> @@ -0,0 +1,80 @@
> +JOBSERVER_FIFO ?= ""
> +JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of creating a per-build server."
> +
> +addhandler jobserver_setup_fifo
> +jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
> +
> +python jobserver_setup_fifo() {
> +    # don't setup a per-build fifo, if an external one is configured
> +    if d.getVar("JOBSERVER_FIFO"):
> +        return
> +
> +    # don't use a job-server if no parallelism is configured
> +    jobs = oe.utils.parallel_make(d)
> +    if jobs in (None, 1):
> +        return
> +
> +    # reduce jobs by one as a token has implicitly been handed to the
> +    # process requesting tokens
> +    jobs -= 1
> +
> +    fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
> +
> +    # and old fifo might be lingering; remove it
> +    if os.path.exists(fifo):
> +        os.remove(fifo)
> +
> +    # create a new fifo to use for communicating tokens
> +    os.mkfifo(fifo)
> +
> +    # fill the fifo with the number of tokens to hand out
> +    wfd = os.open(fifo, os.O_RDWR)
> +    written = os.write(wfd, b"+" * jobs)
> +    if written != (jobs):
> +        bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
> +
> +    # configure the per-build fifo path to use
> +    d.setVar("JOBSERVER_FIFO", fifo)
> +}
> +
> +python () {
> +    # don't configure the fifo if none is defined
> +    fifo = d.getVar("JOBSERVER_FIFO")
> +    if not fifo:
> +        return
> +
> +    # avoid making make-native or its dependencies depend on make-native itself
> +    if d.getVar("PN") in (
> +                "make-native",
> +                "libtool-native",
> +                "pkgconfig-native",
> +                "automake-native",
> +                "autoconf-native",
> +                "m4-native",
> +                "texinfo-dummy-native",
> +                "gettext-minimal-native",
> +                "quilt-native",
> +                "gnu-config-native",

This seems a bit brittle, but I guess make's dependencies doesn't
change often. So it is probably not a big issue.

How does it break if a dependency is missing?

> +            ):
> +        return
> +
> +    # don't make unwilling recipes depend on make-native
> +    if d.getVar('INHIBIT_DEFAULT_DEPS', False):
> +        return
> +
> +    # make other recipes depend on make-native to make sure it is new enough to
> +    # support the --jobserver-auth=fifo:<path> syntax (from make-4.4 and onwards)
> +    d.appendVar("DEPENDS", " virtual/make-native")
> +
> +    # disable the "-j <jobs>" flag, as that overrides the jobserver fifo tokens
> +    d.setVar("PARALLEL_MAKE", "")
> +    d.setVar("PARALLEL_MAKEINST", "")
> +
> +    # set and export the jobserver in the environment
> +    d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
> +    d.setVarFlag("MAKEFLAGS", "export", "1")
> +
> +    # ignore the joberserver argument part of MAKEFLAGS in the hash, as that

s/joberserver/jobserver/

> +    # shouldn't change the build output
> +    d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "| --jobserver-auth=fifo:" + fifo)
> +}
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 6f180d18b0..23a016b31e 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -960,7 +960,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
>      BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
>      SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
>      SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
> -    OMP_NUM_THREADS BB_CURRENTTASK"
> +    OMP_NUM_THREADS BB_CURRENTTASK JOBSERVER_FIFO"
>  BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
>      SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS"
>  BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#197910): https://lists.openembedded.org/g/openembedded-core/message/197910
> Mute This Topic: https://lists.openembedded.org/mt/105304233/7501392
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [andreaskleist@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

/Andreas

Hi Andreas,

On Thu, 2024-04-04 at 09:56 +0200, Andreas Helbech Kleist wrote:
> > +    # avoid making make-native or its dependencies depend on make-
> > native itself
> > +    if d.getVar("PN") in (
> > +                "make-native",
> > +                "libtool-native",
> > +                "pkgconfig-native",
> > +                "automake-native",
> > +                "autoconf-native",
> > +                "m4-native",
> > +                "texinfo-dummy-native",
> > +                "gettext-minimal-native",
> > +                "quilt-native",
> > +                "gnu-config-native",
> 
> This seems a bit brittle, but I guess make's dependencies doesn't
> change often. So it is probably not a big issue.
> 
> How does it break if a dependency is missing?

Bitbake errors out with a dependency loop.

// Martin