diff mbox series

[1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer

Message ID 20220825203451.2074080-1-alex@linutronix.de
State Accepted, archived
Commit f319534dc8fc68dfe120d129154a509f0cd6a3b0
Headers show
Series [1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer | expand

Commit Message

Alexander Kanavin Aug. 25, 2022, 8:34 p.m. UTC
This is the reverse of setting up a build by pointing TEMPLATECONF to a directory
with a template and running '. oe-init-build-env': this takes the config files from build/conf,
replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies
the config files into a specified layer under a specified template name.

In many or perhaps most cases such static prefabricated configurations (that require no
further editing) are just enough, and I believe they should be offered by the
official configuration management. On the other hand, generating build configurations with a
sufficiently versatile tool is a far more complex problem, and one we should try to tackle
once we see where and how static configs fall short.

Tooling to discover and select these templates when setting up a build will be provided later on.

How to use:

alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers save-build-conf ../../meta-alex/ test-1
NOTE: Starting bitbake server...
NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
You can try out the configuration with
TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
alex@Zen2:/srv/work/alex/poky/build-layersetup$

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/bblayers/buildconf.py           | 85 ++++++++++++++++++++++++
 meta/lib/oeqa/selftest/cases/bblayers.py |  5 ++
 2 files changed, 90 insertions(+)
 create mode 100644 meta/lib/bblayers/buildconf.py

Comments

Luca Ceresoli Aug. 26, 2022, 3:25 p.m. UTC | #1
Hi Alex,

On Thu, 25 Aug 2022 22:34:47 +0200
"Alexander Kanavin" <alex.kanavin@gmail.com> wrote:

> This is the reverse of setting up a build by pointing TEMPLATECONF to a directory
> with a template and running '. oe-init-build-env': this takes the config files from build/conf,
> replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies
> the config files into a specified layer under a specified template name.
> 
> In many or perhaps most cases such static prefabricated configurations (that require no
> further editing) are just enough, and I believe they should be offered by the
> official configuration management. On the other hand, generating build configurations with a
> sufficiently versatile tool is a far more complex problem, and one we should try to tackle
> once we see where and how static configs fall short.
> 
> Tooling to discover and select these templates when setting up a build will be provided later on.
> 
> How to use:
> 
> alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers save-build-conf ../../meta-alex/ test-1
> NOTE: Starting bitbake server...
> NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
> Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
> You can try out the configuration with
> TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
> alex@Zen2:/srv/work/alex/poky/build-layersetup$
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>

Testing with this series produced some failures:

  error: pathspec 'yocto-4.0' did not match any file(s) known to git

Looks like is is adding the poky-contrib remote but not poky itself.

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3996/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4010/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4064/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4052/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/26/steps/14/logs/stdio
Alexander Kanavin Aug. 26, 2022, 3:46 p.m. UTC | #2
On Fri, 26 Aug 2022 at 17:25, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote:

> Testing with this series produced some failures:
>
>   error: pathspec 'yocto-4.0' did not match any file(s) known to git
>
> Looks like is is adding the poky-contrib remote but not poky itself.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3996/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4010/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4064/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4052/steps/15/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/26/steps/14/logs/stdio

Yes, it fails when testing from poky-contrib. But it will pass when
tested from poky's master-next :) Perhaps yocto-4.0 should be changed
to a commit id that matches that tag? Or someone could push the tags
to poky-contrib.

Alex
Alexander Kanavin Aug. 27, 2022, 4:41 p.m. UTC | #3
On Fri, 26 Aug 2022 at 17:25, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote:
> Testing with this series produced some failures:
>
>   error: pathspec 'yocto-4.0' did not match any file(s) known to git
>
> Looks like is is adding the poky-contrib remote but not poky itself.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3996/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4010/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4064/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4052/steps/15/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/26/steps/14/logs/stdio

Ok, I sent a followup patch that should correct this.
Luca Ceresoli Aug. 27, 2022, 7:32 p.m. UTC | #4
Hi Alex,

On Fri, 26 Aug 2022 17:25:07 +0200
"Luca Ceresoli via lists.openembedded.org"
<luca.ceresoli=bootlin.com@lists.openembedded.org> wrote:

> Hi Alex,
> 
> On Thu, 25 Aug 2022 22:34:47 +0200
> "Alexander Kanavin" <alex.kanavin@gmail.com> wrote:
> 
> > This is the reverse of setting up a build by pointing TEMPLATECONF to a directory
> > with a template and running '. oe-init-build-env': this takes the config files from build/conf,
> > replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies
> > the config files into a specified layer under a specified template name.
> > 
> > In many or perhaps most cases such static prefabricated configurations (that require no
> > further editing) are just enough, and I believe they should be offered by the
> > official configuration management. On the other hand, generating build configurations with a
> > sufficiently versatile tool is a far more complex problem, and one we should try to tackle
> > once we see where and how static configs fall short.
> > 
> > Tooling to discover and select these templates when setting up a build will be provided later on.
> > 
> > How to use:
> > 
> > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers save-build-conf ../../meta-alex/ test-1
> > NOTE: Starting bitbake server...
> > NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
> > Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
> > You can try out the configuration with
> > TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
> > alex@Zen2:/srv/work/alex/poky/build-layersetup$
> > 
> > Signed-off-by: Alexander Kanavin <alex@linutronix.de>  
> 
> Testing with this series produced some failures:
> 
>   error: pathspec 'yocto-4.0' did not match any file(s) known to git
> 
> Looks like is is adding the poky-contrib remote but not poky itself.
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3996/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4010/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4064/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4052/steps/15/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/26/steps/14/logs/stdio
> 

This series appears to have generated other errors on several qemu
builds:

  ERROR: core-image-sato-1.0-r0 do_testsdkext: Couldn't install the extensible SDK:

To me the correlation with your code was not immediately obvious, but
as Ross suggested I removed your patches and the failure disappeared.

Some logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/5114/steps/16/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/5737/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/5724/steps/16/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5721/steps/16/logs/stdio
Alexander Kanavin Aug. 27, 2022, 9:32 p.m. UTC | #5
On Sat, 27 Aug 2022 at 21:32, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote:
> This series appears to have generated other errors on several qemu
> builds:
>
>   ERROR: core-image-sato-1.0-r0 do_testsdkext: Couldn't install the extensible SDK:
>
> To me the correlation with your code was not immediately obvious, but
> as Ross suggested I removed your patches and the failure disappeared.
>
> Some logs:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/5114/steps/16/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/5737/steps/15/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/5724/steps/16/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5721/steps/16/logs/stdio

Thanks, I've sent a corrected patch for this as well now ( [OE-core]
[PATCH] scripts/oe-setup-builddir: add a check that TEMPLATECONF is
valid ).

Alex
Alexander Kanavin Aug. 28, 2022, 12:01 p.m. UTC | #6
Okay, all this fixing and re-fixing might have become confusing. I
pushed the latest, complete, rebased patchset here, so you can simply
take that:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/setup-layers

Alex

On Sat, 27 Aug 2022 at 23:32, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
>
> On Sat, 27 Aug 2022 at 21:32, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote:
> > This series appears to have generated other errors on several qemu
> > builds:
> >
> >   ERROR: core-image-sato-1.0-r0 do_testsdkext: Couldn't install the extensible SDK:
> >
> > To me the correlation with your code was not immediately obvious, but
> > as Ross suggested I removed your patches and the failure disappeared.
> >
> > Some logs:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/5114/steps/16/logs/stdio
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/5737/steps/15/logs/stdio
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/5724/steps/16/logs/stdio
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5721/steps/16/logs/stdio
>
> Thanks, I've sent a corrected patch for this as well now ( [OE-core]
> [PATCH] scripts/oe-setup-builddir: add a check that TEMPLATECONF is
> valid ).
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#169973): https://lists.openembedded.org/g/openembedded-core/message/169973
> Mute This Topic: https://lists.openembedded.org/mt/93257020/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Luca Ceresoli Aug. 29, 2022, 2:38 p.m. UTC | #7
Hi Alex,

On Sun, 28 Aug 2022 14:01:08 +0200
Alexander Kanavin <alex.kanavin@gmail.com> wrote:

> Okay, all this fixing and re-fixing might have become confusing.

Indeed :-)

> I
> pushed the latest, complete, rebased patchset here, so you can simply
> take that:
> https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/setup-layers

Thanks for cleaning up things. I think having a unique, properly
versioned patch series (v2/V3...) in the mailing list would help.

This week Alex Belloni is in charge of taking patches, thus up to the
two Alexes...

Luca
Alexander Kanavin Aug. 29, 2022, 7:19 p.m. UTC | #8
On Mon, 29 Aug 2022 at 16:38, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote:
> Thanks for cleaning up things. I think having a unique, properly
> versioned patch series (v2/V3...) in the mailing list would help.

I'm afraid it would replace one confusion with another. There's:

- one bitbake patch
- several oe-core patches, one of which does not apply to poky and
needs to be replaced with...
- one poky patch

There's no way to version patch series across several mailing lists.

Alex
diff mbox series

Patch

diff --git a/meta/lib/bblayers/buildconf.py b/meta/lib/bblayers/buildconf.py
new file mode 100644
index 0000000000..e07fc534e1
--- /dev/null
+++ b/meta/lib/bblayers/buildconf.py
@@ -0,0 +1,85 @@ 
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import logging
+import os
+import stat
+import sys
+import shutil
+import json
+
+import bb.utils
+import bb.process
+
+from bblayers.common import LayerPlugin
+
+logger = logging.getLogger('bitbake-layers')
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+import oe.buildcfg
+
+def plugin_init(plugins):
+    return BuildConfPlugin()
+
+class BuildConfPlugin(LayerPlugin):
+    notes_fixme = """FIXME: Please place here the description of this build configuration.
+It will be shown to the users when they set up their builds via TEMPLATECONF.
+"""
+
+    def _save_conf(self, templatename, templatepath, oecorepath, relpaths_to_oecore):
+        confdir = os.path.join(os.environ["BBPATH"], "conf")
+        destdir = os.path.join(templatepath, "conf", "templates", templatename)
+        os.makedirs(destdir, exist_ok=True)
+
+        with open(os.path.join(confdir, "local.conf")) as src:
+            with open(os.path.join(destdir, "local.conf.sample"), 'w') as dest:
+                dest.write(src.read())
+
+        with open(os.path.join(confdir, "bblayers.conf")) as src:
+            with open(os.path.join(destdir, "bblayers.conf.sample"), 'w') as dest:
+                bblayers_data = src.read()
+
+                for (abspath, relpath) in relpaths_to_oecore:
+                    bblayers_data = bblayers_data.replace(abspath, "##OEROOT##/" + relpath)
+                dest.write(bblayers_data)
+
+        with open(os.path.join(destdir, "conf-notes.txt"), 'w') as dest:
+            dest.write(self.notes_fixme)
+
+        logger.info("""Configuration template placed into {}
+Please review the files in there, and particularly provide a configuration description in {}
+You can try out the configuration with
+TEMPLATECONF={} . {}/oe-init-build-env build-try-{}"""
+.format(destdir, os.path.join(destdir, "conf-notes.txt"), destdir, oecorepath, templatename))
+
+    def do_save_build_conf(self, args):
+        """ Save the currently active build configuration (conf/local.conf, conf/bblayers.conf) as a template into a layer.\n This template can later be used for setting up builds via TEMPLATECONF. """
+        repos = {}
+        layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data)
+        targetlayer = None
+        oecore = None
+
+        for l in layers:
+            if l[0] == os.path.abspath(args.layerpath):
+                targetlayer = l[0]
+            if l[1] == 'meta':
+                oecore = os.path.dirname(l[0])
+
+        if not targetlayer:
+            logger.error("Layer {} not in one of the currently enabled layers:\n{}".format(args.layerpath, "\n".join([l[0] for l in layers])))
+        elif not oecore:
+            logger.error("Openembedded-core not in one of the currently enabled layers:\n{}".format("\n".join([l[0] for l in layers])))
+        else:
+            relpaths_to_oecore = [(l[0], os.path.relpath(l[0], start=oecore)) for l in layers]
+            self._save_conf(args.templatename, targetlayer, oecore, relpaths_to_oecore)
+
+    def register_commands(self, sp):
+        parser_build_conf = self.add_command(sp, 'save-build-conf', self.do_save_build_conf, parserecipes=False)
+        parser_build_conf.add_argument('layerpath',
+            help='The path to the layer where the configuration template should be saved.')
+        parser_build_conf.add_argument('templatename',
+            help='The name of the configuration template.')
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 494fa892a3..549abe7d10 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -113,6 +113,11 @@  class BitbakeLayers(OESelftestTestCase):
 
         self.assertEqual(bb_vars['BBFILE_PRIORITY_%s' % layername], str(priority), 'BBFILE_PRIORITY_%s != %d' % (layername, priority))
 
+        result = runCmd('bitbake-layers save-build-conf {} {}'.format(layerpath, "buildconf-1"))
+        for f in ('local.conf.sample', 'bblayers.conf.sample', 'conf-notes.txt'):
+            fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f)
+            self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath))
+
     def get_recipe_basename(self, recipe):
         recipe_file = ""
         result = runCmd("bitbake-layers show-recipes -f %s" % recipe)