| Message ID | 20220701192449.1358325-1-alex@linutronix.de |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC] bitbake-layers: add layer repositories/revisions save and restore tooling (aka 'layer configuration') | expand |
Can you outline what this provides over `git submodules`? It seems pretty similar and I'm struggling to see what this provides that isn't already provided by that tool. On 7/1/22 14:24, Alexander Kanavin wrote: > This addresses a long standing gap in the core offering: > there is no tooling to capture the currently configured layers > with their revisions, or restore the layers from a configuration > file (without using external tools, some of which aren't particularly > suitable for the task). This plugin addresses the gap. > > How to use: > > 1. Saving a layer configuration: > > a) Command line options: > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h > NOTE: Starting bitbake server... > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir > > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. > > positional arguments: > destdir Directory where to write the output > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). > > optional arguments: > -h, --help show this help message and exit > --output OUTPUT, -o OUTPUT > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. > --format {python,json,kas}, -f {python,json,kas} > Format of the output. The options are: > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) > kas - a configuration file for the kas tool that allows the tool to do the same > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. > > b) Running with default choices: > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ > NOTE: Starting bitbake server... > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > 2. Restoring the layers from the saved configuration: > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. > > b) Command line options: > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > optional arguments: > -h, --help show this help message and exit > --force-meta-alex-checkout > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). > --choose-poky-remote {origin,poky-contrib} > Choose a remote server for layer poky (default: origin) > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). > > c) Running with default options: > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel > Note: multiple remotes defined for layer poky, using origin (run with -h to see others). > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky > > And that's it! > > FIXMEs: > - kas config writer not yet implemented > - oe-selftest test cases not yet written > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ > .../templates/setup-layers.py.template | 77 ++++++++++++ > 2 files changed, 194 insertions(+) > create mode 100644 meta/lib/bblayers/makesetup.py > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template > > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py > new file mode 100644 > index 0000000000..3c86eea3c4 > --- /dev/null > +++ b/meta/lib/bblayers/makesetup.py > @@ -0,0 +1,117 @@ > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > +import logging > +import os > +import stat > +import sys > +import shutil > +import json > + > +import bb.utils > +import bb.process > + > +from bblayers.common import LayerPlugin > + > +logger = logging.getLogger('bitbake-layers') > + > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > + > +import oe.buildcfg > + > +def plugin_init(plugins): > + return MakeSetupPlugin() > + > +class MakeSetupPlugin(LayerPlugin): > + > + def _write_python(self, repos, output): > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: > + template = f.read() > + args = sys.argv > + args[0] = os.path.basename(args[0]) > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) > + with open(output, 'w') as f: > + f.write(script) > + st = os.stat(output) > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) > + > + def _write_json(self, repos, output): > + with open(output, 'w') as f: > + json.dump(repos, f, sort_keys=True, indent=4) > + > + def _write_kas(self, repos, output): > + raise NotImplementedError('Kas config writer not implemented yet') > + > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > + > + def _get_repo_path(self, layer_path): > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) > + return repo_path.strip() > + > + def _get_remotes(self, repo_path): > + remotes = [] > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > + for r in remotes_list.split(): > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) > + remotes.append({'name':r,'uri':uri.strip()}) > + return remotes > + > + def _get_describe(self, repo_path): > + try: > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) > + except bb.process.ExecutionError: > + return "" > + return describe.strip() > + > + def _make_repo_config(self, destdir): > + repos = {} > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > + for l in layers: > + if l[1] == 'workspace': > + continue > + if l[4]: > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) > + return > + repo_path = self._get_repo_path(l[0]) > + if repo_path not in repos.keys(): > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} > + if not repos[repo_path]['remotes']: > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) > + return > + if repo_path in os.path.abspath(destdir): > + repos[repo_path]['is_bootstrap'] = True > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > + > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > + if len(repo_dirs) > 1: > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) > + return > + > + repos_nopaths = {} > + for r in repos.keys(): > + r_nopath = os.path.basename(r) > + repos_nopaths[r_nopath] = repos[r] > + return repos_nopaths > + > + def do_make_setup(self, args): > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ > + repos = self._make_repo_config(args.destdir) > + if not repos: > + return > + output = args.output > + if not output: > + output = self._output_filename[args.format] > + output = os.path.join(os.path.abspath(args.destdir),output) > + self._write_config[args.format](self, repos, output) > + logger.info('Created {}'.format(output)) > + > + def register_commands(self, sp): > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) > + parser_setup_layers.add_argument('destdir', > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') > + parser_setup_layers.add_argument('--output', '-o', > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template > new file mode 100644 > index 0000000000..a704ad3d70 > --- /dev/null > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > @@ -0,0 +1,77 @@ > +#!/usr/bin/env python3 > +# > +# This file was generated by running > +# > +# {cmdline} > +# > +# It is recommended that you do not modify it directly, but rather re-run the above command. > +# > + > +layerdata = """ > +{layerdata} > +""" > + > +import argparse > +import json > +import os > +import subprocess > + > +def _do_checkout(args): > + for l_name in layers: > + l_data = layers[l_name] > + if 'is_bootstrap' in l_data.keys(): > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) > + if not args[force_arg]: > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) > + continue > + rev = l_data['rev'] > + desc = l_data['describe'] > + if not desc: > + desc = rev[:10] > + branch = l_data['branch'] > + remotes = l_data['remotes'] > + remote = remotes[0] > + if len(remotes) > 1: > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] > + for r in remotes: > + if r['name'] == remotechoice: > + remote = r > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > + cwd = args['destdir'] > + print("Running '{}' in {}".format(cmd, cwd)) > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > + cmd = 'git checkout -q {}'.format(rev) > + cwd = os.path.join(args['destdir'], l_name) > + print("Running '{}' in {}".format(cmd, cwd)) > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > + > +layers = json.loads(layerdata) > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') > + > +bootstraplayer = None > +for l in layers: > + if 'is_bootstrap' in layers[l]: > + bootstraplayer = l > + > +if bootstraplayer: > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > + > +for l in layers: > + remotes = layers[l]['remotes'] > + if len(remotes) > 1: > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) > + > +try: > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) > +except subprocess.CalledProcessError as e: > + defaultdest = os.path.abspath(".") > + > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > + > +args = parser.parse_args() > + > +_do_checkout(vars(args)) > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#167540): https://lists.openembedded.org/g/openembedded-core/message/167540 > Mute This Topic: https://lists.openembedded.org/mt/92117681/3616693 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [JPEWhacker@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
First of all this provides a single command that does all of the work
to capture the layers and their revisions from currently enabled set
in a bitbake build. No other tool does this because they do not know
anything about bitbake or tinfoil, and so you have to write your
config by hand, with all of them.
Here's how the captured metadata loks like [1]. I find it more useful
for understanding what are we actually checking out than submodules'
bare, unhelpful set of git revisions. It also doesn't require a
super-repo containing all of the submodules: you can just toss the
script into one of the layers and it will do the right thing for
bootstrapping. There was also a comment from one of our customers: "
In practice we learned that recursive submodules are not always
correctly handled by GitLab CI. "
I also find git submodules command line UI supremely confusing.
Actually wait, that's what git as a whole is like: confusing.
https://xkcd.com/1597/
But if you would like, we can add a fourth output format to the
bblayers plugin added in this patch that sets up git submodules
instead of writing a standalone python script with embedded json
(which is what I prefer).
Alex
[1]
{
"meta-alex": {
"branch": "master",
"describe": "",
"is_bootstrap": true,
"layers": [
{
"name": "meta-alex",
"path": ""
}
],
"remotes": [
{
"name": "remote-alex",
"uri": "https://github.com/kanavin/meta-alex"
}
],
"rev": "05b25605fb8b2399e4706d7323828676bf0da0b5"
},
"meta-intel": {
"branch": "master",
"describe": "15.0-hardknott-3.3-310-g0a96edae",
"layers": [
{
"name": "meta-intel",
"path": ""
}
],
"remotes": [
{
"name": "origin",
"uri": "git://git.yoctoproject.org/meta-intel"
}
],
"rev": "0a96edae609a3f48befac36af82cf1eed6786b4a"
},
"poky": {
"branch": "akanavin/setup-layers",
"describe": "4.1_M1-295-gfd524e4984",
"layers": [
{
"name": "meta",
"path": "meta"
},
{
"name": "meta-poky",
"path": "meta-poky"
},
{
"name": "meta-yocto-bsp",
"path": "meta-yocto-bsp"
}
],
"remotes": [
{
"name": "origin",
"uri": "git://git.yoctoproject.org/poky"
},
{
"name": "poky-contrib",
"uri": "ssh://git@push.yoctoproject.org/poky-contrib"
}
],
"rev": "fd524e4984eb535f16a14dd7d79fde661b44aa78"
}
}
On Fri, 1 Jul 2022 at 21:46, Joshua Watt <jpewhacker@gmail.com> wrote:
>
> Can you outline what this provides over `git submodules`? It seems
> pretty similar and I'm struggling to see what this provides that isn't
> already provided by that tool.
>
> On 7/1/22 14:24, Alexander Kanavin wrote:
> > This addresses a long standing gap in the core offering:
> > there is no tooling to capture the currently configured layers
> > with their revisions, or restore the layers from a configuration
> > file (without using external tools, some of which aren't particularly
> > suitable for the task). This plugin addresses the gap.
> >
> > How to use:
> >
> > 1. Saving a layer configuration:
> >
> > a) Command line options:
> >
> > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h
> > NOTE: Starting bitbake server...
> > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir
> >
> > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build.
> >
> > positional arguments:
> > destdir Directory where to write the output
> > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).
> >
> > optional arguments:
> > -h, --help show this help message and exit
> > --output OUTPUT, -o OUTPUT
> > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.
> > --format {python,json,kas}, -f {python,json,kas}
> > Format of the output. The options are:
> > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)
> > kas - a configuration file for the kas tool that allows the tool to do the same
> > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool.
> >
> > b) Running with default choices:
> >
> > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/
> > NOTE: Starting bitbake server...
> > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py
> >
> > 2. Restoring the layers from the saved configuration:
> >
> > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step.
> >
> > b) Command line options:
> >
> > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h
> > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR]
> >
> > A self contained python script that fetches all the needed layers and sets them to correct revisions
> >
> > optional arguments:
> > -h, --help show this help message and exit
> > --force-meta-alex-checkout
> > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place).
> > --choose-poky-remote {origin,poky-contrib}
> > Choose a remote server for layer poky (default: origin)
> > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test).
> >
> > c) Running with default options:
> >
> > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py
> > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override.
> > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel
> > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test
> > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel
> > Note: multiple remotes defined for layer poky, using origin (run with -h to see others).
> > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky
> > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test
> > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky
> >
> > And that's it!
> >
> > FIXMEs:
> > - kas config writer not yet implemented
> > - oe-selftest test cases not yet written
> >
> > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> > ---
> > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++
> > .../templates/setup-layers.py.template | 77 ++++++++++++
> > 2 files changed, 194 insertions(+)
> > create mode 100644 meta/lib/bblayers/makesetup.py
> > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template
> >
> > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py
> > new file mode 100644
> > index 0000000000..3c86eea3c4
> > --- /dev/null
> > +++ b/meta/lib/bblayers/makesetup.py
> > @@ -0,0 +1,117 @@
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +
> > +import logging
> > +import os
> > +import stat
> > +import sys
> > +import shutil
> > +import json
> > +
> > +import bb.utils
> > +import bb.process
> > +
> > +from bblayers.common import LayerPlugin
> > +
> > +logger = logging.getLogger('bitbake-layers')
> > +
> > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
> > +
> > +import oe.buildcfg
> > +
> > +def plugin_init(plugins):
> > + return MakeSetupPlugin()
> > +
> > +class MakeSetupPlugin(LayerPlugin):
> > +
> > + def _write_python(self, repos, output):
> > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f:
> > + template = f.read()
> > + args = sys.argv
> > + args[0] = os.path.basename(args[0])
> > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4))
> > + with open(output, 'w') as f:
> > + f.write(script)
> > + st = os.stat(output)
> > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
> > +
> > + def _write_json(self, repos, output):
> > + with open(output, 'w') as f:
> > + json.dump(repos, f, sort_keys=True, indent=4)
> > +
> > + def _write_kas(self, repos, output):
> > + raise NotImplementedError('Kas config writer not implemented yet')
> > +
> > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas}
> > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"}
> > +
> > + def _get_repo_path(self, layer_path):
> > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path)
> > + return repo_path.strip()
> > +
> > + def _get_remotes(self, repo_path):
> > + remotes = []
> > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path)
> > + for r in remotes_list.split():
> > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path)
> > + remotes.append({'name':r,'uri':uri.strip()})
> > + return remotes
> > +
> > + def _get_describe(self, repo_path):
> > + try:
> > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path)
> > + except bb.process.ExecutionError:
> > + return ""
> > + return describe.strip()
> > +
> > + def _make_repo_config(self, destdir):
> > + repos = {}
> > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data)
> > + for l in layers:
> > + if l[1] == 'workspace':
> > + continue
> > + if l[4]:
> > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0]))
> > + return
> > + repo_path = self._get_repo_path(l[0])
> > + if repo_path not in repos.keys():
> > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)}
> > + if not repos[repo_path]['remotes']:
> > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path))
> > + return
> > + if repo_path in os.path.abspath(destdir):
> > + repos[repo_path]['is_bootstrap'] = True
> > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]})
> > +
> > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()])
> > + if len(repo_dirs) > 1:
> > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs))
> > + return
> > +
> > + repos_nopaths = {}
> > + for r in repos.keys():
> > + r_nopath = os.path.basename(r)
> > + repos_nopaths[r_nopath] = repos[r]
> > + return repos_nopaths
> > +
> > + def do_make_setup(self, args):
> > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """
> > + repos = self._make_repo_config(args.destdir)
> > + if not repos:
> > + return
> > + output = args.output
> > + if not output:
> > + output = self._output_filename[args.format]
> > + output = os.path.join(os.path.abspath(args.destdir),output)
> > + self._write_config[args.format](self, repos, output)
> > + logger.info('Created {}'.format(output))
> > +
> > + def register_commands(self, sp):
> > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False)
> > + parser_setup_layers.add_argument('destdir',
> > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).')
> > + parser_setup_layers.add_argument('--output', '-o',
> > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.')
> > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python',
> > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.')
> > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template
> > new file mode 100644
> > index 0000000000..a704ad3d70
> > --- /dev/null
> > +++ b/meta/lib/bblayers/templates/setup-layers.py.template
> > @@ -0,0 +1,77 @@
> > +#!/usr/bin/env python3
> > +#
> > +# This file was generated by running
> > +#
> > +# {cmdline}
> > +#
> > +# It is recommended that you do not modify it directly, but rather re-run the above command.
> > +#
> > +
> > +layerdata = """
> > +{layerdata}
> > +"""
> > +
> > +import argparse
> > +import json
> > +import os
> > +import subprocess
> > +
> > +def _do_checkout(args):
> > + for l_name in layers:
> > + l_data = layers[l_name]
> > + if 'is_bootstrap' in l_data.keys():
> > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_'))
> > + if not args[force_arg]:
> > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name)))
> > + continue
> > + rev = l_data['rev']
> > + desc = l_data['describe']
> > + if not desc:
> > + desc = rev[:10]
> > + branch = l_data['branch']
> > + remotes = l_data['remotes']
> > + remote = remotes[0]
> > + if len(remotes) > 1:
> > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))]
> > + for r in remotes:
> > + if r['name'] == remotechoice:
> > + remote = r
> > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name']))
> > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri']))
> > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name)
> > + cwd = args['destdir']
> > + print("Running '{}' in {}".format(cmd, cwd))
> > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd)
> > + cmd = 'git checkout -q {}'.format(rev)
> > + cwd = os.path.join(args['destdir'], l_name)
> > + print("Running '{}' in {}".format(cmd, cwd))
> > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd)
> > +
> > +layers = json.loads(layerdata)
> > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions')
> > +
> > +bootstraplayer = None
> > +for l in layers:
> > + if 'is_bootstrap' in layers[l]:
> > + bootstraplayer = l
> > +
> > +if bootstraplayer:
> > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true',
> > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer))
> > +
> > +for l in layers:
> > + remotes = layers[l]['remotes']
> > + if len(remotes) > 1:
> > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'],
> > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name']))
> > +
> > +try:
> > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__)))
> > +except subprocess.CalledProcessError as e:
> > + defaultdest = os.path.abspath(".")
> > +
> > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest))
> > +
> > +args = parser.parse_args()
> > +
> > +_do_checkout(vars(args))
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#167540): https://lists.openembedded.org/g/openembedded-core/message/167540
> > Mute This Topic: https://lists.openembedded.org/mt/92117681/3616693
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [JPEWhacker@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
On Fri, Jul 1, 2022 at 3:29 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > First of all this provides a single command that does all of the work > to capture the layers and their revisions from currently enabled set > in a bitbake build. No other tool does this because they do not know > anything about bitbake or tinfoil, and so you have to write your > config by hand, with all of them. > > Here's how the captured metadata loks like [1]. I find it more useful > for understanding what are we actually checking out than submodules' > bare, unhelpful set of git revisions. It also doesn't require a > super-repo containing all of the submodules: you can just toss the > script into one of the layers and it will do the right thing for > bootstrapping. There was also a comment from one of our customers: " > In practice we learned that recursive submodules are not always > correctly handled by GitLab CI. " > > I also find git submodules command line UI supremely confusing. > Actually wait, that's what git as a whole is like: confusing. > https://xkcd.com/1597/ > > But if you would like, we can add a fourth output format to the > bblayers plugin added in this patch that sets up git submodules > instead of writing a standalone python script with embedded json > (which is what I prefer). Thanks for the write up. I don't particularly care if we go down this route; it seems fine as long as it doesn't become "the one true way" to do layer setup (because I'm still going to use submodules :) ) > > Alex > > [1] > { > "meta-alex": { > "branch": "master", > "describe": "", > "is_bootstrap": true, > "layers": [ > { > "name": "meta-alex", > "path": "" > } > ], > "remotes": [ > { > "name": "remote-alex", > "uri": "https://github.com/kanavin/meta-alex" > } > ], > "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5" > }, > "meta-intel": { > "branch": "master", > "describe": "15.0-hardknott-3.3-310-g0a96edae", > "layers": [ > { > "name": "meta-intel", > "path": "" > } > ], > "remotes": [ > { > "name": "origin", > "uri": "git://git.yoctoproject.org/meta-intel" > } > ], > "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a" > }, > "poky": { > "branch": "akanavin/setup-layers", > "describe": "4.1_M1-295-gfd524e4984", > "layers": [ > { > "name": "meta", > "path": "meta" > }, > { > "name": "meta-poky", > "path": "meta-poky" > }, > { > "name": "meta-yocto-bsp", > "path": "meta-yocto-bsp" > } > > ], > "remotes": [ > { > "name": "origin", > "uri": "git://git.yoctoproject.org/poky" > }, > { > "name": "poky-contrib", > "uri": "ssh://git@push.yoctoproject.org/poky-contrib" > } > ], > "rev": "fd524e4984eb535f16a14dd7d79fde661b44aa78" > } > } > > On Fri, 1 Jul 2022 at 21:46, Joshua Watt <jpewhacker@gmail.com> wrote: > > > > Can you outline what this provides over `git submodules`? It seems > > pretty similar and I'm struggling to see what this provides that isn't > > already provided by that tool. > > > > On 7/1/22 14:24, Alexander Kanavin wrote: > > > This addresses a long standing gap in the core offering: > > > there is no tooling to capture the currently configured layers > > > with their revisions, or restore the layers from a configuration > > > file (without using external tools, some of which aren't particularly > > > suitable for the task). This plugin addresses the gap. > > > > > > How to use: > > > > > > 1. Saving a layer configuration: > > > > > > a) Command line options: > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h > > > NOTE: Starting bitbake server... > > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir > > > > > > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. > > > > > > positional arguments: > > > destdir Directory where to write the output > > > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > --output OUTPUT, -o OUTPUT > > > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. > > > --format {python,json,kas}, -f {python,json,kas} > > > Format of the output. The options are: > > > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) > > > kas - a configuration file for the kas tool that allows the tool to do the same > > > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. > > > > > > b) Running with default choices: > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ > > > NOTE: Starting bitbake server... > > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > > > > > 2. Restoring the layers from the saved configuration: > > > > > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. > > > > > > b) Command line options: > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > > > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > > > > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > --force-meta-alex-checkout > > > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). > > > --choose-poky-remote {origin,poky-contrib} > > > Choose a remote server for layer poky (default: origin) > > > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). > > > > > > c) Running with default options: > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > > > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. > > > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel > > > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test > > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel > > > Note: multiple remotes defined for layer poky, using origin (run with -h to see others). > > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky > > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test > > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky > > > > > > And that's it! > > > > > > FIXMEs: > > > - kas config writer not yet implemented > > > - oe-selftest test cases not yet written > > > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > > --- > > > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ > > > .../templates/setup-layers.py.template | 77 ++++++++++++ > > > 2 files changed, 194 insertions(+) > > > create mode 100644 meta/lib/bblayers/makesetup.py > > > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template > > > > > > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py > > > new file mode 100644 > > > index 0000000000..3c86eea3c4 > > > --- /dev/null > > > +++ b/meta/lib/bblayers/makesetup.py > > > @@ -0,0 +1,117 @@ > > > +# > > > +# SPDX-License-Identifier: GPL-2.0-only > > > +# > > > + > > > +import logging > > > +import os > > > +import stat > > > +import sys > > > +import shutil > > > +import json > > > + > > > +import bb.utils > > > +import bb.process > > > + > > > +from bblayers.common import LayerPlugin > > > + > > > +logger = logging.getLogger('bitbake-layers') > > > + > > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > > > + > > > +import oe.buildcfg > > > + > > > +def plugin_init(plugins): > > > + return MakeSetupPlugin() > > > + > > > +class MakeSetupPlugin(LayerPlugin): > > > + > > > + def _write_python(self, repos, output): > > > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: > > > + template = f.read() > > > + args = sys.argv > > > + args[0] = os.path.basename(args[0]) > > > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) > > > + with open(output, 'w') as f: > > > + f.write(script) > > > + st = os.stat(output) > > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) > > > + > > > + def _write_json(self, repos, output): > > > + with open(output, 'w') as f: > > > + json.dump(repos, f, sort_keys=True, indent=4) > > > + > > > + def _write_kas(self, repos, output): > > > + raise NotImplementedError('Kas config writer not implemented yet') > > > + > > > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} > > > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > > > + > > > + def _get_repo_path(self, layer_path): > > > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) > > > + return repo_path.strip() > > > + > > > + def _get_remotes(self, repo_path): > > > + remotes = [] > > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > > > + for r in remotes_list.split(): > > > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) > > > + remotes.append({'name':r,'uri':uri.strip()}) > > > + return remotes > > > + > > > + def _get_describe(self, repo_path): > > > + try: > > > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) > > > + except bb.process.ExecutionError: > > > + return "" > > > + return describe.strip() > > > + > > > + def _make_repo_config(self, destdir): > > > + repos = {} > > > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > > > + for l in layers: > > > + if l[1] == 'workspace': > > > + continue > > > + if l[4]: > > > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) > > > + return > > > + repo_path = self._get_repo_path(l[0]) > > > + if repo_path not in repos.keys(): > > > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} > > > + if not repos[repo_path]['remotes']: > > > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) > > > + return > > > + if repo_path in os.path.abspath(destdir): > > > + repos[repo_path]['is_bootstrap'] = True > > > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > > > + > > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > > > + if len(repo_dirs) > 1: > > > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) > > > + return > > > + > > > + repos_nopaths = {} > > > + for r in repos.keys(): > > > + r_nopath = os.path.basename(r) > > > + repos_nopaths[r_nopath] = repos[r] > > > + return repos_nopaths > > > + > > > + def do_make_setup(self, args): > > > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ > > > + repos = self._make_repo_config(args.destdir) > > > + if not repos: > > > + return > > > + output = args.output > > > + if not output: > > > + output = self._output_filename[args.format] > > > + output = os.path.join(os.path.abspath(args.destdir),output) > > > + self._write_config[args.format](self, repos, output) > > > + logger.info('Created {}'.format(output)) > > > + > > > + def register_commands(self, sp): > > > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) > > > + parser_setup_layers.add_argument('destdir', > > > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') > > > + parser_setup_layers.add_argument('--output', '-o', > > > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') > > > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', > > > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') > > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template > > > new file mode 100644 > > > index 0000000000..a704ad3d70 > > > --- /dev/null > > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > > > @@ -0,0 +1,77 @@ > > > +#!/usr/bin/env python3 > > > +# > > > +# This file was generated by running > > > +# > > > +# {cmdline} > > > +# > > > +# It is recommended that you do not modify it directly, but rather re-run the above command. > > > +# > > > + > > > +layerdata = """ > > > +{layerdata} > > > +""" > > > + > > > +import argparse > > > +import json > > > +import os > > > +import subprocess > > > + > > > +def _do_checkout(args): > > > + for l_name in layers: > > > + l_data = layers[l_name] > > > + if 'is_bootstrap' in l_data.keys(): > > > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) > > > + if not args[force_arg]: > > > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) > > > + continue > > > + rev = l_data['rev'] > > > + desc = l_data['describe'] > > > + if not desc: > > > + desc = rev[:10] > > > + branch = l_data['branch'] > > > + remotes = l_data['remotes'] > > > + remote = remotes[0] > > > + if len(remotes) > 1: > > > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] > > > + for r in remotes: > > > + if r['name'] == remotechoice: > > > + remote = r > > > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > > > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) > > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > > > + cwd = args['destdir'] > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > + cmd = 'git checkout -q {}'.format(rev) > > > + cwd = os.path.join(args['destdir'], l_name) > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > + > > > +layers = json.loads(layerdata) > > > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') > > > + > > > +bootstraplayer = None > > > +for l in layers: > > > + if 'is_bootstrap' in layers[l]: > > > + bootstraplayer = l > > > + > > > +if bootstraplayer: > > > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', > > > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > > > + > > > +for l in layers: > > > + remotes = layers[l]['remotes'] > > > + if len(remotes) > 1: > > > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], > > > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) > > > + > > > +try: > > > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) > > > +except subprocess.CalledProcessError as e: > > > + defaultdest = os.path.abspath(".") > > > + > > > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > > > + > > > +args = parser.parse_args() > > > + > > > +_do_checkout(vars(args)) > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > Links: You receive all messages sent to this group. > > > View/Reply Online (#167540): https://lists.openembedded.org/g/openembedded-core/message/167540 > > > Mute This Topic: https://lists.openembedded.org/mt/92117681/3616693 > > > Group Owner: openembedded-core+owner@lists.openembedded.org > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [JPEWhacker@gmail.com] > > > -=-=-=-=-=-=-=-=-=-=-=- > > >
On Fri, Jul 1, 2022 at 1:44 PM Joshua Watt <JPEWhacker@gmail.com> wrote: > On Fri, Jul 1, 2022 at 3:29 PM Alexander Kanavin <alex.kanavin@gmail.com> > wrote: > > > > First of all this provides a single command that does all of the work > > to capture the layers and their revisions from currently enabled set > > in a bitbake build. No other tool does this because they do not know > > anything about bitbake or tinfoil, and so you have to write your > > config by hand, with all of them. > > > > Here's how the captured metadata loks like [1]. I find it more useful > > for understanding what are we actually checking out than submodules' > > bare, unhelpful set of git revisions. It also doesn't require a > > super-repo containing all of the submodules: you can just toss the > > script into one of the layers and it will do the right thing for > > bootstrapping. There was also a comment from one of our customers: " > > In practice we learned that recursive submodules are not always > > correctly handled by GitLab CI. " > > > > I also find git submodules command line UI supremely confusing. > > Actually wait, that's what git as a whole is like: confusing. > > https://xkcd.com/1597/ > > > > But if you would like, we can add a fourth output format to the > > bblayers plugin added in this patch that sets up git submodules > > instead of writing a standalone python script with embedded json > > (which is what I prefer). > > Thanks for the write up. I don't particularly care if we go down this > route; it seems fine as long as it doesn't become "the one true way" > to do layer setup (because I'm still going to use submodules :) ) > This is exactly the problem. Everyone bike shedding; beautiful paint btw ;). We MUST find some solution that is a common reference. Otherwise we can just spin wheels forever. I will state for the record that git-repo phoning home on every invocation (and failing in very non helpful ways without proper ssh credentials or corporate proxies) is a non-starter. Yes it works. No, I will die on this mountain fighting against it being the way. No. Bad design. I like git sub modules. Yoe distro is a very good example of how to make this work. I did an ELCE presentation based on a git sub module system. Our entire team of three was git savvy. This is not a guarantee. Downstream from us broke very rapidly because folks didn’t understand how to contribute to the sub modules and keep it in sync. In the long run kas was drastically simpler to roll out and sustain. One remaining question is how do we handle configuration? This seems to address manifest? Git sub modules do not address configuration. Git-repo scripts are the first thing I have to fix/throw away. Do not assume where source and builds happen. My builds are on a different drive than my source. One-line command to fetch the layers and configure a build environment that will JustWork(TM) is the need. Kas is the only tool that does this as far as I have seen. In an opinionated way that overwrites local.conf (bad). For the record, I wholeheartedly endorse this direction, Alex. > > > > Alex > > > > [1] > > { > > "meta-alex": { > > "branch": "master", > > "describe": "", > > "is_bootstrap": true, > > "layers": [ > > { > > "name": "meta-alex", > > "path": "" > > } > > ], > > "remotes": [ > > { > > "name": "remote-alex", > > "uri": "https://github.com/kanavin/meta-alex" > > } > > ], > > "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5" > > }, > > "meta-intel": { > > "branch": "master", > > "describe": "15.0-hardknott-3.3-310-g0a96edae", > > "layers": [ > > { > > "name": "meta-intel", > > "path": "" > > } > > ], > > "remotes": [ > > { > > "name": "origin", > > "uri": "git://git.yoctoproject.org/meta-intel" > > } > > ], > > "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a" > > }, > > "poky": { > > "branch": "akanavin/setup-layers", > > "describe": "4.1_M1-295-gfd524e4984", > > "layers": [ > > { > > "name": "meta", > > "path": "meta" > > }, > > { > > "name": "meta-poky", > > "path": "meta-poky" > > }, > > { > > "name": "meta-yocto-bsp", > > "path": "meta-yocto-bsp" > > } > > > > ], > > "remotes": [ > > { > > "name": "origin", > > "uri": "git://git.yoctoproject.org/poky" > > }, > > { > > "name": "poky-contrib", > > "uri": "ssh://git@push.yoctoproject.org/poky-contrib" > > } > > ], > > "rev": "fd524e4984eb535f16a14dd7d79fde661b44aa78" > > } > > } > > > > On Fri, 1 Jul 2022 at 21:46, Joshua Watt <jpewhacker@gmail.com> wrote: > > > > > > Can you outline what this provides over `git submodules`? It seems > > > pretty similar and I'm struggling to see what this provides that isn't > > > already provided by that tool. > > > > > > On 7/1/22 14:24, Alexander Kanavin wrote: > > > > This addresses a long standing gap in the core offering: > > > > there is no tooling to capture the currently configured layers > > > > with their revisions, or restore the layers from a configuration > > > > file (without using external tools, some of which aren't particularly > > > > suitable for the task). This plugin addresses the gap. > > > > > > > > How to use: > > > > > > > > 1. Saving a layer configuration: > > > > > > > > a) Command line options: > > > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers > create-layers-setup -h > > > > NOTE: Starting bitbake server... > > > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] > [--format {python,json,kas}] destdir > > > > > > > > Writes out a python script/kas config/json config that replicates > the directory structure and revisions of the layers in a current build. > > > > > > > > positional arguments: > > > > destdir Directory where to write the output > > > > (if it is inside one of the layers, the > layer becomes a bootstrap repository and thus will be excluded from > fetching by the script). > > > > > > > > optional arguments: > > > > -h, --help show this help message and exit > > > > --output OUTPUT, -o OUTPUT > > > > File name where to write the output, if the > default (setup-layers.py/.json/.yml) is undesirable. > > > > --format {python,json,kas}, -f {python,json,kas} > > > > Format of the output. The options are: > > > > python - a self contained python > script that fetches all the needed layers and sets them to correct > revisions (default, recommended) > > > > kas - a configuration file for the kas > tool that allows the tool to do the same > > > > json - a json formatted file > containing all the needed metadata to do the same by any external or custom > tool. > > > > > > > > b) Running with default choices: > > > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers > create-layers-setup ../../meta-alex/ > > > > NOTE: Starting bitbake server... > > > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > > > > > > > 2. Restoring the layers from the saved configuration: > > > > > > > > a) Clone meta-alex separately, as a bootstrap layer/repository. It > should already contain setup-layers.py created in the previous step. > > > > > > > > b) Command line options: > > > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > > > > usage: setup-layers.py [-h] [--force-meta-alex-checkout] > [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > > > > > > > A self contained python script that fetches all the needed layers > and sets them to correct revisions > > > > > > > > optional arguments: > > > > -h, --help show this help message and exit > > > > --force-meta-alex-checkout > > > > Force the checkout of the bootstrap layer > meta-alex (by default it is presumed that this script is in it, and so the > layer is already in place). > > > > --choose-poky-remote {origin,poky-contrib} > > > > Choose a remote server for layer poky > (default: origin) > > > > --destdir DESTDIR Where to check out the layers (default is > /srv/work/alex/layers-test). > > > > > > > > c) Running with default options: > > > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > > > > Note: not checking out layer meta-alex, use > --force-meta-alex-checkout to override. > > > > Checking out layer meta-intel, revision > 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git:// > git.yoctoproject.org/meta-intel > > > > Running 'git clone -q git://git.yoctoproject.org/meta-intel > meta-intel' in /srv/work/alex/layers-test > > > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' > in /srv/work/alex/layers-test/meta-intel > > > > Note: multiple remotes defined for layer poky, using origin (run > with -h to see others). > > > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch > akanavin/setup-layers from remote origin at git:// > git.yoctoproject.org/poky > > > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in > /srv/work/alex/layers-test > > > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' > in /srv/work/alex/layers-test/poky > > > > > > > > And that's it! > > > > > > > > FIXMEs: > > > > - kas config writer not yet implemented > > > > - oe-selftest test cases not yet written > > > > > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > > > --- > > > > meta/lib/bblayers/makesetup.py | 117 > ++++++++++++++++++ > > > > .../templates/setup-layers.py.template | 77 ++++++++++++ > > > > 2 files changed, 194 insertions(+) > > > > create mode 100644 meta/lib/bblayers/makesetup.py > > > > create mode 100644 > meta/lib/bblayers/templates/setup-layers.py.template > > > > > > > > diff --git a/meta/lib/bblayers/makesetup.py > b/meta/lib/bblayers/makesetup.py > > > > new file mode 100644 > > > > index 0000000000..3c86eea3c4 > > > > --- /dev/null > > > > +++ b/meta/lib/bblayers/makesetup.py > > > > @@ -0,0 +1,117 @@ > > > > +# > > > > +# SPDX-License-Identifier: GPL-2.0-only > > > > +# > > > > + > > > > +import logging > > > > +import os > > > > +import stat > > > > +import sys > > > > +import shutil > > > > +import json > > > > + > > > > +import bb.utils > > > > +import bb.process > > > > + > > > > +from bblayers.common import LayerPlugin > > > > + > > > > +logger = logging.getLogger('bitbake-layers') > > > > + > > > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > > > > + > > > > +import oe.buildcfg > > > > + > > > > +def plugin_init(plugins): > > > > + return MakeSetupPlugin() > > > > + > > > > +class MakeSetupPlugin(LayerPlugin): > > > > + > > > > + def _write_python(self, repos, output): > > > > + with open(os.path.join(os.path.dirname(__file__), > "templates", "setup-layers.py.template")) as f: > > > > + template = f.read() > > > > + args = sys.argv > > > > + args[0] = os.path.basename(args[0]) > > > > + script = template.replace('{cmdline}', " > ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, > indent=4)) > > > > + with open(output, 'w') as f: > > > > + f.write(script) > > > > + st = os.stat(output) > > > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | > stat.S_IXOTH) > > > > + > > > > + def _write_json(self, repos, output): > > > > + with open(output, 'w') as f: > > > > + json.dump(repos, f, sort_keys=True, indent=4) > > > > + > > > > + def _write_kas(self, repos, output): > > > > + raise NotImplementedError('Kas config writer not > implemented yet') > > > > + > > > > + _write_config = {"python":_write_python, "json":_write_json, > "kas":_write_kas} > > > > + _output_filename = > {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > > > > + > > > > + def _get_repo_path(self, layer_path): > > > > + repo_path, _ = bb.process.run('git rev-parse > --show-toplevel', cwd=layer_path) > > > > + return repo_path.strip() > > > > + > > > > + def _get_remotes(self, repo_path): > > > > + remotes = [] > > > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > > > > + for r in remotes_list.split(): > > > > + uri,_ = bb.process.run('git remote get-url > {r}'.format(r=r), cwd=repo_path) > > > > + remotes.append({'name':r,'uri':uri.strip()}) > > > > + return remotes > > > > + > > > > + def _get_describe(self, repo_path): > > > > + try: > > > > + describe,_ = bb.process.run('git describe --tags', > cwd=repo_path) > > > > + except bb.process.ExecutionError: > > > > + return "" > > > > + return describe.strip() > > > > + > > > > + def _make_repo_config(self, destdir): > > > > + repos = {} > > > > + layers = > oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > > > > + for l in layers: > > > > + if l[1] == 'workspace': > > > > + continue > > > > + if l[4]: > > > > + logger.error("Layer {name} in {path} has > uncommitted modifications or is not in a git > repository.".format(name=l[1],path=l[0])) > > > > + return > > > > + repo_path = self._get_repo_path(l[0]) > > > > + if repo_path not in repos.keys(): > > > > + repos[repo_path] = {'rev':l[3], 'branch':l[2], > 'remotes':self._get_remotes(repo_path), 'layers':[], > 'describe':self._get_describe(repo_path)} > > > > + if not repos[repo_path]['remotes']: > > > > + logger.error("Layer repository in {path} does > not have any remotes configured. Please add at least one with 'git remote > add'.".format(path=repo_path)) > > > > + return > > > > + if repo_path in os.path.abspath(destdir): > > > > + repos[repo_path]['is_bootstrap'] = True > > > > + > repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > > > > + > > > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > > > > + if len(repo_dirs) > 1: > > > > + logger.error("Layer repositories are not all in the > same parent directory: {repo_dirs}. They need to be relocated into the same > directory.".format(repo_dirs=repo_dirs)) > > > > + return > > > > + > > > > + repos_nopaths = {} > > > > + for r in repos.keys(): > > > > + r_nopath = os.path.basename(r) > > > > + repos_nopaths[r_nopath] = repos[r] > > > > + return repos_nopaths > > > > + > > > > + def do_make_setup(self, args): > > > > + """ Writes out a python script/kas config/json config that > replicates the directory structure and revisions of the layers in a current > build. """ > > > > + repos = self._make_repo_config(args.destdir) > > > > + if not repos: > > > > + return > > > > + output = args.output > > > > + if not output: > > > > + output = self._output_filename[args.format] > > > > + output = os.path.join(os.path.abspath(args.destdir),output) > > > > + self._write_config[args.format](self, repos, output) > > > > + logger.info('Created {}'.format(output)) > > > > + > > > > + def register_commands(self, sp): > > > > + parser_setup_layers = self.add_command(sp, > 'create-layers-setup', self.do_make_setup, parserecipes=False) > > > > + parser_setup_layers.add_argument('destdir', > > > > + help='Directory where to write the output\n(if it is > inside one of the layers, the layer becomes a bootstrap repository and thus > will be excluded from fetching by the script).') > > > > + parser_setup_layers.add_argument('--output', '-o', > > > > + help='File name where to write the output, if the > default (setup-layers.py/.json/.yml) is undesirable.') > > > > + parser_setup_layers.add_argument('--format', '-f', > choices=['python', 'json', 'kas'], default='python', > > > > + help='Format of the output. The options are:\n\tpython > - a self contained python script that fetches all the needed layers and > sets them to correct revisions (default, recommended)\n\tkas - a > configuration file for the kas tool that allows the tool to do the > same\n\tjson - a json formatted file containing all the needed metadata to > do the same by any external or custom tool.') > > > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template > b/meta/lib/bblayers/templates/setup-layers.py.template > > > > new file mode 100644 > > > > index 0000000000..a704ad3d70 > > > > --- /dev/null > > > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > > > > @@ -0,0 +1,77 @@ > > > > +#!/usr/bin/env python3 > > > > +# > > > > +# This file was generated by running > > > > +# > > > > +# {cmdline} > > > > +# > > > > +# It is recommended that you do not modify it directly, but rather > re-run the above command. > > > > +# > > > > + > > > > +layerdata = """ > > > > +{layerdata} > > > > +""" > > > > + > > > > +import argparse > > > > +import json > > > > +import os > > > > +import subprocess > > > > + > > > > +def _do_checkout(args): > > > > + for l_name in layers: > > > > + l_data = layers[l_name] > > > > + if 'is_bootstrap' in l_data.keys(): > > > > + force_arg = > 'force_{}_checkout'.format(l_name.replace('-','_')) > > > > + if not args[force_arg]: > > > > + print('Note: not checking out layer {layer}, use > {layerflag} to override.'.format(layer=l_name, > layerflag='--force-{}-checkout'.format(l_name))) > > > > + continue > > > > + rev = l_data['rev'] > > > > + desc = l_data['describe'] > > > > + if not desc: > > > > + desc = rev[:10] > > > > + branch = l_data['branch'] > > > > + remotes = l_data['remotes'] > > > > + remote = remotes[0] > > > > + if len(remotes) > 1: > > > > + remotechoice = > args['choose_{}_remote'.format(l_name.replace('-','_'))] > > > > + for r in remotes: > > > > + if r['name'] == remotechoice: > > > > + remote = r > > > > + print('Note: multiple remotes defined for layer > {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > > > > + print('Checking out layer {}, revision {}, branch {} from > remote {} at {}'.format(l_name, desc, branch, remote['name'], > remote['uri'])) > > > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > > > > + cwd = args['destdir'] > > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > > + cmd = 'git checkout -q {}'.format(rev) > > > > + cwd = os.path.join(args['destdir'], l_name) > > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > > + > > > > +layers = json.loads(layerdata) > > > > +parser = argparse.ArgumentParser(description='A self contained > python script that fetches all the needed layers and sets them to correct > revisions') > > > > + > > > > +bootstraplayer = None > > > > +for l in layers: > > > > + if 'is_bootstrap' in layers[l]: > > > > + bootstraplayer = l > > > > + > > > > +if bootstraplayer: > > > > + > parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), > action='store_true', > > > > + help='Force the checkout of the bootstrap layer > {bootstraplayer} (by default it is presumed that this script is in it, and > so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > > > > + > > > > +for l in layers: > > > > + remotes = layers[l]['remotes'] > > > > + if len(remotes) > 1: > > > > + > parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] > for r in remotes], default=remotes[0]['name'], > > > > + help='Choose a remote server for layer > {multipleremoteslayer} (default: > {defaultremote})'.format(multipleremoteslayer=l, > defaultremote=remotes[0]['name'])) > > > > + > > > > +try: > > > > + defaultdest = os.path.dirname(subprocess.check_output('git > rev-parse --show-toplevel', text=True, shell=True, > cwd=os.path.dirname(__file__))) > > > > +except subprocess.CalledProcessError as e: > > > > + defaultdest = os.path.abspath(".") > > > > + > > > > +parser.add_argument('--destdir', default=defaultdest, help='Where > to check out the layers (default is > {defaultdest}).'.format(defaultdest=defaultdest)) > > > > + > > > > +args = parser.parse_args() > > > > + > > > > +_do_checkout(vars(args)) > > > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#167543): > https://lists.openembedded.org/g/openembedded-core/message/167543 > Mute This Topic: https://lists.openembedded.org/mt/92117681/924729 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > ticotimo@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Sat, 2 Jul 2022 at 07:58, Tim Orling <ticotimo@gmail.com> wrote:
> One remaining question is how do we handle configuration? This seems to address manifest?
Configuration will be handled with TEMPLATECONF and a set of templates
in a product/platform layer. It's already in core, it's simple to
understand and it works. We used it in a very large project. It also
requires a certain bit of discipline, which is a good thing: you don't
get to implement ugly local.conf hacks in a turing complete language,
and must design your configuration (distro, machines and images)
carefully in a static manner. Then local.conf must contain only DISTRO
and MACHINE, and a 'confuguration' is essentially a choice of those
two plus a set of enabled layers plus site-specific settings for
sstate, downloads, proxies and parallelism level. Back to
(maintainable) basics.
And yes, I didn't want to open a discussion about this, I even didn't
describe upfront how I'm going to do this to a customer who's
supporting the work (if said customer is reading this, I appreciate
the trust). I simply sat down and wrote the code. The principle is:
make a tool that fits in a single commit and that I would want to
personally use.
Alex
On Sat, Jul 2, 2022 at 12:45 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > On Sat, 2 Jul 2022 at 07:58, Tim Orling <ticotimo@gmail.com> wrote: > > One remaining question is how do we handle configuration? This seems to > address manifest? > > Configuration will be handled with TEMPLATECONF and a set of templates > in a product/platform layer. It's already in core, it's simple to > understand and it works. We used it in a very large project. It also > requires a certain bit of discipline, which is a good thing: you don't > get to implement ugly local.conf hacks in a turing complete language, > and must design your configuration (distro, machines and images) > carefully in a static manner. Then local.conf must contain only DISTRO > and MACHINE, and a 'confuguration' is essentially a choice of those > two plus a set of enabled layers plus site-specific settings for > sstate, downloads, proxies and parallelism level. Back to > (maintainable) basics. > 100% on board then. I will die on this mountain. > > And yes, I didn't want to open a discussion about this, I even didn't > describe upfront how I'm going to do this to a customer who's > supporting the work (if said customer is reading this, I appreciate > the trust). I simply sat down and wrote the code. The principle is: > make a tool that fits in a single commit and that I would want to > personally use. > > Yes. 1000% yes. I need a way to quickly spin up layers and configuration often in half a dozen ways per day. > Alex >
On Sat, 2 Jul 2022 at 10:28, Tim Orling <ticotimo@gmail.com> wrote: >> Configuration will be handled with TEMPLATECONF and a set of templates >> in a product/platform layer. It's already in core, it's simple to >> understand and it works. We used it in a very large project. It also >> requires a certain bit of discipline, which is a good thing: you don't >> get to implement ugly local.conf hacks in a turing complete language, >> and must design your configuration (distro, machines and images) >> carefully in a static manner. Then local.conf must contain only DISTRO >> and MACHINE, and a 'confuguration' is essentially a choice of those >> two plus a set of enabled layers plus site-specific settings for >> sstate, downloads, proxies and parallelism level. Back to >> (maintainable) basics. > 100% on board then. I will die on this mountain. Ok, my brain couldn't avoid thinking about this, and so I have to write it out before there's any code, and thus invite more bikeshedding :) The way this will work is that there will be a command: $ bitbake-layers save-build-conf path/where name that will take the existing active configuration from build/conf and save it as a template in the specified location: path/where/name. Then 'bitbake-layers create-layer-setup' will refuse to proceed if it can't find any configuration templates in the active layers. meta-poky/conf that we all know and love and use daily doesn't count, but you can force the template-less setup with a command line switch if you so wish but no karma points for it. All template locations will be saved in the json metadata. Once the layer setup script is done setting up the layers, it can suggest a list of available configurations, and a command that you need to run to use one of them to set up a build. And if there aren't any, it will redirect concerns to the layer maintainers for not helping the layer users. Now this is getting somewhere, and that somewhere is more interesting than sticking with submodules, repo or kas :) Alex
I admit I've been putting off this topic for a long time. I actually wrote up something about it about 5 years ago. I sent it to four people to get some opinions. The six opinions I got back made me despair. i have had a while to think about it though. The challenge is that everyone has a workflow they like today and they will tend to dislike anything that looks different. For that reason I think we need a high level tool which can work with the different approaches. Once we have that, I suspect we'll see some grow stronger and some will wither off, which is probably as it should be through natural evolution. The approach Alex has taken here does head in that direction but I'm not sure it goes quite far enough to get the fraction of users we need on board. I'm not going to comment directly on Alex's proposal at this point, I'd like to put some ideas out there and see what people think of them. I think you'll see that I'm in agreement with the idea/direction but I have a slightly bigger idea of how to do it, which will both be harder to implement but also have a better chance of getting more people on board. So, let me go into a proposal of what I think the tool we need looks like. I propose we create a new tool, lets call it "oe-setup". It is a standalone project in it's own git repo and it's first command would be "init" with a command line looking roughly like: oe-setup init <project-name> oe-setup init <project-name> <target-dir> oe-setup init <project-name> <config-name> <target-dir> oe-setup init <project-name> <config-url> <target-dir> oe-setup init <config-url> <target-dir> The idea being that the repository has some "pre-canned" idea of certain project names, e.g. poky, yoe and whatever else we decide to support "out the box" (criteria tbd). Those project names have a default pre-canned config, or set of configs (e.g. taking branch/series names) so I'd want to see these all work: oe-setup init poky . oe-setup init poky dunfell . oe-setup init poky ./my-local-config.json . oe-setup init poky http://someserver/my-remote-config.json . We'd also allow something not in the default like of project names to be used directly with a url, or maybe added with an "add-project" command. This could work against a user local config file, a bit like git does with global config and adding remotes to a repo. You'll note I haven't made any mention of the tooling these use. The reason is that we don't actually care. I'd propose we teach the tool about a few common standards (kas, submodules, repo) in the form of plugins and then hand off to those tools to do the setup. I'd also propose we develop a "native" form where they perhaps aren't needed. the nice thing is we can have several "native" forms if needed so if one approach isn't working or we need to change it, we can. We may also want to consider an optional "sub-config" parameter which gets passed along to the underlying tool. One of my conclusions after thinking about this for a long time is we have a bootstrap problem. If everyone used git and git worked everywhere, things would be easier. They don't and it doesn't. My evidence? The bitbake fetcher. Much of the ugliness the bitbake fetcher deals with applies to layers as well. Some people need proxy support, some need mirror tarballs, some need floating revisions, some need complete lock down and so on. We could simplify the problem and just say those users "can manage". If we do that, we're giving up at the first hurdle on the idea what we can have a (mostly) universal tool. I'm not sure I'm quite ready to do that. Like most more seasoned developers, I dislike code duplication. We do have the fetcher code, which knows something about these issues and it even has a test suite with decent coverage. Most users already know how to configure it too. Whilst I don't like it particularly, I'd propose we include a chunk of bitbake inside oe-setup (maybe just lib/bb?). How needs discussion. I nearly wrote combo-layer here but people might think I'm serious :). More seriously, git subtree might be a potential option. We could then have bitbake master, or a chunk of it, available to the tool. This would then allow us to potentially get out of a variety of different firewall/mirror situations, if the appropriate backends were right. This gives us the option for fix bugs in the fetcher and update oe- setup and still use specific versions of bitbake for the different releases as needed. I'd propose oe-setup be a different kind of project to bitbake, hopefully something we could make available via pip for example, again to try and help the bootstrap issue for people behind firewalls etc. although I would want it usable standalone too. It would be designed to be a separate standalone tool installed somewhere once by a user rather than associated with a particular build/metadata set like bitbake is. Beyond init, the question is probably updating. There are probably two options, one is an update command to oe-setup and the other is deferring to the underlying tool and it should be possible to use either depending on the underlying tool and the circumstances. As for configuration of the build, I've wondered if we add a new level of config file, basically dedicating a config file "slot" to the setup tooling where config from the setup tool would land (i.e. like auto.conf but dedicated to this). I need to think about this piece a bit more. The other side of this would be the generation of the config(s). I'm less worried about this piece, it would be done by the people who can deal with more complexity and I'm sure we could figure it out. I used "json" in the config urls above but I'm aware that yaml is going to be a discussion there too. From the oe-setup perspective, it may support multiple options since once it knows the tool to pass it off to, it doesn't really need to know more. I'm sure there is more I could write on this topic but I'll stop there and see what people think. Cheers, Richard
Hi Alex Thank you for initiating this important discussion with the code. This could be one way to address this issue. However, the discussion here also shows how complicated the issue is and how fragmented the solutions and opinions are. There are already several tools out there, but none of them has proven to be "the only right way". I'm not sure that writing another tool is really the best approach. The complexity of the proposed tool seems to me to be already at the upper limit, where on the other hand Richard suggests to develop it even further and publish it via pip. At the very least, I see the risk of ending up with just another tool that is very complicated, needs maintenance, but still won't be accepted by the community. Personally I really like to build software as simple as git clone --recursive bitbake my-image Setting up layers is basically just about fetching git repos. I don't see the need for creating some configuration files or other complicated tasks during the initial setup. So before introducing a new tool, please let me understand why git submodules have not been very successful in the past. I see some reasons for that: * In the past, there were different RCS systems and the knowledge about git was not everywhere. I think that has fundamentally changed and the acceptance of git (and also git submodules) has massively increased. Today, git may even be the only version control system that needs to be officially supported to manage bitbake layers. * We still use the submodule structures that Tim mentioned. In general, I agree that using Git submodules is unnecessarily complicated. The challenges start when multiple hierarchies of submodules are used. In this use case, I miss a simple command like "git checkout --recursive" that does everything I currently have to do manually with multiple Git submodules sync, init and update and cd commands. * Probably the lack of a simple, recursive command in git is also the reason why some CI implementations are in rare cases not able to checkout git submodules correctly. Do you think there would be a need for a new tool if: * git submodules would be easy to use? * The Yocto manual would suggest to use git submodules for managing the layers and also provide an example folder and submodules structure as a guide line for the users? * If the knowledge of git had been as widespread a few years ago (when the distributions Tim mentions were published) as it is today? I believe that today it may well be possible to establish git submodules as the recommended solution. (Something like an easy to use "git checkout --recursive" command would certainly helpful.) Since the majority of mostly experienced Yocto/OE developers who are participating this discussion tend to develop a new tool, it makes me wonder if I'm missing something. I see the following use cases where layers need to be fetched: * Initial project setup for working with bitbake. * Retrieving layers from an SDK. (I'm not sure if this should remain something special. The PoC which was recently posted by Alex for bootstrapping the SDK directly from the bitbake environment looks very promising to me). * Fetching the layers on CI infrastructures which often call git fetch with fancy options to improve efficiency. (That would probably not work with a Yocto specific fetch tool anyway.) Do you see other use cases for a layer fetching tool? What do you think about trying to optimize git submodules to handle the "layer fetching" use case with a simple command, rather than developing a new Yocto-specific git wrapper? Is it really useful to generate a configuration for KAS? A tool that generates a configuration for another tool that finally does a Git checkout seems a bit over-engineered to me. At least for us, an implementation based on Git submodules would be usable, which would not be the case for a KAS based implementation. Thank you and regards, Adrian On Fri, 2022-07-01 at 21:24 +0200, Alexander Kanavin wrote: > This addresses a long standing gap in the core offering: > there is no tooling to capture the currently configured layers > with their revisions, or restore the layers from a configuration > file (without using external tools, some of which aren't particularly > suitable for the task). This plugin addresses the gap. > > How to use: > > 1. Saving a layer configuration: > > a) Command line options: > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h > NOTE: Starting bitbake server... > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir > > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. > > positional arguments: > destdir Directory where to write the output > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). > > optional arguments: > -h, --help show this help message and exit > --output OUTPUT, -o OUTPUT > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. > --format {python,json,kas}, -f {python,json,kas} > Format of the output. The options are: > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) > kas - a configuration file for the kas tool that allows the tool to do the same > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. > > b) Running with default choices: > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ > NOTE: Starting bitbake server... > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > 2. Restoring the layers from the saved configuration: > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. > > b) Command line options: > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > optional arguments: > -h, --help show this help message and exit > --force-meta-alex-checkout > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). > --choose-poky-remote {origin,poky-contrib} > Choose a remote server for layer poky (default: origin) > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). > > c) Running with default options: > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel > Note: multiple remotes defined for layer poky, using origin (run with -h to see others). > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky > > And that's it! > > FIXMEs: > - kas config writer not yet implemented > - oe-selftest test cases not yet written > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ > .../templates/setup-layers.py.template | 77 ++++++++++++ > 2 files changed, 194 insertions(+) > create mode 100644 meta/lib/bblayers/makesetup.py > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template > > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py > new file mode 100644 > index 0000000000..3c86eea3c4 > --- /dev/null > +++ b/meta/lib/bblayers/makesetup.py > @@ -0,0 +1,117 @@ > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > +import logging > +import os > +import stat > +import sys > +import shutil > +import json > + > +import bb.utils > +import bb.process > + > +from bblayers.common import LayerPlugin > + > +logger = logging.getLogger('bitbake-layers') > + > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > + > +import oe.buildcfg > + > +def plugin_init(plugins): > + return MakeSetupPlugin() > + > +class MakeSetupPlugin(LayerPlugin): > + > + def _write_python(self, repos, output): > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: > + template = f.read() > + args = sys.argv > + args[0] = os.path.basename(args[0]) > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) > + with open(output, 'w') as f: > + f.write(script) > + st = os.stat(output) > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) > + > + def _write_json(self, repos, output): > + with open(output, 'w') as f: > + json.dump(repos, f, sort_keys=True, indent=4) > + > + def _write_kas(self, repos, output): > + raise NotImplementedError('Kas config writer not implemented yet') > + > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > + > + def _get_repo_path(self, layer_path): > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) > + return repo_path.strip() > + > + def _get_remotes(self, repo_path): > + remotes = [] > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > + for r in remotes_list.split(): > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) > + remotes.append({'name':r,'uri':uri.strip()}) > + return remotes > + > + def _get_describe(self, repo_path): > + try: > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) > + except bb.process.ExecutionError: > + return "" > + return describe.strip() > + > + def _make_repo_config(self, destdir): > + repos = {} > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > + for l in layers: > + if l[1] == 'workspace': > + continue > + if l[4]: > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) > + return > + repo_path = self._get_repo_path(l[0]) > + if repo_path not in repos.keys(): > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} > + if not repos[repo_path]['remotes']: > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) > + return > + if repo_path in os.path.abspath(destdir): > + repos[repo_path]['is_bootstrap'] = True > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > + > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > + if len(repo_dirs) > 1: > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) > + return > + > + repos_nopaths = {} > + for r in repos.keys(): > + r_nopath = os.path.basename(r) > + repos_nopaths[r_nopath] = repos[r] > + return repos_nopaths > + > + def do_make_setup(self, args): > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ > + repos = self._make_repo_config(args.destdir) > + if not repos: > + return > + output = args.output > + if not output: > + output = self._output_filename[args.format] > + output = os.path.join(os.path.abspath(args.destdir),output) > + self._write_config[args.format](self, repos, output) > + logger.info('Created {}'.format(output)) > + > + def register_commands(self, sp): > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) > + parser_setup_layers.add_argument('destdir', > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') > + parser_setup_layers.add_argument('--output', '-o', > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template > new file mode 100644 > index 0000000000..a704ad3d70 > --- /dev/null > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > @@ -0,0 +1,77 @@ > +#!/usr/bin/env python3 > +# > +# This file was generated by running > +# > +# {cmdline} > +# > +# It is recommended that you do not modify it directly, but rather re-run the above command. > +# > + > +layerdata = """ > +{layerdata} > +""" > + > +import argparse > +import json > +import os > +import subprocess > + > +def _do_checkout(args): > + for l_name in layers: > + l_data = layers[l_name] > + if 'is_bootstrap' in l_data.keys(): > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) > + if not args[force_arg]: > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) > + continue > + rev = l_data['rev'] > + desc = l_data['describe'] > + if not desc: > + desc = rev[:10] > + branch = l_data['branch'] > + remotes = l_data['remotes'] > + remote = remotes[0] > + if len(remotes) > 1: > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] > + for r in remotes: > + if r['name'] == remotechoice: > + remote = r > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > + cwd = args['destdir'] > + print("Running '{}' in {}".format(cmd, cwd)) > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > + cmd = 'git checkout -q {}'.format(rev) > + cwd = os.path.join(args['destdir'], l_name) > + print("Running '{}' in {}".format(cmd, cwd)) > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > + > +layers = json.loads(layerdata) > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') > + > +bootstraplayer = None > +for l in layers: > + if 'is_bootstrap' in layers[l]: > + bootstraplayer = l > + > +if bootstraplayer: > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > + > +for l in layers: > + remotes = layers[l]['remotes'] > + if len(remotes) > 1: > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) > + > +try: > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) > +except subprocess.CalledProcessError as e: > + defaultdest = os.path.abspath(".") > + > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > + > +args = parser.parse_args() > + > +_do_checkout(vars(args)) > >
Hello Adrian, I am not proposing a whole new standalone tool, RP is :-) What I am proposing is that once there is an active, working build, it is possible to capture the information about the layer structure in that build and available build configurations in the active layers into a metadata file (json), place that file inside a 'bootstrap layer', and then either let the users handle that file any way they please, or provide a self-contained, self-hosting script inside that same bootstrap layer that replicates the layer structure and sets up a build directory using information from the json. It's not really that complex, if both the script and its generator fit inside a single commit :) This is the value over git submodules: a) the layer structure and list of available build configurations can be saved with a single command if that command is using tinfoil to get the needed data about layers from a currently active build. No need to write anything by hand, or make scripts that work only with your specific product, and need to be adjusted every time layer setup changes. There's no way we can add this functionality to git :-) b) we can place anything that can be useful to layer users inside the metadata. Git submodules do not capture any of this layer-specific information (such as what branch we're on? what is the latest tag in that branch? what are other remotes that we can fetch from? where are the templates for build configurations and can I see their descriptions? and so on). All they allow is a single remote, and a single, cryptic revision on that remote for each of the layers. I've worked with that in a customer project, and I disliked it for lack of visibility into what am I actually checking out, and how do I actually set up a build once I ran the checkout (which is another thing that git will never do for you). Does this make sense? Did you try the code? Did you look at what is in the json? Maybe that helps to understand where I want to head with this. Alex On Mon, 4 Jul 2022 at 11:01, Adrian Freihofer <adrian.freihofer@gmail.com> wrote: > > Hi Alex > > Thank you for initiating this important discussion with the code. This > could be one way to address this issue. However, the discussion here > also shows how complicated the issue is and how fragmented the > solutions and opinions are. There are already several tools out there, > but none of them has proven to be "the only right way". I'm not sure > that writing another tool is really the best approach. The complexity > of the proposed tool seems to me to be already at the upper limit, > where on the other hand Richard suggests to develop it even further and > publish it via pip. At the very least, I see the risk of ending up with > just another tool that is very complicated, needs maintenance, but > still won't be accepted by the community. > > Personally I really like to build software as simple as > git clone --recursive > bitbake my-image > > Setting up layers is basically just about fetching git repos. I don't > see the need for creating some configuration files or other complicated > tasks during the initial setup. So before introducing a new tool, > please let me understand why git submodules have not been very > successful in the past. I see some reasons for that: > * In the past, there were different RCS systems and the knowledge > about git was not everywhere. I think that has fundamentally changed > and the acceptance of git (and also git submodules) has massively > increased. Today, git may even be the only version control system > that needs to be officially supported to manage bitbake layers. > * We still use the submodule structures that Tim mentioned. In > general, I agree that using Git submodules is unnecessarily > complicated. The challenges start when multiple hierarchies of > submodules are used. In this use case, I miss a simple command like > "git checkout --recursive" that does everything I currently have to > do manually with multiple Git submodules sync, init and update and > cd commands. > * Probably the lack of a simple, recursive command in git is also the > reason why some CI implementations are in rare cases not able to > checkout git submodules correctly. > > Do you think there would be a need for a new tool if: > * git submodules would be easy to use? > * The Yocto manual would suggest to use git submodules for managing > the layers and also provide an example folder and submodules > structure as a guide line for the users? > * If the knowledge of git had been as widespread a few years ago (when > the distributions Tim mentions were published) as it is today? > I believe that today it may well be possible to establish git > submodules as the recommended solution. (Something like an easy to use > "git checkout --recursive" command would certainly helpful.) > > Since the majority of mostly experienced Yocto/OE developers who are > participating this discussion tend to develop a new tool, it makes me > wonder if I'm missing something. I see the following use cases where > layers need to be fetched: > * Initial project setup for working with bitbake. > * Retrieving layers from an SDK. (I'm not sure if this should remain > something special. The PoC which was recently posted by Alex for > bootstrapping the SDK directly from the bitbake environment looks > very promising to me). > * Fetching the layers on CI infrastructures which often call git fetch > with fancy options to improve efficiency. (That would probably not > work with a Yocto specific fetch tool anyway.) > Do you see other use cases for a layer fetching tool? > > What do you think about trying to optimize git submodules to handle the > "layer fetching" use case with a simple command, rather than developing > a new Yocto-specific git wrapper? > > Is it really useful to generate a configuration for KAS? A tool that > generates a configuration for another tool that finally does a Git > checkout seems a bit over-engineered to me. At least for us, an > implementation based on Git submodules would be usable, which would not > be the case for a KAS based implementation. > > Thank you and regards, > Adrian > > > On Fri, 2022-07-01 at 21:24 +0200, Alexander Kanavin wrote: > > This addresses a long standing gap in the core offering: > > there is no tooling to capture the currently configured layers > > with their revisions, or restore the layers from a configuration > > file (without using external tools, some of which aren't particularly > > suitable for the task). This plugin addresses the gap. > > > > How to use: > > > > 1. Saving a layer configuration: > > > > a) Command line options: > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h > > NOTE: Starting bitbake server... > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir > > > > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. > > > > positional arguments: > > destdir Directory where to write the output > > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). > > > > optional arguments: > > -h, --help show this help message and exit > > --output OUTPUT, -o OUTPUT > > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. > > --format {python,json,kas}, -f {python,json,kas} > > Format of the output. The options are: > > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) > > kas - a configuration file for the kas tool that allows the tool to do the same > > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. > > > > b) Running with default choices: > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ > > NOTE: Starting bitbake server... > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > > > 2. Restoring the layers from the saved configuration: > > > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. > > > > b) Command line options: > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > > > optional arguments: > > -h, --help show this help message and exit > > --force-meta-alex-checkout > > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). > > --choose-poky-remote {origin,poky-contrib} > > Choose a remote server for layer poky (default: origin) > > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). > > > > c) Running with default options: > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. > > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel > > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel > > Note: multiple remotes defined for layer poky, using origin (run with -h to see others). > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky > > > > And that's it! > > > > FIXMEs: > > - kas config writer not yet implemented > > - oe-selftest test cases not yet written > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > --- > > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ > > .../templates/setup-layers.py.template | 77 ++++++++++++ > > 2 files changed, 194 insertions(+) > > create mode 100644 meta/lib/bblayers/makesetup.py > > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template > > > > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py > > new file mode 100644 > > index 0000000000..3c86eea3c4 > > --- /dev/null > > +++ b/meta/lib/bblayers/makesetup.py > > @@ -0,0 +1,117 @@ > > +# > > +# SPDX-License-Identifier: GPL-2.0-only > > +# > > + > > +import logging > > +import os > > +import stat > > +import sys > > +import shutil > > +import json > > + > > +import bb.utils > > +import bb.process > > + > > +from bblayers.common import LayerPlugin > > + > > +logger = logging.getLogger('bitbake-layers') > > + > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > > + > > +import oe.buildcfg > > + > > +def plugin_init(plugins): > > + return MakeSetupPlugin() > > + > > +class MakeSetupPlugin(LayerPlugin): > > + > > + def _write_python(self, repos, output): > > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: > > + template = f.read() > > + args = sys.argv > > + args[0] = os.path.basename(args[0]) > > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) > > + with open(output, 'w') as f: > > + f.write(script) > > + st = os.stat(output) > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) > > + > > + def _write_json(self, repos, output): > > + with open(output, 'w') as f: > > + json.dump(repos, f, sort_keys=True, indent=4) > > + > > + def _write_kas(self, repos, output): > > + raise NotImplementedError('Kas config writer not implemented yet') > > + > > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} > > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > > + > > + def _get_repo_path(self, layer_path): > > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) > > + return repo_path.strip() > > + > > + def _get_remotes(self, repo_path): > > + remotes = [] > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > > + for r in remotes_list.split(): > > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) > > + remotes.append({'name':r,'uri':uri.strip()}) > > + return remotes > > + > > + def _get_describe(self, repo_path): > > + try: > > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) > > + except bb.process.ExecutionError: > > + return "" > > + return describe.strip() > > + > > + def _make_repo_config(self, destdir): > > + repos = {} > > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > > + for l in layers: > > + if l[1] == 'workspace': > > + continue > > + if l[4]: > > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) > > + return > > + repo_path = self._get_repo_path(l[0]) > > + if repo_path not in repos.keys(): > > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} > > + if not repos[repo_path]['remotes']: > > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) > > + return > > + if repo_path in os.path.abspath(destdir): > > + repos[repo_path]['is_bootstrap'] = True > > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > > + > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > > + if len(repo_dirs) > 1: > > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) > > + return > > + > > + repos_nopaths = {} > > + for r in repos.keys(): > > + r_nopath = os.path.basename(r) > > + repos_nopaths[r_nopath] = repos[r] > > + return repos_nopaths > > + > > + def do_make_setup(self, args): > > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ > > + repos = self._make_repo_config(args.destdir) > > + if not repos: > > + return > > + output = args.output > > + if not output: > > + output = self._output_filename[args.format] > > + output = os.path.join(os.path.abspath(args.destdir),output) > > + self._write_config[args.format](self, repos, output) > > + logger.info('Created {}'.format(output)) > > + > > + def register_commands(self, sp): > > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) > > + parser_setup_layers.add_argument('destdir', > > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') > > + parser_setup_layers.add_argument('--output', '-o', > > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') > > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', > > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template > > new file mode 100644 > > index 0000000000..a704ad3d70 > > --- /dev/null > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > > @@ -0,0 +1,77 @@ > > +#!/usr/bin/env python3 > > +# > > +# This file was generated by running > > +# > > +# {cmdline} > > +# > > +# It is recommended that you do not modify it directly, but rather re-run the above command. > > +# > > + > > +layerdata = """ > > +{layerdata} > > +""" > > + > > +import argparse > > +import json > > +import os > > +import subprocess > > + > > +def _do_checkout(args): > > + for l_name in layers: > > + l_data = layers[l_name] > > + if 'is_bootstrap' in l_data.keys(): > > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) > > + if not args[force_arg]: > > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) > > + continue > > + rev = l_data['rev'] > > + desc = l_data['describe'] > > + if not desc: > > + desc = rev[:10] > > + branch = l_data['branch'] > > + remotes = l_data['remotes'] > > + remote = remotes[0] > > + if len(remotes) > 1: > > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] > > + for r in remotes: > > + if r['name'] == remotechoice: > > + remote = r > > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > > + cwd = args['destdir'] > > + print("Running '{}' in {}".format(cmd, cwd)) > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > + cmd = 'git checkout -q {}'.format(rev) > > + cwd = os.path.join(args['destdir'], l_name) > > + print("Running '{}' in {}".format(cmd, cwd)) > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > + > > +layers = json.loads(layerdata) > > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') > > + > > +bootstraplayer = None > > +for l in layers: > > + if 'is_bootstrap' in layers[l]: > > + bootstraplayer = l > > + > > +if bootstraplayer: > > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', > > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > > + > > +for l in layers: > > + remotes = layers[l]['remotes'] > > + if len(remotes) > 1: > > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], > > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) > > + > > +try: > > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) > > +except subprocess.CalledProcessError as e: > > + defaultdest = os.path.abspath(".") > > + > > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > > + > > +args = parser.parse_args() > > + > > +_do_checkout(vars(args)) > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#167540): https://lists.openembedded.org/g/openembedded-core/message/167540 > > Mute This Topic: https://lists.openembedded.org/mt/92117681/4454582 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [adrian.freihofer@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > >
On Mon, 2022-07-04 at 11:01 +0200, Adrian Freihofer wrote: > Thank you for initiating this important discussion with the code. This > could be one way to address this issue. However, the discussion here > also shows how complicated the issue is and how fragmented the > solutions and opinions are. There are already several tools out there, > but none of them has proven to be "the only right way". I'm not sure > that writing another tool is really the best approach. The complexity > of the proposed tool seems to me to be already at the upper limit, > where on the other hand Richard suggests to develop it even further and > publish it via pip. At the very least, I see the risk of ending up with > just another tool that is very complicated, needs maintenance, but > still won't be accepted by the community. > > Personally I really like to build software as simple as > git clone --recursive > bitbake my-image This simply can't work, unless you have a different parent git repo for each configuration you're going to share with users. It can work for a single configuration (e.g. set of layers) only. That works well for your single company single use case, it doesn't work well for developers dealing with multiple layers/configurations. > Setting up layers is basically just about fetching git repos. I don't > see the need for creating some configuration files or other complicated > tasks during the initial setup. So before introducing a new tool, > please let me understand why git submodules have not been very > successful in the past. I see some reasons for that: > * In the past, there were different RCS systems and the knowledge > about git was not everywhere. I think that has fundamentally changed > and the acceptance of git (and also git submodules) has massively > increased. Today, git may even be the only version control system > that needs to be officially supported to manage bitbake layers. I think at this point we're only considering git. That said, there are submodules, repo, subtree and outliers like combo-layer so even limited to git, this isn't straightforward. > * We still use the submodule structures that Tim mentioned. In > general, I agree that using Git submodules is unnecessarily > complicated. The challenges start when multiple hierarchies of > submodules are used. In this use case, I miss a simple command like > "git checkout --recursive" that does everything I currently have to > do manually with multiple Git submodules sync, init and update and > cd commands. > * Probably the lack of a simple, recursive command in git is also the > reason why some CI implementations are in rare cases not able to > checkout git submodules correctly. > > Do you think there would be a need for a new tool if: > * git submodules would be easy to use? > * The Yocto manual would suggest to use git submodules for managing > the layers and also provide an example folder and submodules > structure as a guide line for the users? The Yocto manual cannot mandate submodules. They work for some people, others despise them. > * If the knowledge of git had been as widespread a few years ago (when > the distributions Tim mentions were published) as it is today? > I believe that today it may well be possible to establish git > submodules as the recommended solution. (Something like an easy to use > "git checkout --recursive" command would certainly helpful.) If people were going to standardise on that it would have happened by now. It hasn't and I don't think it will. They work well for some usecases, people want/need/like other tools in other cases. > Since the majority of mostly experienced Yocto/OE developers who are > participating this discussion tend to develop a new tool, it makes me > wonder if I'm missing something. I see the following use cases where > layers need to be fetched: > * Initial project setup for working with bitbake. > * Retrieving layers from an SDK. (I'm not sure if this should remain > something special. The PoC which was recently posted by Alex for > bootstrapping the SDK directly from the bitbake environment looks > very promising to me). > * Fetching the layers on CI infrastructures which often call git fetch > with fancy options to improve efficiency. (That would probably not > work with a Yocto specific fetch tool anyway.) There would be some value in having a common shared way of doing this should as many people need to handle CI at some point. > Do you see other use cases for a layer fetching tool? > > What do you think about trying to optimize git submodules to handle the > "layer fetching" use case with a simple command, rather than developing > a new Yocto-specific git wrapper? I think you'll end up with repo :). There are definitely Yocto specific elements to this which is why we're seeing specific tools. > Is it really useful to generate a configuration for KAS? A tool that > generates a configuration for another tool that finally does a Git > checkout seems a bit over-engineered to me. At least for us, an > implementation based on Git submodules would be usable, which would not > be the case for a KAS based implementation. If the alternative is reinventing kas then it could make sense. See my email for why I think we need something higher level than these tools, which is implementation independent (as much as it can be). Cheers, Richard
On 04/07/2022 11:01:42+0200, Adrian Freihofer wrote: > Hi Alex > > Thank you for initiating this important discussion with the code. This > could be one way to address this issue. However, the discussion here > also shows how complicated the issue is and how fragmented the > solutions and opinions are. There are already several tools out there, > but none of them has proven to be "the only right way". I'm not sure > that writing another tool is really the best approach. The complexity > of the proposed tool seems to me to be already at the upper limit, > where on the other hand Richard suggests to develop it even further and > publish it via pip. At the very least, I see the risk of ending up with > just another tool that is very complicated, needs maintenance, but > still won't be accepted by the community. > > Personally I really like to build software as simple as > git clone --recursive > bitbake my-image > > Setting up layers is basically just about fetching git repos. I don't > see the need for creating some configuration files or other complicated > tasks during the initial setup. So before introducing a new tool, > please let me understand why git submodules have not been very > successful in the past. I see some reasons for that: > * In the past, there were different RCS systems and the knowledge > about git was not everywhere. I think that has fundamentally changed > and the acceptance of git (and also git submodules) has massively > increased. Today, git may even be the only version control system > that needs to be officially supported to manage bitbake layers. > * We still use the submodule structures that Tim mentioned. In > general, I agree that using Git submodules is unnecessarily > complicated. The challenges start when multiple hierarchies of > submodules are used. In this use case, I miss a simple command like > "git checkout --recursive" that does everything I currently have to > do manually with multiple Git submodules sync, init and update and > cd commands. > * Probably the lack of a simple, recursive command in git is also the > reason why some CI implementations are in rare cases not able to > checkout git submodules correctly. > > Do you think there would be a need for a new tool if: > * git submodules would be easy to use? > * The Yocto manual would suggest to use git submodules for managing > the layers and also provide an example folder and submodules > structure as a guide line for the users? > * If the knowledge of git had been as widespread a few years ago (when > the distributions Tim mentions were published) as it is today? > I believe that today it may well be possible to establish git > submodules as the recommended solution. (Something like an easy to use > "git checkout --recursive" command would certainly helpful.) > > Since the majority of mostly experienced Yocto/OE developers who are > participating this discussion tend to develop a new tool, it makes me > wonder if I'm missing something. I see the following use cases where > layers need to be fetched: > * Initial project setup for working with bitbake. > * Retrieving layers from an SDK. (I'm not sure if this should remain > something special. The PoC which was recently posted by Alex for > bootstrapping the SDK directly from the bitbake environment looks > very promising to me). > * Fetching the layers on CI infrastructures which often call git fetch > with fancy options to improve efficiency. (That would probably not > work with a Yocto specific fetch tool anyway.) > Do you see other use cases for a layer fetching tool? > > What do you think about trying to optimize git submodules to handle the > "layer fetching" use case with a simple command, rather than developing > a new Yocto-specific git wrapper? > > Is it really useful to generate a configuration for KAS? A tool that > generates a configuration for another tool that finally does a Git > checkout seems a bit over-engineered to me. At least for us, an > implementation based on Git submodules would be usable, which would not > be the case for a KAS based implementation. > I think the use case you are missing and that you agree just above that it is not well supported using git submodule is tagging a release and moving between those tagged releases. With git submodules, it is a mess and it is very easy to make mistakes or forget to update a module. > Thank you and regards, > Adrian > > > On Fri, 2022-07-01 at 21:24 +0200, Alexander Kanavin wrote: > > This addresses a long standing gap in the core offering: > > there is no tooling to capture the currently configured layers > > with their revisions, or restore the layers from a configuration > > file (without using external tools, some of which aren't particularly > > suitable for the task). This plugin addresses the gap. > > > > How to use: > > > > 1. Saving a layer configuration: > > > > a) Command line options: > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h > > NOTE: Starting bitbake server... > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir > > > > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. > > > > positional arguments: > > destdir Directory where to write the output > > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). > > > > optional arguments: > > -h, --help show this help message and exit > > --output OUTPUT, -o OUTPUT > > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. > > --format {python,json,kas}, -f {python,json,kas} > > Format of the output. The options are: > > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) > > kas - a configuration file for the kas tool that allows the tool to do the same > > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. > > > > b) Running with default choices: > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ > > NOTE: Starting bitbake server... > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > > > 2. Restoring the layers from the saved configuration: > > > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. > > > > b) Command line options: > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > > > optional arguments: > > -h, --help show this help message and exit > > --force-meta-alex-checkout > > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). > > --choose-poky-remote {origin,poky-contrib} > > Choose a remote server for layer poky (default: origin) > > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). > > > > c) Running with default options: > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. > > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel > > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel > > Note: multiple remotes defined for layer poky, using origin (run with -h to see others). > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky > > > > And that's it! > > > > FIXMEs: > > - kas config writer not yet implemented > > - oe-selftest test cases not yet written > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > --- > > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ > > .../templates/setup-layers.py.template | 77 ++++++++++++ > > 2 files changed, 194 insertions(+) > > create mode 100644 meta/lib/bblayers/makesetup.py > > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template > > > > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py > > new file mode 100644 > > index 0000000000..3c86eea3c4 > > --- /dev/null > > +++ b/meta/lib/bblayers/makesetup.py > > @@ -0,0 +1,117 @@ > > +# > > +# SPDX-License-Identifier: GPL-2.0-only > > +# > > + > > +import logging > > +import os > > +import stat > > +import sys > > +import shutil > > +import json > > + > > +import bb.utils > > +import bb.process > > + > > +from bblayers.common import LayerPlugin > > + > > +logger = logging.getLogger('bitbake-layers') > > + > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > > + > > +import oe.buildcfg > > + > > +def plugin_init(plugins): > > + return MakeSetupPlugin() > > + > > +class MakeSetupPlugin(LayerPlugin): > > + > > + def _write_python(self, repos, output): > > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: > > + template = f.read() > > + args = sys.argv > > + args[0] = os.path.basename(args[0]) > > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) > > + with open(output, 'w') as f: > > + f.write(script) > > + st = os.stat(output) > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) > > + > > + def _write_json(self, repos, output): > > + with open(output, 'w') as f: > > + json.dump(repos, f, sort_keys=True, indent=4) > > + > > + def _write_kas(self, repos, output): > > + raise NotImplementedError('Kas config writer not implemented yet') > > + > > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} > > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > > + > > + def _get_repo_path(self, layer_path): > > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) > > + return repo_path.strip() > > + > > + def _get_remotes(self, repo_path): > > + remotes = [] > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > > + for r in remotes_list.split(): > > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) > > + remotes.append({'name':r,'uri':uri.strip()}) > > + return remotes > > + > > + def _get_describe(self, repo_path): > > + try: > > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) > > + except bb.process.ExecutionError: > > + return "" > > + return describe.strip() > > + > > + def _make_repo_config(self, destdir): > > + repos = {} > > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > > + for l in layers: > > + if l[1] == 'workspace': > > + continue > > + if l[4]: > > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) > > + return > > + repo_path = self._get_repo_path(l[0]) > > + if repo_path not in repos.keys(): > > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} > > + if not repos[repo_path]['remotes']: > > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) > > + return > > + if repo_path in os.path.abspath(destdir): > > + repos[repo_path]['is_bootstrap'] = True > > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > > + > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > > + if len(repo_dirs) > 1: > > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) > > + return > > + > > + repos_nopaths = {} > > + for r in repos.keys(): > > + r_nopath = os.path.basename(r) > > + repos_nopaths[r_nopath] = repos[r] > > + return repos_nopaths > > + > > + def do_make_setup(self, args): > > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ > > + repos = self._make_repo_config(args.destdir) > > + if not repos: > > + return > > + output = args.output > > + if not output: > > + output = self._output_filename[args.format] > > + output = os.path.join(os.path.abspath(args.destdir),output) > > + self._write_config[args.format](self, repos, output) > > + logger.info('Created {}'.format(output)) > > + > > + def register_commands(self, sp): > > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) > > + parser_setup_layers.add_argument('destdir', > > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') > > + parser_setup_layers.add_argument('--output', '-o', > > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') > > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', > > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template > > new file mode 100644 > > index 0000000000..a704ad3d70 > > --- /dev/null > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > > @@ -0,0 +1,77 @@ > > +#!/usr/bin/env python3 > > +# > > +# This file was generated by running > > +# > > +# {cmdline} > > +# > > +# It is recommended that you do not modify it directly, but rather re-run the above command. > > +# > > + > > +layerdata = """ > > +{layerdata} > > +""" > > + > > +import argparse > > +import json > > +import os > > +import subprocess > > + > > +def _do_checkout(args): > > + for l_name in layers: > > + l_data = layers[l_name] > > + if 'is_bootstrap' in l_data.keys(): > > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) > > + if not args[force_arg]: > > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) > > + continue > > + rev = l_data['rev'] > > + desc = l_data['describe'] > > + if not desc: > > + desc = rev[:10] > > + branch = l_data['branch'] > > + remotes = l_data['remotes'] > > + remote = remotes[0] > > + if len(remotes) > 1: > > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] > > + for r in remotes: > > + if r['name'] == remotechoice: > > + remote = r > > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > > + cwd = args['destdir'] > > + print("Running '{}' in {}".format(cmd, cwd)) > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > + cmd = 'git checkout -q {}'.format(rev) > > + cwd = os.path.join(args['destdir'], l_name) > > + print("Running '{}' in {}".format(cmd, cwd)) > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > + > > +layers = json.loads(layerdata) > > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') > > + > > +bootstraplayer = None > > +for l in layers: > > + if 'is_bootstrap' in layers[l]: > > + bootstraplayer = l > > + > > +if bootstraplayer: > > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', > > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > > + > > +for l in layers: > > + remotes = layers[l]['remotes'] > > + if len(remotes) > 1: > > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], > > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) > > + > > +try: > > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) > > +except subprocess.CalledProcessError as e: > > + defaultdest = os.path.abspath(".") > > + > > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > > + > > +args = parser.parse_args() > > + > > +_do_checkout(vars(args)) > > > > > > > >
Hello Alex After the discussion turned to the pros and cons of submodules, I was thinking more about the initial setup of the Bitbake environment. The json output of your script has something to do with this topic, but the generated Python script does not. The Python script is meant to replicate a bitbake setup to bootstrap an SDK environment. It is more or less a replacement for the eSDK shell script installer. On Mon, 2022-07-04 at 11:59 +0200, Alexander Kanavin wrote: > Hello Adrian, > > I am not proposing a whole new standalone tool, RP is :-) What I am > proposing is that once there is an active, working build, it is > possible to capture the information about the layer structure in that > build and available build configurations in the active layers into a > metadata file (json), place that file inside a 'bootstrap layer', and > then either let the users handle that file any way they please, or > provide a self-contained, self-hosting script inside that same > bootstrap layer that replicates the layer structure and sets up a > build directory using information from the json. It's not really that > complex, if both the script and its generator fit inside a single > commit :) > > This is the value over git submodules: > a) the layer structure and list of available build configurations can > be saved with a single command if that command is using tinfoil to get > the needed data about layers from a currently active build. No need to > write anything by hand, or make scripts that work only with your > specific product, and need to be adjusted every time layer setup > changes. There's no way we can add this functionality to git :-) > > b) we can place anything that can be useful to layer users inside the > metadata. Git submodules do not capture any of this layer-specific > information (such as what branch we're on? what is the latest tag in > that branch? what are other remotes that we can fetch from? where are > the templates for build configurations and can I see their > descriptions? and so on). All they allow is a single remote, and a > single, cryptic revision on that remote for each of the layers. I've > worked with that in a customer project, and I disliked it for lack of > visibility into what am I actually checking out, and how do I actually > set up a build once I ran the checkout (which is another thing that > git will never do for you). > > Does this make sense? Did you try the code? Did you look at what is in > the json? Maybe that helps to understand where I want to head with > this. Yes, I did. It works nicely for simpler folder structures but I was not able to get it working with more complicated layer setups. I got "ERROR: Layer repositories are not all in the same parent directory". One more detail is that it expects the output folder to be there instead of just creating it if not. It looks really promising to me and I think it will also be appreciated by git submodule fans as an alternative for populate_sdk_ext. Thank you very much. Regards, Adrian > > Alex > > On Mon, 4 Jul 2022 at 11:01, Adrian Freihofer > <adrian.freihofer@gmail.com> wrote: > > > > Hi Alex > > > > Thank you for initiating this important discussion with the code. This > > could be one way to address this issue. However, the discussion here > > also shows how complicated the issue is and how fragmented the > > solutions and opinions are. There are already several tools out there, > > but none of them has proven to be "the only right way". I'm not sure > > that writing another tool is really the best approach. The complexity > > of the proposed tool seems to me to be already at the upper limit, > > where on the other hand Richard suggests to develop it even further and > > publish it via pip. At the very least, I see the risk of ending up with > > just another tool that is very complicated, needs maintenance, but > > still won't be accepted by the community. > > > > Personally I really like to build software as simple as > > git clone --recursive > > bitbake my-image > > > > Setting up layers is basically just about fetching git repos. I don't > > see the need for creating some configuration files or other complicated > > tasks during the initial setup. So before introducing a new tool, > > please let me understand why git submodules have not been very > > successful in the past. I see some reasons for that: > > * In the past, there were different RCS systems and the knowledge > > about git was not everywhere. I think that has fundamentally changed > > and the acceptance of git (and also git submodules) has massively > > increased. Today, git may even be the only version control system > > that needs to be officially supported to manage bitbake layers. > > * We still use the submodule structures that Tim mentioned. In > > general, I agree that using Git submodules is unnecessarily > > complicated. The challenges start when multiple hierarchies of > > submodules are used. In this use case, I miss a simple command like > > "git checkout --recursive" that does everything I currently have to > > do manually with multiple Git submodules sync, init and update and > > cd commands. > > * Probably the lack of a simple, recursive command in git is also the > > reason why some CI implementations are in rare cases not able to > > checkout git submodules correctly. > > > > Do you think there would be a need for a new tool if: > > * git submodules would be easy to use? > > * The Yocto manual would suggest to use git submodules for managing > > the layers and also provide an example folder and submodules > > structure as a guide line for the users? > > * If the knowledge of git had been as widespread a few years ago (when > > the distributions Tim mentions were published) as it is today? > > I believe that today it may well be possible to establish git > > submodules as the recommended solution. (Something like an easy to use > > "git checkout --recursive" command would certainly helpful.) > > > > Since the majority of mostly experienced Yocto/OE developers who are > > participating this discussion tend to develop a new tool, it makes me > > wonder if I'm missing something. I see the following use cases where > > layers need to be fetched: > > * Initial project setup for working with bitbake. > > * Retrieving layers from an SDK. (I'm not sure if this should remain > > something special. The PoC which was recently posted by Alex for > > bootstrapping the SDK directly from the bitbake environment looks > > very promising to me). > > * Fetching the layers on CI infrastructures which often call git fetch > > with fancy options to improve efficiency. (That would probably not > > work with a Yocto specific fetch tool anyway.) > > Do you see other use cases for a layer fetching tool? > > > > What do you think about trying to optimize git submodules to handle the > > "layer fetching" use case with a simple command, rather than developing > > a new Yocto-specific git wrapper? > > > > Is it really useful to generate a configuration for KAS? A tool that > > generates a configuration for another tool that finally does a Git > > checkout seems a bit over-engineered to me. At least for us, an > > implementation based on Git submodules would be usable, which would not > > be the case for a KAS based implementation. > > > > Thank you and regards, > > Adrian > > > > > > On Fri, 2022-07-01 at 21:24 +0200, Alexander Kanavin wrote: > > > This addresses a long standing gap in the core offering: > > > there is no tooling to capture the currently configured layers > > > with their revisions, or restore the layers from a configuration > > > file (without using external tools, some of which aren't particularly > > > suitable for the task). This plugin addresses the gap. > > > > > > How to use: > > > > > > 1. Saving a layer configuration: > > > > > > a) Command line options: > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h > > > NOTE: Starting bitbake server... > > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir > > > > > > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. > > > > > > positional arguments: > > > destdir Directory where to write the output > > > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > --output OUTPUT, -o OUTPUT > > > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. > > > --format {python,json,kas}, -f {python,json,kas} > > > Format of the output. The options are: > > > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) > > > kas - a configuration file for the kas tool that allows the tool to do the same > > > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. > > > > > > b) Running with default choices: > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ > > > NOTE: Starting bitbake server... > > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > > > > > 2. Restoring the layers from the saved configuration: > > > > > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. > > > > > > b) Command line options: > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > > > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > > > > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > --force-meta-alex-checkout > > > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). > > > --choose-poky-remote {origin,poky-contrib} > > > Choose a remote server for layer poky (default: origin) > > > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). > > > > > > c) Running with default options: > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > > > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. > > > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel > > > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test > > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel > > > Note: multiple remotes defined for layer poky, using origin (run with -h to see others). > > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky > > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test > > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky > > > > > > And that's it! > > > > > > FIXMEs: > > > - kas config writer not yet implemented > > > - oe-selftest test cases not yet written > > > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > > --- > > > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ > > > .../templates/setup-layers.py.template | 77 ++++++++++++ > > > 2 files changed, 194 insertions(+) > > > create mode 100644 meta/lib/bblayers/makesetup.py > > > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template > > > > > > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py > > > new file mode 100644 > > > index 0000000000..3c86eea3c4 > > > --- /dev/null > > > +++ b/meta/lib/bblayers/makesetup.py > > > @@ -0,0 +1,117 @@ > > > +# > > > +# SPDX-License-Identifier: GPL-2.0-only > > > +# > > > + > > > +import logging > > > +import os > > > +import stat > > > +import sys > > > +import shutil > > > +import json > > > + > > > +import bb.utils > > > +import bb.process > > > + > > > +from bblayers.common import LayerPlugin > > > + > > > +logger = logging.getLogger('bitbake-layers') > > > + > > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > > > + > > > +import oe.buildcfg > > > + > > > +def plugin_init(plugins): > > > + return MakeSetupPlugin() > > > + > > > +class MakeSetupPlugin(LayerPlugin): > > > + > > > + def _write_python(self, repos, output): > > > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: > > > + template = f.read() > > > + args = sys.argv > > > + args[0] = os.path.basename(args[0]) > > > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) > > > + with open(output, 'w') as f: > > > + f.write(script) > > > + st = os.stat(output) > > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) > > > + > > > + def _write_json(self, repos, output): > > > + with open(output, 'w') as f: > > > + json.dump(repos, f, sort_keys=True, indent=4) > > > + > > > + def _write_kas(self, repos, output): > > > + raise NotImplementedError('Kas config writer not implemented yet') > > > + > > > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} > > > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > > > + > > > + def _get_repo_path(self, layer_path): > > > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) > > > + return repo_path.strip() > > > + > > > + def _get_remotes(self, repo_path): > > > + remotes = [] > > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > > > + for r in remotes_list.split(): > > > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) > > > + remotes.append({'name':r,'uri':uri.strip()}) > > > + return remotes > > > + > > > + def _get_describe(self, repo_path): > > > + try: > > > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) > > > + except bb.process.ExecutionError: > > > + return "" > > > + return describe.strip() > > > + > > > + def _make_repo_config(self, destdir): > > > + repos = {} > > > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > > > + for l in layers: > > > + if l[1] == 'workspace': > > > + continue > > > + if l[4]: > > > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) > > > + return > > > + repo_path = self._get_repo_path(l[0]) > > > + if repo_path not in repos.keys(): > > > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} > > > + if not repos[repo_path]['remotes']: > > > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) > > > + return > > > + if repo_path in os.path.abspath(destdir): > > > + repos[repo_path]['is_bootstrap'] = True > > > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > > > + > > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > > > + if len(repo_dirs) > 1: > > > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) > > > + return > > > + > > > + repos_nopaths = {} > > > + for r in repos.keys(): > > > + r_nopath = os.path.basename(r) > > > + repos_nopaths[r_nopath] = repos[r] > > > + return repos_nopaths > > > + > > > + def do_make_setup(self, args): > > > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ > > > + repos = self._make_repo_config(args.destdir) > > > + if not repos: > > > + return > > > + output = args.output > > > + if not output: > > > + output = self._output_filename[args.format] > > > + output = os.path.join(os.path.abspath(args.destdir),output) > > > + self._write_config[args.format](self, repos, output) > > > + logger.info('Created {}'.format(output)) > > > + > > > + def register_commands(self, sp): > > > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) > > > + parser_setup_layers.add_argument('destdir', > > > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') > > > + parser_setup_layers.add_argument('--output', '-o', > > > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') > > > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', > > > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') > > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template > > > new file mode 100644 > > > index 0000000000..a704ad3d70 > > > --- /dev/null > > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > > > @@ -0,0 +1,77 @@ > > > +#!/usr/bin/env python3 > > > +# > > > +# This file was generated by running > > > +# > > > +# {cmdline} > > > +# > > > +# It is recommended that you do not modify it directly, but rather re-run the above command. > > > +# > > > + > > > +layerdata = """ > > > +{layerdata} > > > +""" > > > + > > > +import argparse > > > +import json > > > +import os > > > +import subprocess > > > + > > > +def _do_checkout(args): > > > + for l_name in layers: > > > + l_data = layers[l_name] > > > + if 'is_bootstrap' in l_data.keys(): > > > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) > > > + if not args[force_arg]: > > > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) > > > + continue > > > + rev = l_data['rev'] > > > + desc = l_data['describe'] > > > + if not desc: > > > + desc = rev[:10] > > > + branch = l_data['branch'] > > > + remotes = l_data['remotes'] > > > + remote = remotes[0] > > > + if len(remotes) > 1: > > > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] > > > + for r in remotes: > > > + if r['name'] == remotechoice: > > > + remote = r > > > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > > > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) > > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > > > + cwd = args['destdir'] > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > + cmd = 'git checkout -q {}'.format(rev) > > > + cwd = os.path.join(args['destdir'], l_name) > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > + > > > +layers = json.loads(layerdata) > > > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') > > > + > > > +bootstraplayer = None > > > +for l in layers: > > > + if 'is_bootstrap' in layers[l]: > > > + bootstraplayer = l > > > + > > > +if bootstraplayer: > > > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', > > > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > > > + > > > +for l in layers: > > > + remotes = layers[l]['remotes'] > > > + if len(remotes) > 1: > > > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], > > > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) > > > + > > > +try: > > > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) > > > +except subprocess.CalledProcessError as e: > > > + defaultdest = os.path.abspath(".") > > > + > > > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > > > + > > > +args = parser.parse_args() > > > + > > > +_do_checkout(vars(args)) > > > -=-=-=-=-=-=-=-=-=-=-=- > > > Links: You receive all messages sent to this group. > > > View/Reply Online (#167540): https://lists.openembedded.org/g/openembedded-core/message/167540 > > > Mute This Topic: https://lists.openembedded.org/mt/92117681/4454582 > > > Group Owner: openembedded-core+owner@lists.openembedded.org > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [adrian.freihofer@gmail.com] > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > >
Yes, for a start I explicitly decided to support only setups where all layers repositories (not the layers!) are in the same top level directory. If there is a repo tree, it needs to be flattened out first and bblayers.conf adjusted accordingly. I see this as more of an implementation detail that can be addressed later. The script is not just a replacement for the eSDK installer: it's meant also to help those who want to bootstrap a regular yocto build. The goal is to make it unnecessary to read the README when you clone some layer to see what needs to be done next: if the layer comes with a generated build setup script, just run it and it will get you to a ready-to-use bitbake environment. The important part is that the script is generated using the standard tooling, and operates from the metadata in a standard format. Alex On Mon, 4 Jul 2022 at 23:59, Adrian Freihofer <adrian.freihofer@gmail.com> wrote: > > Hello Alex > > After the discussion turned to the pros and cons of submodules, I was > thinking more about the initial setup of the Bitbake environment. The > json output of your script has something to do with this topic, but the > generated Python script does not. The Python script is meant to > replicate a bitbake setup to bootstrap an SDK environment. It is more > or less a replacement for the eSDK shell script installer. > > > On Mon, 2022-07-04 at 11:59 +0200, Alexander Kanavin wrote: > > Hello Adrian, > > > > I am not proposing a whole new standalone tool, RP is :-) What I am > > proposing is that once there is an active, working build, it is > > possible to capture the information about the layer structure in that > > build and available build configurations in the active layers into a > > metadata file (json), place that file inside a 'bootstrap layer', and > > then either let the users handle that file any way they please, or > > provide a self-contained, self-hosting script inside that same > > bootstrap layer that replicates the layer structure and sets up a > > build directory using information from the json. It's not really that > > complex, if both the script and its generator fit inside a single > > commit :) > > > > This is the value over git submodules: > > a) the layer structure and list of available build configurations can > > be saved with a single command if that command is using tinfoil to get > > the needed data about layers from a currently active build. No need to > > write anything by hand, or make scripts that work only with your > > specific product, and need to be adjusted every time layer setup > > changes. There's no way we can add this functionality to git :-) > > > > b) we can place anything that can be useful to layer users inside the > > metadata. Git submodules do not capture any of this layer-specific > > information (such as what branch we're on? what is the latest tag in > > that branch? what are other remotes that we can fetch from? where are > > the templates for build configurations and can I see their > > descriptions? and so on). All they allow is a single remote, and a > > single, cryptic revision on that remote for each of the layers. I've > > worked with that in a customer project, and I disliked it for lack of > > visibility into what am I actually checking out, and how do I actually > > set up a build once I ran the checkout (which is another thing that > > git will never do for you). > > > > Does this make sense? Did you try the code? Did you look at what is in > > the json? Maybe that helps to understand where I want to head with > > this. > > Yes, I did. It works nicely for simpler folder structures but I was not > able to get it working with more complicated layer setups. I got > "ERROR: Layer repositories are not all in the same parent directory". > One more detail is that it expects the output folder to be there > instead of just creating it if not. > > It looks really promising to me and I think it will also be appreciated > by git submodule fans as an alternative for populate_sdk_ext. > > Thank you very much. > > Regards, > Adrian > > > > > > Alex > > > > On Mon, 4 Jul 2022 at 11:01, Adrian Freihofer > > <adrian.freihofer@gmail.com> wrote: > > > > > > Hi Alex > > > > > > Thank you for initiating this important discussion with the code. This > > > could be one way to address this issue. However, the discussion here > > > also shows how complicated the issue is and how fragmented the > > > solutions and opinions are. There are already several tools out there, > > > but none of them has proven to be "the only right way". I'm not sure > > > that writing another tool is really the best approach. The complexity > > > of the proposed tool seems to me to be already at the upper limit, > > > where on the other hand Richard suggests to develop it even further and > > > publish it via pip. At the very least, I see the risk of ending up with > > > just another tool that is very complicated, needs maintenance, but > > > still won't be accepted by the community. > > > > > > Personally I really like to build software as simple as > > > git clone --recursive > > > bitbake my-image > > > > > > Setting up layers is basically just about fetching git repos. I don't > > > see the need for creating some configuration files or other complicated > > > tasks during the initial setup. So before introducing a new tool, > > > please let me understand why git submodules have not been very > > > successful in the past. I see some reasons for that: > > > * In the past, there were different RCS systems and the knowledge > > > about git was not everywhere. I think that has fundamentally changed > > > and the acceptance of git (and also git submodules) has massively > > > increased. Today, git may even be the only version control system > > > that needs to be officially supported to manage bitbake layers. > > > * We still use the submodule structures that Tim mentioned. In > > > general, I agree that using Git submodules is unnecessarily > > > complicated. The challenges start when multiple hierarchies of > > > submodules are used. In this use case, I miss a simple command like > > > "git checkout --recursive" that does everything I currently have to > > > do manually with multiple Git submodules sync, init and update and > > > cd commands. > > > * Probably the lack of a simple, recursive command in git is also the > > > reason why some CI implementations are in rare cases not able to > > > checkout git submodules correctly. > > > > > > Do you think there would be a need for a new tool if: > > > * git submodules would be easy to use? > > > * The Yocto manual would suggest to use git submodules for managing > > > the layers and also provide an example folder and submodules > > > structure as a guide line for the users? > > > * If the knowledge of git had been as widespread a few years ago (when > > > the distributions Tim mentions were published) as it is today? > > > I believe that today it may well be possible to establish git > > > submodules as the recommended solution. (Something like an easy to use > > > "git checkout --recursive" command would certainly helpful.) > > > > > > Since the majority of mostly experienced Yocto/OE developers who are > > > participating this discussion tend to develop a new tool, it makes me > > > wonder if I'm missing something. I see the following use cases where > > > layers need to be fetched: > > > * Initial project setup for working with bitbake. > > > * Retrieving layers from an SDK. (I'm not sure if this should remain > > > something special. The PoC which was recently posted by Alex for > > > bootstrapping the SDK directly from the bitbake environment looks > > > very promising to me). > > > * Fetching the layers on CI infrastructures which often call git fetch > > > with fancy options to improve efficiency. (That would probably not > > > work with a Yocto specific fetch tool anyway.) > > > Do you see other use cases for a layer fetching tool? > > > > > > What do you think about trying to optimize git submodules to handle the > > > "layer fetching" use case with a simple command, rather than developing > > > a new Yocto-specific git wrapper? > > > > > > Is it really useful to generate a configuration for KAS? A tool that > > > generates a configuration for another tool that finally does a Git > > > checkout seems a bit over-engineered to me. At least for us, an > > > implementation based on Git submodules would be usable, which would not > > > be the case for a KAS based implementation. > > > > > > Thank you and regards, > > > Adrian > > > > > > > > > On Fri, 2022-07-01 at 21:24 +0200, Alexander Kanavin wrote: > > > > This addresses a long standing gap in the core offering: > > > > there is no tooling to capture the currently configured layers > > > > with their revisions, or restore the layers from a configuration > > > > file (without using external tools, some of which aren't particularly > > > > suitable for the task). This plugin addresses the gap. > > > > > > > > How to use: > > > > > > > > 1. Saving a layer configuration: > > > > > > > > a) Command line options: > > > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h > > > > NOTE: Starting bitbake server... > > > > usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir > > > > > > > > Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. > > > > > > > > positional arguments: > > > > destdir Directory where to write the output > > > > (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). > > > > > > > > optional arguments: > > > > -h, --help show this help message and exit > > > > --output OUTPUT, -o OUTPUT > > > > File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. > > > > --format {python,json,kas}, -f {python,json,kas} > > > > Format of the output. The options are: > > > > python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) > > > > kas - a configuration file for the kas tool that allows the tool to do the same > > > > json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. > > > > > > > > b) Running with default choices: > > > > > > > > alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ > > > > NOTE: Starting bitbake server... > > > > NOTE: Created /srv/work/alex/meta-alex/setup-layers.py > > > > > > > > 2. Restoring the layers from the saved configuration: > > > > > > > > a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. > > > > > > > > b) Command line options: > > > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h > > > > usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] > > > > > > > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > > > > > > > optional arguments: > > > > -h, --help show this help message and exit > > > > --force-meta-alex-checkout > > > > Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). > > > > --choose-poky-remote {origin,poky-contrib} > > > > Choose a remote server for layer poky (default: origin) > > > > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). > > > > > > > > c) Running with default options: > > > > > > > > alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py > > > > Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. > > > > Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel > > > > Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test > > > > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel > > > > Note: multiple remotes defined for layer poky, using origin (run with -h to see others). > > > > Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky > > > > Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test > > > > Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky > > > > > > > > And that's it! > > > > > > > > FIXMEs: > > > > - kas config writer not yet implemented > > > > - oe-selftest test cases not yet written > > > > > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > > > --- > > > > meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ > > > > .../templates/setup-layers.py.template | 77 ++++++++++++ > > > > 2 files changed, 194 insertions(+) > > > > create mode 100644 meta/lib/bblayers/makesetup.py > > > > create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template > > > > > > > > diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py > > > > new file mode 100644 > > > > index 0000000000..3c86eea3c4 > > > > --- /dev/null > > > > +++ b/meta/lib/bblayers/makesetup.py > > > > @@ -0,0 +1,117 @@ > > > > +# > > > > +# SPDX-License-Identifier: GPL-2.0-only > > > > +# > > > > + > > > > +import logging > > > > +import os > > > > +import stat > > > > +import sys > > > > +import shutil > > > > +import json > > > > + > > > > +import bb.utils > > > > +import bb.process > > > > + > > > > +from bblayers.common import LayerPlugin > > > > + > > > > +logger = logging.getLogger('bitbake-layers') > > > > + > > > > +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) > > > > + > > > > +import oe.buildcfg > > > > + > > > > +def plugin_init(plugins): > > > > + return MakeSetupPlugin() > > > > + > > > > +class MakeSetupPlugin(LayerPlugin): > > > > + > > > > + def _write_python(self, repos, output): > > > > + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: > > > > + template = f.read() > > > > + args = sys.argv > > > > + args[0] = os.path.basename(args[0]) > > > > + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) > > > > + with open(output, 'w') as f: > > > > + f.write(script) > > > > + st = os.stat(output) > > > > + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) > > > > + > > > > + def _write_json(self, repos, output): > > > > + with open(output, 'w') as f: > > > > + json.dump(repos, f, sort_keys=True, indent=4) > > > > + > > > > + def _write_kas(self, repos, output): > > > > + raise NotImplementedError('Kas config writer not implemented yet') > > > > + > > > > + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} > > > > + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} > > > > + > > > > + def _get_repo_path(self, layer_path): > > > > + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) > > > > + return repo_path.strip() > > > > + > > > > + def _get_remotes(self, repo_path): > > > > + remotes = [] > > > > + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) > > > > + for r in remotes_list.split(): > > > > + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) > > > > + remotes.append({'name':r,'uri':uri.strip()}) > > > > + return remotes > > > > + > > > > + def _get_describe(self, repo_path): > > > > + try: > > > > + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) > > > > + except bb.process.ExecutionError: > > > > + return "" > > > > + return describe.strip() > > > > + > > > > + def _make_repo_config(self, destdir): > > > > + repos = {} > > > > + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) > > > > + for l in layers: > > > > + if l[1] == 'workspace': > > > > + continue > > > > + if l[4]: > > > > + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) > > > > + return > > > > + repo_path = self._get_repo_path(l[0]) > > > > + if repo_path not in repos.keys(): > > > > + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} > > > > + if not repos[repo_path]['remotes']: > > > > + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) > > > > + return > > > > + if repo_path in os.path.abspath(destdir): > > > > + repos[repo_path]['is_bootstrap'] = True > > > > + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) > > > > + > > > > + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) > > > > + if len(repo_dirs) > 1: > > > > + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) > > > > + return > > > > + > > > > + repos_nopaths = {} > > > > + for r in repos.keys(): > > > > + r_nopath = os.path.basename(r) > > > > + repos_nopaths[r_nopath] = repos[r] > > > > + return repos_nopaths > > > > + > > > > + def do_make_setup(self, args): > > > > + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ > > > > + repos = self._make_repo_config(args.destdir) > > > > + if not repos: > > > > + return > > > > + output = args.output > > > > + if not output: > > > > + output = self._output_filename[args.format] > > > > + output = os.path.join(os.path.abspath(args.destdir),output) > > > > + self._write_config[args.format](self, repos, output) > > > > + logger.info('Created {}'.format(output)) > > > > + > > > > + def register_commands(self, sp): > > > > + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) > > > > + parser_setup_layers.add_argument('destdir', > > > > + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') > > > > + parser_setup_layers.add_argument('--output', '-o', > > > > + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') > > > > + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', > > > > + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') > > > > diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template > > > > new file mode 100644 > > > > index 0000000000..a704ad3d70 > > > > --- /dev/null > > > > +++ b/meta/lib/bblayers/templates/setup-layers.py.template > > > > @@ -0,0 +1,77 @@ > > > > +#!/usr/bin/env python3 > > > > +# > > > > +# This file was generated by running > > > > +# > > > > +# {cmdline} > > > > +# > > > > +# It is recommended that you do not modify it directly, but rather re-run the above command. > > > > +# > > > > + > > > > +layerdata = """ > > > > +{layerdata} > > > > +""" > > > > + > > > > +import argparse > > > > +import json > > > > +import os > > > > +import subprocess > > > > + > > > > +def _do_checkout(args): > > > > + for l_name in layers: > > > > + l_data = layers[l_name] > > > > + if 'is_bootstrap' in l_data.keys(): > > > > + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) > > > > + if not args[force_arg]: > > > > + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) > > > > + continue > > > > + rev = l_data['rev'] > > > > + desc = l_data['describe'] > > > > + if not desc: > > > > + desc = rev[:10] > > > > + branch = l_data['branch'] > > > > + remotes = l_data['remotes'] > > > > + remote = remotes[0] > > > > + if len(remotes) > 1: > > > > + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] > > > > + for r in remotes: > > > > + if r['name'] == remotechoice: > > > > + remote = r > > > > + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) > > > > + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) > > > > + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) > > > > + cwd = args['destdir'] > > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > > + cmd = 'git checkout -q {}'.format(rev) > > > > + cwd = os.path.join(args['destdir'], l_name) > > > > + print("Running '{}' in {}".format(cmd, cwd)) > > > > + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) > > > > + > > > > +layers = json.loads(layerdata) > > > > +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') > > > > + > > > > +bootstraplayer = None > > > > +for l in layers: > > > > + if 'is_bootstrap' in layers[l]: > > > > + bootstraplayer = l > > > > + > > > > +if bootstraplayer: > > > > + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', > > > > + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) > > > > + > > > > +for l in layers: > > > > + remotes = layers[l]['remotes'] > > > > + if len(remotes) > 1: > > > > + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], > > > > + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) > > > > + > > > > +try: > > > > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) > > > > +except subprocess.CalledProcessError as e: > > > > + defaultdest = os.path.abspath(".") > > > > + > > > > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > > > > + > > > > +args = parser.parse_args() > > > > + > > > > +_do_checkout(vars(args)) > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > Links: You receive all messages sent to this group. > > > > View/Reply Online (#167540): https://lists.openembedded.org/g/openembedded-core/message/167540 > > > > Mute This Topic: https://lists.openembedded.org/mt/92117681/4454582 > > > > Group Owner: openembedded-core+owner@lists.openembedded.org > > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [adrian.freihofer@gmail.com] > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > >
On Sat, Jul 2, 2022 at 4:13 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > I admit I've been putting off this topic for a long time. I actually > wrote up something about it about 5 years ago. I sent it to four people > to get some opinions. The six opinions I got back made me despair. i > have had a while to think about it though. > > The challenge is that everyone has a workflow they like today and they > will tend to dislike anything that looks different. For that reason I > think we need a high level tool which can work with the different > approaches. Once we have that, I suspect we'll see some grow stronger > and some will wither off, which is probably as it should be through > natural evolution. The approach Alex has taken here does head in that > direction but I'm not sure it goes quite far enough to get the fraction > of users we need on board. > > I'm not going to comment directly on Alex's proposal at this point, I'd > like to put some ideas out there and see what people think of them. I > think you'll see that I'm in agreement with the idea/direction but I > have a slightly bigger idea of how to do it, which will both be harder > to implement but also have a better chance of getting more people on > board. > > So, let me go into a proposal of what I think the tool we need looks > like. I propose we create a new tool, lets call it "oe-setup". It is a > standalone project in it's own git repo and it's first command would be > "init" with a command line looking roughly like: > > oe-setup init <project-name> > oe-setup init <project-name> <target-dir> > oe-setup init <project-name> <config-name> <target-dir> > oe-setup init <project-name> <config-url> <target-dir> > oe-setup init <config-url> <target-dir> > > The idea being that the repository has some "pre-canned" idea of > certain project names, e.g. poky, yoe and whatever else we decide to > support "out the box" (criteria tbd). Those project names have a > default pre-canned config, or set of configs (e.g. taking branch/series > names) so I'd want to see these all work: My proposal would be to try to reuse Linux kconfig. Users know how to run "make oldconfig", "make menuconfig", etc and I believe all key distro / machine configuration along with which meta layers should be enabled, which branch to use, etc can all be captured within a kconfig .config file. The task at hand would then become writing a tool to translate from a .config file to a set of distro, machine, local.conf config files (and an image recipe?) + drive whichever tool will be used to fetch meta layer git repos. The "pre-canned" knowledge of poky, yoe, etc then just becomes a set of alternative defconfig files. Although if we can generate distro configs on the fly from a defconfig file then separate meta layers to hold distro configs starts to become redundant... As a reminder, here's the Buildroot README: https://github.com/buildroot/buildroot/blob/master/README With those first 2 steps a Buildroot user can effectively create their own custom distro and machine config... all from within the user-friendly menuconfig environment. The OE equivalent (Yocto Quick Start) is not only more complex and only tells a user how to build someone else's preset configurations. Creating a custom distro and machine config in OE is beyond the quick start and requires a full knowledge of bitbake syntax etc, etc. > oe-setup init poky . > oe-setup init poky dunfell . > oe-setup init poky ./my-local-config.json . > oe-setup init poky http://someserver/my-remote-config.json . > > We'd also allow something not in the default like of project names to > be used directly with a url, or maybe added with an "add-project" > command. This could work against a user local config file, a bit like > git does with global config and adding remotes to a repo. > > You'll note I haven't made any mention of the tooling these use. The > reason is that we don't actually care. I'd propose we teach the tool > about a few common standards (kas, submodules, repo) in the form of > plugins and then hand off to those tools to do the setup. I'd also > propose we develop a "native" form where they perhaps aren't needed. > the nice thing is we can have several "native" forms if needed so if > one approach isn't working or we need to change it, we can. > > We may also want to consider an optional "sub-config" parameter which > gets passed along to the underlying tool. > > One of my conclusions after thinking about this for a long time is we > have a bootstrap problem. If everyone used git and git worked > everywhere, things would be easier. They don't and it doesn't. My > evidence? The bitbake fetcher. Much of the ugliness the bitbake fetcher > deals with applies to layers as well. Some people need proxy support, > some need mirror tarballs, some need floating revisions, some need > complete lock down and so on. > > We could simplify the problem and just say those users "can manage". If > we do that, we're giving up at the first hurdle on the idea what we can > have a (mostly) universal tool. I'm not sure I'm quite ready to do > that. > > Like most more seasoned developers, I dislike code duplication. We do > have the fetcher code, which knows something about these issues and it > even has a test suite with decent coverage. Most users already know how > to configure it too. > > Whilst I don't like it particularly, I'd propose we include a chunk of > bitbake inside oe-setup (maybe just lib/bb?). How needs discussion. I > nearly wrote combo-layer here but people might think I'm serious :). > More seriously, git subtree might be a potential option. We could then > have bitbake master, or a chunk of it, available to the tool. This > would then allow us to potentially get out of a variety of different > firewall/mirror situations, if the appropriate backends were right. > This gives us the option for fix bugs in the fetcher and update oe- > setup and still use specific versions of bitbake for the different > releases as needed. > > I'd propose oe-setup be a different kind of project to bitbake, > hopefully something we could make available via pip for example, again > to try and help the bootstrap issue for people behind firewalls etc. > although I would want it usable standalone too. It would be designed to > be a separate standalone tool installed somewhere once by a user rather > than associated with a particular build/metadata set like bitbake is. > > Beyond init, the question is probably updating. There are probably two > options, one is an update command to oe-setup and the other is > deferring to the underlying tool and it should be possible to use > either depending on the underlying tool and the circumstances. > > As for configuration of the build, I've wondered if we add a new level > of config file, basically dedicating a config file "slot" to the setup > tooling where config from the setup tool would land (i.e. like > auto.conf but dedicated to this). I need to think about this piece a > bit more. > > The other side of this would be the generation of the config(s). I'm > less worried about this piece, it would be done by the people who can > deal with more complexity and I'm sure we could figure it out. I used > "json" in the config urls above but I'm aware that yaml is going to be > a discussion there too. From the oe-setup perspective, it may support > multiple options since once it knows the tool to pass it off to, it > doesn't really need to know more. > > I'm sure there is more I could write on this topic but I'll stop there > and see what people think. > > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#167548): https://lists.openembedded.org/g/openembedded-core/message/167548 > Mute This Topic: https://lists.openembedded.org/mt/92117681/3619030 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [armccurdy@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Tue, 5 Jul 2022 at 05:09, Andre McCurdy <armccurdy@gmail.com> wrote: > My proposal would be to try to reuse Linux kconfig. Users know how to > run "make oldconfig", "make menuconfig", etc and I believe all key > distro / machine configuration along with which meta layers should be > enabled, which branch to use, etc can all be captured within a kconfig > .config file. The task at hand would then become writing a tool to > translate from a .config file to a set of distro, machine, local.conf > config files (and an image recipe?) + drive whichever tool will be > used to fetch meta layer git repos. At this point, I have to remind everyone about our harsh reality. Which is having lots of core pieces without maintainers: https://git.yoctoproject.org/poky/tree/MAINTAINERS.md#n45 Let's focus on adding foundational pieces of layer and configuration management in a controlled, testable and tested manner for now. Which is what I am trying to do. I think discussing grandiose designs for things that try to please everybody is just a bit premature, honestly. Alex
On Tue, 5 Jul 2022 at 10:34, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > At this point, I have to remind everyone about our harsh reality. > Which is having lots of core pieces without maintainers: > https://git.yoctoproject.org/poky/tree/MAINTAINERS.md#n45 > > Let's focus on adding foundational pieces of layer and configuration > management in a controlled, testable and tested manner for now. Which > is what I am trying to do. I think discussing grandiose designs for > things that try to please everybody is just a bit premature, honestly. And let me just go ahead and say it, and I understand this may be upsetting. I'd rather try really hard to completely avoid the standalone setup tool. That tool, and the way it's just been described sounds like an invitation to compatibility issues between various tool versions and the metadata and backends (kas alone has more than 12 config format versions and requires to specify the version explicitly?) coming from all kinds of places, maintainability issues (we are already pressed to the limit with many pieces getting no maintenance), leaky abstractions, not quite covering all of the ways people want to use it with their existing setups, and overall greatly increased complexity in a project that is already notorious for it. I'd rather avoid repeating the story with devtool, which is both more complex and has less use than what we'd hope for - and has no maintainer. Alex
On Tue, 2022-07-05 at 10:34 +0200, Alexander Kanavin wrote: > On Tue, 5 Jul 2022 at 05:09, Andre McCurdy <armccurdy@gmail.com> wrote: > > My proposal would be to try to reuse Linux kconfig. Users know how to > > run "make oldconfig", "make menuconfig", etc and I believe all key > > distro / machine configuration along with which meta layers should be > > enabled, which branch to use, etc can all be captured within a kconfig > > .config file. The task at hand would then become writing a tool to > > translate from a .config file to a set of distro, machine, local.conf > > config files (and an image recipe?) + drive whichever tool will be > > used to fetch meta layer git repos. > > At this point, I have to remind everyone about our harsh reality. > Which is having lots of core pieces without maintainers: > https://git.yoctoproject.org/poky/tree/MAINTAINERS.md#n45 > > Let's focus on adding foundational pieces of layer and configuration > management in a controlled, testable and tested manner for now. Which > is what I am trying to do. I think discussing grandiose designs for > things that try to please everybody is just a bit premature, honestly. There is a time and a place for incremental improvement and also a time and a place where we need to step back, look at the bigger picture/problem and try and design for that. It is no secret that I've been avoiding the layer setup issue, I know it is a painful topic and will be hard to solve. I'm also acutely aware that we can't ignore it for that much longer, we need to do something about it. The challenge with your approach is that it is bolted directly onto bitbake-layers, which makes it not only OE-Core API but bitbake API too. There is a standard such changes need to reach and I'm not convinced that is doesn't tie our hands in future as is. I suspect in reality, we could probably have the same functionality in a separate script in OE-Core, maybe improving the APIs/library functions from bitbake-layers if needed. That would lower the bar and mean we could have something focused on eSDK needs rather than the wider problem. Of course that potentially misses an opportunity to potentially address the wider issue. I did at least take the time (on a weekend) to write down my thoughts, which wasn't particularly easy. As for kconfig, I don't think we need another config file format. I have proven that json can encapsulate all the complexities of the configurations of the autobuilder and kas has also proven the same think with yaml. Yes, the menus are nice but at some point users will need to see our config files so I don't think its a much of a win as people think, unless the entire project used it which it doesn't and is unlikely to. Cheers, Richard
On Tue, 2022-07-05 at 10:39 +0200, Alexander Kanavin wrote: > On Tue, 5 Jul 2022 at 10:34, Alexander Kanavin via > lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> > wrote: > > At this point, I have to remind everyone about our harsh reality. > > Which is having lots of core pieces without maintainers: > > https://git.yoctoproject.org/poky/tree/MAINTAINERS.md#n45 > > > > Let's focus on adding foundational pieces of layer and configuration > > management in a controlled, testable and tested manner for now. Which > > is what I am trying to do. I think discussing grandiose designs for > > things that try to please everybody is just a bit premature, honestly. > > And let me just go ahead and say it, and I understand this may be > upsetting. I'd rather try really hard to completely avoid the > standalone setup tool. I've been trying that for years, it is looking like it is no longer an option as if we don't do something, I think the project will suffer more damage than a tool would do. > That tool, and the way it's just been described > sounds like an invitation to compatibility issues between various tool > versions and the metadata and backends (kas alone has more than 12 > config format versions and requires to specify the version > explicitly?) coming from all kinds of places, maintainability issues > (we are already pressed to the limit with many pieces getting no > maintenance), leaky abstractions, not quite covering all of the ways > people want to use it with their existing setups, and overall greatly > increased complexity in a project that is already notorious for it. In defence of the tool idea, we do have some experience of what compatibility looks like through the autobuilder. There are a lot of parallels in what it does and how it does it to what we need in that standalone tool. Yes, there are differences needed but there is a decent chunk of experience which can be used to hopefully largely avoid those compatibility issues. > I'd rather avoid repeating the story with devtool, which is both more > complex and has less use than what we'd hope for - and has no > maintainer. I don't think the setup tool should impact devtool, those are two different components which should be orthogonal to each other. Yes, there is impact on the eSDK but I'd hope in general that would be a simplification of the existing setup code. Cheers, Richard
On Tue, 5 Jul 2022 at 10:51, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > The challenge with your approach is that it is bolted directly onto > bitbake-layers, which makes it not only OE-Core API but bitbake API > too. There is a standard such changes need to reach and I'm not > convinced that is doesn't tie our hands in future as is. > > I suspect in reality, we could probably have the same functionality in > a separate script in OE-Core, maybe improving the APIs/library > functions from bitbake-layers if needed. That would lower the bar and > mean we could have something focused on eSDK needs rather than the > wider problem. I added the layer config generator as a bitbake-layers plugin for these reasons: 1. It is layer management functionality, and so logically belongs in that tool. 2. It is more visible and discoverable when it is in there, as opposed to all those executables in poky/scripts/ which most people have no idea about. 3. It needs to get a list of active layers and their locations programmatically, which is trivial from a bitbake-layers plugin, less so from other places. 4. it can reuse code from meta/lib/oe if/when needed. There are a few things in there that I'd already like to do differently, and there's also the 'save the active build/conf in TEMPLATECONF format' piece missing, so there'll be a reworked version, arriving in bite-sized chunks for easier review/discussion. Alex
On Tue, 5 Jul 2022 at 10:51, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > As for kconfig, I don't think we need another config file format. I > have proven that json can encapsulate all the complexities of the > configurations of the autobuilder and kas has also proven the same > think with yaml. Yes, the menus are nice but at some point users will > need to see our config files so I don't think its a much of a win as > people think, unless the entire project used it which it doesn't and is > unlikely to. What is more important to me is that each machine, distro and templateconf comes with a *description* right next to the actual .conf file. They all must have conf-notes.txt, and that should be a requirement for YP compatibility actually. Then if anyone wants to write a tool that discovers the available items and presents a UI for choosing them, the documentation for each of the item is already in place and not maintained separately (which inevitably results in things getting out of sync). Alex
On Tue, Jul 5, 2022 at 4:16 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > On Tue, 5 Jul 2022 at 10:51, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > The challenge with your approach is that it is bolted directly onto > > bitbake-layers, which makes it not only OE-Core API but bitbake API > > too. There is a standard such changes need to reach and I'm not > > convinced that is doesn't tie our hands in future as is. > > > > I suspect in reality, we could probably have the same functionality in > > a separate script in OE-Core, maybe improving the APIs/library > > functions from bitbake-layers if needed. That would lower the bar and > > mean we could have something focused on eSDK needs rather than the > > wider problem. > > I added the layer config generator as a bitbake-layers plugin for these reasons: > 1. It is layer management functionality, and so logically belongs in that tool. > 2. It is more visible and discoverable when it is in there, as opposed > to all those executables in poky/scripts/ which most people have no > idea about. > 3. It needs to get a list of active layers and their locations > programmatically, which is trivial from a bitbake-layers plugin, less > so from other places. > 4. it can reuse code from meta/lib/oe if/when needed. Focusing on the data model helps with that; as long as it's reasonable to maintain the configuration manually, having the tool to create the layer setup from your current layers is a useful way to initially populate/maintain that. An easily consumable data model makes it easier for 3rd party tools (e.g. kas) to pivot to using it instead of making their own ad hoc, and makes it possible for whatever flavor of other tooling to grow up around it. In that regard, I'd prefer that the JSON file be what defines a layer setup and included in layers. The setup python script is nice (and convenient), but I think it would be better as an additional file alongside the JSON file (or that someone can download independently if the layer doesn't have one). I think a few tweaks would make it possible to write the script so it can operate generically on any JSON file input without needing templating. I agree, it is harder to define a stable data model.... but probably worth it in the long run? Also, sticking to submodules doesn't mean I don't see the value here. We need something for layer CI, first-time startup guides, et al, and I think that either has to be "adopt kas", or take this approach. I _much_ prefer this approach to kas. I apologize if it came off that I didn't want to do this at all. > > There are a few things in there that I'd already like to do > differently, and there's also the 'save the active build/conf in > TEMPLATECONF format' piece missing, so there'll be a reworked version, > arriving in bite-sized chunks for easier review/discussion. > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#167645): https://lists.openembedded.org/g/openembedded-core/message/167645 > Mute This Topic: https://lists.openembedded.org/mt/92117681/3616693 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [JPEWhacker@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Tue, 5 Jul 2022 at 16:28, Joshua Watt <jpewhacker@gmail.com> wrote: > In that regard, I'd prefer that the JSON file be what defines a layer > setup and included in layers. The setup python script is nice (and > convenient), but I think it would be better as an additional file > alongside the JSON file (or that someone can download independently if > the layer doesn't have one). I think a few tweaks would make it > possible to write the script so it can operate generically on any JSON > file input without needing templating. You have that choice when you run the generator: write out the bare json, write out the script that embeds the json (and is guaranteed to be compatible with it), or both. Additionally, the script can print the json inside itself with --print-config. If you prefer to include just the json into the layer, that's fine, but I don't particularly want to solve the problem of where to find tools that can do something with it (without exploding with cryptic compatibility errors), and I can imagine most people, and especially beginners, would just want to run the script directly. Alex
On Sat, 2 Jul 2022 at 10:28, Tim Orling <ticotimo@gmail.com> wrote: >> Configuration will be handled with TEMPLATECONF and a set of templates >> in a product/platform layer. It's already in core, it's simple to >> understand and it works. We used it in a very large project. It also >> requires a certain bit of discipline, which is a good thing: you don't >> get to implement ugly local.conf hacks in a turing complete language, >> and must design your configuration (distro, machines and images) >> carefully in a static manner. Then local.conf must contain only DISTRO >> and MACHINE, and a 'confuguration' is essentially a choice of those >> two plus a set of enabled layers plus site-specific settings for >> sstate, downloads, proxies and parallelism level. Back to >> (maintainable) basics. I just sent out patches for this bit (build configuration). The layer configuration generator/script needs a few changes which I'll do next. Alex
On Tue, Jul 5, 2022 at 11:04 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > On Tue, 5 Jul 2022 at 16:28, Joshua Watt <jpewhacker@gmail.com> wrote: > > In that regard, I'd prefer that the JSON file be what defines a layer > > setup and included in layers. The setup python script is nice (and > > convenient), but I think it would be better as an additional file > > alongside the JSON file (or that someone can download independently if > > the layer doesn't have one). I think a few tweaks would make it > > possible to write the script so it can operate generically on any JSON > > file input without needing templating. > > You have that choice when you run the generator: write out the bare > json, write out the script that embeds the json (and is guaranteed to > be compatible with it), or both. Additionally, the script can print > the json inside itself with --print-config. If you prefer to include > just the json into the layer, that's fine, but I don't particularly > want to solve the problem of where to find tools that can do something > with it (without exploding with cryptic compatibility errors), and I > can imagine most people, and especially beginners, would just want to > run the script directly. I think my interest here is more in the JSON schema, and I think that's the more important piece for consolidating tools around a common standard in the long run. I've pushed up an RFC patch with a proposed schema and example layer JSON file to review. It would be nice if the script outlined here didn't have to be generated from bblayers and was a static script that could operate on any JSON file, since that is more generally useful; it would also be a good starting point for a generic layer setup tool that we can point users to and can be replaced by something more comprehensive (like what Richard outlines) later without having to change the actual layer data (the JSON). Or put another way, I'd prefer the data be kept separate from the code. Note that my schema is missing the "is_bootstrap" option yours has; I wasn't quite clear if that meant anything other than "don't fetch this layer by default"? > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#167667): https://lists.openembedded.org/g/openembedded-core/message/167667 > Mute This Topic: https://lists.openembedded.org/mt/92117681/3616693 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [JPEWhacker@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, 6 Jul 2022 at 21:54, Joshua Watt <jpewhacker@gmail.com> wrote: > I think my interest here is more in the JSON schema, and I think > that's the more important piece for consolidating tools around a > common standard in the long run. I've pushed up an RFC patch with a > proposed schema and example layer JSON file to review. It would be > nice if the script outlined here didn't have to be generated from > bblayers and was a static script that could operate on any JSON file, > since that is more generally useful; it would also be a good starting > point for a generic layer setup tool that we can point users to and > can be replaced by something more comprehensive (like what Richard > outlines) later without having to change the actual layer data (the > JSON). Or put another way, I'd prefer the data be kept separate from > the code. Defining the format formally in a verifiable way is quite useful, thanks. I'll integrate it into my patchset. And yes, I have a different viewpoint: I do not want separate tools that need to be pointed to, or any extra steps, or data decoupled from code. What I do want is to clone a layer, run a single script once, and be dropped into an environment where I can run sstate-accelerated bitbake. Maybe making a choice about what target I want to build for on the way. And without ever having to read a (often) badly written, unhelpful layer README, or asking anyone for help. If it's not like that, it's not good enough. > Note that my schema is missing the "is_bootstrap" option yours has; I > wasn't quite clear if that meant anything other than "don't fetch this > layer by default"? Yes. The idea is that the tools shouldn't have to guess if the json itself is hosted inside one of the layers it describes, and so that fact should be written into the json. It can be named 'contains_this_file' or something like that. Alex
On 7/4/22 22:09, Andre McCurdy wrote: > On Sat, Jul 2, 2022 at 4:13 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: >> >> I admit I've been putting off this topic for a long time. I actually >> wrote up something about it about 5 years ago. I sent it to four people >> to get some opinions. The six opinions I got back made me despair. i >> have had a while to think about it though. >> >> The challenge is that everyone has a workflow they like today and they >> will tend to dislike anything that looks different. For that reason I >> think we need a high level tool which can work with the different >> approaches. Once we have that, I suspect we'll see some grow stronger >> and some will wither off, which is probably as it should be through >> natural evolution. The approach Alex has taken here does head in that >> direction but I'm not sure it goes quite far enough to get the fraction >> of users we need on board. >> >> I'm not going to comment directly on Alex's proposal at this point, I'd >> like to put some ideas out there and see what people think of them. I >> think you'll see that I'm in agreement with the idea/direction but I >> have a slightly bigger idea of how to do it, which will both be harder >> to implement but also have a better chance of getting more people on >> board. >> >> So, let me go into a proposal of what I think the tool we need looks >> like. I propose we create a new tool, lets call it "oe-setup". It is a >> standalone project in it's own git repo and it's first command would be >> "init" with a command line looking roughly like: >> >> oe-setup init <project-name> >> oe-setup init <project-name> <target-dir> >> oe-setup init <project-name> <config-name> <target-dir> >> oe-setup init <project-name> <config-url> <target-dir> >> oe-setup init <config-url> <target-dir> >> >> The idea being that the repository has some "pre-canned" idea of >> certain project names, e.g. poky, yoe and whatever else we decide to >> support "out the box" (criteria tbd). Those project names have a >> default pre-canned config, or set of configs (e.g. taking branch/series >> names) so I'd want to see these all work: > > My proposal would be to try to reuse Linux kconfig. Users know how to > run "make oldconfig", "make menuconfig", etc and I believe all key I'm not sure that is true. Many current users do not come from the kernel dev community so, they are not familiar with how kconfig operates. Philip > distro / machine configuration along with which meta layers should be > enabled, which branch to use, etc can all be captured within a kconfig > .config file. The task at hand would then become writing a tool to > translate from a .config file to a set of distro, machine, local.conf > config files (and an image recipe?) + drive whichever tool will be > used to fetch meta layer git repos. > > The "pre-canned" knowledge of poky, yoe, etc then just becomes a set > of alternative defconfig files. Although if we can generate distro > configs on the fly from a defconfig file then separate meta layers to > hold distro configs starts to become redundant... > > As a reminder, here's the Buildroot README: > > https://github.com/buildroot/buildroot/blob/master/README > > With those first 2 steps a Buildroot user can effectively create their > own custom distro and machine config... all from within the > user-friendly menuconfig environment. The OE equivalent (Yocto Quick > Start) is not only more complex and only tells a user how to build > someone else's preset configurations. Creating a custom distro and > machine config in OE is beyond the quick start and requires a full > knowledge of bitbake syntax etc, etc. > >> oe-setup init poky . >> oe-setup init poky dunfell . >> oe-setup init poky ./my-local-config.json . >> oe-setup init poky http://someserver/my-remote-config.json . >> >> We'd also allow something not in the default like of project names to >> be used directly with a url, or maybe added with an "add-project" >> command. This could work against a user local config file, a bit like >> git does with global config and adding remotes to a repo. >> >> You'll note I haven't made any mention of the tooling these use. The >> reason is that we don't actually care. I'd propose we teach the tool >> about a few common standards (kas, submodules, repo) in the form of >> plugins and then hand off to those tools to do the setup. I'd also >> propose we develop a "native" form where they perhaps aren't needed. >> the nice thing is we can have several "native" forms if needed so if >> one approach isn't working or we need to change it, we can. >> >> We may also want to consider an optional "sub-config" parameter which >> gets passed along to the underlying tool. >> >> One of my conclusions after thinking about this for a long time is we >> have a bootstrap problem. If everyone used git and git worked >> everywhere, things would be easier. They don't and it doesn't. My >> evidence? The bitbake fetcher. Much of the ugliness the bitbake fetcher >> deals with applies to layers as well. Some people need proxy support, >> some need mirror tarballs, some need floating revisions, some need >> complete lock down and so on. >> >> We could simplify the problem and just say those users "can manage". If >> we do that, we're giving up at the first hurdle on the idea what we can >> have a (mostly) universal tool. I'm not sure I'm quite ready to do >> that. >> >> Like most more seasoned developers, I dislike code duplication. We do >> have the fetcher code, which knows something about these issues and it >> even has a test suite with decent coverage. Most users already know how >> to configure it too. >> >> Whilst I don't like it particularly, I'd propose we include a chunk of >> bitbake inside oe-setup (maybe just lib/bb?). How needs discussion. I >> nearly wrote combo-layer here but people might think I'm serious :). >> More seriously, git subtree might be a potential option. We could then >> have bitbake master, or a chunk of it, available to the tool. This >> would then allow us to potentially get out of a variety of different >> firewall/mirror situations, if the appropriate backends were right. >> This gives us the option for fix bugs in the fetcher and update oe- >> setup and still use specific versions of bitbake for the different >> releases as needed. >> >> I'd propose oe-setup be a different kind of project to bitbake, >> hopefully something we could make available via pip for example, again >> to try and help the bootstrap issue for people behind firewalls etc. >> although I would want it usable standalone too. It would be designed to >> be a separate standalone tool installed somewhere once by a user rather >> than associated with a particular build/metadata set like bitbake is. >> >> Beyond init, the question is probably updating. There are probably two >> options, one is an update command to oe-setup and the other is >> deferring to the underlying tool and it should be possible to use >> either depending on the underlying tool and the circumstances. >> >> As for configuration of the build, I've wondered if we add a new level >> of config file, basically dedicating a config file "slot" to the setup >> tooling where config from the setup tool would land (i.e. like >> auto.conf but dedicated to this). I need to think about this piece a >> bit more. >> >> The other side of this would be the generation of the config(s). I'm >> less worried about this piece, it would be done by the people who can >> deal with more complexity and I'm sure we could figure it out. I used >> "json" in the config urls above but I'm aware that yaml is going to be >> a discussion there too. From the oe-setup perspective, it may support >> multiple options since once it knows the tool to pass it off to, it >> doesn't really need to know more. >> >> I'm sure there is more I could write on this topic but I'll stop there >> and see what people think. >> >> Cheers, >> >> Richard >> >> >> >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#167633): https://lists.openembedded.org/g/openembedded-core/message/167633 >> Mute This Topic: https://lists.openembedded.org/mt/92117681/384425 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [philip@balister.org] >> -=-=-=-=-=-=-=-=-=-=-=- >>
Okay, the reworked version of the layer tooling is now out as well. I've addressed most of the feedback, and placed the things still TBD in FIXME notes in the commits. Alex On Wed, 6 Jul 2022 at 20:34, Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > On Sat, 2 Jul 2022 at 10:28, Tim Orling <ticotimo@gmail.com> wrote: > >> Configuration will be handled with TEMPLATECONF and a set of templates > >> in a product/platform layer. It's already in core, it's simple to > >> understand and it works. We used it in a very large project. It also > >> requires a certain bit of discipline, which is a good thing: you don't > >> get to implement ugly local.conf hacks in a turing complete language, > >> and must design your configuration (distro, machines and images) > >> carefully in a static manner. Then local.conf must contain only DISTRO > >> and MACHINE, and a 'confuguration' is essentially a choice of those > >> two plus a set of enabled layers plus site-specific settings for > >> sstate, downloads, proxies and parallelism level. Back to > >> (maintainable) basics. > > I just sent out patches for this bit (build configuration). The layer > configuration generator/script needs a few changes which I'll do next. > > Alex
diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py new file mode 100644 index 0000000000..3c86eea3c4 --- /dev/null +++ b/meta/lib/bblayers/makesetup.py @@ -0,0 +1,117 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil +import json + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return MakeSetupPlugin() + +class MakeSetupPlugin(LayerPlugin): + + def _write_python(self, repos, output): + with open(os.path.join(os.path.dirname(__file__), "templates", "setup-layers.py.template")) as f: + template = f.read() + args = sys.argv + args[0] = os.path.basename(args[0]) + script = template.replace('{cmdline}', " ".join(args)).replace('{layerdata}', json.dumps(repos, sort_keys=True, indent=4)) + with open(output, 'w') as f: + f.write(script) + st = os.stat(output) + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + def _write_json(self, repos, output): + with open(output, 'w') as f: + json.dump(repos, f, sort_keys=True, indent=4) + + def _write_kas(self, repos, output): + raise NotImplementedError('Kas config writer not implemented yet') + + _write_config = {"python":_write_python, "json":_write_json, "kas":_write_kas} + _output_filename = {"python":"setup-layers.py","json":"setup-layers.json","kas":"setup-layers.kas.yaml"} + + def _get_repo_path(self, layer_path): + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) + return repo_path.strip() + + def _get_remotes(self, repo_path): + remotes = [] + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) + for r in remotes_list.split(): + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) + remotes.append({'name':r,'uri':uri.strip()}) + return remotes + + def _get_describe(self, repo_path): + try: + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) + except bb.process.ExecutionError: + return "" + return describe.strip() + + def _make_repo_config(self, destdir): + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + for l in layers: + if l[1] == 'workspace': + continue + if l[4]: + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l[1],path=l[0])) + return + repo_path = self._get_repo_path(l[0]) + if repo_path not in repos.keys(): + repos[repo_path] = {'rev':l[3], 'branch':l[2], 'remotes':self._get_remotes(repo_path), 'layers':[], 'describe':self._get_describe(repo_path)} + if not repos[repo_path]['remotes']: + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) + return + if repo_path in os.path.abspath(destdir): + repos[repo_path]['is_bootstrap'] = True + repos[repo_path]['layers'].append({'name':l[1],'path':l[0].replace(repo_path,'')[1:]}) + + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) + if len(repo_dirs) > 1: + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) + return + + repos_nopaths = {} + for r in repos.keys(): + r_nopath = os.path.basename(r) + repos_nopaths[r_nopath] = repos[r] + return repos_nopaths + + def do_make_setup(self, args): + """ Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. """ + repos = self._make_repo_config(args.destdir) + if not repos: + return + output = args.output + if not output: + output = self._output_filename[args.format] + output = os.path.join(os.path.abspath(args.destdir),output) + self._write_config[args.format](self, repos, output) + logger.info('Created {}'.format(output)) + + def register_commands(self, sp): + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) + parser_setup_layers.add_argument('destdir', + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') + parser_setup_layers.add_argument('--output', '-o', + help='File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable.') + parser_setup_layers.add_argument('--format', '-f', choices=['python', 'json', 'kas'], default='python', + help='Format of the output. The options are:\n\tpython - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended)\n\tkas - a configuration file for the kas tool that allows the tool to do the same\n\tjson - a json formatted file containing all the needed metadata to do the same by any external or custom tool.') diff --git a/meta/lib/bblayers/templates/setup-layers.py.template b/meta/lib/bblayers/templates/setup-layers.py.template new file mode 100644 index 0000000000..a704ad3d70 --- /dev/null +++ b/meta/lib/bblayers/templates/setup-layers.py.template @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +# +# This file was generated by running +# +# {cmdline} +# +# It is recommended that you do not modify it directly, but rather re-run the above command. +# + +layerdata = """ +{layerdata} +""" + +import argparse +import json +import os +import subprocess + +def _do_checkout(args): + for l_name in layers: + l_data = layers[l_name] + if 'is_bootstrap' in l_data.keys(): + force_arg = 'force_{}_checkout'.format(l_name.replace('-','_')) + if not args[force_arg]: + print('Note: not checking out layer {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-{}-checkout'.format(l_name))) + continue + rev = l_data['rev'] + desc = l_data['describe'] + if not desc: + desc = rev[:10] + branch = l_data['branch'] + remotes = l_data['remotes'] + remote = remotes[0] + if len(remotes) > 1: + remotechoice = args['choose_{}_remote'.format(l_name.replace('-','_'))] + for r in remotes: + if r['name'] == remotechoice: + remote = r + print('Note: multiple remotes defined for layer {}, using {} (run with -h to see others).'.format(l_name, r['name'])) + print('Checking out layer {}, revision {}, branch {} from remote {} at {}'.format(l_name, desc, branch, remote['name'], remote['uri'])) + cmd = 'git clone -q {} {}'.format(remote['uri'], l_name) + cwd = args['destdir'] + print("Running '{}' in {}".format(cmd, cwd)) + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) + cmd = 'git checkout -q {}'.format(rev) + cwd = os.path.join(args['destdir'], l_name) + print("Running '{}' in {}".format(cmd, cwd)) + subprocess.check_output(cmd, text=True, shell=True, cwd=cwd) + +layers = json.loads(layerdata) +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') + +bootstraplayer = None +for l in layers: + if 'is_bootstrap' in layers[l]: + bootstraplayer = l + +if bootstraplayer: + parser.add_argument('--force-{bootstraplayer}-checkout'.format(bootstraplayer=bootstraplayer), action='store_true', + help='Force the checkout of the bootstrap layer {bootstraplayer} (by default it is presumed that this script is in it, and so the layer is already in place).'.format(bootstraplayer=bootstraplayer)) + +for l in layers: + remotes = layers[l]['remotes'] + if len(remotes) > 1: + parser.add_argument('--choose-{multipleremoteslayer}-remote'.format(multipleremoteslayer=l),choices=[r['name'] for r in remotes], default=remotes[0]['name'], + help='Choose a remote server for layer {multipleremoteslayer} (default: {defaultremote})'.format(multipleremoteslayer=l, defaultremote=remotes[0]['name'])) + +try: + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) +except subprocess.CalledProcessError as e: + defaultdest = os.path.abspath(".") + +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) + +args = parser.parse_args() + +_do_checkout(vars(args))
This addresses a long standing gap in the core offering: there is no tooling to capture the currently configured layers with their revisions, or restore the layers from a configuration file (without using external tools, some of which aren't particularly suitable for the task). This plugin addresses the gap. How to use: 1. Saving a layer configuration: a) Command line options: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h NOTE: Starting bitbake server... usage: bitbake-layers create-layers-setup [-h] [--output OUTPUT] [--format {python,json,kas}] destdir Writes out a python script/kas config/json config that replicates the directory structure and revisions of the layers in a current build. positional arguments: destdir Directory where to write the output (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). optional arguments: -h, --help show this help message and exit --output OUTPUT, -o OUTPUT File name where to write the output, if the default (setup-layers.py/.json/.yml) is undesirable. --format {python,json,kas}, -f {python,json,kas} Format of the output. The options are: python - a self contained python script that fetches all the needed layers and sets them to correct revisions (default, recommended) kas - a configuration file for the kas tool that allows the tool to do the same json - a json formatted file containing all the needed metadata to do the same by any external or custom tool. b) Running with default choices: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup ../../meta-alex/ NOTE: Starting bitbake server... NOTE: Created /srv/work/alex/meta-alex/setup-layers.py 2. Restoring the layers from the saved configuration: a) Clone meta-alex separately, as a bootstrap layer/repository. It should already contain setup-layers.py created in the previous step. b) Command line options: alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py -h usage: setup-layers.py [-h] [--force-meta-alex-checkout] [--choose-poky-remote {origin,poky-contrib}] [--destdir DESTDIR] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --help show this help message and exit --force-meta-alex-checkout Force the checkout of the bootstrap layer meta-alex (by default it is presumed that this script is in it, and so the layer is already in place). --choose-poky-remote {origin,poky-contrib} Choose a remote server for layer poky (default: origin) --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/layers-test). c) Running with default options: alex@Zen2:/srv/work/alex/layers-test/meta-alex$ ./setup-layers.py Note: not checking out layer meta-alex, use --force-meta-alex-checkout to override. Checking out layer meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master from remote origin at git://git.yoctoproject.org/meta-intel Running 'git clone -q git://git.yoctoproject.org/meta-intel meta-intel' in /srv/work/alex/layers-test Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/layers-test/meta-intel Note: multiple remotes defined for layer poky, using origin (run with -h to see others). Checking out layer poky, revision 4.1_M1-295-g6850b29806, branch akanavin/setup-layers from remote origin at git://git.yoctoproject.org/poky Running 'git clone -q git://git.yoctoproject.org/poky poky' in /srv/work/alex/layers-test Running 'git checkout -q 4cc94de99230201c3c39b924219113157ff47006' in /srv/work/alex/layers-test/poky And that's it! FIXMEs: - kas config writer not yet implemented - oe-selftest test cases not yet written Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/bblayers/makesetup.py | 117 ++++++++++++++++++ .../templates/setup-layers.py.template | 77 ++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 meta/lib/bblayers/makesetup.py create mode 100644 meta/lib/bblayers/templates/setup-layers.py.template