| Message ID | 20220817131023.4093773-4-alex@linutronix.de |
|---|---|
| State | Accepted, archived |
| Commit | 58f94471675aef9ac6d15637ac5d8e69cc304c7a |
| Headers | show |
| Series | [1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer | expand |
On 8/17/22 09:10, Alexander Kanavin wrote: > This script can be used directly from poky, or can be copied directly into a Does this work if you are not using poky? Philip > layer or any other repository - it is self-suffucient and requires only python3 > and git on the host where it will run. It is also copied by the bitbake-layers > layers-setup plugin together with the json, unless requested otherwise. > > 1. How to restore the layers from the saved configuration: > > a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. > (use 'bitbake-layers create-layer-setup' from the previous commit to create them) > > b) Running with default options: > (note: this will work to update an existing checkout as well) > > alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers > Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. > > Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master > Running 'git init -q /srv/work/alex/my-build/meta-intel' > Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel > Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel > > Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers > Running 'git init -q /srv/work/alex/my-build/poky' > Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky > Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky > Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky > Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky > Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky > > Available build configurations: > /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget > /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo > /srv/work/alex/my-build/poky/meta-poky/conf > > You can set up a build with: > TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env > > 2. Command line options: > > alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h > usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > optional arguments: > -h, --help show this help message and exit > --force-bootstraplayer-checkout > Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). > --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > scripts/oe-setup-layers | 89 +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 89 insertions(+) > create mode 100755 scripts/oe-setup-layers > > diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers > new file mode 100755 > index 0000000000..fe948898b0 > --- /dev/null > +++ b/scripts/oe-setup-layers > @@ -0,0 +1,89 @@ > +#!/usr/bin/env python3 > +# > +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running > +# > +# bitbake-layers create-layers-setup destdir > +# > +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. > +# > + > +import argparse > +import json > +import os > +import subprocess > + > +def _do_checkout(args, json): > + layers = json['sources'] > + buildconfs = [] > + oecorepath = "" > + for l_name in layers: > + l_data = layers[l_name] > + layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) > + > + for ll_name in l_data['layers']: > + if ll_name == 'meta': > + oecorepath = layerdir > + ll_data = l_data['layers'][ll_name] > + if 'buildconfigs' in ll_data: > + for c in ll_data['buildconfigs']: > + buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) > + > + if 'contains_this_file' in l_data.keys(): > + force_arg = 'force_bootstraplayer_checkout' > + if not args[force_arg]: > + print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) > + continue > + l_remote = l_data['git-remote'] > + rev = l_remote['rev'] > + desc = l_remote['describe'] > + if not desc: > + desc = rev[:10] > + branch = l_remote['branch'] > + remotes = l_remote['remotes'] > + > + print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) > + cmd = 'git init -q {}'.format(layerdir) > + print("Running '{}'".format(cmd)) > + subprocess.check_output(cmd, shell=True) > + > + for remote in remotes: > + cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri']) > + print("Running '{}' in {}".format(cmd, layerdir)) > + subprocess.check_output(cmd, shell=True, cwd=layerdir) > + > + cmd = "git fetch -q {} || true".format(remote) > + print("Running '{}' in {}".format(cmd, layerdir)) > + subprocess.check_output(cmd, shell=True, cwd=layerdir) > + > + cmd = 'git checkout -q {}'.format(rev) > + print("Running '{}' in {}".format(cmd, layerdir)) > + subprocess.check_output(cmd, shell=True, cwd=layerdir) > + > + > + print("\nAvailable build configurations:\n{}\n".format("\n".join(buildconfs))) > + print("You can set up a build with:\nTEMPLATECONF=one/of/the/above . {}/oe-init-build-env".format(oecorepath)) > + > + > +parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") > + > +parser.add_argument('--force-bootstraplayer-checkout', action='store_true', > + help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).') > + > +try: > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', universal_newlines=True, shell=True, cwd=os.path.dirname(__file__))) > +except subprocess.CalledProcessError as e: > + defaultdest = os.path.abspath(".") > + > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > +parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) > + > +args = parser.parse_args() > + > +with open(args.jsondata) as f: > + json = json.load(f) > + > +supported_versions = ["1.0"] > +if json["version"] not in supported_versions: > + raise Exception("File {} has version {}, which is not in supported versions: {}".format(args.jsondata, json["version"], supported_versions)) > + > +_do_checkout(vars(args), json) > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#169481): https://lists.openembedded.org/g/openembedded-core/message/169481 > Mute This Topic: https://lists.openembedded.org/mt/93080238/384425 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [philip@balister.org] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, Aug 17, 2022 at 6:10 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > This script can be used directly from poky, or can be copied directly into > a > layer or any other repository - it is self-suffucient and requires only > python3 > and git on the host where it will run. It is also copied by the > bitbake-layers > layers-setup plugin together with the json, unless requested otherwise. > I need to review the code a bit closer, but on the face of it, this appears to solve the repository pinning problem quite elegantly. This matters a great deal in tightly configuration controlled environments, such as where regulatory oversight is involved. One feedback item I might suggest is to check if the layer is already pulled and to avoid pulling again. Same goes for the commit hash - no reason to checkout the hash if HEAD is already pointing at it. Adding those checks would enable your script to be added as a pre-build step in a developer workflow wrapper script. Our environment takes a different approach to repository pinning, but if setup-layers had those checks, we could probably switch to this approach. ..Ch:W..
On Mon, 22 Aug 2022 at 00:04, Philip Balister <philip@balister.org> wrote: > > On 8/17/22 09:10, Alexander Kanavin wrote: > > This script can be used directly from poky, or can be copied directly into a > > Does this work if you are not using poky? Yes. It's a case of 'poky chauvinism' on my part. I'll edit the commit messages to reassure everyone :) https://en.wikipedia.org/wiki/Carbon_chauvinism Alex
On Mon, 22 Aug 2022 at 03:20, Chuck Wolber <chuckwolber@gmail.com> wrote: > One feedback item I might suggest is to check if the layer is already pulled and to avoid pulling > again. Same goes for the commit hash - no reason to checkout the hash if HEAD is already pointing > at it. The script does 'git fetch' which is an incremental operation, and becomes cheap and quick after the initial fetch. Same for the checkout, which is trivial. I'd rather not complicate the code with additional conditions, unless there's some scenario I am not aware of. The script needs to be robust on a bulletproof level. > Adding those checks would enable your script to be added as a pre-build step in a developer > workflow wrapper script. Our environment takes a different approach to repository pinning, but if > setup-layers had those checks, we could probably switch to this approach. Please explain the details, I'd like to understand where the problem is specifically. Alex
diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 0000000000..fe948898b0 --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. +# + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): + layers = json['sources'] + buildconfs = [] + oecorepath = "" + for l_name in layers: + l_data = layers[l_name] + layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + + for ll_name in l_data['layers']: + if ll_name == 'meta': + oecorepath = layerdir + ll_data = l_data['layers'][ll_name] + if 'buildconfigs' in ll_data: + for c in ll_data['buildconfigs']: + buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + + if 'contains_this_file' in l_data.keys(): + force_arg = 'force_bootstraplayer_checkout' + if not args[force_arg]: + print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) + continue + l_remote = l_data['git-remote'] + rev = l_remote['rev'] + desc = l_remote['describe'] + if not desc: + desc = rev[:10] + branch = l_remote['branch'] + remotes = l_remote['remotes'] + + print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) + cmd = 'git init -q {}'.format(layerdir) + print("Running '{}'".format(cmd)) + subprocess.check_output(cmd, shell=True) + + for remote in remotes: + cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri']) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + cmd = "git fetch -q {} || true".format(remote) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + cmd = 'git checkout -q {}'.format(rev) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + + print("\nAvailable build configurations:\n{}\n".format("\n".join(buildconfs))) + print("You can set up a build with:\nTEMPLATECONF=one/of/the/above . {}/oe-init-build-env".format(oecorepath)) + + +parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") + +parser.add_argument('--force-bootstraplayer-checkout', action='store_true', + help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).') + +try: + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', universal_newlines=True, shell=True, cwd=os.path.dirname(__file__))) +except subprocess.CalledProcessError as e: + defaultdest = os.path.abspath(".") + +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) +parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) + +args = parser.parse_args() + +with open(args.jsondata) as f: + json = json.load(f) + +supported_versions = ["1.0"] +if json["version"] not in supported_versions: + raise Exception("File {} has version {}, which is not in supported versions: {}".format(args.jsondata, json["version"], supported_versions)) + +_do_checkout(vars(args), json)
This script can be used directly from poky, or can be copied directly into a layer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky Available build configurations: /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo /srv/work/alex/my-build/poky/meta-poky/conf You can set up a build with: TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --help show this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- scripts/oe-setup-layers | 89 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100755 scripts/oe-setup-layers