| Message ID | 20231206092536.2874022-1-andrey.z@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | scripts/oe-setup-layers: add option to skip revs in json | expand |
Unfortunately I have to say no to this. The problem is that this will update to latest revisions on all the layers, which I believe is not what people would want, as everything including poky etc. will be shifting uncontrollably. Conceptually, I would not want oe-setup-layers to suffer from feature creep and become bloated and difficult to maintain. Every little enhancement is pushing in that direction. It has a json, it does what the json prescribes, end of story. We've previously discussed with Jermain how this can be solved on the bitbake-layers side, e.g. that tool would write out a json where some layers don't have a hardcoded revision. I think work to implement that has stalled, not sure. But it's not difficult to add. Alex On Wed, 6 Dec 2023 at 10:25, Andrey Zhizhikin <andrey.z@gmail.com> wrote: > > Current script implementation uses revisions recorded in JSON file to > replicate previously recorded setup. While this is useful for complete > layer setup reproducibility, this does not allow the setup "upgrade" where > this script can be used to use remote branch information and latest HEAD, > have the setup restored, and then saved using updated revisions. > > Add new '--skip-revs' parameter which can be used to obtain the updated > layer setup. > > Following set of operations could be executed to achieve this: > 1. Restore initial layer setup using remote branch HEAD instead of recorded > revisions by specifying --skip-revs. > 2. After layer setup is created, JSON file containing updated revisions is > retrieved via `bitbake-layers create-layers-setup` > 3. Updated JSON file can be checked in to bootstrap repository and used > later without '--skip-revs' to replicated updated setup. > > Signed-off-by: Andrey Zhizhikin <andrey.z@gmail.com> > --- > scripts/oe-setup-layers | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers > index 6d49688a32..a4941450d3 100755 > --- a/scripts/oe-setup-layers > +++ b/scripts/oe-setup-layers > @@ -69,6 +69,13 @@ def _do_checkout(args, json): > remotes = r_remote['remotes'] > > print('\nSetting up source {}, revision {}, branch {}'.format(r_name, desc, branch)) > + if args['skip_revs']: > + if branch and (branch != 'HEAD'): > + print('\nSkip revision option used, remote {} branch {} HEAD will be used instead of {}'.format(r_name, branch, rev)) > + rev = branch > + else: > + print('\nBranch info is missing in json, --skip-revs option is not used') > + > if not _is_repo_git_repo(repodir): > cmd = 'git init -q {}'.format(repodir) > print("Running '{}'".format(cmd)) > @@ -108,6 +115,7 @@ except subprocess.CalledProcessError as e: > > parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) > +parser.add_argument('--skip-revs', action='store_true', help='Skip revisions recorded in json manifest, and use latest revision on branch to restore setup') > > args = parser.parse_args() > > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#191879): https://lists.openembedded.org/g/openembedded-core/message/191879 > Mute This Topic: https://lists.openembedded.org/mt/103009912/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hello Alex, On Wed, Dec 6, 2023 at 11:08 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > Unfortunately I have to say no to this. The problem is that this will > update to latest revisions on all the layers, which I believe is not > what people would want, as everything including poky etc. will be > shifting uncontrollably. Correct, and that is exactly the reason why this mechanism is only offered via additional switch. If it is not supplied - then behavior is exactly the same, and this is intended. Only those users who would really like to exercise the shift would use it. I've also prepared a patch for documenting this switch, but held it back until the functionality is in. There I do explain the case for CI systems which would want to capture updated setup of all layers in collection. > > Conceptually, I would not want oe-setup-layers to suffer from feature > creep and become bloated and difficult to maintain. Every little > enhancement is pushing in that direction. It has a json, it does what > the json prescribes, end of story. In this regard I think that json prescription is too strict. While this totally suites the needs and purpose of passing over the layer setup to any other client, it really limits the usability of setup-layers only to that specific case. If this was originally an intended design, then perhaps a note in documentation would be needed. > > We've previously discussed with Jermain how this can be solved on the > bitbake-layers side, e.g. that tool would write out a json where some > layers don't have a hardcoded revision. I think work to implement that > has stalled, not sure. But it's not difficult to add. Is there any mail thread or patch stack where this has been discussed? I would like to take a look at what was proposed and perhaps re-use and re-establish that part instead. Thanks a lot! > > Alex > > > > On Wed, 6 Dec 2023 at 10:25, Andrey Zhizhikin <andrey.z@gmail.com> wrote: > > > > Current script implementation uses revisions recorded in JSON file to > > replicate previously recorded setup. While this is useful for complete > > layer setup reproducibility, this does not allow the setup "upgrade" where > > this script can be used to use remote branch information and latest HEAD, > > have the setup restored, and then saved using updated revisions. > > > > Add new '--skip-revs' parameter which can be used to obtain the updated > > layer setup. > > > > Following set of operations could be executed to achieve this: > > 1. Restore initial layer setup using remote branch HEAD instead of recorded > > revisions by specifying --skip-revs. > > 2. After layer setup is created, JSON file containing updated revisions is > > retrieved via `bitbake-layers create-layers-setup` > > 3. Updated JSON file can be checked in to bootstrap repository and used > > later without '--skip-revs' to replicated updated setup. > > > > Signed-off-by: Andrey Zhizhikin <andrey.z@gmail.com> > > --- > > scripts/oe-setup-layers | 8 ++++++++ > > 1 file changed, 8 insertions(+) > > > > diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers > > index 6d49688a32..a4941450d3 100755 > > --- a/scripts/oe-setup-layers > > +++ b/scripts/oe-setup-layers > > @@ -69,6 +69,13 @@ def _do_checkout(args, json): > > remotes = r_remote['remotes'] > > > > print('\nSetting up source {}, revision {}, branch {}'.format(r_name, desc, branch)) > > + if args['skip_revs']: > > + if branch and (branch != 'HEAD'): > > + print('\nSkip revision option used, remote {} branch {} HEAD will be used instead of {}'.format(r_name, branch, rev)) > > + rev = branch > > + else: > > + print('\nBranch info is missing in json, --skip-revs option is not used') > > + > > if not _is_repo_git_repo(repodir): > > cmd = 'git init -q {}'.format(repodir) > > print("Running '{}'".format(cmd)) > > @@ -108,6 +115,7 @@ except subprocess.CalledProcessError as e: > > > > parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > > parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) > > +parser.add_argument('--skip-revs', action='store_true', help='Skip revisions recorded in json manifest, and use latest revision on branch to restore setup') > > > > args = parser.parse_args() > > > > -- > > 2.34.1 > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#191879): https://lists.openembedded.org/g/openembedded-core/message/191879 > > Mute This Topic: https://lists.openembedded.org/mt/103009912/1686489 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > >
On Wed, 6 Dec 2023 at 12:51, Andrey Zhizhikin <andrey.z@gmail.com> wrote: > > Unfortunately I have to say no to this. The problem is that this will > > update to latest revisions on all the layers, which I believe is not > > what people would want, as everything including poky etc. will be > > shifting uncontrollably. > > Correct, and that is exactly the reason why this mechanism is only > offered via additional switch. If it is not supplied - then behavior is > exactly the same, and this is intended. Only those users who would > really like to exercise the shift would use it. If you want to add something like this, it can't be an all or nothing switch. There has to be a way to update only some of the layers, and have others stay where they are. Also, you might want to update to latest tag on a branch, rather than latest revision. There are many scenarios, and there's no way oe-setup-layers can cover them, and it shouldn't try. > I've also prepared a patch for documenting this switch, but held it back > until the functionality is in. There I do explain the case for CI systems > which would want to capture updated setup of all layers in collection. I understand and agree with the use case, I just don't think updating all layers to their latest revisions is what many users want, or that oe-setup-layers is the right place for this functionality. > In this regard I think that json prescription is too strict. > > While this totally suites the needs and purpose of passing over the layer > setup to any other client, it really limits the usability of setup-layers only > to that specific case. If this was originally an intended design, then > perhaps a note in documentation would be needed. json schema does not require having precise revisions of the layers. You can specify tags, branches or anything else that 'git checkout' understands, and oe-setup-layers would follow that. It's just that we currently don't have tools to write out anything except precise revisions. But you can edit the json after the fact to point to something else, then give it to oe-setup-layers, and it will work. > Is there any mail thread or patch stack where this has been discussed? > > I would like to take a look at what was proposed and perhaps re-use and > re-establish that part instead. Look for the patchset '[RFC 0/7] bitbake-layers: Add update-layers-setup' posted on 7 November, particularly the comments to the individual patches, where we get to the design that should work and cover the use cases. Alex
> Look for the patchset '[RFC 0/7] bitbake-layers: Add > update-layers-setup' posted on 7 November, particularly the comments > to the individual patches, where we get to the design that should work > and cover the use cases. I had to work on some other issues that needed my attention, so not too much progress since, however, I have reserved some time this month and was planning to work on this. Sincerely, Jermain Horsman
On Wed, 6 Dec 2023 at 13:38, Jermain Horsman <jermain.horsman@nedap.com> wrote: > > > Look for the patchset '[RFC 0/7] bitbake-layers: Add > > update-layers-setup' posted on 7 November, particularly the comments > > to the individual patches, where we get to the design that should work > > and cover the use cases. > > I had to work on some other issues that needed my attention, > so not too much progress since, however, I have reserved some > time this month and was planning to work on this. The patches are now in testing so Andrey can take a look and check if they handle he use case he wants to solve: https://git.yoctoproject.org/poky-contrib/log/?h=abelloni%2Fmaster-next&qt=author&q=horsman Alex
On Wed, 31 Jan 2024 at 15:30, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > > Look for the patchset '[RFC 0/7] bitbake-layers: Add > > > update-layers-setup' posted on 7 November, particularly the comments > > > to the individual patches, where we get to the design that should work > > > and cover the use cases. > > > > I had to work on some other issues that needed my attention, > > so not too much progress since, however, I have reserved some > > time this month and was planning to work on this. > > The patches are now in testing so Andrey can take a look and check if > they handle he use case he wants to solve: > > https://git.yoctoproject.org/poky-contrib/log/?h=abelloni%2Fmaster-next&qt=author&q=horsman And now looks like they're not. Alexandre, was there some reason to drop them? [PATCH 1/2] bitbake-layers: Add ability to update the reference of repositories [PATCH 2/2] bitbake-layers: Add test case layers setup for custom references Alex
On 06/02/2024 15:54:27+0100, Alexander Kanavin wrote: > On Wed, 31 Jan 2024 at 15:30, Alexander Kanavin via > lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> > wrote: > > > > Look for the patchset '[RFC 0/7] bitbake-layers: Add > > > > update-layers-setup' posted on 7 November, particularly the comments > > > > to the individual patches, where we get to the design that should work > > > > and cover the use cases. > > > > > > I had to work on some other issues that needed my attention, > > > so not too much progress since, however, I have reserved some > > > time this month and was planning to work on this. > > > > The patches are now in testing so Andrey can take a look and check if > > they handle he use case he wants to solve: > > > > https://git.yoctoproject.org/poky-contrib/log/?h=abelloni%2Fmaster-next&qt=author&q=horsman > > And now looks like they're not. Alexandre, was there some reason to drop them? > > [PATCH 1/2] bitbake-layers: Add ability to update the reference of repositories > [PATCH 2/2] bitbake-layers: Add test case layers setup for custom references > RP didn't take them for a while and they conflict with your other series that I wanted to test. I moved those to my staging branch so I remember to come back to them.
On Tue, 6 Feb 2024 at 20:02, Alexandre Belloni <alexandre.belloni@bootlin.com> wrote: > RP didn't take them for a while and they conflict with your other series > that I wanted to test. I moved those to my staging branch so I remember > to come back to them. Thanks, that's fine. Alex
> RP didn't take them for a while and they conflict with your other series > that I wanted to test. I moved those to my staging branch so I remember > to come back to them. At a quick glance it's probably because we both updated the end of the selftest file. Anyway let me know if/when I need to update anything. Sincerely, Jermain Horsman
On 06/02/2024 19:23:58+0000, Jermain Horsman wrote: > > RP didn't take them for a while and they conflict with your other series > > that I wanted to test. I moved those to my staging branch so I remember > > to come back to them. > > At a quick glance it's probably because we both updated the end of the selftest file. > Anyway let me know if/when I need to update anything. > Yes it would be great if you could rebase on top of my master-next. > Sincerely, > > Jermain Horsman
diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers index 6d49688a32..a4941450d3 100755 --- a/scripts/oe-setup-layers +++ b/scripts/oe-setup-layers @@ -69,6 +69,13 @@ def _do_checkout(args, json): remotes = r_remote['remotes'] print('\nSetting up source {}, revision {}, branch {}'.format(r_name, desc, branch)) + if args['skip_revs']: + if branch and (branch != 'HEAD'): + print('\nSkip revision option used, remote {} branch {} HEAD will be used instead of {}'.format(r_name, branch, rev)) + rev = branch + else: + print('\nBranch info is missing in json, --skip-revs option is not used') + if not _is_repo_git_repo(repodir): cmd = 'git init -q {}'.format(repodir) print("Running '{}'".format(cmd)) @@ -108,6 +115,7 @@ except subprocess.CalledProcessError as e: parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) +parser.add_argument('--skip-revs', action='store_true', help='Skip revisions recorded in json manifest, and use latest revision on branch to restore setup') args = parser.parse_args()
Current script implementation uses revisions recorded in JSON file to replicate previously recorded setup. While this is useful for complete layer setup reproducibility, this does not allow the setup "upgrade" where this script can be used to use remote branch information and latest HEAD, have the setup restored, and then saved using updated revisions. Add new '--skip-revs' parameter which can be used to obtain the updated layer setup. Following set of operations could be executed to achieve this: 1. Restore initial layer setup using remote branch HEAD instead of recorded revisions by specifying --skip-revs. 2. After layer setup is created, JSON file containing updated revisions is retrieved via `bitbake-layers create-layers-setup` 3. Updated JSON file can be checked in to bootstrap repository and used later without '--skip-revs' to replicated updated setup. Signed-off-by: Andrey Zhizhikin <andrey.z@gmail.com> --- scripts/oe-setup-layers | 8 ++++++++ 1 file changed, 8 insertions(+)