| Message ID | 20220901142328.268769-1-alex@linutronix.de |
|---|---|
| State | Accepted, archived |
| Commit | 9da0dd350a451676e3d4a1f09f90ec0234047ab7 |
| Headers | show |
| Series | scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations | expand |
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Alexander Kanavin > Sent: den 1 september 2022 16:23 > To: openembedded-core@lists.openembedded.org > Cc: Alexander Kanavin <alex@linutronix.de> > Subject: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations > > This is done only for default oe-core/poky templates; for anything > else the locations themselves need to be migrated first, and there > is no way to tell where they would be. > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > scripts/oe-setup-builddir | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir > index bf832ee0ca..32bc6580eb 100755 > --- a/scripts/oe-setup-builddir > +++ b/scripts/oe-setup-builddir > @@ -40,6 +40,12 @@ cd "$BUILDDIR" > > if [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then > TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg") > + # The following two are no longer valid; unsetting them will automatically get them replaced > + # with correct ones. > + if [ $TEMPLATECONF = "meta/conf" -o $TEMPLATECONF = "meta-poky/conf" ]; then > + unset TEMPLATECONF > + rm $BUILDDIR/conf/templateconf.cfg > + fi Since the test you added previously now requires that the path contains "/templates/", we might as well do this for any faulty configuration. I.e.: # The path in $TEMPLATECONF must contain "/templates/". If it doesn't, # unset it to have it replaced by a (hopefully) correct path. case $TEMPLATECONF in */templates/*) ;; *) unset TEMPLATECONF rm "$BUILDDIR/conf/templateconf.cfg" ;; esac If .templateconf hasn't been updated to comply with the new rules, the test later will point this out. But if it has been updated, the case statement above will avoid the problem of an old, non-complying configuration regardless of which layer it comes from. > fi > > . "$OEROOT"/.templateconf > -- > 2.30.2 //Peter
On Fri, 2 Sept 2022 at 02:12, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > Since the test you added previously now requires that the path contains > "/templates/", we might as well do this for any faulty configuration. I.e.: > > # The path in $TEMPLATECONF must contain "/templates/". If it doesn't, > # unset it to have it replaced by a (hopefully) correct path. > case $TEMPLATECONF in > */templates/*) > ;; > *) > unset TEMPLATECONF > rm "$BUILDDIR/conf/templateconf.cfg" > ;; > esac > > If .templateconf hasn't been updated to comply with the new rules, the > test later will point this out. But if it has been updated, the case > statement above will avoid the problem of an old, non-complying > configuration regardless of which layer it comes from. .templateconf is not a universal file that can be placed into any layer, it is specific to poky/oe-core, as it is read from $OEROOT, which in turn is defined as location of oe-init-build-env. So if the fixing up is made generic as suggested, it will only replace incorrect but at least helpful value of the original custom template location with the generic default template location in poky/core. We can however add a hint to check what is in build/conf/templateconf.cfg if the test fails, as it is not obvious that TEMPLATECONF obtained from there takes priority over other sources (which are, in order, TEMPLATECONF passed via environment, then .templateconf next to oe-init-build-env). Alex
On Fri, 2 Sept 2022 at 06:32, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > We can however add a hint to check what is in > build/conf/templateconf.cfg if the test fails, as it is not obvious > that TEMPLATECONF obtained from there takes priority over other > sources (which are, in order, TEMPLATECONF passed via environment, > then .templateconf next to oe-init-build-env). Ok, the more I think about this, the more I'm convinced that oe-setup-builddir needs a complete refactor. Specifically: 1. Let's rewrite it in python; shell just isn't expressive or readable enough for what it needs to do. 2. Let's have a clear, logical priority order: TEMPLATECONF env var, build/conf/templateconf.cfg, dirname('oe-init-build-env') + '/.templateconf' 3. build/conf/templateconf.cfg becomes read-only; the final selected value for TEMPLATECONF is written into a separate file (for having a reference to where the configs came from). 4. oe-setup-builddir can also later be extended to support 'configuration fragments' management, see my email to oe-archtecture from yesterday. It's just too late to do this in this release cycle, but worth looking into afterwards. Alex
On Fri, 2022-09-02 at 09:16 +0200, Alexander Kanavin wrote: > On Fri, 2 Sept 2022 at 06:32, Alexander Kanavin via > lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> > wrote: > > We can however add a hint to check what is in > > build/conf/templateconf.cfg if the test fails, as it is not obvious > > that TEMPLATECONF obtained from there takes priority over other > > sources (which are, in order, TEMPLATECONF passed via environment, > > then .templateconf next to oe-init-build-env). > > Ok, the more I think about this, the more I'm convinced that > oe-setup-builddir needs a complete refactor. That script is meant to be really really simple and push any heavy lifting off to bitbake or other tools. Ideally we were never even meant to need such a script and the tools would "just work". I'd prefer it went away, not rewrite and grow it. Cheers, Richard
On Fri, 2 Sept 2022 at 09:37, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > > We can however add a hint to check what is in > > > build/conf/templateconf.cfg if the test fails, as it is not obvious > > > that TEMPLATECONF obtained from there takes priority over other > > > sources (which are, in order, TEMPLATECONF passed via environment, > > > then .templateconf next to oe-init-build-env). > > > > Ok, the more I think about this, the more I'm convinced that > > oe-setup-builddir needs a complete refactor. > > That script is meant to be really really simple and push any heavy > lifting off to bitbake or other tools. Ideally we were never even meant > to need such a script and the tools would "just work". I'd prefer it > went away, not rewrite and grow it. Rewriting it in python will make it twice (or more) shorter, and the order of templateconfs does need clarity and logic. We can leave the 'config fragments' or anything more advanced out. Alex
Ok, I think I overstated things :) a couple of small targeted fixes incoming, no python rewrite. Alex On Fri 2. Sep 2022 at 9.41, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > On Fri, 2 Sept 2022 at 09:37, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > We can however add a hint to check what is in > > > > build/conf/templateconf.cfg if the test fails, as it is not obvious > > > > that TEMPLATECONF obtained from there takes priority over other > > > > sources (which are, in order, TEMPLATECONF passed via environment, > > > > then .templateconf next to oe-init-build-env). > > > > > > Ok, the more I think about this, the more I'm convinced that > > > oe-setup-builddir needs a complete refactor. > > > > That script is meant to be really really simple and push any heavy > > lifting off to bitbake or other tools. Ideally we were never even meant > > to need such a script and the tools would "just work". I'd prefer it > > went away, not rewrite and grow it. > > Rewriting it in python will make it twice (or more) shorter, and the > order of templateconfs does need clarity and logic. We can leave the > 'config fragments' or anything more advanced out. > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#170238): > https://lists.openembedded.org/g/openembedded-core/message/170238 > Mute This Topic: https://lists.openembedded.org/mt/93396426/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 2 september 2022 06:32 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: openembedded-core@lists.openembedded.org; Alexander Kanavin > <alex@linutronix.de> > Subject: Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate > build/conf/templateconf.cfg to new template locations > > On Fri, 2 Sept 2022 at 02:12, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > Since the test you added previously now requires that the path contains > > "/templates/", we might as well do this for any faulty configuration. > I.e.: > > > > # The path in $TEMPLATECONF must contain "/templates/". If it doesn't, > > # unset it to have it replaced by a (hopefully) correct path. > > case $TEMPLATECONF in > > */templates/*) > > ;; > > *) > > unset TEMPLATECONF > > rm "$BUILDDIR/conf/templateconf.cfg" > > ;; > > esac > > > > If .templateconf hasn't been updated to comply with the new rules, the > > test later will point this out. But if it has been updated, the case > > statement above will avoid the problem of an old, non-complying > > configuration regardless of which layer it comes from. > > .templateconf is not a universal file that can be placed into any > layer, it is specific to poky/oe-core, as it is read from $OEROOT, > which in turn is defined as location of oe-init-build-env. That assumption doesn't hold true. Our environment is set up to mimic Poky, but with extra layers and our own version of ".templateconf" (installed via a <copyfile> element in the repo manifest). So everything works and looks like Poky, except there are more layers available. With my suggested change above the code just works, without any artificial limits such as only working for plain OE/Poky. > So if the fixing up is made generic as suggested, it will only replace > incorrect but at least helpful value of the original custom template > location with the generic default template location in poky/core. The expectation is of course that if one has one's own .templateconf file, then it will of course have to be updated to match the changes in OE. Nothing unusual with that. But that is something I do as maintainer of our environment. For our developers it will then just work. > > We can however add a hint to check what is in > build/conf/templateconf.cfg if the test fails, as it is not obvious > that TEMPLATECONF obtained from there takes priority over other > sources (which are, in order, TEMPLATECONF passed via environment, > then .templateconf next to oe-init-build-env). > > Alex //Peter
On Fri, 2 Sept 2022 at 16:47, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > The expectation is of course that if one has one's own .templateconf > file, then it will of course have to be updated to match the changes > in OE. Nothing unusual with that. But that is something I do as > maintainer of our environment. For our developers it will then just > work. Please consider that there are three different ways to obtain the location of the template, and they must all work. For those who have created a build/conf by supplying TEMPLATECONF on the command line or templateconf.cfg (which I would say is everyone, except you) the suggested code will only make things worse, not better. It would take wrong but helpful value in templateconf.cfg that points to a custom template, and replace it with meta/conf/templates/default, because that is what they have in .templateconf. Then all trace of where the template came from is lost. And as discussed previously, you don't even need to jump through the extra hoops with generated template configurations to begin with, and can write the dynamically generated configs directly into build/conf. Alex
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 2 september 2022 17:09 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: openembedded-core@lists.openembedded.org; Alexander Kanavin > <alex@linutronix.de> > Subject: Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate > build/conf/templateconf.cfg to new template locations > > On Fri, 2 Sept 2022 at 16:47, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > > The expectation is of course that if one has one's own .templateconf > > file, then it will of course have to be updated to match the changes > > in OE. Nothing unusual with that. But that is something I do as > > maintainer of our environment. For our developers it will then just > > work. > > Please consider that there are three different ways to obtain the > location of the template, and they must all work. For those who have > created a build/conf by supplying TEMPLATECONF on the command line or > templateconf.cfg (which I would say is everyone, except you) the > suggested code will only make things worse, not better. It would take > wrong but helpful value in templateconf.cfg that points to a custom > template, and replace it with meta/conf/templates/default, because > that is what they have in .templateconf. Then all trace of where the > template came from is lost. Actually, that does not correspond to what the documentation says. The way we do things (i.e., modify .templateconf to define a default TEMPLATECONF) _is_ what the documentation says one should do. See: https://docs.yoctoproject.org/dev/singleindex.html#creating-a-custom-template-configuration-directory Also note that templateconf.cfg is not documented at all. My guess (based on the current code) is that it is only supposed to be used internally as a way to store the value of a previous build where TEMPLATECONF was explicitly set. So based on the documentation, the officially supported ways to configure the template directory is to either specify TEMPLATECONF explicitly or by modifying .templateconf to set a default value. That said, I do agree with your suggested change to prefer an explicitly set TEMPLATECONF before a value from templateconf.cfg, which also agrees with the idea that templateconf.cfg is only an internal way of storing the last TEMPLATECONF used. > > And as discussed previously, you don't even need to jump through the > extra hoops with generated template configurations to begin with, and > can write the dynamically generated configs directly into build/conf. > > Alex //Peter
On Fri, 2 Sept 2022 at 17:41, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > So based on the documentation, the officially supported ways to > configure the template directory is to either specify TEMPLATECONF > explicitly or by modifying .templateconf to set a default value. Okay, I read that section, and I still think the requirement to have .templateconf next to oe-init-build-env makes it altogether impractical as a way to change the default from what poky/oe-core prescribes, unless you want to maintain yet another custom script doing non-standard magic before a build can start. I'm going to modify the section so that TEMPLATECONF passed in explicitly is emphasized as the standard method to set the template, and .templateconf is described as primarily a fallback to the standard core/poky template. Maybe we can decouple .templateconf from oe-init-build-env but I don't know how. Alex
diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir index bf832ee0ca..32bc6580eb 100755 --- a/scripts/oe-setup-builddir +++ b/scripts/oe-setup-builddir @@ -40,6 +40,12 @@ cd "$BUILDDIR" if [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg") + # The following two are no longer valid; unsetting them will automatically get them replaced + # with correct ones. + if [ $TEMPLATECONF = "meta/conf" -o $TEMPLATECONF = "meta-poky/conf" ]; then + unset TEMPLATECONF + rm $BUILDDIR/conf/templateconf.cfg + fi fi . "$OEROOT"/.templateconf
This is done only for default oe-core/poky templates; for anything else the locations themselves need to be migrated first, and there is no way to tell where they would be. Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- scripts/oe-setup-builddir | 6 ++++++ 1 file changed, 6 insertions(+)