| Message ID | 20220302190554.1953968-1-dwagenknecht@emlix.com |
|---|---|
| State | Accepted, archived |
| Commit | 16d330d42e03085769eddb1b60ba1df7228baf36 |
| Headers | show |
| Series | copy_buildsystem: allow more layer paths | expand |
Hi Daniel, Could you please give here the examples how the layer structure looks before and after change? I want to see how transformation looks like. Regards, Andrej On Wed, 2022-03-02 at 20:05 +0100, Daniel Wagenknecht wrote: > Layers could be located anywhere. The eSDK should work with them even > if > they are not located in TOPDIR or in the same parent directory as > COREBASE. > > For layers located in the same parent directory as COREBASE this > preserves > the intent from the previous > copy_buildsystem: include layer tree during build structure > creation > commit. > > Related OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b > > Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> > --- > This patch resolves issues like > ERROR: my-image-1.0-r0 do_populate_sdk_ext: Failed to generate > filtered task list for extensible SDK: > > ### Shell environment set up for builds. ### > [...] > > ERROR: bitbake failed: > ERROR: The following layer directories do not exist: > ERROR: /build/tmp/work/my-board-linux/my-image/1.0-r0/sdk- > ext/image/tmp-renamed-sdk/layers/../../../repo/layers/meta-my-layer > ERROR: Please check BBLAYERS in /build/tmp/work/my-board-linux/my- > image/1.0-r0/sdk-ext/image/tmp-renamed-sdk/conf/bblayers.conf > ERROR: Logfile of failure stored in: /build/tmp/work/my-board- > linux/my-image/1.0-r0/temp/log.do_populate_sdk_ext.68844 > > I tried to preserve the special casing to preserve the layer tree > e.g. get the > following layer-structure in the eSDK: > layers/poky/meta > layers/meta-openembedded/meta-networking > layers/meta-openembedded/meta-oe > For layers that are not located in a directory tree right next to > COREBASE we > don't have an anchor to determine what part of the layer tree we > should keep, > thus the layer tree will be flattened. > > Alternative to this patch: > Delete the whole else / elseif block. This would lead to a layer > structure in > the eSDK like > layers/poky/meta > layers/meta-networking > layers/meta-oe > thus flattening the layer tree. I'm personally not opposed to this > approach, > but both 5a59a6997f41e606d088e3e86812de56f72f543b and > 55ecf6988d3e3c0935cb6324a6ad2c75f1191a1d (OE-Core) show that there > seems to be > a need / preference for keeping the layer tree. > > > meta/lib/oe/copy_buildsystem.py | 12 ++++-------- > 1 file changed, 4 insertions(+), 8 deletions(-) > > diff --git a/meta/lib/oe/copy_buildsystem.py > b/meta/lib/oe/copy_buildsystem.py > index d97bf9d1b9..79642fd76a 100644 > --- a/meta/lib/oe/copy_buildsystem.py > +++ b/meta/lib/oe/copy_buildsystem.py > @@ -45,9 +45,6 @@ class BuildSystem(object): > > corebase = os.path.abspath(self.d.getVar('COREBASE')) > layers.append(corebase) > - # Get relationship between TOPDIR and COREBASE > - # Layers should respect it > - corebase_relative = > os.path.dirname(os.path.relpath(os.path.abspath(self.d.getVar('TOPDIR > ')), corebase)) > # The bitbake build system uses the meta-skeleton layer as a > layout > # for common recipies, e.g: the recipetool script to create > kernel recipies > # Add the meta-skeleton layer to be included as part of the > eSDK installation > @@ -100,11 +97,10 @@ class BuildSystem(object): > layerdestpath = destdir > if corebase == os.path.dirname(layer): > layerdestpath += '/' + os.path.basename(corebase) > - else: > - layer_relative = os.path.relpath(layer, corebase) > - if os.path.dirname(layer_relative) == > corebase_relative: > - layer_relative = > os.path.dirname(corebase_relative) + '/' + layernewname > - layer_relative = os.path.basename(corebase) + '/' + > layer_relative > + # If the layer is located somewhere under the same > parent directory > + # as corebase we keep the layer structure. > + elif os.path.commonpath([layer, corebase]) == > os.path.dirname(corebase): > + layer_relative = os.path.relpath(layer, > os.path.dirname(corebase)) > if os.path.dirname(layer_relative) != layernewname: > layerdestpath += '/' + > os.path.dirname(layer_relative) >
Hello Andrej, On Thu, 2022-03-03 at 06:35 +0000, Andrej Valek wrote: > Hi Daniel, > > Could you please give here the examples how the layer structure looks > before and after change? I want to see how transformation looks like. With a directory-structure like / ├── repo │ └── layers │ └── meta-my-layer └── work ├── build └── layers └── external ├── meta-openembedded │ ├── meta-networking │ └── meta-oe └── poky └── meta and # Set through bitbake itself COREBASE = "/work/layers/external/poky" TOPDIR = "/work/build" # Set in bblayers.conf BBLAYERS = " \ /repo/layers/meta-my-layer \ /work/layers/external/meta-openembedded/meta-networking \ /work/layers/external/meta-openembedded/meta-oe \ /work/layers/external/poky/meta" The resulting eSDK layers directory will look like this: . ├── meta-my-layer ├── meta-openembedded │ ├── meta-networking │ └── meta-oe └── poky └── meta Without this patch the /repo/meta-my-layer layer broke the build: > > This patch resolves issues like > > ERROR: my-image-1.0-r0 do_populate_sdk_ext: Failed to generate > > filtered task list for extensible SDK: > > > > ### Shell environment set up for builds. ### > > [...] > > > > ERROR: bitbake failed: > > ERROR: The following layer directories do not exist: > > ERROR: /build/tmp/work/my-board-linux/my-image/1.0-r0/sdk- > > ext/image/tmp-renamed-sdk/layers/../../../repo/layers/meta-my-layer > > ERROR: Please check BBLAYERS in /build/tmp/work/my-board-linux/my- > > image/1.0-r0/sdk-ext/image/tmp-renamed-sdk/conf/bblayers.conf > > ERROR: Logfile of failure stored in: /build/tmp/work/my-board- > > linux/my-image/1.0-r0/temp/log.do_populate_sdk_ext.68844 > > Without meta-my-layer this patch should not cause any change. The alternative > > Alternative to this patch: > > Delete the whole else / elseif block. This would lead to a layer > > structure in > > the eSDK like > > layers/poky/meta > > layers/meta-networking > > layers/meta-oe > > thus flattening the layer tree. would remove the special casing in the implementation (except for COREBASE sublayers), thus resulting in the following layer structure in the eSDKs layers directory: . ├── meta-my-layer ├── meta-networking ├── meta-oe └── poky └── meta -- Sincerely Daniel Wagenknecht
Hi Daniel,
thanks for the explanation. To be honest, when I was dealing with the
layer structure, I didn't take a case about the layer outside of "work"
directory.
Basically it's the same as mine, but respect the external layers
outside of work, right? So, your proposal makes sense to me.
layers
├── meta-my-layer
├── meta-openembedded
│ ├── meta-networking
│ └── meta-oe
└── poky
└── meta
Forget about the last variant. We don't want to remove the layer
structure.
Regards,
Andrej
On Fri, 2022-03-04 at 10:26 +0100, Daniel Wagenknecht wrote:
> Hello Andrej,
>
> On Thu, 2022-03-03 at 06:35 +0000, Andrej Valek wrote:
> > Hi Daniel,
> >
> > Could you please give here the examples how the layer structure looks
> > before and after change? I want to see how transformation looks like.
>
> With a directory-structure like
>
> /
> ├── repo
> │ └── layers
> │ └── meta-my-layer
> └── work
> ├── build
> └── layers
> └── external
> ├── meta-openembedded
> │ ├── meta-networking
> │ └── meta-oe
> └── poky
> └── meta
>
> and
>
> # Set through bitbake itself
> COREBASE = "/work/layers/external/poky"
> TOPDIR = "/work/build"
> # Set in bblayers.conf
> BBLAYERS = " \
> /repo/layers/meta-my-layer \
> /work/layers/external/meta-openembedded/meta-networking \
> /work/layers/external/meta-openembedded/meta-oe \
> /work/layers/external/poky/meta"
>
> The resulting eSDK layers directory will look like this:
>
> .
> ├── meta-my-layer
> ├── meta-openembedded
> │ ├── meta-networking
> │ └── meta-oe
> └── poky
> └── meta
>
> Without this patch the /repo/meta-my-layer layer broke the build:
> > > This patch resolves issues like
> > > ERROR: my-image-1.0-r0 do_populate_sdk_ext: Failed to generate
> > > filtered task list for extensible SDK:
> > >
> > > ### Shell environment set up for builds. ###
> > > [...]
> > >
> > > ERROR: bitbake failed:
> > > ERROR: The following layer directories do not exist:
> > > ERROR: /build/tmp/work/my-board-linux/my-image/1.0-r0/sdk-
> > > ext/image/tmp-renamed-sdk/layers/../../../repo/layers/meta-my-layer
> > > ERROR: Please check BBLAYERS in /build/tmp/work/my-board-
> > > linux/my-
> > > image/1.0-r0/sdk-ext/image/tmp-renamed-sdk/conf/bblayers.conf
> > > ERROR: Logfile of failure stored in: /build/tmp/work/my-board-
> > > linux/my-image/1.0-r0/temp/log.do_populate_sdk_ext.68844
> > >
>
> Without meta-my-layer this patch should not cause any change.
>
> The alternative
> > > Alternative to this patch:
> > > Delete the whole else / elseif block. This would lead to a layer
> > > structure in
> > > the eSDK like
> > > layers/poky/meta
> > > layers/meta-networking
> > > layers/meta-oe
> > > thus flattening the layer tree.
> would remove the special casing in the implementation (except for
> COREBASE
> sublayers), thus resulting in the following layer structure in the
> eSDKs layers
> directory:
>
> .
> ├── meta-my-layer
> ├── meta-networking
> ├── meta-oe
> └── poky
> └── meta
>
> --
> Sincerely
> Daniel Wagenknecht
>
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py index d97bf9d1b9..79642fd76a 100644 --- a/meta/lib/oe/copy_buildsystem.py +++ b/meta/lib/oe/copy_buildsystem.py @@ -45,9 +45,6 @@ class BuildSystem(object): corebase = os.path.abspath(self.d.getVar('COREBASE')) layers.append(corebase) - # Get relationship between TOPDIR and COREBASE - # Layers should respect it - corebase_relative = os.path.dirname(os.path.relpath(os.path.abspath(self.d.getVar('TOPDIR')), corebase)) # The bitbake build system uses the meta-skeleton layer as a layout # for common recipies, e.g: the recipetool script to create kernel recipies # Add the meta-skeleton layer to be included as part of the eSDK installation @@ -100,11 +97,10 @@ class BuildSystem(object): layerdestpath = destdir if corebase == os.path.dirname(layer): layerdestpath += '/' + os.path.basename(corebase) - else: - layer_relative = os.path.relpath(layer, corebase) - if os.path.dirname(layer_relative) == corebase_relative: - layer_relative = os.path.dirname(corebase_relative) + '/' + layernewname - layer_relative = os.path.basename(corebase) + '/' + layer_relative + # If the layer is located somewhere under the same parent directory + # as corebase we keep the layer structure. + elif os.path.commonpath([layer, corebase]) == os.path.dirname(corebase): + layer_relative = os.path.relpath(layer, os.path.dirname(corebase)) if os.path.dirname(layer_relative) != layernewname: layerdestpath += '/' + os.path.dirname(layer_relative)
Layers could be located anywhere. The eSDK should work with them even if they are not located in TOPDIR or in the same parent directory as COREBASE. For layers located in the same parent directory as COREBASE this preserves the intent from the previous copy_buildsystem: include layer tree during build structure creation commit. Related OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> --- This patch resolves issues like ERROR: my-image-1.0-r0 do_populate_sdk_ext: Failed to generate filtered task list for extensible SDK: ### Shell environment set up for builds. ### [...] ERROR: bitbake failed: ERROR: The following layer directories do not exist: ERROR: /build/tmp/work/my-board-linux/my-image/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/../../../repo/layers/meta-my-layer ERROR: Please check BBLAYERS in /build/tmp/work/my-board-linux/my-image/1.0-r0/sdk-ext/image/tmp-renamed-sdk/conf/bblayers.conf ERROR: Logfile of failure stored in: /build/tmp/work/my-board-linux/my-image/1.0-r0/temp/log.do_populate_sdk_ext.68844 I tried to preserve the special casing to preserve the layer tree e.g. get the following layer-structure in the eSDK: layers/poky/meta layers/meta-openembedded/meta-networking layers/meta-openembedded/meta-oe For layers that are not located in a directory tree right next to COREBASE we don't have an anchor to determine what part of the layer tree we should keep, thus the layer tree will be flattened. Alternative to this patch: Delete the whole else / elseif block. This would lead to a layer structure in the eSDK like layers/poky/meta layers/meta-networking layers/meta-oe thus flattening the layer tree. I'm personally not opposed to this approach, but both 5a59a6997f41e606d088e3e86812de56f72f543b and 55ecf6988d3e3c0935cb6324a6ad2c75f1191a1d (OE-Core) show that there seems to be a need / preference for keeping the layer tree. meta/lib/oe/copy_buildsystem.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)