| Message ID | 20230920105802.1008778-3-richard.purdie@linuxfoundation.org |
|---|---|
| State | Accepted, archived |
| Commit | 15f98f72e1730fed5164fcd539737a7a7b96a1a2 |
| Headers | show |
| Series | [1/6,v2] license/license_image: Fix license file layout to avoid overlapping files | expand |
Just FYI I think this change is now causing few more recipes to be mutually
exclusive, when they build the same library (even when it's packaged in
differently named package), in world builds I'm seeing e.g. libslirp and
libslirp-virt (from meta-virtualization) causing packagedata failure for
one of them (depending which one was built second):
DEBUG: Staging files from
TOPDIR/BUILD/work/raspberrypi4_64-oe-linux/libslirp-virt/4.6.1+git/pkgdata-pdata-input
to TOPDIR/BUILD/pkgdata/raspberrypi4-64
ERROR: Recipe libslirp-virt is trying to install files into a shared area
when those files already exist. The files and the manifests listing them
are:
TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dev
(matched in manifest-raspberrypi4_64-libslirp.packagedata)
TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp0
(matched in manifest-raspberrypi4_64-libslirp.packagedata)
TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dbg
(matched in manifest-raspberrypi4_64-libslirp.packagedata)
TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-src
(matched in manifest-raspberrypi4_64-libslirp.packagedata)
Please adjust the recipes so only one recipe provides a given file.
DEBUG: Python function sstate_task_postfunc finished
Bruce is 4.6.1 version in meta-virtualization still needed or can you
update to libslirp 4.7.0 from oe-core?
From the git log
https://git.yoctoproject.org/meta-virtualization/log/recipes-networking/slirp
it looks like it was originally imported from meta-retro and later renamed
from libslirp to libslirt-virt until the oe-core version is validated in
runtime.
And I'm seeing the same with some internal recipes (e.g. we have
faultmanager recipe which provides libfm - completely different from libfm
from oe-core, just library name coincidence).
Cheers,
On Wed, Sep 20, 2023 at 12:58 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> When originally implemented, overlapping symlinks in DEPLOY_DIR were
> common. That
> is no longer the case and these overlapping links are causing bugs in
> other areas
> (e.g. bug 14123).
>
> Therefore start showing errors for overlapping symlinks in shared areas.
> Whilst here,
> fix a broken file reference in the grep command to match current file
> layouts and
> update the message shown to users to match current times. Most of the
> message content
> is obsolete now due to other advances and changes in the way the staging
> code
> now works.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> meta/classes-global/sstate.bbclass | 30 ++++++------------------------
> 1 file changed, 6 insertions(+), 24 deletions(-)
>
> diff --git a/meta/classes-global/sstate.bbclass
> b/meta/classes-global/sstate.bbclass
> index 706c2ae9388..afcda2980b3 100644
> --- a/meta/classes-global/sstate.bbclass
> +++ b/meta/classes-global/sstate.bbclass
> @@ -266,7 +266,7 @@ def sstate_install(ss, d):
> overlap_allowed = (d.getVar("SSTATE_ALLOW_OVERLAP_FILES") or
> "").split()
> match = []
> for f in sharedfiles:
> - if os.path.exists(f) and not os.path.islink(f):
> + if os.path.exists(f):
> f = os.path.normpath(f)
> realmatch = True
> for w in overlap_allowed:
> @@ -276,36 +276,18 @@ def sstate_install(ss, d):
> break
> if realmatch:
> match.append(f)
> - sstate_search_cmd = "grep -rlF '%s' %s
> --exclude=master.list | sed -e 's:^.*/::'" % (f,
> d.expand("${SSTATE_MANIFESTS}"))
> + sstate_search_cmd = "grep -rlF '%s' %s --exclude=index-*
> | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
> search_output = subprocess.Popen(sstate_search_cmd,
> shell=True, stdout=subprocess.PIPE).communicate()[0]
> if search_output:
> match.append(" (matched in %s)" %
> search_output.decode('utf-8').rstrip())
> else:
> match.append(" (not matched to any task)")
> if match:
> - bb.error("The recipe %s is trying to install files into a shared
> " \
> - "area when those files already exist. Those files and their
> manifest " \
> - "location are:\n %s\nPlease verify which recipe should provide
> the " \
> - "above files.\n\nThe build has stopped, as continuing in this
> scenario WILL " \
> - "break things - if not now, possibly in the future (we've seen
> builds fail " \
> - "several months later). If the system knew how to recover from
> this " \
> - "automatically it would, however there are several different
> scenarios " \
> - "which can result in this and we don't know which one this is.
> It may be " \
> - "you have switched providers of something like virtual/kernel
> (e.g. from " \
> - "linux-yocto to linux-yocto-dev), in that case you need to
> execute the " \
> - "clean task for both recipes and it will resolve this error. It
> may be " \
> - "you changed DISTRO_FEATURES from systemd to udev or vice
> versa. Cleaning " \
> - "those recipes should again resolve this error, however
> switching " \
> - "DISTRO_FEATURES on an existing build directory is not
> supported - you " \
> - "should really clean out tmp and rebuild (reusing sstate should
> be safe). " \
> - "It could be the overlapping files detected are harmless in
> which case " \
> - "adding them to SSTATE_ALLOW_OVERLAP_FILES may be the correct
> solution. It could " \
> - "also be your build is including two different conflicting
> versions of " \
> - "things (e.g. bluez 4 and bluez 5 and the correct solution for
> that would " \
> - "be to resolve the conflict. If in doubt, please ask on the
> mailing list, " \
> - "sharing the error and filelist above." % \
> + bb.fatal("Recipe %s is trying to install files into a shared " \
> + "area when those files already exist. The files and the
> manifests listing " \
> + "them are:\n %s\n"
> + "Please adjust the recipes so only one recipe provides a given
> file. " % \
> (d.getVar('PN'), "\n ".join(match)))
> - bb.fatal("If the above message is too much, the simpler version
> is you're advised to wipe out tmp and rebuild (reusing sstate is fine).
> That will likely fix things in most (but not all) cases.")
>
> if ss['fixmedir'] and os.path.exists(ss['fixmedir'] +
> "/fixmepath.cmd"):
> sharedfiles.append(ss['fixmedir'] + "/fixmepath.cmd")
> --
> 2.39.2
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#187919):
> https://lists.openembedded.org/g/openembedded-core/message/187919
> Mute This Topic: https://lists.openembedded.org/mt/101475773/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
On Sun, 2023-09-24 at 11:14 +0200, Martin Jansa wrote: > Just FYI I think this change is now causing few more recipes to be mutually exclusive, when they build the same library (even when it's packaged in differently named package), in world builds I'm seeing e.g. libslirp and libslirp-virt (from meta-virtualization) causing packagedata failure for one of them (depending which one was built second): > > DEBUG: Staging files from TOPDIR/BUILD/work/raspberrypi4_64-oe-linux/libslirp-virt/4.6.1+git/pkgdata-pdata-input to TOPDIR/BUILD/pkgdata/raspberrypi4-64 > ERROR: Recipe libslirp-virt is trying to install files into a shared area when those files already exist. The files and the manifests listing them are: > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dev > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp0 > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dbg > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-src > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > Please adjust the recipes so only one recipe provides a given file. > DEBUG: Python function sstate_task_postfunc finished > > Bruce is 4.6.1 version in meta-virtualization still needed or can you update to libslirp 4.7.0 from oe-core? > From the git log https://git.yoctoproject.org/meta-virtualization/log/recipes-networking/slirp it looks like it was originally imported from meta-retro and later renamed from libslirp to libslirt-virt until the oe-core version is validated in runtime. > > And I'm seeing the same with some internal recipes (e.g. we have faultmanager recipe which provides libfm - completely different from libfm from oe-core, just library name coincidence). This might be safe to exclude due to the way pkgdata works, it is handled per workdir now. I'd need to check a few things but offhand I think it will be ok to allow specifically. Cheers, Richard
On Sun, 2023-09-24 at 11:14 +0200, Martin Jansa wrote: > Just FYI I think this change is now causing few more recipes to be mutually exclusive, when they build the same library (even when it's packaged in differently named package), in world builds I'm seeing e.g. libslirp and libslirp-virt (from meta-virtualization) causing packagedata failure for one of them (depending which one was built second): > > DEBUG: Staging files from TOPDIR/BUILD/work/raspberrypi4_64-oe-linux/libslirp-virt/4.6.1+git/pkgdata-pdata-input to TOPDIR/BUILD/pkgdata/raspberrypi4-64 > ERROR: Recipe libslirp-virt is trying to install files into a shared area when those files already exist. The files and the manifests listing them are: > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dev > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp0 > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dbg > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-src > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > Please adjust the recipes so only one recipe provides a given file. > DEBUG: Python function sstate_task_postfunc finished > > Bruce is 4.6.1 version in meta-virtualization still needed or can you update to libslirp 4.7.0 from oe-core? > From the git log https://git.yoctoproject.org/meta-virtualization/log/recipes-networking/slirp it looks like it was originally imported from meta-retro and later renamed from libslirp to libslirt-virt until the oe-core version is validated in runtime. > > And I'm seeing the same with some internal recipes (e.g. we have faultmanager recipe which provides libfm - completely different from libfm from oe-core, just library name coincidence). I did look into this and it *is* a real issue/bug. The output will be non-deterministic depending upon which is built first. The issue would "corrupt" anything which is reading data from pkgdata related to the recipes in question. The error is therefore correct and we need to do something about this. Cheers, Richard
In the cases I've seen in our internal recipes I can just PNBLACKLIST the one we never use (libfm) to make sure it's never built with faultmanager (as it was built only as part of world build anyway, nothing in our images really depends on it). So it wasn't something which would happen in regular builds, but I agree that ihe error is real and good that it will force us to make this a bit more deterministic. For some other internal cases I can also add SkipRecipe to make them mutually exclusive (they are MACHINE_ARCH implementations of the same thing, just not restricted with COMPATIBLE_MACHINE or such to make sure that only the preferred provider is built for given MACHINE). And for libslirp and libslirp-virt it looks like it was meant to be only temporary and this error will force it to be resolved a bit sooner. Regards, On Mon, Sep 25, 2023 at 4:59 PM Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Sun, 2023-09-24 at 11:14 +0200, Martin Jansa wrote: > > Just FYI I think this change is now causing few more recipes to be > mutually exclusive, when they build the same library (even when it's > packaged in differently named package), in world builds I'm seeing e.g. > libslirp and libslirp-virt (from meta-virtualization) causing packagedata > failure for one of them (depending which one was built second): > > > > DEBUG: Staging files from > TOPDIR/BUILD/work/raspberrypi4_64-oe-linux/libslirp-virt/4.6.1+git/pkgdata-pdata-input > to TOPDIR/BUILD/pkgdata/raspberrypi4-64 > > ERROR: Recipe libslirp-virt is trying to install files into a shared > area when those files already exist. The files and the manifests listing > them are: > > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dev > > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp0 > > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-dbg > > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > > TOPDIR/BUILD/pkgdata/raspberrypi4-64/runtime-reverse/libslirp-src > > (matched in manifest-raspberrypi4_64-libslirp.packagedata) > > Please adjust the recipes so only one recipe provides a given file. > > DEBUG: Python function sstate_task_postfunc finished > > > > Bruce is 4.6.1 version in meta-virtualization still needed or can you > update to libslirp 4.7.0 from oe-core? > > From the git log > https://git.yoctoproject.org/meta-virtualization/log/recipes-networking/slirp > it looks like it was originally imported from meta-retro and later renamed > from libslirp to libslirt-virt until the oe-core version is validated in > runtime. > > > > And I'm seeing the same with some internal recipes (e.g. we have > faultmanager recipe which provides libfm - completely different from libfm > from oe-core, just library name coincidence). > > I did look into this and it *is* a real issue/bug. The output will be > non-deterministic depending upon which is built first. The issue would > "corrupt" anything which is reading data from pkgdata related to the > recipes in question. > > The error is therefore correct and we need to do something about this. > > Cheers, > > Richard > >
diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 706c2ae9388..afcda2980b3 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -266,7 +266,7 @@ def sstate_install(ss, d): overlap_allowed = (d.getVar("SSTATE_ALLOW_OVERLAP_FILES") or "").split() match = [] for f in sharedfiles: - if os.path.exists(f) and not os.path.islink(f): + if os.path.exists(f): f = os.path.normpath(f) realmatch = True for w in overlap_allowed: @@ -276,36 +276,18 @@ def sstate_install(ss, d): break if realmatch: match.append(f) - sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}")) + sstate_search_cmd = "grep -rlF '%s' %s --exclude=index-* | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}")) search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0] if search_output: match.append(" (matched in %s)" % search_output.decode('utf-8').rstrip()) else: match.append(" (not matched to any task)") if match: - bb.error("The recipe %s is trying to install files into a shared " \ - "area when those files already exist. Those files and their manifest " \ - "location are:\n %s\nPlease verify which recipe should provide the " \ - "above files.\n\nThe build has stopped, as continuing in this scenario WILL " \ - "break things - if not now, possibly in the future (we've seen builds fail " \ - "several months later). If the system knew how to recover from this " \ - "automatically it would, however there are several different scenarios " \ - "which can result in this and we don't know which one this is. It may be " \ - "you have switched providers of something like virtual/kernel (e.g. from " \ - "linux-yocto to linux-yocto-dev), in that case you need to execute the " \ - "clean task for both recipes and it will resolve this error. It may be " \ - "you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning " \ - "those recipes should again resolve this error, however switching " \ - "DISTRO_FEATURES on an existing build directory is not supported - you " \ - "should really clean out tmp and rebuild (reusing sstate should be safe). " \ - "It could be the overlapping files detected are harmless in which case " \ - "adding them to SSTATE_ALLOW_OVERLAP_FILES may be the correct solution. It could " \ - "also be your build is including two different conflicting versions of " \ - "things (e.g. bluez 4 and bluez 5 and the correct solution for that would " \ - "be to resolve the conflict. If in doubt, please ask on the mailing list, " \ - "sharing the error and filelist above." % \ + bb.fatal("Recipe %s is trying to install files into a shared " \ + "area when those files already exist. The files and the manifests listing " \ + "them are:\n %s\n" + "Please adjust the recipes so only one recipe provides a given file. " % \ (d.getVar('PN'), "\n ".join(match))) - bb.fatal("If the above message is too much, the simpler version is you're advised to wipe out tmp and rebuild (reusing sstate is fine). That will likely fix things in most (but not all) cases.") if ss['fixmedir'] and os.path.exists(ss['fixmedir'] + "/fixmepath.cmd"): sharedfiles.append(ss['fixmedir'] + "/fixmepath.cmd")
When originally implemented, overlapping symlinks in DEPLOY_DIR were common. That is no longer the case and these overlapping links are causing bugs in other areas (e.g. bug 14123). Therefore start showing errors for overlapping symlinks in shared areas. Whilst here, fix a broken file reference in the grep command to match current file layouts and update the message shown to users to match current times. Most of the message content is obsolete now due to other advances and changes in the way the staging code now works. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- meta/classes-global/sstate.bbclass | 30 ++++++------------------------ 1 file changed, 6 insertions(+), 24 deletions(-)