| Message ID | 20240201025621.1632547-2-maanyagoenka@linux.microsoft.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | usrmerge: add the distro feature to nativesdk builds when added to target and enable symlink creation for both | expand |
Hello, This causes failures on the autobuilders: https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/8540/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/5665/steps/12/logs/stdio On 01/02/2024 02:56:22+0000, Maanya Goenka wrote: > From: Maanya Goenka <maanyagoenka@microsoft.com> > > Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency), > since we have come across cases where the distro feature would be required by both contexts. > As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage, > these need to be created for both the target and native sysroots within the SDK. > > Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com> > Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com> > --- > meta/classes-recipe/image.bbclass | 2 ++ > meta/conf/bitbake.conf | 4 ++-- > 2 files changed, 4 insertions(+), 2 deletions(-) > > diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass > index 7231fad940..df59133f77 100644 > --- a/meta/classes-recipe/image.bbclass > +++ b/meta/classes-recipe/image.bbclass > @@ -653,8 +653,10 @@ create_merged_usr_symlinks_rootfs() { > create_merged_usr_symlinks ${IMAGE_ROOTFS} > } > > +# Create the usrmerge symlink for the host in addition to the target build > create_merged_usr_symlinks_sdk() { > create_merged_usr_symlinks ${SDK_OUTPUT}${SDKTARGETSYSROOT} > + create_merged_usr_symlinks ${SDK_OUTPUT}${SDKPATHNATIVE} > } > > ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks_rootfs', '',d)}" > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 6f180d18b0..ff9f7ab407 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -920,8 +920,8 @@ DISTRO_FEATURES_NATIVESDK ?= "x11" > > # Normally target distro features will not be applied to native builds: > # Native distro features on this list will use the target feature value > -DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl wayland" > -DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl wayland" > +DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl usrmerge wayland" > +DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl usrmerge wayland" > > DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit gobject-introspection-data ldconfig" > MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode" > -- > 2.25.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#194548): https://lists.openembedded.org/g/openembedded-core/message/194548 > Mute This Topic: https://lists.openembedded.org/mt/104089708/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Thu, 2024-02-01 at 02:56 +0000, Maanya Goenka wrote: > From: Maanya Goenka <maanyagoenka@microsoft.com> > > Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency), > since we have come across cases where the distro feature would be required by both contexts. > As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage, > these need to be created for both the target and native sysroots within the SDK. > > Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com> > Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com> > --- > meta/classes-recipe/image.bbclass | 2 ++ > meta/conf/bitbake.conf | 4 ++-- > 2 files changed, 4 insertions(+), 2 deletions(-) This doesn't quite follow. "target", "nativesdk" and "native" are all different separate filesystems and not connected. Yes, you can use usrmerge on some (or all?) of them but you don't have to have them match. Forcing them to match is likely just working around an underlying issue which needs to be fixed. What were the real failures? For example, this would stop native or nativesdk sstate being reused between differently configured builds when it could be. Cheers, Richard
On 2/6/24 03:27, Richard Purdie wrote: > On Thu, 2024-02-01 at 02:56 +0000, Maanya Goenka wrote: >> From: Maanya Goenka<maanyagoenka@microsoft.com> >> >> Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency), >> since we have come across cases where the distro feature would be required by both contexts. >> As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage, >> these need to be created for both the target and native sysroots within the SDK. >> >> Signed-off-by: Maanya Goenka<maanyagoenka@microsoft.com> >> Signed-off-by : Alejandro Hernandez Samaniego<alhe@linux.microsoft.com> >> --- >> meta/classes-recipe/image.bbclass | 2 ++ >> meta/conf/bitbake.conf | 4 ++-- >> 2 files changed, 4 insertions(+), 2 deletions(-) > This doesn't quite follow. "target", "nativesdk" and "native" are all > different separate filesystems and not connected. Yes, you can use > usrmerge on some (or all?) of them but you don't have to have them > match. > > Forcing them to match is likely just working around an underlying issue > which needs to be fixed. > > What were the real failures? Hello Richard, Apologies for the late response. I do agree that target, native and nativesdk are all separate and that you could use usrmerge on some or all. I understand your point and I'm to blame for the suggestion to add usrmerge to the NATIVE case for consistency. The proposed patch is trying to fix an issue for a use case related to systemd. Newer versions of systemd do not function properly without usrmerge because upstream systemd is dropping support for non-usrmerge cases, which is typically fine because usrmerge works well for the TARGET. https://lists.freedesktop.org/archives/systemd-devel/2022-September/048352.html However, we are using some NATIVESDK systemd components that require usrmerge to work as well hence why we needed to add usrmerge to the NATIVESDK filter since systemd components where expecting usr/foo locations in the nativesdk filesystem. While I understand that using nativesdk systemd components may not be widely common, I do believe its a valid usecase. I'd like to clarify that I'm not aware of any native systemd usecase and that my suggestion to add usrmerge to the NATIVE filter was only for consistency, but not required (apologies for that again). I've reproduced the autobuilder failures locally and I see that taking out usrmerge from the NATIVE filter does get rid of the errors, if you agree with my statement that nativesdk systemd components are a valid usecase, we can send a v2 that adds usrmerge to the NATIVESDK filter only to fix such cases. If you prefer, we could also go one step above that and create a condition to add usrmerge to the NATIVESDK filter only if systemd is being used or something similar. Cheers, Alejandro > > For example, this would stop native or nativesdk sstate being reused > between differently configured builds when it could be. > > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#194983):https://lists.openembedded.org/g/openembedded-core/message/194983 > Mute This Topic:https://lists.openembedded.org/mt/104089708/4354175 > Group Owner:openembedded-core+owner@lists.openembedded.org > Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [alhe@linux.microsoft.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Alejandro, On Thu, 2024-02-15 at 12:10 -0700, Alejandro Enedino Hernandez Samaniego wrote: > On 2/6/24 03:27, Richard Purdie wrote: > > > > On Thu, 2024-02-01 at 02:56 +0000, Maanya Goenka wrote: > > > From: Maanya Goenka <maanyagoenka@microsoft.com> > > > > > > Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency), > > > since we have come across cases where the distro feature would be required by both contexts. > > > As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage, > > > these need to be created for both the target and native sysroots within the SDK. > > > > > > Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com> > > > Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com> > > > --- > > > meta/classes-recipe/image.bbclass | 2 ++ > > > meta/conf/bitbake.conf | 4 ++-- > > > 2 files changed, 4 insertions(+), 2 deletions(-) > > > > This doesn't quite follow. "target", "nativesdk" and "native" are all > > different separate filesystems and not connected. Yes, you can use > > usrmerge on some (or all?) of them but you don't have to have them > > match. > > > > Forcing them to match is likely just working around an underlying issue > > which needs to be fixed. > > > > What were the real failures? > > > Apologies for the late response. > > I do agree that target, native and nativesdk are all separate and that you could use usrmerge on some > or all. I understand your point and I'm to blame for the suggestion to add usrmerge to the NATIVE case > for consistency. > > The proposed patch is trying to fix an issue for a use case related to systemd. > > Newer versions of systemd do not function properly without usrmerge because upstream systemd is dropping > support for non-usrmerge cases, which is typically fine because usrmerge works well for the TARGET. > > https://lists.freedesktop.org/archives/systemd-devel/2022-September/048352.html > > However, we are using some NATIVESDK systemd components that require usrmerge to work as well hence why > we needed to add usrmerge to the NATIVESDK filter since systemd components where expecting usr/foo locations > in the nativesdk filesystem. > > While I understand that using nativesdk systemd components may not be widely common, I do believe its a > valid usecase. > > I'd like to clarify that I'm not aware of any native systemd usecase and that my suggestion to add usrmerge > to the NATIVE filter was only for consistency, but not required (apologies for that again). > > I've reproduced the autobuilder failures locally and I see that taking out usrmerge from the NATIVE filter does > get rid of the errors, if you agree with my statement that nativesdk systemd components are a valid usecase, > we can send a v2 that adds usrmerge to the NATIVESDK filter only to fix such cases. > > If you prefer, we could also go one step above that and create a condition to add usrmerge to the NATIVESDK filter > only if systemd is being used or something similar. I think there is a piece of this that is still missing. Changing usrmerge in nativesdk fundamentally alters the layout of the SDK. The layout of the SDK is meant to be independent of the target (think multilibs and so on as well). What that means is that the tools are meant to work with any target system and not need to "match" their layout. This effectively means that systemd's tools aren't working properly for environments which have cross layouts. The SDK layout is already drastically different from that of the target system in that everything is installed into a deep prefix path. How long will it be before systemd drops support for that too? I don't really buy into this "systemd should dictate the layout of everything" situation we're ending up in. It does mean any given SDK can only support systemd or not systemd but not both so it breaks multiconfig too. What is systemd's position on supporting cross environments? That is effectively the real question here and it sounds like they don't support that? We already have big issues with this in the "native" tooling side of things since that cross support doesn't exist and we had to do our own emulation. So whilst I understand your usecase, I do believe this is a sign of a much deeper problem unfortunately, namely that systemd doesn't support cross environments or usage. We could probably hack around it like this for now but I'm far from sure it won't just break again in the future in worse ways. Cheers, Richard
diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass index 7231fad940..df59133f77 100644 --- a/meta/classes-recipe/image.bbclass +++ b/meta/classes-recipe/image.bbclass @@ -653,8 +653,10 @@ create_merged_usr_symlinks_rootfs() { create_merged_usr_symlinks ${IMAGE_ROOTFS} } +# Create the usrmerge symlink for the host in addition to the target build create_merged_usr_symlinks_sdk() { create_merged_usr_symlinks ${SDK_OUTPUT}${SDKTARGETSYSROOT} + create_merged_usr_symlinks ${SDK_OUTPUT}${SDKPATHNATIVE} } ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks_rootfs', '',d)}" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 6f180d18b0..ff9f7ab407 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -920,8 +920,8 @@ DISTRO_FEATURES_NATIVESDK ?= "x11" # Normally target distro features will not be applied to native builds: # Native distro features on this list will use the target feature value -DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl wayland" -DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl wayland" +DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl usrmerge wayland" +DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl usrmerge wayland" DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit gobject-introspection-data ldconfig" MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"