| Message ID | 20220823085636.30868-2-peter@berginkonsult.se |
|---|---|
| State | Accepted, archived |
| Commit | 85a676da586f9b1085e62ef1325c9a58168390ae |
| Headers | show |
| Series | [1/2] packagegroup-rust-cross-canadian: add native compiler environment | expand |
On Tue, 2022-08-23 at 10:56 +0200, Peter Bergin wrote: > The test for rust in the SDK is extended with the simplest > possible build script. This will make use of the host toolchain > for building build.rs before building the rust package for target. > > Signed-off-by: Peter Bergin <peter@berginkonsult.se> > --- > meta/lib/oeqa/sdk/files/rust/hello/build.rs | 3 +++ > 1 file changed, 3 insertions(+) > create mode 100644 meta/lib/oeqa/sdk/files/rust/hello/build.rs > > diff --git a/meta/lib/oeqa/sdk/files/rust/hello/build.rs b/meta/lib/oeqa/sdk/files/rust/hello/build.rs > new file mode 100644 > index 00000000000..b1a533d5dfa > --- /dev/null > +++ b/meta/lib/oeqa/sdk/files/rust/hello/build.rs > @@ -0,0 +1,3 @@ > +/* This is the simplest build script just to invoke host compiler > + in the build process. */ > +fn main() {} This seemed to break everywhere :( https://autobuilder.yoctoproject.org/typhoon/#/builders/48/builds/5773/steps/13/logs/stdio and many others. Cheers, Richard
Hi Richard, On 2022-08-24 10:52, Richard Purdie wrote: > On Tue, 2022-08-23 at 10:56 +0200, Peter Bergin wrote: >> The test for rust in the SDK is extended with the simplest >> possible build script. This will make use of the host toolchain >> for building build.rs before building the rust package for target. >> >> Signed-off-by: Peter Bergin <peter@berginkonsult.se> >> --- >> meta/lib/oeqa/sdk/files/rust/hello/build.rs | 3 +++ >> 1 file changed, 3 insertions(+) >> create mode 100644 meta/lib/oeqa/sdk/files/rust/hello/build.rs >> >> diff --git a/meta/lib/oeqa/sdk/files/rust/hello/build.rs b/meta/lib/oeqa/sdk/files/rust/hello/build.rs >> new file mode 100644 >> index 00000000000..b1a533d5dfa >> --- /dev/null >> +++ b/meta/lib/oeqa/sdk/files/rust/hello/build.rs >> @@ -0,0 +1,3 @@ >> +/* This is the simplest build script just to invoke host compiler >> + in the build process. */ >> +fn main() {} > This seemed to break everywhere :( Not good. :-| > > https://autobuilder.yoctoproject.org/typhoon/#/builders/48/builds/5773/steps/13/logs/stdio > > and many others. I've tried locally to reproduce something. I've built and tested genericx86 and qemuarm64 now on core-image-sato sdk that I saw was the target for the autobuilder. Both tests passes. The failure I see in the autobuilder logs is that the build script can not be executed. On my machine I have that file and it can clearly be executed: $ find tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/ -name build-script-build tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build $ tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build $ file tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2, BuildID[sha1]=c5d5e70657f8342addf5343d0206c77d9d767fd8, for GNU/Linux 3.2.0, with debug_info, not stripped Found one interesting link here https://github.com/rust-lang/cargo/issues/3553. Unfortunately without answer. But also checked the interpreter in my build which looks correct? $ readelf -a tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build | grep interpreter [Requesting program interpreter: /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] So there are some differences between my machine and the autobuilder setup that I can't get. I would need help here as I'm not that familiar with the autobuilder setup. Can it still be some host dependency? I'm running on Ubuntu 22.04. Is it possible to check in a autobuilder setup if the file 'build-script-build' is present and possible to execute? Best regards, /Peter
Hi Peter, On Thu, 2022-08-25 at 09:17 +0200, Peter Bergin wrote: > On 2022-08-24 10:52, Richard Purdie wrote: > > On Tue, 2022-08-23 at 10:56 +0200, Peter Bergin wrote: > > > The test for rust in the SDK is extended with the simplest > > > possible build script. This will make use of the host toolchain > > > for building build.rs before building the rust package for target. > > > > > > Signed-off-by: Peter Bergin <peter@berginkonsult.se> > > > --- > > > meta/lib/oeqa/sdk/files/rust/hello/build.rs | 3 +++ > > > 1 file changed, 3 insertions(+) > > > create mode 100644 meta/lib/oeqa/sdk/files/rust/hello/build.rs > > > > > > diff --git a/meta/lib/oeqa/sdk/files/rust/hello/build.rs b/meta/lib/oeqa/sdk/files/rust/hello/build.rs > > > new file mode 100644 > > > index 00000000000..b1a533d5dfa > > > --- /dev/null > > > +++ b/meta/lib/oeqa/sdk/files/rust/hello/build.rs > > > @@ -0,0 +1,3 @@ > > > +/* This is the simplest build script just to invoke host compiler > > > + in the build process. */ > > > +fn main() {} > > This seemed to break everywhere :( > Not good. :-| > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/48/builds/5773/steps/13/logs/stdio > > > > and many others. > > I've tried locally to reproduce something. I've built and tested > genericx86 and qemuarm64 now on core-image-sato sdk that I saw was the > target for the autobuilder. Both tests passes. The failure I see in the > autobuilder logs is that the build script can not be executed. On my > machine I have that file and it can clearly be executed: > > $ find > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/ > -name build-script-build > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > $ > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > $ file > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build: > ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically > linked, interpreter > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2, > BuildID[sha1]=c5d5e70657f8342addf5343d0206c77d9d767fd8, for GNU/Linux > 3.2.0, with debug_info, not stripped > > Found one interesting link here > https://github.com/rust-lang/cargo/issues/3553. Unfortunately without > answer. But also checked the interpreter in my build which looks correct? > > $ readelf -a > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > grep interpreter > [Requesting program interpreter: > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] > > > So there are some differences between my machine and the autobuilder > setup that I can't get. I would need help here as I'm not that familiar > with the autobuilder setup. Can it still be some host dependency? I'm > running on Ubuntu 22.04. Is it possible to check in a autobuilder setup > if the file 'build-script-build' is present and possible to execute? After staring at this for an hour, I think the pattern is that is it failing on builds with: SDKMACHINE = "i686" which probably means the linker isn't linking against the libc and loader in the SDK properly. (i686 SDK binaries should run on x86_64 hosts since we provide our own loader and libc) Cheers, Richard
Hi Richard, On 2022-08-25 10:21, Richard Purdie wrote: > <snip> >> I've tried locally to reproduce something. I've built and tested >> genericx86 and qemuarm64 now on core-image-sato sdk that I saw was the >> target for the autobuilder. Both tests passes. The failure I see in the >> autobuilder logs is that the build script can not be executed. On my >> machine I have that file and it can clearly be executed: >> >> $ find >> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/ >> -name build-script-build >> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >> >> $ >> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >> >> $ file >> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >> >> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build: >> ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically >> linked, interpreter >> /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2, >> BuildID[sha1]=c5d5e70657f8342addf5343d0206c77d9d767fd8, for GNU/Linux >> 3.2.0, with debug_info, not stripped >> >> Found one interesting link here >> https://github.com/rust-lang/cargo/issues/3553. Unfortunately without >> answer. But also checked the interpreter in my build which looks correct? >> >> $ readelf -a >> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >>> grep interpreter >> [Requesting program interpreter: >> /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] >> >> >> So there are some differences between my machine and the autobuilder >> setup that I can't get. I would need help here as I'm not that familiar >> with the autobuilder setup. Can it still be some host dependency? I'm >> running on Ubuntu 22.04. Is it possible to check in a autobuilder setup >> if the file 'build-script-build' is present and possible to execute? > After staring at this for an hour, I think the pattern is that is it > failing on builds with: > > SDKMACHINE = "i686" > > which probably means the linker isn't linking against the libc and > loader in the SDK properly. > > (i686 SDK binaries should run on x86_64 hosts since we provide our own > loader and libc) Thanks a lot for the reproducer. You are correct, with SDKMACHINE i686 I was able to reproduce. I want to share some weird stuff and see if someone can get a clue for a solution. Did same analyze on the build-script-build file as I did on a working one: $ /storage/yocto/esp5-platform/build/qemuarm64/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-2c1c69e3bd7e958d/build-script-build -bash: /storage/yocto/esp5-platform/build/qemuarm64/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-2c1c69e3bd7e958d/build-script-build: No such file or directory $ file /storage/yocto/esp5-platform/build/qemuarm64/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-2c1c69e3bd7e958d/build-script-build /storage/yocto/esp5-platform/build/qemuarm64/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-2c1c69e3bd7e958d/build-script-build: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2, BuildID[sha1]=4c32d67e8597f30ce774e2bb32ea5170b9878957, for GNU/Linux 3.2.0, with debug_info, not stripped $ ldd /storage/yocto/esp5-platform/build/qemuarm64/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-2c1c69e3bd7e958d/build-script-build linux-gate.so.1 (0xf7f62000) libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7ecd000) libc.so.6 => /lib32/libc.so.6 (0xf7c9c000) /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xf7f64000) $ readelf -a /storage/yocto/esp5-platform/build/qemuarm64/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-2c1c69e3bd7e958d/build-script-build | grep interpreter [Requesting program interpreter: /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2] The interpreter is wrong and pointing to a non existing file. It has '/usr/local/oe-sdk-hardcoded-buildpath' in its path which seems really bad and is equal to SDKPATH variable. Best regards, /Peter
On Thu, 2022-08-25 at 14:03 +0200, Peter Bergin wrote: > Hi Richard, > > On 2022-08-25 10:21, Richard Purdie wrote: > > <snip> > > > I've tried locally to reproduce something. I've built and tested > > > genericx86 and qemuarm64 now on core-image-sato sdk that I saw was the > > > target for the autobuilder. Both tests passes. The failure I see in the > > > autobuilder logs is that the build script can not be executed. On my > > > machine I have that file and it can clearly be executed: > > > > > > $ find > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/ > > > -name build-script-build > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > $ > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > $ file > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build: > > > ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically > > > linked, interpreter > > > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2, > > > BuildID[sha1]=c5d5e70657f8342addf5343d0206c77d9d767fd8, for GNU/Linux > > > 3.2.0, with debug_info, not stripped > > > > > > Found one interesting link here > > > https://github.com/rust-lang/cargo/issues/3553. Unfortunately without > > > answer. But also checked the interpreter in my build which looks correct? > > > > > > $ readelf -a > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > grep interpreter > > > [Requesting program interpreter: > > > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] > > > > > > > > > So there are some differences between my machine and the autobuilder > > > setup that I can't get. I would need help here as I'm not that familiar > > > with the autobuilder setup. Can it still be some host dependency? I'm > > > running on Ubuntu 22.04. Is it possible to check in a autobuilder setup > > > if the file 'build-script-build' is present and possible to execute? > > After staring at this for an hour, I think the pattern is that is it > > failing on builds with: > > > > SDKMACHINE = "i686" > > > > which probably means the linker isn't linking against the libc and > > loader in the SDK properly. > > > > (i686 SDK binaries should run on x86_64 hosts since we provide our own > > loader and libc) > > Thanks a lot for the reproducer. You are correct, with SDKMACHINE i686 I > was able to reproduce. I want to share some weird stuff and see if > someone can get a clue for a solution. > > Did same analyze on the build-script-build file as I did on a working one: As I suspected, the dynamic loader on binaries built using the toolchain in the SDK isn't working correctly. "/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk- linux/lib/ld-linux.so.2" is the value used at build time. This is supposed to be relocated into the SDK's install location at runtime. The question is therefore where is rust getting this value from? You can specify it to the linker at compile time along the lines of: -Wl,--dynamic-linker=/path/to/ld-linux.so.2 or it can be the default that comes from toolchain. It is important it points at the one in the SDK, not on the host system. The bit I don't understand is whether: a) the compiler in the SDK is generating the wrong loader (in which case buildtools-extended-tarball would break which it doesn't) b) rust is hardcoding some dynamic-linker option somewhere it shouldn't c) whether the toolchain is relocating paths correctly or some path isn't relocating and is being used here. I'd suggest first checking that a binary generated by nativesdk-gcc in the SDK has the right loader. If that is the case, see what compiler options are being used to create the build tool and go from there. FWIW you can see/change the interpreter with patchelf. Cheers, Richard
Hi Richard, (also adding Khem Raj to CC as maintainer for gcc in oe-core) On 2022-08-25 17:25, Richard Purdie wrote: > On Thu, 2022-08-25 at 14:03 +0200, Peter Bergin wrote: >> Hi Richard, >> >> On 2022-08-25 10:21, Richard Purdie wrote: >>> <snip> >>>> I've tried locally to reproduce something. I've built and tested >>>> genericx86 and qemuarm64 now on core-image-sato sdk that I saw was the >>>> target for the autobuilder. Both tests passes. The failure I see in the >>>> autobuilder logs is that the build script can not be executed. On my >>>> machine I have that file and it can clearly be executed: >>>> >>>> $ find >>>> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/ >>>> -name build-script-build >>>> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >>>> >>>> $ >>>> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >>>> >>>> $ file >>>> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >>>> >>>> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build: >>>> ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically >>>> linked, interpreter >>>> /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2, >>>> BuildID[sha1]=c5d5e70657f8342addf5343d0206c77d9d767fd8, for GNU/Linux >>>> 3.2.0, with debug_info, not stripped >>>> >>>> Found one interesting link here >>>> https://github.com/rust-lang/cargo/issues/3553. Unfortunately without >>>> answer. But also checked the interpreter in my build which looks correct? >>>> >>>> $ readelf -a >>>> tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build >>>>> grep interpreter >>>> [Requesting program interpreter: >>>> /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] >>>> >>>> >>>> So there are some differences between my machine and the autobuilder >>>> setup that I can't get. I would need help here as I'm not that familiar >>>> with the autobuilder setup. Can it still be some host dependency? I'm >>>> running on Ubuntu 22.04. Is it possible to check in a autobuilder setup >>>> if the file 'build-script-build' is present and possible to execute? >>> After staring at this for an hour, I think the pattern is that is it >>> failing on builds with: >>> >>> SDKMACHINE = "i686" >>> >>> which probably means the linker isn't linking against the libc and >>> loader in the SDK properly. >>> >>> (i686 SDK binaries should run on x86_64 hosts since we provide our own >>> loader and libc) >> Thanks a lot for the reproducer. You are correct, with SDKMACHINE i686 I >> was able to reproduce. I want to share some weird stuff and see if >> someone can get a clue for a solution. >> >> Did same analyze on the build-script-build file as I did on a working one: > As I suspected, the dynamic loader on binaries built using the > toolchain in the SDK isn't working correctly. > > "/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk- > linux/lib/ld-linux.so.2" > > is the value used at build time. This is supposed to be relocated into > the SDK's install location at runtime. > > The question is therefore where is rust getting this value from? You > can specify it to the linker at compile time along the lines of: > > -Wl,--dynamic-linker=/path/to/ld-linux.so.2 > > or it can be the default that comes from toolchain. It is important it > points at the one in the SDK, not on the host system. > > The bit I don't understand is whether: > > a) the compiler in the SDK is generating the wrong loader (in which > case buildtools-extended-tarball would break which it doesn't) > b) rust is hardcoding some dynamic-linker option somewhere it shouldn't > c) whether the toolchain is relocating paths correctly or some path > isn't relocating and is being used here. > > I'd suggest first checking that a binary generated by nativesdk-gcc in > the SDK has the right loader. If that is the case, see what compiler > options are being used to create the build tool and go from there. > > FWIW you can see/change the interpreter with patchelf. > > Cheers, > > Richard I've tried to narrow this down further and I can say now that this error when building the native rust build script is not directly related to cargo/rust but to the native gcc compiler shipped in the SDK. I will next try to dig down in the gcc build process but before spending time there I would like to share my finding and see if someone have ideas. I have two SDKs one with SDKMACHINE="x86_64" and one with SDKMACHINE="i686". I have the hosts gcc compiler installed in them and use them to compile a minimal c code example with hello world. The one for "x86_64" works well and is using the program interpreter from the SDK. The executable compiled and linked with "i686" toolchain will have a program interpreter pointing to "/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2". I will paste in output from the two experiments here below: x86_64: $ <sdk-install>/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-pokysdk-linux-gcc main.c $ ./a.out Hello world! $ readelf -a a.out | grep interpreter [Requesting program interpreter: <sdk-install>/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] $ <sdk-install>/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-pokysdk-linux-gcc -dumpspecs | grep -A 1 *dynamic_linker i686: $ <sdk-install>/sysroots/i686-pokysdk-linux/usr/bin/i686-pokysdk-linux-gcc main.c $ ./a.out -bash: ./a.out: No such file or directory $ readelf -a a.out | grep interpreter [Requesting program interpreter: /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2] $ <sdk-install>/sysroots/i686-pokysdk-linux/usr/bin/i686-pokysdk-linux-gcc -dumpspecs | grep -A 1 *dynamic_linker *dynamic_linker: %{muclibc:%rld-uClibc.so.0;:%{mbionic:/system/bin/linker;:%{mmusl:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-musl-i386.so.1;:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2}}} Noted above is that the spec-file (-dumpspecs) for i686 have a sections with "*dynamic_linker" but that one is missing in x86_64. So my conclusion around the results above is that something seems missconfigured in the nativesdk-gcc build for i686. When cargo/rust is using the native compiler to build its build script it is set to use wrong program interpreter and when executed it fails just as my a.out above does. I hope this can give you some more ideas where to look next and ultimately how to solve this. Best regards, /Peter
On Mon, 2022-08-29 at 13:09 +0200, Peter Bergin wrote: > Hi Richard, > > (also adding Khem Raj to CC as maintainer for gcc in oe-core) > > On 2022-08-25 17:25, Richard Purdie wrote: > > On Thu, 2022-08-25 at 14:03 +0200, Peter Bergin wrote: > > > Hi Richard, > > > > > > On 2022-08-25 10:21, Richard Purdie wrote: > > > > <snip> > > > > > I've tried locally to reproduce something. I've built and tested > > > > > genericx86 and qemuarm64 now on core-image-sato sdk that I saw was the > > > > > target for the autobuilder. Both tests passes. The failure I see in the > > > > > autobuilder logs is that the build script can not be executed. On my > > > > > machine I have that file and it can clearly be executed: > > > > > > > > > > $ find > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/ > > > > > -name build-script-build > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > > > > > $ > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > > > > > $ file > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build: > > > > > ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically > > > > > linked, interpreter > > > > > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2, > > > > > BuildID[sha1]=c5d5e70657f8342addf5343d0206c77d9d767fd8, for GNU/Linux > > > > > 3.2.0, with debug_info, not stripped > > > > > > > > > > Found one interesting link here > > > > > https://github.com/rust-lang/cargo/issues/3553. Unfortunately without > > > > > answer. But also checked the interpreter in my build which looks correct? > > > > > > > > > > $ readelf -a > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > grep interpreter > > > > > [Requesting program interpreter: > > > > > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] > > > > > > > > > > > > > > > So there are some differences between my machine and the autobuilder > > > > > setup that I can't get. I would need help here as I'm not that familiar > > > > > with the autobuilder setup. Can it still be some host dependency? I'm > > > > > running on Ubuntu 22.04. Is it possible to check in a autobuilder setup > > > > > if the file 'build-script-build' is present and possible to execute? > > > > After staring at this for an hour, I think the pattern is that is it > > > > failing on builds with: > > > > > > > > SDKMACHINE = "i686" > > > > > > > > which probably means the linker isn't linking against the libc and > > > > loader in the SDK properly. > > > > > > > > (i686 SDK binaries should run on x86_64 hosts since we provide our own > > > > loader and libc) > > > Thanks a lot for the reproducer. You are correct, with SDKMACHINE i686 I > > > was able to reproduce. I want to share some weird stuff and see if > > > someone can get a clue for a solution. > > > > > > Did same analyze on the build-script-build file as I did on a working one: > > As I suspected, the dynamic loader on binaries built using the > > toolchain in the SDK isn't working correctly. > > > > "/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk- > > linux/lib/ld-linux.so.2" > > > > is the value used at build time. This is supposed to be relocated into > > the SDK's install location at runtime. > > > > The question is therefore where is rust getting this value from? You > > can specify it to the linker at compile time along the lines of: > > > > -Wl,--dynamic-linker=/path/to/ld-linux.so.2 > > > > or it can be the default that comes from toolchain. It is important it > > points at the one in the SDK, not on the host system. > > > > The bit I don't understand is whether: > > > > a) the compiler in the SDK is generating the wrong loader (in which > > case buildtools-extended-tarball would break which it doesn't) > > b) rust is hardcoding some dynamic-linker option somewhere it shouldn't > > c) whether the toolchain is relocating paths correctly or some path > > isn't relocating and is being used here. > > > > I'd suggest first checking that a binary generated by nativesdk-gcc in > > the SDK has the right loader. If that is the case, see what compiler > > options are being used to create the build tool and go from there. > > > > FWIW you can see/change the interpreter with patchelf. > > > > Cheers, > > > > Richard > > I've tried to narrow this down further and I can say now that this error > when building the native rust build script is not directly related to > cargo/rust but to the native gcc compiler shipped in the SDK. I will > next try to dig down in the gcc build process but before spending time > there I would like to share my finding and see if someone have ideas. > > I have two SDKs one with SDKMACHINE="x86_64" and one with > SDKMACHINE="i686". I have the hosts gcc compiler installed in them and > use them to compile a minimal c code example with hello world. The one > for "x86_64" works well and is using the program interpreter from the > SDK. The executable compiled and linked with "i686" toolchain will have > a program interpreter pointing to > "/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2". > I will paste in output from the two experiments here below: > > x86_64: > > $ > <sdk-install>/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-pokysdk-linux-gcc > main.c > $ ./a.out > Hello world! > $ readelf -a a.out | grep interpreter > [Requesting program interpreter: > <sdk-install>/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] > $ > <sdk-install>/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-pokysdk-linux-gcc > -dumpspecs | grep -A 1 *dynamic_linker > > i686: > > $ > <sdk-install>/sysroots/i686-pokysdk-linux/usr/bin/i686-pokysdk-linux-gcc > main.c > $ ./a.out > -bash: ./a.out: No such file or directory > $ readelf -a a.out | grep interpreter > [Requesting program interpreter: > /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2] > > $ > <sdk-install>/sysroots/i686-pokysdk-linux/usr/bin/i686-pokysdk-linux-gcc > -dumpspecs | grep -A 1 *dynamic_linker > *dynamic_linker: > %{muclibc:%rld-uClibc.so.0;:%{mbionic:/system/bin/linker;:%{mmusl:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-musl-i386.so.1;:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2}}} > > > Noted above is that the spec-file (-dumpspecs) for i686 have a sections > with "*dynamic_linker" but that one is missing in x86_64. > > So my conclusion around the results above is that something seems > missconfigured in the nativesdk-gcc build for i686. When cargo/rust is > using the native compiler to build its build script it is set to use > wrong program interpreter and when executed it fails just as my a.out > above does. I hope this can give you some more ideas where to look next > and ultimately how to solve this. > I couldn't remeber the details of how this worked and there has been a lot going on so I only just got around to experimenting. This might be really simple fix: diff --git a/meta/recipes-devtools/gcc/gcc-multilib-config.inc b/meta/recipes-devtools/gcc/gcc-multilib-config.inc index 26bfed9507b..2dbbc23c940 100644 --- a/meta/recipes-devtools/gcc/gcc-multilib-config.inc +++ b/meta/recipes-devtools/gcc/gcc-multilib-config.inc @@ -154,7 +154,7 @@ python gcc_multilib_setup() { gcc_header_config_files = { 'x86_64' : ['gcc/config/linux.h', 'gcc/config/i386/linux.h', 'gcc/config/i386/linux64.h'], 'i586' : ['gcc/config/linux.h', 'gcc/config/i386/linux.h', 'gcc/config/i386/linux64.h'], - 'i686' : ['gcc/config/linux.h', 'gcc/config/i386/linux64.h'], + 'i686' : ['gcc/config/linux.h', 'gcc/config/i386/linux.h', 'gcc/config/i386/linux64.h'], 'mips' : ['gcc/config/linux.h', 'gcc/config/mips/linux.h', 'gcc/config/mips/linux64.h'], 'mips64' : ['gcc/config/linux.h', 'gcc/config/mips/linux.h', 'gcc/config/mips/linux64.h'], 'powerpc' : ['gcc/config/linux.h', 'gcc/config/rs6000/linux64.h'], which changes the linux.h header to have a %r instead of a SYSTEMLIBS_DIR hardcoded definition. That will the relocate with the toolchain. I've not tested this yet but seems like the right thing to fix. Cheers, Richard
On Fri, 2022-09-02 at 14:33 +0100, Richard Purdie via lists.openembedded.org wrote: > On Mon, 2022-08-29 at 13:09 +0200, Peter Bergin wrote: > > Hi Richard, > > > > (also adding Khem Raj to CC as maintainer for gcc in oe-core) > > > > On 2022-08-25 17:25, Richard Purdie wrote: > > > On Thu, 2022-08-25 at 14:03 +0200, Peter Bergin wrote: > > > > Hi Richard, > > > > > > > > On 2022-08-25 10:21, Richard Purdie wrote: > > > > > <snip> > > > > > > I've tried locally to reproduce something. I've built and tested > > > > > > genericx86 and qemuarm64 now on core-image-sato sdk that I saw was the > > > > > > target for the autobuilder. Both tests passes. The failure I see in the > > > > > > autobuilder logs is that the build script can not be executed. On my > > > > > > machine I have that file and it can clearly be executed: > > > > > > > > > > > > $ find > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/ > > > > > > -name build-script-build > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > > > > > > > $ > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > > > > > > > $ file > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build: > > > > > > ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically > > > > > > linked, interpreter > > > > > > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2, > > > > > > BuildID[sha1]=c5d5e70657f8342addf5343d0206c77d9d767fd8, for GNU/Linux > > > > > > 3.2.0, with debug_info, not stripped > > > > > > > > > > > > Found one interesting link here > > > > > > https://github.com/rust-lang/cargo/issues/3553. Unfortunately without > > > > > > answer. But also checked the interpreter in my build which looks correct? > > > > > > > > > > > > $ readelf -a > > > > > > tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/hello/target/debug/build/hello-4dbf26d86b93a892/build-script-build > > > > > > > grep interpreter > > > > > > [Requesting program interpreter: > > > > > > /storage/yocto/esp5-platform/build/genericx86/tmp/work/genericx86-poky-linux/core-image-sato/1.0-r0/testimage-sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] > > > > > > > > > > > > > > > > > > So there are some differences between my machine and the autobuilder > > > > > > setup that I can't get. I would need help here as I'm not that familiar > > > > > > with the autobuilder setup. Can it still be some host dependency? I'm > > > > > > running on Ubuntu 22.04. Is it possible to check in a autobuilder setup > > > > > > if the file 'build-script-build' is present and possible to execute? > > > > > After staring at this for an hour, I think the pattern is that is it > > > > > failing on builds with: > > > > > > > > > > SDKMACHINE = "i686" > > > > > > > > > > which probably means the linker isn't linking against the libc and > > > > > loader in the SDK properly. > > > > > > > > > > (i686 SDK binaries should run on x86_64 hosts since we provide our own > > > > > loader and libc) > > > > Thanks a lot for the reproducer. You are correct, with SDKMACHINE i686 I > > > > was able to reproduce. I want to share some weird stuff and see if > > > > someone can get a clue for a solution. > > > > > > > > Did same analyze on the build-script-build file as I did on a working one: > > > As I suspected, the dynamic loader on binaries built using the > > > toolchain in the SDK isn't working correctly. > > > > > > "/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk- > > > linux/lib/ld-linux.so.2" > > > > > > is the value used at build time. This is supposed to be relocated into > > > the SDK's install location at runtime. > > > > > > The question is therefore where is rust getting this value from? You > > > can specify it to the linker at compile time along the lines of: > > > > > > -Wl,--dynamic-linker=/path/to/ld-linux.so.2 > > > > > > or it can be the default that comes from toolchain. It is important it > > > points at the one in the SDK, not on the host system. > > > > > > The bit I don't understand is whether: > > > > > > a) the compiler in the SDK is generating the wrong loader (in which > > > case buildtools-extended-tarball would break which it doesn't) > > > b) rust is hardcoding some dynamic-linker option somewhere it shouldn't > > > c) whether the toolchain is relocating paths correctly or some path > > > isn't relocating and is being used here. > > > > > > I'd suggest first checking that a binary generated by nativesdk-gcc in > > > the SDK has the right loader. If that is the case, see what compiler > > > options are being used to create the build tool and go from there. > > > > > > FWIW you can see/change the interpreter with patchelf. > > > > > > Cheers, > > > > > > Richard > > > > I've tried to narrow this down further and I can say now that this error > > when building the native rust build script is not directly related to > > cargo/rust but to the native gcc compiler shipped in the SDK. I will > > next try to dig down in the gcc build process but before spending time > > there I would like to share my finding and see if someone have ideas. > > > > I have two SDKs one with SDKMACHINE="x86_64" and one with > > SDKMACHINE="i686". I have the hosts gcc compiler installed in them and > > use them to compile a minimal c code example with hello world. The one > > for "x86_64" works well and is using the program interpreter from the > > SDK. The executable compiled and linked with "i686" toolchain will have > > a program interpreter pointing to > > "/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2". > > I will paste in output from the two experiments here below: > > > > x86_64: > > > > $ > > <sdk-install>/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-pokysdk-linux-gcc > > main.c > > $ ./a.out > > Hello world! > > $ readelf -a a.out | grep interpreter > > [Requesting program interpreter: > > <sdk-install>/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2] > > $ > > <sdk-install>/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-pokysdk-linux-gcc > > -dumpspecs | grep -A 1 *dynamic_linker > > > > i686: > > > > $ > > <sdk-install>/sysroots/i686-pokysdk-linux/usr/bin/i686-pokysdk-linux-gcc > > main.c > > $ ./a.out > > -bash: ./a.out: No such file or directory > > $ readelf -a a.out | grep interpreter > > [Requesting program interpreter: > > /usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2] > > > > $ > > <sdk-install>/sysroots/i686-pokysdk-linux/usr/bin/i686-pokysdk-linux-gcc > > -dumpspecs | grep -A 1 *dynamic_linker > > *dynamic_linker: > > %{muclibc:%rld-uClibc.so.0;:%{mbionic:/system/bin/linker;:%{mmusl:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-musl-i386.so.1;:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/lib/ld-linux.so.2}}} > > > > > > Noted above is that the spec-file (-dumpspecs) for i686 have a sections > > with "*dynamic_linker" but that one is missing in x86_64. > > > > So my conclusion around the results above is that something seems > > missconfigured in the nativesdk-gcc build for i686. When cargo/rust is > > using the native compiler to build its build script it is set to use > > wrong program interpreter and when executed it fails just as my a.out > > above does. I hope this can give you some more ideas where to look next > > and ultimately how to solve this. > > > > I couldn't remeber the details of how this worked and there has been a > lot going on so I only just got around to experimenting. This might be > really simple fix: > > diff --git a/meta/recipes-devtools/gcc/gcc-multilib-config.inc b/meta/recipes-devtools/gcc/gcc-multilib-config.inc > index 26bfed9507b..2dbbc23c940 100644 > --- a/meta/recipes-devtools/gcc/gcc-multilib-config.inc > +++ b/meta/recipes-devtools/gcc/gcc-multilib-config.inc > @@ -154,7 +154,7 @@ python gcc_multilib_setup() { > gcc_header_config_files = { > 'x86_64' : ['gcc/config/linux.h', 'gcc/config/i386/linux.h', 'gcc/config/i386/linux64.h'], > 'i586' : ['gcc/config/linux.h', 'gcc/config/i386/linux.h', 'gcc/config/i386/linux64.h'], > - 'i686' : ['gcc/config/linux.h', 'gcc/config/i386/linux64.h'], > + 'i686' : ['gcc/config/linux.h', 'gcc/config/i386/linux.h', 'gcc/config/i386/linux64.h'], > 'mips' : ['gcc/config/linux.h', 'gcc/config/mips/linux.h', 'gcc/config/mips/linux64.h'], > 'mips64' : ['gcc/config/linux.h', 'gcc/config/mips/linux.h', 'gcc/config/mips/linux64.h'], > 'powerpc' : ['gcc/config/linux.h', 'gcc/config/rs6000/linux64.h'], > > > which changes the linux.h header to have a %r instead of a > SYSTEMLIBS_DIR hardcoded definition. That will the relocate with the > toolchain. > > I've not tested this yet but seems like the right thing to fix. I was able to confirm that fixed a simple test case for nativesdk-gcc on i686 so I'll queue it up to reset the rust changes. Cheers, Richard
<snip> > I was able to confirm that fixed a simple test case for nativesdk-gcc > on i686 so I'll queue it up to reset the rust changes. > > Cheers, > > Richard Thank you Richard! Great that you could fix this and we could get the rust tests in. /Peter
diff --git a/meta/lib/oeqa/sdk/files/rust/hello/build.rs b/meta/lib/oeqa/sdk/files/rust/hello/build.rs new file mode 100644 index 00000000000..b1a533d5dfa --- /dev/null +++ b/meta/lib/oeqa/sdk/files/rust/hello/build.rs @@ -0,0 +1,3 @@ +/* This is the simplest build script just to invoke host compiler + in the build process. */ +fn main() {}
The test for rust in the SDK is extended with the simplest possible build script. This will make use of the host toolchain for building build.rs before building the rust package for target. Signed-off-by: Peter Bergin <peter@berginkonsult.se> --- meta/lib/oeqa/sdk/files/rust/hello/build.rs | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 meta/lib/oeqa/sdk/files/rust/hello/build.rs