| Message ID | 20231011040550.1392831-1-sundeep.kokkonda@windriver.com |
|---|---|
| State | Accepted, archived |
| Commit | 6ae62259afbbe861ed74211dab18a27b8c8d8b7a |
| Headers | show |
| Series | rust: reproducibility issue fix | expand |
Excellent, thanks! Alex On Wed, 11 Oct 2023 at 06:06, Sundeep KOKKONDA via lists.openembedded.org <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote: > > [Yocto#14875] > The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary. > However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on. > So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'. > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> > --- > meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- > meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + > 2 files changed, 1 insertion(+), 2 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py > index 84c6c3a05f..9b4a0887dd 100644 > --- a/meta/lib/oeqa/selftest/cases/reproducible.py > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py > @@ -16,8 +16,6 @@ import os > import datetime > > exclude_packages = [ > - 'rust', > - 'rust-dbg' > ] > > def is_excluded(package): > diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb b/meta/recipes-devtools/rust/rust_1.70.0.bb > index 3b9c05a19f..16d433910f 100644 > --- a/meta/recipes-devtools/rust/rust_1.70.0.bb > +++ b/meta/recipes-devtools/rust/rust_1.70.0.bb > @@ -130,6 +130,7 @@ python do_configure() { > # [rust] > config.add_section("rust") > config.set("rust", "rpath", e(True)) > + config.set("rust", "remap-debuginfo", e(True)) > config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}"))) > > # Whether or not to optimize the compiler and standard library > -- > 2.39.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#188940): https://lists.openembedded.org/g/openembedded-core/message/188940 > Mute This Topic: https://lists.openembedded.org/mt/101890610/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Tue, 2023-10-10 at 21:05 -0700, Sundeep KOKKONDA via lists.openembedded.org wrote: > [Yocto#14875] > The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary. > However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on. > So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'. > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> > --- > meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- > meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + > 2 files changed, 1 insertion(+), 2 deletions(-) Thanks for that! I know this has taken a lot of work to get to the bottom of. It is great to have OE-Core at 100% reproducible again! :) Cheers, Richard
On 11 Oct 2023, at 05:05, Sundeep KOKKONDA via lists.openembedded.org <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote: > > [Yocto#14875] > The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary. > However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on. > So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix’. Love it when literally months of work results in a one-liner… So glad you managed to figure this out! Ross
On Wed, Oct 11, 2023 at 5:06 AM Sundeep KOKKONDA via lists.openembedded.org <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote: > > [Yocto#14875] > The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary. > However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on. > So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'. > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> > --- > meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- > meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + > 2 files changed, 1 insertion(+), 2 deletions(-) > Blimey, well done!
very great news to read ! On Wed, 11 Oct 2023 at 13:03, Alex Kiernan <alex.kiernan@gmail.com> wrote: > On Wed, Oct 11, 2023 at 5:06 AM Sundeep KOKKONDA via > lists.openembedded.org > <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote: > > > > [Yocto#14875] > > The '--remap-path-prefix' option removes all references to build > directory structure in the debug information within the compiled output for > Cargo dependencies and the project's binary. > > However, some references to build directories remains in the final > binary in .rustc section in the form of compressed metadata and this makes > the build output dependent on the folder structure of the computer it's > compiled on. > > So, for reproducible builds, use the configuration option > 'remap-debuginfo = true' along with the '--remap-path-prefix'. > > > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> > > --- > > meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- > > meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + > > 2 files changed, 1 insertion(+), 2 deletions(-) > > > > Blimey, well done! > > -- > Alex Kiernan > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#188954): > https://lists.openembedded.org/g/openembedded-core/message/188954 > Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > frederic.martinsons@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Wed, 11 Oct 2023 at 14:27, Frederic Martinsons via lists.openembedded.org <frederic.martinsons=gmail.com@lists.openembedded.org> wrote: > very great news to read ! > > > On Wed, 11 Oct 2023 at 13:03, Alex Kiernan <alex.kiernan@gmail.com> wrote: > >> On Wed, Oct 11, 2023 at 5:06 AM Sundeep KOKKONDA via >> lists.openembedded.org >> <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote: >> > >> > [Yocto#14875] >> > The '--remap-path-prefix' option removes all references to build >> directory structure in the debug information within the compiled output for >> Cargo dependencies and the project's binary. >> > However, some references to build directories remains in the final >> binary in .rustc section in the form of compressed metadata and this makes >> the build output dependent on the folder structure of the computer it's >> compiled on. >> > So, for reproducible builds, use the configuration option >> 'remap-debuginfo = true' along with the '--remap-path-prefix'. >> > >> > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> >> > --- >> > meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- >> > meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + >> > 2 files changed, 1 insertion(+), 2 deletions(-) >> > >> >> Blimey, well done! >> >> -- >> Alex Kiernan >> >> >> We may want to remove the EXCLUDE_FROM_WORLD in zvariant recipe <https://git.openembedded.org/openembedded-core/tree/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb> since it was introduced for reproducibility issue (#15090 which had been marked as a duplicate of the main rust issue #14875). What do you think sundeep ? >> > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#188956): > https://lists.openembedded.org/g/openembedded-core/message/188956 > Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > frederic.martinsons@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Wed, 2023-10-11 at 14:37 +0200, Frederic Martinsons wrote: > We may want to remove the EXCLUDE_FROM_WORLD in zvariant > recipe since it was introduced for reproducibility issue (#15090 > which had been marked as a duplicate of the main rust issue #14875). > What do you think sundeep ? We should certainly test it, it really depends if there are further issues present or not. This certainly is a good first step though! Cheers, Richard
I looked briefly into the issue 15090 (zvariant recipe) description and as per me it should be tested before removing EXCLUDE_FROM_WORLD to ensure the builds are reproducible.
Le mer. 11 oct. 2023, 16:32, Sundeep KOKKONDA via lists.openembedded.org <sundeep.kokkonda=windriver.com@lists.openembedded.org> a écrit : > I looked briefly into the issue 15090 (zvariant recipe) description and as > per me it should be tested before removing EXCLUDE_FROM_WORLD to ensure the > builds are reproducible. I'll try do that in the coming days and submit the patch if it is OK. > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#188961): > https://lists.openembedded.org/g/openembedded-core/message/188961 > Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > frederic.martinsons@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Tue, 2023-10-10 at 21:05 -0700, Sundeep KOKKONDA via lists.openembedded.org wrote: > [Yocto#14875] > The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary. > However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on. > So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'. > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> > --- > meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- > meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + > 2 files changed, 1 insertion(+), 2 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py > index 84c6c3a05f..9b4a0887dd 100644 > --- a/meta/lib/oeqa/selftest/cases/reproducible.py > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py > @@ -16,8 +16,6 @@ import os > import datetime > > exclude_packages = [ > - 'rust', > - 'rust-dbg' > ] > > def is_excluded(package): > diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb b/meta/recipes-devtools/rust/rust_1.70.0.bb > index 3b9c05a19f..16d433910f 100644 > --- a/meta/recipes-devtools/rust/rust_1.70.0.bb > +++ b/meta/recipes-devtools/rust/rust_1.70.0.bb > @@ -130,6 +130,7 @@ python do_configure() { > # [rust] > config.add_section("rust") > config.set("rust", "rpath", e(True)) > + config.set("rust", "remap-debuginfo", e(True)) > config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}"))) > > # Whether or not to optimize the compiler and standard library Unfortunately I have some less good news, the rust reproducibility tests do still have some kind of failure: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3736/steps/12/logs/stdio http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20231011-7wbq1syl/packages/diff-html/ This is a bit different to the previous issue since the rlib files are not changing and this is rustdoc. At a quick glance, it looks like the symbols are changing order in the file, which can sometimes be caused by the link command not being in the same order on all builds. This could be for example something doing: $LD *.so and hence the order of the linked objects depends on the order of the files on disk and which order glob found them in. It could also be something else. So we've made great progress and this will help our SPDX issues as the filenames are now consistent but we're not quite there yet I'm afraid. Unless we can find it quickly we'll have to put rust back on the exclusion list for the release. Cheers, Richard
On Thu, 12 Oct 2023 at 08:37, Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Tue, 2023-10-10 at 21:05 -0700, Sundeep KOKKONDA via > lists.openembedded.org wrote: > > [Yocto#14875] > > The '--remap-path-prefix' option removes all references to build > directory structure in the debug information within the compiled output for > Cargo dependencies and the project's binary. > > However, some references to build directories remains in the final > binary in .rustc section in the form of compressed metadata and this makes > the build output dependent on the folder structure of the computer it's > compiled on. > > So, for reproducible builds, use the configuration option > 'remap-debuginfo = true' along with the '--remap-path-prefix'. > > > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> > > --- > > meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- > > meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + > > 2 files changed, 1 insertion(+), 2 deletions(-) > > > > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py > b/meta/lib/oeqa/selftest/cases/reproducible.py > > index 84c6c3a05f..9b4a0887dd 100644 > > --- a/meta/lib/oeqa/selftest/cases/reproducible.py > > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py > > @@ -16,8 +16,6 @@ import os > > import datetime > > > > exclude_packages = [ > > - 'rust', > > - 'rust-dbg' > > ] > > > > def is_excluded(package): > > diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb > b/meta/recipes-devtools/rust/rust_1.70.0.bb > > index 3b9c05a19f..16d433910f 100644 > > --- a/meta/recipes-devtools/rust/rust_1.70.0.bb > > +++ b/meta/recipes-devtools/rust/rust_1.70.0.bb > > @@ -130,6 +130,7 @@ python do_configure() { > > # [rust] > > config.add_section("rust") > > config.set("rust", "rpath", e(True)) > > + config.set("rust", "remap-debuginfo", e(True)) > > config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}"))) > > > > # Whether or not to optimize the compiler and standard library > > Unfortunately I have some less good news, the rust reproducibility > tests do still have some kind of failure: > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3736/steps/12/logs/stdio > > > http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20231011-7wbq1syl/packages/diff-html/ > > This is a bit different to the previous issue since the rlib files are > not changing and this is rustdoc. At a quick glance, it looks like the > symbols are changing order in the file, which can sometimes be caused > by the link command not being in the same order on all builds. This > could be for example something doing: > > $LD *.so > > and hence the order of the linked objects depends on the order of the > files on disk and which order glob found them in. It could also be > something else. > > So we've made great progress and this will help our SPDX issues as the > filenames are now consistent but we're not quite there yet I'm afraid. > Unless we can find it quickly we'll have to put rust back on the > exclusion list for the release. > > Cheers, > > Richard > > Hello, just to confirm that I still diff for zvariant package , I tested with the followin setup on poky master: local;conf: SANITY_TESTED_DISTROS = "" OEQA_REPRODUCIBLE_TEST_TARGET = "zvariant" OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS = "rust-native cargo-native" and OEQA_DEBUGGING_SAVED_OUTPUT=/home/jenkins/yocto-poky-master/reproducible_saved_output oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#188988): > https://lists.openembedded.org/g/openembedded-core/message/188988 > Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > frederic.martinsons@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py index 84c6c3a05f..9b4a0887dd 100644 --- a/meta/lib/oeqa/selftest/cases/reproducible.py +++ b/meta/lib/oeqa/selftest/cases/reproducible.py @@ -16,8 +16,6 @@ import os import datetime exclude_packages = [ - 'rust', - 'rust-dbg' ] def is_excluded(package): diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb b/meta/recipes-devtools/rust/rust_1.70.0.bb index 3b9c05a19f..16d433910f 100644 --- a/meta/recipes-devtools/rust/rust_1.70.0.bb +++ b/meta/recipes-devtools/rust/rust_1.70.0.bb @@ -130,6 +130,7 @@ python do_configure() { # [rust] config.add_section("rust") config.set("rust", "rpath", e(True)) + config.set("rust", "remap-debuginfo", e(True)) config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}"))) # Whether or not to optimize the compiler and standard library
[Yocto#14875] The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary. However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on. So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'. Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com> --- meta/lib/oeqa/selftest/cases/reproducible.py | 2 -- meta/recipes-devtools/rust/rust_1.70.0.bb | 1 + 2 files changed, 1 insertion(+), 2 deletions(-)