Message ID | 20231224230934.831-1-alex.kiernan@gmail.com |
---|---|
Headers | show |
Series | Stepwise rust upgrade 1.71.1 -> 1.74.1 | expand |
On Sun, 2023-12-24 at 23:09 +0000, Alex Kiernan wrote: > This is the 1.74.1 rust series rebased to include a revert of > https://github.com/rust-lang/cc-rs/commit/c4f414f449bb7cffba3bc923f277704d1d08a8ec > which I'm pretty sure is what's causing our filename churn. I've checked > 1.72.0 and 1.74.1 for the absvdi2.o intrinsic (one of many intrinsics > which had the issue) and in both cases we're not generating the dirname > based prefix - hopefully that means that the interim commits are fine > too, if not we can do the step back through the commits to find the next > issue. > > I've dropped the zvariant tests as upgrading it isn't useful (since it > no longer includes git crate dependency) and spurious oe-selftest > failures aren't helpful. I guess we need to include either something > synthetic which tests git crates, or pull in > https://github.com/jthornber/thin-provisioning-tools/ from > meta-openembedded which includes a live example (though who knows for > how long!) > > Assuming this does actually fix the reproducibility issue, I'll look at > how we fix the issue properly, rather than just reverting the commit > which I think is our problem, but I'd like to try and get us back on the > rust release train if we can! > > > Alex Kiernan (6): > meta-selftest: Drop zvariant recipe > rust: Upgrade 1.71.1 -> 1.72.0 > rust: Upgrade 1.72.0 -> 1.72.1 > rust: Upgrade 1.72.1 -> 1.73.0 > rust: Upgrade 1.73.0 -> 1.74.0 > rust: Upgrade 1.74.0 -> 1.74.1 Sounds like a good plan, I've started a reproducibility test: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4142 With zvariant, we might be better replacing with a synthetic example for testing purposes? Cheers, Richard
On Sun, Dec 24, 2023 at 3:09 PM Alex Kiernan <alex.kiernan@gmail.com> wrote: > This is the 1.74.1 rust series rebased to include a revert of > > https://github.com/rust-lang/cc-rs/commit/c4f414f449bb7cffba3bc923f277704d1d08a8ec I think this is definitely the culprit kudos to track it down. I think this could be improved by generating hash of file content instead of the pathname to prefix the relative path and we will fix both the issues <https://github.com/rust-lang/cc-rs/commit/c4f414f449bb7cffba3bc923f277704d1d08a8ec> > which I'm pretty sure is what's causing our filename churn. I've checked > 1.72.0 and 1.74.1 for the absvdi2.o intrinsic (one of many intrinsics > which had the issue) and in both cases we're not generating the dirname > based prefix - hopefully that means that the interim commits are fine > too, if not we can do the step back through the commits to find the next > issue. > > I've dropped the zvariant tests as upgrading it isn't useful (since it > no longer includes git crate dependency) and spurious oe-selftest > failures aren't helpful. I guess we need to include either something > synthetic which tests git crates, or pull in > https://github.com/jthornber/thin-provisioning-tools/ from > meta-openembedded which includes a live example (though who knows for > how long!) > > Assuming this does actually fix the reproducibility issue, I'll look at > how we fix the issue properly, rather than just reverting the commit > which I think is our problem, but I'd like to try and get us back on the > rust release train if we can! > > > Alex Kiernan (6): > meta-selftest: Drop zvariant recipe > rust: Upgrade 1.71.1 -> 1.72.0 > rust: Upgrade 1.72.0 -> 1.72.1 > rust: Upgrade 1.72.1 -> 1.73.0 > rust: Upgrade 1.73.0 -> 1.74.0 > rust: Upgrade 1.74.0 -> 1.74.1 > > .../zvariant/zvariant-crates.inc | 258 ---- > .../zvariant/zvariant-git-crates.inc | 14 - > .../0001-Tweak-zvariant-crate-config.patch | 1292 ----------------- > .../zvariant/zvariant_3.12.0.bb | 37 - > meta/conf/distro/include/tcmode-default.inc | 2 +- > meta/lib/oeqa/selftest/cases/devtool.py | 93 -- > .../rust/{cargo_1.71.1.bb => cargo_1.74.1.bb} | 0 > ...-Do-not-use-LFS64-on-linux-with-musl.patch | 164 --- > ...0001-Don-t-use-LFS64-symbols-on-musl.patch | 163 +++ > ...e-absolute-paths-to-OUT_DIR-as-relat.patch | 67 + > ...Define-SOCK_NONBLOCK-with-O_NONBLOCK.patch | 122 ++ > ...efine-SOCK_SEQPACKET-in-common-place.patch | 114 -- > ...ine-F_SETLK-F_SETLKW-and-fix-F_GETLK.patch | 41 + > ...GETLK-F_OFD_SETLK-and-F_OFD_SETLKW-t.patch | 205 +++ > ...-musl-Define-O_LARGEFILE-for-riscv32.patch | 32 + > ...efine-SOCK_SEQPACKET-in-common-place.patch | 115 ++ > .../rust/files/getrandom-open64.patch | 50 - > .../rust/files/hardcodepaths.patch | 14 +- > .../rust/files/zlib-off64_t.patch | 17 +- > ...ibstd-rs_1.71.1.bb => libstd-rs_1.74.1.bb} | 0 > ....71.1.bb => rust-cross-canadian_1.74.1.bb} | 0 > ...ust-llvm_1.71.1.bb => rust-llvm_1.74.1.bb} | 0 > meta/recipes-devtools/rust/rust-snapshot.inc | 64 +- > meta/recipes-devtools/rust/rust-source.inc | 12 +- > .../rust/{rust_1.71.1.bb => rust_1.74.1.bb} | 1 + > 25 files changed, 802 insertions(+), 2075 deletions(-) > delete mode 100644 > meta-selftest/recipes-extended/zvariant/zvariant-crates.inc > delete mode 100644 > meta-selftest/recipes-extended/zvariant/zvariant-git-crates.inc > delete mode 100644 > meta-selftest/recipes-extended/zvariant/zvariant/0001-Tweak-zvariant-crate-config.patch > delete mode 100644 meta-selftest/recipes-extended/zvariant/ > zvariant_3.12.0.bb > rename meta/recipes-devtools/rust/{cargo_1.71.1.bb => cargo_1.74.1.bb} > (100%) > delete mode 100644 > meta/recipes-devtools/rust/files/0001-Do-not-use-LFS64-on-linux-with-musl.patch > create mode 100644 > meta/recipes-devtools/rust/files/0001-Don-t-use-LFS64-symbols-on-musl.patch > create mode 100644 > meta/recipes-devtools/rust/files/0001-Revert-Map-source-absolute-paths-to-OUT_DIR-as-relat.patch > create mode 100644 > meta/recipes-devtools/rust/files/0001-musl-Define-SOCK_NONBLOCK-with-O_NONBLOCK.patch > delete mode 100644 > meta/recipes-devtools/rust/files/0001-musl-Define-SOCK_SEQPACKET-in-common-place.patch > create mode 100644 > meta/recipes-devtools/rust/files/0002-musl-riscv32-Define-F_SETLK-F_SETLKW-and-fix-F_GETLK.patch > create mode 100644 > meta/recipes-devtools/rust/files/0003-musl-Move-F_OFD_GETLK-F_OFD_SETLK-and-F_OFD_SETLKW-t.patch > create mode 100644 > meta/recipes-devtools/rust/files/0004-musl-Define-O_LARGEFILE-for-riscv32.patch > create mode 100644 > meta/recipes-devtools/rust/files/0005-musl-Define-SOCK_SEQPACKET-in-common-place.patch > delete mode 100644 meta/recipes-devtools/rust/files/getrandom-open64.patch > rename meta/recipes-devtools/rust/{libstd-rs_1.71.1.bb => > libstd-rs_1.74.1.bb} (100%) > rename meta/recipes-devtools/rust/{rust-cross-canadian_1.71.1.bb => > rust-cross-canadian_1.74.1.bb} (100%) > rename meta/recipes-devtools/rust/{rust-llvm_1.71.1.bb => > rust-llvm_1.74.1.bb} (100%) > rename meta/recipes-devtools/rust/{rust_1.71.1.bb => rust_1.74.1.bb} > (99%) > > -- > 2.39.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#192895): > https://lists.openembedded.org/g/openembedded-core/message/192895 > Mute This Topic: https://lists.openembedded.org/mt/103354255/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Mon, 2023-12-25 at 08:40 +0000, Richard Purdie via lists.openembedded.org wrote: > On Sun, 2023-12-24 at 23:09 +0000, Alex Kiernan wrote: > > This is the 1.74.1 rust series rebased to include a revert of > > https://github.com/rust-lang/cc-rs/commit/c4f414f449bb7cffba3bc923f277704d1d08a8ec > > which I'm pretty sure is what's causing our filename churn. I've checked > > 1.72.0 and 1.74.1 for the absvdi2.o intrinsic (one of many intrinsics > > which had the issue) and in both cases we're not generating the dirname > > based prefix - hopefully that means that the interim commits are fine > > too, if not we can do the step back through the commits to find the next > > issue. > > > > I've dropped the zvariant tests as upgrading it isn't useful (since it > > no longer includes git crate dependency) and spurious oe-selftest > > failures aren't helpful. I guess we need to include either something > > synthetic which tests git crates, or pull in > > https://github.com/jthornber/thin-provisioning-tools/ from > > meta-openembedded which includes a live example (though who knows for > > how long!) > > > > Assuming this does actually fix the reproducibility issue, I'll look at > > how we fix the issue properly, rather than just reverting the commit > > which I think is our problem, but I'd like to try and get us back on the > > rust release train if we can! > > > > > > Alex Kiernan (6): > > meta-selftest: Drop zvariant recipe > > rust: Upgrade 1.71.1 -> 1.72.0 > > rust: Upgrade 1.72.0 -> 1.72.1 > > rust: Upgrade 1.72.1 -> 1.73.0 > > rust: Upgrade 1.73.0 -> 1.74.0 > > rust: Upgrade 1.74.0 -> 1.74.1 > > Sounds like a good plan, I've started a reproducibility test: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4142 > > With zvariant, we might be better replacing with a synthetic example > for testing purposes? That build went green so I ran an a-full to double check and check everything else: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6374 Curl ptest warning was the only issue (unrelated) so it also looks good. Looks like you might have tracked down the issue, nice work! :) Should I merge this as is or is there something we should do first now we know where the issue is? Cheers, Richard
On Mon, Dec 25, 2023 at 10:02 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Mon, 2023-12-25 at 08:40 +0000, Richard Purdie via > lists.openembedded.org wrote: > > On Sun, 2023-12-24 at 23:09 +0000, Alex Kiernan wrote: > > > This is the 1.74.1 rust series rebased to include a revert of > > > https://github.com/rust-lang/cc-rs/commit/c4f414f449bb7cffba3bc923f277704d1d08a8ec > > > which I'm pretty sure is what's causing our filename churn. I've checked > > > 1.72.0 and 1.74.1 for the absvdi2.o intrinsic (one of many intrinsics > > > which had the issue) and in both cases we're not generating the dirname > > > based prefix - hopefully that means that the interim commits are fine > > > too, if not we can do the step back through the commits to find the next > > > issue. > > > > > > I've dropped the zvariant tests as upgrading it isn't useful (since it > > > no longer includes git crate dependency) and spurious oe-selftest > > > failures aren't helpful. I guess we need to include either something > > > synthetic which tests git crates, or pull in > > > https://github.com/jthornber/thin-provisioning-tools/ from > > > meta-openembedded which includes a live example (though who knows for > > > how long!) > > > > > > Assuming this does actually fix the reproducibility issue, I'll look at > > > how we fix the issue properly, rather than just reverting the commit > > > which I think is our problem, but I'd like to try and get us back on the > > > rust release train if we can! > > > > > > > > > Alex Kiernan (6): > > > meta-selftest: Drop zvariant recipe > > > rust: Upgrade 1.71.1 -> 1.72.0 > > > rust: Upgrade 1.72.0 -> 1.72.1 > > > rust: Upgrade 1.72.1 -> 1.73.0 > > > rust: Upgrade 1.73.0 -> 1.74.0 > > > rust: Upgrade 1.74.0 -> 1.74.1 > > > > Sounds like a good plan, I've started a reproducibility test: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4142 > > > > With zvariant, we might be better replacing with a synthetic example > > for testing purposes? > > That build went green so I ran an a-full to double check and check > everything else: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6374 > > Curl ptest warning was the only issue (unrelated) so it also looks > good. > > Looks like you might have tracked down the issue, nice work! :) > > Should I merge this as is or is there something we should do first now > we know where the issue is? > Obviously we need the upstream pieces sorting, but I don't think that's a blocker to merging here. Just ripping out zvariant and all the associated infrastructure I guess wants fixing first, though I think don't we need anything more complicated than: https://github.com/akiernan/hello-rs I'll try and have a look at that later.
On Tue, Dec 26, 2023 at 8:59 AM Alex Kiernan via lists.openembedded.org <alex.kiernan=gmail.com@lists.openembedded.org> wrote: > > On Mon, Dec 25, 2023 at 10:02 PM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Mon, 2023-12-25 at 08:40 +0000, Richard Purdie via > > lists.openembedded.org wrote: > > > On Sun, 2023-12-24 at 23:09 +0000, Alex Kiernan wrote: > > > > This is the 1.74.1 rust series rebased to include a revert of > > > > https://github.com/rust-lang/cc-rs/commit/c4f414f449bb7cffba3bc923f277704d1d08a8ec > > > > which I'm pretty sure is what's causing our filename churn. I've checked > > > > 1.72.0 and 1.74.1 for the absvdi2.o intrinsic (one of many intrinsics > > > > which had the issue) and in both cases we're not generating the dirname > > > > based prefix - hopefully that means that the interim commits are fine > > > > too, if not we can do the step back through the commits to find the next > > > > issue. > > > > > > > > I've dropped the zvariant tests as upgrading it isn't useful (since it > > > > no longer includes git crate dependency) and spurious oe-selftest > > > > failures aren't helpful. I guess we need to include either something > > > > synthetic which tests git crates, or pull in > > > > https://github.com/jthornber/thin-provisioning-tools/ from > > > > meta-openembedded which includes a live example (though who knows for > > > > how long!) > > > > > > > > Assuming this does actually fix the reproducibility issue, I'll look at > > > > how we fix the issue properly, rather than just reverting the commit > > > > which I think is our problem, but I'd like to try and get us back on the > > > > rust release train if we can! > > > > > > > > > > > > Alex Kiernan (6): > > > > meta-selftest: Drop zvariant recipe > > > > rust: Upgrade 1.71.1 -> 1.72.0 > > > > rust: Upgrade 1.72.0 -> 1.72.1 > > > > rust: Upgrade 1.72.1 -> 1.73.0 > > > > rust: Upgrade 1.73.0 -> 1.74.0 > > > > rust: Upgrade 1.74.0 -> 1.74.1 > > > > > > Sounds like a good plan, I've started a reproducibility test: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4142 > > > > > > With zvariant, we might be better replacing with a synthetic example > > > for testing purposes? > > > > That build went green so I ran an a-full to double check and check > > everything else: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6374 > > > > Curl ptest warning was the only issue (unrelated) so it also looks > > good. > > > > Looks like you might have tracked down the issue, nice work! :) > > > > Should I merge this as is or is there something we should do first now > > we know where the issue is? > > > > Obviously we need the upstream pieces sorting, but I don't think > that's a blocker to merging here. Just ripping out zvariant and all > the associated infrastructure I guess wants fixing first, though I > think don't we need anything more complicated than: > > https://github.com/akiernan/hello-rs > > I'll try and have a look at that later. > The pieces I have are: https://github.com/akiernan/hello-bin https://github.com/akiernan/hello-lib Which build/work fine, but the swapping it in for the zvariant tests fails with things totally unrelated to upgrading rust, which suggest we have more stuff to fix, so after two days looking at it, on and off, I think just skipping the relevant test is the right thing for now. 1.75.0 lands tomorrow, and from a quick test breaks everything again :|
On Wed, Dec 27, 2023 at 6:46 PM Alex Kiernan via lists.openembedded.org <alex.kiernan=gmail.com@lists.openembedded.org> wrote: > > On Tue, Dec 26, 2023 at 8:59 AM Alex Kiernan via > lists.openembedded.org <alex.kiernan=gmail.com@lists.openembedded.org> > wrote: > > > > On Mon, Dec 25, 2023 at 10:02 PM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Mon, 2023-12-25 at 08:40 +0000, Richard Purdie via > > > lists.openembedded.org wrote: > > > > On Sun, 2023-12-24 at 23:09 +0000, Alex Kiernan wrote: > > > > > This is the 1.74.1 rust series rebased to include a revert of > > > > > https://github.com/rust-lang/cc-rs/commit/c4f414f449bb7cffba3bc923f277704d1d08a8ec > > > > > which I'm pretty sure is what's causing our filename churn. I've checked > > > > > 1.72.0 and 1.74.1 for the absvdi2.o intrinsic (one of many intrinsics > > > > > which had the issue) and in both cases we're not generating the dirname > > > > > based prefix - hopefully that means that the interim commits are fine > > > > > too, if not we can do the step back through the commits to find the next > > > > > issue. > > > > > > > > > > I've dropped the zvariant tests as upgrading it isn't useful (since it > > > > > no longer includes git crate dependency) and spurious oe-selftest > > > > > failures aren't helpful. I guess we need to include either something > > > > > synthetic which tests git crates, or pull in > > > > > https://github.com/jthornber/thin-provisioning-tools/ from > > > > > meta-openembedded which includes a live example (though who knows for > > > > > how long!) > > > > > > > > > > Assuming this does actually fix the reproducibility issue, I'll look at > > > > > how we fix the issue properly, rather than just reverting the commit > > > > > which I think is our problem, but I'd like to try and get us back on the > > > > > rust release train if we can! > > > > > > > > > > > > > > > Alex Kiernan (6): > > > > > meta-selftest: Drop zvariant recipe > > > > > rust: Upgrade 1.71.1 -> 1.72.0 > > > > > rust: Upgrade 1.72.0 -> 1.72.1 > > > > > rust: Upgrade 1.72.1 -> 1.73.0 > > > > > rust: Upgrade 1.73.0 -> 1.74.0 > > > > > rust: Upgrade 1.74.0 -> 1.74.1 > > > > > > > > Sounds like a good plan, I've started a reproducibility test: > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4142 > > > > > > > > With zvariant, we might be better replacing with a synthetic example > > > > for testing purposes? > > > > > > That build went green so I ran an a-full to double check and check > > > everything else: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6374 > > > > > > Curl ptest warning was the only issue (unrelated) so it also looks > > > good. > > > > > > Looks like you might have tracked down the issue, nice work! :) > > > > > > Should I merge this as is or is there something we should do first now > > > we know where the issue is? > > > > > > > Obviously we need the upstream pieces sorting, but I don't think > > that's a blocker to merging here. Just ripping out zvariant and all > > the associated infrastructure I guess wants fixing first, though I > > think don't we need anything more complicated than: > > > > https://github.com/akiernan/hello-rs > > > > I'll try and have a look at that later. > > > > The pieces I have are: > > https://github.com/akiernan/hello-bin > https://github.com/akiernan/hello-lib > > Which build/work fine, but the swapping it in for the zvariant tests > fails with things totally unrelated to upgrading rust, which suggest > we have more stuff to fix, so after two days looking at it, on and > off, I think just skipping the relevant test is the right thing for > now. > Unsurprisingly this was a complete idiot mistake I'd made... step away from it and look at it fresh and its obvious why it didn't work! Will send a v3 w/ a replacement for zvariant.