| Message ID | 20220203172619.1299363-1-enrico.scholz@sigma-chemnitz.de |
|---|---|
| State | New |
| Headers | show |
| Series | gcc: enable zstd compression of LTO bytecode | expand |
On Thu, Feb 3, 2022 at 9:26 AM Enrico Scholz via lists.openembedded.org <enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote: > > gcc-10+ supports zstd compression of LTO bytecode. Install the > corresponding package to enable this feature in a deterministic way. > > NOTE: previously built LTO object files (without this compression) > must be regenerated; gcc will fail else with > > | lto1: internal compiler error: original not compressed with zstd > > in this case. > > NOTE: it seems to be possible that zstd support is enabled non > deterministically (perhaps by host system pollution). > > I had the problem that the SDK gcc was built with zstd while the > cross gcc was built without it. Libraries (built by cross gcc and > uncompressed hence) could not be used with the SDK gcc. > this means regenerating entire shared state isnt it ? > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> > --- > meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc > index f07a5be507b3..62db035760aa 100644 > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc > @@ -11,7 +11,7 @@ BINV = "11.2.0" > FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:" > > DEPENDS =+ "mpfr gmp libmpc zlib flex-native" > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native" > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native" > > LICENSE = "GPL-3.0-with-GCC-exception & GPLv3" > > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#161292): https://lists.openembedded.org/g/openembedded-core/message/161292 > Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Khem Raj <raj.khem@gmail.com> writes: >> gcc-10+ supports zstd compression of LTO bytecode. Install the >> corresponding package to enable this feature in a deterministic way. >> >> NOTE: previously built LTO object files (without this compression) >> must be regenerated; gcc will fail else with >> >> | lto1: internal compiler error: original not compressed with zstd >> >> in this case. >> >> NOTE: it seems to be possible that zstd support is enabled non >> deterministically (perhaps by host system pollution). >> >> I had the problem that the SDK gcc was built with zstd while the >> cross gcc was built without it. Libraries (built by cross gcc and >> uncompressed hence) could not be used with the SDK gcc. > > this means regenerating entire shared state isnt it ? yes (at least when lto.inc is used (which is *not* by default)). When a bad sstate really matters for 'master', perhaps there can be wait for a change in glibc or so before applying this patch. Or do you think that adding a DISTRO_FEATURE like 'lto-zstd' would make sense? When this flag is missing, '--without-zstd' must be added to EXTRA_OECONF. Also some early [vardeps] must be set (where?) to avoid the sstate problem. Globally enabling 'zstd' appears easier because drawbacks (costs for compression + decompression) should be negligible for zstd. Enrico
On Thu, Feb 3, 2022 at 1:40 PM Enrico Scholz <enrico.scholz@sigma-chemnitz.de> wrote: > > Khem Raj <raj.khem@gmail.com> writes: > > >> gcc-10+ supports zstd compression of LTO bytecode. Install the > >> corresponding package to enable this feature in a deterministic way. > >> > >> NOTE: previously built LTO object files (without this compression) > >> must be regenerated; gcc will fail else with > >> > >> | lto1: internal compiler error: original not compressed with zstd > >> > >> in this case. > >> > >> NOTE: it seems to be possible that zstd support is enabled non > >> deterministically (perhaps by host system pollution). > >> > >> I had the problem that the SDK gcc was built with zstd while the > >> cross gcc was built without it. Libraries (built by cross gcc and > >> uncompressed hence) could not be used with the SDK gcc. > > > > this means regenerating entire shared state isnt it ? > > yes (at least when lto.inc is used (which is *not* by default)). When a > bad sstate really matters for 'master', perhaps there can be wait for a > change in glibc or so before applying this patch. > > Or do you think that adding a DISTRO_FEATURE like 'lto-zstd' would make > sense? When this flag is missing, '--without-zstd' must be added to > EXTRA_OECONF. Also some early [vardeps] must be set (where?) to avoid > the sstate problem. > > Globally enabling 'zstd' appears easier because drawbacks (costs for > compression + decompression) should be negligible for zstd. I think switching to zstd is fine. > > > > Enrico
On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via lists.openembedded.org wrote: > gcc-10+ supports zstd compression of LTO bytecode. Install the > corresponding package to enable this feature in a deterministic way. > > NOTE: previously built LTO object files (without this compression) > must be regenerated; gcc will fail else with > > | lto1: internal compiler error: original not compressed with zstd > > in this case. > > NOTE: it seems to be possible that zstd support is enabled non > deterministically (perhaps by host system pollution). > > I had the problem that the SDK gcc was built with zstd while the > cross gcc was built without it. Libraries (built by cross gcc and > uncompressed hence) could not be used with the SDK gcc. > > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> > --- > meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc > index f07a5be507b3..62db035760aa 100644 > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc > @@ -11,7 +11,7 @@ BINV = "11.2.0" > FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:" > > DEPENDS =+ "mpfr gmp libmpc zlib flex-native" > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native" > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native" > > LICENSE = "GPL-3.0-with-GCC-exception & GPLv3" > Should we be passing some flag to gcc explicitly to configure this deterministicly? Cheers, Richard
On Thu, Feb 3, 2022 at 1:59 PM Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via > lists.openembedded.org > wrote: > > gcc-10+ supports zstd compression of LTO bytecode. Install the > > corresponding package to enable this feature in a deterministic way. > > > > NOTE: previously built LTO object files (without this compression) > > must be regenerated; gcc will fail else with > > > > | lto1: internal compiler error: original not compressed with zstd > > > > in this case. > > > > NOTE: it seems to be possible that zstd support is enabled non > > deterministically (perhaps by host system pollution). > > > > I had the problem that the SDK gcc was built with zstd while the > > cross gcc was built without it. Libraries (built by cross gcc and > > uncompressed hence) could not be used with the SDK gcc. > > > > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> > > --- > > meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc > b/meta/recipes-devtools/gcc/gcc-11.2.inc > > index f07a5be507b3..62db035760aa 100644 > > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc > > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc > > @@ -11,7 +11,7 @@ BINV = "11.2.0" > > FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:" > > > > DEPENDS =+ "mpfr gmp libmpc zlib flex-native" > > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native > flex-native" > > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native > flex-native zstd-native" > > > > LICENSE = "GPL-3.0-with-GCC-exception & GPLv3" > > > > Should we be passing some flag to gcc explicitly to configure this > deterministicly? Perhaps use —with-zstd configure switch maybe via package config but it’s better to enable it always > > > Cheers, > > Richard > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#161323): > https://lists.openembedded.org/g/openembedded-core/message/161323 > Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Thu, 2022-02-03 at 14:11 -0800, Khem Raj wrote: > > > On Thu, Feb 3, 2022 at 1:59 PM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via lists.openembedded.org > > wrote: > > > gcc-10+ supports zstd compression of LTO bytecode. Install the > > > corresponding package to enable this feature in a deterministic way. > > > > > > NOTE: previously built LTO object files (without this compression) > > > must be regenerated; gcc will fail else with > > > > > > | lto1: internal compiler error: original not compressed with zstd > > > > > > in this case. > > > > > > NOTE: it seems to be possible that zstd support is enabled non > > > deterministically (perhaps by host system pollution). > > > > > > I had the problem that the SDK gcc was built with zstd while the > > > cross gcc was built without it. Libraries (built by cross gcc and > > > uncompressed hence) could not be used with the SDK gcc. > > > > > > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> > > > --- > > > meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes- > > devtools/gcc/gcc-11.2.inc > > > index f07a5be507b3..62db035760aa 100644 > > > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc > > > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc > > > @@ -11,7 +11,7 @@ BINV = "11.2.0" > > > FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:" > > > > > > DEPENDS =+ "mpfr gmp libmpc zlib flex-native" > > > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex- > > > native" > > > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex- > > > native > > zstd-native" > > > > > > LICENSE = "GPL-3.0-with-GCC-exception & GPLv3" > > > > > > > Should we be passing some flag to gcc explicitly to configure this > > deterministicly? > > > > > Perhaps use —with-zstd configure switch maybe via package config but it’s > better to enable it always > Agreed, I'm fine with enabling it, I'd just like to be explicit about what we're doing. Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> writes: >> | lto1: internal compiler error: original not compressed with zstd >> >> DEPENDS =+ "mpfr gmp libmpc zlib flex-native" >> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native" >> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native" >> >> LICENSE = "GPL-3.0-with-GCC-exception & GPLv3" >> > > Should we be passing some flag to gcc explicitly to configure this > deterministicly? Both '--with-zstd' and '--without-zstd' make it deterministic (require zstd.h and abort when non-existing resp. ignore it when existing). Perhaps '--without-zstd' should be added in the hardknott and honister branches to prevent pollution by host environment. Enrico
On Fri, Feb 4, 2022 at 11:00 AM Enrico Scholz via lists.openembedded.org <enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote: > > Richard Purdie <richard.purdie@linuxfoundation.org> writes: > > >> | lto1: internal compiler error: original not compressed with zstd > >> > >> DEPENDS =+ "mpfr gmp libmpc zlib flex-native" > >> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native" > >> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native" > >> > >> LICENSE = "GPL-3.0-with-GCC-exception & GPLv3" > >> > > > > Should we be passing some flag to gcc explicitly to configure this > > deterministicly? > > Both '--with-zstd' and '--without-zstd' make it deterministic (require > zstd.h and abort when non-existing resp. ignore it when existing). > > Perhaps '--without-zstd' should be added in the hardknott and honister > branches to prevent pollution by host environment. > yes makes sense. > > > Enrico > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#161379): https://lists.openembedded.org/g/openembedded-core/message/161379 > Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc index f07a5be507b3..62db035760aa 100644 --- a/meta/recipes-devtools/gcc/gcc-11.2.inc +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc @@ -11,7 +11,7 @@ BINV = "11.2.0" FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:" DEPENDS =+ "mpfr gmp libmpc zlib flex-native" -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native" +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native" LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
gcc-10+ supports zstd compression of LTO bytecode. Install the corresponding package to enable this feature in a deterministic way. NOTE: previously built LTO object files (without this compression) must be regenerated; gcc will fail else with | lto1: internal compiler error: original not compressed with zstd in this case. NOTE: it seems to be possible that zstd support is enabled non deterministically (perhaps by host system pollution). I had the problem that the SDK gcc was built with zstd while the cross gcc was built without it. Libraries (built by cross gcc and uncompressed hence) could not be used with the SDK gcc. Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> --- meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)