| Message ID | VI1PR04MB537371C2BED481FEDB533DB89102A@VI1PR04MB5373.eurprd04.prod.outlook.com |
|---|---|
| State | New |
| Headers | show |
| Series | ODP: [OE-Core][PATCH v10][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr | expand |
Hello Piotr, On 24/07/2023 20:50:50+0000, Piotr Łobacz wrote: > > Hello Alexandre, > this is a final patchest which works for me, even with reproducible builds, but additionally, with posix format of tar archives. > > I hope that this time it will work for autobuild as well. We are almost there, there are still reproducibility failures which I guess where hidding in all the other other ones: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3251/steps/12/logs/stdio This time, diffoscope ran: https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230724-5av30jl8/packages/diff-html/ > > BR > Piotr > > Od: Piotr Łobacz <p.lobacz@welotec.com> > Wysłane: poniedziałek, 24 lipca 2023 22:46 > Do: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> > DW: Piotr Łobacz <p.lobacz@welotec.com> > Temat: [OE-Core][PATCH v10][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr > > Extend OPKGBUILDCMD variable, with additional parameters, depending > on target distro features, in order to support ACLs and xattr. > > With fix pushed to the opkg-devel: > https://groups.google.com/g/opkg-devel/c/dYNHrLjDwg8 > opkg-build is able to create tar archives with ACLs and xattr. > > Signed-off-by: Piotr Łobacz <p.lobacz@welotec.com> > --- > meta/classes-global/package_ipk.bbclass | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/classes-global/package_ipk.bbclass b/meta/classes-global/package_ipk.bbclass > index b4b7bc9ac2..a0f106e4ad 100644 > --- a/meta/classes-global/package_ipk.bbclass > +++ b/meta/classes-global/package_ipk.bbclass > @@ -15,7 +15,7 @@ IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf" > PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks" > > # Program to be used to build opkg packages > -OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}"' > +OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}" ${@bb.utils.contains('DISTRO_FEATURES', 'acl', '-A', '', d)} ${@bb.utils.contains('DISTRO_FEATURES', 'xattr', '-X', '', d)}' > > OPKG_ARGS += "--force_postinstall --prefer-arch-to-version" > OPKG_ARGS += "${@['', '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS") == "1"]}" > -- > 2.34.1 > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#184818): https://lists.openembedded.org/g/openembedded-core/message/184818 > Mute This Topic: https://lists.openembedded.org/mt/100337925/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hmmm, this is odd because yesterday I have checked this and the additional parameters added to tar have fixed the issue for me. But I have added them only for posix tar format, because for gnu format there is no need. BTW. only these packages are different? AssertionError: The following ipk packages are different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/acpid-src_2.0.34-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/avahi-src_0.8-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/bind-src_9.18.16-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/debugedit-src_5.0-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/flac-src_1.4.3-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libjson-glib-1.0-dev_1.6.6-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libjson-glib-1.0-ptest_1.6.6-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libjson-glib-1.0-src_1.6.6-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libmnl-src_1.0.5-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libuuid-src_2.38.1-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/log4cplus-src_2.1.0-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/lttng-tools-src_2.13.9-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-numpy-src_1.25.1-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-pygobject-dev_3.44.1-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-pygobject-src_3.44.1-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-pygobject_3.44.1-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/util-linux-ptest_2.38.1-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/util-linux-src_2.38.1-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/webkitgtk-src_2.40.2-r0_core2-64.ipk Od: Alexandre Belloni <alexandre.belloni@bootlin.com> Wysłane: wtorek, 25 lipca 2023 11:28 Do: Piotr Łobacz <p.lobacz@welotec.com> DW: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> Temat: Re: ODP: [OE-Core][PATCH v10][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr Hello Piotr, On 24/07/2023 20:50:50+0000, Piotr Łobacz wrote: > > Hello Alexandre, > this is a final patchest which works for me, even with reproducible builds, but additionally, with posix format of tar archives. > > I hope that this time it will work for autobuild as well. We are almost there, there are still reproducibility failures which I guess where hidding in all the other other ones: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3251/steps/12/logs/stdio This time, diffoscope ran: https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230724-5av30jl8/packages/diff-html/ > > BR > Piotr > > Od: Piotr Łobacz <p.lobacz@welotec.com> > Wysłane: poniedziałek, 24 lipca 2023 22:46 > Do: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> > DW: Piotr Łobacz <p.lobacz@welotec.com> > Temat: [OE-Core][PATCH v10][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr > > Extend OPKGBUILDCMD variable, with additional parameters, depending > on target distro features, in order to support ACLs and xattr. > > With fix pushed to the opkg-devel: > https://groups.google.com/g/opkg-devel/c/dYNHrLjDwg8 > opkg-build is able to create tar archives with ACLs and xattr. > > Signed-off-by: Piotr Łobacz <p.lobacz@welotec.com> > --- > meta/classes-global/package_ipk.bbclass | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/classes-global/package_ipk.bbclass b/meta/classes-global/package_ipk.bbclass > index b4b7bc9ac2..a0f106e4ad 100644 > --- a/meta/classes-global/package_ipk.bbclass > +++ b/meta/classes-global/package_ipk.bbclass > @@ -15,7 +15,7 @@ IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf" > PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks" > > # Program to be used to build opkg packages > -OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}"' > +OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}" ${@bb.utils.contains('DISTRO_FEATURES', 'acl', '-A', '', d)} ${@bb.utils.contains('DISTRO_FEATURES', 'xattr', '-X', '', d)}' > > OPKG_ARGS += "--force_postinstall --prefer-arch-to-version" > OPKG_ARGS += "${@['', '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS") == "1"]}" > -- > 2.34.1 > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#184818): https://lists.openembedded.org/g/openembedded-core/message/184818 > Mute This Topic: https://lists.openembedded.org/mt/100337925/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com/
Yeah, I have just checked it for acpid-src packages and diffoscope worked without any errors. Question, is this test being run with acl or/and xattr? Because from what I remember it was not. Can you check? If this will confirm, then something is still wrong with the gnu format, because right now all posix - meaning with acl/xattr support - packages are working for me. BR Piotr
On Tue, 25 Jul 2023 at 11:58, Piotr Łobacz <p.lobacz@welotec.com> wrote: > > Yeah, I have just checked it for acpid-src packages and diffoscope worked without any errors. > > Question, is this test being run with acl or/and xattr? Because from what I remember it was not. Can you check? > If this will confirm, then something is still wrong with the gnu format, because right now all posix - meaning with acl/xattr support - packages are working for me. Hello Piotr, may I suggest: you need to learn to reproduce the autobuilder failures locally. Integrators are stretched very thin, they juggle a ton of patches (other than the ones you want to get in), and the expectation is that when they tell you something fails, it is on you to do further investigation. Here's how you do it: - start by finding the failed build in the list of their definitions: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json You can see that it's this one: "reproducible" : { "MACHINE" : "qemux86-64", "SDKMACHINE" : "x86_64", "step1" : { "shortname" : "Reproducible Selftest", "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -r reproducible"], "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] } What that means is you need to set up a plain poky build (no extra layers, no custom configs specific to your organization), then run 'oe-selftest -r reproducible'. That is however a heavy test as it builds everything, so we need to narrow it down to just the failing pieces. - the failure log shows that reproducibility is tested via meta/lib/oeqa/selftest/cases/reproducible.py. If you open that file, you can see OEQA_REPRODUCIBLE_TEST_TARGET narrows down the targets used to test reproducibility, and you can also set OEQA_REPRODUCIBLE_TEST_PACKAGE similarly. - now you can run oe-selftest -r reproducible. Once it finishes and confirms the failure, head to build-st/reproducibleA and build-st/reproducibleB, to see where the difference first appears. You can also replicate reproducibility test by manually setting up two different build directories with identical configs. Alex
Hi Alex,
at the beginning I would like to thank you for your suggestions, and I want to mention that this is what I wanted to do that from the very beginning,
but I could not find any good HowTo or manual how to do that. Right now, I have this poky-contrib:abelloni/master-next branch downloaded,
which also has the patches given by me.
Now, I'm trying to run it using these commands:
. ./oe-init-build-env build
and than:
oe-selftest -r reproducible
but for some reason it produces me an error:
raceback (most recent call last):
File "/home/plobacz/workspace/poky-contrib/scripts/oe-selftest", line 60, in <module>
ret = main()
File "/home/plobacz/workspace/poky-contrib/scripts/oe-selftest", line 47, in main
results = args.func(logger, args)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/selftest/context.py", line 406, in run
self._process_args(logger, args)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/selftest/context.py", line 252, in _process_args
bbvars = get_bb_vars()
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 248, in get_bb_vars
bbenv = get_bb_env(target, postconfig=postconfig)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 244, in get_bb_env
return bitbake("-e", postconfig=postconfig).output
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 234, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake -e' returned non-zero exit status 1:
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
chrpath
But:
echo $PATH
/home/plobacz/workspace/poky-contrib/scripts:/home/plobacz/workspace/poky-contrib/bitbake/bin:/home/plobacz/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Maybe I'm doing it somehow in wrong order?
BR
Piotr
Oh jesus I'm a morron I haven't noticed that I'm missing chrpath
Do not bother this messages:P
Cheers,
Piotr
________________________________
Od: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> w imieniu użytkownika Piotr Łobacz via lists.openembedded.org <p.lobacz=welotec.com@lists.openembedded.org>
Wysłane: wtorek, 25 lipca 2023 17:22
Do: Alexander Kanavin <alex.kanavin@gmail.com>
DW: Alexandre Belloni <alexandre.belloni@bootlin.com>; openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org>
Temat: ODP: ODP: [OE-Core][PATCH v10][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr
Hi Alex,
at the beginning I would like to thank you for your suggestions, and I want to mention that this is what I wanted to do that from the very beginning,
but I could not find any good HowTo or manual how to do that. Right now, I have this poky-contrib:abelloni/master-next branch downloaded,
which also has the patches given by me.
Now, I'm trying to run it using these commands:
. ./oe-init-build-env build
and than:
oe-selftest -r reproducible
but for some reason it produces me an error:
raceback (most recent call last):
File "/home/plobacz/workspace/poky-contrib/scripts/oe-selftest", line 60, in <module>
ret = main()
File "/home/plobacz/workspace/poky-contrib/scripts/oe-selftest", line 47, in main
results = args.func(logger, args)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/selftest/context.py", line 406, in run
self._process_args(logger, args)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/selftest/context.py", line 252, in _process_args
bbvars = get_bb_vars()
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 248, in get_bb_vars
bbenv = get_bb_env(target, postconfig=postconfig)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 244, in get_bb_env
return bitbake("-e", postconfig=postconfig).output
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 234, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
File "/home/plobacz/workspace/poky-contrib/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake -e' returned non-zero exit status 1:
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
chrpath
But:
echo $PATH
/home/plobacz/workspace/poky-contrib/scripts:/home/plobacz/workspace/poky-contrib/bitbake/bin:/home/plobacz/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Maybe I'm doing it somehow in wrong order?
BR
Piotr
________________________________
Od: Alexander Kanavin <alex.kanavin@gmail.com>
Wysłane: wtorek, 25 lipca 2023 12:28
Do: Piotr Łobacz <p.lobacz@welotec.com>
DW: Alexandre Belloni <alexandre.belloni@bootlin.com>; openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org>
Temat: Re: ODP: [OE-Core][PATCH v10][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr
On Tue, 25 Jul 2023 at 11:58, Piotr Łobacz <p.lobacz@welotec.com> wrote:
>
> Yeah, I have just checked it for acpid-src packages and diffoscope worked without any errors.
>
> Question, is this test being run with acl or/and xattr? Because from what I remember it was not. Can you check?
> If this will confirm, then something is still wrong with the gnu format, because right now all posix - meaning with acl/xattr support - packages are working for me.
Hello Piotr,
may I suggest: you need to learn to reproduce the autobuilder failures
locally. Integrators are stretched very thin, they juggle a ton of
patches (other than the ones you want to get in), and the expectation
is that when they tell you something fails, it is on you to do further
investigation. Here's how you do it:
- start by finding the failed build in the list of their definitions:
https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.yoctoproject.org%2Fyocto-autobuilder-helper%2Ftree%2Fconfig.json&data=05%7C01%7Cp.lobacz%40welotec.com%7C29e17992f2b148e9206c08db8cf9e54c%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638258777149507734%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3czxntfPZq68MvQxKQtJT6Rk5la4O0cca9osxYGcVqI%3D&reserved=0<https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json>
You can see that it's this one:
"reproducible" : {
"MACHINE" : "qemux86-64",
"SDKMACHINE" : "x86_64",
"step1" : {
"shortname" : "Reproducible Selftest",
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/
DISPLAY=:1 oe-selftest -r reproducible"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
}
What that means is you need to set up a plain poky build (no extra
layers, no custom configs specific to your organization), then run
'oe-selftest -r reproducible'. That is however a heavy test as it
builds everything, so we need to narrow it down to just the failing
pieces.
- the failure log shows that reproducibility is tested via
meta/lib/oeqa/selftest/cases/reproducible.py. If you open that file,
you can see OEQA_REPRODUCIBLE_TEST_TARGET narrows down the targets
used to test reproducibility, and you can also set
OEQA_REPRODUCIBLE_TEST_PACKAGE similarly.
- now you can run oe-selftest -r reproducible. Once it finishes and
confirms the failure, head to build-st/reproducibleA and
build-st/reproducibleB, to see where the difference first appears.
You can also replicate reproducibility test by manually setting up two
different build directories with identical configs.
Alex
H
diff --git a/meta/classes-global/package_ipk.bbclass b/meta/classes-global/package_ipk.bbclass index b4b7bc9ac2..a0f106e4ad 100644 --- a/meta/classes-global/package_ipk.bbclass +++ b/meta/classes-global/package_ipk.bbclass @@ -15,7 +15,7 @@ IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf" PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks" # Program to be used to build opkg packages -OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}"' +OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}" ${@bb.utils.contains('DISTRO_FEATURES', 'acl', '-A', '', d)} ${@bb.utils.contains('DISTRO_FEATURES', 'xattr', '-X', '', d)}' OPKG_ARGS += "--force_postinstall --prefer-arch-to-version" OPKG_ARGS += "${@['', '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS") == "1"]}"
Hello Alexandre, this is a final patchest which works for me, even with reproducible builds, but additionally, with posix format of tar archives. I hope that this time it will work for autobuild as well. BR Piotr Od: Piotr Łobacz <p.lobacz@welotec.com> Wysłane: poniedziałek, 24 lipca 2023 22:46 Do: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> DW: Piotr Łobacz <p.lobacz@welotec.com> Temat: [OE-Core][PATCH v10][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr Extend OPKGBUILDCMD variable, with additional parameters, depending on target distro features, in order to support ACLs and xattr. With fix pushed to the opkg-devel: https://groups.google.com/g/opkg-devel/c/dYNHrLjDwg8 opkg-build is able to create tar archives with ACLs and xattr. Signed-off-by: Piotr Łobacz <p.lobacz@welotec.com> --- meta/classes-global/package_ipk.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)