| Message ID | 20231213085657.2552223-1-yoann.congal@smile.fr |
|---|---|
| State | Accepted, archived |
| Commit | 5dbfeea1c90c4dab9291d27da5a7ed1706e2ac2e |
| Headers | show |
| Series | [v3] strace: Disable bluetooth support by default | expand |
On Wed, 13 Dec 2023 at 09:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > The bluetooth support adds a bluez5 dependency (and,recursively, a lot > of other stuff). Disable it by default to avoid having to build all of > this when it is not needed. > > This decrease the number of tasks run for a core-image-minimal build by > ~1000 (-21%). > > To re-enable bluetooth support in strace, add "bluez" to strace > PACKAGECONFIG. For example, in local.conf: > PACKAGECONFIG:append:pn-strace = " bluez" > > Fixes [YOCTO #15323] I'm afraid I have to raise objections. First, this needs an explanation: what functionality in strace does this disable? Is that functionality important from the point of having bluetooth in DISTRO_FEATURES? Not respecting DISTRO_FEATURES sets a bad precedent, and should be more carefully justified and treated as an exception. Second, why is strace even needed in the context of core-image-minimal? It's not installed into the image, so I went and checked: util-linux-ptest needs mdadm mdadm-ptest needs strace. Which begs the question. Should we continue to enable ptest by default in poky? Should we create and use a ptest-less distro configuration? It does pull in a ton of extra stuff all over the place which lengthens the builds a lot. And the resulting ptest packages aren't even used until one explicitly requests one of the ptest images. As an example, I saw an oe-selftest-armhost yesterday, which ran nearly 17 hours: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2587 If you go and look at what tests in it took the longest time, you see: 2023-12-13 15:45:22,812 - oe-selftest - INFO - RESULTS - runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd: PASSED (32620.46s) 2023-12-13 15:45:22,794 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_override_db: PASSED (12789.45s) 2023-12-13 15:45:22,789 - oe-selftest - INFO - RESULTS - overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: PASSED (35205.79s) 2023-12-13 15:45:22,781 - oe-selftest - INFO - RESULTS - minidebuginfo.Minidebuginfo.test_minidebuginfo: PASSED (15395.76s) 2023-12-13 15:45:22,782 - oe-selftest - INFO - RESULTS - multiconfig.MultiConfig.test_multiconfig: PASSED (11098.06s) 2023-12-13 15:45:22,776 - oe-selftest - INFO - RESULTS - incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license: PASSED (24012.67s) 2023-12-13 15:45:22,775 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_mandb: PASSED (19394.74s) 2023-12-13 15:45:22,762 - oe-selftest - INFO - RESULTS - devtool.DevtoolExtractTests.test_devtool_build_image: PASSED (27341.75s) 2023-12-13 15:45:22,760 - oe-selftest - INFO - RESULTS - debuginfod.Debuginfod.test_debuginfod_qemu: PASSED (25784.11s) 2023-12-13 15:45:22,759 - oe-selftest - INFO - RESULTS - containerimage.ContainerImageTests.test_expected_files: PASSED (19453.84s) 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - buildoptions.SanityOptionsTest.test_options_warnqa_errorqa_switch: PASSED (10492.68s) 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - buildoptions.ToolchainOptions.test_toolchain_fortran: PASSED (11367.46s) 2023-12-13 15:45:22,743 - oe-selftest - INFO - RESULTS - baremetal.BaremetalTest.test_baremetal: PASSED (24196.04s) The same a-full selftest, but on a x86 host has these times, quicker than arm but still measured in hours: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6179/steps/15/logs/stdio What do most of these tests do? They do indeed build core-image-minimal (sometimes full-cmdline or some other images), and sometimes in multiple variants within a single test. But they never use ptest. And so we need to find a way to make it happen faster. I'm going to get some numbers, first without any changes, then with your proposed change, then with ptest dropped - this will take a bit of time, so I wanted to get the concerns written and sent first. Alex
On Thu, 2023-12-14 at 10:09 +0100, Alexander Kanavin wrote: > On Wed, 13 Dec 2023 at 09:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > > > The bluetooth support adds a bluez5 dependency (and,recursively, a lot > > of other stuff). Disable it by default to avoid having to build all of > > this when it is not needed. > > > > This decrease the number of tasks run for a core-image-minimal build by > > ~1000 (-21%). > > > > To re-enable bluetooth support in strace, add "bluez" to strace > > PACKAGECONFIG. For example, in local.conf: > > PACKAGECONFIG:append:pn-strace = " bluez" > > > > Fixes [YOCTO #15323] > > I'm afraid I have to raise objections. > > First, this needs an explanation: what functionality in strace does > this disable? Is that functionality important from the point of having > bluetooth in DISTRO_FEATURES? Not respecting DISTRO_FEATURES sets a > bad precedent, and should be more carefully justified and treated as > an exception. This was raised as a question on the call on Tuesday. I appreciate you weren't there and the commit message above does give the reasoning but let me elaborate. The bluez support in strace is basically for protocol decoding. This is not something most users of strace use, I personally can never remember using it. Yes, if you need it, it is extremely useful. If you know how to debug bluetooth wireless, you can probably work out how to turn on the packageconfig. The downside to having this enabled by default is a significant dependency chain increase (21%). Sometimes we need to think about the big picture and whether one single packageconfig is really worth the increased build cycles it places upon everyone by default. > Second, why is strace even needed in the context of > core-image-minimal? It's not installed into the image, so I went and > checked: > util-linux-ptest needs mdadm > mdadm-ptest needs strace. > > Which begs the question. Should we continue to enable ptest by default > in poky? Should we create and use a ptest-less distro configuration? > It does pull in a ton of extra stuff all over the place which > lengthens the builds a lot. And the resulting ptest packages aren't > even used until one explicitly requests one of the ptest images. This was also asked. We did use to have it off by default but then nearly every package upgrade broke ptest since the upgraders didn't remember to turn it on for testing. I decided on for poky, off for oe- core was a better compromise. > As an example, I saw an oe-selftest-armhost yesterday, which ran > nearly 17 hours: > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2587 > > If you go and look at what tests in it took the longest time, you see: > 2023-12-13 15:45:22,812 - oe-selftest - INFO - RESULTS - > runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd: PASSED > (32620.46s) > 2023-12-13 15:45:22,794 - oe-selftest - INFO - RESULTS - > prservice.BitbakePrTests.test_import_export_override_db: PASSED > (12789.45s) > 2023-12-13 15:45:22,789 - oe-selftest - INFO - RESULTS - > overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: > PASSED (35205.79s) > 2023-12-13 15:45:22,781 - oe-selftest - INFO - RESULTS - > minidebuginfo.Minidebuginfo.test_minidebuginfo: PASSED (15395.76s) > 2023-12-13 15:45:22,782 - oe-selftest - INFO - RESULTS - > multiconfig.MultiConfig.test_multiconfig: PASSED (11098.06s) > 2023-12-13 15:45:22,776 - oe-selftest - INFO - RESULTS - > incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license: > PASSED (24012.67s) > 2023-12-13 15:45:22,775 - oe-selftest - INFO - RESULTS - > imagefeatures.ImageFeatures.test_mandb: PASSED (19394.74s) > 2023-12-13 15:45:22,762 - oe-selftest - INFO - RESULTS - > devtool.DevtoolExtractTests.test_devtool_build_image: PASSED > (27341.75s) > 2023-12-13 15:45:22,760 - oe-selftest - INFO - RESULTS - > debuginfod.Debuginfod.test_debuginfod_qemu: PASSED (25784.11s) > 2023-12-13 15:45:22,759 - oe-selftest - INFO - RESULTS - > containerimage.ContainerImageTests.test_expected_files: PASSED > (19453.84s) > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > buildoptions.SanityOptionsTest.test_options_warnqa_errorqa_switch: > PASSED (10492.68s) > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > buildoptions.ToolchainOptions.test_toolchain_fortran: PASSED > (11367.46s) > 2023-12-13 15:45:22,743 - oe-selftest - INFO - RESULTS - > baremetal.BaremetalTest.test_baremetal: PASSED (24196.04s) > > The same a-full selftest, but on a x86 host has these times, quicker > than arm but still measured in hours: > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6179/steps/15/logs/stdio > > What do most of these tests do? They do indeed build > core-image-minimal (sometimes full-cmdline or some other images), and > sometimes in multiple variants within a single test. But they never > use ptest. And so we need to find a way to make it happen faster. Once a given build has run through the system, things do run much faster but this is basically the performance issue I've been mentioning in the weekly status reports. Even the above packageconfig change in this patch would actually speed a lot of these up, but you're objecting to that. The tests themselves are actually quite valuable as we're way beyond the point I can work out which patch will break which features. Some of the tests could undoubtedly be improved. If we disable the ptests for the selftests, we run the risk of not reusing sstate so a better question might be, why are all the tests not reusing sstate more efficiently? > I'm going to get some numbers, first without any changes, then with > your proposed change, then with ptest dropped - this will take a bit > of time, so I wanted to get the concerns written and sent first. I don't think dropping the ptests is the right approach, we should likely focus on sstate reuse? I am also pretty in favour of this patch. Cheers, Richard
On Thu, 14 Dec 2023 at 10:50, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > Once a given build has run through the system, things do run much > faster but this is basically the performance issue I've been mentioning > in the weekly status reports. Even the above packageconfig change in > this patch would actually speed a lot of these up, but you're objecting > to that. I no longer object :) the above explanations are fine with me. I did run the benchmarks though, so for the sake of having a fuller picture: default poky, core-image-minimal: 5134 tasks real 26m26.874s poky with this patch, core-image-minimal 4051 tasks real 25m53.513s poky without ptests, core-image-minimal 2997 tasks real 21m23.684s > The tests themselves are actually quite valuable as we're way beyond > the point I can work out which patch will break which features. Some of > the tests could undoubtedly be improved. If we disable the ptests for > the selftests, we run the risk of not reusing sstate so a better > question might be, why are all the tests not reusing sstate more > efficiently? Looking at that a-full again, it was abelloni's master-next which had something that triggered a mass rebuild, e.g. qemux86-64: Sstate summary: Wanted 8139 Local 2083 Mirrors 0 Missed 6056 Current 0 (25% match, 0% complete) (and that took over 5 hours to fulfil: https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/8224 ) My theory is that when all of the builders start at the same time in that situation, they will all rebuild the same things, and become very slow and overloaded. Including image builds in selftest as well, as it's parallelized in itself. I have a proposal: splitting a-full into stages. First, run the build steps from qemux86-64 and qemuarm64 (and arm64 on armhost), which would serve two purposes: - pre-populate sstate for all the the builders that will run in the next stage, accelerating them. It's possible some of the builders would still do full rebuilds, but then it becomes possible to look separately at why they can't reuse the sstate from the first stage. - do a quick, lightweight smoke check on the changes under test: if something breaks the build really badly, it will be caught there and then, the build will stop, and unneeded AB overload will be avoided. Which will in turn speed up everything else that's running at the same time. Alex
On Thu, 2023-12-14 at 13:03 +0100, Alexander Kanavin wrote: > On Thu, 14 Dec 2023 at 10:50, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > Once a given build has run through the system, things do run much > > faster but this is basically the performance issue I've been mentioning > > in the weekly status reports. Even the above packageconfig change in > > this patch would actually speed a lot of these up, but you're objecting > > to that. > > I no longer object :) the above explanations are fine with me. I did > run the benchmarks though, so for the sake of having a fuller picture: > > default poky, core-image-minimal: > 5134 tasks > real 26m26.874s > > poky with this patch, core-image-minimal > 4051 tasks > real 25m53.513s > > > poky without ptests, core-image-minimal > 2997 tasks > real 21m23.684s Thanks, that is useful data to have, > > > The tests themselves are actually quite valuable as we're way beyond > > the point I can work out which patch will break which features. Some of > > the tests could undoubtedly be improved. If we disable the ptests for > > the selftests, we run the risk of not reusing sstate so a better > > question might be, why are all the tests not reusing sstate more > > efficiently? > > Looking at that a-full again, it was abelloni's master-next which had > something that triggered a mass rebuild, e.g. qemux86-64: > > Sstate summary: Wanted 8139 Local 2083 Mirrors 0 Missed 6056 Current 0 > (25% match, 0% complete) > (and that took over 5 hours to fulfil: > https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/8224 > ) > > My theory is that when all of the builders start at the same time in > that situation, they will all rebuild the same things, and become very > slow and overloaded. Including image builds in selftest as well, as > it's parallelized in itself. Maybe, maybe not. By that theory if you run the selftests in series they should be quite fast as they reuse previously built artefacts. Experience suggests there are quite a few slow on the first run. > I have a proposal: splitting a-full into stages. First, run the build > steps from qemux86-64 and qemuarm64 (and arm64 on armhost), which > would serve two purposes: > > - pre-populate sstate for all the the builders that will run in the > next stage, accelerating them. It's possible some of the builders > would still do full rebuilds, but then it becomes possible to look > separately at why they can't reuse the sstate from the first stage. > > - do a quick, lightweight smoke check on the changes under test: if > something breaks the build really badly, it will be caught there and > then, the build will stop, and unneeded AB overload will be avoided. > Which will in turn speed up everything else that's running at the same > time. I've wondered about this. The "prebuild" ends up quite slow as it would involve rust for example and when I last tried this, it ended up being a pain and just increased overall testing times even of the overall load on the AB might have been lower. So I'm really not sure. I suspect it wouldn't help the pain points as much as you think as there are other sstate reuse issues at play too. Cheers, Richard
On Thu, 14 Dec 2023 at 13:11, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > I've wondered about this. The "prebuild" ends up quite slow as it would > involve rust for example and when I last tried this, it ended up being > a pain and just increased overall testing times even of the overall > load on the AB might have been lower. So I'm really not sure. I suspect > it wouldn't help the pain points as much as you think as there are > other sstate reuse issues at play too. But is it worth experimenting with it anyway, to collect some data? The more we know, the more we can rule out. Alex
On Thu, 2023-12-14 at 13:17 +0100, Alexander Kanavin wrote: > On Thu, 14 Dec 2023 at 13:11, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > I've wondered about this. The "prebuild" ends up quite slow as it would > > involve rust for example and when I last tried this, it ended up being > > a pain and just increased overall testing times even of the overall > > load on the AB might have been lower. So I'm really not sure. I suspect > > it wouldn't help the pain points as much as you think as there are > > other sstate reuse issues at play too. > > But is it worth experimenting with it anyway, to collect some data? > The more we know, the more we can rule out. I think it would have a huge impact on the throughput of patches :( An hour or two before the majority of the tests even start. Cheers, Richard
On Thu, 14 Dec 2023 at 13:30, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > I think it would have a huge impact on the throughput of patches :( An > hour or two before the majority of the tests even start. Then the only remaining idea I have is adding more hardware. So that there's enough of it to rarely if ever have a situation where several builds run on the same worker. Alex
On Thu, 2023-12-14 at 13:34 +0100, Alexander Kanavin wrote: > On Thu, 14 Dec 2023 at 13:30, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > I think it would have a huge impact on the throughput of patches :( An > > hour or two before the majority of the tests even start. > > Then the only remaining idea I have is adding more hardware. So that > there's enough of it to rarely if ever have a situation where several > builds run on the same worker. We are in the process of trying to work out which direction the infrastructure will take as we are due a hardware refresh FWIW. I do still also think we need to take a look at the sstate matches/mismatches... Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quinta, 14/12/2023 à(s) 12:37: > On Thu, 2023-12-14 at 13:34 +0100, Alexander Kanavin wrote: > > On Thu, 14 Dec 2023 at 13:30, Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > I think it would have a huge impact on the throughput of patches :( An > > > hour or two before the majority of the tests even start. > > > > Then the only remaining idea I have is adding more hardware. So that > > there's enough of it to rarely if ever have a situation where several > > builds run on the same worker. > > We are in the process of trying to work out which direction the > infrastructure will take as we are due a hardware refresh FWIW. > > I do still also think we need to take a look at the sstate > matches/mismatches... > I've thought a few times that having an argument on bitbake or something else to just execute the native targets would be very useful. With something like that we can split the builds for multiple machines in two steps building first the common part, In this first step we have some of the havey like clang-native rust-native rust-llvm-native. To conclude the first build would only run and build the native targets and the following builds will get all from the sstate. Jose > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#192364): > https://lists.openembedded.org/g/openembedded-core/message/192364 > Mute This Topic: https://lists.openembedded.org/mt/103146402/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Wed, 2023-12-13 at 09:56 +0100, Yoann Congal wrote: > The bluetooth support adds a bluez5 dependency (and,recursively, a lot > of other stuff). Disable it by default to avoid having to build all of > this when it is not needed. > > This decrease the number of tasks run for a core-image-minimal build by > ~1000 (-21%). > > To re-enable bluetooth support in strace, add "bluez" to strace > PACKAGECONFIG. For example, in local.conf: > PACKAGECONFIG:append:pn-strace = " bluez" > > Fixes [YOCTO #15323] > > Signed-off-by: Yoann Congal <yoann.congal@smile.fr> > Suggested-by: Ross Burton <ross.burton@arm.com> > --- > v1->v2: > * Fixed local.conf example (thanks Tim) > * Added Suggested-by: Ross (thanks Ross) > * Removed an extra empty line > v2->v3: > * Added missing leading space in local.conf suggestion (thanks Martin) > --- > meta/recipes-devtools/strace/strace_6.6.bb | 4 ---- > 1 file changed, 4 deletions(-) > > diff --git a/meta/recipes-devtools/strace/strace_6.6.bb b/meta/recipes-devtools/strace/strace_6.6.bb > index a3de7941cf..eeffe477b3 100644 > --- a/meta/recipes-devtools/strace/strace_6.6.bb > +++ b/meta/recipes-devtools/strace/strace_6.6.bb > @@ -22,10 +22,6 @@ inherit autotools ptest > # Not yet ported to rv32 > COMPATIBLE_HOST:riscv32 = "null" > > -PACKAGECONFIG:class-target ??= "\ > - ${@bb.utils.contains('DISTRO_FEATURES', 'bluetooth', 'bluez', '', d)} \ > -" > - > PACKAGECONFIG[bluez] = "ac_cv_header_bluetooth_bluetooth_h=yes,ac_cv_header_bluetooth_bluetooth_h=no,bluez5" > PACKAGECONFIG[libunwind] = "--with-libunwind,--without-libunwind,libunwind" Thanks. I've tweaked the version I queued in master-next to leave a comment in the recipe about why we've done this so someone in a few years time understands. Cheers, Richard
On Thu, Dec 14, 2023 at 1:50 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Thu, 2023-12-14 at 10:09 +0100, Alexander Kanavin wrote: > > On Wed, 13 Dec 2023 at 09:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > > > > > The bluetooth support adds a bluez5 dependency (and,recursively, a lot > > > of other stuff). Disable it by default to avoid having to build all of > > > this when it is not needed. > > > > > > This decrease the number of tasks run for a core-image-minimal build by > > > ~1000 (-21%). > > > > > > To re-enable bluetooth support in strace, add "bluez" to strace > > > PACKAGECONFIG. For example, in local.conf: > > > PACKAGECONFIG:append:pn-strace = " bluez" > > > > > > Fixes [YOCTO #15323] > > > > I'm afraid I have to raise objections. > > > > First, this needs an explanation: what functionality in strace does > > this disable? Is that functionality important from the point of having > > bluetooth in DISTRO_FEATURES? Not respecting DISTRO_FEATURES sets a > > bad precedent, and should be more carefully justified and treated as > > an exception. > > This was raised as a question on the call on Tuesday. I appreciate you > weren't there and the commit message above does give the reasoning but > let me elaborate. > > The bluez support in strace is basically for protocol decoding. This is > not something most users of strace use, I personally can never remember > using it. Yes, if you need it, it is extremely useful. If you know how > to debug bluetooth wireless, you can probably work out how to turn on > the packageconfig. For more context see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=882413 > > The downside to having this enabled by default is a significant > dependency chain increase (21%). Sometimes we need to think about the > big picture and whether one single packageconfig is really worth the > increased build cycles it places upon everyone by default. > > > Second, why is strace even needed in the context of > > core-image-minimal? It's not installed into the image, so I went and > > checked: > > util-linux-ptest needs mdadm > > mdadm-ptest needs strace. > > > > Which begs the question. Should we continue to enable ptest by default > > in poky? Should we create and use a ptest-less distro configuration? > > It does pull in a ton of extra stuff all over the place which > > lengthens the builds a lot. And the resulting ptest packages aren't > > even used until one explicitly requests one of the ptest images. > > This was also asked. We did use to have it off by default but then > nearly every package upgrade broke ptest since the upgraders didn't > remember to turn it on for testing. I decided on for poky, off for oe- > core was a better compromise. > > > As an example, I saw an oe-selftest-armhost yesterday, which ran > > nearly 17 hours: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2587 > > > > If you go and look at what tests in it took the longest time, you see: > > 2023-12-13 15:45:22,812 - oe-selftest - INFO - RESULTS - > > runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd: PASSED > > (32620.46s) > > 2023-12-13 15:45:22,794 - oe-selftest - INFO - RESULTS - > > prservice.BitbakePrTests.test_import_export_override_db: PASSED > > (12789.45s) > > 2023-12-13 15:45:22,789 - oe-selftest - INFO - RESULTS - > > overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: > > PASSED (35205.79s) > > 2023-12-13 15:45:22,781 - oe-selftest - INFO - RESULTS - > > minidebuginfo.Minidebuginfo.test_minidebuginfo: PASSED (15395.76s) > > 2023-12-13 15:45:22,782 - oe-selftest - INFO - RESULTS - > > multiconfig.MultiConfig.test_multiconfig: PASSED (11098.06s) > > 2023-12-13 15:45:22,776 - oe-selftest - INFO - RESULTS - > > incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license: > > PASSED (24012.67s) > > 2023-12-13 15:45:22,775 - oe-selftest - INFO - RESULTS - > > imagefeatures.ImageFeatures.test_mandb: PASSED (19394.74s) > > 2023-12-13 15:45:22,762 - oe-selftest - INFO - RESULTS - > > devtool.DevtoolExtractTests.test_devtool_build_image: PASSED > > (27341.75s) > > 2023-12-13 15:45:22,760 - oe-selftest - INFO - RESULTS - > > debuginfod.Debuginfod.test_debuginfod_qemu: PASSED (25784.11s) > > 2023-12-13 15:45:22,759 - oe-selftest - INFO - RESULTS - > > containerimage.ContainerImageTests.test_expected_files: PASSED > > (19453.84s) > > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > > buildoptions.SanityOptionsTest.test_options_warnqa_errorqa_switch: > > PASSED (10492.68s) > > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > > buildoptions.ToolchainOptions.test_toolchain_fortran: PASSED > > (11367.46s) > > 2023-12-13 15:45:22,743 - oe-selftest - INFO - RESULTS - > > baremetal.BaremetalTest.test_baremetal: PASSED (24196.04s) > > > > The same a-full selftest, but on a x86 host has these times, quicker > > than arm but still measured in hours: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6179/steps/15/logs/stdio > > > > What do most of these tests do? They do indeed build > > core-image-minimal (sometimes full-cmdline or some other images), and > > sometimes in multiple variants within a single test. But they never > > use ptest. And so we need to find a way to make it happen faster. > > Once a given build has run through the system, things do run much > faster but this is basically the performance issue I've been mentioning > in the weekly status reports. Even the above packageconfig change in > this patch would actually speed a lot of these up, but you're objecting > to that. > > The tests themselves are actually quite valuable as we're way beyond > the point I can work out which patch will break which features. Some of > the tests could undoubtedly be improved. If we disable the ptests for > the selftests, we run the risk of not reusing sstate so a better > question might be, why are all the tests not reusing sstate more > efficiently? > > > I'm going to get some numbers, first without any changes, then with > > your proposed change, then with ptest dropped - this will take a bit > > of time, so I wanted to get the concerns written and sent first. > > I don't think dropping the ptests is the right approach, we should > likely focus on sstate reuse? > > I am also pretty in favour of this patch. > > Cheers, > > Richard > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#192343): https://lists.openembedded.org/g/openembedded-core/message/192343 > Mute This Topic: https://lists.openembedded.org/mt/103146402/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Thu, Dec 14, 2023 at 1:10 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > On Wed, 13 Dec 2023 at 09:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > > > The bluetooth support adds a bluez5 dependency (and,recursively, a lot > > of other stuff). Disable it by default to avoid having to build all of > > this when it is not needed. > > > > This decrease the number of tasks run for a core-image-minimal build by > > ~1000 (-21%). > > > > To re-enable bluetooth support in strace, add "bluez" to strace > > PACKAGECONFIG. For example, in local.conf: > > PACKAGECONFIG:append:pn-strace = " bluez" > > > > Fixes [YOCTO #15323] > > I'm afraid I have to raise objections. > > First, this needs an explanation: what functionality in strace does > this disable? Is that functionality important from the point of having > bluetooth in DISTRO_FEATURES? Not respecting DISTRO_FEATURES sets a > bad precedent, and should be more carefully justified and treated as > an exception. > > Second, why is strace even needed in the context of > core-image-minimal? It's not installed into the image, so I went and > checked: > util-linux-ptest needs mdadm > mdadm-ptest needs strace. I looked briefly into util-linux pertaining to mdadm needs in tests, there are 4 tests needing it. md-raid0-whole, md-raid1-part, md-raid1-whole, align-512-4K-md and all of them are marked as TS_KNOWN_FAIL="yes" See https://github.com/util-linux/util-linux/commit/7519c3edab120b14623931d5ddb16fdc6e7cad5d I think we can skip running these tests as well and safely avoid depending upon mdadm for util-linux ptests which can break the depchain as well. > > Which begs the question. Should we continue to enable ptest by default > in poky? Should we create and use a ptest-less distro configuration? > It does pull in a ton of extra stuff all over the place which > lengthens the builds a lot. And the resulting ptest packages aren't > even used until one explicitly requests one of the ptest images. > > As an example, I saw an oe-selftest-armhost yesterday, which ran > nearly 17 hours: > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2587 > > If you go and look at what tests in it took the longest time, you see: > 2023-12-13 15:45:22,812 - oe-selftest - INFO - RESULTS - > runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd: PASSED > (32620.46s) > 2023-12-13 15:45:22,794 - oe-selftest - INFO - RESULTS - > prservice.BitbakePrTests.test_import_export_override_db: PASSED > (12789.45s) > 2023-12-13 15:45:22,789 - oe-selftest - INFO - RESULTS - > overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: > PASSED (35205.79s) > 2023-12-13 15:45:22,781 - oe-selftest - INFO - RESULTS - > minidebuginfo.Minidebuginfo.test_minidebuginfo: PASSED (15395.76s) > 2023-12-13 15:45:22,782 - oe-selftest - INFO - RESULTS - > multiconfig.MultiConfig.test_multiconfig: PASSED (11098.06s) > 2023-12-13 15:45:22,776 - oe-selftest - INFO - RESULTS - > incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license: > PASSED (24012.67s) > 2023-12-13 15:45:22,775 - oe-selftest - INFO - RESULTS - > imagefeatures.ImageFeatures.test_mandb: PASSED (19394.74s) > 2023-12-13 15:45:22,762 - oe-selftest - INFO - RESULTS - > devtool.DevtoolExtractTests.test_devtool_build_image: PASSED > (27341.75s) > 2023-12-13 15:45:22,760 - oe-selftest - INFO - RESULTS - > debuginfod.Debuginfod.test_debuginfod_qemu: PASSED (25784.11s) > 2023-12-13 15:45:22,759 - oe-selftest - INFO - RESULTS - > containerimage.ContainerImageTests.test_expected_files: PASSED > (19453.84s) > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > buildoptions.SanityOptionsTest.test_options_warnqa_errorqa_switch: > PASSED (10492.68s) > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > buildoptions.ToolchainOptions.test_toolchain_fortran: PASSED > (11367.46s) > 2023-12-13 15:45:22,743 - oe-selftest - INFO - RESULTS - > baremetal.BaremetalTest.test_baremetal: PASSED (24196.04s) > > The same a-full selftest, but on a x86 host has these times, quicker > than arm but still measured in hours: > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6179/steps/15/logs/stdio > > What do most of these tests do? They do indeed build > core-image-minimal (sometimes full-cmdline or some other images), and > sometimes in multiple variants within a single test. But they never > use ptest. And so we need to find a way to make it happen faster. > > I'm going to get some numbers, first without any changes, then with > your proposed change, then with ptest dropped - this will take a bit > of time, so I wanted to get the concerns written and sent first. > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#192339): https://lists.openembedded.org/g/openembedded-core/message/192339 > Mute This Topic: https://lists.openembedded.org/mt/103146402/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 14/12/2023 10:53:05-0800, Khem Raj wrote: > On Thu, Dec 14, 2023 at 1:10 AM Alexander Kanavin > <alex.kanavin@gmail.com> wrote: > > > > On Wed, 13 Dec 2023 at 09:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > > > > > The bluetooth support adds a bluez5 dependency (and,recursively, a lot > > > of other stuff). Disable it by default to avoid having to build all of > > > this when it is not needed. > > > > > > This decrease the number of tasks run for a core-image-minimal build by > > > ~1000 (-21%). > > > > > > To re-enable bluetooth support in strace, add "bluez" to strace > > > PACKAGECONFIG. For example, in local.conf: > > > PACKAGECONFIG:append:pn-strace = " bluez" > > > > > > Fixes [YOCTO #15323] > > > > I'm afraid I have to raise objections. > > > > First, this needs an explanation: what functionality in strace does > > this disable? Is that functionality important from the point of having > > bluetooth in DISTRO_FEATURES? Not respecting DISTRO_FEATURES sets a > > bad precedent, and should be more carefully justified and treated as > > an exception. > > > > Second, why is strace even needed in the context of > > core-image-minimal? It's not installed into the image, so I went and > > checked: > > util-linux-ptest needs mdadm > > mdadm-ptest needs strace. > > I looked briefly into util-linux pertaining to mdadm needs in tests, there > are 4 tests needing it. > md-raid0-whole, md-raid1-part, md-raid1-whole, align-512-4K-md > > and all of them are marked as > TS_KNOWN_FAIL="yes" > > See https://github.com/util-linux/util-linux/commit/7519c3edab120b14623931d5ddb16fdc6e7cad5d > > I think we can skip running these tests as well and safely avoid > depending upon mdadm for util-linux ptests which can break the depchain as well. > I think dropping the mdam ptests was the plan seeing the amount of breakage this does on the AB. > > > > Which begs the question. Should we continue to enable ptest by default > > in poky? Should we create and use a ptest-less distro configuration? > > It does pull in a ton of extra stuff all over the place which > > lengthens the builds a lot. And the resulting ptest packages aren't > > even used until one explicitly requests one of the ptest images. > > > > As an example, I saw an oe-selftest-armhost yesterday, which ran > > nearly 17 hours: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2587 > > > > If you go and look at what tests in it took the longest time, you see: > > 2023-12-13 15:45:22,812 - oe-selftest - INFO - RESULTS - > > runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd: PASSED > > (32620.46s) > > 2023-12-13 15:45:22,794 - oe-selftest - INFO - RESULTS - > > prservice.BitbakePrTests.test_import_export_override_db: PASSED > > (12789.45s) > > 2023-12-13 15:45:22,789 - oe-selftest - INFO - RESULTS - > > overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: > > PASSED (35205.79s) > > 2023-12-13 15:45:22,781 - oe-selftest - INFO - RESULTS - > > minidebuginfo.Minidebuginfo.test_minidebuginfo: PASSED (15395.76s) > > 2023-12-13 15:45:22,782 - oe-selftest - INFO - RESULTS - > > multiconfig.MultiConfig.test_multiconfig: PASSED (11098.06s) > > 2023-12-13 15:45:22,776 - oe-selftest - INFO - RESULTS - > > incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license: > > PASSED (24012.67s) > > 2023-12-13 15:45:22,775 - oe-selftest - INFO - RESULTS - > > imagefeatures.ImageFeatures.test_mandb: PASSED (19394.74s) > > 2023-12-13 15:45:22,762 - oe-selftest - INFO - RESULTS - > > devtool.DevtoolExtractTests.test_devtool_build_image: PASSED > > (27341.75s) > > 2023-12-13 15:45:22,760 - oe-selftest - INFO - RESULTS - > > debuginfod.Debuginfod.test_debuginfod_qemu: PASSED (25784.11s) > > 2023-12-13 15:45:22,759 - oe-selftest - INFO - RESULTS - > > containerimage.ContainerImageTests.test_expected_files: PASSED > > (19453.84s) > > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > > buildoptions.SanityOptionsTest.test_options_warnqa_errorqa_switch: > > PASSED (10492.68s) > > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > > buildoptions.ToolchainOptions.test_toolchain_fortran: PASSED > > (11367.46s) > > 2023-12-13 15:45:22,743 - oe-selftest - INFO - RESULTS - > > baremetal.BaremetalTest.test_baremetal: PASSED (24196.04s) > > > > The same a-full selftest, but on a x86 host has these times, quicker > > than arm but still measured in hours: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6179/steps/15/logs/stdio > > > > What do most of these tests do? They do indeed build > > core-image-minimal (sometimes full-cmdline or some other images), and > > sometimes in multiple variants within a single test. But they never > > use ptest. And so we need to find a way to make it happen faster. > > > > I'm going to get some numbers, first without any changes, then with > > your proposed change, then with ptest dropped - this will take a bit > > of time, so I wanted to get the concerns written and sent first. > > > > Alex > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#192339): https://lists.openembedded.org/g/openembedded-core/message/192339 > > Mute This Topic: https://lists.openembedded.org/mt/103146402/1997914 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > >
On Thu, Dec 14, 2023 at 11:33 AM Alexandre Belloni <alexandre.belloni@bootlin.com> wrote: > > On 14/12/2023 10:53:05-0800, Khem Raj wrote: > > On Thu, Dec 14, 2023 at 1:10 AM Alexander Kanavin > > <alex.kanavin@gmail.com> wrote: > > > > > > On Wed, 13 Dec 2023 at 09:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > > > > > > > The bluetooth support adds a bluez5 dependency (and,recursively, a lot > > > > of other stuff). Disable it by default to avoid having to build all of > > > > this when it is not needed. > > > > > > > > This decrease the number of tasks run for a core-image-minimal build by > > > > ~1000 (-21%). > > > > > > > > To re-enable bluetooth support in strace, add "bluez" to strace > > > > PACKAGECONFIG. For example, in local.conf: > > > > PACKAGECONFIG:append:pn-strace = " bluez" > > > > > > > > Fixes [YOCTO #15323] > > > > > > I'm afraid I have to raise objections. > > > > > > First, this needs an explanation: what functionality in strace does > > > this disable? Is that functionality important from the point of having > > > bluetooth in DISTRO_FEATURES? Not respecting DISTRO_FEATURES sets a > > > bad precedent, and should be more carefully justified and treated as > > > an exception. > > > > > > Second, why is strace even needed in the context of > > > core-image-minimal? It's not installed into the image, so I went and > > > checked: > > > util-linux-ptest needs mdadm > > > mdadm-ptest needs strace. > > > > I looked briefly into util-linux pertaining to mdadm needs in tests, there > > are 4 tests needing it. > > md-raid0-whole, md-raid1-part, md-raid1-whole, align-512-4K-md > > > > and all of them are marked as > > TS_KNOWN_FAIL="yes" > > > > See https://github.com/util-linux/util-linux/commit/7519c3edab120b14623931d5ddb16fdc6e7cad5d > > > > I think we can skip running these tests as well and safely avoid > > depending upon mdadm for util-linux ptests which can break the depchain as well. > > > > I think dropping the mdam ptests was the plan seeing the amount of > breakage this does on the AB. Good deal, I also see them happening on musl so I went ahead and cooked a patch and posted it on ml. Musl util-linux ptests have 4 less fails now. > > > > > > > Which begs the question. Should we continue to enable ptest by default > > > in poky? Should we create and use a ptest-less distro configuration? > > > It does pull in a ton of extra stuff all over the place which > > > lengthens the builds a lot. And the resulting ptest packages aren't > > > even used until one explicitly requests one of the ptest images. > > > > > > As an example, I saw an oe-selftest-armhost yesterday, which ran > > > nearly 17 hours: > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2587 > > > > > > If you go and look at what tests in it took the longest time, you see: > > > 2023-12-13 15:45:22,812 - oe-selftest - INFO - RESULTS - > > > runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd: PASSED > > > (32620.46s) > > > 2023-12-13 15:45:22,794 - oe-selftest - INFO - RESULTS - > > > prservice.BitbakePrTests.test_import_export_override_db: PASSED > > > (12789.45s) > > > 2023-12-13 15:45:22,789 - oe-selftest - INFO - RESULTS - > > > overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: > > > PASSED (35205.79s) > > > 2023-12-13 15:45:22,781 - oe-selftest - INFO - RESULTS - > > > minidebuginfo.Minidebuginfo.test_minidebuginfo: PASSED (15395.76s) > > > 2023-12-13 15:45:22,782 - oe-selftest - INFO - RESULTS - > > > multiconfig.MultiConfig.test_multiconfig: PASSED (11098.06s) > > > 2023-12-13 15:45:22,776 - oe-selftest - INFO - RESULTS - > > > incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license: > > > PASSED (24012.67s) > > > 2023-12-13 15:45:22,775 - oe-selftest - INFO - RESULTS - > > > imagefeatures.ImageFeatures.test_mandb: PASSED (19394.74s) > > > 2023-12-13 15:45:22,762 - oe-selftest - INFO - RESULTS - > > > devtool.DevtoolExtractTests.test_devtool_build_image: PASSED > > > (27341.75s) > > > 2023-12-13 15:45:22,760 - oe-selftest - INFO - RESULTS - > > > debuginfod.Debuginfod.test_debuginfod_qemu: PASSED (25784.11s) > > > 2023-12-13 15:45:22,759 - oe-selftest - INFO - RESULTS - > > > containerimage.ContainerImageTests.test_expected_files: PASSED > > > (19453.84s) > > > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > > > buildoptions.SanityOptionsTest.test_options_warnqa_errorqa_switch: > > > PASSED (10492.68s) > > > 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - > > > buildoptions.ToolchainOptions.test_toolchain_fortran: PASSED > > > (11367.46s) > > > 2023-12-13 15:45:22,743 - oe-selftest - INFO - RESULTS - > > > baremetal.BaremetalTest.test_baremetal: PASSED (24196.04s) > > > > > > The same a-full selftest, but on a x86 host has these times, quicker > > > than arm but still measured in hours: > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6179/steps/15/logs/stdio > > > > > > What do most of these tests do? They do indeed build > > > core-image-minimal (sometimes full-cmdline or some other images), and > > > sometimes in multiple variants within a single test. But they never > > > use ptest. And so we need to find a way to make it happen faster. > > > > > > I'm going to get some numbers, first without any changes, then with > > > your proposed change, then with ptest dropped - this will take a bit > > > of time, so I wanted to get the concerns written and sent first. > > > > > > Alex > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > Links: You receive all messages sent to this group. > > > View/Reply Online (#192339): https://lists.openembedded.org/g/openembedded-core/message/192339 > > > Mute This Topic: https://lists.openembedded.org/mt/103146402/1997914 > > > Group Owner: openembedded-core+owner@lists.openembedded.org > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
Le 14/12/2023 à 20:36, Khem Raj a écrit : > On Thu, Dec 14, 2023 at 11:33 AM Alexandre Belloni > <alexandre.belloni@bootlin.com> wrote: >> >> On 14/12/2023 10:53:05-0800, Khem Raj wrote: >>> On Thu, Dec 14, 2023 at 1:10 AM Alexander Kanavin >>> <alex.kanavin@gmail.com> wrote: >>>> >>>> On Wed, 13 Dec 2023 at 09:57, Yoann Congal <yoann.congal@smile.fr> wrote: >>>>> >>>>> The bluetooth support adds a bluez5 dependency (and,recursively, a lot >>>>> of other stuff). Disable it by default to avoid having to build all of >>>>> this when it is not needed. >>>>> >>>>> This decrease the number of tasks run for a core-image-minimal build by >>>>> ~1000 (-21%). >>>>> >>>>> To re-enable bluetooth support in strace, add "bluez" to strace >>>>> PACKAGECONFIG. For example, in local.conf: >>>>> PACKAGECONFIG:append:pn-strace = " bluez" >>>>> >>>>> Fixes [YOCTO #15323] >>>> >>>> I'm afraid I have to raise objections. >>>> >>>> First, this needs an explanation: what functionality in strace does >>>> this disable? Is that functionality important from the point of having >>>> bluetooth in DISTRO_FEATURES? Not respecting DISTRO_FEATURES sets a >>>> bad precedent, and should be more carefully justified and treated as >>>> an exception. >>>> >>>> Second, why is strace even needed in the context of >>>> core-image-minimal? It's not installed into the image, so I went and >>>> checked: >>>> util-linux-ptest needs mdadm >>>> mdadm-ptest needs strace. >>> >>> I looked briefly into util-linux pertaining to mdadm needs in tests, there >>> are 4 tests needing it. >>> md-raid0-whole, md-raid1-part, md-raid1-whole, align-512-4K-md >>> >>> and all of them are marked as >>> TS_KNOWN_FAIL="yes" >>> >>> See https://github.com/util-linux/util-linux/commit/7519c3edab120b14623931d5ddb16fdc6e7cad5d >>> >>> I think we can skip running these tests as well and safely avoid >>> depending upon mdadm for util-linux ptests which can break the depchain as well. >>> >> >> I think dropping the mdam ptests was the plan seeing the amount of >> breakage this does on the AB. > > Good deal, I also see them happening on musl so I went ahead and cooked a patch > and posted it on ml. Musl util-linux ptests have 4 less fails now. FYI, since Khem's patch ([OE-core] [PATCH] util-linux: Delete md-raid tests) also prevent bluez to be built in a default core-image-minimal build, we can skip this one (strace: Disable bluetooth support by default) and still fix [YOCTO #15323]. >> >>>> >>>> Which begs the question. Should we continue to enable ptest by default >>>> in poky? Should we create and use a ptest-less distro configuration? >>>> It does pull in a ton of extra stuff all over the place which >>>> lengthens the builds a lot. And the resulting ptest packages aren't >>>> even used until one explicitly requests one of the ptest images. >>>> >>>> As an example, I saw an oe-selftest-armhost yesterday, which ran >>>> nearly 17 hours: >>>> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2587 >>>> >>>> If you go and look at what tests in it took the longest time, you see: >>>> 2023-12-13 15:45:22,812 - oe-selftest - INFO - RESULTS - >>>> runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd: PASSED >>>> (32620.46s) >>>> 2023-12-13 15:45:22,794 - oe-selftest - INFO - RESULTS - >>>> prservice.BitbakePrTests.test_import_export_override_db: PASSED >>>> (12789.45s) >>>> 2023-12-13 15:45:22,789 - oe-selftest - INFO - RESULTS - >>>> overlayfs.OverlayFSEtcRunTimeTests.test_all_required_variables_set: >>>> PASSED (35205.79s) >>>> 2023-12-13 15:45:22,781 - oe-selftest - INFO - RESULTS - >>>> minidebuginfo.Minidebuginfo.test_minidebuginfo: PASSED (15395.76s) >>>> 2023-12-13 15:45:22,782 - oe-selftest - INFO - RESULTS - >>>> multiconfig.MultiConfig.test_multiconfig: PASSED (11098.06s) >>>> 2023-12-13 15:45:22,776 - oe-selftest - INFO - RESULTS - >>>> incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license: >>>> PASSED (24012.67s) >>>> 2023-12-13 15:45:22,775 - oe-selftest - INFO - RESULTS - >>>> imagefeatures.ImageFeatures.test_mandb: PASSED (19394.74s) >>>> 2023-12-13 15:45:22,762 - oe-selftest - INFO - RESULTS - >>>> devtool.DevtoolExtractTests.test_devtool_build_image: PASSED >>>> (27341.75s) >>>> 2023-12-13 15:45:22,760 - oe-selftest - INFO - RESULTS - >>>> debuginfod.Debuginfod.test_debuginfod_qemu: PASSED (25784.11s) >>>> 2023-12-13 15:45:22,759 - oe-selftest - INFO - RESULTS - >>>> containerimage.ContainerImageTests.test_expected_files: PASSED >>>> (19453.84s) >>>> 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - >>>> buildoptions.SanityOptionsTest.test_options_warnqa_errorqa_switch: >>>> PASSED (10492.68s) >>>> 2023-12-13 15:45:22,758 - oe-selftest - INFO - RESULTS - >>>> buildoptions.ToolchainOptions.test_toolchain_fortran: PASSED >>>> (11367.46s) >>>> 2023-12-13 15:45:22,743 - oe-selftest - INFO - RESULTS - >>>> baremetal.BaremetalTest.test_baremetal: PASSED (24196.04s) >>>> >>>> The same a-full selftest, but on a x86 host has these times, quicker >>>> than arm but still measured in hours: >>>> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6179/steps/15/logs/stdio >>>> >>>> What do most of these tests do? They do indeed build >>>> core-image-minimal (sometimes full-cmdline or some other images), and >>>> sometimes in multiple variants within a single test. But they never >>>> use ptest. And so we need to find a way to make it happen faster. >>>> >>>> I'm going to get some numbers, first without any changes, then with >>>> your proposed change, then with ptest dropped - this will take a bit >>>> of time, so I wanted to get the concerns written and sent first. >>>> >>>> Alex >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> Links: You receive all messages sent to this group. >>>> View/Reply Online (#192339): https://lists.openembedded.org/g/openembedded-core/message/192339 >>>> Mute This Topic: https://lists.openembedded.org/mt/103146402/1997914 >>>> Group Owner: openembedded-core+owner@lists.openembedded.org >>>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> >> >> -- >> Alexandre Belloni, co-owner and COO, Bootlin >> Embedded Linux and Kernel engineering >> https://bootlin.com
On 15 Dec 2023, at 10:49, Yoann Congal <yoann.congal@smile.fr> wrote:
> FYI, since Khem's patch ([OE-core] [PATCH] util-linux: Delete md-raid tests) also prevent bluez to be built in a default core-image-minimal build, we can skip this one (strace: Disable bluetooth support by default) and still fix [YOCTO #15323].
I’m a fan of both. Bluetooth support in strace is a niche usecase and easily enabled.
Ross
On Fri, 2023-12-15 at 11:59 +0000, Ross Burton wrote: > On 15 Dec 2023, at 10:49, Yoann Congal <yoann.congal@smile.fr> wrote: > > FYI, since Khem's patch ([OE-core] [PATCH] util-linux: Delete md- > > raid tests) also prevent bluez to be built in a default core-image- > > minimal build, we can skip this one (strace: Disable bluetooth > > support by default) and still fix [YOCTO #15323]. > > I’m a fan of both. Bluetooth support in strace is a niche usecase and easily enabled. I have a half drafted email saying I was thinking of both too! (I then got distracted). Cheers, Richard
diff --git a/meta/recipes-devtools/strace/strace_6.6.bb b/meta/recipes-devtools/strace/strace_6.6.bb index a3de7941cf..eeffe477b3 100644 --- a/meta/recipes-devtools/strace/strace_6.6.bb +++ b/meta/recipes-devtools/strace/strace_6.6.bb @@ -22,10 +22,6 @@ inherit autotools ptest # Not yet ported to rv32 COMPATIBLE_HOST:riscv32 = "null" -PACKAGECONFIG:class-target ??= "\ - ${@bb.utils.contains('DISTRO_FEATURES', 'bluetooth', 'bluez', '', d)} \ -" - PACKAGECONFIG[bluez] = "ac_cv_header_bluetooth_bluetooth_h=yes,ac_cv_header_bluetooth_bluetooth_h=no,bluez5" PACKAGECONFIG[libunwind] = "--with-libunwind,--without-libunwind,libunwind"
The bluetooth support adds a bluez5 dependency (and,recursively, a lot of other stuff). Disable it by default to avoid having to build all of this when it is not needed. This decrease the number of tasks run for a core-image-minimal build by ~1000 (-21%). To re-enable bluetooth support in strace, add "bluez" to strace PACKAGECONFIG. For example, in local.conf: PACKAGECONFIG:append:pn-strace = " bluez" Fixes [YOCTO #15323] Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Suggested-by: Ross Burton <ross.burton@arm.com> --- v1->v2: * Fixed local.conf example (thanks Tim) * Added Suggested-by: Ross (thanks Ross) * Removed an extra empty line v2->v3: * Added missing leading space in local.conf suggestion (thanks Martin) --- meta/recipes-devtools/strace/strace_6.6.bb | 4 ---- 1 file changed, 4 deletions(-)