[7/8] core-image-ptest: Switch to BBCLASSEXTEND parallel execution

Replace the two images where tests are run serially with an image
per ptest in the form core-image-ptest-XXX using class extensions.

The original two images are then replaced with dependencies on these
images. This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating
this number of class extensions is slow but making it easier to execute
and collect test data should outweigh that.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../images/core-image-ptest-all.bb            | 29 ++++++--------
 .../images/core-image-ptest-fast.bb           | 24 ++----------
 meta/recipes-core/images/core-image-ptest.bb  | 39 +++++++++++++++++++
 3 files changed, 53 insertions(+), 39 deletions(-)
 create mode 100644 meta/recipes-core/images/core-image-ptest.bb

Nice! Is there some other low(ish) hanging fruit for making a-full faster?

(one thing that springs to mind is more arm hardware; the one
available seems overloaded)

Alex

On Sat, 11 Mar 2023 at 23:42, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Replace the two images where tests are run serially with an image
> per ptest in the form core-image-ptest-XXX using class extensions.
>
> The original two images are then replaced with dependencies on these
> images. This allows parallel execution of the tests.
>
> The downside to this approach is the parsing time of an image generating
> this number of class extensions is slow but making it easier to execute
> and collect test data should outweigh that.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  .../images/core-image-ptest-all.bb            | 29 ++++++--------
>  .../images/core-image-ptest-fast.bb           | 24 ++----------
>  meta/recipes-core/images/core-image-ptest.bb  | 39 +++++++++++++++++++
>  3 files changed, 53 insertions(+), 39 deletions(-)
>  create mode 100644 meta/recipes-core/images/core-image-ptest.bb
>
> diff --git a/meta/recipes-core/images/core-image-ptest-all.bb b/meta/recipes-core/images/core-image-ptest-all.bb
> index 1472451675e..1a3e979d7e3 100644
> --- a/meta/recipes-core/images/core-image-ptest-all.bb
> +++ b/meta/recipes-core/images/core-image-ptest-all.bb
> @@ -1,25 +1,18 @@
> -inherit features_check
> +DESCRIPTION = "Recipe to trigger execution of all ptest images."
> +HOMEPAGE = "https://www.yoctoproject.org/"
> +
> +LICENSE = "MIT"
> +
> +inherit features_check nopackages
>  REQUIRED_DISTRO_FEATURES = "ptest"
>
> -require core-image-minimal.bb
>  require conf/distro/include/ptest-packagelists.inc
>
> -DESCRIPTION += "Also includes ptest packages."
> -HOMEPAGE = "https://www.yoctoproject.org/"
> -
>  # Include the full set of ptests
> -IMAGE_INSTALL += "${PTESTS_FAST} ${PTESTS_SLOW}"
> -
> -# This image is sufficiently large (~1.8GB) that we need to be careful that it fits in a live
> -# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
> -# box) and explicitly add just 1100MB.
> -# strace-ptest in particular needs more than 500MB
> -IMAGE_OVERHEAD_FACTOR = "1.0"
> -IMAGE_ROOTFS_EXTRA_SPACE = "1124288"
> +PTESTS = "${PTESTS_FAST} ${PTESTS_SLOW}"
>
> -# ptests need more memory than standard to avoid the OOM killer
> -# also lttng-tools needs /tmp that has at least 2G
> -QB_MEM = "-m 4096"
> +addtask do_testimage
> +do_testimage[noexec] = "1"
> +do_testimage[depends] = "${@' '.join(['core-image-ptest-'+x.replace('-ptest', '')+':do_testimage' for x in d.getVar('PTESTS').split()])}"
>
> -# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
> -PTEST_EXPECT_FAILURE = "1"
> +do_build[depends] = "${@' '.join(['core-image-ptest-'+x.replace('-ptest', '')+':do_build' for x in d.getVar('PTESTS').split()])}"
> diff --git a/meta/recipes-core/images/core-image-ptest-fast.bb b/meta/recipes-core/images/core-image-ptest-fast.bb
> index 40df274c4c1..e768afb302b 100644
> --- a/meta/recipes-core/images/core-image-ptest-fast.bb
> +++ b/meta/recipes-core/images/core-image-ptest-fast.bb
> @@ -1,24 +1,6 @@
> -inherit features_check
> -REQUIRED_DISTRO_FEATURES = "ptest"
> +require core-image-ptest-all.bb
>
> -require core-image-minimal.bb
> -require conf/distro/include/ptest-packagelists.inc
> +DESCRIPTION = "Recipe to trigger execution of all fast ptest images."
>
> -# Include only fast (< 30 sec) ptests
> -IMAGE_INSTALL += "${PTESTS_FAST}"
> +PTESTS = "${PTESTS_FAST}"
>
> -DESCRIPTION += "Also includes ptest packages with fast execution times to allow for more automated QA."
> -HOMEPAGE = "https://www.yoctoproject.org/"
> -
> -# This image is sufficiently large (~1.8GB) that it can't actually fit in a live
> -# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
> -# box) and explicitly add just 1500MB.
> -# strace-ptest in particular needs more than 500MB
> -IMAGE_OVERHEAD_FACTOR = "1.0"
> -IMAGE_ROOTFS_EXTRA_SPACE = "1524288"
> -
> -# ptests need more memory than standard to avoid the OOM killer
> -QB_MEM = "-m 1024"
> -
> -# Sadly at the moment the fast set of ptests is not robust enough and sporadically fails in a couple of places
> -PTEST_EXPECT_FAILURE = "1"
> diff --git a/meta/recipes-core/images/core-image-ptest.bb b/meta/recipes-core/images/core-image-ptest.bb
> new file mode 100644
> index 00000000000..0c67943b057
> --- /dev/null
> +++ b/meta/recipes-core/images/core-image-ptest.bb
> @@ -0,0 +1,39 @@
> +inherit features_check
> +REQUIRED_DISTRO_FEATURES = "ptest"
> +
> +require core-image-minimal.bb
> +require conf/distro/include/ptest-packagelists.inc
> +
> +DESCRIPTION += "Also including the ${MCNAME} ptest package."
> +HOMEPAGE = "https://www.yoctoproject.org/"
> +
> +PTESTS = "${PTESTS_SLOW} ${PTESTS_FAST}"
> +
> +IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
> +
> +BBCLASSEXTEND = "${@' '.join(['mcextend:'+x.replace('-ptest', '') for x in d.getVar('PTESTS').split()])}"
> +
> +# The image can sufficiently large (~1.8GB) that we need to be careful that it fits in a live
> +# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
> +# box) and explicitly add up to 1500MB.
> +# strace-ptest in particular needs more than 500MB
> +IMAGE_OVERHEAD_FACTOR = "1.0"
> +IMAGE_ROOTFS_EXTRA_SPACE = "324288"
> +IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-strace = "1024288"
> +IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-lttng-tools = "1524288"
> +
> +# ptests need more memory than standard to avoid the OOM killer
> +QB_MEM = "-m 1024"
> +QB_MEM:virtclass-mcextend-lttng-tools = "-m 4096"
> +QB_MEM:virtclass-mcextend-python3-cryptography = "-m 4096"
> +
> +TEST_SUITES = "ping ssh parselogs ptest"
> +
> +# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
> +PTEST_EXPECT_FAILURE = "1"
> +
> +python () {
> +    if not d.getVar("MCNAME"):
> +        raise bb.parse.SkipRecipe("No class extension set")
> +}
> +
> --
> 2.38.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#178373): https://lists.openembedded.org/g/openembedded-core/message/178373
> Mute This Topic: https://lists.openembedded.org/mt/97550253/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Sun, 2023-03-12 at 14:17 +0100, Alexander Kanavin wrote:
> Nice! Is there some other low(ish) hanging fruit for making a-full faster?
> 
> (one thing that springs to mind is more arm hardware; the one
> available seems overloaded)

I suspect it isn't arm hardware that is the bottleneck. I have some
patches locally to split up the oe-selftest sstate tests which is one
easy improvement.

The next would be to have reproducible builds reuse rust-native from
sstate. We don't test reproducibility of native recipes anyway.

After that, it would be looking at the toolchain tests, i.e. this step
in qemu machine builds:

https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6780/steps/19/logs/stdio

from the command:

oe-selftest -a -t machine -t toolchain-user -j 15

That step takes a couple of hours which is 2/3rds of the time of that
target. I was thinking it could be parallelised but I notice we already
did that! Maybe we should move it to a separate target so it is in
parallel with everything else?

Cheers,

Richard

On Sun, 12 Mar 2023 at 14:36, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I suspect it isn't arm hardware that is the bottleneck. I have some
> patches locally to split up the oe-selftest sstate tests which is one
> easy improvement.
>
> The next would be to have reproducible builds reuse rust-native from
> sstate. We don't test reproducibility of native recipes anyway.
>
> After that, it would be looking at the toolchain tests, i.e. this step
> in qemu machine builds:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6780/steps/19/logs/stdio
>
> from the command:
>
> oe-selftest -a -t machine -t toolchain-user -j 15
>
> That step takes a couple of hours which is 2/3rds of the time of that
> target. I was thinking it could be parallelised but I notice we already
> did that! Maybe we should move it to a separate target so it is in
> parallel with everything else?

I also just sent a (tested) patchset that takes mesa-native (and
friends) out of qemu-system-native dependencies by default, like it
was before we had virgl support - should help a bit with bootstrapping
qemu targets.

Alex