[yocto-autobuilder-helper] Add toaster test builder

From: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>

Add a toaster test builder that runs the tox test suite using a new
run-toaster-tests script.

Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
CC: richard.purdie@linuxfoundation.org
---
 config.json               |  3 +++
 scripts/run-toaster-tests | 22 ++++++++++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100644 scripts/run-toaster-tests

On Mon, 2023-11-13 at 13:21 -0500,
alexander.lussier-cullen@savoirfairelinux.com wrote:
> From: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
> 
> Add a toaster test builder that runs the tox test suite using a new
> run-toaster-tests script.
> 
> Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
> CC: richard.purdie@linuxfoundation.org
> ---
>  config.json               |  3 +++
>  scripts/run-toaster-tests | 22 ++++++++++++++++++++++
>  2 files changed, 25 insertions(+)
>  create mode 100644 scripts/run-toaster-tests
> 
> diff --git a/config.json b/config.json
> index d10a3a8..0c581d1 100644
> --- a/config.json
> +++ b/config.json
> @@ -1675,6 +1675,9 @@
>              "step1" : {
>                  "BBTARGETS" : "core-image-patchtest"
>              }
> +        },
> +        "toaster" : {
> +            "EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/.."]
>          }
>      },
>      "repo-defaults" : {
> diff --git a/scripts/run-toaster-tests b/scripts/run-toaster-tests
> new file mode 100644
> index 0000000..a785c36
> --- /dev/null
> +++ b/scripts/run-toaster-tests
> @@ -0,0 +1,22 @@
> +#!/bin/bash
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# Called with $1 as the build directory
> +#             $2 as the path to poky
> +
> +set -e
> +set -u
> +set -o pipefail
> +set -x
> +
> +builddir=$(realpath "$1")
> +pokydir=$(realpath "$2")
> +
> +cd $builddir
> +mkdir -p toaster_logs
> +python3 -m venv venv --without-pip --system-site-packages
> +source venv/bin/activate
> +python3 -m pip install tox
> +
> +tox -c $pokydir/bitbake/lib/toaster/tox.ini

Thanks! I tested this and the first issue was:

https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/1

so I added a chmod a+x onto the commit which is on master-next of
yocto-autobuilder-helper.

This lead to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/2
and
https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/3

which were because a worker with python 3.8/3.9/3.10 wasn't available
at the time of testing and I'm not restricting the worker set the
target is using.

I put a commit onto bitbake/poky master-next to add 3.11 and 3.12 to
tox.ini which leads to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/4

which does appear to be running some testing with some errors.

So we're making progress :)

Cheers,

Richard

On Mon, 2023-11-13 at 22:30 +0000, Richard Purdie via
lists.yoctoproject.org wrote:
> On Mon, 2023-11-13 at 13:21 -0500,
> alexander.lussier-cullen@savoirfairelinux.com wrote:
> > From: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
> > 
> > Add a toaster test builder that runs the tox test suite using a new
> > run-toaster-tests script.
> > 
> > Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
> > CC: richard.purdie@linuxfoundation.org
> > ---
> >  config.json               |  3 +++
> >  scripts/run-toaster-tests | 22 ++++++++++++++++++++++
> >  2 files changed, 25 insertions(+)
> >  create mode 100644 scripts/run-toaster-tests
> > 
> > diff --git a/config.json b/config.json
> > index d10a3a8..0c581d1 100644
> > --- a/config.json
> > +++ b/config.json
> > @@ -1675,6 +1675,9 @@
> >              "step1" : {
> >                  "BBTARGETS" : "core-image-patchtest"
> >              }
> > +        },
> > +        "toaster" : {
> > +            "EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/.."]
> >          }
> >      },
> >      "repo-defaults" : {
> > diff --git a/scripts/run-toaster-tests b/scripts/run-toaster-tests
> > new file mode 100644
> > index 0000000..a785c36
> > --- /dev/null
> > +++ b/scripts/run-toaster-tests
> > @@ -0,0 +1,22 @@
> > +#!/bin/bash
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# Called with $1 as the build directory
> > +#             $2 as the path to poky
> > +
> > +set -e
> > +set -u
> > +set -o pipefail
> > +set -x
> > +
> > +builddir=$(realpath "$1")
> > +pokydir=$(realpath "$2")
> > +
> > +cd $builddir
> > +mkdir -p toaster_logs
> > +python3 -m venv venv --without-pip --system-site-packages
> > +source venv/bin/activate
> > +python3 -m pip install tox
> > +
> > +tox -c $pokydir/bitbake/lib/toaster/tox.ini
> 
> Thanks! I tested this and the first issue was:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/1
> 
> so I added a chmod a+x onto the commit which is on master-next of
> yocto-autobuilder-helper.
> 
> This lead to:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/2
> and
> https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/3
> 
> which were because a worker with python 3.8/3.9/3.10 wasn't available
> at the time of testing and I'm not restricting the worker set the
> target is using.
> 
> I put a commit onto bitbake/poky master-next to add 3.11 and 3.12 to
> tox.ini which leads to:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/4
> 
> which does appear to be running some testing with some errors.
> 
> So we're making progress :)

https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/4

(on ubuntu 23.04 with python3.11) ran to completion in 45 minutes,
failures=2, errors=18. 

It'd note that it didn't use SSTATE_DIR or DL_DIR from the parent
environment which is partly why it was so slow, we need to fix that.

I tried adding some of the pending toaster patches to master-next and
rebuilding. There were more workers available so it tried some
different combinations which failed:

https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/5

opensuse 15.4 python 3.11

  File "/home/pokybuild/yocto-
worker/toaster/build/build/.tox/py311/lib/python3.11/site-
packages/django/utils/timezone.py", line 12, in <module>
    from backports import zoneinfo
ModuleNotFoundError: No module named 'backports'

https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/6

ubuntu 20.04 python 3.8

  File "/usr/lib/python3.8/importlib/metadata.py", line 79, in load
    return functools.reduce(getattr, attrs, module)
AttributeError: module 'virtualenv.create.via_global_ref.builtin.cpython.mac_os' has no attribute 'CPython2macOsFramework'


I've set https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/7
on ubuntu 23.04 away so we can see if things improve or not with the patches.

Cheers,

Richard