From patchwork Wed Dec 6 15:54:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 35786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E704EC46CA0 for ; Wed, 6 Dec 2023 15:54:38 +0000 (UTC) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by mx.groups.io with SMTP id smtpd.web11.34966.1701878075191306485 for ; Wed, 06 Dec 2023 07:54:35 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=E1uIRInS; spf=pass (domain: bootlin.com, ip: 217.70.183.196, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id E570AE0007; Wed, 6 Dec 2023 15:54:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1701878073; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q1XB+x8aMBrHoDezAvCBrzpH8aRsrCFxzuwUggnFh6c=; b=E1uIRInS3Idapu+Bqj+Od6YvLRWkJguyG7UWgknT3jkTD7dKhRnnkR0m/6nnCGd7bH/JbX H2n0j3m0YTcv0NpWvV/7FfdqYktuQCt2YmMxbKOhEJ7ijMvP8sK7yAfMgepYrsYBktWwtj aKqzo7z4FN3BPJXNO/ISXZamfdsjdRqPwLGGeUBY7Gm9xnDzcEShk/aYkQup9EwbAVG0Sd xP7lULhvphc9B2vkgUt+uYxx19XtraSTxZziRhnGj8NjZx+wDP/hc9h/5r2baR7c76Td7V JDefposuAU+X0IfkOwfSCCIRxNYlnjzTOKCsWGVqbWFLstQogHYtEooHIZMnUA== From: michael.opdenacker@bootlin.com To: docs@lists.yoctoproject.org Cc: Michael Opdenacker Subject: [PATCH 1/6] test-manual: text and formatting fixes Date: Wed, 6 Dec 2023 16:54:22 +0100 Message-Id: <20231206155427.279612-2-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231206155427.279612-1-michael.opdenacker@bootlin.com> References: <20231206155427.279612-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 06 Dec 2023 15:54:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/4657 From: Michael Opdenacker Signed-off-by: Michael Opdenacker --- documentation/test-manual/intro.rst | 44 +++++++++---------- documentation/test-manual/test-process.rst | 12 ++--- .../test-manual/understand-autobuilder.rst | 33 +++++++------- 3 files changed, 45 insertions(+), 44 deletions(-) diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index aaf64ae017..f27210642c 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst @@ -14,7 +14,7 @@ release works as intended. All the project's testing infrastructure and processes are publicly visible and available so that the community can see what testing is being performed, how it's being done and the current status of the tests and the project at any given time. It is intended -that Other organizations can leverage off the process and testing +that other organizations can leverage off the process and testing environment used by the Yocto Project to create their own automated, production test environment, building upon the foundations from the project core. @@ -93,8 +93,8 @@ the following types of tests: - *Build Testing:* Tests whether specific configurations build by varying :term:`MACHINE`, :term:`DISTRO`, other configuration - options, and the specific target images being built (or world). Used - to trigger builds of all the different test configurations on the + options, and the specific target images being built (or ``world``). This is + used to trigger builds of all the different test configurations on the Autobuilder. Builds usually cover many different targets for different architectures, machines, and distributions, as well as different configurations, such as different init systems. The @@ -120,7 +120,7 @@ the following types of tests: $ bitbake image -c testsdkext - The tests utilize the :ref:`ref-classes-testsdk` class and the + The tests use the :ref:`ref-classes-testsdk` class and the ``do_testsdkext`` task. - *Feature Testing:* Various scenario-based tests are run through the @@ -131,7 +131,7 @@ the following types of tests: $ bitbake image -c testimage - The tests utilize the :ref:`ref-classes-testimage` + The tests use the :ref:`ref-classes-testimage` class and the :ref:`ref-tasks-testimage` task. - *Layer Testing:* The Autobuilder has the possibility to test whether @@ -151,7 +151,7 @@ the following types of tests: $ bitbake image -c testsdk - The tests utilize the :ref:`ref-classes-testsdk` class and + The tests use the :ref:`ref-classes-testsdk` class and the ``do_testsdk`` task. - *Unit Testing:* Unit tests on various components of the system run @@ -190,7 +190,7 @@ Tests map into the codebase as follows: $ bitbake-selftest -v To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` - variable when running "bitbake-selftest" as follows:: + variable when running ``bitbake-selftest`` as follows:: $ BB_SKIP_NETTESTS=yes bitbake-selftest @@ -215,7 +215,7 @@ Tests map into the codebase as follows: - These tests use OE to test the workflows, which include testing specific features, behaviors of tasks, and API unit tests. - - The tests can take advantage of parallelism through the "-j" + - The tests can take advantage of parallelism through the ``-j`` option, which can specify a number of threads to spread the tests across. Note that all tests from a given class of tests will run in the same thread. To parallelize large numbers of tests you can @@ -231,18 +231,18 @@ Tests map into the codebase as follows: $ oe-selftest -a - To run a specific test, use the following command form where - testname is the name of the specific test:: + ``testname`` is the name of the specific test:: $ oe-selftest -r - For example, the following command would run the tinfoil - getVar API test:: + For example, the following command would run the ``tinfoil`` + ``getVar`` API test:: $ oe-selftest -r tinfoil.TinfoilTests.test_getvar It is also possible to run a set of tests. For example the following command will run all of the - tinfoil tests:: + ``tinfoil`` tests:: $ oe-selftest -r tinfoil @@ -391,14 +391,14 @@ so tests within a given test class should always run in the same build, while tests in different classes or modules may be split into different builds. There is no data store available for these tests since the tests launch the ``bitbake`` command and exist outside of its context. As a -result, common bitbake library functions (bb.\*) are also unavailable. +result, common BitBake library functions (``bb.\*``) are also unavailable. ``testimage`` ------------- These tests are run once an image is up and running, either on target hardware or under QEMU. As a result, they are assumed to be running in a -target image environment, as opposed to a host build environment. A +target image environment, as opposed to in a host build environment. A simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains the following:: @@ -413,19 +413,19 @@ the following:: In this example, the ``OERuntimeTestCase`` class wraps ``unittest.TestCase``. Within the test, ``self.target`` represents the -target system, where commands can be run on it using the ``run()`` +target system, where commands can be run using the ``run()`` method. -To ensure certain test or package dependencies are met, you can use the +To ensure certain tests or package dependencies are met, you can use the ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test -in this example would only make sense if python3-core is installed in +in this example would only make sense if ``python3-core`` is installed in the image. ``testsdk_ext`` --------------- These tests are run against built extensible SDKs (eSDKs). The tests can -assume that the eSDK environment has already been setup. An example from +assume that the eSDK environment has already been set up. An example from ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: class DevtoolTest(OESDKExtTestCase): @@ -472,9 +472,9 @@ following:: output = self._run(cmd) self.assertEqual(output, "Hello, world\n") -In this example, if nativesdk-python3-core has been installed into the SDK, the code runs -the python3 interpreter with a basic command to check it is working -correctly. The test would only run if Python3 is installed in the SDK. +In this example, if ``nativesdk-python3-core`` has been installed into the SDK, +the code runs the ``python3`` interpreter with a basic command to check it is +working correctly. The test would only run if Python3 is installed in the SDK. ``oe-build-perf-test`` ---------------------- @@ -520,7 +520,7 @@ an isolated directory. This can delete files from :term:`SSTATE_DIR` which would potentially break other builds running in parallel. If this is required, :term:`SSTATE_DIR` must -be set to an isolated directory. Alternatively, you can use the "-f" +be set to an isolated directory. Alternatively, you can use the ``-f`` option with the ``bitbake`` command to "taint" tasks by changing the sstate checksums to ensure sstate cache items will not be reused. diff --git a/documentation/test-manual/test-process.rst b/documentation/test-manual/test-process.rst index 4c3b32bfea..0c560efe39 100644 --- a/documentation/test-manual/test-process.rst +++ b/documentation/test-manual/test-process.rst @@ -42,16 +42,16 @@ our sstate up to date. On the weekend, there is a master-next build instead to ensure the test results are updated for the less frequently run targets. -Performance builds (buildperf-\* targets in the console) are triggered +Performance builds (``buildperf-\*`` targets in the console) are triggered separately every six hours and automatically push their results to the buildstats repository at: :yocto_git:`/yocto-buildstats/`. -The 'quick' targets have been selected to be the ones which catch the -most failures or give the most valuable data. We run 'fast' ptests in +The "quick" targets have been selected to be the ones which catch the +most failures or give the most valuable data. We run "fast" ptests in this case for example but not the ones which take a long time. The quick -target doesn't include \*-lsb builds for all architectures, some world -builds and doesn't trigger performance tests or ltp testing. The full +target doesn't include ``\*-lsb`` builds for all architectures, some ``world`` +builds and doesn't trigger performance tests or ``ltp`` testing. The full build includes all these things and is slower but more comprehensive. Release Builds @@ -67,7 +67,7 @@ that in :ref:`test-manual/test-process:day to day development`, in that the a-full target of the Autobuilder is used but in addition the form is configured to generate and publish artifacts and the milestone number, version, release candidate number and other information is entered. The -box to "generate an email to QA"is also checked. +box to "generate an email to QA" is also checked. When the build completes, an email is sent out using the send-qa-email script in the ``yocto-autobuilder-helper`` repository to the list of diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst index 7a6cb2443b..a3fff29aca 100644 --- a/documentation/test-manual/understand-autobuilder.rst +++ b/documentation/test-manual/understand-autobuilder.rst @@ -32,8 +32,8 @@ which looks like:: } }, -And to expand that, you need the "arch-qemu" entry from -the "templates" section, which looks like:: +And to expand that, you need the ``arch-qemu`` entry from +the ``templates`` section, which looks like:: "arch-qemu" : { "BUILDINFO" : true, @@ -54,11 +54,11 @@ the "templates" section, which looks like:: } }, -Combining these two entries you can see that "qemux86-64" is a three step build where the -``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for -``MACHINE="qemux86-64"`` but with differing :term:`SDKMACHINE` settings. In step -1 an extra variable is added to the ``auto.conf`` file to enable wic -image generation. +Combining these two entries you can see that ``qemux86-64`` is a three step +build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` +for each step; all for ``MACHINE="qemux86-64"`` but with differing +:term:`SDKMACHINE` settings. In step 1, an extra variable is added to the +``auto.conf`` file to enable wic image generation. While not every detail of this is covered here, you can see how the template mechanism allows quite complex configurations to be built up @@ -163,8 +163,9 @@ Autobuilder Worker Janitor -------------------------- This is a process running on each Worker that performs two basic -operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and -maintenance of a cache of cloned repositories to improve the speed +operations, including background file deletion at IO idle (see +"Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`) +and maintenance of a cache of cloned repositories to improve the speed the system can checkout repositories. Shared DL_DIR @@ -172,7 +173,7 @@ Shared DL_DIR The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared between them. This reduces network accesses from the system and allows -the build to be sped up. Usage of the directory within the build system +the build to be sped up. The usage of the directory within the build system is designed to be able to be shared over NFS. Shared SSTATE_DIR @@ -180,8 +181,8 @@ Shared SSTATE_DIR The Workers are all connected over NFS which allows the ``sstate`` directory to be shared between them. This means once a Worker has built -an artifact, all the others can benefit from it. Usage of the directory -within the directory is designed for sharing over NFS. +an artifact, all the others can benefit from it. The usage of the directory +within the build system is designed for sharing over NFS. Resulttool ---------- @@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as failure logs or the time taken to run the tests, may also be included. Resulttool is part of OpenEmbedded-Core and is used to manipulate these -json results files. It has the ability to merge files together, display +JSON results files. It has the ability to merge files together, display reports of the test results and compare different result files. For details, see :yocto_wiki:`/Resulttool`. @@ -206,7 +207,7 @@ are general setup steps that are run once and include: #. Set up any :term:`buildtools` tarball if configured. -#. Call "buildhistory-init" if :ref:`ref-classes-buildhistory` is configured. +#. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured. For each step that is configured in ``config.json``, it will perform the following: @@ -258,7 +259,7 @@ it is inevitable that users will end up needing to heavily customise the ``yocto-autobuilder-helper`` repository, particularly the ``config.json`` file as they will want to define their own test matrix. -The Autobuilder supports wo customization options: +The Autobuilder supports two customization options: - variable substitution @@ -278,7 +279,7 @@ environment:: $ ABHELPER_JSON="config.json /some/location/local.json" One issue users often run into is validation of the ``config.json`` files. A -tip for minimizing issues from invalid json files is to use a Git +tip for minimizing issues from invalid JSON files is to use a Git ``pre-commit-hook.sh`` script to verify the JSON file before committing it. Create a symbolic link as follows:: From patchwork Wed Dec 6 15:54:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 35785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF324C10DC1 for ; Wed, 6 Dec 2023 15:54:38 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web10.35293.1701878077570561350 for ; Wed, 06 Dec 2023 07:54:38 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=JqMisB6E; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id B4E8BC0003; Wed, 6 Dec 2023 15:54:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1701878075; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zeVH8zD45s0XhjB8pNu78cfzj45xv5t2TBlXRLBxwAA=; b=JqMisB6EE6rZkX5+vnsKHTyyUegYe789GAXfNn190wwOn/hpdU4JmjZ3eOp1dbr/WdNiro cg43Hic8v6Vxt0BS7ziY9yJAHqGpcrP7jonkhJ+zE76idocZ1tjej7s+GF0cPZ4i2sgpAn lgotw3bOVBRzaE1GeRJWM6m9urtVnaYDvvxH0tNIt+nPIpC5FI+FYne+JM+VmmMCnTn8Vg 8V+H5a945H4PaX2oVupSu+2PwvzC1k0Qe69ToNwgNzdTEHgtAX1cxCFfvY9Hvu6VzazdF8 t2BmWv+pKlEk3N4T/S+wlPDPQCM4EFGpwrIZ257lIo80FULi1pv/iZeUmR/gDQ== From: michael.opdenacker@bootlin.com To: docs@lists.yoctoproject.org Cc: Michael Opdenacker Subject: [PATCH 2/6] test-manual: resource updates Date: Wed, 6 Dec 2023 16:54:23 +0100 Message-Id: <20231206155427.279612-3-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231206155427.279612-1-michael.opdenacker@bootlin.com> References: <20231206155427.279612-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 06 Dec 2023 15:54:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/4658 From: Michael Opdenacker Signed-off-by: Michael Opdenacker --- documentation/test-manual/intro.rst | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index f27210642c..d14aa20f76 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst @@ -19,10 +19,8 @@ environment used by the Yocto Project to create their own automated, production test environment, building upon the foundations from the project core. -Currently, the Yocto Project Test Environment Manual has no projected -release date. This manual is a work-in-progress and is being initially -loaded with information from the README files and notes from key -engineers: +This manual is a work-in-progress and is being initially loaded with +information from the README files and notes from key engineers: - *yocto-autobuilder2:* This :yocto_git:`README.md ` @@ -39,7 +37,7 @@ engineers: As a result, it can be used by any Continuous Improvement (CI) system to run builds, support getting the correct code revisions, configure builds and layers, run builds, and collect results. The code is - independent of any CI system, which means the code can work `Buildbot `__, + independent of any CI system, which means the code can work `Buildbot `__, Jenkins, or others. This repository has a branch per release of the project defining the tests to run on a per release basis. @@ -54,8 +52,8 @@ the Autobuilder tests if things work. The Autobuilder builds all test targets and runs all the tests. The Yocto Project uses now uses standard upstream -`Buildbot `__ (version 9) to -drive its integration and testing. Buildbot Nine has a plug-in interface +Buildbot (`version 3.8 `__) to +drive its integration and testing. Buildbot has a plug-in interface that the Yocto Project customizes using code from the ``yocto-autobuilder2`` repository, adding its own console UI plugin. The resulting UI plug-in allows you to visualize builds in a way suited to @@ -304,11 +302,11 @@ Tests map into the codebase as follows: Git repository. Use the ``oe-build-perf-report`` command to generate text reports - and HTML reports with graphs of the performance data. For - examples, see - :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` + and HTML reports with graphs of the performance data. See + :yocto_dl:`html ` and - :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. + :yocto_dl:`txt ` + examples. - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. From patchwork Wed Dec 6 15:54:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 35784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 073B7C10F07 for ; Wed, 6 Dec 2023 15:54:39 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web11.34970.1701878078376320681 for ; Wed, 06 Dec 2023 07:54:38 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=Q9lj2nB9; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id E8F5AC000E; Wed, 6 Dec 2023 15:54:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1701878077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9lLOcUSh5ewAmgOh7JEWRzCBKPMGEV6AI4TeYj7aGAQ=; b=Q9lj2nB9kzmnfqwg+g0aZUNHfNwEi2vxTnDIGQ8lpMczZuMPjV+XFlFkMlwE/0c97F6PN0 fQdrCkmv7IG6pG8GKrpVB8gezxZ/wBRmPWwaUUN/lpOEYI7KfLETy7WBgzJQ1Wjw1WF1ko ByJJw1dIrb1Bg7va7wOyaTTY2hj+RHF5VUY5EKCEE7G6gYNuD51Jc95aBIhpJu24adXp2G xWE+t9HwkjY30T75smMpLXfvrzwjw51s1ZrfjYlNhfWwadzQnrMBPuq9F4S2S+riiBPWKE 8r4GO9VS20K9XRxn9+4zTQgHMYPzIvYCkGd/radw2j97UxADewRfCl9cbMNnBw== From: michael.opdenacker@bootlin.com To: docs@lists.yoctoproject.org Cc: Michael Opdenacker Subject: [PATCH 3/6] test-manual: use working example Date: Wed, 6 Dec 2023 16:54:24 +0100 Message-Id: <20231206155427.279612-4-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231206155427.279612-1-michael.opdenacker@bootlin.com> References: <20231206155427.279612-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 06 Dec 2023 15:54:39 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/4659 From: Michael Opdenacker Signed-off-by: Michael Opdenacker --- documentation/test-manual/intro.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index d14aa20f76..e72a51172c 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst @@ -195,15 +195,15 @@ Tests map into the codebase as follows: Use this option when you wish to skip tests that access the network, which are mostly necessary to test the fetcher modules. To specify individual test modules to run, append the test module name to the - "bitbake-selftest" command. For example, to specify the tests for the - bb.data.module, run:: + ``bitbake-selftest`` command. For example, to specify the tests for + ``bb.tests.data.DataExpansions``, run:: - $ bitbake-selftest bb.test.data.module + $ bitbake-selftest bb.tests.data.DataExpansions You can also specify individual tests by defining the full name and module plus the class path of the test, for example:: - $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override + $ bitbake-selftest bb.tests.data.DataExpansions.test_one_var The tests are based on `Python unittest `__. From patchwork Wed Dec 6 15:54:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 35788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F14D4C4167B for ; Wed, 6 Dec 2023 15:54:48 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web10.35298.1701878080280281610 for ; Wed, 06 Dec 2023 07:54:40 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=SAS5ewoV; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 517E160008; Wed, 6 Dec 2023 15:54:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1701878078; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fWBzfsGfZ/lKPZo0QozcDhJ2Cra4P7AfIKCH09Lgwuw=; b=SAS5ewoVXv2U62BAJ0YP7VqOkVR4VsCn7iBKARE8I39Sul3YppPE6mURiPLFD8q932nnHe ek+fCDJA5QEdSEI/vyG9nvoBfFkTgZ1EIzzUxU39Ui7Z39SAlrH3Iwu8/7yD/G0OQESwvx vzSY7aemeaoMl+fefEwOTpCpGsoaQUR6f98sGxttO3SF8UP9GYrWfKM3S2WN4Qg8hoHDXJ OMJkH437pD2DiVKzSQzyHdkXKkkPTCtH71RgNI/tKeLqxeUK8qahU5X5oLfSDaMxba8Jey 2gwJ8bSlPZyOsqdVauAS0kKxmHa9gh9LRtd29HIX/PppZaIV2MKW2mL4vx2bbg== From: michael.opdenacker@bootlin.com To: docs@lists.yoctoproject.org Cc: Michael Opdenacker Subject: [PATCH 4/6] test-manual: add links to python unittest Date: Wed, 6 Dec 2023 16:54:25 +0100 Message-Id: <20231206155427.279612-5-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231206155427.279612-1-michael.opdenacker@bootlin.com> References: <20231206155427.279612-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 06 Dec 2023 15:54:48 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/4660 From: Michael Opdenacker Better than using "python unittest" without any special formatting. Signed-off-by: Michael Opdenacker --- documentation/test-manual/intro.rst | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index e72a51172c..a2560f4f53 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst @@ -205,8 +205,8 @@ Tests map into the codebase as follows: $ bitbake-selftest bb.tests.data.DataExpansions.test_one_var - The tests are based on `Python - unittest `__. + The tests are based on + `Python unittest `__. - *oe-selftest:* @@ -219,7 +219,8 @@ Tests map into the codebase as follows: in the same thread. To parallelize large numbers of tests you can split the class into multiple units. - - The tests are based on Python unittest. + - The tests are based on + `Python unittest `__. - The code for the tests resides in ``meta/lib/oeqa/selftest/cases/``. @@ -338,21 +339,24 @@ A simple test example from ``lib/bb/tests/data.py`` is:: val = self.d.expand("${foo}") self.assertEqual(str(val), "value_of_foo") -In this example, a ``DataExpansions`` class of tests is created, -derived from standard Python unittest. The class has a common ``setUp`` -function which is shared by all the tests in the class. A simple test is -then added to test that when a variable is expanded, the correct value -is found. +In this example, a ``DataExpansions`` class of tests is created, derived from +standard `Python unittest `__. +The class has a common ``setUp`` function which is shared by all the tests in +the class. A simple test is then added to test that when a variable is +expanded, the correct value is found. -BitBake selftests are straightforward Python unittest. Refer to the -Python unittest documentation for additional information on writing -these tests at: https://docs.python.org/3/library/unittest.html. +BitBake selftests are straightforward +`Python unittest `__. +Refer to the `Python unittest documentation +`__ for additional information +on writing such tests. ``oe-selftest`` --------------- These tests are more complex due to the setup required behind the scenes -for full builds. Rather than directly using Python's unittest, the code +for full builds. Rather than directly using `Python unittest +`__, the code wraps most of the standard objects. The tests can be simple, such as testing a command from within the OE build environment using the following example:: From patchwork Wed Dec 6 15:54:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 35787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03084C10F05 for ; Wed, 6 Dec 2023 15:54:49 +0000 (UTC) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by mx.groups.io with SMTP id smtpd.web10.35299.1701878081669464294 for ; Wed, 06 Dec 2023 07:54:41 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=hC26pXIk; spf=pass (domain: bootlin.com, ip: 217.70.183.200, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id B3B9C20008; Wed, 6 Dec 2023 15:54:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1701878079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J/lXpgIFopJqtnxonxDzxk661INKWehrgYotNBogoMw=; b=hC26pXIk/WQkwMZsM9VP2Hnt21fmMmKt3vVxyn3O1vUdpIoy4fsNmsghC+gnsTapUpCIeR UV7IW12qhlNs8RvnOCb4MZCLayz0ZfRcjflkTlALfFe1BlE4qEkwXIW0GEro8tIkGkS3A2 6Uz1yKwAepbeBbemNfr1DTyJnGJwf//0FCoLKOa8wcXHjNS2V52B1lDeYC4XgJjKwQPp4o VWXMP4FERwVTmobAL7Bk5caprbi5rBF3imcxbRsfuiRksShbZDcUIP9RMpJS5UwnfnM8e9 16T5d5SCZdVTz67f9t2uQlOOSWCHypbnBclzCsP/rVBk7+FucJyj2+cOzwmpDQ== From: michael.opdenacker@bootlin.com To: docs@lists.yoctoproject.org Cc: Michael Opdenacker Subject: [PATCH 5/6] test-manual: explicit or fix file paths Date: Wed, 6 Dec 2023 16:54:26 +0100 Message-Id: <20231206155427.279612-6-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231206155427.279612-1-michael.opdenacker@bootlin.com> References: <20231206155427.279612-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 06 Dec 2023 15:54:49 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/4661 From: Michael Opdenacker Signed-off-by: Michael Opdenacker --- documentation/test-manual/intro.rst | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index a2560f4f53..c31fd11c7a 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst @@ -276,7 +276,7 @@ Tests map into the codebase as follows: - These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. - - The code for these tests resides in ``meta/lib/oeqa/esdk``. + - The code for these tests resides in ``meta/lib/oeqa/sdkext/cases/``. - To run the tests, use the following command form:: @@ -309,7 +309,7 @@ Tests map into the codebase as follows: :yocto_dl:`txt ` examples. - - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. + - The tests are contained in ``meta/lib/oeqa/buildperf/test_basic.py``. Test Examples ============= @@ -317,16 +317,14 @@ Test Examples This section provides example tests for each of the tests listed in the :ref:`test-manual/intro:How Tests Map to Areas of Code` section. -For oeqa tests, testcases for each area reside in the main test -directory at ``meta/lib/oeqa/selftest/cases`` directory. +- ``oe-selftest`` testcases reside in the ``meta/lib/oeqa/selftest/cases`` directory. -For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/`` -directory. +- ``bitbake-selftest`` testcases reside in the ``bitbake/lib/bb/tests/`` directory. ``bitbake-selftest`` -------------------- -A simple test example from ``lib/bb/tests/data.py`` is:: +A simple test example from ``bitbake/lib/bb/tests/data.py`` is:: class DataExpansions(unittest.TestCase): def setUp(self): From patchwork Wed Dec 6 15:54:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 35789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A5F6C10DC1 for ; Wed, 6 Dec 2023 15:54:49 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web11.34973.1701878082184932947 for ; Wed, 06 Dec 2023 07:54:42 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=bsNzIiG6; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id B0F8AC0003; Wed, 6 Dec 2023 15:54:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1701878080; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kXHXW6Er/Izq1qIVo7wM5l/7geWhnYWErJGrf7lov3s=; b=bsNzIiG6M2op9wAv1slypuiVEug75zvfxToG9OCbx6udQT/FbMDPOxXAELriYnYCuRnrAM VAVqs+/RRv+Dkd3tieslLlt2H5IBwXgpPmptnSVIL7EuUxsGDyTiWkSz1dOo3ub2RWIWME L78c8XB+KeSOwEWLrJKIUr6FyP7PUJ2Xp2kQfM3s/becPQMFW7UeBKKMVULganaPdQZmbs Xwgv05yFY3vmYE6Ne3gy2dzg98/Q4BaJtbQU1zSMvbDxf0pN739IOJAUQ43MJCP9GTKaAf xn+nX8oFbOjtW4eKuzh2dMIQgeiZzeJdiOO/qsl+6w+Eq03lD9sv6sw8rPpe7A== From: michael.opdenacker@bootlin.com To: docs@lists.yoctoproject.org Cc: Michael Opdenacker Subject: [PATCH 6/6] test-manual: add or improve hyperlinks Date: Wed, 6 Dec 2023 16:54:27 +0100 Message-Id: <20231206155427.279612-7-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231206155427.279612-1-michael.opdenacker@bootlin.com> References: <20231206155427.279612-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 06 Dec 2023 15:54:49 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/4662 From: Michael Opdenacker Signed-off-by: Michael Opdenacker --- documentation/test-manual/test-process.rst | 32 +++++++++---------- .../test-manual/understand-autobuilder.rst | 31 +++++++++--------- 2 files changed, 31 insertions(+), 32 deletions(-) diff --git a/documentation/test-manual/test-process.rst b/documentation/test-manual/test-process.rst index 0c560efe39..7bec5ba828 100644 --- a/documentation/test-manual/test-process.rst +++ b/documentation/test-manual/test-process.rst @@ -20,8 +20,8 @@ helps review and test patches and this is his testing tree). We have two broad categories of test builds, including "full" and "quick". On the Autobuilder, these can be seen as "a-quick" and "a-full", simply for ease of sorting in the UI. Use our Autobuilder -console view to see where me manage most test-related items, available -at: :yocto_ab:`/typhoon/#/console`. +:yocto_ab:`console view ` to see where we manage most +test-related items. Builds are triggered manually when the test branches are ready. The builds are monitored by the SWAT team. For additional information, see @@ -34,18 +34,15 @@ which the result was required. The Autobuilder does build the ``master`` branch once daily for several reasons, in particular, to ensure the current ``master`` branch does -build, but also to keep ``yocto-testresults`` -(:yocto_git:`/yocto-testresults/`), -buildhistory -(:yocto_git:`/poky-buildhistory/`), and -our sstate up to date. On the weekend, there is a master-next build +build, but also to keep (:yocto_git:`yocto-testresults `), +(:yocto_git:`buildhistory `), and +our sstate up to date. On the weekend, there is a ``master-next`` build instead to ensure the test results are updated for the less frequently run targets. Performance builds (``buildperf-\*`` targets in the console) are triggered separately every six hours and automatically push their results to the -buildstats repository at: -:yocto_git:`/yocto-buildstats/`. +:yocto_git:`buildstats ` repository. The "quick" targets have been selected to be the ones which catch the most failures or give the most valuable data. We run "fast" ptests in @@ -69,10 +66,10 @@ configured to generate and publish artifacts and the milestone number, version, release candidate number and other information is entered. The box to "generate an email to QA" is also checked. -When the build completes, an email is sent out using the send-qa-email -script in the ``yocto-autobuilder-helper`` repository to the list of -people configured for that release. Release builds are placed into a -directory in https://autobuilder.yocto.io/pub/releases on the +When the build completes, an email is sent out using the ``send-qa-email`` +script in the :yocto_git:`yocto-autobuilder-helper ` +repository to the list of people configured for that release. Release builds +are placed into a directory in https://autobuilder.yocto.io/pub/releases on the Autobuilder which is included in the email. The process from here is more manual and control is effectively passed to release engineering. The next steps include: @@ -80,14 +77,15 @@ The next steps include: - QA teams respond to the email saying which tests they plan to run and when the results will be available. -- QA teams run their tests and share their results in the yocto- - testresults-contrib repository, along with a summary of their - findings. +- QA teams run their tests and share their results in the + :yocto_git:`yocto-testresults-contrib ` + repository, along with a summary of their findings. - Release engineering prepare the release as per their process. - Test results from the QA teams are included into the release in - separate directories and also uploaded to the yocto-testresults + separate directories and also uploaded to the + :yocto_git:`yocto-testresults ` repository alongside the other test results for the given revision. - The QA report in the final release is regenerated using resulttool to diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst index a3fff29aca..6b4fab4f0b 100644 --- a/documentation/test-manual/understand-autobuilder.rst +++ b/documentation/test-manual/understand-autobuilder.rst @@ -9,8 +9,8 @@ Execution Flow within the Autobuilder The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and it makes sense to follow the process through the system -starting there. This is best visualized from the Autobuilder Console -view (:yocto_ab:`/typhoon/#/console`). +starting there. This is best visualized from the :yocto_ab:`Autobuilder +Console view `. Each item along the top of that view represents some "target build" and these targets are all run in parallel. The 'full' build will trigger the @@ -18,9 +18,9 @@ majority of them, the "quick" build will trigger some subset of them. The Autobuilder effectively runs whichever configuration is defined for each of those targets on a separate buildbot worker. To understand the configuration, you need to look at the entry on ``config.json`` file -within the ``yocto-autobuilder-helper`` repository. The targets are -defined in the ‘overrides' section, a quick example could be qemux86-64 -which looks like:: +within the :yocto_git:`yocto-autobuilder-helper ` +repository. The targets are defined in the ``overrides`` section, a quick +example could be ``qemux86-64`` which looks like:: "qemux86-64" : { "MACHINE" : "qemux86-64", @@ -88,9 +88,9 @@ roughly consist of: #. *Obtain yocto-autobuilder-helper* - This step clones the ``yocto-autobuilder-helper`` git repository. - This is necessary to prevent the requirement to maintain all the - release or project-specific code within Buildbot. The branch chosen + This step clones the :yocto_git:`yocto-autobuilder-helper ` + git repository. This is necessary to avoid the requirement to maintain all + the release or project-specific code within Buildbot. The branch chosen matches the release being built so we can support older releases and still make changes in newer ones. @@ -251,13 +251,14 @@ Deploying Yocto Autobuilder =========================== The most up to date information about how to setup and deploy your own -Autobuilder can be found in README.md in the ``yocto-autobuilder2`` -repository. - -We hope that people can use the ``yocto-autobuilder2`` code directly but -it is inevitable that users will end up needing to heavily customise the -``yocto-autobuilder-helper`` repository, particularly the -``config.json`` file as they will want to define their own test matrix. +Autobuilder can be found in :yocto_git:`README.md ` +in the :yocto_git:`yocto-autobuilder2 ` repository. + +We hope that people can use the :yocto_git:`yocto-autobuilder2 ` +code directly but it is inevitable that users will end up needing to heavily +customize the :yocto_git:`yocto-autobuilder-helper ` +repository, particularly the ``config.json`` file as they will want to define +their own test matrix. The Autobuilder supports two customization options: