test-manual code block, link, and format update

Submitted by Mark Morton on Sept. 14, 2020, 11:11 p.m. | Patch ID: 176514

Details

Message ID 20200914231131.32571-1-mark.morton@windriver.com
State New
Headers show

Commit Message

Mark Morton Sept. 14, 2020, 11:11 p.m.
Signed-off-by: Mark Morton <mark.morton@windriver.com>
---
 documentation/conf.py                         |   4 +-
 .../test-manual/test-manual-intro.rst         | 266 +++++++++++-------
 .../test-manual-understand-autobuilder.rst    | 102 ++++---
 3 files changed, 226 insertions(+), 146 deletions(-)

Patch hide | download patch | download mbox

diff --git a/documentation/conf.py b/documentation/conf.py
index e1a7e5e3d..e825fbf6e 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -42,8 +42,8 @@  sys.path.insert(0, os.path.abspath('sphinx'))
 extensions = [
     'sphinx.ext.autosectionlabel',
     'sphinx.ext.extlinks',
-    'sphinx.ext.intersphinx',
-    'yocto-vars'
+    'sphinx.ext.intersphinx'
+ #   'yocto-vars'
 ]
 autosectionlabel_prefix_document = True
 
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst
index 2f11e2beb..d68ab73f4 100644
--- a/documentation/test-manual/test-manual-intro.rst
+++ b/documentation/test-manual/test-manual-intro.rst
@@ -23,26 +23,25 @@  project core.
 
 Currently, the Yocto Project Test Environment Manual has no projected
 release date. This manual is a work-in-progress and is being initially
-loaded with information from the `README <>`__ files and notes from key
+loaded with information from the README files and notes from key
 engineers:
 
--  *``yocto-autobuilder2``:* This
-   ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
+-  *yocto-autobuilder2:* This
+   `README.md <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md>`__
    is the main README which detials how to set up the Yocto Project
    Autobuilder. The ``yocto-autobuilder2`` repository represents the
    Yocto Project's console UI plugin to Buildbot and the configuration
    necessary to configure Buildbot to perform the testing the project
    requires.
 
--  *``yocto-autobuilder-helper``:* This
-   ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README
+-  *yocto-autobuilder-helper:* This `README <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README/>`__
    and repository contains Yocto Project Autobuilder Helper scripts and
    configuration. The ``yocto-autobuilder-helper`` repository contains
    the "glue" logic that defines which tests to run and how to run them.
    As a result, it can be used by any Continuous Improvement (CI) system
    to run builds, support getting the correct code revisions, configure
    builds and layers, run builds, and collect results. The code is
-   independent of any CI system, which means the code can work Buildbot,
+   independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
    Jenkins, or others. This repository has a branch per release of the
    project defining the tests to run on a per release basis.
 
@@ -66,9 +65,9 @@  that the Yocto Project customizes using code from the
 resulting UI plug-in allows you to visualize builds in a way suited to
 the project's needs.
 
-A ``helper`` layer provides configuration and job management through
+A helper layer provides configuration and job management through
 scripts found in the ``yocto-autobuilder-helper`` repository. The
-``helper`` layer contains the bulk of the build configuration
+helper layer contains the bulk of the build configuration
 information and is release-specific, which makes it highly customizable
 on a per-project basis. The layer is CI system-agnostic and contains a
 number of Helper scripts that can generate build configurations from
@@ -124,18 +123,21 @@  thefollowing types of tests:
    ensure we have no single point of failure and can ensure the
    different distros work effectively.
 
--  *eSDK Testing:* Image tests initiated through the following command:
-   $ bitbake image -c testsdkext The tests utilize the ``testsdkext``
-   class and the ``do_testsdkext`` task.
+-  *eSDK Testing:* Image tests initiated through the following command::
+
+   $ bitbake image -c testsdkext 
+   
+   The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
 
 -  *Feature Testing:* Various scenario-based tests are run through the
-   `OpenEmbedded
-   Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__
-   (oe-selftest). We test oe-selftest on each of the main distrubutions
+   :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
    we support.
 
--  *Image Testing:* Image tests initiated through the following command:
-   $ bitbake image -c testimage The tests utilize the
+-  *Image Testing:* Image tests initiated through the following command::
+
+   $ bitbake image -c testimage 
+   
+   The tests utilize the
    :ref:`testimage* <ref-classes-testimage*>`
    classes and the
    :ref:`ref-tasks-testimage` task.
@@ -147,8 +149,8 @@  thefollowing types of tests:
 
 -  *Package Testing:* A Package Test (ptest) runs tests against packages
    built by the OpenEmbedded build system on the target machine. See the
-   "`Testing Packages With
-   ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section
+   :ref:`Testing Packages With
+   ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section
    in the Yocto Project Development Tasks Manual and the
    ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
    information on Ptest.
@@ -159,8 +161,8 @@  thefollowing types of tests:
    the ``do_testsdk`` task.
 
 -  *Unit Testing:* Unit tests on various components of the system run
-   through ``oe-selftest`` and
-   ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__.
+   through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and
+   :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`.
 
 -  *Automatic Upgrade Helper:* This target tests whether new versions of
    software are available and whether we can automatically upgrade to
@@ -174,36 +176,43 @@  How Tests Map to Areas of Code
 
 Tests map into the codebase as follows:
 
--  *bitbake-selftest*:
+-  *bitbake-selftest:*
 
    These tests are self-contained and test BitBake as well as its APIs,
    which include the fetchers. The tests are located in
    ``bitbake/lib/*/tests``.
 
-   From within the BitBake repository, run the following: $
-   bitbake-selftest
+   From within the BitBake repository, run the following::   
+   
+   $ bitbake-selftest
 
    To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
-   variable when running "bitbake-selftest" as follows: $
-   BB_SKIP_NETTEST=yes bitbake-selftest
+   variable when running "bitbake-selftest" as follows:: 
+   
+   $ BB_SKIP_NETTEST=yes bitbake-selftest
 
    The default output is quiet and just prints a summary of what was
-   run. To see more information, there is a verbose option:$
-   bitbake-selftest -v
+   run. To see more information, there is a verbose option::
+    
+   $ bitbake-selftest -v
 
    Use this option when you wish to skip tests that access the network,
    which are mostly necessary to test the fetcher modules. To specify
    individual test modules to run, append the test module name to the
    "bitbake-selftest" command. For example, to specify the tests for the
-   bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can
-   also specify individual tests by defining the full name and module
-   plus the class path of the test, for example: $ bitbake-selftest
-   bb.tests.data.TestOverrides.test_one_override
+   bb.data.module, run:: 
+   
+   $ bitbake-selftest bb.test.data.module
+   
+   You can also specify individual tests by defining the full name and module
+   plus the class path of the test, for example:: 
+   
+   $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
 
    The tests are based on `Python
    unittest <https://docs.python.org/3/library/unittest.html>`__.
 
--  *oe-selftest*:
+-  *oe-selftest:*
 
    -  These tests use OE to test the workflows, which include testing
       specific features, behaviors of tasks, and API unit tests.
@@ -219,16 +228,25 @@  Tests map into the codebase as follows:
    -  The code for the tests resides in
       ``meta/lib/oeqa/selftest/cases/``.
 
-   -  To run all the tests, enter the following command: $ oe-selftest
-      -a
+   -  To run all the tests, enter the following command:: 
+   
+      $ oe-selftest -a
 
    -  To run a specific test, use the following command form where
-      testname is the name of the specific test: $ oe-selftest -r
-      testname For example, the following command would run the tinfoil
-      getVar API test:$ oe-selftest -r
-      tinfoil.TinfoilTests.test_getvarIt is also possible to run a set
+      testname is the name of the specific test:: 
+      
+      $ oe-selftest -r <testname> 
+      
+      For example, the following command would run the tinfoil
+      getVar API test::
+      
+      $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
+      
+      It is also possible to run a set
       of tests. For example the following command will run all of the
-      tinfoil tests:$ oe-selftest -r tinfoil
+      tinfoil tests::
+      
+      $ oe-selftest -r tinfoil
 
 -  *testimage:*
 
@@ -240,10 +258,13 @@  Tests map into the codebase as follows:
 
    -  You need to set the
       :term:`IMAGE_CLASSES`
-      variable as follows: IMAGE_CLASSES += "testimage"
+      variable as follows::
+      
+      IMAGE_CLASSES += "testimage"
 
-   -  Run the tests using the following command form: $ bitbake image -c
-      testimage
+   -  Run the tests using the following command form::
+   
+      $ bitbake image -c testimage
 
 -  *testsdk:*
 
@@ -252,8 +273,9 @@  Tests map into the codebase as follows:
 
    -  The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
 
-   -  Run the test using the following command form: $ bitbake image -c
-      testsdk
+   -  Run the test using the following command form::
+   
+      $ bitbake image -c testsdk
 
 -  *testsdk_ext:*
 
@@ -262,8 +284,9 @@  Tests map into the codebase as follows:
 
    -  The code for these tests resides in ``meta/lib/oeqa/esdk``.
 
-   -  To run the tests, use the following command form: $ bitbake image
-      -c testsdkext
+   -  To run the tests, use the following command form::
+   
+      $ bitbake image -c testsdkext
 
 -  *oe-build-perf-test:*
 
@@ -272,8 +295,11 @@  Tests map into the codebase as follows:
 
    -  The code for these tests resides in ``meta/lib/oeqa/buildperf``.
 
-   -  To run the tests, use the following command form: $
-      oe-build-perf-test optionsThe command takes a number of options,
+   -  To run the tests, use the following command form::
+   
+      $ oe-build-perf-test <options>
+      
+      The command takes a number of options,
       such as where to place the test results. The Autobuilder Helper
       Scripts include the ``build-perf-test-wrapper`` script with
       examples of how to use the oe-build-perf-test from the command
@@ -308,14 +334,20 @@  directory.
 ``bitbake-selftest``
 --------------------
 
-A simple test example from ``lib/bb/tests/data.py`` is: class
-DataExpansions(unittest.TestCase): def setUp(self): self.d =
-bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] =
-"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def
-test_one_var(self): val = self.d.expand("${foo}")
-self.assertEqual(str(val), "value_of_foo")
+A simple test example from ``lib/bb/tests/data.py`` is::
 
-In this example, a ```DataExpansions`` <>`__ class of tests is created,
+   class DataExpansions(unittest.TestCase): 
+      def setUp(self):
+            self.d = bb.data.init() 
+            self.d["foo"] = "value_of_foo" 
+            self.d["bar"] = "value_of_bar" 
+            self.d["value_of_foo"] = "value_of_'value_of_foo'" 
+   
+      def test_one_var(self):
+            val = self.d.expand("${foo}")
+            self.assertEqual(str(val), "value_of_foo")
+
+In this example, a ``DataExpansions`` class of tests is created,
 derived from standard python unittest. The class has a common ``setUp``
 function which is shared by all the tests in the class. A simple test is
 then added to test that when a variable is expanded, the correct value
@@ -334,14 +366,15 @@  These tests are more complex due to the setup required behind the scenes
 for full builds. Rather than directly using Python's unittest, the code
 wraps most of the standard objects. The tests can be simple, such as
 testing a command from within the OE build environment using the
-following example:class BitbakeLayers(OESelftestTestCase): def
-test_bitbakelayers_showcrossdepends(self): result =
-runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in
-result.output, msg = "No dependencies were shown. bitbake-layers
-show-cross-depends output: %s"% result.output)
+following example::
+
+   class BitbakeLayers(OESelftestTestCase): 
+      def test_bitbakelayers_showcrossdepends(self): 
+            result = runCmd('bitbake-layers show-cross-depends') 
+            self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
 
 This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
-creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived
+creates a testcase from the ``OESelftestTestCase`` class, derived
 from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
 and checks the output to ensure it contains something we know should be
 here.
@@ -367,7 +400,7 @@  so tests within a given test class should always run in the same build,
 while tests in different classes or modules may be split into different
 builds. There is no data store available for these tests since the tests
 launch the ``bitbake`` command and exist outside of its context. As a
-result, common bitbake library functions (bb.*) are also unavailable.
+result, common bitbake library functions (bb.\*) are also unavailable.
 
 .. _testimage-example:
 
@@ -378,14 +411,18 @@  These tests are run once an image is up and running, either on target
 hardware or under QEMU. As a result, they are assumed to be running in a
 target image environment, as opposed to a host build environment. A
 simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
-the following:class PythonTest(OERuntimeTestCase):
-@OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core'])
-def test_python3(self): cmd = "python3 -c \\"import codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output =
-self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output
-self.assertEqual(status, 0, msg=msg)
-
-In this example, the ```OERuntimeTestCase`` <>`__ class wraps
+the following::
+
+   class PythonTest(OERuntimeTestCase):
+      @OETestDepends(['ssh.SSHTest.test_ssh']) 
+      @OEHasPackage(['python3-core'])
+      def test_python3(self): 
+         cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" 
+         status, output = self.target.run(cmd) 
+         msg = 'Exit status was not 0. Output: %s' % output
+         self.assertEqual(status, 0, msg=msg)
+
+In this example, the ``OERuntimeTestCase`` class wraps
 ``unittest.TestCase``. Within the test, ``self.target`` represents the
 target system, where commands can be run on it using the ``run()``
 method.
@@ -402,19 +439,30 @@  the image.
 
 These tests are run against built extensible SDKs (eSDKs). The tests can
 assume that the eSDK environment has already been setup. An example from
-``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class
-DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls):
-myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst =
-os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src,
-cls.myapp_dst) subprocess.check_output(['git', 'init', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m',
-"'test commit'"], cwd=cls.myapp_dst) @classmethod def
-tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def
-\_test_devtool_build(self, directory): self._run('devtool add myapp %s'
-% directory) try: self._run('devtool build myapp') finally:
-self._run('devtool reset myapp') def test_devtool_build_make(self):
-self._test_devtool_build(self.myapp_dst)In this example, the ``devtool``
+``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
+
+   class DevtoolTest(OESDKExtTestCase): 
+      @classmethod def setUpClass(cls):
+         myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") 
+         cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") 
+         shutil.copytree(myapp_src, cls.myapp_dst)
+         subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) 
+         subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
+         subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) 
+      
+      @classmethod 
+      def tearDownClass(cls): 
+         shutil.rmtree(cls.myapp_dst) 
+      def _test_devtool_build(self, directory): 
+         self._run('devtool add myapp %s' % directory) 
+         try: 
+         self._run('devtool build myapp') 
+         finally:
+         self._run('devtool reset myapp') 
+      def test_devtool_build_make(self):
+         self._test_devtool_build(self.myapp_dst)
+      
+In this example, the ``devtool``
 command is tested to see whether a sample application can be built with
 the ``devtool build`` command within the eSDK.
 
@@ -426,14 +474,20 @@  the ``devtool build`` command within the eSDK.
 These tests are run against built SDKs. The tests can assume that an SDK
 has already been extracted and its environment file has been sourced. A
 simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
-following:class Python3Test(OESDKTestCase): def setUp(self): if not
-(self.tc.hasHostPackage("nativesdk-python3-core") or
-self.tc.hasHostPackage("python3-core-native")): raise
-unittest.SkipTest("No python3 package in the SDK") def
-test_python3(self): cmd = "python3 -c \\"import codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd)
-self.assertEqual(output, "Hello, world\n")In this example, if
-nativesdk-python3-core has been installed into the SDK, the code runs
+following::
+
+   class Python3Test(OESDKTestCase): 
+      def setUp(self): 
+            if not (self.tc.hasHostPackage("nativesdk-python3-core") or
+                  self.tc.hasHostPackage("python3-core-native")): 
+               raise unittest.SkipTest("No python3 package in the SDK") 
+   
+      def test_python3(self): 
+            cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" 
+            output = self._run(cmd)
+            self.assertEqual(output, "Hello, world\n")
+         
+In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
 the python3 interpreter with a basic command to check it is working
 correctly. The test would only run if python3 is installed in the SDK.
 
@@ -444,17 +498,25 @@  correctly. The test would only run if python3 is installed in the SDK.
 
 The performance tests usually measure how long operations take and the
 resource utilisation as that happens. An example from
-``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class
-Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake
--p)""" # Drop all caches and parse self.rm_cache()
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no
-caches)') # Drop tmp/cache
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no
-tmp/cache)') # Parse with fully cached data
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p
-(cached)')This example shows how three specific parsing timings are
+``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
+
+   class Test3(BuildPerfTestCase): 
+      def test3(self): 
+            """Bitbake parsing (bitbake -p)""" 
+            # Drop all caches and parse 
+            self.rm_cache()
+            oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+            self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 
+                     'bitbake -p (no caches)')          
+            # Drop tmp/cache
+            oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+            self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 
+                     'bitbake -p (no tmp/cache)')
+            # Parse with fully cached data
+            self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 
+                     'bitbake -p (cached)')
+                  
+This example shows how three specific parsing timings are
 measured, with and without various caches, to show how BitBake’s parsing
 performance trends over time.
 
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst
index 8d3c84415..0d987d245 100644
--- a/documentation/test-manual/test-manual-understand-autobuilder.rst
+++ b/documentation/test-manual/test-manual-understand-autobuilder.rst
@@ -20,32 +20,48 @@  each of those targets on a seperate buildbot worker. To understand the
 configuration, you need to look at the entry on ``config.json`` file
 within the ``yocto-autobuilder-helper`` repository. The targets are
 defined in the ‘overrides’ section, a quick example could be qemux86-64
-which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" :
-"arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic
-wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from
-the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" :
-true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato
-core-image-sato-dev core-image-sato-sdk core-image-minimal
-core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS"
-: "core-image-minimal:do_testimage core-image-sato:do_testimage
-core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2"
-: { "SDKMACHINE" : "x86_64", "BBTARGETS" :
-"core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext
-core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" :
-"core-image-sato:do_testsdk core-image-minimal:do_testsdkext
-core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false,
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest
-${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" :
-["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can
-see that “qemux86-64” is a three step build where the
-``bitbake BBTARGETS`` would be run, then ``bitbake
-                SANITYTARGETS`` for each step; all for
+which looks like::
+
+   "qemux86-64" : { 
+         "MACHINE" : "qemux86-64", 
+         "TEMPLATE" : "arch-qemu", 
+         "step1" : { 
+               "extravars" : [ 
+                     "IMAGE_FSTYPES_append = ' wic wic.bmap'" 
+                    ] 
+        } 
+   },
+   
+And to expand that, you need the “arch-qemu” entry from
+the “templates” section, which looks like::
+
+   "arch-qemu" : { 
+         "BUILDINFO" : true, 
+         "BUILDHISTORY" : true, 
+         "step1" : { 
+               "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", 
+         "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" 
+         }, 
+         "step2" : { 
+               "SDKMACHINE" : "x86_64", 
+               "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", 
+               "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" 
+         }, 
+         "step3" : { 
+               "BUILDHISTORY" : false, 
+               "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], 
+               "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] 
+         } 
+   },
+   
+Combining these two entries you can see that “qemux86-64” is a three step build where the
+``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
 ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step
 1 an extra variable is added to the ``auto.conf`` file to enable wic
 image generation.
 
 While not every detail of this is covered here, you can see how the
-templating mechanism allows quite complex configurations to be built up
+template mechanism allows quite complex configurations to be built up
 yet allows duplication and repetition to be kept to a minimum.
 
 The different build targets are designed to allow for parallelisation,
@@ -70,7 +86,7 @@  roughly consist of:
 
    This cleans out any previous build. Old builds are left around to
    allow easier debugging of failed builds. For additional information,
-   see ```clobberdir`` <#test-clobberdir>`__.
+   see :ref:`test-clobberdir`.
 
 2. *Obtain yocto-autobuilder-helper*
 
@@ -94,7 +110,7 @@  roughly consist of:
    from the ``layerinfo.json`` file to help understand the
    configuration. It will also use a local cache of repositories to
    speed up the clone checkouts. For additional information, see
-   `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__.
+   :ref:`test-autobuilder-clone-cache`.
 
    This step has two possible modes of operation. If the build is part
    of a parent build, its possible that all the repositories needed may
@@ -137,8 +153,7 @@  special script that moves files to a special location, rather than
 deleting them. Files in this location are deleted by an ``rm`` command,
 which is run under ``ionice -c 3``. For example, the deletion only
 happens when there is idle IO capacity on the Worker. The Autobuilder
-Worker Janitor runs this deletion. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor runs this deletion. See :ref:`test-autobuilder-worker-janitor`.
 
 .. _test-autobuilder-clone-cache:
 
@@ -150,8 +165,8 @@  on the Autobuilder. We therefore have a stash of commonly used
 repositories pre-cloned on the Workers. Data is fetched from these
 during clones first, then "topped up" with later revisions from any
 upstream when necesary. The cache is maintained by the Autobuilder
-Worker Janitor. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor. See :ref:`Autobuilder Worker
+Janitor`.
 
 .. _test-autobuilder-worker-janitor:
 
@@ -159,8 +174,8 @@  Autobuilder Worker Janitor
 --------------------------
 
 This is a process running on each Worker that performs two basic
-operations, including background file deletion at IO idle (see `Target
-Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and
+operations, including background file deletion at IO idle (see Target
+Execution: clobberdir :ref:`test-list-tgt-exec-clobberdir`) and
 maintainenance of a cache of cloned repositories to improve the speed
 the system can checkout repositories.
 
@@ -216,9 +231,6 @@  are general setup steps that are run once and include:
 For each step that is configured in ``config.json``, it will perform the
 following:
 
-## WRITER's question: What does "logging in as stepXa" and others refer
-to below? ##
-
 1. Add any layers that are specified using the
    ``bitbake-layers add-layer`` command (logging as stepXa)
 
@@ -235,9 +247,8 @@  to below? ##
 6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
    BitBake build environment (logging as stepXd)
 
-7. Remove any layers added in `step
-   1 <#test-run-config-add-layers-step>`__ using the
-   ``bitbake-layers remove-layer`` command (logging as stepXa)
+7. Remove any layers added in step
+   1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
 
 Once the execution steps above complete, ``run-config`` executes a set
 of post-build steps, including:
@@ -252,7 +263,7 @@  of post-build steps, including:
    generated to the remote server.
 
 4. Cleanup the build directory using
-   ```clobberdir`` <#test-clobberdir>`__ if the build was successful,
+   :ref:`test-clobberdir` if the build was successful,
    else rename it to “build-renamed” for potential future debugging.
 
 .. _test-deploying-yp-autobuilder:
@@ -279,11 +290,18 @@  The standard ``config.json`` minimally attempts to allow substitution of
 the paths. The Helper script repository includes a
 ``local-example.json`` file to show how you could override these from a
 separate configuration file. Pass the following into the environment of
-the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As
-another example, you could also pass the following into the
-environment:$ ABHELPER_JSON="config.json /some/location/local.json"One
-issue users often run into is validation of the ``config.json`` files. A
+the Autobuilder::
+
+   $ ABHELPER_JSON="config.json local-example.json"
+   
+As another example, you could also pass the following into the
+environment::
+
+   $ ABHELPER_JSON="config.json /some/location/local.json"
+   
+One issue users often run into is validation of the ``config.json`` files. A
 tip for minimizing issues from invalid json files is to use a Git
 ``pre-commit-hook.sh`` script to verify the JSON file before committing
-it. Create a symbolic link as follows:$ ln -s
-../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
+it. Create a symbolic link as follows:: 
+
+   $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit

Comments

Nicolas Dechesne Sept. 15, 2020, 6:08 a.m.
hi Mark,

many thanks for your help!

On Tue, Sep 15, 2020 at 1:16 AM Mark Morton <Mark.Morton@windriver.com>
wrote:

> Signed-off-by: Mark Morton <mark.morton@windriver.com>
> ---
>  documentation/conf.py                         |   4 +-
>  .../test-manual/test-manual-intro.rst         | 266 +++++++++++-------
>  .../test-manual-understand-autobuilder.rst    | 102 ++++---
>  3 files changed, 226 insertions(+), 146 deletions(-)
>
> diff --git a/documentation/conf.py b/documentation/conf.py
> index e1a7e5e3d..e825fbf6e 100644
> --- a/documentation/conf.py
> +++ b/documentation/conf.py
> @@ -42,8 +42,8 @@ sys.path.insert(0, os.path.abspath('sphinx'))
>  extensions = [
>      'sphinx.ext.autosectionlabel',
>      'sphinx.ext.extlinks',
> -    'sphinx.ext.intersphinx',
> -    'yocto-vars'
> +    'sphinx.ext.intersphinx'
> + #   'yocto-vars'
>  ]
>

This extension doesn't work with Python2. To be honest, I don't think i
want to bother with Python2 at this point. I am not even sure how well
Sphinx is expected to work with Python2, so we should make clear  that our
doc system requires Python3, and we need to write clear instructions for
users who have Python3 on their host as 'default' vs users who still have
Python2 (which is the problem you faced)

 autosectionlabel_prefix_document = True
>
> diff --git a/documentation/test-manual/test-manual-intro.rst
> b/documentation/test-manual/test-manual-intro.rst
> index 2f11e2beb..d68ab73f4 100644
> --- a/documentation/test-manual/test-manual-intro.rst
> +++ b/documentation/test-manual/test-manual-intro.rst
> @@ -23,26 +23,25 @@ project core.
>
>  Currently, the Yocto Project Test Environment Manual has no projected
>  release date. This manual is a work-in-progress and is being initially
> -loaded with information from the `README <>`__ files and notes from key
> +loaded with information from the README files and notes from key
>  engineers:
>
> --  *``yocto-autobuilder2``:* This
> -   ```README.md`
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
> +-  *yocto-autobuilder2:* This
> +   `README.md <
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
> >`__
>

that works. However for links to git trees, we are using the :yocto_git:
extlink , such as (from the BSP guide):
:yocto_git:`README.md </cgit.cgi/meta-raspberrypi/tree/README.md>`
We should standardize on that.
I think i will eventually make sure we don't even need /cgit.git in the
link name, but for now it's needed.

    is the main README which detials how to set up the Yocto Project
>     Autobuilder. The ``yocto-autobuilder2`` repository represents the
>     Yocto Project's console UI plugin to Buildbot and the configuration
>     necessary to configure Buildbot to perform the testing the project
>     requires.
>
> --  *``yocto-autobuilder-helper``:* This
> -   ```README`
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README
> +-  *yocto-autobuilder-helper:* This `README <
> http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README/
> >`__
>     and repository contains Yocto Project Autobuilder Helper scripts and
>     configuration. The ``yocto-autobuilder-helper`` repository contains
>     the "glue" logic that defines which tests to run and how to run them.
>     As a result, it can be used by any Continuous Improvement (CI) system
>     to run builds, support getting the correct code revisions, configure
>     builds and layers, run builds, and collect results. The code is
> -   independent of any CI system, which means the code can work Buildbot,
> +   independent of any CI system, which means the code can work `Buildbot <
> https://docs.buildbot.net/0.9.15.post1/>`__,
>

this is a new link you created which was not in the original Docbook,
right? Perhaps best to have that in a separate patch if you think it's
needed.


>     Jenkins, or others. This repository has a branch per release of the
>     project defining the tests to run on a per release basis.
>
> @@ -66,9 +65,9 @@ that the Yocto Project customizes using code from the
>  resulting UI plug-in allows you to visualize builds in a way suited to
>  the project's needs.
>
> -A ``helper`` layer provides configuration and job management through
> +A helper layer provides configuration and job management through
>

Why? 'helper' is inside a <filename> tag in docbook to make it verbatim in
the output, why did you remove it?

 scripts found in the ``yocto-autobuilder-helper`` repository. The
> -``helper`` layer contains the bulk of the build configuration
> +helper layer contains the bulk of the build configuration
>  information and is release-specific, which makes it highly customizable
>  on a per-project basis. The layer is CI system-agnostic and contains a
>  number of Helper scripts that can generate build configurations from
> @@ -124,18 +123,21 @@ thefollowing types of tests:
>     ensure we have no single point of failure and can ensure the
>     different distros work effectively.
>
> --  *eSDK Testing:* Image tests initiated through the following command:
> -   $ bitbake image -c testsdkext The tests utilize the ``testsdkext``
> -   class and the ``do_testsdkext`` task.
> +-  *eSDK Testing:* Image tests initiated through the following command::
>

does that work? I've always done ": ::" to keep the semi column, and I
thought the :: needed a space before.

> +
> +   $ bitbake image -c testsdkext
>
Indentation is not typical here. you need 3 space starting from the
previous indentation level.

> +
> +   The tests utilize the ``testsdkext`` class and the ``do_testsdkext``
> task.
>
this sentence above is no in the code-block, it should not be at the same
indentation level as the code block.


>
>  -  *Feature Testing:* Various scenario-based tests are run through the
> -   `OpenEmbedded
> -   Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__
> -   (oe-selftest). We test oe-selftest on each of the main distrubutions
> +   :ref:`OpenEmbedded Self test (oe-selftest)
> <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test
> oe-selftest on each of the main distrubutions
>     we support.
>

> --  *Image Testing:* Image tests initiated through the following command:
> -   $ bitbake image -c testimage The tests utilize the
> +-  *Image Testing:* Image tests initiated through the following command::
> +
> +   $ bitbake image -c testimage
> +
> +   The tests utilize the
>
Need to fix indentation here

    :ref:`testimage* <ref-classes-testimage*>`
>     classes and the
>     :ref:`ref-tasks-testimage` task.
> @@ -147,8 +149,8 @@ thefollowing types of tests:
>
>  -  *Package Testing:* A Package Test (ptest) runs tests against packages
>     built by the OpenEmbedded build system on the target machine. See the
> -   "`Testing Packages With
> -   ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section
> +   :ref:`Testing Packages With
> +   ptest <dev-manual/dev-manual-common-tasks:Testing Packages With
> ptest>` section
>     in the Yocto Project Development Tasks Manual and the
>     ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
>     information on Ptest.
> @@ -159,8 +161,8 @@ thefollowing types of tests:
>     the ``do_testsdk`` task.
>
>  -  *Unit Testing:* Unit tests on various components of the system run
> -   through ``oe-selftest`` and
> -   ```bitbake-selftest``
> <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__.
> +   through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing
> and Quality Assurance>` and
> +   :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality
> Assurance>`.
>

the link is on bitbake-selftest, not oe-selftest (you added it).


>
>  -  *Automatic Upgrade Helper:* This target tests whether new versions of
>     software are available and whether we can automatically upgrade to
> @@ -174,36 +176,43 @@ How Tests Map to Areas of Code
>
>  Tests map into the codebase as follows:
>
> --  *bitbake-selftest*:
> +-  *bitbake-selftest:*
>
>     These tests are self-contained and test BitBake as well as its APIs,
>     which include the fetchers. The tests are located in
>     ``bitbake/lib/*/tests``.
>
> -   From within the BitBake repository, run the following: $
> -   bitbake-selftest
> +   From within the BitBake repository, run the following::
> +
> +   $ bitbake-selftest
>

indentation and use of :: are not right (please check all other occurrences)


>
>     To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
> -   variable when running "bitbake-selftest" as follows: $
> -   BB_SKIP_NETTEST=yes bitbake-selftest
> +   variable when running "bitbake-selftest" as follows::
> +
> +   $ BB_SKIP_NETTEST=yes bitbake-selftest
>
>     The default output is quiet and just prints a summary of what was
> -   run. To see more information, there is a verbose option:$
> -   bitbake-selftest -v
> +   run. To see more information, there is a verbose option::
> +
> +   $ bitbake-selftest -v
>
>     Use this option when you wish to skip tests that access the network,
>     which are mostly necessary to test the fetcher modules. To specify
>     individual test modules to run, append the test module name to the
>     "bitbake-selftest" command. For example, to specify the tests for the
> -   bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can
> -   also specify individual tests by defining the full name and module
> -   plus the class path of the test, for example: $ bitbake-selftest
> -   bb.tests.data.TestOverrides.test_one_override
> +   bb.data.module, run::
> +
> +   $ bitbake-selftest bb.test.data.module
> +
> +   You can also specify individual tests by defining the full name and
> module
> +   plus the class path of the test, for example::
> +
> +   $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
>
>     The tests are based on `Python
>     unittest <https://docs.python.org/3/library/unittest.html>`__.
>
> --  *oe-selftest*:
> +-  *oe-selftest:*
>
>     -  These tests use OE to test the workflows, which include testing
>        specific features, behaviors of tasks, and API unit tests.
> @@ -219,16 +228,25 @@ Tests map into the codebase as follows:
>     -  The code for the tests resides in
>        ``meta/lib/oeqa/selftest/cases/``.
>
> -   -  To run all the tests, enter the following command: $ oe-selftest
> -      -a
> +   -  To run all the tests, enter the following command::
> +
> +      $ oe-selftest -a
>
>     -  To run a specific test, use the following command form where
> -      testname is the name of the specific test: $ oe-selftest -r
> -      testname For example, the following command would run the tinfoil
> -      getVar API test:$ oe-selftest -r
> -      tinfoil.TinfoilTests.test_getvarIt is also possible to run a set
> +      testname is the name of the specific test::
> +
> +      $ oe-selftest -r <testname>
> +
> +      For example, the following command would run the tinfoil
> +      getVar API test::
> +
> +      $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
> +
> +      It is also possible to run a set
>        of tests. For example the following command will run all of the
> -      tinfoil tests:$ oe-selftest -r tinfoil
> +      tinfoil tests::
> +
> +      $ oe-selftest -r tinfoil
>
>  -  *testimage:*
>
> @@ -240,10 +258,13 @@ Tests map into the codebase as follows:
>
>     -  You need to set the
>        :term:`IMAGE_CLASSES`
> -      variable as follows: IMAGE_CLASSES += "testimage"
> +      variable as follows::
> +
> +      IMAGE_CLASSES += "testimage"
>
> -   -  Run the tests using the following command form: $ bitbake image -c
> -      testimage
> +   -  Run the tests using the following command form::
> +
> +      $ bitbake image -c testimage
>
>  -  *testsdk:*
>
> @@ -252,8 +273,9 @@ Tests map into the codebase as follows:
>
>     -  The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
>
> -   -  Run the test using the following command form: $ bitbake image -c
> -      testsdk
> +   -  Run the test using the following command form::
> +
> +      $ bitbake image -c testsdk
>
>  -  *testsdk_ext:*
>
> @@ -262,8 +284,9 @@ Tests map into the codebase as follows:
>
>     -  The code for these tests resides in ``meta/lib/oeqa/esdk``.
>
> -   -  To run the tests, use the following command form: $ bitbake image
> -      -c testsdkext
> +   -  To run the tests, use the following command form::
> +
> +      $ bitbake image -c testsdkext
>
>  -  *oe-build-perf-test:*
>
> @@ -272,8 +295,11 @@ Tests map into the codebase as follows:
>
>     -  The code for these tests resides in ``meta/lib/oeqa/buildperf``.
>
> -   -  To run the tests, use the following command form: $
> -      oe-build-perf-test optionsThe command takes a number of options,
> +   -  To run the tests, use the following command form::
> +
> +      $ oe-build-perf-test <options>
> +
> +      The command takes a number of options,
>        such as where to place the test results. The Autobuilder Helper
>        Scripts include the ``build-perf-test-wrapper`` script with
>        examples of how to use the oe-build-perf-test from the command
> @@ -308,14 +334,20 @@ directory.
>  ``bitbake-selftest``
>  --------------------
>
> -A simple test example from ``lib/bb/tests/data.py`` is: class
> -DataExpansions(unittest.TestCase): def setUp(self): self.d =
> -bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] =
> -"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def
> -test_one_var(self): val = self.d.expand("${foo}")
> -self.assertEqual(str(val), "value_of_foo")
> +A simple test example from ``lib/bb/tests/data.py`` is::
>
> -In this example, a ```DataExpansions`` <>`__ class of tests is created,
> +   class DataExpansions(unittest.TestCase):
> +      def setUp(self):
> +            self.d = bb.data.init()
> +            self.d["foo"] = "value_of_foo"
> +            self.d["bar"] = "value_of_bar"
> +            self.d["value_of_foo"] = "value_of_'value_of_foo'"
> +
> +      def test_one_var(self):
> +            val = self.d.expand("${foo}")
> +            self.assertEqual(str(val), "value_of_foo")
> +
> +In this example, a ``DataExpansions`` class of tests is created,
>  derived from standard python unittest. The class has a common ``setUp``
>  function which is shared by all the tests in the class. A simple test is
>  then added to test that when a variable is expanded, the correct value
> @@ -334,14 +366,15 @@ These tests are more complex due to the setup
> required behind the scenes
>  for full builds. Rather than directly using Python's unittest, the code
>  wraps most of the standard objects. The tests can be simple, such as
>  testing a command from within the OE build environment using the
> -following example:class BitbakeLayers(OESelftestTestCase): def
> -test_bitbakelayers_showcrossdepends(self): result =
> -runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in
> -result.output, msg = "No dependencies were shown. bitbake-layers
> -show-cross-depends output: %s"% result.output)
> +following example::
> +
> +   class BitbakeLayers(OESelftestTestCase):
> +      def test_bitbakelayers_showcrossdepends(self):
> +            result = runCmd('bitbake-layers show-cross-depends')
> +            self.assertTrue('aspell' in result.output, msg = "No
> dependencies were shown. bitbake-layers show-cross-depends output: %s"%
> result.output)
>
>  This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
> -creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived
> +creates a testcase from the ``OESelftestTestCase`` class, derived
>  from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
>  and checks the output to ensure it contains something we know should be
>  here.
> @@ -367,7 +400,7 @@ so tests within a given test class should always run
> in the same build,
>  while tests in different classes or modules may be split into different
>  builds. There is no data store available for these tests since the tests
>  launch the ``bitbake`` command and exist outside of its context. As a
> -result, common bitbake library functions (bb.*) are also unavailable.
> +result, common bitbake library functions (bb.\*) are also unavailable.
>
>  .. _testimage-example:
>
> @@ -378,14 +411,18 @@ These tests are run once an image is up and running,
> either on target
>  hardware or under QEMU. As a result, they are assumed to be running in a
>  target image environment, as opposed to a host build environment. A
>  simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
> -the following:class PythonTest(OERuntimeTestCase):
> -@OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core'])
> -def test_python3(self): cmd = "python3 -c \\"import codecs;
> -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output =
> -self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output
> -self.assertEqual(status, 0, msg=msg)
> -
> -In this example, the ```OERuntimeTestCase`` <>`__ class wraps
> +the following::
> +
> +   class PythonTest(OERuntimeTestCase):
> +      @OETestDepends(['ssh.SSHTest.test_ssh'])
> +      @OEHasPackage(['python3-core'])
> +      def test_python3(self):
> +         cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb,
> jbeyq', 'rot13'))\""
> +         status, output = self.target.run(cmd)
> +         msg = 'Exit status was not 0. Output: %s' % output
> +         self.assertEqual(status, 0, msg=msg)
> +
> +In this example, the ``OERuntimeTestCase`` class wraps
>  ``unittest.TestCase``. Within the test, ``self.target`` represents the
>  target system, where commands can be run on it using the ``run()``
>  method.
> @@ -402,19 +439,30 @@ the image.
>
>  These tests are run against built extensible SDKs (eSDKs). The tests can
>  assume that the eSDK environment has already been setup. An example from
> -``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class
> -DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls):
> -myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst =
> -os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src,
> -cls.myapp_dst) subprocess.check_output(['git', 'init', '.'],
> -cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'],
> -cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m',
> -"'test commit'"], cwd=cls.myapp_dst) @classmethod def
> -tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def
> -\_test_devtool_build(self, directory): self._run('devtool add myapp %s'
> -% directory) try: self._run('devtool build myapp') finally:
> -self._run('devtool reset myapp') def test_devtool_build_make(self):
> -self._test_devtool_build(self.myapp_dst)In this example, the ``devtool``
> +``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
> +
> +   class DevtoolTest(OESDKExtTestCase):
> +      @classmethod def setUpClass(cls):
> +         myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
> +         cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
> +         shutil.copytree(myapp_src, cls.myapp_dst)
> +         subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
> +         subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
> +         subprocess.check_output(['git', 'commit', '-m', "'test
> commit'"], cwd=cls.myapp_dst)
> +
> +      @classmethod
> +      def tearDownClass(cls):
> +         shutil.rmtree(cls.myapp_dst)
> +      def _test_devtool_build(self, directory):
> +         self._run('devtool add myapp %s' % directory)
> +         try:
> +         self._run('devtool build myapp')
> +         finally:
> +         self._run('devtool reset myapp')
> +      def test_devtool_build_make(self):
> +         self._test_devtool_build(self.myapp_dst)
> +
> +In this example, the ``devtool``
>  command is tested to see whether a sample application can be built with
>  the ``devtool build`` command within the eSDK.
>
> @@ -426,14 +474,20 @@ the ``devtool build`` command within the eSDK.
>  These tests are run against built SDKs. The tests can assume that an SDK
>  has already been extracted and its environment file has been sourced. A
>  simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
> -following:class Python3Test(OESDKTestCase): def setUp(self): if not
> -(self.tc.hasHostPackage("nativesdk-python3-core") or
> -self.tc.hasHostPackage("python3-core-native")): raise
> -unittest.SkipTest("No python3 package in the SDK") def
> -test_python3(self): cmd = "python3 -c \\"import codecs;
> -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd)
> -self.assertEqual(output, "Hello, world\n")In this example, if
> -nativesdk-python3-core has been installed into the SDK, the code runs
> +following::
> +
> +   class Python3Test(OESDKTestCase):
> +      def setUp(self):
> +            if not (self.tc.hasHostPackage("nativesdk-python3-core") or
> +                  self.tc.hasHostPackage("python3-core-native")):
> +               raise unittest.SkipTest("No python3 package in the SDK")
> +
> +      def test_python3(self):
> +            cmd = "python3 -c \\"import codecs;
> print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
> +            output = self._run(cmd)
> +            self.assertEqual(output, "Hello, world\n")
> +
> +In this example, if nativesdk-python3-core has been installed into the
> SDK, the code runs
>  the python3 interpreter with a basic command to check it is working
>  correctly. The test would only run if python3 is installed in the SDK.
>
> @@ -444,17 +498,25 @@ correctly. The test would only run if python3 is
> installed in the SDK.
>
>  The performance tests usually measure how long operations take and the
>  resource utilisation as that happens. An example from
> -``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class
> -Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake
> --p)""" # Drop all caches and parse self.rm_cache()
> -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
> -self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no
> -caches)') # Drop tmp/cache
> -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
> -self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no
> -tmp/cache)') # Parse with fully cached data
> -self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p
> -(cached)')This example shows how three specific parsing timings are
> +``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
> +
> +   class Test3(BuildPerfTestCase):
> +      def test3(self):
> +            """Bitbake parsing (bitbake -p)"""
> +            # Drop all caches and parse
> +            self.rm_cache()
> +            oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),
> True)
> +            self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
> +                     'bitbake -p (no caches)')
> +            # Drop tmp/cache
> +            oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),
> True)
> +            self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
> +                     'bitbake -p (no tmp/cache)')
> +            # Parse with fully cached data
> +            self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
> +                     'bitbake -p (cached)')
> +
> +This example shows how three specific parsing timings are
>  measured, with and without various caches, to show how BitBake’s parsing
>  performance trends over time.
>
> diff --git
> a/documentation/test-manual/test-manual-understand-autobuilder.rst
> b/documentation/test-manual/test-manual-understand-autobuilder.rst
> index 8d3c84415..0d987d245 100644
> --- a/documentation/test-manual/test-manual-understand-autobuilder.rst
> +++ b/documentation/test-manual/test-manual-understand-autobuilder.rst
> @@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker.
> To understand the
>  configuration, you need to look at the entry on ``config.json`` file
>  within the ``yocto-autobuilder-helper`` repository. The targets are
>  defined in the ‘overrides’ section, a quick example could be qemux86-64
> -which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" :
> -"arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic
> -wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from
> -the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" :
> -true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato
> -core-image-sato-dev core-image-sato-sdk core-image-minimal
> -core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS"
> -: "core-image-minimal:do_testimage core-image-sato:do_testimage
> -core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2"
> -: { "SDKMACHINE" : "x86_64", "BBTARGETS" :
> -"core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext
> -core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" :
> -"core-image-sato:do_testsdk core-image-minimal:do_testsdkext
> -core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false,
> -"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest
> -${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" :
> -["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can
> -see that “qemux86-64” is a three step build where the
> -``bitbake BBTARGETS`` would be run, then ``bitbake
> -                SANITYTARGETS`` for each step; all for
> +which looks like::
> +
> +   "qemux86-64" : {
> +         "MACHINE" : "qemux86-64",
> +         "TEMPLATE" : "arch-qemu",
> +         "step1" : {
> +               "extravars" : [
> +                     "IMAGE_FSTYPES_append = ' wic wic.bmap'"
> +                    ]
> +        }
> +   },
> +
> +And to expand that, you need the “arch-qemu” entry from
> +the “templates” section, which looks like::
> +
> +   "arch-qemu" : {
> +         "BUILDINFO" : true,
> +         "BUILDHISTORY" : true,
> +         "step1" : {
> +               "BBTARGETS" : "core-image-sato core-image-sato-dev
> core-image-sato-sdk core-image-minimal core-image-minimal-dev
> core-image-sato:do_populate_sdk",
> +         "SANITYTARGETS" : "core-image-minimal:do_testimage
> core-image-sato:do_testimage core-image-sato-sdk:do_testimage
> core-image-sato:do_testsdk"
> +         },
> +         "step2" : {
> +               "SDKMACHINE" : "x86_64",
> +               "BBTARGETS" : "core-image-sato:do_populate_sdk
> core-image-minimal:do_populate_sdk_ext
> core-image-sato:do_populate_sdk_ext",
> +               "SANITYTARGETS" : "core-image-sato:do_testsdk
> core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
> +         },
> +         "step3" : {
> +               "BUILDHISTORY" : false,
> +               "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1
> oe-selftest ${HELPERSTMACHTARGS} -j 15"],
> +               "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
> +         }
> +   },
> +
> +Combining these two entries you can see that “qemux86-64” is a three step
> build where the
> +``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for
> each step; all for
>  ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step
>  1 an extra variable is added to the ``auto.conf`` file to enable wic
>  image generation.
>
>  While not every detail of this is covered here, you can see how the
> -templating mechanism allows quite complex configurations to be built up
> +template mechanism allows quite complex configurations to be built up
>

why? let's stick to the original .xml file, we can fix the doc later.


>  yet allows duplication and repetition to be kept to a minimum.
>
>  The different build targets are designed to allow for parallelisation,
> @@ -70,7 +86,7 @@ roughly consist of:
>
>     This cleans out any previous build. Old builds are left around to
>     allow easier debugging of failed builds. For additional information,
> -   see ```clobberdir`` <#test-clobberdir>`__.
> +   see :ref:`test-clobberdir`.
>

It works. However we are now using autosectionlabel which creates sections
labels for headings automatically. So I would like to get rid of all custom
labels, as much as possible. the following is preferred:
:ref:`test-manual/test-manual-understand-autobuilder:clobberdir>`

I know we've left some links like that in other manuals. So it's not
strictly required to change it now.


>  2. *Obtain yocto-autobuilder-helper*
>
> @@ -94,7 +110,7 @@ roughly consist of:
>     from the ``layerinfo.json`` file to help understand the
>     configuration. It will also use a local cache of repositories to
>     speed up the clone checkouts. For additional information, see
> -   `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__.
> +   :ref:`test-autobuilder-clone-cache`.
>

same here (and probably in other places)


>
>     This step has two possible modes of operation. If the build is part
>     of a parent build, its possible that all the repositories needed may
> @@ -137,8 +153,7 @@ special script that moves files to a special location,
> rather than
>  deleting them. Files in this location are deleted by an ``rm`` command,
>  which is run under ``ionice -c 3``. For example, the deletion only
>  happens when there is idle IO capacity on the Worker. The Autobuilder
> -Worker Janitor runs this deletion. See `Autobuilder Worker
> -Janitor <#test-autobuilder-worker-janitor>`__.
> +Worker Janitor runs this deletion. See
> :ref:`test-autobuilder-worker-janitor`.
>
>  .. _test-autobuilder-clone-cache:
>
> @@ -150,8 +165,8 @@ on the Autobuilder. We therefore have a stash of
> commonly used
>  repositories pre-cloned on the Workers. Data is fetched from these
>  during clones first, then "topped up" with later revisions from any
>  upstream when necesary. The cache is maintained by the Autobuilder
> -Worker Janitor. See `Autobuilder Worker
> -Janitor <#test-autobuilder-worker-janitor>`__.
> +Worker Janitor. See :ref:`Autobuilder Worker
> +Janitor`.
>
>  .. _test-autobuilder-worker-janitor:
>
> @@ -159,8 +174,8 @@ Autobuilder Worker Janitor
>  --------------------------
>
>  This is a process running on each Worker that performs two basic
> -operations, including background file deletion at IO idle (see `Target
> -Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and
> +operations, including background file deletion at IO idle (see Target
> +Execution: clobberdir :ref:`test-list-tgt-exec-clobberdir`) and
>

the link and text look wrong here. the link is not on the 'text'


>  maintainenance of a cache of cloned repositories to improve the speed
>  the system can checkout repositories.
>
> @@ -216,9 +231,6 @@ are general setup steps that are run once and include:
>  For each step that is configured in ``config.json``, it will perform the
>  following:
>
> -## WRITER's question: What does "logging in as stepXa" and others refer
> -to below? ##
>

you can keep comment with the ".. xxx" syntax.


> -
>  1. Add any layers that are specified using the
>     ``bitbake-layers add-layer`` command (logging as stepXa)
>
> @@ -235,9 +247,8 @@ to below? ##
>  6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
>     BitBake build environment (logging as stepXd)
>
> -7. Remove any layers added in `step
> -   1 <#test-run-config-add-layers-step>`__ using the
> -   ``bitbake-layers remove-layer`` command (logging as stepXa)
> +7. Remove any layers added in step
> +   1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
>

you removed a link here, no?

>
>  Once the execution steps above complete, ``run-config`` executes a set
>  of post-build steps, including:
> @@ -252,7 +263,7 @@ of post-build steps, including:
>     generated to the remote server.
>
>  4. Cleanup the build directory using
> -   ```clobberdir`` <#test-clobberdir>`__ if the build was successful,
> +   :ref:`test-clobberdir` if the build was successful,
>     else rename it to “build-renamed” for potential future debugging.
>
>  .. _test-deploying-yp-autobuilder:
> @@ -279,11 +290,18 @@ The standard ``config.json`` minimally attempts to
> allow substitution of
>  the paths. The Helper script repository includes a
>  ``local-example.json`` file to show how you could override these from a
>  separate configuration file. Pass the following into the environment of
> -the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As
> -another example, you could also pass the following into the
> -environment:$ ABHELPER_JSON="config.json /some/location/local.json"One
> -issue users often run into is validation of the ``config.json`` files. A
> +the Autobuilder::
> +
> +   $ ABHELPER_JSON="config.json local-example.json"
> +
> +As another example, you could also pass the following into the
> +environment::
> +
> +   $ ABHELPER_JSON="config.json /some/location/local.json"
> +
> +One issue users often run into is validation of the ``config.json``
> files. A
>  tip for minimizing issues from invalid json files is to use a Git
>  ``pre-commit-hook.sh`` script to verify the JSON file before committing
> -it. Create a symbolic link as follows:$ ln -s
> -../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
> +it. Create a symbolic link as follows::
> +
> +   $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
> --
> 2.17.1
>
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#354): https://lists.yoctoproject.org/g/docs/message/354
Mute This Topic: https://lists.yoctoproject.org/mt/76854579/3617530
Group Owner: docs+owner@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Mark Morton Sept. 15, 2020, 7:06 p.m.
From: Nicolas Dechesne <nicolas.dechesne@linaro.org>

Sent: Monday, September 14, 2020 11:08 PM
To: Morton, Mark <Mark.Morton@windriver.com>
Cc: docs@lists.yoctoproject.org
Subject: Re: [docs] [PATCH] test-manual code block, link, and format update

hi Mark,

many thanks for your help!

On Tue, Sep 15, 2020 at 1:16 AM Mark Morton <Mark.Morton@windriver.com<mailto:Mark.Morton@windriver.com>> wrote:
Signed-off-by: Mark Morton <mark.morton@windriver.com<mailto:mark.morton@windriver.com>>

---
 documentation/conf.py                         |   4 +-
 .../test-manual/test-manual-intro.rst         | 266 +++++++++++-------
 .../test-manual-understand-autobuilder.rst    | 102 ++++---
 3 files changed, 226 insertions(+), 146 deletions(-)

It’s just more grammatically correct. I feel like I should have caught this the first time around ☺

 yet allows duplication and repetition to be kept to a minimum.

 The different build targets are designed to allow for parallelisation,
@@ -70,7 +86,7 @@ roughly consist of:

    This cleans out any previous build. Old builds are left around to
    allow easier debugging of failed builds. For additional information,
-   see ```clobberdir`` <#test-clobberdir>`__.
+   see :ref:`test-clobberdir`.

It works. However we are now using autosectionlabel which creates sections labels for headings automatically. So I would like to get rid of all custom labels, as much as possible. the following is preferred:
:ref:`test-manual/test-manual-understand-autobuilder:clobberdir>`

I’ve updated all these instances

I know we've left some links like that in other manuals. So it's not strictly required to change it now.


 2. *Obtain yocto-autobuilder-helper*

@@ -94,7 +110,7 @@ roughly consist of:
    from the ``layerinfo.json`` file to help understand the
    configuration. It will also use a local cache of repositories to
    speed up the clone checkouts. For additional information, see
-   `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__.
+   :ref:`test-autobuilder-clone-cache`.

same here (and probably in other places)

I updated all of them


    This step has two possible modes of operation. If the build is part
    of a parent build, its possible that all the repositories needed may
@@ -137,8 +153,7 @@ special script that moves files to a special location, rather than
 deleting them. Files in this location are deleted by an ``rm`` command,
 which is run under ``ionice -c 3``. For example, the deletion only
 happens when there is idle IO capacity on the Worker. The Autobuilder
-Worker Janitor runs this deletion. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor runs this deletion. See :ref:`test-autobuilder-worker-janitor`.

 .. _test-autobuilder-clone-cache:

@@ -150,8 +165,8 @@ on the Autobuilder. We therefore have a stash of commonly used
 repositories pre-cloned on the Workers. Data is fetched from these
 during clones first, then "topped up" with later revisions from any
 upstream when necesary. The cache is maintained by the Autobuilder
-Worker Janitor. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor. See :ref:`Autobuilder Worker
+Janitor`.

 .. _test-autobuilder-worker-janitor:

@@ -159,8 +174,8 @@ Autobuilder Worker Janitor
 --------------------------

 This is a process running on each Worker that performs two basic
-operations, including background file deletion at IO idle (see `Target
-Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and
+operations, including background file deletion at IO idle (see Target
+Execution: clobberdir :ref:`test-list-tgt-exec-clobberdir`) and

the link and text look wrong here. the link is not on the 'text'

It was to an old anchor link for the step that did not convert. I updated the link to point to the main section.

 maintainenance of a cache of cloned repositories to improve the speed
 the system can checkout repositories.

@@ -216,9 +231,6 @@ are general setup steps that are run once and include:
 For each step that is configured in ``config.json``, it will perform the
 following:

-## WRITER's question: What does "logging in as stepXa" and others refer
-to below? ##

you can keep comment with the ".. xxx" syntax.

It was no longer necessary – should I add it back in?

-
 1. Add any layers that are specified using the
    ``bitbake-layers add-layer`` command (logging as stepXa)

@@ -235,9 +247,8 @@ to below? ##
 6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
    BitBake build environment (logging as stepXd)

-7. Remove any layers added in `step
-   1 <#test-run-config-add-layers-step>`__ using the
-   ``bitbake-layers remove-layer`` command (logging as stepXa)
+7. Remove any layers added in step
+   1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)

you removed a link here, no?

Yes – it pointed to a step, so I just used the step number instead, as it’s part of the same procedure.

 Once the execution steps above complete, ``run-config`` executes a set
 of post-build steps, including:
@@ -252,7 +263,7 @@ of post-build steps, including:
    generated to the remote server.

 4. Cleanup the build directory using
-   ```clobberdir`` <#test-clobberdir>`__ if the build was successful,
+   :ref:`test-clobberdir` if the build was successful,
    else rename it to “build-renamed” for potential future debugging.

 .. _test-deploying-yp-autobuilder:
@@ -279,11 +290,18 @@ The standard ``config.json`` minimally attempts to allow substitution of
 the paths. The Helper script repository includes a
 ``local-example.json`` file to show how you could override these from a
 separate configuration file. Pass the following into the environment of
-the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As
-another example, you could also pass the following into the
-environment:$ ABHELPER_JSON="config.json /some/location/local.json"One
-issue users often run into is validation of the ``config.json`` files. A
+the Autobuilder::
+
+   $ ABHELPER_JSON="config.json local-example.json"
+
+As another example, you could also pass the following into the
+environment::
+
+   $ ABHELPER_JSON="config.json /some/location/local.json"
+
+One issue users often run into is validation of the ``config.json`` files. A
 tip for minimizing issues from invalid json files is to use a Git
 ``pre-commit-hook.sh`` script to verify the JSON file before committing
-it. Create a symbolic link as follows:$ ln -s
-../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
+it. Create a symbolic link as follows::
+
+   $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#359): https://lists.yoctoproject.org/g/docs/message/359
Mute This Topic: https://lists.yoctoproject.org/mt/76854579/3617530
Group Owner: docs+owner@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-