| Message ID | 20240425154607.566716-1-michael.opdenacker@bootlin.com |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC] oeqa/runtime/cases: new image_upgrade test | expand |
Hi Michael, At least at a first read and without running it, this does look like a reasonable direction. I suspect that anyone else looking at this would have a lot of questions about why we'd do it this way but given the various constraints, it does make sense to me. What we do need to think about is how someone else would reuse this as currently it is very poky specific. Our aim is to make it easy for others to use too. With that in mind: * We probably want to "tag" this test with something so we can exclude it from the normal oe-selftest runs on the autobuilder and allow it to run on a per machine basis. There are other oe-selftests we already do this with (like toolchain testing or machine specific environment file tests). * The configuration about what to test probably needs to come from the distro (i.e. which DISTRO/MACHINE/image combinations). * We probably need to parameterise it so that a list of images can be tested rather than just a single one. I did wonder if we could have it dynamically add tests for each image configured. * We don't want to test on all MACHINE (e.g. qemumips and qemuppc are not going to be included). * We need to find a better way to share the code with autobuilder- helper, I don't like duplicating code. * The image url code is also highly poky specific. That probably needs to come from the poky repository alongside the configuration. Does that all make sense? Cheers, Richard
On Thu, 25 Apr 2024 at 17:46, Michael Opdenacker via lists.openembedded.org <michael.opdenacker=bootlin.com@lists.openembedded.org> wrote: > +++ b/meta/lib/oeqa/runtime/cases/opkg_sysupgrade.py This should perhaps go to meta-selftest/lib/oeqa/runtime/cases/ as it needs special setup via selftest, and not useful in standalone '-c testimage' runs. > + def test_image_upgrade(self): > + """ > + Summary: Test that generated ipk packages can > + be used to upgrade an older image version. > + This is done by generating an image but then replacing it > + by an older image shared by the Yocto Project autobuilder. > + We then run QEMU on the old image and replace the original > + original package feeds by our own. > + """ > + > + image = 'core-image-full-cmdline' > + machine = get_bb_var("MACHINE") > + > + features = 'DISTRO = "poky-altcfg"\n' > + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' > + features += 'PACKAGE_CLASSES = "package_ipk"\n' > + features += 'IMAGE_CLASSES += "testimage"\n' > + features += 'TEST_SUITES="opkg_sysupgrade"\n' > + self.write_config(features) > + > + # Need to build a full image to build the .json file needed by QEMU. > + # Therefore, it is not sufficient to run only "package_write_ipk" for the image. > + > + self.logger.info("Generating '%s' and package index for latest commit in this branch..." % image) > + bitbake(image) > + bitbake('package-index') > + > + # Download previously generated image > + > + image_file = '%s-%s.rootfs.ext4' % (image, machine) > + image_path = '%s/tmp/deploy/images/%s/%s' % (self.builddir, machine, image_file) > + machine_variant = '-alt' > + > + os.remove(image_path) > + image_url = get_latest_image_url(machine, machine_variant, image_file) > + self.logger.info("Downloading image: %s..." % image_url) > + cmd = 'wget -O %s %s' % (image_path, image_url) > + result = runCmd(cmd) > + self.assertEqual(0, result.status, cmd + ' returned a non 0 status: %s' % result.output) > + > + # Now run the upgrade tests on the old image > + > + self.logger.info("Running upgrade tests on the downloaded image, using the package feeds generated here...") > + bitbake(image + ' -c testimage') RP perhaps already pointed out this, but this mashes together parameters and implementation. Please split the implementation into 'non-opinionated' functions that take everything that can change via arguments, and have a short and sweet top level test definition that does something like this: def test_poky_altcfg_cmdline_update(self): config = 'DISTRO=..., etc' imagelocation = find_yocto_image(...) run_update_test(config, imagelocation, ...) As an example, CDN sstate tests are arranged similarly: https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n920 Then we can further improve this by defining those parameters somewhere external, that's a separate decision. Alex
Hi Richard, Many thanks for your feedback. I started by implementing Alex Kanavin's suggestions, which are a little more specific to get started ;-) On 4/25/24 at 22:40, Richard Purdie wrote: > Hi Michael, > > At least at a first read and without running it, this does look like a > reasonable direction. I suspect that anyone else looking at this would > have a lot of questions about why we'd do it this way but given the > various constraints, it does make sense to me. > > What we do need to think about is how someone else would reuse this as > currently it is very poky specific. Our aim is to make it easy for > others to use too. With that in mind: > > * We probably want to "tag" this test with something so we can exclude > it from the normal oe-selftest runs on the autobuilder and allow it to > run on a per machine basis. There are other oe-selftests we already do > this with (like toolchain testing or machine specific environment file > tests). > > * The configuration about what to test probably needs to come from the > distro (i.e. which DISTRO/MACHINE/image combinations). > > * We probably need to parameterise it so that a list of images can be > tested rather than just a single one. I did wonder if we could have it > dynamically add tests for each image configured. > > * We don't want to test on all MACHINE (e.g. qemumips and qemuppc are > not going to be included). > > * We need to find a better way to share the code with autobuilder- > helper, I don't like duplicating code. Is there example code somewhere doing something similar, i.e. importing code from an external repository? > > * The image url code is also highly poky specific. That probably needs > to come from the poky repository alongside the configuration. I'm going to submit a new version that doesn't assume any specific URL scheme. More soon. Thanks, Michael.
On Mon, 2024-04-29 at 17:21 +0200, Michael Opdenacker wrote: > On 4/25/24 at 22:40, Richard Purdie wrote: > > Hi Michael, > > > > At least at a first read and without running it, this does look like a > > reasonable direction. I suspect that anyone else looking at this would > > have a lot of questions about why we'd do it this way but given the > > various constraints, it does make sense to me. > > > > What we do need to think about is how someone else would reuse this as > > currently it is very poky specific. Our aim is to make it easy for > > others to use too. With that in mind: > > > > * We probably want to "tag" this test with something so we can exclude > > it from the normal oe-selftest runs on the autobuilder and allow it to > > run on a per machine basis. There are other oe-selftests we already do > > this with (like toolchain testing or machine specific environment file > > tests). > > > > * The configuration about what to test probably needs to come from the > > distro (i.e. which DISTRO/MACHINE/image combinations). > > > > * We probably need to parameterise it so that a list of images can be > > tested rather than just a single one. I did wonder if we could have it > > dynamically add tests for each image configured. > > > > * We don't want to test on all MACHINE (e.g. qemumips and qemuppc are > > not going to be included). > > > > * We need to find a better way to share the code with autobuilder- > > helper, I don't like duplicating code. > > Is there example code somewhere doing something similar, i.e. importing > code from an external repository? What needs to happen is the code needs to be moved into meta/lib/oe, or probably better, meta-poky/lib/oe in meta-yocto and then autobuilder- helper needs to understand how to access it from there. That is probably beyond what we have time for with the current work :( > > * The image url code is also highly poky specific. That probably needs > > to come from the poky repository alongside the configuration. > > I'm going to submit a new version that doesn't assume any specific URL > scheme. > More soon. Some assumptions about the url format will have to be made. I still maintain this code should be in meta-yocto since it is poky specific but should call code with only the configuration coming from poky/meta- yocto. Cheers, Richard
diff --git a/meta/lib/oeqa/runtime/cases/opkg_sysupgrade.py b/meta/lib/oeqa/runtime/cases/opkg_sysupgrade.py new file mode 100644 index 0000000000..05b5847b4a --- /dev/null +++ b/meta/lib/oeqa/runtime/cases/opkg_sysupgrade.py @@ -0,0 +1,68 @@ +# +# Copyright OpenEmbedded Contributors +# +# Test that generated ipk packages can be used to upgrade +# an older image version. +# +# This is done by the meta/lib/oeqa/selftest/cases/image_upgrade.py oe-selftest +# replacing the newly generated image by an older image +# generated by the Yocto Project autobuilder. +# +# Here, we replace the package feeds in our image by our own +# +# This test is not meant to be used as a regular "testimage" test +# run on the fresh image. +# +# SPDX-License-Identifier: MIT +# + +import os +from oeqa.utils.httpserver import HTTPService +from oeqa.runtime.case import OERuntimeTestCase +from oeqa.core.decorator.data import skipIfNotDataVar, skipIfNotFeature, skipIfFeature +from oeqa.runtime.decorator.package import OEHasPackage + +class OpkgSysUpgradeTest(OERuntimeTestCase): + + def pkg(self, command, expected = 0): + command = 'opkg %s' % command + status, output = self.target.run(command, 1500) + message = os.linesep.join([command, output]) + self.assertEqual(status, expected, message) + return output + +class OpkgRepoTest(OpkgSysUpgradeTest): + + @classmethod + def setUp(cls): + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_IPK']) + cls.repo_server = HTTPService(service_repo, + '0.0.0.0', port=cls.tc.target.server_port, + logger=cls.tc.logger) + cls.repo_server.start() + + @classmethod + def tearDown(cls): + cls.repo_server.stop() + + def setup_source_config_for_package_install(self): + source_server = 'http://%s:%s' % (self.tc.target.server_ip, self.repo_server.port) + sourceslist_dir = '/etc/opkg' + pkgarch = self.tc.td["TUNE_PKGARCH"] + machinedir = self.tc.td["MACHINE"].replace("-", "_") + self.target.run('cd %s; echo src/gz all %s/all > base-feeds.conf' % (sourceslist_dir, source_server)) + self.target.run('cd %s; echo src/gz %s %s/%s >> base-feeds.conf' % (sourceslist_dir, pkgarch, source_server, pkgarch)) + self.target.run('cd %s; echo src/gz %s %s/%s >> base-feeds.conf' % (sourceslist_dir, machinedir, source_server, machinedir)) + + @skipIfNotFeature('package-management', + 'Test requires package-management to be in IMAGE_FEATURES') + @skipIfNotDataVar('IMAGE_PKGTYPE', 'ipk', + 'IPK is not the primary package manager') + @skipIfFeature('read-only-rootfs', + 'Test does not work with read-only-rootfs in IMAGE_FEATURES') + @OEHasPackage(['opkg']) + def test_opkg_system_upgrade_from_repo(self): + self.setup_source_config_for_package_install() + self.pkg('update') + self.pkg('upgrade') + diff --git a/meta/lib/oeqa/selftest/cases/image_upgrade.py b/meta/lib/oeqa/selftest/cases/image_upgrade.py new file mode 100755 index 0000000000..3e26c7eed0 --- /dev/null +++ b/meta/lib/oeqa/selftest/cases/image_upgrade.py @@ -0,0 +1,124 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import os +import subprocess +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import bitbake, runCmd, get_bb_var +from oeqa.core.decorator.data import skipIfNotQemu + +basepath = os.path.abspath(os.path.dirname(__file__) + '/../../../../../') + +# Version string utilities copied from yocto-autobuilder-helper/scripts/utils.py + +def get_string_from_version(version, milestone=None, rc=None): + """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, + those are major releases + """ + if len(version) == 3 and version[-1] == 0: + version = version[:-1] + + result = ".".join(list(map(str, version))) + if milestone: + result += "_M" + str(milestone) + if rc: + result += ".rc" + str(rc) + return result + +def get_tag_from_version(version, milestone): + if not milestone: + return "yocto-" + get_string_from_version(version, milestone) + return get_string_from_version(version, milestone) + +def get_version_from_string(raw_version): + """ Get version as list of int from raw_version. + + Raw version _can_ be prefixed by "yocto-", + Raw version _can_ be suffixed by "_MX" + Raw version _can_ be suffixed by ".rcY" + """ + version = None + milestone = None + rc = None + if raw_version[:6] == "yocto-": + raw_version = raw_version[6:] + raw_version = raw_version.split(".") + if raw_version[-1][:2] == "rc": + rc = int(raw_version[-1][-1]) + raw_version = raw_version[:-1] + if raw_version[-1][-3:-1] == "_M": + milestone = int(raw_version[-1][-1]) + raw_version = raw_version[:-1] + [raw_version[-1][:-3]] + version = list(map(int, raw_version)) + """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, + those are major releases + """ + if len(version) == 3 and version[-1] == 0: + version = version[:-1] + return version, milestone, rc + +def get_latest_image_url(machine, machine_variant, image_file): + + """Returns the URL of the latest generated image for the current branch""" + + baseversion, milestone, _ = get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=basepath).decode('utf-8').strip()) + tag = get_tag_from_version(baseversion, milestone) + downloads_base = "https://downloads.yoctoproject.org/releases/yocto/" + + if milestone is not None: + downloads_base += "milestones/yocto-%s" % tag + else: + downloads_base += tag + + return "%s/machines/qemu/%s%s/%s" % (downloads_base, machine, machine_variant, image_file) + +class ImageIpkUpgrade(OESelftestTestCase): + + @skipIfNotQemu() + def test_image_upgrade(self): + """ + Summary: Test that generated ipk packages can + be used to upgrade an older image version. + This is done by generating an image but then replacing it + by an older image shared by the Yocto Project autobuilder. + We then run QEMU on the old image and replace the original + original package feeds by our own. + """ + + image = 'core-image-full-cmdline' + machine = get_bb_var("MACHINE") + + features = 'DISTRO = "poky-altcfg"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_ipk"\n' + features += 'IMAGE_CLASSES += "testimage"\n' + features += 'TEST_SUITES="opkg_sysupgrade"\n' + self.write_config(features) + + # Need to build a full image to build the .json file needed by QEMU. + # Therefore, it is not sufficient to run only "package_write_ipk" for the image. + + self.logger.info("Generating '%s' and package index for latest commit in this branch..." % image) + bitbake(image) + bitbake('package-index') + + # Download previously generated image + + image_file = '%s-%s.rootfs.ext4' % (image, machine) + image_path = '%s/tmp/deploy/images/%s/%s' % (self.builddir, machine, image_file) + machine_variant = '-alt' + + os.remove(image_path) + image_url = get_latest_image_url(machine, machine_variant, image_file) + self.logger.info("Downloading image: %s..." % image_url) + cmd = 'wget -O %s %s' % (image_path, image_url) + result = runCmd(cmd) + self.assertEqual(0, result.status, cmd + ' returned a non 0 status: %s' % result.output) + + # Now run the upgrade tests on the old image + + self.logger.info("Running upgrade tests on the downloaded image, using the package feeds generated here...") + bitbake(image + ' -c testimage')