diff mbox series

[v2,2/9] testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

Message ID 20230823061025.3952909-2-mikko.rapeli@linaro.org
State Accepted, archived
Commit f4e865062cec06586d8c38c05d86a5b0d727625c
Headers show
Series [v2,1/9] core-image-minimal: increase extra space to pass df.py oeqa runtime test | expand

Commit Message

Mikko Rapeli Aug. 23, 2023, 6:10 a.m. UTC 
Also set target and server IP addresses correctly to 127.0.0.1
so that TEST_TARGET_IP and TEST_SERVER_IP don't
need to be set manually with slirp.

Users run qemu images with slirp networking like this:

$ bitbake core-image-minimal
$ ../poky/scripts/runqemu slirp

slirp networking means that only one TCP port is forwarded from host
system to the qemu machine so that ssh into the machine works. ping
and other low level networking from host to target machine do not work,
but networking from target via host does work. This is much more easy to
setup than the bridge networking used by default, which I've never
managed to get running on various host machines.

To use slirp networking wita qemu and core-image-minimal to run testimage.bbclass tests, add
these to local.conf:

IMAGE_CLASSES += "testimage"
IMAGE_FEATURES += "ssh-server-dropbear"
TEST_RUNQEMUPARAMS += "slirp"

Then image can be compiled and tested with:

$ bitbake core-image-minimal
$ bitbake -c testimage core-image-minimal

Note that only few of the current oeqa runtime tests work against
core-image-minimal, but test results look like:

RESULTS:
RESULTS - date.DateTest.test_date: PASSED (3.64s)
RESULTS - df.DfTest.test_df: PASSED (0.55s)
RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s)
RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s)
RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s)
RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s)
RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s)
RESULTS - ping.PingTest.test_ping: PASSED (0.05s)
RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s)
RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s)
RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s)
RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s)
RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s)
RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s)
RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s)
RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s)
RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s)
RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s)
RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s)
RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s)
RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s)
RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s)
RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s)
RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s)
RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s)
RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s)
RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s)
RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s)
RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s)
RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED (0.00s)
RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s)
RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s)
RESULTS - rpm.RpmBasicTest.test_rpm_help: SKIPPED (0.00s)
RESULTS - rpm.RpmBasicTest.test_rpm_query: SKIPPED (0.00s)
RESULTS - rpm.RpmBasicTest.test_rpm_query_nonroot: SKIPPED (0.00s)
RESULTS - rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size: SKIPPED (0.00s)
RESULTS - rpm.RpmInstallRemoveTest.test_rpm_install: SKIPPED (0.00s)
RESULTS - rpm.RpmInstallRemoveTest.test_rpm_remove: SKIPPED (0.00s)
RESULTS - rust.RustCompileTest.test_cargo_compile: SKIPPED (0.00s)
RESULTS - rust.RustCompileTest.test_rust_compile: SKIPPED (0.00s)
RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
SUMMARY:
core-image-minimal () - Ran 70 tests in 22.173s
core-image-minimal - OK - All required tests passed (successes=14, skipped=56, failures=0, errors=0)
NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun and all succeeded.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/testimage.bbclass | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

v2: set TEST_SERVER_IP correctly to default qemu dhcp server IP 10.0.2.2
so that testimage selftests pass

v1: https://lists.openembedded.org/g/openembedded-core/message/186424

Comments

Khem Raj Aug. 23, 2023, 6:25 a.m. UTC | #1 
On Tue, Aug 22, 2023 at 11:10 PM Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
>
> Also set target and server IP addresses correctly to 127.0.0.1
> so that TEST_TARGET_IP and TEST_SERVER_IP don't
> need to be set manually with slirp.
>
> Users run qemu images with slirp networking like this:
>
> $ bitbake core-image-minimal
> $ ../poky/scripts/runqemu slirp
>
> slirp networking means that only one TCP port is forwarded from host
> system to the qemu machine so that ssh into the machine works. ping
> and other low level networking from host to target machine do not work,
> but networking from target via host does work. This is much more easy to
> setup than the bridge networking used by default, which I've never
> managed to get running on various host machines.
>
> To use slirp networking wita qemu and core-image-minimal to run testimage.bbclass tests, add
> these to local.conf:
>
> IMAGE_CLASSES += "testimage"
> IMAGE_FEATURES += "ssh-server-dropbear"
> TEST_RUNQEMUPARAMS += "slirp"
>
> Then image can be compiled and tested with:
>
> $ bitbake core-image-minimal
> $ bitbake -c testimage core-image-minimal
>
> Note that only few of the current oeqa runtime tests work against
> core-image-minimal, but test results look like:
>
> RESULTS:
> RESULTS - date.DateTest.test_date: PASSED (3.64s)
> RESULTS - df.DfTest.test_df: PASSED (0.55s)
> RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s)
> RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s)
> RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s)
> RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s)
> RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s)
> RESULTS - ping.PingTest.test_ping: PASSED (0.05s)
> RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s)
> RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s)
> RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s)
> RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s)
> RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s)
> RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s)
> RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s)
> RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s)
> RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s)
> RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s)
> RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s)
> RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s)
> RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s)
> RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s)
> RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s)
> RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s)
> RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s)
> RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s)
> RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s)
> RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s)
> RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s)
> RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED (0.00s)
> RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s)
> RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s)
> RESULTS - rpm.RpmBasicTest.test_rpm_help: SKIPPED (0.00s)
> RESULTS - rpm.RpmBasicTest.test_rpm_query: SKIPPED (0.00s)
> RESULTS - rpm.RpmBasicTest.test_rpm_query_nonroot: SKIPPED (0.00s)
> RESULTS - rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size: SKIPPED (0.00s)
> RESULTS - rpm.RpmInstallRemoveTest.test_rpm_install: SKIPPED (0.00s)
> RESULTS - rpm.RpmInstallRemoveTest.test_rpm_remove: SKIPPED (0.00s)
> RESULTS - rust.RustCompileTest.test_cargo_compile: SKIPPED (0.00s)
> RESULTS - rust.RustCompileTest.test_rust_compile: SKIPPED (0.00s)
> RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
> RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
> RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
> RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
> RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
> SUMMARY:
> core-image-minimal () - Ran 70 tests in 22.173s
> core-image-minimal - OK - All required tests passed (successes=14, skipped=56, failures=0, errors=0)
> NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
> NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun and all succeeded.
>
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/classes-recipe/testimage.bbclass | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)
>
> v2: set TEST_SERVER_IP correctly to default qemu dhcp server IP 10.0.2.2
> so that testimage selftests pass
>
> v1: https://lists.openembedded.org/g/openembedded-core/message/186424
>
> diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass
> index e3068348ff..7340996788 100644
> --- a/meta/classes-recipe/testimage.bbclass
> +++ b/meta/classes-recipe/testimage.bbclass
> @@ -322,7 +322,7 @@ def testimage_main(d):
>      ovmf = d.getVar("QEMU_USE_OVMF")
>
>      slirp = False
> -    if d.getVar("QEMU_USE_SLIRP"):
> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
>          slirp = True
>
>      # TODO: We use the current implementation of qemu runner because of
> @@ -369,10 +369,18 @@ def testimage_main(d):
>      # runtime use network for download projects for build
>      export_proxies(d)
>
> +    if slirp:
> +        target_ip = "127.0.0.1"

will this work when running multiple instances of qemu ?
e.g.  try bitbake core-image-ptest-all

> +        # from qemu target to host with default DHCP server
> +        server_ip = "10.0.2.2"
> +    else:
> +        target_ip = d.getVar("TEST_TARGET_IP")
> +        server_ip = d.getVar("TEST_SERVER_IP")
> +
>      # the robot dance
>      target = OERuntimeTestContextExecutor.getTarget(
> -        d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
> -        d.getVar("TEST_SERVER_IP"), **target_kwargs)
> +        d.getVar("TEST_TARGET"), logger, target_ip,
> +        server_ip, **target_kwargs)
>
>      # test context
>      tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186547): https://lists.openembedded.org/g/openembedded-core/message/186547
> Mute This Topic: https://lists.openembedded.org/mt/100910036/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Mikko Rapeli Aug. 23, 2023, 6:48 a.m. UTC | #2 
Hi,

On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> On Tue, Aug 22, 2023 at 11:10 PM Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> >
> > Also set target and server IP addresses correctly to 127.0.0.1
> > so that TEST_TARGET_IP and TEST_SERVER_IP don't
> > need to be set manually with slirp.
> >
> > Users run qemu images with slirp networking like this:
> >
> > $ bitbake core-image-minimal
> > $ ../poky/scripts/runqemu slirp
> >
> > slirp networking means that only one TCP port is forwarded from host
> > system to the qemu machine so that ssh into the machine works. ping
> > and other low level networking from host to target machine do not work,
> > but networking from target via host does work. This is much more easy to
> > setup than the bridge networking used by default, which I've never
> > managed to get running on various host machines.
> >
> > To use slirp networking wita qemu and core-image-minimal to run testimage.bbclass tests, add
> > these to local.conf:
> >
> > IMAGE_CLASSES += "testimage"
> > IMAGE_FEATURES += "ssh-server-dropbear"
> > TEST_RUNQEMUPARAMS += "slirp"
> >
> > Then image can be compiled and tested with:
> >
> > $ bitbake core-image-minimal
> > $ bitbake -c testimage core-image-minimal
> >
> > Note that only few of the current oeqa runtime tests work against
> > core-image-minimal, but test results look like:
> >
> > RESULTS:
> > RESULTS - date.DateTest.test_date: PASSED (3.64s)
> > RESULTS - df.DfTest.test_df: PASSED (0.55s)
> > RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s)
> > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s)
> > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s)
> > RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s)
> > RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s)
> > RESULTS - ping.PingTest.test_ping: PASSED (0.05s)
> > RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s)
> > RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s)
> > RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s)
> > RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s)
> > RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s)
> > RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s)
> > RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s)
> > RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s)
> > RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s)
> > RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s)
> > RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s)
> > RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s)
> > RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s)
> > RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s)
> > RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s)
> > RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s)
> > RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s)
> > RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s)
> > RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s)
> > RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s)
> > RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s)
> > RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED (0.00s)
> > RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s)
> > RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> > RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s)
> > RESULTS - rpm.RpmBasicTest.test_rpm_help: SKIPPED (0.00s)
> > RESULTS - rpm.RpmBasicTest.test_rpm_query: SKIPPED (0.00s)
> > RESULTS - rpm.RpmBasicTest.test_rpm_query_nonroot: SKIPPED (0.00s)
> > RESULTS - rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size: SKIPPED (0.00s)
> > RESULTS - rpm.RpmInstallRemoveTest.test_rpm_install: SKIPPED (0.00s)
> > RESULTS - rpm.RpmInstallRemoveTest.test_rpm_remove: SKIPPED (0.00s)
> > RESULTS - rust.RustCompileTest.test_cargo_compile: SKIPPED (0.00s)
> > RESULTS - rust.RustCompileTest.test_rust_compile: SKIPPED (0.00s)
> > RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s)
> > RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s)
> > RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s)
> > RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s)
> > RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s)
> > SUMMARY:
> > core-image-minimal () - Ran 70 tests in 22.173s
> > core-image-minimal - OK - All required tests passed (successes=14, skipped=56, failures=0, errors=0)
> > NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded
> > NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun and all succeeded.
> >
> > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > ---
> >  meta/classes-recipe/testimage.bbclass | 14 +++++++++++---
> >  1 file changed, 11 insertions(+), 3 deletions(-)
> >
> > v2: set TEST_SERVER_IP correctly to default qemu dhcp server IP 10.0.2.2
> > so that testimage selftests pass
> >
> > v1: https://lists.openembedded.org/g/openembedded-core/message/186424
> >
> > diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass
> > index e3068348ff..7340996788 100644
> > --- a/meta/classes-recipe/testimage.bbclass
> > +++ b/meta/classes-recipe/testimage.bbclass
> > @@ -322,7 +322,7 @@ def testimage_main(d):
> >      ovmf = d.getVar("QEMU_USE_OVMF")
> >
> >      slirp = False
> > -    if d.getVar("QEMU_USE_SLIRP"):
> > +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> >          slirp = True
> >
> >      # TODO: We use the current implementation of qemu runner because of
> > @@ -369,10 +369,18 @@ def testimage_main(d):
> >      # runtime use network for download projects for build
> >      export_proxies(d)
> >
> > +    if slirp:
> > +        target_ip = "127.0.0.1"
> 
> will this work when running multiple instances of qemu ?
> e.g.  try bitbake core-image-ptest-all

The port forwarding settings are same for each of the qemu virtual machine instances
currently.

So one instance reserves the 127.0.0.1 and TCP port 2222, by default.
Another instance will fail to reserve the port 2222 and qemu startup fails.

So I don't think "slirp" works with multiple qemu instances on a single machine.

This could be changed if the hostfwd port number selection was changed to be dynamic.
runqemu currently hard codes this as:

ostfwd = ",hostfwd=tcp:127.0.0.1:2222-:22,hostfwd=tcp:127.0.0.1:2323-:23"

A retry with different, increasing port numbers could be added.

Then qemurunner.py could be changed to detect the effective port from qemu startup logs,
as it already now partially does as a fallback, or maybe the current code would even work
with dynamic port numbers.

Cheers,

-Mikko

> 
> > +        # from qemu target to host with default DHCP server
> > +        server_ip = "10.0.2.2"
> > +    else:
> > +        target_ip = d.getVar("TEST_TARGET_IP")
> > +        server_ip = d.getVar("TEST_SERVER_IP")
> > +
> >      # the robot dance
> >      target = OERuntimeTestContextExecutor.getTarget(
> > -        d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
> > -        d.getVar("TEST_SERVER_IP"), **target_kwargs)
> > +        d.getVar("TEST_TARGET"), logger, target_ip,
> > +        server_ip, **target_kwargs)
> >
> >      # test context
> >      tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)
> > --
> > 2.34.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#186547): https://lists.openembedded.org/g/openembedded-core/message/186547
> > Mute This Topic: https://lists.openembedded.org/mt/100910036/1997914
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
Mikko Rapeli Aug. 23, 2023, 7:31 a.m. UTC | #3 
Hi,

On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> will this work when running multiple instances of qemu ?
> e.g.  try bitbake core-image-ptest-all

I was not aware of core-image-ptest-all. Tried to build it but it doesn't
seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
needed to test core-image-minimal:

Error: 
 Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
  - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
  - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
  - conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
just be skipped.

Cheers,

-Mikko
Richard Purdie Aug. 23, 2023, 9:06 a.m. UTC | #4 
On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > will this work when running multiple instances of qemu ?
> > e.g.  try bitbake core-image-ptest-all
> 
> I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> needed to test core-image-minimal:
> 
> Error: 
>  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
>   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
>   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
>   - conflicting requests
> (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> 
> oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> just be skipped.

The autobuilder actually does that, the minimal image is just tested
with the small number of non-network tests. The main thing was to test
it does actually boot to a login prompt. We have other tests which test
the other areas with other images.

The reason for the above is that there will be ptest openssh images
which conflict with the dropbear ones. You can likely avoid that by
using:

IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"

The ptest images are designed to only include the ptest in question so
in theory are otherwise as minimal as the dependencies allow.

Cheers,

Richard
Mikko Rapeli Aug. 23, 2023, 9:47 a.m. UTC | #5 
Hi,

On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > Hi,
> > 
> > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > will this work when running multiple instances of qemu ?
> > > e.g.  try bitbake core-image-ptest-all
> > 
> > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > needed to test core-image-minimal:
> > 
> > Error: 
> >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> >   - conflicting requests
> > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > 
> > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > just be skipped.
> 
> The autobuilder actually does that, the minimal image is just tested
> with the small number of non-network tests. The main thing was to test
> it does actually boot to a login prompt. We have other tests which test
> the other areas with other images.

Yes, granted it's enough to test that boot to serial console login works.

> The reason for the above is that there will be ptest openssh images
> which conflict with the dropbear ones. You can likely avoid that by
> using:
> 
> IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> 
> The ptest images are designed to only include the ptest in question so
> in theory are otherwise as minimal as the dependencies allow.

Alright, this I could try. But I fear there is a log more missing from my
plain poky and default machine target to get the selftests and tests running.
This magic is somewhere in the autobuilder related git repositories, but from plain
poky checkout with a specific commit from master branch I don't know which versions
and repos to use so that the tests would be passing.

With these modifications in local.conf:

IMAGE_CLASSES += "testimage"
TEST_RUNQEMUPARAMS += "slirp"
IMAGE_FEATURES += "ssh-server-dropbear"
# update kernel to latest available in poky
PREFERRED_VERSION_linux-yocto = ""
SANITY_TESTED_DISTROS = ""

at least runtime_test.TestImage are passing with slirp now.

Without MAGE_FEATURES += "ssh-server-dropbear", "bitbake core-image-ptest-all" now succeeds
and "bitbake -c testimage core-image-ptest-all" is running the tests, seeminly in series.
At least there are no multiple qemu instances running in parallel and no failures related to
slirp ssh port being reserved by a single qemu instance. But the tests are reporting only skips
so maybe the autobuilder scripts have some settings which I don't have correctly set:

Cannot run ptests without @expectedFailure as ptests are expected to fail
QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
Cannot run ptests without @expectedFailure as ptests are expected to fail
QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
Cannot run ptests without @expectedFailure as ptests are expected to fail
RESULTS:
RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
SUMMARY:
core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)

The ptest execution seems to be skipped for all images.

Cheers,

-Mikko
Alexander Kanavin Aug. 23, 2023, 9:57 a.m. UTC | #6 
On Wed, 23 Aug 2023 at 11:48, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:

> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
> QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> RESULTS:
> RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
> RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
> RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> SUMMARY:
> core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
> core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
>
> The ptest execution seems to be skipped for all images.

This is actually correct and ptest did run. We have a soft and hard
fail options for ptests, and it defaults to soft fail
(PTEST_EXPECT_FAILURE = "1" in core-image-ptest), i.e. warnings on the
autobuilder.

Alex
Mikko Rapeli Aug. 23, 2023, 10:49 a.m. UTC | #7 
Hi,

On Wed, Aug 23, 2023 at 11:57:50AM +0200, Alexander Kanavin wrote:
> On Wed, 23 Aug 2023 at 11:48, Mikko Rapeli <builder@linaro.org> wrote:
> 
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
> > QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
> > Cannot run ptests without @expectedFailure as ptests are expected to fail
> > RESULTS:
> > RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
> > RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
> > RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
> > RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> > SUMMARY:
> > core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
> > core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
> >
> > The ptest execution seems to be skipped for all images.
> 
> This is actually correct and ptest did run. We have a soft and hard
> fail options for ptests, and it defaults to soft fail
> (PTEST_EXPECT_FAILURE = "1" in core-image-ptest), i.e. warnings on the
> autobuilder.

OK, that is good to know. In this case with local.conf additions for "slirp":

IMAGE_CLASSES += "testimage"
#IMAGE_FEATURES += "ssh-server-dropbear"
TEST_RUNQEMUPARAMS += "slirp"
# update kernel to latest available in poky
PREFERRED_VERSION_linux-yocto = ""
SANITY_TESTED_DISTROS = ""

The tests passed with one warning, which I'll try to check:

WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
    self.fail(failmsg)
AssertionError: 
Failed ptests:
{'openssh': ['banner']}
...
SUMMARY:
core-image-ptest-lttng-tools () - Ran 5 tests in 2289.072s
core-image-ptest-lttng-tools - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
NOTE: recipe core-image-ptest-lttng-tools-1.0-r0: task do_testimage: Succeeded
NOTE: Running noexec task 1405 of 1405 (/home/builder/src/poky/meta/recipes-core/images/core-image-ptest-all.bb:do_testimage)
NOTE: Tasks Summary: Attempted 1405 tasks of which 1304 didn't need to be rerun and all succeeded.

Summary: There was 1 WARNING message.

Cheers,

-Mikko
Richard Purdie Aug. 23, 2023, 10:49 a.m. UTC | #8 
On Wed, 2023-08-23 at 12:47 +0300, Mikko Rapeli wrote:
> On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > > Hi,
> > > 
> > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > > will this work when running multiple instances of qemu ?
> > > > e.g.  try bitbake core-image-ptest-all
> > > 
> > > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > > needed to test core-image-minimal:
> > > 
> > > Error: 
> > >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> > >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> > >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> > >   - conflicting requests
> > > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > > 
> > > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > > just be skipped.
> > 
> > The autobuilder actually does that, the minimal image is just tested
> > with the small number of non-network tests. The main thing was to test
> > it does actually boot to a login prompt. We have other tests which test
> > the other areas with other images.
> 
> Yes, granted it's enough to test that boot to serial console login works.
> 
> > The reason for the above is that there will be ptest openssh images
> > which conflict with the dropbear ones. You can likely avoid that by
> > using:
> > 
> > IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> > 
> > The ptest images are designed to only include the ptest in question so
> > in theory are otherwise as minimal as the dependencies allow.
> 
> Alright, this I could try. But I fear there is a log more missing from my
> plain poky and default machine target to get the selftests and tests running.

There is no secret magic config the autobuilder uses. You keep asking
me for this and there isn't anything. It is actually starting to annoy
me a bit as there isn't anything "hidden".

The configurations used are all from this file:

https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json

Yes, there is a block of high level config around numbers of threads,
disk space monitoring, pressure regulation values and so on but we
purposefully keep the config to be as close to standard poky as we can.

When we run selftest we do a couple of things. Firstly we split the
machine and toolchain targets into separate areas.  We also split
reproducibility to it's own target and test mirroring elsewhere too.
This results in a slightly more complex selftest invocation:

OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

The only test which I don't think we run anywhere any more is the
test_checkpkg target.

You can see all this from the logs buildbot shows from it's UI on the
autobuilder too.

> This magic is somewhere in the autobuilder related git repositories, but from plain
> poky checkout with a specific commit from master branch I don't know which versions
> and repos to use so that the tests would be passing.
> 
> With these modifications in local.conf:
> 
> IMAGE_CLASSES += "testimage"
> TEST_RUNQEMUPARAMS += "slirp"

We do not use slirp on the autobuilder. We never have and we're
unlikely ever to do so and it is not something we officially support
for this. This is likely the biggest source of problems.

I appreciate that gives some networking challenges for people in
constrained environments but we did that primarily to allow for
simplifications in the rest of the setup.

> IMAGE_FEATURES += "ssh-server-dropbear"

I've already explained that this one does likely cause problems. We
simply don't run many tests against minimal images. 

> # update kernel to latest available in poky
> PREFERRED_VERSION_linux-yocto = ""

Not sure why this is needed?

> SANITY_TESTED_DISTROS = ""

This one we've discussed. It really should be fixed in a better way but
isn't anywhere near the top of the priority list.

> at least runtime_test.TestImage are passing with slirp now.
> 
> Without MAGE_FEATURES += "ssh-server-dropbear", "bitbake core-image-ptest-all" now succeeds
> and "bitbake -c testimage core-image-ptest-all" is running the tests, seeminly in series.
> At least there are no multiple qemu instances running in parallel and no failures related to
> slirp ssh port being reserved by a single qemu instance. But the tests are reporting only skips
> so maybe the autobuilder scripts have some settings which I don't have correctly set:
> 
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP released QEMU at 08/23/23 10:26:03 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> QMP connected to QEMU at 08/23/23 10:26:04 and took 0.60 seconds
> QMP released QEMU at 08/23/23 10:26:04 and took 0.13 seconds from connect
> Cannot run ptests without @expectedFailure as ptests are expected to fail
> RESULTS:
> RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.30s)
> RESULTS - ping.PingTest.test_ping: PASSED (0.04s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: PASSED (1.55s)
> RESULTS - ssh.SSHTest.test_ssh: PASSED (1.01s)
> RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
> SUMMARY:
> core-image-ptest-libtry-tiny-perl () - Ran 5 tests in 7.208s
> core-image-ptest-libtry-tiny-perl - OK - All required tests passed (successes=3, skipped=1, failures=0, errors=0)
> 
> The ptest execution seems to be skipped for all images.

I think Alex covers this.

You can compare it to what is shown on the autobuilder output. You can
also compare the testresults.json file too using "resulttool report" to
compare results with what the autobuilder runs.

Cheers,

Richard
Mikko Rapeli Aug. 23, 2023, 11:47 a.m. UTC | #9 
Hi,

On Wed, Aug 23, 2023 at 11:49:39AM +0100, Richard Purdie wrote:
> On Wed, 2023-08-23 at 12:47 +0300, Mikko Rapeli wrote:
> > On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> > > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > > > Hi,
> > > > 
> > > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > > > will this work when running multiple instances of qemu ?
> > > > > e.g.  try bitbake core-image-ptest-all
> > > > 
> > > > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > > > needed to test core-image-minimal:
> > > > 
> > > > Error: 
> > > >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> > > >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> > > >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> > > >   - conflicting requests
> > > > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > > > 
> > > > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > > > just be skipped.
> > > 
> > > The autobuilder actually does that, the minimal image is just tested
> > > with the small number of non-network tests. The main thing was to test
> > > it does actually boot to a login prompt. We have other tests which test
> > > the other areas with other images.
> > 
> > Yes, granted it's enough to test that boot to serial console login works.
> > 
> > > The reason for the above is that there will be ptest openssh images
> > > which conflict with the dropbear ones. You can likely avoid that by
> > > using:
> > > 
> > > IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> > > 
> > > The ptest images are designed to only include the ptest in question so
> > > in theory are otherwise as minimal as the dependencies allow.
> > 
> > Alright, this I could try. But I fear there is a log more missing from my
> > plain poky and default machine target to get the selftests and tests running.
> 
> There is no secret magic config the autobuilder uses. You keep asking
> me for this and there isn't anything. It is actually starting to annoy
> me a bit as there isn't anything "hidden".

If I can't run the tests for poky the way yocto upstream does, then I'm afraid
I can't really help with the issues you've seeing with the tests.

> The configurations used are all from this file:
> 
> https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json

Ok so https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1948
for example uses yocto-autobuilder-helper at commit
af5d072a654a060c3ee61b5f394f52632e20200b. And the full repo
URL is visible not in the snippet there but in the full stdio download
of "Fetch yocto-autobuilder-helper" step. It was the web UI confusing me.

I'm sorry if these are all obvious to you and other developers but not
to me.

> Yes, there is a block of high level config around numbers of threads,
> disk space monitoring, pressure regulation values and so on but we
> purposefully keep the config to be as close to standard poky as we can.
>
> When we run selftest we do a couple of things. Firstly we split the
> machine and toolchain targets into separate areas.  We also split
> reproducibility to it's own target and test mirroring elsewhere too.
> This results in a slightly more complex selftest invocation:
> 
> OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15
> 
> The only test which I don't think we run anywhere any more is the
> test_checkpkg target.
> 
> You can see all this from the logs buildbot shows from it's UI on the
> autobuilder too.

Thanks, this explains alot!

I'm still wondering how the build hosts are setup with "modprobe vgem" and that
build user account is in "render" group (on Ubuntu at least). Should these
be added to some documentation too?

> > This magic is somewhere in the autobuilder related git repositories, but from plain
> > poky checkout with a specific commit from master branch I don't know which versions
> > and repos to use so that the tests would be passing.
> > 
> > With these modifications in local.conf:
> > 
> > IMAGE_CLASSES += "testimage"
> > TEST_RUNQEMUPARAMS += "slirp"
> 
> We do not use slirp on the autobuilder. We never have and we're
> unlikely ever to do so and it is not something we officially support
> for this. This is likely the biggest source of problems.

Sadly I can't figure out how to setup runqemu and testimage.bbclass and
oe-selftests networking without slirp and for my testing needs it has been good
enough, now also with oe-selftests.

> I appreciate that gives some networking challenges for people in
> constrained environments but we did that primarily to allow for
> simplifications in the rest of the setup.
> 
> > IMAGE_FEATURES += "ssh-server-dropbear"
> 
> I've already explained that this one does likely cause problems. We
> simply don't run many tests against minimal images. 
>
> > # update kernel to latest available in poky
> > PREFERRED_VERSION_linux-yocto = ""
> 
> Not sure why this is needed?

I'm trying to test the 6.4 kernel update and reproduce the issues you've seen
there too.

> > SANITY_TESTED_DISTROS = ""
> 
> This one we've discussed. It really should be fixed in a better way but
> isn't anywhere near the top of the priority list.

I'll try to get to this.

Cheers,

-Mikko
Mikko Rapeli Aug. 23, 2023, 1:01 p.m. UTC | #10 
Hi,

On Wed, Aug 23, 2023 at 01:49:39PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> The tests passed with one warning, which I'll try to check:
> 
> WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
> Traceback (most recent call last):
>   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
>     self.do_ptestrunner()
>   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
>     self.fail(failmsg)
> AssertionError: 
> Failed ptests:
> {'openssh': ['banner']}

Has this openssh banner test on x86_64 been failing on yocto autobuilders?

Log tmp/work/qemux86_64-poky-linux/core-image-ptest-openssh/1.0/temp/log.do_testimage
shows:

PASS: environment passing
run test transfer.sh ...
PASS: transfer data
run test banner.sh ...
test banner: missing banner file
test banner: size 0
test banner: size 10
test banner: size 100
test banner: size 1000
cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
banner size 1000 mismatch
test banner: size 10000
test banner: size 100000
test banner: suppress banner (-q)
FAIL:  banner
make: *** [Makefile:223: t-exec] Error 1

On manual execution this passes:

$ runqemu slirp nographic novga core-image-ptest-openssh
...
root@qemux86-64:~# ptest-runner
START: ptest-runner
2023-08-23T12:04
BEGIN: /usr/lib/openssh/ptest
...
PASS: transfer data
run test banner.sh ...
test banner: missing banner file
test banner: size 0
test banner: size 10
test banner: size 100
test banner: size 1000
test banner: size 10000
test banner: size 100000
test banner: suppress banner (-q)
PASS: banner
...

client rekey tests are still running.

Cheers,

-Mikko
Richard Purdie Aug. 23, 2023, 1:17 p.m. UTC | #11 
On Wed, 2023-08-23 at 14:47 +0300, Mikko Rapeli wrote:
> On Wed, Aug 23, 2023 at 11:49:39AM +0100, Richard Purdie wrote:
> > On Wed, 2023-08-23 at 12:47 +0300, Mikko Rapeli wrote:
> > > On Wed, Aug 23, 2023 at 10:06:41AM +0100, Richard Purdie wrote:
> > > > On Wed, 2023-08-23 at 10:31 +0300, Mikko Rapeli wrote:
> > > > > Hi,
> > > > > 
> > > > > On Tue, Aug 22, 2023 at 11:25:58PM -0700, Khem Raj wrote:
> > > > > > will this work when running multiple instances of qemu ?
> > > > > > e.g.  try bitbake core-image-ptest-all
> > > > > 
> > > > > I was not aware of core-image-ptest-all. Tried to build it but it doesn't
> > > > > seem to be compatible with IMAGE_FEATURES += "ssh-server-dropbear" which is
> > > > > needed to test core-image-minimal:
> > > > > 
> > > > > Error: 
> > > > >  Problem: package packagegroup-core-ssh-dropbear-1.0-r1.noarch from oe-repo requires dropbear, but none of the providers can be installed
> > > > >   - package dropbear-2022.83-r0.core2_64 from oe-repo conflicts with openssh provided by openssh-9.3p2-r0.core2_64 from oe-repo
> > > > >   - package openssh-9.3p2-r0.core2_64 from oe-repo conflicts with dropbear provided by dropbear-2022.83-r0.core2_64 from oe-repo
> > > > >   - conflicting requests
> > > > > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
> > > > > 
> > > > > oeqa runtime testing core-image-minimal without ssh server doesn't make sense as all tests will
> > > > > just be skipped.
> > > > 
> > > > The autobuilder actually does that, the minimal image is just tested
> > > > with the small number of non-network tests. The main thing was to test
> > > > it does actually boot to a login prompt. We have other tests which test
> > > > the other areas with other images.
> > > 
> > > Yes, granted it's enough to test that boot to serial console login works.
> > > 
> > > > The reason for the above is that there will be ptest openssh images
> > > > which conflict with the dropbear ones. You can likely avoid that by
> > > > using:
> > > > 
> > > > IMAGE_FEATURES:append:pn-core-image-minimal =  " ssh-server-dropbear"
> > > > 
> > > > The ptest images are designed to only include the ptest in question so
> > > > in theory are otherwise as minimal as the dependencies allow.
> > > 
> > > Alright, this I could try. But I fear there is a log more missing from my
> > > plain poky and default machine target to get the selftests and tests running.
> > 
> > There is no secret magic config the autobuilder uses. You keep asking
> > me for this and there isn't anything. It is actually starting to annoy
> > me a bit as there isn't anything "hidden".
> 
> If I can't run the tests for poky the way yocto upstream does, then I'm afraid
> I can't really help with the issues you've seeing with the tests.

If that isn't possible we need to fix the docs.

> > The configurations used are all from this file:
> > 
> > https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json
> 
> Ok so https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1948
> for example uses yocto-autobuilder-helper at commit
> af5d072a654a060c3ee61b5f394f52632e20200b. And the full repo
> URL is visible not in the snippet there but in the full stdio download
> of "Fetch yocto-autobuilder-helper" step. It was the web UI confusing me.
> 
> I'm sorry if these are all obvious to you and other developers but not
> to me.
> 
> > Yes, there is a block of high level config around numbers of threads,
> > disk space monitoring, pressure regulation values and so on but we
> > purposefully keep the config to be as close to standard poky as we can.
> > 
> > When we run selftest we do a couple of things. Firstly we split the
> > machine and toolchain targets into separate areas.  We also split
> > reproducibility to it's own target and test mirroring elsewhere too.
> > This results in a slightly more complex selftest invocation:
> > 
> > OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15
> > 
> > The only test which I don't think we run anywhere any more is the
> > test_checkpkg target.
> > 
> > You can see all this from the logs buildbot shows from it's UI on the
> > autobuilder too.
> 
> Thanks, this explains alot!
> 
> I'm still wondering how the build hosts are setup with "modprobe vgem" and that
> build user account is in "render" group (on Ubuntu at least). Should these
> be added to some documentation too?

Yes, they should.

> 
> > > This magic is somewhere in the autobuilder related git repositories, but from plain
> > > poky checkout with a specific commit from master branch I don't know which versions
> > > and repos to use so that the tests would be passing.
> > > 
> > > With these modifications in local.conf:
> > > 
> > > IMAGE_CLASSES += "testimage"
> > > TEST_RUNQEMUPARAMS += "slirp"
> > 
> > We do not use slirp on the autobuilder. We never have and we're
> > unlikely ever to do so and it is not something we officially support
> > for this. This is likely the biggest source of problems.
> 
> Sadly I can't figure out how to setup runqemu and testimage.bbclass and
> oe-selftests networking without slirp and for my testing needs it has been good
> enough, now also with oe-selftests.

As root you need to run:

runqemu-gen-tapdevs <uid> <gid> <number tap devs>

where uid is your user ID and gid is the group you want to be able to
access those devices. Once done once at boot, the builds will use those
devices. The autobuilder has these pre-setup.

You raise a good point that this should be documented, probably in the
testing manual. As we're all aware, documentation is hard so we should
either file a bug for that (and the vgem issue) or write a docs patch.

https://docs.yoctoproject.org/test-manual/index.html

> > I appreciate that gives some networking challenges for people in
> > constrained environments but we did that primarily to allow for
> > simplifications in the rest of the setup.
> > 
> > > IMAGE_FEATURES += "ssh-server-dropbear"
> > 
> > I've already explained that this one does likely cause problems. We
> > simply don't run many tests against minimal images. 
> > 
> > > # update kernel to latest available in poky
> > > PREFERRED_VERSION_linux-yocto = ""
> > 
> > Not sure why this is needed?
> 
> I'm trying to test the 6.4 kernel update and reproduce the issues you've seen
> there too.

Ok, I did change the default to 6.4 now.

> > > SANITY_TESTED_DISTROS = ""
> > 
> > This one we've discussed. It really should be fixed in a better way but
> > isn't anywhere near the top of the priority list.
> 
> I'll try to get to this.

Thanks,

Richard
Alexander Kanavin Aug. 23, 2023, 1:24 p.m. UTC | #12 
I've just seen this too. I think to trigger this you need to run the
test via bitbake -c testimage rather than directly over console - the
output is then piped over ssh, and something goes wrong due to the
large volume of banner data perhaps? Would *really* appreciate if you
can get to the bottom of it.

Alex

On Wed, 23 Aug 2023 at 15:01, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
>
> Hi,
>
> On Wed, Aug 23, 2023 at 01:49:39PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > The tests passed with one warning, which I'll try to check:
> >
> > WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
> > Traceback (most recent call last):
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
> >     self.do_ptestrunner()
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
> >     self.fail(failmsg)
> > AssertionError:
> > Failed ptests:
> > {'openssh': ['banner']}
>
> Has this openssh banner test on x86_64 been failing on yocto autobuilders?
>
> Log tmp/work/qemux86_64-poky-linux/core-image-ptest-openssh/1.0/temp/log.do_testimage
> shows:
>
> PASS: environment passing
> run test transfer.sh ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
> banner size 1000 mismatch
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> FAIL:  banner
> make: *** [Makefile:223: t-exec] Error 1
>
> On manual execution this passes:
>
> $ runqemu slirp nographic novga core-image-ptest-openssh
> ...
> root@qemux86-64:~# ptest-runner
> START: ptest-runner
> 2023-08-23T12:04
> BEGIN: /usr/lib/openssh/ptest
> ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> PASS: banner
> ...
>
> client rekey tests are still running.
>
> Cheers,
>
> -Mikko
Richard Purdie Aug. 23, 2023, 1:26 p.m. UTC | #13 
On Wed, 2023-08-23 at 16:01 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Wed, Aug 23, 2023 at 01:49:39PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > The tests passed with one warning, which I'll try to check:
> > 
> > WARNING: core-image-ptest-openssh-1.0-r0 do_testimage: There were failing ptests.
> > Traceback (most recent call last):
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
> >     self.do_ptestrunner()
> >   File "/home/builder/src/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
> >     self.fail(failmsg)
> > AssertionError: 
> > Failed ptests:
> > {'openssh': ['banner']}
> 
> Has this openssh banner test on x86_64 been failing on yocto autobuilders?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178

so yes, it has. Not always but sometimes. We do try and ensure there
are open bugs for any issues although I'm aware the current qemuppc one
does not have a bug as yet.

Cheers,

Richard

> 
> Log tmp/work/qemux86_64-poky-linux/core-image-ptest-openssh/1.0/temp/log.do_testimage
> shows:
> 
> PASS: environment passing
> run test transfer.sh ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> cmp: EOF on /usr/lib/openssh/ptest/regress/banner.in
> banner size 1000 mismatch
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> FAIL:  banner
> make: *** [Makefile:223: t-exec] Error 1
> 
> On manual execution this passes:
> 
> $ runqemu slirp nographic novga core-image-ptest-openssh
> ...
> root@qemux86-64:~# ptest-runner
> START: ptest-runner
> 2023-08-23T12:04
> BEGIN: /usr/lib/openssh/ptest
> ...
> PASS: transfer data
> run test banner.sh ...
> test banner: missing banner file
> test banner: size 0
> test banner: size 10
> test banner: size 100
> test banner: size 1000
> test banner: size 10000
> test banner: size 100000
> test banner: suppress banner (-q)
> PASS: banner
> ...
> 
> client rekey tests are still running.
> 
> Cheers,
> 
> -Mikko
>
Mikko Rapeli Aug. 23, 2023, 1:28 p.m. UTC | #14 
Hi,

On Wed, Aug 23, 2023 at 03:24:43PM +0200, Alexander Kanavin wrote:
> I've just seen this too. I think to trigger this you need to run the
> test via bitbake -c testimage rather than directly over console - the
> output is then piped over ssh, and something goes wrong due to the
> large volume of banner data perhaps? Would *really* appreciate if you
> can get to the bottom of it.

Ok, I'll try to have a look.

I also just found out that:

https://wiki.yoctoproject.org/wiki/Weekly_Status

-> https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT

-> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15178
   Reported: 	2023-08-02 14:32 UTC by Alexandre Belloni 

So it's a known issue.

Cheers,

-Mikko
Ross Burton Sept. 12, 2023, 11:25 a.m. UTC | #15 
On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> -    if d.getVar("QEMU_USE_SLIRP"):
> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
>         slirp = True

This means we can remove the QEMU_USE_SLIRP variable too, right?

Ross
Mikko Rapeli Sept. 12, 2023, 11:29 a.m. UTC | #16 
Hi,

On Tue, Sep 12, 2023 at 11:25:28AM +0000, Ross Burton wrote:
> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> > -    if d.getVar("QEMU_USE_SLIRP"):
> > +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> >         slirp = True
> 
> This means we can remove the QEMU_USE_SLIRP variable too, right?

Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with testimage.bbclass.

Cheers,

-Mikko
Ross Burton Sept. 12, 2023, 11:30 a.m. UTC | #17 
On 12 Sep 2023, at 12:29, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> 
> Hi,
> 
> On Tue, Sep 12, 2023 at 11:25:28AM +0000, Ross Burton wrote:
>> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
>>> -    if d.getVar("QEMU_USE_SLIRP"):
>>> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
>>>        slirp = True
>> 
>> This means we can remove the QEMU_USE_SLIRP variable too, right?
> 
> Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with testimage.bbclass.

Can you send a followup to do that shortly?

Ross
Mikko Rapeli Sept. 12, 2023, 11:37 a.m. UTC | #18 
On Tue, Sep 12, 2023 at 11:30:57AM +0000, Ross Burton wrote:
> On 12 Sep 2023, at 12:29, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > 
> > Hi,
> > 
> > On Tue, Sep 12, 2023 at 11:25:28AM +0000, Ross Burton wrote:
> >> On 23 Aug 2023, at 07:10, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> >>> -    if d.getVar("QEMU_USE_SLIRP"):
> >>> +    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
> >>>        slirp = True
> >> 
> >> This means we can remove the QEMU_USE_SLIRP variable too, right?
> > 
> > Yes. TEST_RUNQEMUPARAMS += "slirp" would be enough to run slirp with testimage.bbclass.
> 
> Can you send a followup to do that shortly?

Yes, I can do that.

In my developer builds I have following changes from this patch series:

oeqa selftest runtime_test.py: append to TEST_RUNQEMUPARAMS
oeqa dnf_runtime.py: fix HTTP server IP address and port
testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMS

Rest are already applied or rejected.

Cheers,

-Mikko
diff mbox series

Patch

diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass
index e3068348ff..7340996788 100644
--- a/meta/classes-recipe/testimage.bbclass
+++ b/meta/classes-recipe/testimage.bbclass
@@ -322,7 +322,7 @@  def testimage_main(d):
     ovmf = d.getVar("QEMU_USE_OVMF")
 
     slirp = False
-    if d.getVar("QEMU_USE_SLIRP"):
+    if d.getVar("QEMU_USE_SLIRP") or bb.utils.contains('TEST_RUNQEMUPARAMS', 'slirp', True, False, d):
         slirp = True
 
     # TODO: We use the current implementation of qemu runner because of
@@ -369,10 +369,18 @@  def testimage_main(d):
     # runtime use network for download projects for build
     export_proxies(d)
 
+    if slirp:
+        target_ip = "127.0.0.1"
+        # from qemu target to host with default DHCP server
+        server_ip = "10.0.2.2"
+    else:
+        target_ip = d.getVar("TEST_TARGET_IP")
+        server_ip = d.getVar("TEST_SERVER_IP")
+
     # the robot dance
     target = OERuntimeTestContextExecutor.getTarget(
-        d.getVar("TEST_TARGET"), logger, d.getVar("TEST_TARGET_IP"),
-        d.getVar("TEST_SERVER_IP"), **target_kwargs)
+        d.getVar("TEST_TARGET"), logger, target_ip,
+        server_ip, **target_kwargs)
 
     # test context
     tc = OERuntimeTestContext(td, logger, target, image_packages, extract_dir)