From patchwork Fri Mar 15 14:37:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 41010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F307CC54E6E for ; Fri, 15 Mar 2024 14:37:50 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.23978.1710513461500340617 for ; Fri, 15 Mar 2024 07:37:41 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 16977C15 for ; Fri, 15 Mar 2024 07:38:16 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6DF6F3F73F for ; Fri, 15 Mar 2024 07:37:40 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/3] curl: improve run-ptest Date: Fri, 15 Mar 2024 14:37:36 +0000 Message-Id: <20240315143738.1234858-1-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 15 Mar 2024 14:37:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/197133 From: Ross Burton There's no need to run the output of runtests.pl through a sed to get automake-style output, as you can pass -am to get this formatting. Don't run timing dependent tests, as the ptests can run on loaded systems. Add a dependency on the en_US locale because some of the tests require this. Signed-off-by: Ross Burton --- meta/recipes-support/curl/curl/run-ptest | 13 +++++++++---- meta/recipes-support/curl/curl_8.6.0.bb | 1 + 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/meta/recipes-support/curl/curl/run-ptest b/meta/recipes-support/curl/curl/run-ptest index 8f9c20f34df..acd2892f80e 100644 --- a/meta/recipes-support/curl/curl/run-ptest +++ b/meta/recipes-support/curl/curl/run-ptest @@ -1,6 +1,11 @@ #!/bin/sh + cd tests -{ ./runtests.pl -a -n -s -j4 !flaky || echo "FAIL: curl" ; } | sed \ - -e 's|\([^ ]* *\) \([^ ]* *\)...OK|PASS: \1 \2|' \ - -e 's|\([^ ]* *\) \([^ ]* *\)...FAILED|FAIL: \1 \2|' \ - -e 's/Warning: test[0-9]\+ not present in tests\/data\/Makefile.inc//' + +# Run all tests, don't stop on first failure +# Don't use valgrind if it is found +# Use automake-style output +# Run four tests in parallel +# Print log output on failure +# Don't run the flaky or timing dependent tests +./runtests.pl -a -n -am -j4 -p '!flaky !timing-dependent' diff --git a/meta/recipes-support/curl/curl_8.6.0.bb b/meta/recipes-support/curl/curl_8.6.0.bb index b01f4542a22..d4473d868e2 100644 --- a/meta/recipes-support/curl/curl_8.6.0.bb +++ b/meta/recipes-support/curl/curl_8.6.0.bb @@ -125,6 +125,7 @@ RDEPENDS:${PN}-ptest += " \ perl-module-storable \ perl-module-time-hires \ " +RDEPENDS:${PN}-ptest:append:libc-glibc = " locale-base-en-us" PACKAGES =+ "lib${BPN}" From patchwork Fri Mar 15 14:37:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 41008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0D91C54E58 for ; Fri, 15 Mar 2024 14:37:50 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.24116.1710513461747993713 for ; Fri, 15 Mar 2024 07:37:41 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B0DDFDA7 for ; Fri, 15 Mar 2024 07:38:16 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 159C33F73F for ; Fri, 15 Mar 2024 07:37:40 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/3] curl: increase test timeouts Date: Fri, 15 Mar 2024 14:37:37 +0000 Message-Id: <20240315143738.1234858-2-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240315143738.1234858-1-ross.burton@arm.com> References: <20240315143738.1234858-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 15 Mar 2024 14:37:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/197134 From: Ross Burton We often see multiple curl tests fail during ptest runs, the actual test varies but the output is like this: FAIL: 337: protoc! There was no content at all in the file log/3/server.input. Server glitch? Total curl failure? Returned: 28 Error code 28 is CURLE_OPERATION_TIMEDOUT, so this is almost certainly due to a loaded machine resulting in the tests running slowly. It is notable that the test runner explicitly passes --max-time=13 to curl, so experiment and change this to 600 to see if this solves the problem. [ YOCTO #15268 ] Signed-off-by: Ross Burton --- .../curl/curl/no-test-timeout.patch | 18 ++++++++++++++++++ meta/recipes-support/curl/curl_8.6.0.bb | 1 + 2 files changed, 19 insertions(+) create mode 100644 meta/recipes-support/curl/curl/no-test-timeout.patch diff --git a/meta/recipes-support/curl/curl/no-test-timeout.patch b/meta/recipes-support/curl/curl/no-test-timeout.patch new file mode 100644 index 00000000000..b4cfe716db7 --- /dev/null +++ b/meta/recipes-support/curl/curl/no-test-timeout.patch @@ -0,0 +1,18 @@ +Set the max-time timeout to 600 so the timeout is 10 minutes instead of 13 seconds. + +Upstream-Status: Inappropriate +Signed-off-by: Ross Burton + +diff --git a/tests/servers.pm b/tests/servers.pm +index d4472d509..aeab62c47 100644 +--- a/tests/servers.pm ++++ b/tests/servers.pm +@@ -120,7 +120,7 @@ my $sshdverstr; # for socks server, ssh daemon version string + my $sshderror; # for socks server, ssh daemon version error + my %doesntrun; # servers that don't work, identified by pidfile + my %PORT = (nolisten => 47); # port we use for a local non-listening service +-my $server_response_maxtime=13; ++my $server_response_maxtime=600; + my $httptlssrv = find_httptlssrv(); + my %run; # running server + my %runcert; # cert file currently in use by an ssl running server diff --git a/meta/recipes-support/curl/curl_8.6.0.bb b/meta/recipes-support/curl/curl_8.6.0.bb index d4473d868e2..ea69164d765 100644 --- a/meta/recipes-support/curl/curl_8.6.0.bb +++ b/meta/recipes-support/curl/curl_8.6.0.bb @@ -13,6 +13,7 @@ SRC_URI = " \ https://curl.se/download/${BP}.tar.xz \ file://run-ptest \ file://disable-tests \ + file://no-test-timeout.patch \ " SRC_URI[sha256sum] = "3ccd55d91af9516539df80625f818c734dc6f2ecf9bada33c76765e99121db15" From patchwork Fri Mar 15 14:37:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 41009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1CBFC54E67 for ; Fri, 15 Mar 2024 14:37:50 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.24117.1710513462452193509 for ; Fri, 15 Mar 2024 07:37:42 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5806BC15 for ; Fri, 15 Mar 2024 07:38:17 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B127E3F73F for ; Fri, 15 Mar 2024 07:37:41 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/3] gstreamer1.0: improve test reliability Date: Fri, 15 Mar 2024 14:37:38 +0000 Message-Id: <20240315143738.1234858-3-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240315143738.1234858-1-ross.burton@arm.com> References: <20240315143738.1234858-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 15 Mar 2024 14:37:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/197135 From: Ross Burton First, libcheck has the ability to increase all test timeouts by an arbitrary multiplier. Because we run our tests on loaded build machines, increase all timeouts by 10x to reduce the chance of load causing failures. Second, use GST_CHECKS_IGNORE to list test cases that should be skipped. Drop skip-aggregator-test.patch as this is now redundant, and also skip gstnetclientclock.c:test_functioning as this is very sensitive to load. [ YOCTO #14808 ] Signed-off-by: Ross Burton --- .../gstreamer/gstreamer1.0/run-ptest | 16 ++++++++- .../gstreamer1.0/skip-aggregator-test.patch | 35 ------------------- .../gstreamer/gstreamer1.0_1.22.10.bb | 1 - 3 files changed, 15 insertions(+), 37 deletions(-) delete mode 100644 meta/recipes-multimedia/gstreamer/gstreamer1.0/skip-aggregator-test.patch diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0/run-ptest b/meta/recipes-multimedia/gstreamer/gstreamer1.0/run-ptest index 0cfa955f033..7d0312005f4 100755 --- a/meta/recipes-multimedia/gstreamer/gstreamer1.0/run-ptest +++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0/run-ptest @@ -1,2 +1,16 @@ -#!/usr/bin/env sh +#! /bin/sh + +# Multiply all timeouts by ten so they're more likely to work +# on a loaded system. +export CK_TIMEOUT_MULTIPLIER=5 + +# Skip some tests that we know are problematic +export GST_CHECKS_IGNORE="" +# gstnetclientclock.c:test_functioning is very sensitive to load +GST_CHECKS_IGNORE="$GST_CHECKS_IGNORE,test_functioning" + +# aggregator.c:test_infinite_seek_50_src_live is known to be flaky +# https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/410 +GST_CHECKS_IGNORE="$GST_CHECKS_IGNORE,test_infinite_seek_50_src_live" + gnome-desktop-testing-runner gstreamer diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0/skip-aggregator-test.patch b/meta/recipes-multimedia/gstreamer/gstreamer1.0/skip-aggregator-test.patch deleted file mode 100644 index 81337512fdd..00000000000 --- a/meta/recipes-multimedia/gstreamer/gstreamer1.0/skip-aggregator-test.patch +++ /dev/null @@ -1,35 +0,0 @@ -From 9b72aa7cdbc2a81cffc6f855933afe90c81046d5 Mon Sep 17 00:00:00 2001 -From: Ross Burton -Date: Wed, 28 Feb 2024 12:40:34 +0000 -Subject: [PATCH] Skip aggregator test - -This test case is known to be flaky upstream[1] and often fails on the -autobuilder[2], so skip it until this has been resolved upstream. - -[1] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/410 -[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15054 - -Upstream-Status: Inappropriate -Signed-off-by: Ross Burton ---- - tests/check/libs/aggregator.c | 4 +++- - 1 file changed, 3 insertions(+), 1 deletion(-) - -diff --git a/tests/check/libs/aggregator.c b/tests/check/libs/aggregator.c -index 1f2c5b4..27b3ac7 100644 ---- a/tests/check/libs/aggregator.c -+++ b/tests/check/libs/aggregator.c -@@ -1475,7 +1475,9 @@ gst_aggregator_suite (void) - tcase_add_test (general, test_flushing_seek); - tcase_add_test (general, test_infinite_seek); - tcase_add_test (general, test_infinite_seek_50_src); -- tcase_add_test (general, test_infinite_seek_50_src_live); -+ // This test case is known to be flaky, remove it until resolved: -+ // https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/410 -+ // tcase_add_test (general, test_infinite_seek_50_src_live); - tcase_add_test (general, test_linear_pipeline); - tcase_add_test (general, test_two_src_pipeline); - tcase_add_test (general, test_timeout_pipeline); --- -2.34.1 - diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.22.10.bb b/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.22.10.bb index c16f0de3b72..0c944e9e110 100644 --- a/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.22.10.bb +++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.22.10.bb @@ -21,7 +21,6 @@ SRC_URI = "https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-${PV}.tar.x file://0002-tests-add-support-for-install-the-tests.patch \ file://0003-tests-use-a-dictionaries-for-environment.patch \ file://0004-tests-add-helper-script-to-run-the-installed_tests.patch \ - file://skip-aggregator-test.patch \ " SRC_URI[sha256sum] = "969aaef396f252ce925132a4be2aa004e0320f5c1baf0acaaae09c544a6759f4"