From patchwork Fri Oct 6 13:59:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 31773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46ED5E81DFE for ; Fri, 6 Oct 2023 13:58:20 +0000 (UTC) Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by mx.groups.io with SMTP id smtpd.web11.13369.1696600690583776568 for ; Fri, 06 Oct 2023 06:58:11 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=e6OMHWgb; spf=pass (domain: bootlin.com, ip: 217.70.183.193, mailfrom: alexis.lothore@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 9F574240005; Fri, 6 Oct 2023 13:58:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1696600688; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=k25nQ+N7A512fiZ2A5mZYXR9TW2i7IMYrnRaFNmADmI=; b=e6OMHWgb8tkVw4YFqjChOZ0yEUxM3DVjoY/I3aQts9148NmhAAVS/hot9gqL3re8nTH8KY a+3WVIkiSdo0qO7WEIQlzRgo55x1dFQmyeT7W6DGObTn9BNjnEuOgxEi25TPTglABvxoUo XaZFwn2Qd++qppWpNyUZisGFRSMvAJ2/Uu829z6wMhTA7JPqTIBFxywsVcF2fClc9humRD kQddNfkK3NJqKvRD+ZuIHUkg9y4sJpP8bNzzINUkyKrZaG4XzXrlGAyQpcNYum8Lbpmmv1 Ri7rp13wGWOnTHKiA/S+YheE6yXLktmBjnQYfDbz++3Lw+6qqo93Nyria6MzyQ== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [PATCH] scripts/resulttool: do not try to parse metadata as tests results Date: Fri, 6 Oct 2023 15:59:03 +0200 Message-ID: <20231006135903.96652-1-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 X-GND-Sasl: alexis.lothore@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Oct 2023 13:58:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/188770 From: Alexis Lothoré When regression report is computed during a CI build, a lot of errors often appears regarding missing test status: ERROR: Failed to retrieved base test case status: ptestresult.sections ERROR: Failed to retrieved base test case status: ptestresult.sections ERROR: Failed to retrieved base test case status: reproducible ERROR: Failed to retrieved base test case status: reproducible.rawlogs [...] Those errors are caused by entries in test results which are not exactly test results (i.e. an entry with a relevant "status" field containing value such as "PASSED", "FAILED, "SKIPPED", etc) but additional data, which depends on the log parser associated to the test, or tests which store results in a different way. For example, the ptestresult.sections entry is generated by the ptest log parser and can contain additional info about ptest such as "begin", "end", "duration", "exitcode" or "timeout". Another example is a "reproducible" section, which does not have a "status" field but rather contains a big "files" entry containing lists of identical, missing, or different files between two builds. Remove those errors by adding a list of known entries which do not hold test results as expected by resulttool, and by ignoring those keys when encountered during test results comparison. I could also have completely removed the warning about missing test case status, but that would silently hide any real future issue with relevant test results Signed-off-by: Alexis Lothoré --- scripts/lib/resulttool/regression.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/scripts/lib/resulttool/regression.py b/scripts/lib/resulttool/regression.py index 3d64b8f4af7c..e15a268c0206 100644 --- a/scripts/lib/resulttool/regression.py +++ b/scripts/lib/resulttool/regression.py @@ -78,6 +78,16 @@ STATUS_STRINGS = { "None": "No matching test result" } +TEST_KEY_WHITELIST = [ + "ltpposixresult.rawlogs", + "ltpposixresult.sections", + "ltpresult.rawlogs", + "ltpresult.sections", + "ptestresult.sections", + "reproducible", + "reproducible.rawlogs" +] + def test_has_at_least_one_matching_tag(test, tag_list): return "oetags" in test and any(oetag in tag_list for oetag in test["oetags"]) @@ -189,6 +199,10 @@ def compare_result(logger, base_name, target_name, base_result, target_result): if base_result and target_result: for k in base_result: + # Some entries present in test results are known not to be test + # results but metadata about tests + if k in TEST_KEY_WHITELIST: + continue base_testcase = base_result[k] base_status = base_testcase.get('status') if base_status: