diff mbox series

scripts/resulttool: do not try to parse metadata as tests results

Message ID 20231006135903.96652-1-alexis.lothore@bootlin.com
State New
Headers show
Series scripts/resulttool: do not try to parse metadata as tests results | expand

Commit Message

Alexis Lothoré Oct. 6, 2023, 1:59 p.m. UTC 
From: Alexis Lothoré <alexis.lothore@bootlin.com>

When regression report is computed during a CI build, a lot of errors
often appears regarding missing test status:

ERROR: Failed to retrieved base test case status: ptestresult.sections
ERROR: Failed to retrieved base test case status: ptestresult.sections
ERROR: Failed to retrieved base test case status: reproducible
ERROR: Failed to retrieved base test case status: reproducible.rawlogs
[...]

Those errors are caused by entries in test results which are not exactly
test results (i.e. an entry with a relevant "status" field containing value such as
"PASSED", "FAILED, "SKIPPED", etc) but additional data, which depends on
the log parser associated to the test, or tests which store results in a
different way. For example, the ptestresult.sections entry is generated by
the ptest log parser and can contain additional info about ptest such as
"begin", "end", "duration", "exitcode" or "timeout". Another example is a
"reproducible" section, which does not have a "status" field but rather
contains a big "files" entry containing lists of identical, missing, or
different files between two builds.

Remove those errors by adding a list of known entries which do not hold
test results as expected by resulttool, and by ignoring those keys when
encountered during test results comparison. I could also have completely
removed the warning about missing test case status, but that would
silently hide any real future issue with relevant test results

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
 scripts/lib/resulttool/regression.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
diff mbox series

Patch

diff --git a/scripts/lib/resulttool/regression.py b/scripts/lib/resulttool/regression.py
index 3d64b8f4af7c..e15a268c0206 100644
--- a/scripts/lib/resulttool/regression.py
+++ b/scripts/lib/resulttool/regression.py
@@ -78,6 +78,16 @@  STATUS_STRINGS = {
     "None": "No matching test result"
 }
 
+TEST_KEY_WHITELIST = [
+    "ltpposixresult.rawlogs",
+    "ltpposixresult.sections",
+    "ltpresult.rawlogs",
+    "ltpresult.sections",
+    "ptestresult.sections",
+    "reproducible",
+    "reproducible.rawlogs"
+]
+
 def test_has_at_least_one_matching_tag(test, tag_list):
     return "oetags" in test and any(oetag in tag_list for oetag in test["oetags"])
 
@@ -189,6 +199,10 @@  def compare_result(logger, base_name, target_name, base_result, target_result):
 
     if base_result and target_result:
         for k in base_result:
+            # Some entries present in test results are known not to be test
+            # results but metadata about tests
+            if k in TEST_KEY_WHITELIST:
+                continue
             base_testcase = base_result[k]
             base_status = base_testcase.get('status')
             if base_status: