mbox series

[v2,0/4] Add a display limit for regression report generation

Message ID 20231022174938.7012-1-alexis.lothore@bootlin.com
Headers show
Series Add a display limit for regression report generation | expand

Message

Alexis Lothoré Oct. 22, 2023, 5:49 p.m. UTC 
It has been observed that useful information in regression report can be
drowned in huge regression lists which are often false-positives (for
example, a whole set of tests has been temporarily disabled). Additionally,
the regression report usually starts with not-so-useful information when
searching for regressions (basically, big lists of tests which have not
regressed).

This series aims to improve regression display by rearranging info order as
well as bringing a default limit to how many changes are displayed per
base/target comparison. This new version follows up suggestions discussed
in [1].

One consequence of this series is the new available option for
yocto_testresults_query:
- yocto_testresults_query 4.3_M1 4.3_M2
  -> will display at most 50 regressions per test type
- yocto_testresults_query -l 10 4.3_M1 4.3_M2
  -> override the display limit and reduce it to 10 regressions per test
     type
- yocto_testresults_query -l 0 4.3_M1 4.3_M2
  -> disable the display limit, print all regressions

An example of regression report with those modifications can be found at
[2]

Note: this series does not take care yet of improving base/target pairs
selection based on machine/distro (issue raised by Ross  in [1]),
because I still need to have a better look at this before tweaking anything
there.

Changes in v2:
- use the display limit for each test type instead of using it as a global
  limit
- display regressions summaries before detailed list
- display regressions first, display matched/improved tests after, as
  "additional data"
- Reduce less useful info size
- fix testresult selftests

[1] https://lore.kernel.org/yocto/90D7FE40-32EB-4D67-B099-9E0C764FC258@arm.com/
[2] https://pastebin.com/JHFLF0sK

Alexis Lothoré (4):
  scripts/resulttool: limit the number of changes displayed per test
  scripts/resulttool: rearrange regressions report order
  scripts/resulttool: make additional info more compact
  scripts/yocto_testresults_query: add option to change display limit

 scripts/lib/resulttool/regression.py | 60 +++++++++++++++++++++++-----
 scripts/yocto_testresults_query.py   | 13 ++++--
 2 files changed, 60 insertions(+), 13 deletions(-)