mbox series

[0/2] Add a display limit for regression report generation

Message ID 20231019095352.25923-1-alexis.lothore@bootlin.com
Headers show
Series Add a display limit for regression report generation | expand

Message

Alexis Lothoré Oct. 19, 2023, 9:53 a.m. UTC 
It has been observed that useful information in regression report can be
drowned in huge regression lists which are often false-positives (for
example, a whole set of tests has been temporarily disabled).

This series brings a default limit to how many changes are displayed per
base/target comparison. This default can still be overriden on commandline,
for example to have a better look at the whole regression list when trying
to debug an issue (i.e. by disabling the limit)

First commit implement the limit, its default value and the corresponding
commandline option in resulttool.
Second commit allow yocto_testresults_query.py to drive this value.

As a result, one can for example do the following:
- yocto_testresults_query 4.3_M1 4.3_M2
  -> will display at most 50 regressions per test
- yocto_testresults_query -l 10 4.3_M1 4.3_M2
  -> override the display limit and reduce it to 10 regressions per pair.
- yocto_testresults_query -l 0 4.3_M1 4.3_M2
  -> disable the display limit, print all regressions

An example of regression report with display limit can be found here:
https://pastebin.com/6QbfGstR

Alexis Lothoré (2):
  scripts/resulttool: limit the number of changes displayed per test
  scripts/yocto_testresults_query: add option to change display limit

 scripts/lib/resulttool/regression.py | 23 +++++++++++++++++++----
 scripts/yocto_testresults_query.py   | 13 ++++++++++---
 2 files changed, 29 insertions(+), 7 deletions(-)

Comments

Alexandre Belloni Oct. 20, 2023, 6:05 a.m. UTC | #1 
Hello Alexis,

On 19/10/2023 11:53:50+0200, Alexis Lothor� via lists.openembedded.org wrote:
> It has been observed that useful information in regression report can be
> drowned in huge regression lists which are often false-positives (for
> example, a whole set of tests has been temporarily disabled).
> 
> This series brings a default limit to how many changes are displayed per
> base/target comparison. This default can still be overriden on commandline,
> for example to have a better look at the whole regression list when trying
> to debug an issue (i.e. by disabling the limit)
> 
> First commit implement the limit, its default value and the corresponding
> commandline option in resulttool.
> Second commit allow yocto_testresults_query.py to drive this value.
> 
> As a result, one can for example do the following:
> - yocto_testresults_query 4.3_M1 4.3_M2
>   -> will display at most 50 regressions per test
> - yocto_testresults_query -l 10 4.3_M1 4.3_M2
>   -> override the display limit and reduce it to 10 regressions per pair.
> - yocto_testresults_query -l 0 4.3_M1 4.3_M2
>   -> disable the display limit, print all regressions
> 
> An example of regression report with display limit can be found here:
> https://pastebin.com/6QbfGstR
> 
> Alexis Lothor� (2):
>   scripts/resulttool: limit the number of changes displayed per test
>   scripts/yocto_testresults_query: add option to change display limit
> 

This causes:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5886/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5935/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5952/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5936/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2296/steps/14/logs/stdio


2023-10-19 07:28:44,229 - oe-selftest - INFO - resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names (subunit.RemotedTestCase)
2023-10-19 07:28:44,229 - oe-selftest - INFO -  ... ERROR
Stderr:
2023-10-19 07:05:57,080 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-271681/conf/local.conf
2023-10-19 07:05:57,081 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 07:28:44,229 - oe-selftest - INFO - 6: 24/42 185/543 (0.01s) (0 failed) (resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names)
2023-10-19 07:28:44,230 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/resulttooltests.py", line 372, in test_can_match_non_static_ptest_names
    result, resultstring = regression.compare_result(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: compare_result() missing 1 required positional argument: 'display_limit'


2023-10-19 07:28:44,311 - oe-selftest - INFO - resulttooltests.ResultToolTests.test_regression_can_get_regression_result (subunit.RemotedTestCase)
2023-10-19 07:28:44,311 - oe-selftest - INFO -  ... ERROR
Stderr:
2023-10-19 07:05:57,080 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-271681/conf/local.conf
2023-10-19 07:05:57,081 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 07:28:44,312 - oe-selftest - INFO - 6: 31/42 192/543 (0.01s) (2 failed) (resulttooltests.ResultToolTests.test_regression_can_get_regression_result)
2023-10-19 07:28:44,312 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/resulttooltests.py", line 85, in test_regression_can_get_regression_result
    result, text = regression.compare_result(self.logger, "BaseTestRunName", "TargetTestRunName", base_result_data, target_result_data)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: compare_result() missing 1 required positional argument: 'display_limit'


>  scripts/lib/resulttool/regression.py | 23 +++++++++++++++++++----
>  scripts/yocto_testresults_query.py   | 13 ++++++++++---
>  2 files changed, 29 insertions(+), 7 deletions(-)
> 
> -- 
> 2.42.0
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189440): https://lists.openembedded.org/g/openembedded-core/message/189440
> Mute This Topic: https://lists.openembedded.org/mt/102057050/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Alexis Lothoré Oct. 20, 2023, 6:32 a.m. UTC | #2 
Hello Alexandre,

On 10/20/23 08:05, Alexandre Belloni via lists.openembedded.org wrote:
> Hello Alexis,> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5886/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5935/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5952/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5936/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2296/steps/14/logs/stdio
> 
> 
> 2023-10-19 07:28:44,229 - oe-selftest - INFO - resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names (subunit.RemotedTestCase)
> 2023-10-19 07:28:44,229 - oe-selftest - INFO -  ... ERROR
> Stderr:
> 2023-10-19 07:05:57,080 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-271681/conf/local.conf
> 2023-10-19 07:05:57,081 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
> 2023-10-19 07:28:44,229 - oe-selftest - INFO - 6: 24/42 185/543 (0.01s) (0 failed) (resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names)
> 2023-10-19 07:28:44,230 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
>   File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/resulttooltests.py", line 372, in test_can_match_non_static_ptest_names
>     result, resultstring = regression.compare_result(
>                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
> TypeError: compare_result() missing 1 required positional argument: 'display_limit'

Looks like I forgot to update/run selftests locally before pushing... Sorry for
the mess and thanks for the notification.
Anyway, Ross and Richard gave some relevant feedback
(https://lore.kernel.org/yocto/90D7FE40-32EB-4D67-B099-9E0C764FC258@arm.com/) on
this so I will prepare a v2, which fill also fix this selftest.

Thanks,

Alexis