| Message ID | 20231004125731.1785803-1-yoann.congal@smile.fr |
|---|---|
| Headers | show |
| Series | insane: Add unimplemented-ptest checks | expand |
On Wed, 4 Oct 2023 at 14:57, Yoann Congal <yoann.congal@smile.fr> wrote: > Currently, this check find: > * 309 unimplemented ptest in oe-core/meta-poky/meta-yocto-bsp > * 827 unimplemented ptest in meta-openembedded Does this mean there will be this many warnings in everyone's builds? If so, this cannot be enabled by default; any new QA warning must either not be triggered at all on the current revision (and is added specifically to prevent regressions), or come with a realistic plan to address the warnings. Implementing hundreds of ptests in a short time frame is not realistic. On the other hand, the output could go into some kind of report, I'm just not sure which one. Alex
On Thu, 2023-10-05 at 10:47 +0200, Alexander Kanavin wrote: > On Wed, 4 Oct 2023 at 14:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > Currently, this check find: > > * 309 unimplemented ptest in oe-core/meta-poky/meta-yocto-bsp > > * 827 unimplemented ptest in meta-openembedded > > Does this mean there will be this many warnings in everyone's builds? > If so, this cannot be enabled by default; any new QA warning must > either not be triggered at all on the current revision (and is added > specifically to prevent regressions), or come with a realistic plan to > address the warnings. Implementing hundreds of ptests in a short time > frame is not realistic. > > On the other hand, the output could go into some kind of report, I'm > just not sure which one. It isn't enabled in default builds and can't be for the reason you mention so I think the intention is that we can do reporting using this periodically. Ultimately we might have it as a new "log" or "report" level and just generate reports during the build. We did used to do that for previous "insane" tests. We just need to think about how that would work with sstate (warnings are already an issue with sstate). Cheers, Richard
Hi, Le jeu. 5 oct. 2023 à 10:47, Alexander Kanavin <alex.kanavin@gmail.com> a écrit : > On Wed, 4 Oct 2023 at 14:57, Yoann Congal <yoann.congal@smile.fr> wrote: > > Currently, this check find: > > * 309 unimplemented ptest in oe-core/meta-poky/meta-yocto-bsp > > * 827 unimplemented ptest in meta-openembedded > > Does this mean there will be this many warnings in everyone's builds? > If so, this cannot be enabled by default; any new QA warning must > either not be triggered at all on the current revision (and is added > specifically to prevent regressions), or come with a realistic plan to > address the warnings. Implementing hundreds of ptests in a short time > frame is not realistic. > Of course! The warning is not enabled by default. One has to add to local.conf: WARN_QA += "unimplemented-ptest" See [PATCH v2 1/4] insane: Add unimplemented-ptest infrastructure : https://lists.openembedded.org/g/openembedded-core/message/188681 > > On the other hand, the output could go into some kind of report, I'm > just not sure which one. > > Alex >
On Wed, 2023-10-04 at 14:57 +0200, Yoann Congal wrote: > To increase ptest coverage we can check if the sources of a recipe looks like > it contains unittest and warn the user that a test may be implemented there. > > This series provide the check infrastructure as a package QA check and some checks for : > python pytest, perl Test::, meson, cmake, autotools-based tests... as well as > the naive check of "Is there a test/ directory in the sources?" which work > surprisingly well. > > Currently, this check find: > * 309 unimplemented ptest in oe-core/meta-poky/meta-yocto-bsp > * 827 unimplemented ptest in meta-openembedded > > Full list : https://gist.githubusercontent.com/ycongal-smile/0969e92aec1ca37e61b6cdbcaf1f6885/raw/b1be8a5b576e144b6563f5481d31b32de9b22368/gistfile1.txt > > v1->v2: > * Corrected: shortlog tag > * Fixed infinite loop on source package containing symlink loops (e.g. md5deep) > * Added test result > > Jérémy Rosen (4): > insane: Add unimplemented-ptest infrastructure > insane: Detect python and perl based tests > insane: Detect build-system test harnesses > insane: Add a naive heuristic to detect test subdirectories > > meta/classes-global/insane.bbclass | 50 ++++++++++++++++++++++++++++++ > 1 file changed, 50 insertions(+) Somehow this series causes: oe-selftest -r sstatetests.SStateHashSameSigs2.test_sstate_allarch_samesigs to fail and the other test failures in: https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5816/steps/14/logs/stdio I've not looked into why, just narrowed it to this series. Cheers, Richard
Le ven. 6 oct. 2023 à 09:20, Richard Purdie < richard.purdie@linuxfoundation.org> a écrit : > On Wed, 2023-10-04 at 14:57 +0200, Yoann Congal wrote: > > To increase ptest coverage we can check if the sources of a recipe looks > like > > it contains unittest and warn the user that a test may be implemented > there. > > > > This series provide the check infrastructure as a package QA check and > some checks for : > > python pytest, perl Test::, meson, cmake, autotools-based tests... as > well as > > the naive check of "Is there a test/ directory in the sources?" which > work > > surprisingly well. > > > > Currently, this check find: > > * 309 unimplemented ptest in oe-core/meta-poky/meta-yocto-bsp > > * 827 unimplemented ptest in meta-openembedded > > > > Full list : > https://gist.githubusercontent.com/ycongal-smile/0969e92aec1ca37e61b6cdbcaf1f6885/raw/b1be8a5b576e144b6563f5481d31b32de9b22368/gistfile1.txt > > > > v1->v2: > > * Corrected: shortlog tag > > * Fixed infinite loop on source package containing symlink loops (e.g. > md5deep) > > * Added test result > > > > Jérémy Rosen (4): > > insane: Add unimplemented-ptest infrastructure > > insane: Detect python and perl based tests > > insane: Detect build-system test harnesses > > insane: Add a naive heuristic to detect test subdirectories > > > > meta/classes-global/insane.bbclass | 50 ++++++++++++++++++++++++++++++ > > 1 file changed, 50 insertions(+) > > Somehow this series causes: > > oe-selftest -r > sstatetests.SStateHashSameSigs2.test_sstate_allarch_samesigs > > to fail and the other test failures in: > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5816/steps/14/logs/stdio > > I've not looked into why, just narrowed it to this series. > Ok, thanks! We'll look into this.