| Message ID | 20230614085614.27951-1-alexis.lothore@bootlin.com |
|---|---|
| Headers | show |
| Series | fix test results storage for mickledore | expand |
On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via lists.yoctoproject.org wrote: > From: Alexis Lothoré <alexis.lothore@bootlin.com> > > This series is a follow-up for the 4.3_M1.rc1 regression report issue. > > It has been observed that the report is empty. This issue is linked to > configuration description in yocto-autobuilder-helper, and has been > identified through the following steps: > - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3) > and 4.3_M1.rc1 > - yocto-4.2 results are almost empty: we only find test results from Intel > QA (pushed _after_ the AB build) and not the AB test results > - tests results are managed by send-qa-email.send-qa-email uses resulttool > to systematically gather and store test results in local git directory > - however, it looks for basebranch/comparebranch to know if those results > can be pushed onto git server, and those variables depend on config.json > content > - yocto-4.2 (4.2.rc3) has been built on release branch mickledore > (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212) > - since mickledore is not yet described in config.json, send-qa-email > considers it as a "work" branch (contrary to a "release" branch) and does > not push test results > > As a consequence: > - first commit brings in python logger > - second commit adds a warning when such case happen, since we are able to > detect it > - third fix actually adds mickledore as a release branch to properly store > again test results > > There must be a more robust rework to do (because the issue will likely > happen on each major delivery), but I aimed for the quick and small fix to > quickly bring back tests results storage without breaking other things in > the process > > Alexis Lothoré (3): > scripts/send-qa-email: use logger instead of raw prints > scripts/send-qa-email: print warning when test results are not stored > config.json: add mickledore as direct push branch for test results Thanks for the analysis. I agree we need to somehow fix this properly. One solution might be to always push for poky if the branch name doesn't end with -next? Since we have the release artefacts for the release, could we add the test results after the fact now? Id' be interested to see the 4.3 M1 to 4.2 comparison rerun with that added. Cheers, Richard
On 6/14/23 12:31, Richard Purdie wrote: > On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via > lists.yoctoproject.org wrote: >> From: Alexis Lothoré <alexis.lothore@bootlin.com> >> >> This series is a follow-up for the 4.3_M1.rc1 regression report issue. >> >> It has been observed that the report is empty. This issue is linked to >> configuration description in yocto-autobuilder-helper, and has been >> identified through the following steps: >> - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3) >> and 4.3_M1.rc1 >> - yocto-4.2 results are almost empty: we only find test results from Intel >> QA (pushed _after_ the AB build) and not the AB test results >> - tests results are managed by send-qa-email.send-qa-email uses resulttool >> to systematically gather and store test results in local git directory >> - however, it looks for basebranch/comparebranch to know if those results >> can be pushed onto git server, and those variables depend on config.json >> content >> - yocto-4.2 (4.2.rc3) has been built on release branch mickledore >> (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212) >> - since mickledore is not yet described in config.json, send-qa-email >> considers it as a "work" branch (contrary to a "release" branch) and does >> not push test results >> >> As a consequence: >> - first commit brings in python logger >> - second commit adds a warning when such case happen, since we are able to >> detect it >> - third fix actually adds mickledore as a release branch to properly store >> again test results >> >> There must be a more robust rework to do (because the issue will likely >> happen on each major delivery), but I aimed for the quick and small fix to >> quickly bring back tests results storage without breaking other things in >> the process >> >> Alexis Lothoré (3): >> scripts/send-qa-email: use logger instead of raw prints >> scripts/send-qa-email: print warning when test results are not stored >> config.json: add mickledore as direct push branch for test results > > Thanks for the analysis. I agree we need to somehow fix this properly. > One solution might be to always push for poky if the branch name > doesn't end with -next? That might work indeed. If we are sure enough that no custom/feature branch will be used in poky with send-qa-email (ie, only in poky-contrib), I can do the fix this way > > Since we have the release artefacts for the release, could we add the > test results after the fact now?> > Id' be interested to see the 4.3 M1 to 4.2 comparison rerun with that > added. I am not sure about where to find those artifacts for yocto-4.2 ? If you are referring to https://autobuilder.yocto.io/pub/, yocto-4.2 has already been removed from there. And if you are referring to the archived release on main site (https://downloads.yoctoproject.org/releases/yocto/yocto-4.2/poky-21790e71d55f417f27cd51fae9dd47549758d4a0.tar.bz2), it does contain a single, 40 line testresults.json, so that's definitely not the full AB tests results. > > Cheers, > > Richard > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#60297): https://lists.yoctoproject.org/g/yocto/message/60297 > Mute This Topic: https://lists.yoctoproject.org/mt/99523809/7394887 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/12378809/7394887/1227806781/xyzzy [alexis.lothore@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 14/06/2023 14:15:54+0200, Alexis Lothor� wrote: > On 6/14/23 12:31, Richard Purdie wrote: > > On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothor� via > > lists.yoctoproject.org wrote: > >> From: Alexis Lothor� <alexis.lothore@bootlin.com> > >> > >> This series is a follow-up for the 4.3_M1.rc1 regression report issue. > >> > >> It has been observed that the report is empty. This issue is linked to > >> configuration description in yocto-autobuilder-helper, and has been > >> identified through the following steps: > >> - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3) > >> and 4.3_M1.rc1 > >> - yocto-4.2 results are almost empty: we only find test results from Intel > >> QA (pushed _after_ the AB build) and not the AB test results > >> - tests results are managed by send-qa-email.send-qa-email uses resulttool > >> to systematically gather and store test results in local git directory > >> - however, it looks for basebranch/comparebranch to know if those results > >> can be pushed onto git server, and those variables depend on config.json > >> content > >> - yocto-4.2 (4.2.rc3) has been built on release branch mickledore > >> (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212) > >> - since mickledore is not yet described in config.json, send-qa-email > >> considers it as a "work" branch (contrary to a "release" branch) and does > >> not push test results > >> > >> As a consequence: > >> - first commit brings in python logger > >> - second commit adds a warning when such case happen, since we are able to > >> detect it > >> - third fix actually adds mickledore as a release branch to properly store > >> again test results > >> > >> There must be a more robust rework to do (because the issue will likely > >> happen on each major delivery), but I aimed for the quick and small fix to > >> quickly bring back tests results storage without breaking other things in > >> the process > >> > >> Alexis Lothor� (3): > >> scripts/send-qa-email: use logger instead of raw prints > >> scripts/send-qa-email: print warning when test results are not stored > >> config.json: add mickledore as direct push branch for test results > > > > Thanks for the analysis. I agree we need to somehow fix this properly. > > One solution might be to always push for poky if the branch name > > doesn't end with -next? > > That might work indeed. If we are sure enough that no custom/feature branch will > be used in poky with send-qa-email (ie, only in poky-contrib), I can do the fix > this way I sometimes use a different branch name when testing things out (like 64 bit time) but as long as we all know, we can probably ensure this ends in -next. > > > > Since we have the release artefacts for the release, could we add the > > test results after the fact now?> > > Id' be interested to see the 4.3 M1 to 4.2 comparison rerun with that > > added. > > I am not sure about where to find those artifacts for yocto-4.2 ? If you are > referring to https://autobuilder.yocto.io/pub/, yocto-4.2 has already been > removed from there. And if you are referring to the archived release on main > site > (https://downloads.yoctoproject.org/releases/yocto/yocto-4.2/poky-21790e71d55f417f27cd51fae9dd47549758d4a0.tar.bz2), > it does contain a single, 40 line testresults.json, so that's definitely not the > full AB tests results. > > > > > Cheers, > > > > Richard > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#60297): https://lists.yoctoproject.org/g/yocto/message/60297 > > Mute This Topic: https://lists.yoctoproject.org/mt/99523809/7394887 > > Group Owner: yocto+owner@lists.yoctoproject.org > > Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/12378809/7394887/1227806781/xyzzy [alexis.lothore@bootlin.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > > > -- > Alexis Lothor�, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com >
On Wed, 2023-06-14 at 16:29 +0200, Alexandre Belloni wrote: > On 14/06/2023 14:15:54+0200, Alexis Lothoré wrote: > > On 6/14/23 12:31, Richard Purdie wrote: > > > On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via > > > lists.yoctoproject.org wrote: > > > > From: Alexis Lothoré <alexis.lothore@bootlin.com> > > > > > > > > This series is a follow-up for the 4.3_M1.rc1 regression report issue. > > > > > > > > It has been observed that the report is empty. This issue is linked to > > > > configuration description in yocto-autobuilder-helper, and has been > > > > identified through the following steps: > > > > - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3) > > > > and 4.3_M1.rc1 > > > > - yocto-4.2 results are almost empty: we only find test results from Intel > > > > QA (pushed _after_ the AB build) and not the AB test results > > > > - tests results are managed by send-qa-email.send-qa-email uses resulttool > > > > to systematically gather and store test results in local git directory > > > > - however, it looks for basebranch/comparebranch to know if those results > > > > can be pushed onto git server, and those variables depend on config.json > > > > content > > > > - yocto-4.2 (4.2.rc3) has been built on release branch mickledore > > > > (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212) > > > > - since mickledore is not yet described in config.json, send-qa-email > > > > considers it as a "work" branch (contrary to a "release" branch) and does > > > > not push test results > > > > > > > > As a consequence: > > > > - first commit brings in python logger > > > > - second commit adds a warning when such case happen, since we are able to > > > > detect it > > > > - third fix actually adds mickledore as a release branch to properly store > > > > again test results > > > > > > > > There must be a more robust rework to do (because the issue will likely > > > > happen on each major delivery), but I aimed for the quick and small fix to > > > > quickly bring back tests results storage without breaking other things in > > > > the process > > > > > > > > Alexis Lothoré (3): > > > > scripts/send-qa-email: use logger instead of raw prints > > > > scripts/send-qa-email: print warning when test results are not stored > > > > config.json: add mickledore as direct push branch for test results > > > > > > Thanks for the analysis. I agree we need to somehow fix this properly. > > > One solution might be to always push for poky if the branch name > > > doesn't end with -next? > > > > That might work indeed. If we are sure enough that no custom/feature branch will > > be used in poky with send-qa-email (ie, only in poky-contrib), I can do the fix > > this way > > I sometimes use a different branch name when testing things out (like 64 > bit time) but as long as we all know, we can probably ensure this ends in > -next. That would always be in poky-contrib though? Cheers, Richard
On 14/06/2023 15:32:25+0100, Richard Purdie wrote: > On Wed, 2023-06-14 at 16:29 +0200, Alexandre Belloni wrote: > > On 14/06/2023 14:15:54+0200, Alexis Lothor� wrote: > > > On 6/14/23 12:31, Richard Purdie wrote: > > > > On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothor� via > > > > lists.yoctoproject.org wrote: > > > > > From: Alexis Lothor� <alexis.lothore@bootlin.com> > > > > > > > > > > This series is a follow-up for the 4.3_M1.rc1 regression report issue. > > > > > > > > > > It has been observed that the report is empty. This issue is linked to > > > > > configuration description in yocto-autobuilder-helper, and has been > > > > > identified through the following steps: > > > > > - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3) > > > > > and 4.3_M1.rc1 > > > > > - yocto-4.2 results are almost empty: we only find test results from Intel > > > > > QA (pushed _after_ the AB build) and not the AB test results > > > > > - tests results are managed by send-qa-email.send-qa-email uses resulttool > > > > > to systematically gather and store test results in local git directory > > > > > - however, it looks for basebranch/comparebranch to know if those results > > > > > can be pushed onto git server, and those variables depend on config.json > > > > > content > > > > > - yocto-4.2 (4.2.rc3) has been built on release branch mickledore > > > > > (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212) > > > > > - since mickledore is not yet described in config.json, send-qa-email > > > > > considers it as a "work" branch (contrary to a "release" branch) and does > > > > > not push test results > > > > > > > > > > As a consequence: > > > > > - first commit brings in python logger > > > > > - second commit adds a warning when such case happen, since we are able to > > > > > detect it > > > > > - third fix actually adds mickledore as a release branch to properly store > > > > > again test results > > > > > > > > > > There must be a more robust rework to do (because the issue will likely > > > > > happen on each major delivery), but I aimed for the quick and small fix to > > > > > quickly bring back tests results storage without breaking other things in > > > > > the process > > > > > > > > > > Alexis Lothor� (3): > > > > > scripts/send-qa-email: use logger instead of raw prints > > > > > scripts/send-qa-email: print warning when test results are not stored > > > > > config.json: add mickledore as direct push branch for test results > > > > > > > > Thanks for the analysis. I agree we need to somehow fix this properly. > > > > One solution might be to always push for poky if the branch name > > > > doesn't end with -next? > > > > > > That might work indeed. If we are sure enough that no custom/feature branch will > > > be used in poky with send-qa-email (ie, only in poky-contrib), I can do the fix > > > this way > > > > I sometimes use a different branch name when testing things out (like 64 > > bit time) but as long as we all know, we can probably ensure this ends in > > -next. > > That would always be in poky-contrib though? > Indeed! > Cheers, > > Richard
On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via lists.yoctoproject.org wrote: > From: Alexis Lothoré <alexis.lothore@bootlin.com> > > This series is a follow-up for the 4.3_M1.rc1 regression report issue. > > It has been observed that the report is empty. This issue is linked to > configuration description in yocto-autobuilder-helper, and has been > identified through the following steps: > - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3) > and 4.3_M1.rc1 > - yocto-4.2 results are almost empty: we only find test results from Intel > QA (pushed _after_ the AB build) and not the AB test results > - tests results are managed by send-qa-email.send-qa-email uses resulttool > to systematically gather and store test results in local git directory > - however, it looks for basebranch/comparebranch to know if those results > can be pushed onto git server, and those variables depend on config.json > content > - yocto-4.2 (4.2.rc3) has been built on release branch mickledore > (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212) > - since mickledore is not yet described in config.json, send-qa-email > considers it as a "work" branch (contrary to a "release" branch) and does > not push test results > > As a consequence: > - first commit brings in python logger > - second commit adds a warning when such case happen, since we are able to > detect it > - third fix actually adds mickledore as a release branch to properly store > again test results > > There must be a more robust rework to do (because the issue will likely > happen on each major delivery), but I aimed for the quick and small fix to > quickly bring back tests results storage without breaking other things in > the process Thanks, I've merged this as it is a good first set of steps. As I mentioned, I think we should hardcode poky + "not ending with - next" as the test, then we shouldn't run into this issue again. I'd also like to retroactively push the test results for 4.2 since we have them and should be able to merge them onto the branch. I'd then like to see what the revised 4.3 M1 report looks like. Cheers, Richard
Hello Richard, Michael, On 6/15/23 15:41, Richard Purdie wrote: > On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via lists.yoctoproject.org wrote: >> From: Alexis Lothoré <alexis.lothore@bootlin.com> >> >> There must be a more robust rework to do (because the issue will likely >> happen on each major delivery), but I aimed for the quick and small fix to >> quickly bring back tests results storage without breaking other things in >> the process > > Thanks, I've merged this as it is a good first set of steps. > > As I mentioned, I think we should hardcode poky + "not ending with - > next" as the test, then we shouldn't run into this issue again. ACK, will do the fix > > I'd also like to retroactively push the test results for 4.2 since we > have them and should be able to merge them onto the branch. I'd then > like to see what the revised 4.3 M1 report looks like. I have started importing the archive kindly prepared by Michael in poky-contrib test-results repository, but I am struggling a bit regarding regression report generation with freshly imported result. I still have to confirm if it is the generated tag that is faulty or if it is a kind of an edge case in resulttool Kind regards, > Cheers, > > Richard
On 6/15/23 22:34, Alexis Lothoré wrote: > Hello Richard, Michael, > On 6/15/23 15:41, Richard Purdie wrote: >> On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via lists.yoctoproject.org wrote: >>> From: Alexis Lothoré <alexis.lothore@bootlin.com> >>> >>> There must be a more robust rework to do (because the issue will likely >>> happen on each major delivery), but I aimed for the quick and small fix to >>> quickly bring back tests results storage without breaking other things in >>> the process >> >> Thanks, I've merged this as it is a good first set of steps. >> >> As I mentioned, I think we should hardcode poky + "not ending with - >> next" as the test, then we shouldn't run into this issue again. > > ACK, will do the fix >> >> I'd also like to retroactively push the test results for 4.2 since we >> have them and should be able to merge them onto the branch. I'd then >> like to see what the revised 4.3 M1 report looks like. > > I have started importing the archive kindly prepared by Michael in poky-contrib > test-results repository, but I am struggling a bit regarding regression report > generation with freshly imported result. I still have to confirm if it is the > generated tag that is faulty or if it is a kind of an edge case in resulttool So, I have managed to generate the regression report locally (there's likely a tag issue for older tests stored in test-results to be circumvented in resulttool), and it is a bit disappointing. The report is 13MB large, and is filled once again with false positive likely due to non static ptest names, likely due to leaky build logs. Here's a sample ptestresult.gcc-g++-user.c-c++-common/Wbidi-chars-ranges.c -std=gnu++14 expected multiline pattern lines 13-17 was found: "\s*/\*<U\+202E> \} <U\+2066>if \(isAdmin\)<U\+2069> <U\+2066> begin admins only \*/[^\n\r]*\n ~~~~~~~~ ~~~~~~~~ \^\n \| \| \|[^\n\r]*\n \| \| end of bidirectional context[^\n\r]*\n U\+202E \(RIGHT-TO-LEFT OVERRIDE\) U\+2066 \(LEFT-TO-RIGHT ISOLATE\)[^\n\r]*\n": PASS -> None ptestresult.gcc-g++-user.c-c++-common/Wbidi-chars-ranges.c -std=gnu++14 expected multiline pattern lines 26-31 was found: " /\* end admins only <U\+202E> \{ <U\+2066>\*/[^\n\r]*\n ~~~~~~~~ ~~~~~~~~ \^\n \| \| \|[^\n\r]*\n \| \| end of bidirectional context[^\n\r]*\n \| U\+2066 \(LEFT-TO-RIGHT ISOLATE\)[^\n\r]*\n U\+202E \(RIGHT-TO-LEFT OVERRIDE\)[^\n\r]*\n": PASS -> None Most of this noise is about gcc ptests, there is also a bit about python3 and ltp. I manually trimmed gcc false positive to reach a reasonable size, here it is: https://pastebin.com/rYZ3qYMK > > Kind regards, > >> Cheers, >> >> Richard > >
On Fri, 2023-06-16 at 16:58 +0200, Alexis Lothoré wrote: > On 6/15/23 22:34, Alexis Lothoré wrote: > > Hello Richard, Michael, > > On 6/15/23 15:41, Richard Purdie wrote: > > > On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via lists.yoctoproject.org wrote: > > > > From: Alexis Lothoré <alexis.lothore@bootlin.com> > > > > > > > > There must be a more robust rework to do (because the issue will likely > > > > happen on each major delivery), but I aimed for the quick and small fix to > > > > quickly bring back tests results storage without breaking other things in > > > > the process > > > > > > Thanks, I've merged this as it is a good first set of steps. > > > > > > As I mentioned, I think we should hardcode poky + "not ending with - > > > next" as the test, then we shouldn't run into this issue again. > > > > ACK, will do the fix > > > > > > I'd also like to retroactively push the test results for 4.2 since we > > > have them and should be able to merge them onto the branch. I'd then > > > like to see what the revised 4.3 M1 report looks like. > > > > I have started importing the archive kindly prepared by Michael in poky-contrib > > test-results repository, but I am struggling a bit regarding regression report > > generation with freshly imported result. I still have to confirm if it is the > > generated tag that is faulty or if it is a kind of an edge case in resulttool > > So, I have managed to generate the regression report locally (there's likely a > tag issue for older tests stored in test-results to be circumvented in > resulttool), and it is a bit disappointing. The report is 13MB large, and is > filled once again with false positive likely due to non static ptest names, > likely due to leaky build logs. Here's a sample > > ptestresult.gcc-g++-user.c-c++-common/Wbidi-chars-ranges.c -std=gnu++14 > expected multiline pattern lines 13-17 was found: "\s*/\*<U\+202E> \} > <U\+2066>if \(isAdmin\)<U\+2069> <U\+2066> begin admins only \*/[^\n\r]*\n > ~~~~~~~~ ~~~~~~~~ \^\n > \| \| > \|[^\n\r]*\n \| \| > end of bidirectional context[^\n\r]*\n U\+202E \(RIGHT-TO-LEFT > OVERRIDE\) U\+2066 \(LEFT-TO-RIGHT ISOLATE\)[^\n\r]*\n": PASS -> None > ptestresult.gcc-g++-user.c-c++-common/Wbidi-chars-ranges.c -std=gnu++14 > expected multiline pattern lines 26-31 was found: " /\* end admins only > <U\+202E> \{ <U\+2066>\*/[^\n\r]*\n ~~~~~~~~ ~~~~~~~~ > \^\n \| \| \|[^\n\r]*\n > \| \| end of bidirectional context[^\n\r]*\n > \| U\+2066 \(LEFT-TO-RIGHT ISOLATE\)[^\n\r]*\n > U\+202E \(RIGHT-TO-LEFT OVERRIDE\)[^\n\r]*\n": PASS -> None > > Most of this noise is about gcc ptests, there is also a bit about python3 and > ltp. I manually trimmed gcc false positive to reach a reasonable size, here it is: > https://pastebin.com/rYZ3qYMK Thanks for getting us the diff! Going through the details there, most of it is "expected" due to changes in version of the components. I did wonder if we could somehow show that version change? I'm starting to wonder if we should: a) file two bugs for cleaning up the python3 and gcc test results b) summarise the python3 and gcc test results in the processing rather than printing in full if the differences exceed some threshold (40 changes?) Basically we need to make this report useful somehow, even if we have to exclude some data for now until we can better process it. I'm open to other ideas... Cheers, Richard
On 6/16/23 18:30, Richard Purdie wrote: > On Fri, 2023-06-16 at 16:58 +0200, Alexis Lothoré wrote: >> On 6/15/23 22:34, Alexis Lothoré wrote: >>> Hello Richard, Michael, >>> On 6/15/23 15:41, Richard Purdie wrote: >>>> On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothoré via lists.yoctoproject.org wrote: >>>>> From: Alexis Lothoré <alexis.lothore@bootlin.com> >>>>> >>>>> There must be a more robust rework to do (because the issue will likely >>>>> happen on each major delivery), but I aimed for the quick and small fix to >>>>> quickly bring back tests results storage without breaking other things in >>>>> the process >>>> >>>> Thanks, I've merged this as it is a good first set of steps. >>>> >>>> As I mentioned, I think we should hardcode poky + "not ending with - >>>> next" as the test, then we shouldn't run into this issue again. >>> >>> ACK, will do the fix >>>> >>>> I'd also like to retroactively push the test results for 4.2 since we >>>> have them and should be able to merge them onto the branch. I'd then >>>> like to see what the revised 4.3 M1 report looks like. >>> >>> I have started importing the archive kindly prepared by Michael in poky-contrib >>> test-results repository, but I am struggling a bit regarding regression report >>> generation with freshly imported result. I still have to confirm if it is the >>> generated tag that is faulty or if it is a kind of an edge case in resulttool >> >> So, I have managed to generate the regression report locally (there's likely a >> tag issue for older tests stored in test-results to be circumvented in >> resulttool), and it is a bit disappointing. The report is 13MB large, and is >> filled once again with false positive likely due to non static ptest names, >> likely due to leaky build logs. Here's a sample >> >> ptestresult.gcc-g++-user.c-c++-common/Wbidi-chars-ranges.c -std=gnu++14 >> expected multiline pattern lines 13-17 was found: "\s*/\*<U\+202E> \} >> <U\+2066>if \(isAdmin\)<U\+2069> <U\+2066> begin admins only \*/[^\n\r]*\n >> ~~~~~~~~ ~~~~~~~~ \^\n >> \| \| >> \|[^\n\r]*\n \| \| >> end of bidirectional context[^\n\r]*\n U\+202E \(RIGHT-TO-LEFT >> OVERRIDE\) U\+2066 \(LEFT-TO-RIGHT ISOLATE\)[^\n\r]*\n": PASS -> None >> ptestresult.gcc-g++-user.c-c++-common/Wbidi-chars-ranges.c -std=gnu++14 >> expected multiline pattern lines 26-31 was found: " /\* end admins only >> <U\+202E> \{ <U\+2066>\*/[^\n\r]*\n ~~~~~~~~ ~~~~~~~~ >> \^\n \| \| \|[^\n\r]*\n >> \| \| end of bidirectional context[^\n\r]*\n >> \| U\+2066 \(LEFT-TO-RIGHT ISOLATE\)[^\n\r]*\n >> U\+202E \(RIGHT-TO-LEFT OVERRIDE\)[^\n\r]*\n": PASS -> None >> >> Most of this noise is about gcc ptests, there is also a bit about python3 and >> ltp. I manually trimmed gcc false positive to reach a reasonable size, here it is: >> https://pastebin.com/rYZ3qYMK > > Thanks for getting us the diff! > > Going through the details there, most of it is "expected" due to > changes in version of the components. I did wonder if we could somehow > show that version change? > > I'm starting to wonder if we should: > > a) file two bugs for cleaning up the python3 and gcc test results > b) summarise the python3 and gcc test results in the processing rather > than printing in full if the differences exceed some threshold (40 > changes?) I would say yes and yes, and I like the idea of setting a general threshold, either an absolute one or as a percentage of total number of test cases in current test. > > Basically we need to make this report useful somehow, even if we have > to exclude some data for now until we can better process it. Absolutely. I will use this report as a base to bring a new batch of improvements. I will also add the stats I have been talking about earlier, to know for example if for a test case, the generated noise is really affecting the whole test or is a drop in the sea > > I'm open to other ideas... > > Cheers, > > Richard > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#60328): https://lists.yoctoproject.org/g/yocto/message/60328 > Mute This Topic: https://lists.yoctoproject.org/mt/99523809/7394887 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/12378809/7394887/1227806781/xyzzy [alexis.lothore@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
From: Alexis Lothoré <alexis.lothore@bootlin.com> This series is a follow-up for the 4.3_M1.rc1 regression report issue. It has been observed that the report is empty. This issue is linked to configuration description in yocto-autobuilder-helper, and has been identified through the following steps: - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3) and 4.3_M1.rc1 - yocto-4.2 results are almost empty: we only find test results from Intel QA (pushed _after_ the AB build) and not the AB test results - tests results are managed by send-qa-email.send-qa-email uses resulttool to systematically gather and store test results in local git directory - however, it looks for basebranch/comparebranch to know if those results can be pushed onto git server, and those variables depend on config.json content - yocto-4.2 (4.2.rc3) has been built on release branch mickledore (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212) - since mickledore is not yet described in config.json, send-qa-email considers it as a "work" branch (contrary to a "release" branch) and does not push test results As a consequence: - first commit brings in python logger - second commit adds a warning when such case happen, since we are able to detect it - third fix actually adds mickledore as a release branch to properly store again test results There must be a more robust rework to do (because the issue will likely happen on each major delivery), but I aimed for the quick and small fix to quickly bring back tests results storage without breaking other things in the process Alexis Lothoré (3): scripts/send-qa-email: use logger instead of raw prints scripts/send-qa-email: print warning when test results are not stored config.json: add mickledore as direct push branch for test results config.json | 2 +- scripts/send_qa_email.py | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 6 deletions(-)