| Message ID | 20230706134556.2824413-1-tgamblin@baylibre.com |
|---|---|
| State | New |
| Headers | show |
| Series | python3: disable test_cppext ptest | expand |
I don't understand. We run python ptest all the time, and it doesn't report fails, so why do we need to disable tests that aren't being reported as failing? Besides that, I believe that yes, adding toolchain bits to the image is actually fine. Alex On Thu, 6 Jul 2023 at 15:46, Trevor Gamblin <tgamblin@baylibre.com> wrote: > > This ptest requires gcc, g++, and binutils to pass. This is a lot to add > for one test, so disable it instead. > > Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> > --- > ...-test_cppext-omit-from-python3-ptest.patch | 37 +++++++++++++++++++ > .../recipes-devtools/python/python3_3.11.4.bb | 1 + > 2 files changed, 38 insertions(+) > create mode 100644 meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch > > diff --git a/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch > new file mode 100644 > index 0000000000..9fca815368 > --- /dev/null > +++ b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch > @@ -0,0 +1,37 @@ > +From 11f77795efa9aa15b0459006a16e3bbc72ec69b9 Mon Sep 17 00:00:00 2001 > +From: Trevor Gamblin <tgamblin@baylibre.com> > +Date: Wed, 5 Jul 2023 14:29:33 -0400 > +Subject: [PATCH] test_cppext: omit from python3 ptest > + > +These tests fail when running python3 ptests because gcc, g++, and > +binutils aren't present. This is a lot to add to the RDEPENDS, so > +disable the tests instead. > + > +[YOCTO # 13298] > + > +Upstream-Status: Inappropriate (OE-Specific) > + > +Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> > +--- > + Lib/test/test_cppext.py | 2 ++ > + 1 file changed, 2 insertions(+) > + > +diff --git a/Lib/test/test_cppext.py b/Lib/test/test_cppext.py > +index 465894d24e..93b71d5100 100644 > +--- a/Lib/test/test_cppext.py > ++++ b/Lib/test/test_cppext.py > +@@ -17,9 +17,11 @@ > + > + @support.requires_subprocess() > + class TestCPPExt(unittest.TestCase): > ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') > + def test_build_cpp11(self): > + self.check_build(False, '_testcpp11ext') > + > ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') > + def test_build_cpp03(self): > + self.check_build(True, '_testcpp03ext') > + > +-- > +2.41.0 > + > diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb > index 6b074c48cf..6a471e706d 100644 > --- a/meta/recipes-devtools/python/python3_3.11.4.bb > +++ b/meta/recipes-devtools/python/python3_3.11.4.bb > @@ -32,6 +32,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ > file://deterministic_imports.patch \ > file://0001-Avoid-shebang-overflow-on-python-config.py.patch \ > file://0001-Update-test_sysconfig-for-posix_user-purelib.patch \ > + file://0001-test_cppext-omit-from-python3-ptest.patch \ > " > > SRC_URI:append:class-native = " \ > -- > 2.41.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#183947): https://lists.openembedded.org/g/openembedded-core/message/183947 > Mute This Topic: https://lists.openembedded.org/mt/99985618/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 2023-07-06 09:50, Alexander Kanavin wrote: > I don't understand. We run python ptest all the time, and it doesn't > report fails, so why do we need to disable tests that aren't being > reported as failing? This is what I see when running the ptests with core-image-ptest-python3 (I've already got another patch in to fix the cgitb and zipapp failures): ---------------------------------------------------------------------- Ran 210 tests in 0.382s OK (skipped=26) == Tests result: FAILURE == 403 tests OK. 3 tests failed: test_cgitb test_cppext test_zipapp 28 tests skipped: test_asdl_parser test_check_c_globals test_clinic test_curses test_devpoll test_gdb test_idle test_kqueue test_launcher test_msilib test_ossaudiodev test_readline test_smtpnet test_socketserver test_startfile test_tcl test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle test_urllib2net test_urllibnet test_winconsoleio test_winreg test_winsound test_xmlrpc_net test_zipfile64 Total duration: 23 min 1 sec Tests result: FAILURE > > Besides that, I believe that yes, adding toolchain bits to the image > is actually fine. It could be. I discussed this with RP and thought he didn't want to be adding those dependencies for python3 ptests, but I may have misunderstood, so maybe when he's back we can discuss it. The core-image-sato-sdk image doesn't have this problem. Trevor > > Alex > > On Thu, 6 Jul 2023 at 15:46, Trevor Gamblin <tgamblin@baylibre.com> wrote: >> This ptest requires gcc, g++, and binutils to pass. This is a lot to add >> for one test, so disable it instead. >> >> Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> >> --- >> ...-test_cppext-omit-from-python3-ptest.patch | 37 +++++++++++++++++++ >> .../recipes-devtools/python/python3_3.11.4.bb | 1 + >> 2 files changed, 38 insertions(+) >> create mode 100644 meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch >> >> diff --git a/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch >> new file mode 100644 >> index 0000000000..9fca815368 >> --- /dev/null >> +++ b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch >> @@ -0,0 +1,37 @@ >> +From 11f77795efa9aa15b0459006a16e3bbc72ec69b9 Mon Sep 17 00:00:00 2001 >> +From: Trevor Gamblin <tgamblin@baylibre.com> >> +Date: Wed, 5 Jul 2023 14:29:33 -0400 >> +Subject: [PATCH] test_cppext: omit from python3 ptest >> + >> +These tests fail when running python3 ptests because gcc, g++, and >> +binutils aren't present. This is a lot to add to the RDEPENDS, so >> +disable the tests instead. >> + >> +[YOCTO # 13298] >> + >> +Upstream-Status: Inappropriate (OE-Specific) >> + >> +Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> >> +--- >> + Lib/test/test_cppext.py | 2 ++ >> + 1 file changed, 2 insertions(+) >> + >> +diff --git a/Lib/test/test_cppext.py b/Lib/test/test_cppext.py >> +index 465894d24e..93b71d5100 100644 >> +--- a/Lib/test/test_cppext.py >> ++++ b/Lib/test/test_cppext.py >> +@@ -17,9 +17,11 @@ >> + >> + @support.requires_subprocess() >> + class TestCPPExt(unittest.TestCase): >> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') >> + def test_build_cpp11(self): >> + self.check_build(False, '_testcpp11ext') >> + >> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') >> + def test_build_cpp03(self): >> + self.check_build(True, '_testcpp03ext') >> + >> +-- >> +2.41.0 >> + >> diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb >> index 6b074c48cf..6a471e706d 100644 >> --- a/meta/recipes-devtools/python/python3_3.11.4.bb >> +++ b/meta/recipes-devtools/python/python3_3.11.4.bb >> @@ -32,6 +32,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ >> file://deterministic_imports.patch \ >> file://0001-Avoid-shebang-overflow-on-python-config.py.patch \ >> file://0001-Update-test_sysconfig-for-posix_user-purelib.patch \ >> + file://0001-test_cppext-omit-from-python3-ptest.patch \ >> " >> >> SRC_URI:append:class-native = " \ >> -- >> 2.41.0 >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#183947): https://lists.openembedded.org/g/openembedded-core/message/183947 >> Mute This Topic: https://lists.openembedded.org/mt/99985618/1686489 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >>
The issue here seems to be that python ptest failures are not reported as such? Does '-c testimage' succeed for you, and do the logs for it then still contain the fails? That lack of fail reporting must be addressed first, then we can look into the fails themselves. We have plenty of ptests which do pull in the target toolchain (that's why you always see it building even for images that aren't going to have it), so I dont think it would be a problem for python. Alex On Thu, 6 Jul 2023 at 16:30, Trevor Gamblin <tgamblin@baylibre.com> wrote: > > > On 2023-07-06 09:50, Alexander Kanavin wrote: > > I don't understand. We run python ptest all the time, and it doesn't > > report fails, so why do we need to disable tests that aren't being > > reported as failing? > > This is what I see when running the ptests with core-image-ptest-python3 > (I've already got another patch in to fix the cgitb and zipapp failures): > > ---------------------------------------------------------------------- > Ran 210 tests in 0.382s > > OK (skipped=26) > > == Tests result: FAILURE == > > 403 tests OK. > > 3 tests failed: > test_cgitb test_cppext test_zipapp > > 28 tests skipped: > test_asdl_parser test_check_c_globals test_clinic test_curses > test_devpoll test_gdb test_idle test_kqueue test_launcher > test_msilib test_ossaudiodev test_readline test_smtpnet > test_socketserver test_startfile test_tcl test_tix test_tk > test_ttk_guionly test_ttk_textonly test_turtle test_urllib2net > test_urllibnet test_winconsoleio test_winreg test_winsound > test_xmlrpc_net test_zipfile64 > > Total duration: 23 min 1 sec > Tests result: FAILURE > > > > > Besides that, I believe that yes, adding toolchain bits to the image > > is actually fine. > > It could be. I discussed this with RP and thought he didn't want to be > adding those dependencies for python3 ptests, but I may have > misunderstood, so maybe when he's back we can discuss it. The > core-image-sato-sdk image doesn't have this problem. > > Trevor > > > > > Alex > > > > On Thu, 6 Jul 2023 at 15:46, Trevor Gamblin <tgamblin@baylibre.com> wrote: > >> This ptest requires gcc, g++, and binutils to pass. This is a lot to add > >> for one test, so disable it instead. > >> > >> Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> > >> --- > >> ...-test_cppext-omit-from-python3-ptest.patch | 37 +++++++++++++++++++ > >> .../recipes-devtools/python/python3_3.11.4.bb | 1 + > >> 2 files changed, 38 insertions(+) > >> create mode 100644 meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch > >> > >> diff --git a/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch > >> new file mode 100644 > >> index 0000000000..9fca815368 > >> --- /dev/null > >> +++ b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch > >> @@ -0,0 +1,37 @@ > >> +From 11f77795efa9aa15b0459006a16e3bbc72ec69b9 Mon Sep 17 00:00:00 2001 > >> +From: Trevor Gamblin <tgamblin@baylibre.com> > >> +Date: Wed, 5 Jul 2023 14:29:33 -0400 > >> +Subject: [PATCH] test_cppext: omit from python3 ptest > >> + > >> +These tests fail when running python3 ptests because gcc, g++, and > >> +binutils aren't present. This is a lot to add to the RDEPENDS, so > >> +disable the tests instead. > >> + > >> +[YOCTO # 13298] > >> + > >> +Upstream-Status: Inappropriate (OE-Specific) > >> + > >> +Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> > >> +--- > >> + Lib/test/test_cppext.py | 2 ++ > >> + 1 file changed, 2 insertions(+) > >> + > >> +diff --git a/Lib/test/test_cppext.py b/Lib/test/test_cppext.py > >> +index 465894d24e..93b71d5100 100644 > >> +--- a/Lib/test/test_cppext.py > >> ++++ b/Lib/test/test_cppext.py > >> +@@ -17,9 +17,11 @@ > >> + > >> + @support.requires_subprocess() > >> + class TestCPPExt(unittest.TestCase): > >> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') > >> + def test_build_cpp11(self): > >> + self.check_build(False, '_testcpp11ext') > >> + > >> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') > >> + def test_build_cpp03(self): > >> + self.check_build(True, '_testcpp03ext') > >> + > >> +-- > >> +2.41.0 > >> + > >> diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb > >> index 6b074c48cf..6a471e706d 100644 > >> --- a/meta/recipes-devtools/python/python3_3.11.4.bb > >> +++ b/meta/recipes-devtools/python/python3_3.11.4.bb > >> @@ -32,6 +32,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ > >> file://deterministic_imports.patch \ > >> file://0001-Avoid-shebang-overflow-on-python-config.py.patch \ > >> file://0001-Update-test_sysconfig-for-posix_user-purelib.patch \ > >> + file://0001-test_cppext-omit-from-python3-ptest.patch \ > >> " > >> > >> SRC_URI:append:class-native = " \ > >> -- > >> 2.41.0 > >> > >> > >> -=-=-=-=-=-=-=-=-=-=-=- > >> Links: You receive all messages sent to this group. > >> View/Reply Online (#183947): https://lists.openembedded.org/g/openembedded-core/message/183947 > >> Mute This Topic: https://lists.openembedded.org/mt/99985618/1686489 > >> Group Owner: openembedded-core+owner@lists.openembedded.org > >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > >> -=-=-=-=-=-=-=-=-=-=-=- > >>
On 6 Jul 2023, at 14:45, Trevor Gamblin via lists.openembedded.org <tgamblin=baylibre.com@lists.openembedded.org> wrote: > + class TestCPPExt(unittest.TestCase): > ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') > + def test_build_cpp11(self): > + self.check_build(False, '_testcpp11ext') > + > ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') > + def test_build_cpp03(self): > + self.check_build(True, '_testcpp03ext’) So torn. First, it’s probably better to do the excludes in the run-ptest script instead, python3 -mtest takes an —exclude argument so you can filter out tests there. Also, it seems that the test should be skipped if the tools it needs are not present, and that’s potentially upstreamable. As Alex says, we’ve several other ptests that pull the toolchain in, so I’m not against adding the dependencies. Ross
On 2023-07-06 10:41, Alexander Kanavin wrote: > The issue here seems to be that python ptest failures are not reported > as such? Does '-c testimage' succeed for you, and do the logs for it > then still contain the fails? That lack of fail reporting must be > addressed first, then we can look into the fails themselves. Until recently yes, but it seems that a different issue has crept in that is preventing me from seeing the logs in the same way. I'm looking into it. Interestingly, running ptest-runner manually used to break on a broken pipe for me before, but now it's not happening... > > We have plenty of ptests which do pull in the target toolchain (that's > why you always see it building even for images that aren't going to > have it), so I dont think it would be a problem for python. > > Alex > > On Thu, 6 Jul 2023 at 16:30, Trevor Gamblin <tgamblin@baylibre.com> wrote: >> >> On 2023-07-06 09:50, Alexander Kanavin wrote: >>> I don't understand. We run python ptest all the time, and it doesn't >>> report fails, so why do we need to disable tests that aren't being >>> reported as failing? >> This is what I see when running the ptests with core-image-ptest-python3 >> (I've already got another patch in to fix the cgitb and zipapp failures): >> >> ---------------------------------------------------------------------- >> Ran 210 tests in 0.382s >> >> OK (skipped=26) >> >> == Tests result: FAILURE == >> >> 403 tests OK. >> >> 3 tests failed: >> test_cgitb test_cppext test_zipapp >> >> 28 tests skipped: >> test_asdl_parser test_check_c_globals test_clinic test_curses >> test_devpoll test_gdb test_idle test_kqueue test_launcher >> test_msilib test_ossaudiodev test_readline test_smtpnet >> test_socketserver test_startfile test_tcl test_tix test_tk >> test_ttk_guionly test_ttk_textonly test_turtle test_urllib2net >> test_urllibnet test_winconsoleio test_winreg test_winsound >> test_xmlrpc_net test_zipfile64 >> >> Total duration: 23 min 1 sec >> Tests result: FAILURE >> >>> Besides that, I believe that yes, adding toolchain bits to the image >>> is actually fine. >> It could be. I discussed this with RP and thought he didn't want to be >> adding those dependencies for python3 ptests, but I may have >> misunderstood, so maybe when he's back we can discuss it. The >> core-image-sato-sdk image doesn't have this problem. >> >> Trevor >> >>> Alex >>> >>> On Thu, 6 Jul 2023 at 15:46, Trevor Gamblin <tgamblin@baylibre.com> wrote: >>>> This ptest requires gcc, g++, and binutils to pass. This is a lot to add >>>> for one test, so disable it instead. >>>> >>>> Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> >>>> --- >>>> ...-test_cppext-omit-from-python3-ptest.patch | 37 +++++++++++++++++++ >>>> .../recipes-devtools/python/python3_3.11.4.bb | 1 + >>>> 2 files changed, 38 insertions(+) >>>> create mode 100644 meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch >>>> >>>> diff --git a/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch >>>> new file mode 100644 >>>> index 0000000000..9fca815368 >>>> --- /dev/null >>>> +++ b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch >>>> @@ -0,0 +1,37 @@ >>>> +From 11f77795efa9aa15b0459006a16e3bbc72ec69b9 Mon Sep 17 00:00:00 2001 >>>> +From: Trevor Gamblin <tgamblin@baylibre.com> >>>> +Date: Wed, 5 Jul 2023 14:29:33 -0400 >>>> +Subject: [PATCH] test_cppext: omit from python3 ptest >>>> + >>>> +These tests fail when running python3 ptests because gcc, g++, and >>>> +binutils aren't present. This is a lot to add to the RDEPENDS, so >>>> +disable the tests instead. >>>> + >>>> +[YOCTO # 13298] >>>> + >>>> +Upstream-Status: Inappropriate (OE-Specific) >>>> + >>>> +Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> >>>> +--- >>>> + Lib/test/test_cppext.py | 2 ++ >>>> + 1 file changed, 2 insertions(+) >>>> + >>>> +diff --git a/Lib/test/test_cppext.py b/Lib/test/test_cppext.py >>>> +index 465894d24e..93b71d5100 100644 >>>> +--- a/Lib/test/test_cppext.py >>>> ++++ b/Lib/test/test_cppext.py >>>> +@@ -17,9 +17,11 @@ >>>> + >>>> + @support.requires_subprocess() >>>> + class TestCPPExt(unittest.TestCase): >>>> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') >>>> + def test_build_cpp11(self): >>>> + self.check_build(False, '_testcpp11ext') >>>> + >>>> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') >>>> + def test_build_cpp03(self): >>>> + self.check_build(True, '_testcpp03ext') >>>> + >>>> +-- >>>> +2.41.0 >>>> + >>>> diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb >>>> index 6b074c48cf..6a471e706d 100644 >>>> --- a/meta/recipes-devtools/python/python3_3.11.4.bb >>>> +++ b/meta/recipes-devtools/python/python3_3.11.4.bb >>>> @@ -32,6 +32,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ >>>> file://deterministic_imports.patch \ >>>> file://0001-Avoid-shebang-overflow-on-python-config.py.patch \ >>>> file://0001-Update-test_sysconfig-for-posix_user-purelib.patch \ >>>> + file://0001-test_cppext-omit-from-python3-ptest.patch \ >>>> " >>>> >>>> SRC_URI:append:class-native = " \ >>>> -- >>>> 2.41.0 >>>> >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> Links: You receive all messages sent to this group. >>>> View/Reply Online (#183947): https://lists.openembedded.org/g/openembedded-core/message/183947 >>>> Mute This Topic: https://lists.openembedded.org/mt/99985618/1686489 >>>> Group Owner: openembedded-core+owner@lists.openembedded.org >>>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>
On 2023-07-06 10:50, Ross Burton wrote: > On 6 Jul 2023, at 14:45, Trevor Gamblin via lists.openembedded.org <tgamblin=baylibre.com@lists.openembedded.org> wrote: >> + class TestCPPExt(unittest.TestCase): >> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') >> + def test_build_cpp11(self): >> + self.check_build(False, '_testcpp11ext') >> + >> ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') >> + def test_build_cpp03(self): >> + self.check_build(True, '_testcpp03ext’) > So torn. > > First, it’s probably better to do the excludes in the run-ptest script instead, python3 -mtest takes an —exclude argument so you can filter out tests there. > > Also, it seems that the test should be skipped if the tools it needs are not present, and that’s potentially upstreamable. > > As Alex says, we’ve several other ptests that pull the toolchain in, so I’m not against adding the dependencies. Alright. I'll look at the other options for skipping the tests for now. I'll wait before submitting again just to be sure I wasn't misunderstanding the original conversation. > > Ross >
On Thu, 6 Jul 2023 at 19:22, Trevor Gamblin <tgamblin@baylibre.com> wrote: > Until recently yes, but it seems that a different issue has crept in > that is preventing me from seeing the logs in the same way. I'm looking > into it. Interestingly, running ptest-runner manually used to break on a > broken pipe for me before, but now it's not happening... I took a look. The culprit is what run-ptest does: SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v | sed -u -e '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g' This has two problems: 1. Processing output with sed tends to quietly regress (not to mention it's difficult to read), and we should discourage use of sed across yocto stronger than we currently do. Particularly here the tests fail in a way that the sed expression isn't catching (ERROR without leading dots I think). 2. This would've been okay if the overall command would exit with a non-zero, but alas it doesn't - shell pipes obscure any problems in the pipe except in the very last command. I think we need to eliminate the use of sed first, but I'm not yet sure how. I'll try to look into whether -m test can be convinced to output the results in a compatible way, or write them into a structured file that can be processed to report failures. Alex
On 7 Jul 2023, at 11:20, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > On Thu, 6 Jul 2023 at 19:22, Trevor Gamblin <tgamblin@baylibre.com> wrote: >> Until recently yes, but it seems that a different issue has crept in >> that is preventing me from seeing the logs in the same way. I'm looking >> into it. Interestingly, running ptest-runner manually used to break on a >> broken pipe for me before, but now it's not happening... > > I took a look. The culprit is what run-ptest does: > > SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v | sed -u -e > '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' > -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. > ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g' > > This has two problems: > > 1. Processing output with sed tends to quietly regress (not to mention > it's difficult to read), and we should discourage use of sed across > yocto stronger than we currently do. Particularly here the tests fail > in a way that the sed expression isn't catching (ERROR without leading > dots I think). So I started on this, and there’s a unittest runner in python3-unittest-automake-output which has our output format. The problem with python’s test module is that it also has its own runner, so the trick would be interfacing the two. Ross
On Fri, 7 Jul 2023 at 12:22, Ross Burton <Ross.Burton@arm.com> wrote:
> So I started on this, and there’s a unittest runner in python3-unittest-automake-output which has our output format. The problem with python’s test module is that it also has its own runner, so the trick would be interfacing the two.
Until that is ready, I think we should simply use '-m test -W' with no
post-processing. It would stay quiet for the succeeding tests (which
is 99.9% of them), but output everything for the failing ones. As long
as that output is captured and reported as an 'overall fail', we can
do without per-test split in the qa reports. I'm testing a patch.
Alex
diff --git a/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch new file mode 100644 index 0000000000..9fca815368 --- /dev/null +++ b/meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch @@ -0,0 +1,37 @@ +From 11f77795efa9aa15b0459006a16e3bbc72ec69b9 Mon Sep 17 00:00:00 2001 +From: Trevor Gamblin <tgamblin@baylibre.com> +Date: Wed, 5 Jul 2023 14:29:33 -0400 +Subject: [PATCH] test_cppext: omit from python3 ptest + +These tests fail when running python3 ptests because gcc, g++, and +binutils aren't present. This is a lot to add to the RDEPENDS, so +disable the tests instead. + +[YOCTO # 13298] + +Upstream-Status: Inappropriate (OE-Specific) + +Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> +--- + Lib/test/test_cppext.py | 2 ++ + 1 file changed, 2 insertions(+) + +diff --git a/Lib/test/test_cppext.py b/Lib/test/test_cppext.py +index 465894d24e..93b71d5100 100644 +--- a/Lib/test/test_cppext.py ++++ b/Lib/test/test_cppext.py +@@ -17,9 +17,11 @@ + + @support.requires_subprocess() + class TestCPPExt(unittest.TestCase): ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') + def test_build_cpp11(self): + self.check_build(False, '_testcpp11ext') + ++ @unittest.skip('requires gcc, g++ - excessive for ptest build') + def test_build_cpp03(self): + self.check_build(True, '_testcpp03ext') + +-- +2.41.0 + diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb index 6b074c48cf..6a471e706d 100644 --- a/meta/recipes-devtools/python/python3_3.11.4.bb +++ b/meta/recipes-devtools/python/python3_3.11.4.bb @@ -32,6 +32,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ file://deterministic_imports.patch \ file://0001-Avoid-shebang-overflow-on-python-config.py.patch \ file://0001-Update-test_sysconfig-for-posix_user-purelib.patch \ + file://0001-test_cppext-omit-from-python3-ptest.patch \ " SRC_URI:append:class-native = " \
This ptest requires gcc, g++, and binutils to pass. This is a lot to add for one test, so disable it instead. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> --- ...-test_cppext-omit-from-python3-ptest.patch | 37 +++++++++++++++++++ .../recipes-devtools/python/python3_3.11.4.bb | 1 + 2 files changed, 38 insertions(+) create mode 100644 meta/recipes-devtools/python/python3/0001-test_cppext-omit-from-python3-ptest.patch