| Message ID | 20231030195037.232912-8-alassane.yattara@savoirfairelinux.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/8] Toaster: Update toaster-requirements.txt to add pytest and some plugin's | expand |
On Mon, Oct 30, 2023 at 12:51 PM Alassane Yattara < alassane.yattara@savoirfairelinux.com> wrote: > Added pytest init file to configure pytest and generate test reports. > Typo: pytest.ini file? > > Objective > - We propose to integrate the plugin Pytest to generate test reports for > Toaster application. > - To keep configurations to a minimal, we propose to integrate Pytest with > Tox(tox.ini). > - Tox will automate the execution of the Pytest test cases. > > Pros > - Generate test reports > - Create a historical record of test results over time > - Track Toasters stability and quality > - Documentation and Transparency > - Debugging, troubleshooting : identify regressions (input, output,) > > Cons > - Limited predefined report options. Although Pytest provides different > report > formats, its default reports are rather straightforward, We might have to > put extra effort into crafting them ourselves. > One of the benefits of pytest is in fact that report plugins are available and fairly straightforward to implement if we need our own custom report format (although we should avoid custom plugins and use upstream maintained ones wherever possible). We do need to ensure that plain text output is generated for consumption by the AutoBuilder. Fancy html output is going to clutter the testresults.json that will be where these test results will ultimately be stored: https://git.yoctoproject.org/poky/plain/scripts/resulttool https://git.yoctoproject.org/yocto-testresults/ https://docs.yoctoproject.org/test-manual/understand-autobuilder.html?highlight=resulttool#resulttool The reason is that the testresults.json is consumed by tooling to look for regressions. It must be machine readable and easily processed. > Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com> > --- > lib/toaster/pytest.ini | 19 +++++++++++++++++++ > 1 file changed, 19 insertions(+) > create mode 100644 lib/toaster/pytest.ini > > diff --git a/lib/toaster/pytest.ini b/lib/toaster/pytest.ini > new file mode 100644 > index 00000000..f07076b7 > --- /dev/null > +++ b/lib/toaster/pytest.ini > @@ -0,0 +1,19 @@ > +# -- FILE: pytest.ini (or tox.ini) > +[pytest] > +DJANGO_SETTINGS_MODULE = toastermain.settings_test > + > +python_files = db/test_*.py commands/test_*.py views/test_*.py > browser/test_*.py functional/test_*.py > + > +# --create-db - force re creation of the test database > +# > https://pytest-django.readthedocs.io/en/latest/database.html#create-db-force-re-creation-of-the-test-database > + > +# --html=report.html --self-contained-html > +# https://docs.pytest.org/en/latest/usage.html#creating-html-reports > +# > https://pytest-html.readthedocs.io/en/latest/user_guide.html#creating-a-self-contained-report > +addopts = --create-db --html="Toaster Tests Report.html" > --self-contained-html > + > +# Define environment variables using pytest-env > +# A pytest plugin that enables you to set environment variables in the > pytest.ini file. > +# https://pypi.org/project/pytest-env/ > +env = > + TOASTER_BUILDSERVER=1 > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#15332): > https://lists.openembedded.org/g/bitbake-devel/message/15332 > Mute This Topic: https://lists.openembedded.org/mt/102283466/924729 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ > ticotimo@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Tue, Oct 31, 2023 at 8:16 AM Tim Orling via lists.yoctoproject.org <ticotimo=gmail.com@lists.yoctoproject.org> wrote: > > > On Mon, Oct 30, 2023 at 12:51 PM Alassane Yattara < > alassane.yattara@savoirfairelinux.com> wrote: > >> Added pytest init file to configure pytest and generate test reports. >> > > Typo: pytest.ini file? > > >> >> Objective >> - We propose to integrate the plugin Pytest to generate test reports for >> Toaster application. >> - To keep configurations to a minimal, we propose to integrate Pytest >> with Tox(tox.ini). >> - Tox will automate the execution of the Pytest test cases. >> >> Pros >> - Generate test reports >> - Create a historical record of test results over time >> - Track Toasters stability and quality >> - Documentation and Transparency >> - Debugging, troubleshooting : identify regressions (input, output,) >> >> Cons >> - Limited predefined report options. Although Pytest provides different >> report >> formats, its default reports are rather straightforward, We might have to >> put extra effort into crafting them ourselves. >> > > One of the benefits of pytest is in fact that report plugins are available > and fairly straightforward to implement if we need our own custom report > format (although we should avoid custom plugins and use upstream maintained > ones wherever possible). > > We do need to ensure that plain text output is generated for consumption > by the AutoBuilder. Fancy html output is going to clutter the > testresults.json that will be where these test results will ultimately be > stored: > https://git.yoctoproject.org/poky/plain/scripts/resulttool > https://git.yoctoproject.org/yocto-testresults/ > > https://docs.yoctoproject.org/test-manual/understand-autobuilder.html?highlight=resulttool#resulttool > > The reason is that the testresults.json is consumed by tooling to look for > regressions. It must be machine readable and easily processed. > > >> Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com> >> --- >> lib/toaster/pytest.ini | 19 +++++++++++++++++++ >> 1 file changed, 19 insertions(+) >> create mode 100644 lib/toaster/pytest.ini >> >> diff --git a/lib/toaster/pytest.ini b/lib/toaster/pytest.ini >> new file mode 100644 >> index 00000000..f07076b7 >> --- /dev/null >> +++ b/lib/toaster/pytest.ini >> @@ -0,0 +1,19 @@ >> +# -- FILE: pytest.ini (or tox.ini) >> +[pytest] >> +DJANGO_SETTINGS_MODULE = toastermain.settings_test >> + >> +python_files = db/test_*.py commands/test_*.py views/test_*.py >> browser/test_*.py functional/test_*.py >> + >> +# --create-db - force re creation of the test database >> +# >> https://pytest-django.readthedocs.io/en/latest/database.html#create-db-force-re-creation-of-the-test-database >> + >> +# --html=report.html --self-contained-html >> +# https://docs.pytest.org/en/latest/usage.html#creating-html-reports >> +# >> https://pytest-html.readthedocs.io/en/latest/user_guide.html#creating-a-self-contained-report >> +addopts = --create-db --html="Toaster Tests Report.html" >> --self-contained-html > > Your patch had a trailing whitespace at the end of the above line. Please double check for that in the future. > >> + >> +# Define environment variables using pytest-env >> +# A pytest plugin that enables you to set environment variables in the >> pytest.ini file. >> +# https://pypi.org/project/pytest-env/ >> +env = >> + TOASTER_BUILDSERVER=1 >> -- >> 2.34.1 >> >> >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#5921): > https://lists.yoctoproject.org/g/toaster/message/5921 > Mute This Topic: https://lists.yoctoproject.org/mt/102299113/924729 > Group Owner: toaster+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/toaster/unsub [ > ticotimo@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Tue, 2023-10-31 at 08:16 -0700, Tim Orling wrote: > > > On Mon, Oct 30, 2023 at 12:51 PM Alassane Yattara > <alassane.yattara@savoirfairelinux.com> wrote: > > Added pytest init file to configure pytest and generate test > > reports. > > > > > Typo: pytest.ini file? > > > > > Objective > > - We propose to integrate the plugin Pytest to generate test > > reports for Toaster application. > > - To keep configurations to a minimal, we propose to integrate > > Pytest with Tox(tox.ini). > > - Tox will automate the execution of the Pytest test cases. > > > > Pros > > - Generate test reports > > - Create a historical record of test results over time > > - Track Toasters stability and quality > > - Documentation and Transparency > > - Debugging, troubleshooting : identify regressions (input, > > output,) > > > > Cons > > - Limited predefined report options. Although Pytest provides > > different report > > formats, its default reports are rather straightforward, We might > > have to put extra effort into crafting them ourselves. > > > > > One of the benefits of pytest is in fact that report plugins are > available and fairly straightforward to implement if we need our own > custom report format (although we should avoid custom plugins and use > upstream maintained ones wherever possible). > > We do need to ensure that plain text output is generated for > consumption by the AutoBuilder. Fancy html output is going to clutter > the testresults.json that will be where these test results will > ultimately be stored: > https://git.yoctoproject.org/poky/plain/scripts/resulttool > https://git.yoctoproject.org/yocto-testresults/ > https://docs.yoctoproject.org/test-manual/understand-autobuilder.html?highlight=resulttool#resulttool > > The reason is that the testresults.json is consumed by tooling to > look for regressions. It must be machine readable and easily > processed. Tim is correct that we do need text/machine readable results for tesresults.json but there is also a case for generating graphical/html reports alongside the results. An example would be our build performance tests: https://autobuilder.yocto.io/pub/non-release/20231030-22/testresults/buildperf-alma8/perf-alma8_master_20231030150044_28324600cb.html where we've saved text data but also show it more visually. Cheers, Richard
Tim, > > Typo: pytest.ini file? Non, pytest.init is a config file for pytest, next step is to merge this file with tox.init. tox.ini file are the configuration files of the tox project, and can also be used to hold pytest configuration if they have a [pytest] section. ----- Mail original ----- De: "Richard Purdie" <richard.purdie@linuxfoundation.org> À: "Tim Orling" <ticotimo@gmail.com>, "Alassane Yattara" <alassane.yattara@savoirfairelinux.com> Cc: "bitbake-devel" <bitbake-devel@lists.openembedded.org>, "toaster" <toaster@lists.yoctoproject.org> Envoyé: Mardi 31 Octobre 2023 16:36:42 Objet: Re: [Toaster] [bitbake-devel] [PATCH 8/8] Toaster: Added pytest.init file On Tue, 2023-10-31 at 08:16 -0700, Tim Orling wrote: > > > On Mon, Oct 30, 2023 at 12:51 PM Alassane Yattara > <alassane.yattara@savoirfairelinux.com> wrote: > > Added pytest init file to configure pytest and generate test > > reports. > > > > > Typo: pytest.ini file? > > > > > Objective > > - We propose to integrate the plugin Pytest to generate test > > reports for Toaster application. > > - To keep configurations to a minimal, we propose to integrate > > Pytest with Tox(tox.ini). > > - Tox will automate the execution of the Pytest test cases. > > > > Pros > > - Generate test reports > > - Create a historical record of test results over time > > - Track Toasters stability and quality > > - Documentation and Transparency > > - Debugging, troubleshooting : identify regressions (input, > > output,) > > > > Cons > > - Limited predefined report options. Although Pytest provides > > different report > > formats, its default reports are rather straightforward, We might > > have to put extra effort into crafting them ourselves. > > > > > One of the benefits of pytest is in fact that report plugins are > available and fairly straightforward to implement if we need our own > custom report format (although we should avoid custom plugins and use > upstream maintained ones wherever possible). > > We do need to ensure that plain text output is generated for > consumption by the AutoBuilder. Fancy html output is going to clutter > the testresults.json that will be where these test results will > ultimately be stored: > https://git.yoctoproject.org/poky/plain/scripts/resulttool > https://git.yoctoproject.org/yocto-testresults/ > https://docs.yoctoproject.org/test-manual/understand-autobuilder.html?highlight=resulttool#resulttool > > The reason is that the testresults.json is consumed by tooling to > look for regressions. It must be machine readable and easily > processed. Tim is correct that we do need text/machine readable results for tesresults.json but there is also a case for generating graphical/html reports alongside the results. An example would be our build performance tests: https://autobuilder.yocto.io/pub/non-release/20231030-22/testresults/buildperf-alma8/perf-alma8_master_20231030150044_28324600cb.html where we've saved text data but also show it more visually. Cheers, Richard
On Tue, 2023-10-31 at 13:43 -0400, Alassane Yattara wrote: > Tim, > > > > Typo: pytest.ini file? > Non, pytest.init is a config file for pytest, next step is to merge this file with tox.init. > tox.ini file are the configuration files of the tox project, and can also be used to hold pytest configuration if they have a [pytest] section. The subject line shortlog says "pytest.init" but the patch adds a "pytest.ini" file. The tox settings file is called "tox.ini", not "tox.init" too. We need to get the names right... Cheers, Richard
diff --git a/lib/toaster/pytest.ini b/lib/toaster/pytest.ini new file mode 100644 index 00000000..f07076b7 --- /dev/null +++ b/lib/toaster/pytest.ini @@ -0,0 +1,19 @@ +# -- FILE: pytest.ini (or tox.ini) +[pytest] +DJANGO_SETTINGS_MODULE = toastermain.settings_test + +python_files = db/test_*.py commands/test_*.py views/test_*.py browser/test_*.py functional/test_*.py + +# --create-db - force re creation of the test database +# https://pytest-django.readthedocs.io/en/latest/database.html#create-db-force-re-creation-of-the-test-database + +# --html=report.html --self-contained-html +# https://docs.pytest.org/en/latest/usage.html#creating-html-reports +# https://pytest-html.readthedocs.io/en/latest/user_guide.html#creating-a-self-contained-report +addopts = --create-db --html="Toaster Tests Report.html" --self-contained-html + +# Define environment variables using pytest-env +# A pytest plugin that enables you to set environment variables in the pytest.ini file. +# https://pypi.org/project/pytest-env/ +env = + TOASTER_BUILDSERVER=1
Added pytest init file to configure pytest and generate test reports. Objective - We propose to integrate the plugin Pytest to generate test reports for Toaster application. - To keep configurations to a minimal, we propose to integrate Pytest with Tox(tox.ini). - Tox will automate the execution of the Pytest test cases. Pros - Generate test reports - Create a historical record of test results over time - Track Toasters stability and quality - Documentation and Transparency - Debugging, troubleshooting : identify regressions (input, output,) Cons - Limited predefined report options. Although Pytest provides different report formats, its default reports are rather straightforward, We might have to put extra effort into crafting them ourselves. Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com> --- lib/toaster/pytest.ini | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 lib/toaster/pytest.ini