Patchwork [0/4] Script and layer for running tests

login
register
mail settings
Submitter Stanacar, StefanX
Date Nov. 27, 2013, 5:08 p.m.
Message ID <cover.1385571298.git.stefanx.stanacar@intel.com>
Download mbox
Permalink /patch/62487/
State New
Headers show

Pull-request

git://git.yoctoproject.org/poky-contrib stefans/selftest

Comments

Stanacar, StefanX - Nov. 27, 2013, 5:08 p.m.
Hello,

This series adds an oe-selftest script, some modules and a new layer meta-selftest.
which are meant to help in writing tests (using python unittest) for various
bitbake tools/scripts as well as simple output checks or do
complete builds with different options (with the emphasis that everything checked
is done outside of bitbake context, just like a human would do.) For more details,
please see YOCTO #4740.

Cheers,
Stefan

Changes since the RFC:
 - more tests added
 - tests are now in meta/lib/oeqa/selftest instead of scripts/lib/selftest/tests
 - small changes to some of the files in the meta-selftest layer
 - more cleanups done by default in the base class


The following changes since commit 32adaac34a614d106e6dd3e9f1130f4e94ff39ae:

  libpng: set reasonable SUMMARY (2013-11-27 11:51:25 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib stefans/selftest
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/selftest

Alexandru Palalau (1):
  lib/oeqa/selftest: buildoptions.py: add simple image build tests

Corneliu Stoicescu (2):
  meta-selftest: create a new test layer to be used by oe-selftest
    script
  lib/oeqa/selftest: add test modules for expected bitbake output and
    bitbake-layers

Stefan Stanacar (1):
  scripts/oe-selftest: script to run builds as unittest against bitbake
    or various scripts

 .gitignore                                         |   1 +
 meta-selftest/COPYING.MIT                          |  17 +++
 meta-selftest/README                               |   3 +
 meta-selftest/classes/test_events.bbclass          |  16 +++
 meta-selftest/conf/layer.conf                      |  10 ++
 .../recipes-test/aspell/aspell_0.0.0.1.bb          |  28 ++++
 .../recipes-test/aspell/aspell_0.60.6.1.bbappend   |   2 +
 meta-selftest/recipes-test/m4/m4_1.4.17.bbappend   |   2 +
 .../recipes-test/man/man/man-1.5h1-make.patch      |  16 +++
 meta-selftest/recipes-test/man/man_1.6g.bbappend   |   2 +
 .../xcursor-transparent-theme_0.1.1.bbappend       |   2 +
 meta/lib/oeqa/selftest/__init__.py                 |   2 +
 meta/lib/oeqa/selftest/base.py                     |  98 ++++++++++++++
 meta/lib/oeqa/selftest/bblayers.py                 |  37 ++++++
 meta/lib/oeqa/selftest/bbtests.py                  |  97 ++++++++++++++
 meta/lib/oeqa/selftest/buildoptions.py             |  86 ++++++++++++
 meta/lib/oeqa/utils/commands.py                    | 137 +++++++++++++++++++
 meta/lib/oeqa/utils/ftools.py                      |  27 ++++
 scripts/oe-selftest                                | 148 +++++++++++++++++++++
 19 files changed, 731 insertions(+)
 create mode 100644 meta-selftest/COPYING.MIT
 create mode 100644 meta-selftest/README
 create mode 100644 meta-selftest/classes/test_events.bbclass
 create mode 100644 meta-selftest/conf/layer.conf
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.60.6.1.bbappend
 create mode 100644 meta-selftest/recipes-test/m4/m4_1.4.17.bbappend
 create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
 create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend
 create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
 create mode 100644 meta/lib/oeqa/selftest/__init__.py
 create mode 100644 meta/lib/oeqa/selftest/base.py
 create mode 100644 meta/lib/oeqa/selftest/bblayers.py
 create mode 100644 meta/lib/oeqa/selftest/bbtests.py
 create mode 100644 meta/lib/oeqa/selftest/buildoptions.py
 create mode 100644 meta/lib/oeqa/utils/commands.py
 create mode 100644 meta/lib/oeqa/utils/ftools.py
 create mode 100755 scripts/oe-selftest
Chris Larson - Nov. 27, 2013, 6:42 p.m.
On Wed, Nov 27, 2013 at 10:08 AM, Stefan Stanacar <
stefanx.stanacar@intel.com> wrote:

> This series adds an oe-selftest script, some modules and a new layer
> meta-selftest.
> which are meant to help in writing tests (using python unittest) for
> various
> bitbake tools/scripts as well as simple output checks or do
> complete builds with different options (with the emphasis that everything
> checked
> is done outside of bitbake context, just like a human would do.) For more
> details,
> please see YOCTO #4740.
>

Will this have its own repository like all the other poky integrated
layers? It really should, in my opinion.
Paul Eggleton - Nov. 28, 2013, 10:17 a.m.
Hi Chris,

On Wednesday 27 November 2013 11:42:06 Chris Larson wrote:
> On Wed, Nov 27, 2013 at 10:08 AM, Stefan Stanacar <
> stefanx.stanacar@intel.com> wrote:
> > This series adds an oe-selftest script, some modules and a new layer
> > meta-selftest.
> > which are meant to help in writing tests (using python unittest) for
> > various
> > bitbake tools/scripts as well as simple output checks or do
> > complete builds with different options (with the emphasis that everything
> > checked
> > is done outside of bitbake context, just like a human would do.) For more
> > details,
> > please see YOCTO #4740.
> 
> Will this have its own repository like all the other poky integrated
> layers? It really should, in my opinion.

It really needs to stay in the same place as the tests that rely on it; they 
can't work without it. It might be suggested that the tests should go 
elsewhere with them; but I don't think that would make sense - we want people 
to be able to run these tests easily if they make changes to OE-Core to check 
if they have caused any regressions.

If you have a look at the layer, there's not a lot in it, and I wouldn't 
expect a lot to be added in future. One possible change would be to just have 
the layer created on the fly in some temporary directory by the tests 
themselves, rather than having it in the repository; on the other hand it does 
make it a bit more difficult to look into outside of the test environment.

Thoughts?

Cheers,
Paul
Chris Larson - Nov. 28, 2013, 8:29 p.m.
On Thu, Nov 28, 2013 at 3:17 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> On Wednesday 27 November 2013 11:42:06 Chris Larson wrote:
> > On Wed, Nov 27, 2013 at 10:08 AM, Stefan Stanacar <
> > stefanx.stanacar@intel.com> wrote:
> > > This series adds an oe-selftest script, some modules and a new layer
> > > meta-selftest.
> > > which are meant to help in writing tests (using python unittest) for
> > > various
> > > bitbake tools/scripts as well as simple output checks or do
> > > complete builds with different options (with the emphasis that
> everything
> > > checked
> > > is done outside of bitbake context, just like a human would do.) For
> more
> > > details,
> > > please see YOCTO #4740.
> >
> > Will this have its own repository like all the other poky integrated
> > layers? It really should, in my opinion.
>
> It really needs to stay in the same place as the tests that rely on it;
> they
> can't work without it. It might be suggested that the tests should go
> elsewhere with them; but I don't think that would make sense - we want
> people
> to be able to run these tests easily if they make changes to OE-Core to
> check
> if they have caused any regressions.
>
> If you have a look at the layer, there's not a lot in it, and I wouldn't
> expect a lot to be added in future. One possible change would be to just
> have
> the layer created on the fly in some temporary directory by the tests
> themselves, rather than having it in the repository; on the other hand it
> does
> make it a bit more difficult to look into outside of the test environment.


I understand what you’re saying, but if this script is supposed to be
generally useful for OE/Yocto, not just Poky, then the poky repository
shouldn’t be its definitive location. Further, patches against oe-core
should be submitted against the oe-core repository, not the poky
repository, and anything poky specific should go to the poky list.
Paul Eggleton - Nov. 28, 2013, 10:30 p.m.
On Thursday 28 November 2013 13:29:32 Chris Larson wrote:
> On Thu, Nov 28, 2013 at 3:17 AM, Paul Eggleton <
> paul.eggleton@linux.intel.com> wrote:
> > On Wednesday 27 November 2013 11:42:06 Chris Larson wrote:
> > > Will this have its own repository like all the other poky integrated
> > > layers? It really should, in my opinion.
> > 
> > It really needs to stay in the same place as the tests that rely on it;
> > they can't work without it. It might be suggested that the tests should go
> > elsewhere with them; but I don't think that would make sense - we want
> > people to be able to run these tests easily if they make changes to OE-
> > Core to check if they have caused any regressions.
> > 
> > If you have a look at the layer, there's not a lot in it, and I wouldn't
> > expect a lot to be added in future. One possible change would be to just
> > have the layer created on the fly in some temporary directory by the tests
> > themselves, rather than having it in the repository; on the other hand it
> > does make it a bit more difficult to look into outside of the test
> > environment.
> 
> I understand what you’re saying, but if this script is supposed to be
> generally useful for OE/Yocto, not just Poky, then the poky repository
> shouldn’t be its definitive location.

Poky isn't its definitive location, it has been sent as a pull request to the 
OE-Core mailing list to be applied on top of OE-Core.

> Further, patches against oe-core should be submitted against the oe-core
> repository, not the poky repository, and anything poky specific should go to
> the poky list.

Point taken, but there isn't actually anything poky-specific in this series. 
Can we please concentrate on the content of the patches?

Cheers,
Paul