Patchwork [2/5] Move check that all installed files are shipped into insane.bbclass

Checking that all installed files are shipped is in reality a QA check.
It would benefit from mechanisms like ERROR_QA/WARNING_QA. So move it
into insane.bbclass. If some of the files are installed but should not
be shipped for some reasons, one can add them to the variable
IGNORE_UNSHIPPED_FILES.

Signed-off-by: Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
---
 meta/classes/insane.bbclass  |   50 ++++++++++++++++++++++++++++++++++++++++-
 meta/classes/package.bbclass |   15 ------------
 2 files changed, 48 insertions(+), 17 deletions(-)

On Sunday, December 18, 2011, Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
wrote:
> Checking that all installed files are shipped is in reality a QA check.
> It would benefit from mechanisms like ERROR_QA/WARNING_QA. So move it
> into insane.bbclass. If some of the files are installed but should not
> be shipped for some reasons, one can add them to the variable
> IGNORE_UNSHIPPED_FILES.
>

This needs to be documented somewhere too and may be added to
sample.conf.extended

> Signed-off-by: Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
> ---
>  meta/classes/insane.bbclass  |   50
++++++++++++++++++++++++++++++++++++++++-
>  meta/classes/package.bbclass |   15 ------------
>  2 files changed, 48 insertions(+), 17 deletions(-)
>
> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> index 5726e69..41b815c 100644
> --- a/meta/classes/insane.bbclass
> +++ b/meta/classes/insane.bbclass
> @@ -100,7 +100,7 @@ def package_qa_get_machine_dict():
>
>
>  # Currently not being used by default "desktop"
> -WARN_QA ?= "ldflags useless-rpaths rpaths"
> +WARN_QA ?= "ldflags useless-rpaths rpaths unshipped"
>  ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch la2 pkgconfig
la perms"
>
>  def package_qa_clean_path(path,d):
> @@ -485,6 +485,49 @@ def package_qa_check_rdepends(pkg, pkgdest, skip, d):
>
>     return sane
>
> +IGNORE_UNSHIPPED_FILES ??= ""
> +
> +def packages_qa_unshipped_files(d):
> +    warn = (d.getVar('WARN_QA', True) or "").split()
> +    err = (d.getVar('ERROR_QA', True) or "").split()
> +    if not "unshipped" in warn + err:
> +        return True
> +
> +    seen = d.getVar('IGNORE_UNSHIPPED_FILES', True).split()
> +    unshipped = []
> +    dvar = d.getVar('PKGD', True)
> +    destvar = d.getVar('PKGDEST', True)
> +    packages = d.getVar('PACKAGES', True).split()
> +    for p in packages:
> +        pdir = os.path.join(destvar, p)
> +        for root, dirs, files in os.walk(pdir):
> +            dir = root[len(pdir):]
> +            if not dir:
> +                dir = os.sep
> +            for f in (files + dirs):
> +                path = os.path.join(dir, f)
> +                if path not in seen
> +                    seen.append(path)
> +
> +    for root, dirs, files in os.walk(dvar):
> +        dir = root[len(dvar):]
> +        if not dir:
> +            dir = os.sep
> +        for f in (files + dirs):
> +            path = os.path.join(dir, f)
> +            if path not in seen:
> +                unshipped.append(path)
> +
> +    pn = d.getVar('PN', True)
> +
> +    if unshipped == []:
> +        return True
> +
> +    ret = package_qa_handle_error("unshipped", "For recipe %s, the
following files/directories were installed but not shipped in any package:"
% pn, d)
> +    for f in unshipped:
> +        package_qa_handle_error("unshipped", f, d)
> +    return ret
> +
>  # The PACKAGE FUNC to scan each package
>  python do_package_qa () {
>     bb.note("DO PACKAGE QA")
> @@ -522,6 +565,7 @@ python do_package_qa () {
>     g = globals()
>     walk_sane = True
>     rdepends_sane = True
> +    shipped_sane = True
>     for package in packages.split():
>         skip = (d.getVar('INSANE_SKIP_' + package, True) or "").split()
>         if skip:
> @@ -546,8 +590,10 @@ python do_package_qa () {
>         if not package_qa_check_rdepends(package, pkgdest, skip, d):
>             rdepends_sane = False
>
> +    if not packages_qa_unshipped_files(d):
> +        shipped_sane = False
>
> -    if not walk_sane or not rdepends_sane:
> +    if not walk_sane or not rdepends_sane or not shipped_sane:
>         bb.fatal("QA run found fatal errors. Please consider fixing
them.")
>     bb.note("DONE with PACKAGE QA")
>  }
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 39c1d4b..fbea9c6 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -957,21 +957,6 @@ python populate_packages () {
>                del localdata
>        os.chdir(workdir)
>
> -       unshipped = []
> -       for root, dirs, files in os.walk(dvar):
> -               dir = root[len(dvar):]
> -               if not dir:
> -                       dir = os.sep
> -               for f in (files + dirs):
> -                       path = os.path.join(dir, f)
> -                       if ('.' + path) not in seen:
> -                               unshipped.append(path)
> -
> -       if unshipped != []:
> -               bb.warn("For recipe %s, the following files/directories
were installed but not shipped in any package:" % pn)
> -               for f in unshipped:
> -                       bb.warn("  " + f)
> -
>        bb.build.exec_func("package_name_hook", d)
>
>        for pkg in package_list:
> --
> 1.7.7.3
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

On Sun, 2011-12-18 at 15:29 -0800, Khem Raj wrote:
> On Sunday, December 18, 2011, Dmitry Eremin-Solenikov
> <dbaryshkov@gmail.com> wrote:
> > Checking that all installed files are shipped is in reality a QA
> check.
> > It would benefit from mechanisms like ERROR_QA/WARNING_QA. So move
> it
> > into insane.bbclass. If some of the files are installed but should
> not
> > be shipped for some reasons, one can add them to the variable
> > IGNORE_UNSHIPPED_FILES.
> >
> This needs to be documented somewhere too and may be added to
> sample.conf.extended

This doesn't make sense for sample.conf.extended. We're talking about a
recipe specific configuration variable, not distro/machine/conf config
which is more appropriate there.

I'd suggest adding to documentation.conf and/or the various manuals.

Cheers,

Richard

On Sun, Dec 18, 2011 at 4:02 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Sun, 2011-12-18 at 15:29 -0800, Khem Raj wrote:
>> On Sunday, December 18, 2011, Dmitry Eremin-Solenikov
>> <dbaryshkov@gmail.com> wrote:
>> > Checking that all installed files are shipped is in reality a QA
>> check.
>> > It would benefit from mechanisms like ERROR_QA/WARNING_QA. So move
>> it
>> > into insane.bbclass. If some of the files are installed but should
>> not
>> > be shipped for some reasons, one can add them to the variable
>> > IGNORE_UNSHIPPED_FILES.
>> >
>> This needs to be documented somewhere too and may be added to
>> sample.conf.extended
>
> This doesn't make sense for sample.conf.extended. We're talking about a
> recipe specific configuration variable, not distro/machine/conf config
> which is more appropriate there.

yes forgot that

>
> I'd suggest adding to documentation.conf and/or the various manuals.
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core