Patchwork [1/2] insane.bbclass: QAPATHTEST: check file depends on bash, perl and python

login
register
mail settings
Submitter Robert Yang
Date July 14, 2014, 9:41 a.m.
Message ID <bb94f3eb6cee9ad053d6a938766e45e9fab2d167.1405330780.git.liezhi.yang@windriver.com>
Download mbox | patch
Permalink /patch/75447/
State New
Headers show

Comments

Robert Yang - July 14, 2014, 9:41 a.m.
If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't
have bash, python or perl, then warn.

This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the
its "Requires" section if it is a perl script, but ipk or deb can't
depend on file such as "/usr/bin/perl", they can only depend on pkg, so
they would know nothing about perl, then there would be depend issues.

This check can help us figure out the depends, it nearly has no effect
on performance when enabled.

[YOCTO #1662]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
Mark Hatle - July 14, 2014, 8:02 p.m.
On 7/14/14, 4:41 AM, Robert Yang wrote:
> If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't
> have bash, python or perl, then warn.
>
> This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the
> its "Requires" section if it is a perl script, but ipk or deb can't
> depend on file such as "/usr/bin/perl", they can only depend on pkg, so
> they would know nothing about perl, then there would be depend issues.
>
> This check can help us figure out the depends, it nearly has no effect
> on performance when enabled.
>
> [YOCTO #1662]
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
>   1 file changed, 30 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> index 106ace7..a14e668 100644
> --- a/meta/classes/insane.bbclass
> +++ b/meta/classes/insane.bbclass
> @@ -19,7 +19,7 @@
>
>   # unsafe-references-in-binaries requires prelink-rtld from
>   # prelink-native, but we don't want this DEPENDS for -native builds
> -QADEPENDS = "prelink-native"
> +QADEPENDS = "prelink-native file-native"

If the perfile dependency processing is enabled, run in: package_do_filedeps

The system will already have gathered the information needed for the QA 
processing.  You can look at:

# Collect perfile run-time dependency metadata
# Output:
#  FILERPROVIDESFLIST_pkg - list of all files w/ deps
#  FILERPROVIDES_filepath_pkg - per file dep
#
#  FILERDEPENDSFLIST_pkg - list of all files w/ deps
#  FILERDEPENDS_filepath_pkg - per file dep

variables for the per-file dependencies and then warn if a dependency is not 
already declared in the manual RDEPENDS -- and/or promote them to real 
dependencies automatically.

This should be faster then iterating through and calling file directly.

--Mark

>   QADEPENDS_class-native = ""
>   QADEPENDS_class-nativesdk = ""
>   QA_SANE = "True"
> @@ -29,7 +29,7 @@ QA_SANE = "True"
>   WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
>               textrel already-stripped incompatible-license files-invalid \
>               installed-vs-shipped compile-host-path install-host-path \
> -            pn-overrides infodir \
> +            pn-overrides infodir file-depends \
>               "
>   ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
>               perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
> @@ -605,6 +605,34 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf, messages):
>                   trimmed = path.replace(os.path.join (d.getVar("PKGDEST", True), name), "")
>                   messages.append("Symlink %s in %s points to TMPDIR" % (trimmed, name))
>
> +QAPATHTEST[file-depends] = "package_qa_check_file_depends"
> +def package_qa_check_file_depends(path, name, d, elf, messages):
> +    """
> +    Check that the package doesn't miss the RDEPENDS on perl, python and bash
> +    """
> +
> +    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
> +        return
> +
> +    import subprocess
> +    import re
> +
> +    # The file command may return:
> +    # - Perl or perl, Python or python.
> +    # - "Bourne-Again shell" script or "bash" for bash script
> +    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
> +                "Bourne-Again shell": "bash"}
> +
> +    file_output = bb.process.Popen(["file", "-b", path], stdout=subprocess.PIPE).stdout.read()
> +    for p in prog_maps:
> +        if re.search('%s .*script, ASCII text executable' % p, file_output):
> +            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name, True) or "")
> +            if prog_maps[p] not in rdepends:
> +                if prog_maps[p] == "python" and "python-core" in rdepends:
> +                    continue
> +                messages.append("%s is a %s script, but %s is not in RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))
> +                break
> +
>   def package_qa_check_license(workdir, d):
>       """
>       Check for changes in the license files
>
Robert Yang - July 15, 2014, 1:24 a.m.
On 07/15/2014 04:02 AM, Mark Hatle wrote:
> On 7/14/14, 4:41 AM, Robert Yang wrote:
>> If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't
>> have bash, python or perl, then warn.
>>
>> This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the
>> its "Requires" section if it is a perl script, but ipk or deb can't
>> depend on file such as "/usr/bin/perl", they can only depend on pkg, so
>> they would know nothing about perl, then there would be depend issues.
>>
>> This check can help us figure out the depends, it nearly has no effect
>> on performance when enabled.
>>
>> [YOCTO #1662]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
>>   1 file changed, 30 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
>> index 106ace7..a14e668 100644
>> --- a/meta/classes/insane.bbclass
>> +++ b/meta/classes/insane.bbclass
>> @@ -19,7 +19,7 @@
>>
>>   # unsafe-references-in-binaries requires prelink-rtld from
>>   # prelink-native, but we don't want this DEPENDS for -native builds
>> -QADEPENDS = "prelink-native"
>> +QADEPENDS = "prelink-native file-native"
>
> If the perfile dependency processing is enabled, run in: package_do_filedeps
>
> The system will already have gathered the information needed for the QA
> processing.  You can look at:
>
> # Collect perfile run-time dependency metadata
> # Output:
> #  FILERPROVIDESFLIST_pkg - list of all files w/ deps
> #  FILERPROVIDES_filepath_pkg - per file dep
> #
> #  FILERDEPENDSFLIST_pkg - list of all files w/ deps
> #  FILERDEPENDS_filepath_pkg - per file dep
>
> variables for the per-file dependencies and then warn if a dependency is not
> already declared in the manual RDEPENDS -- and/or promote them to real
> dependencies automatically.
>
> This should be faster then iterating through and calling file directly.
>

Thanks, I will only warn from package_do_filedeps and let the user add the
RDEPENDS manually, I will also fix a few of the RDEPENDS in V2.

Please feel free to let me know if others have any suggestions.

// Robert


> --Mark
>
>>   QADEPENDS_class-native = ""
>>   QADEPENDS_class-nativesdk = ""
>>   QA_SANE = "True"
>> @@ -29,7 +29,7 @@ QA_SANE = "True"
>>   WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
>>               textrel already-stripped incompatible-license files-invalid \
>>               installed-vs-shipped compile-host-path install-host-path \
>> -            pn-overrides infodir \
>> +            pn-overrides infodir file-depends \
>>               "
>>   ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
>>               perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
>> @@ -605,6 +605,34 @@ def package_qa_check_symlink_to_sysroot(path, name, d,
>> elf, messages):
>>                   trimmed = path.replace(os.path.join (d.getVar("PKGDEST",
>> True), name), "")
>>                   messages.append("Symlink %s in %s points to TMPDIR" %
>> (trimmed, name))
>>
>> +QAPATHTEST[file-depends] = "package_qa_check_file_depends"
>> +def package_qa_check_file_depends(path, name, d, elf, messages):
>> +    """
>> +    Check that the package doesn't miss the RDEPENDS on perl, python and bash
>> +    """
>> +
>> +    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
>> +        return
>> +
>> +    import subprocess
>> +    import re
>> +
>> +    # The file command may return:
>> +    # - Perl or perl, Python or python.
>> +    # - "Bourne-Again shell" script or "bash" for bash script
>> +    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
>> +                "Bourne-Again shell": "bash"}
>> +
>> +    file_output = bb.process.Popen(["file", "-b", path],
>> stdout=subprocess.PIPE).stdout.read()
>> +    for p in prog_maps:
>> +        if re.search('%s .*script, ASCII text executable' % p, file_output):
>> +            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name,
>> True) or "")
>> +            if prog_maps[p] not in rdepends:
>> +                if prog_maps[p] == "python" and "python-core" in rdepends:
>> +                    continue
>> +                messages.append("%s is a %s script, but %s is not in
>> RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))
>> +                break
>> +
>>   def package_qa_check_license(workdir, d):
>>       """
>>       Check for changes in the license files
>>
>

Patch

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 106ace7..a14e668 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -19,7 +19,7 @@ 
 
 # unsafe-references-in-binaries requires prelink-rtld from
 # prelink-native, but we don't want this DEPENDS for -native builds
-QADEPENDS = "prelink-native"
+QADEPENDS = "prelink-native file-native"
 QADEPENDS_class-native = ""
 QADEPENDS_class-nativesdk = ""
 QA_SANE = "True"
@@ -29,7 +29,7 @@  QA_SANE = "True"
 WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
             textrel already-stripped incompatible-license files-invalid \
             installed-vs-shipped compile-host-path install-host-path \
-            pn-overrides infodir \
+            pn-overrides infodir file-depends \
             "
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
@@ -605,6 +605,34 @@  def package_qa_check_symlink_to_sysroot(path, name, d, elf, messages):
                 trimmed = path.replace(os.path.join (d.getVar("PKGDEST", True), name), "")
                 messages.append("Symlink %s in %s points to TMPDIR" % (trimmed, name))
 
+QAPATHTEST[file-depends] = "package_qa_check_file_depends"
+def package_qa_check_file_depends(path, name, d, elf, messages):
+    """
+    Check that the package doesn't miss the RDEPENDS on perl, python and bash
+    """
+
+    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
+        return
+
+    import subprocess
+    import re
+
+    # The file command may return:
+    # - Perl or perl, Python or python.
+    # - "Bourne-Again shell" script or "bash" for bash script
+    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
+                "Bourne-Again shell": "bash"}
+
+    file_output = bb.process.Popen(["file", "-b", path], stdout=subprocess.PIPE).stdout.read()
+    for p in prog_maps:
+        if re.search('%s .*script, ASCII text executable' % p, file_output):
+            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name, True) or "")
+            if prog_maps[p] not in rdepends:
+                if prog_maps[p] == "python" and "python-core" in rdepends:
+                    continue
+                messages.append("%s is a %s script, but %s is not in RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))
+                break
+
 def package_qa_check_license(workdir, d):
     """
     Check for changes in the license files