recipetool: support python3 instead of python2

Submitted by Maciej Pijanowski on June 7, 2019, 7:36 a.m. | Patch ID: 162030

Details

Message ID 1559892984-24780-1-git-send-email-maciej.pijanowski@3mdeb.com
State New
Headers show

Commit Message

Maciej Pijanowski June 7, 2019, 7:36 a.m.
Add support for generating python3 recipes using the recipetool / devtool.
Drop python2 support at the same time.

Tested with:

devtool add --also-native python3-python-magic \
https://files.pythonhosted.org/packages/84/30/\
  80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/\
  python-magic-0.4.15.tar.gz

No python2 dependencies were pulled in

Fixes Bug 13264

Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
---
 scripts/lib/recipetool/create_buildsys_python.py | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

Patch hide | download patch | download mbox

diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index ec5449bee99f..f24dce23918a 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -41,11 +41,11 @@  def tinfoil_init(instance):
 
 
 class PythonRecipeHandler(RecipeHandler):
-    base_pkgdeps = ['python-core']
-    excluded_pkgdeps = ['python-dbg']
-    # os.path is provided by python-core
+    base_pkgdeps = ['python3-core']
+    excluded_pkgdeps = ['python3-dbg']
+    # os.path is provided by python3-core
     assume_provided = ['builtins', 'os.path']
-    # Assumes that the host python builtin_module_names is sane for target too
+    # Assumes that the host python3 builtin_module_names is sane for target too
     assume_provided = assume_provided + list(sys.builtin_module_names)
 
     bbvar_map = {
@@ -225,9 +225,9 @@  class PythonRecipeHandler(RecipeHandler):
         self.apply_info_replacements(info)
 
         if uses_setuptools:
-            classes.append('setuptools')
+            classes.append('setuptools3')
         else:
-            classes.append('distutils')
+            classes.append('distutils3')
 
         if license_str:
             for i, line in enumerate(lines_before):
@@ -292,7 +292,7 @@  class PythonRecipeHandler(RecipeHandler):
                 for feature, feature_reqs in extras_req.items():
                     unmapped_deps.difference_update(feature_reqs)
 
-                    feature_req_deps = ('python-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
+                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
                     lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
 
         inst_reqs = set()
@@ -303,7 +303,7 @@  class PythonRecipeHandler(RecipeHandler):
             if inst_reqs:
                 unmapped_deps.difference_update(inst_reqs)
 
-                inst_req_deps = ('python-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
+                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
                 lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
                 lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
                 lines_after.append('RDEPENDS_${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
@@ -364,7 +364,7 @@  class PythonRecipeHandler(RecipeHandler):
         return info, 'setuptools' in imported_modules, non_literals, extensions
 
     def get_setup_args_info(self, setupscript='./setup.py'):
-        cmd = ['python', setupscript]
+        cmd = ['python3', setupscript]
         info = {}
         keys = set(self.bbvar_map.keys())
         keys |= set(self.setuparg_list_fields)
@@ -398,7 +398,7 @@  class PythonRecipeHandler(RecipeHandler):
     def get_setup_byline(self, fields, setupscript='./setup.py'):
         info = {}
 
-        cmd = ['python', setupscript]
+        cmd = ['python3', setupscript]
         cmd.extend('--' + self.setuparg_map.get(f, f.lower()) for f in fields)
         try:
             info_lines = self.run_command(cmd, cwd=os.path.dirname(setupscript)).splitlines()
@@ -535,7 +535,7 @@  class PythonRecipeHandler(RecipeHandler):
         pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
 
         ldata = tinfoil.config_data.createCopy()
-        bb.parse.handle('classes/python-dir.bbclass', ldata, True)
+        bb.parse.handle('classes/python3-dir.bbclass', ldata, True)
         python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
 
         dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')

Comments

Ross Burton June 7, 2019, 8:35 a.m.
My first thought was "won't this break the test" but then I discovered
that there's no test.

Would you be able to write a small test for this, along the lines of
test_recipetool_create_*() in
meta/lib/oeqa/selftest/cases/recipetool.py?

Ross

On Fri, 7 Jun 2019 at 08:56, Maciej Pijanowski
<maciej.pijanowski@3mdeb.com> wrote:
>
> Add support for generating python3 recipes using the recipetool / devtool.
> Drop python2 support at the same time.
>
> Tested with:
>
> devtool add --also-native python3-python-magic \
> https://files.pythonhosted.org/packages/84/30/\
>   80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/\
>   python-magic-0.4.15.tar.gz
>
> No python2 dependencies were pulled in
>
> Fixes Bug 13264
>
> Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
> ---
>  scripts/lib/recipetool/create_buildsys_python.py | 22 +++++++++++-----------
>  1 file changed, 11 insertions(+), 11 deletions(-)
>
> diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
> index ec5449bee99f..f24dce23918a 100644
> --- a/scripts/lib/recipetool/create_buildsys_python.py
> +++ b/scripts/lib/recipetool/create_buildsys_python.py
> @@ -41,11 +41,11 @@ def tinfoil_init(instance):
>
>
>  class PythonRecipeHandler(RecipeHandler):
> -    base_pkgdeps = ['python-core']
> -    excluded_pkgdeps = ['python-dbg']
> -    # os.path is provided by python-core
> +    base_pkgdeps = ['python3-core']
> +    excluded_pkgdeps = ['python3-dbg']
> +    # os.path is provided by python3-core
>      assume_provided = ['builtins', 'os.path']
> -    # Assumes that the host python builtin_module_names is sane for target too
> +    # Assumes that the host python3 builtin_module_names is sane for target too
>      assume_provided = assume_provided + list(sys.builtin_module_names)
>
>      bbvar_map = {
> @@ -225,9 +225,9 @@ class PythonRecipeHandler(RecipeHandler):
>          self.apply_info_replacements(info)
>
>          if uses_setuptools:
> -            classes.append('setuptools')
> +            classes.append('setuptools3')
>          else:
> -            classes.append('distutils')
> +            classes.append('distutils3')
>
>          if license_str:
>              for i, line in enumerate(lines_before):
> @@ -292,7 +292,7 @@ class PythonRecipeHandler(RecipeHandler):
>                  for feature, feature_reqs in extras_req.items():
>                      unmapped_deps.difference_update(feature_reqs)
>
> -                    feature_req_deps = ('python-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
> +                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
>                      lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
>
>          inst_reqs = set()
> @@ -303,7 +303,7 @@ class PythonRecipeHandler(RecipeHandler):
>              if inst_reqs:
>                  unmapped_deps.difference_update(inst_reqs)
>
> -                inst_req_deps = ('python-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
> +                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
>                  lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
>                  lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
>                  lines_after.append('RDEPENDS_${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
> @@ -364,7 +364,7 @@ class PythonRecipeHandler(RecipeHandler):
>          return info, 'setuptools' in imported_modules, non_literals, extensions
>
>      def get_setup_args_info(self, setupscript='./setup.py'):
> -        cmd = ['python', setupscript]
> +        cmd = ['python3', setupscript]
>          info = {}
>          keys = set(self.bbvar_map.keys())
>          keys |= set(self.setuparg_list_fields)
> @@ -398,7 +398,7 @@ class PythonRecipeHandler(RecipeHandler):
>      def get_setup_byline(self, fields, setupscript='./setup.py'):
>          info = {}
>
> -        cmd = ['python', setupscript]
> +        cmd = ['python3', setupscript]
>          cmd.extend('--' + self.setuparg_map.get(f, f.lower()) for f in fields)
>          try:
>              info_lines = self.run_command(cmd, cwd=os.path.dirname(setupscript)).splitlines()
> @@ -535,7 +535,7 @@ class PythonRecipeHandler(RecipeHandler):
>          pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
>
>          ldata = tinfoil.config_data.createCopy()
> -        bb.parse.handle('classes/python-dir.bbclass', ldata, True)
> +        bb.parse.handle('classes/python3-dir.bbclass', ldata, True)
>          python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
>
>          dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
Maciej Pijanowski June 7, 2019, 8:45 a.m.
On 07.06.2019 10:35, Burton, Ross wrote:
> My first thought was "won't this break the test" but then I discovered
> that there's no test.
>
> Would you be able to write a small test for this, along the lines of
> test_recipetool_create_*() in
> meta/lib/oeqa/selftest/cases/recipetool.py?
Sure, I will try that. I haven't been contributing to the QA parts yet.
Is there any documentation / guidelines I should refer to?
Or looking at other test cases should be enough to get it done?
>
> Ross
>
> On Fri, 7 Jun 2019 at 08:56, Maciej Pijanowski
> <maciej.pijanowski@3mdeb.com> wrote:
>> Add support for generating python3 recipes using the recipetool / devtool.
>> Drop python2 support at the same time.
>>
>> Tested with:
>>
>> devtool add --also-native python3-python-magic \
>> https://files.pythonhosted.org/packages/84/30/\
>>   80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/\
>>   python-magic-0.4.15.tar.gz
>>
>> No python2 dependencies were pulled in
>>
>> Fixes Bug 13264
>>
>> Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
>> ---
>>  scripts/lib/recipetool/create_buildsys_python.py | 22 +++++++++++-----------
>>  1 file changed, 11 insertions(+), 11 deletions(-)
>>
>> diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
>> index ec5449bee99f..f24dce23918a 100644
>> --- a/scripts/lib/recipetool/create_buildsys_python.py
>> +++ b/scripts/lib/recipetool/create_buildsys_python.py
>> @@ -41,11 +41,11 @@ def tinfoil_init(instance):
>>
>>
>>  class PythonRecipeHandler(RecipeHandler):
>> -    base_pkgdeps = ['python-core']
>> -    excluded_pkgdeps = ['python-dbg']
>> -    # os.path is provided by python-core
>> +    base_pkgdeps = ['python3-core']
>> +    excluded_pkgdeps = ['python3-dbg']
>> +    # os.path is provided by python3-core
>>      assume_provided = ['builtins', 'os.path']
>> -    # Assumes that the host python builtin_module_names is sane for target too
>> +    # Assumes that the host python3 builtin_module_names is sane for target too
>>      assume_provided = assume_provided + list(sys.builtin_module_names)
>>
>>      bbvar_map = {
>> @@ -225,9 +225,9 @@ class PythonRecipeHandler(RecipeHandler):
>>          self.apply_info_replacements(info)
>>
>>          if uses_setuptools:
>> -            classes.append('setuptools')
>> +            classes.append('setuptools3')
>>          else:
>> -            classes.append('distutils')
>> +            classes.append('distutils3')
>>
>>          if license_str:
>>              for i, line in enumerate(lines_before):
>> @@ -292,7 +292,7 @@ class PythonRecipeHandler(RecipeHandler):
>>                  for feature, feature_reqs in extras_req.items():
>>                      unmapped_deps.difference_update(feature_reqs)
>>
>> -                    feature_req_deps = ('python-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
>> +                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
>>                      lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
>>
>>          inst_reqs = set()
>> @@ -303,7 +303,7 @@ class PythonRecipeHandler(RecipeHandler):
>>              if inst_reqs:
>>                  unmapped_deps.difference_update(inst_reqs)
>>
>> -                inst_req_deps = ('python-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
>> +                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
>>                  lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
>>                  lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
>>                  lines_after.append('RDEPENDS_${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
>> @@ -364,7 +364,7 @@ class PythonRecipeHandler(RecipeHandler):
>>          return info, 'setuptools' in imported_modules, non_literals, extensions
>>
>>      def get_setup_args_info(self, setupscript='./setup.py'):
>> -        cmd = ['python', setupscript]
>> +        cmd = ['python3', setupscript]
>>          info = {}
>>          keys = set(self.bbvar_map.keys())
>>          keys |= set(self.setuparg_list_fields)
>> @@ -398,7 +398,7 @@ class PythonRecipeHandler(RecipeHandler):
>>      def get_setup_byline(self, fields, setupscript='./setup.py'):
>>          info = {}
>>
>> -        cmd = ['python', setupscript]
>> +        cmd = ['python3', setupscript]
>>          cmd.extend('--' + self.setuparg_map.get(f, f.lower()) for f in fields)
>>          try:
>>              info_lines = self.run_command(cmd, cwd=os.path.dirname(setupscript)).splitlines()
>> @@ -535,7 +535,7 @@ class PythonRecipeHandler(RecipeHandler):
>>          pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
>>
>>          ldata = tinfoil.config_data.createCopy()
>> -        bb.parse.handle('classes/python-dir.bbclass', ldata, True)
>> +        bb.parse.handle('classes/python3-dir.bbclass', ldata, True)
>>          python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
>>
>>          dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
>> --
>> 2.7.4
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
Ross Burton June 7, 2019, 8:57 a.m.
On Fri, 7 Jun 2019 at 09:46, Maciej Pijanowski
<maciej.pijanowski@3mdeb.com> wrote:
> > Would you be able to write a small test for this, along the lines of
> > test_recipetool_create_*() in
> > meta/lib/oeqa/selftest/cases/recipetool.py?

> Sure, I will try that. I haven't been contributing to the QA parts yet.
> Is there any documentation / guidelines I should refer to?
> Or looking at other test cases should be enough to get it done?

Thanks, that's much appreciated.  Looking at the other
test_recipetool_create_* tests in that file should show you enough to
write the test case, the hardest bit will most likely be picking a
suitable Python module for the test case!

To execute the tests, use oe-selftest:

# This runs all tests, will take quite a long time
$ oe-selftest -a

# Run just the recipetool tests
$ oe-selftest -r recipetool

# Run just test_recipetool_create_git_http
$ oe-selftest -r recipetool.RecipetoolTests.test_recipetool_create_git_http

Ross