Patchwork [00/15] Add python 3 support

login
register
mail settings
Submitter Khem Raj
Date Jan. 30, 2014, 4:09 a.m.
Message ID <cover.1391054829.git.raj.khem@gmail.com>
Download mbox
Permalink /patch/66055/
State New
Headers show

Pull-request

git://git.openembedded.org/openembedded-core-contrib kraj/python3

Comments

Khem Raj - Jan. 30, 2014, 4:09 a.m.
Post V5

Upgrade to 3.3.3

Fix the warnings like

WARNING: The recipe python3 is trying to install files into a shared
area when those files already exist. Those files and their manifest
location are:

/srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/include/python3.3m/pyconfig.h

/srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/libpython3.3m.so.1.0

/srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/libpython3.3m.so

/srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python3.3/config/Makefile

and

WARNING: The recipe python-setuptools is trying to install files into a 
shared area when those files already exist. Those files and their 
manifest location are:
 
/srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python2.7/site-packages/setuptools.pth
    Matched in manifest-qemux86-64-python-distribute
 
/srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python2.7/site-packages/site.pyc
    Matched in manifest-qemux86-64-python-distribute
 
/srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python2.7/site-packages/site.py
    Matched in manifest-qemux86-64-python-distribute


The following changes since commit c3f3fed1b397b6ee6d44e7d39e63d084e4a88e30:

  toaster.bbclass: save original package name (2014-01-29 17:40:29 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib kraj/python3
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kraj/python3

Khem Raj (15):
  generate-manifest-3.3.py: Add script to generate python 3.3 manifests
  python-3.3-manifest: Add python3 manifest file
  python3native.bbclass: Add python3 abstraction class
  package_rpm.bbclass: Trigger the regexp on exact keyword match
  classes/distutils: Introduce PYTHON_PN
  classes: Add distutils for python3
  distutils: Introduce PYTHON_ABI variable
  distutils3: Do build_ext as separate step during compile
  python3: Add target and native recipes
  python_2.7.3.bb: Inherit python-dir
  python-distribute: Add recipes for python2 and python3
  distutils3.bbclass: Fix typo and use proper values for target sysroot
  distutils3.bbclass: Use MACHINE for sysroot when not building for
    build host
  python3: Fix staging warnings due to duplicate install
  python-setuptools: Remove its provided by python-distribute

 meta/classes/distutils-base.bbclass                |    5 +-
 meta/classes/distutils-native-base.bbclass         |    4 +-
 meta/classes/distutils-tools.bbclass               |   77 ++++
 meta/classes/distutils.bbclass                     |   23 +-
 meta/classes/distutils3-base.bbclass               |    8 +
 meta/classes/distutils3-native-base.bbclass        |    4 +
 meta/classes/distutils3.bbclass                    |   98 +++++
 meta/classes/package_rpm.bbclass                   |    2 +-
 meta/classes/python-dir.bbclass                    |    2 +
 meta/classes/python3native.bbclass                 |    7 +
 meta/classes/pythonnative.bbclass                  |    9 +-
 meta/classes/setuptools.bbclass                    |    3 +-
 meta/classes/setuptools3.bbclass                   |    8 +
 .../python/python-3.3-manifest.inc                 |  260 ++++++++++++++
 .../python/python-distribute_0.6.32.bb             |   50 +++
 .../python/python-setuptools_1.4.bb                |   40 ---
 .../python/python3-distribute_0.6.32.bb            |   49 +++
 .../python/python3-native_3.3.3.bb                 |   70 ++++
 .../python/python3/000-cross-compile.patch         |   83 +++++
 ...sue-13032-where-it-fails-with-UnicodeDeco.patch |   37 ++
 .../python3/020-dont-compile-python-files.patch    |   37 ++
 .../python/python3/03-fix-tkinter-detection.patch  |   42 +++
 .../python/python3/030-fixup-include-dirs.patch    |   33 ++
 .../python/python3/04-default-is-optimized.patch   |   58 +++
 .../python3/06-ctypes-libffi-fix-configure.patch   |   44 +++
 .../python3/070-dont-clean-ipkg-install.patch      |   36 ++
 .../python3/080-distutils-dont_adjust_files.patch  |   92 +++++
 .../python/python3/110-enable-zlib.patch           |   21 ++
 ...2-distutils-prefix-is-inside-staging-area.patch |   78 ++++
 .../python/python3/130-readline-setup.patch        |   55 +++
 .../python/python3/150-fix-setupterm.patch         |   17 +
 .../python3/avoid_warning_about_tkinter.patch      |   25 ++
 meta/recipes-devtools/python/python3/cgi_py.patch  |   23 ++
 .../python3/fix_for_using_different_libdir.patch   |   54 +++
 .../python3/host_include_contamination.patch       |   28 ++
 .../python/python3/python-3.3-multilib.patch       |  336 +++++++++++++++++
 .../python/python3/remove_sqlite_rpath.patch       |   19 +
 .../python/python3/setuptweaks.patch               |   57 +++
 .../python/python3/shutil-follow-symlink-fix.patch |   17 +
 .../python/python3/sitecustomize.py                |   37 ++
 .../sys_platform_is_now_always_linux2.patch        |   29 ++
 .../python/python3/sysroot-include-headers.patch   |   35 ++
 .../python/python3/unixccompiler.patch             |   33 ++
 meta/recipes-devtools/python/python3_3.3.3.bb      |  197 ++++++++++
 meta/recipes-devtools/python/python_2.7.3.bb       |    2 +-
 .../packagegroups/packagegroup-toolset-native.bb   |    2 +-
 scripts/contrib/python/generate-manifest-3.3.py    |  376 ++++++++++++++++++++
 47 files changed, 2558 insertions(+), 64 deletions(-)
 create mode 100644 meta/classes/distutils-tools.bbclass
 create mode 100644 meta/classes/distutils3-base.bbclass
 create mode 100644 meta/classes/distutils3-native-base.bbclass
 create mode 100644 meta/classes/distutils3.bbclass
 create mode 100644 meta/classes/python3native.bbclass
 create mode 100644 meta/classes/setuptools3.bbclass
 create mode 100644 meta/recipes-devtools/python/python-3.3-manifest.inc
 create mode 100644 meta/recipes-devtools/python/python-distribute_0.6.32.bb
 delete mode 100644 meta/recipes-devtools/python/python-setuptools_1.4.bb
 create mode 100644 meta/recipes-devtools/python/python3-distribute_0.6.32.bb
 create mode 100644 meta/recipes-devtools/python/python3-native_3.3.3.bb
 create mode 100644 meta/recipes-devtools/python/python3/000-cross-compile.patch
 create mode 100644 meta/recipes-devtools/python/python3/0001-h2py-Fix-issue-13032-where-it-fails-with-UnicodeDeco.patch
 create mode 100644 meta/recipes-devtools/python/python3/020-dont-compile-python-files.patch
 create mode 100644 meta/recipes-devtools/python/python3/03-fix-tkinter-detection.patch
 create mode 100644 meta/recipes-devtools/python/python3/030-fixup-include-dirs.patch
 create mode 100644 meta/recipes-devtools/python/python3/04-default-is-optimized.patch
 create mode 100644 meta/recipes-devtools/python/python3/06-ctypes-libffi-fix-configure.patch
 create mode 100644 meta/recipes-devtools/python/python3/070-dont-clean-ipkg-install.patch
 create mode 100644 meta/recipes-devtools/python/python3/080-distutils-dont_adjust_files.patch
 create mode 100644 meta/recipes-devtools/python/python3/110-enable-zlib.patch
 create mode 100644 meta/recipes-devtools/python/python3/12-distutils-prefix-is-inside-staging-area.patch
 create mode 100644 meta/recipes-devtools/python/python3/130-readline-setup.patch
 create mode 100644 meta/recipes-devtools/python/python3/150-fix-setupterm.patch
 create mode 100644 meta/recipes-devtools/python/python3/avoid_warning_about_tkinter.patch
 create mode 100644 meta/recipes-devtools/python/python3/cgi_py.patch
 create mode 100644 meta/recipes-devtools/python/python3/fix_for_using_different_libdir.patch
 create mode 100644 meta/recipes-devtools/python/python3/host_include_contamination.patch
 create mode 100644 meta/recipes-devtools/python/python3/python-3.3-multilib.patch
 create mode 100644 meta/recipes-devtools/python/python3/remove_sqlite_rpath.patch
 create mode 100644 meta/recipes-devtools/python/python3/setuptweaks.patch
 create mode 100644 meta/recipes-devtools/python/python3/shutil-follow-symlink-fix.patch
 create mode 100644 meta/recipes-devtools/python/python3/sitecustomize.py
 create mode 100644 meta/recipes-devtools/python/python3/sys_platform_is_now_always_linux2.patch
 create mode 100644 meta/recipes-devtools/python/python3/sysroot-include-headers.patch
 create mode 100644 meta/recipes-devtools/python/python3/unixccompiler.patch
 create mode 100644 meta/recipes-devtools/python/python3_3.3.3.bb
 create mode 100755 scripts/contrib/python/generate-manifest-3.3.py
Khem Raj - Jan. 30, 2014, 9:04 a.m.
On Wed, Jan 29, 2014 at 8:12 PM, Khem Raj <raj.khem@gmail.com> wrote:
> Post V5
>
> Upgrade to 3.3.3
>
> Fix the warnings like
>
> WARNING: The recipe python3 is trying to install files into a shared
> area when those files already exist. Those files and their manifest
> location are:
>
> /srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/include/python3.3m/pyconfig.h
>
> /srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/libpython3.3m.so.1.0
>
> /srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/libpython3.3m.so
>
> /srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python3.3/config/Makefile
>
> and
>
> WARNING: The recipe python-setuptools is trying to install files into a
> shared area when those files already exist. Those files and their
> manifest location are:
>
> /srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python2.7/site-packages/setuptools.pth
>     Matched in manifest-qemux86-64-python-distribute
>
> /srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python2.7/site-packages/site.pyc
>     Matched in manifest-qemux86-64-python-distribute
>
> /srv/ssd/sgw/builds/world/tmp/sysroots/qemux86-64/usr/lib/python2.7/site-packages/site.py
>     Matched in manifest-qemux86-64-python-distribute
>

I have fixed two more issues 1 where cross build failed on hosts which
had ncursesw installed
secondly the in tree python modules (.so) were not being built due to
wrong linker paths

I have updated the pull branch with these two fixes.

>
> The following changes since commit c3f3fed1b397b6ee6d44e7d39e63d084e4a88e30:
>
>   toaster.bbclass: save original package name (2014-01-29 17:40:29 +0000)
>
> are available in the git repository at:
>
>   git://git.openembedded.org/openembedded-core-contrib kraj/python3
>   http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kraj/python3
>
> Khem Raj (15):
>   generate-manifest-3.3.py: Add script to generate python 3.3 manifests
>   python-3.3-manifest: Add python3 manifest file
>   python3native.bbclass: Add python3 abstraction class
>   package_rpm.bbclass: Trigger the regexp on exact keyword match
>   classes/distutils: Introduce PYTHON_PN
>   classes: Add distutils for python3
>   distutils: Introduce PYTHON_ABI variable
>   distutils3: Do build_ext as separate step during compile
>   python3: Add target and native recipes
>   python_2.7.3.bb: Inherit python-dir
>   python-distribute: Add recipes for python2 and python3
>   distutils3.bbclass: Fix typo and use proper values for target sysroot
>   distutils3.bbclass: Use MACHINE for sysroot when not building for
>     build host
>   python3: Fix staging warnings due to duplicate install
>   python-setuptools: Remove its provided by python-distribute
>
>  meta/classes/distutils-base.bbclass                |    5 +-
>  meta/classes/distutils-native-base.bbclass         |    4 +-
>  meta/classes/distutils-tools.bbclass               |   77 ++++
>  meta/classes/distutils.bbclass                     |   23 +-
>  meta/classes/distutils3-base.bbclass               |    8 +
>  meta/classes/distutils3-native-base.bbclass        |    4 +
>  meta/classes/distutils3.bbclass                    |   98 +++++
>  meta/classes/package_rpm.bbclass                   |    2 +-
>  meta/classes/python-dir.bbclass                    |    2 +
>  meta/classes/python3native.bbclass                 |    7 +
>  meta/classes/pythonnative.bbclass                  |    9 +-
>  meta/classes/setuptools.bbclass                    |    3 +-
>  meta/classes/setuptools3.bbclass                   |    8 +
>  .../python/python-3.3-manifest.inc                 |  260 ++++++++++++++
>  .../python/python-distribute_0.6.32.bb             |   50 +++
>  .../python/python-setuptools_1.4.bb                |   40 ---
>  .../python/python3-distribute_0.6.32.bb            |   49 +++
>  .../python/python3-native_3.3.3.bb                 |   70 ++++
>  .../python/python3/000-cross-compile.patch         |   83 +++++
>  ...sue-13032-where-it-fails-with-UnicodeDeco.patch |   37 ++
>  .../python3/020-dont-compile-python-files.patch    |   37 ++
>  .../python/python3/03-fix-tkinter-detection.patch  |   42 +++
>  .../python/python3/030-fixup-include-dirs.patch    |   33 ++
>  .../python/python3/04-default-is-optimized.patch   |   58 +++
>  .../python3/06-ctypes-libffi-fix-configure.patch   |   44 +++
>  .../python3/070-dont-clean-ipkg-install.patch      |   36 ++
>  .../python3/080-distutils-dont_adjust_files.patch  |   92 +++++
>  .../python/python3/110-enable-zlib.patch           |   21 ++
>  ...2-distutils-prefix-is-inside-staging-area.patch |   78 ++++
>  .../python/python3/130-readline-setup.patch        |   55 +++
>  .../python/python3/150-fix-setupterm.patch         |   17 +
>  .../python3/avoid_warning_about_tkinter.patch      |   25 ++
>  meta/recipes-devtools/python/python3/cgi_py.patch  |   23 ++
>  .../python3/fix_for_using_different_libdir.patch   |   54 +++
>  .../python3/host_include_contamination.patch       |   28 ++
>  .../python/python3/python-3.3-multilib.patch       |  336 +++++++++++++++++
>  .../python/python3/remove_sqlite_rpath.patch       |   19 +
>  .../python/python3/setuptweaks.patch               |   57 +++
>  .../python/python3/shutil-follow-symlink-fix.patch |   17 +
>  .../python/python3/sitecustomize.py                |   37 ++
>  .../sys_platform_is_now_always_linux2.patch        |   29 ++
>  .../python/python3/sysroot-include-headers.patch   |   35 ++
>  .../python/python3/unixccompiler.patch             |   33 ++
>  meta/recipes-devtools/python/python3_3.3.3.bb      |  197 ++++++++++
>  meta/recipes-devtools/python/python_2.7.3.bb       |    2 +-
>  .../packagegroups/packagegroup-toolset-native.bb   |    2 +-
>  scripts/contrib/python/generate-manifest-3.3.py    |  376 ++++++++++++++++++++
>  47 files changed, 2558 insertions(+), 64 deletions(-)
>  create mode 100644 meta/classes/distutils-tools.bbclass
>  create mode 100644 meta/classes/distutils3-base.bbclass
>  create mode 100644 meta/classes/distutils3-native-base.bbclass
>  create mode 100644 meta/classes/distutils3.bbclass
>  create mode 100644 meta/classes/python3native.bbclass
>  create mode 100644 meta/classes/setuptools3.bbclass
>  create mode 100644 meta/recipes-devtools/python/python-3.3-manifest.inc
>  create mode 100644 meta/recipes-devtools/python/python-distribute_0.6.32.bb
>  delete mode 100644 meta/recipes-devtools/python/python-setuptools_1.4.bb
>  create mode 100644 meta/recipes-devtools/python/python3-distribute_0.6.32.bb
>  create mode 100644 meta/recipes-devtools/python/python3-native_3.3.3.bb
>  create mode 100644 meta/recipes-devtools/python/python3/000-cross-compile.patch
>  create mode 100644 meta/recipes-devtools/python/python3/0001-h2py-Fix-issue-13032-where-it-fails-with-UnicodeDeco.patch
>  create mode 100644 meta/recipes-devtools/python/python3/020-dont-compile-python-files.patch
>  create mode 100644 meta/recipes-devtools/python/python3/03-fix-tkinter-detection.patch
>  create mode 100644 meta/recipes-devtools/python/python3/030-fixup-include-dirs.patch
>  create mode 100644 meta/recipes-devtools/python/python3/04-default-is-optimized.patch
>  create mode 100644 meta/recipes-devtools/python/python3/06-ctypes-libffi-fix-configure.patch
>  create mode 100644 meta/recipes-devtools/python/python3/070-dont-clean-ipkg-install.patch
>  create mode 100644 meta/recipes-devtools/python/python3/080-distutils-dont_adjust_files.patch
>  create mode 100644 meta/recipes-devtools/python/python3/110-enable-zlib.patch
>  create mode 100644 meta/recipes-devtools/python/python3/12-distutils-prefix-is-inside-staging-area.patch
>  create mode 100644 meta/recipes-devtools/python/python3/130-readline-setup.patch
>  create mode 100644 meta/recipes-devtools/python/python3/150-fix-setupterm.patch
>  create mode 100644 meta/recipes-devtools/python/python3/avoid_warning_about_tkinter.patch
>  create mode 100644 meta/recipes-devtools/python/python3/cgi_py.patch
>  create mode 100644 meta/recipes-devtools/python/python3/fix_for_using_different_libdir.patch
>  create mode 100644 meta/recipes-devtools/python/python3/host_include_contamination.patch
>  create mode 100644 meta/recipes-devtools/python/python3/python-3.3-multilib.patch
>  create mode 100644 meta/recipes-devtools/python/python3/remove_sqlite_rpath.patch
>  create mode 100644 meta/recipes-devtools/python/python3/setuptweaks.patch
>  create mode 100644 meta/recipes-devtools/python/python3/shutil-follow-symlink-fix.patch
>  create mode 100644 meta/recipes-devtools/python/python3/sitecustomize.py
>  create mode 100644 meta/recipes-devtools/python/python3/sys_platform_is_now_always_linux2.patch
>  create mode 100644 meta/recipes-devtools/python/python3/sysroot-include-headers.patch
>  create mode 100644 meta/recipes-devtools/python/python3/unixccompiler.patch
>  create mode 100644 meta/recipes-devtools/python/python3_3.3.3.bb
>  create mode 100755 scripts/contrib/python/generate-manifest-3.3.py
>
> --
> 1.7.10.4
>
Ross Burton - Jan. 30, 2014, 6:14 p.m.
On 30 January 2014 04:09, Khem Raj <raj.khem@gmail.com> wrote:
>   classes: Add distutils for python3
>   distutils3: Do build_ext as separate step during compile
>   distutils3.bbclass: Fix typo and use proper values for target sysroot
>   distutils3.bbclass: Use MACHINE for sysroot when not building for
>     build host

Please try and squash patches like this set that simply fix bugs in
previous patches.

Ross
Khem Raj - Jan. 30, 2014, 6:26 p.m.
On Thu, Jan 30, 2014 at 10:14 AM, Burton, Ross <ross.burton@intel.com> wrote:
> On 30 January 2014 04:09, Khem Raj <raj.khem@gmail.com> wrote:
>>   classes: Add distutils for python3
>>   distutils3: Do build_ext as separate step during compile
>>   distutils3.bbclass: Fix typo and use proper values for target sysroot
>>   distutils3.bbclass: Use MACHINE for sysroot when not building for
>>     build host
>
> Please try and squash patches like this set that simply fix bugs in
> previous patches.

Thanks, updated pull branch which squashed them and also another set
of 4 python3 recipe specific patches

>
> Ross
Khem Raj - Jan. 31, 2014, 9:45 p.m.
There was another report where collections were packaged in python3-misc which
make python3 to not work without installing python3-misc I have fixed
that and updated
the pull tree.

On Thu, Jan 30, 2014 at 10:26 AM, Khem Raj <raj.khem@gmail.com> wrote:
> On Thu, Jan 30, 2014 at 10:14 AM, Burton, Ross <ross.burton@intel.com> wrote:
>> On 30 January 2014 04:09, Khem Raj <raj.khem@gmail.com> wrote:
>>>   classes: Add distutils for python3
>>>   distutils3: Do build_ext as separate step during compile
>>>   distutils3.bbclass: Fix typo and use proper values for target sysroot
>>>   distutils3.bbclass: Use MACHINE for sysroot when not building for
>>>     build host
>>
>> Please try and squash patches like this set that simply fix bugs in
>> previous patches.
>
> Thanks, updated pull branch which squashed them and also another set
> of 4 python3 recipe specific patches
>
>>
>> Ross
Paul Eggleton - Feb. 3, 2014, 12:03 p.m.
Hi Khem,

On Wednesday 29 January 2014 20:12:24 Khem Raj wrote:
> Post V5
> 
> Upgrade to 3.3.3

I see this series has been merged at last - great! Could you also send Scott 
some information about documentation changes that we should have for this?

Thanks,
Paul
Khem Raj - Feb. 3, 2014, 4:49 p.m.
On Mon, Feb 3, 2014 at 4:03 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> Hi Khem,
>
> On Wednesday 29 January 2014 20:12:24 Khem Raj wrote:
>> Post V5
>>
>> Upgrade to 3.3.3
>
> I see this series has been merged at last - great! Could you also send Scott
> some information about documentation changes that we should have for this?

I will send another patch to define a python3 packagegroup and I think
it would be then
good to document how to include python3 feature in images.

>
> Thanks,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
Paul Eggleton - Feb. 3, 2014, 4:55 p.m.
On Monday 03 February 2014 08:49:36 Khem Raj wrote:
> On Mon, Feb 3, 2014 at 4:03 AM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
> > I see this series has been merged at last - great! Could you also send
> > Scott some information about documentation changes that we should have
> > for this?
>
> I will send another patch to define a python3 packagegroup and I think
> it would be then
> good to document how to include python3 feature in images.

Well, there's that, but you've also additional variables (e.g. PYTHON_ABI and 
PYTHON_PN).

Cheers,
Paul