Patchwork [CONSOLIDATED,PULL,17/17] python: skip setup.py 'import check' when cross-compiling

login
register
mail settings
Submitter Saul Wold
Date Nov. 8, 2011, 7:22 p.m.
Message ID <9f8d5c15eb59611c86dee83180642f29dcc7a361.1320780004.git.sgw@linux.intel.com>
Download mbox | patch
Permalink /patch/14563/
State New
Headers show

Comments

Saul Wold - Nov. 8, 2011, 7:22 p.m.
From: Tom Zanussi <tom.zanussi@intel.com>

build_extension() in setup.py, as part of the build process, does an
'import check' on the built extension.  The import check in turn
dlopen()'s the shared library associated with the extension, which
isn't something that makes sense if that library was cross-compiled
for a different architecture.

This was noticed with an x86_64 target that was compiled with avx
support, because it caused 'illegal instruction' exceptions:

| /bin/sh: line 1: 14575 Illegal instruction ... -E ./setup.py -q build

For other target architectures, it doesn't necessarily cause illegal
instruction exceptions, but still fails.  For example, on arm, the
failure pathway causes this warning:

*** WARNING: renaming "cmath" since importing it failed: .../cmath.so:
    wrong ELF class: ELFCLASS32

This patch to setup.py and the associated recipe changes allow the
whole 'import check' logic to be skipped when cross-compiling.

Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
---
 .../python/setup_py_skip_cross_import_check.patch  |   27 ++++++++++++++++++++
 meta/recipes-devtools/python/python_2.7.2.bb       |    5 +++
 2 files changed, 32 insertions(+), 0 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch
Koen Kooi - Nov. 9, 2011, 9:39 a.m.
Missing PR bump!

Op 8 nov. 2011, om 20:22 heeft Saul Wold het volgende geschreven:

> From: Tom Zanussi <tom.zanussi@intel.com>
> 
> build_extension() in setup.py, as part of the build process, does an
> 'import check' on the built extension.  The import check in turn
> dlopen()'s the shared library associated with the extension, which
> isn't something that makes sense if that library was cross-compiled
> for a different architecture.
> 
> This was noticed with an x86_64 target that was compiled with avx
> support, because it caused 'illegal instruction' exceptions:
> 
> | /bin/sh: line 1: 14575 Illegal instruction ... -E ./setup.py -q build
> 
> For other target architectures, it doesn't necessarily cause illegal
> instruction exceptions, but still fails.  For example, on arm, the
> failure pathway causes this warning:
> 
> *** WARNING: renaming "cmath" since importing it failed: .../cmath.so:
>    wrong ELF class: ELFCLASS32
> 
> This patch to setup.py and the associated recipe changes allow the
> whole 'import check' logic to be skipped when cross-compiling.
> 
> Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
> ---
> .../python/setup_py_skip_cross_import_check.patch  |   27 ++++++++++++++++++++
> meta/recipes-devtools/python/python_2.7.2.bb       |    5 +++
> 2 files changed, 32 insertions(+), 0 deletions(-)
> create mode 100644 meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch
> 
> diff --git a/meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch
> new file mode 100644
> index 0000000..6ccdb94
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch
> @@ -0,0 +1,27 @@
> +This patch skips over the 'import check' setup.py does when building
> +extensions.  This generally won't work when cross-compiling.
> +
> +Upstream-Status: Inappropriate [embedded-specific]
> +
> +Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
> +
> +Index: Python-2.7.2/setup.py
> +===================================================================
> +--- Python-2.7.2.orig/setup.py	2011-11-04 16:46:34.553796410 -0500
> ++++ Python-2.7.2/setup.py	2011-11-04 16:59:49.692802313 -0500
> +@@ -287,6 +287,15 @@
> +                           (ext.name, sys.exc_info()[1]))
> +             self.failed.append(ext.name)
> +             return
> ++
> ++        # If we're cross-compiling, we want to skip the import check
> ++        # i.e. we shouldn't be dynamically loading target shared libs
> ++        if os.environ.get('CROSS_COMPILE') is not None:
> ++            self.announce(
> ++                'WARNING: skipping import check for cross-compiled "%s"' %
> ++                ext.name)
> ++            return
> ++
> +         # Workaround for Mac OS X: The Carbon-based modules cannot be
> +         # reliably imported into a command-line Python
> +         if 'Carbon' in ext.extra_link_args:
> diff --git a/meta/recipes-devtools/python/python_2.7.2.bb b/meta/recipes-devtools/python/python_2.7.2.bb
> index bb088a4..360e64e 100644
> --- a/meta/recipes-devtools/python/python_2.7.2.bb
> +++ b/meta/recipes-devtools/python/python_2.7.2.bb
> @@ -18,6 +18,7 @@ SRC_URI += "\
>   file://multilib.patch \
>   file://cgi_py.patch \
>   file://remove_sqlite_rpath.patch \
> +  file://setup_py_skip_cross_import_check.patch \
> "
> 
> S = "${WORKDIR}/Python-${PV}"
> @@ -57,6 +58,8 @@ do_compile() {
> 	# then call do_install twice we get Makefile.orig == Makefile.sysroot
> 	install -m 0644 Makefile Makefile.sysroot
> 
> +	export CROSS_COMPILE="${TARGET_PREFIX}"
> +
> 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
> 		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \
> 		STAGING_LIBDIR=${STAGING_LIBDIR} \
> @@ -78,6 +81,8 @@ do_install() {
> 	# make install needs the original Makefile, or otherwise the inclues would
> 	# go to ${D}${STAGING...}/...
> 	install -m 0644 Makefile.orig Makefile
> +
> +	export CROSS_COMPILE="${TARGET_PREFIX}"
> 	
> 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
> 		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \
> -- 
> 1.7.6.4
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Dexuan Cui - Nov. 9, 2011, 10:34 a.m.
How I wish I could notice the patch this morning so I could save 1 day!

Anyway, thanks TomZ for the patch!  :-)

Thanks,
-- Dexuan


> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> Of Saul Wold
> Sent: Wednesday, November 09, 2011 3:23 AM
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [CONSOLIDATED PULL 17/17] python: skip setup.py
> 'import check' when cross-compiling
> 
> From: Tom Zanussi <tom.zanussi@intel.com>
> 
> build_extension() in setup.py, as part of the build process, does an
> 'import check' on the built extension.  The import check in turn
> dlopen()'s the shared library associated with the extension, which
> isn't something that makes sense if that library was cross-compiled
> for a different architecture.
> 
> This was noticed with an x86_64 target that was compiled with avx
> support, because it caused 'illegal instruction' exceptions:
> 
> | /bin/sh: line 1: 14575 Illegal instruction ... -E ./setup.py -q build
> 
> For other target architectures, it doesn't necessarily cause illegal
> instruction exceptions, but still fails.  For example, on arm, the
> failure pathway causes this warning:
> 
> *** WARNING: renaming "cmath" since importing it failed: .../cmath.so:
>     wrong ELF class: ELFCLASS32
> 
> This patch to setup.py and the associated recipe changes allow the
> whole 'import check' logic to be skipped when cross-compiling.
> 
> Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
> ---
>  .../python/setup_py_skip_cross_import_check.patch  |   27
> ++++++++++++++++++++
>  meta/recipes-devtools/python/python_2.7.2.bb       |    5 +++
>  2 files changed, 32 insertions(+), 0 deletions(-)
>  create mode 100644
> meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.
> patch
> 
> diff --git
> a/meta/recipes-devtools/python/python/setup_py_skip_cross_import_chec
> k.patch
> b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_chec
> k.patch
> new file mode 100644
> index 0000000..6ccdb94
> --- /dev/null
> +++
> b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_chec
> k.patch
> @@ -0,0 +1,27 @@
> +This patch skips over the 'import check' setup.py does when building
> +extensions.  This generally won't work when cross-compiling.
> +
> +Upstream-Status: Inappropriate [embedded-specific]
> +
> +Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
> +
> +Index: Python-2.7.2/setup.py
> +==========================================================
> =========
> +--- Python-2.7.2.orig/setup.py	2011-11-04 16:46:34.553796410 -0500
> ++++ Python-2.7.2/setup.py	2011-11-04 16:59:49.692802313 -0500
> +@@ -287,6 +287,15 @@
> +                           (ext.name, sys.exc_info()[1]))
> +             self.failed.append(ext.name)
> +             return
> ++
> ++        # If we're cross-compiling, we want to skip the import check
> ++        # i.e. we shouldn't be dynamically loading target shared libs
> ++        if os.environ.get('CROSS_COMPILE') is not None:
> ++            self.announce(
> ++                'WARNING: skipping import check for cross-compiled
> "%s"' %
> ++                ext.name)
> ++            return
> ++
> +         # Workaround for Mac OS X: The Carbon-based modules cannot
> be
> +         # reliably imported into a command-line Python
> +         if 'Carbon' in ext.extra_link_args:
> diff --git a/meta/recipes-devtools/python/python_2.7.2.bb
> b/meta/recipes-devtools/python/python_2.7.2.bb
> index bb088a4..360e64e 100644
> --- a/meta/recipes-devtools/python/python_2.7.2.bb
> +++ b/meta/recipes-devtools/python/python_2.7.2.bb
> @@ -18,6 +18,7 @@ SRC_URI += "\
>    file://multilib.patch \
>    file://cgi_py.patch \
>    file://remove_sqlite_rpath.patch \
> +  file://setup_py_skip_cross_import_check.patch \
>  "
> 
>  S = "${WORKDIR}/Python-${PV}"
> @@ -57,6 +58,8 @@ do_compile() {
>  	# then call do_install twice we get Makefile.orig == Makefile.sysroot
>  	install -m 0644 Makefile Makefile.sysroot
> 
> +	export CROSS_COMPILE="${TARGET_PREFIX}"
> +
>  	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
>  		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \
>  		STAGING_LIBDIR=${STAGING_LIBDIR} \
> @@ -78,6 +81,8 @@ do_install() {
>  	# make install needs the original Makefile, or otherwise the inclues
> would
>  	# go to ${D}${STAGING...}/...
>  	install -m 0644 Makefile.orig Makefile
> +
> +	export CROSS_COMPILE="${TARGET_PREFIX}"
> 
>  	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
>  		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \
> --
> 1.7.6.4
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Tom Zanussi - Nov. 9, 2011, 1:42 p.m.
On Wed, 2011-11-09 at 02:34 -0800, Cui, Dexuan wrote:
> How I wish I could notice the patch this morning so I could save 1 day!
> 

??? Can you please explain what you mean?

> Anyway, thanks TomZ for the patch!  :-)
> 

You're welcome, I guess!

Tom

> Thanks,
> -- Dexuan
> 
> 
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > Of Saul Wold
> > Sent: Wednesday, November 09, 2011 3:23 AM
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [CONSOLIDATED PULL 17/17] python: skip setup.py
> > 'import check' when cross-compiling
> > 
> > From: Tom Zanussi <tom.zanussi@intel.com>
> > 
> > build_extension() in setup.py, as part of the build process, does an
> > 'import check' on the built extension.  The import check in turn
> > dlopen()'s the shared library associated with the extension, which
> > isn't something that makes sense if that library was cross-compiled
> > for a different architecture.
> > 
> > This was noticed with an x86_64 target that was compiled with avx
> > support, because it caused 'illegal instruction' exceptions:
> > 
> > | /bin/sh: line 1: 14575 Illegal instruction ... -E ./setup.py -q build
> > 
> > For other target architectures, it doesn't necessarily cause illegal
> > instruction exceptions, but still fails.  For example, on arm, the
> > failure pathway causes this warning:
> > 
> > *** WARNING: renaming "cmath" since importing it failed: .../cmath.so:
> >     wrong ELF class: ELFCLASS32
> > 
> > This patch to setup.py and the associated recipe changes allow the
> > whole 'import check' logic to be skipped when cross-compiling.
> > 
> > Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
> > ---
> >  .../python/setup_py_skip_cross_import_check.patch  |   27
> > ++++++++++++++++++++
> >  meta/recipes-devtools/python/python_2.7.2.bb       |    5 +++
> >  2 files changed, 32 insertions(+), 0 deletions(-)
> >  create mode 100644
> > meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.
> > patch
> > 
> > diff --git
> > a/meta/recipes-devtools/python/python/setup_py_skip_cross_import_chec
> > k.patch
> > b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_chec
> > k.patch
> > new file mode 100644
> > index 0000000..6ccdb94
> > --- /dev/null
> > +++
> > b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_chec
> > k.patch
> > @@ -0,0 +1,27 @@
> > +This patch skips over the 'import check' setup.py does when building
> > +extensions.  This generally won't work when cross-compiling.
> > +
> > +Upstream-Status: Inappropriate [embedded-specific]
> > +
> > +Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
> > +
> > +Index: Python-2.7.2/setup.py
> > +==========================================================
> > =========
> > +--- Python-2.7.2.orig/setup.py	2011-11-04 16:46:34.553796410 -0500
> > ++++ Python-2.7.2/setup.py	2011-11-04 16:59:49.692802313 -0500
> > +@@ -287,6 +287,15 @@
> > +                           (ext.name, sys.exc_info()[1]))
> > +             self.failed.append(ext.name)
> > +             return
> > ++
> > ++        # If we're cross-compiling, we want to skip the import check
> > ++        # i.e. we shouldn't be dynamically loading target shared libs
> > ++        if os.environ.get('CROSS_COMPILE') is not None:
> > ++            self.announce(
> > ++                'WARNING: skipping import check for cross-compiled
> > "%s"' %
> > ++                ext.name)
> > ++            return
> > ++
> > +         # Workaround for Mac OS X: The Carbon-based modules cannot
> > be
> > +         # reliably imported into a command-line Python
> > +         if 'Carbon' in ext.extra_link_args:
> > diff --git a/meta/recipes-devtools/python/python_2.7.2.bb
> > b/meta/recipes-devtools/python/python_2.7.2.bb
> > index bb088a4..360e64e 100644
> > --- a/meta/recipes-devtools/python/python_2.7.2.bb
> > +++ b/meta/recipes-devtools/python/python_2.7.2.bb
> > @@ -18,6 +18,7 @@ SRC_URI += "\
> >    file://multilib.patch \
> >    file://cgi_py.patch \
> >    file://remove_sqlite_rpath.patch \
> > +  file://setup_py_skip_cross_import_check.patch \
> >  "
> > 
> >  S = "${WORKDIR}/Python-${PV}"
> > @@ -57,6 +58,8 @@ do_compile() {
> >  	# then call do_install twice we get Makefile.orig == Makefile.sysroot
> >  	install -m 0644 Makefile Makefile.sysroot
> > 
> > +	export CROSS_COMPILE="${TARGET_PREFIX}"
> > +
> >  	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
> >  		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \
> >  		STAGING_LIBDIR=${STAGING_LIBDIR} \
> > @@ -78,6 +81,8 @@ do_install() {
> >  	# make install needs the original Makefile, or otherwise the inclues
> > would
> >  	# go to ${D}${STAGING...}/...
> >  	install -m 0644 Makefile.orig Makefile
> > +
> > +	export CROSS_COMPILE="${TARGET_PREFIX}"
> > 
> >  	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
> >  		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \
> > --
> > 1.7.6.4
> > 
> > 
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Dexuan Cui - Nov. 10, 2011, 12:57 a.m.
Zanussi, Tom wrote on 2011-11-09:
> On Wed, 2011-11-09 at 02:34 -0800, Cui, Dexuan wrote:
>> How I wish I could notice the patch this morning so I could save 1 day!
>> 
> 
> ??? Can you please explain what you mean?
I meant I spent 1 day on debugging the same issue and got the cause but wasn't sure how the patch should be made, so I wanted to ask for suggestion in the ML and before that I tried to find out if anybody reported the same issue and I saw your patch that was sent 1.5 days ago... :-)

>> Anyway, thanks TomZ for the patch!  :-)
>> 
> 
> You're welcome, I guess!
> 
> Tom
> 

Thanks,
-- Dexuan
Tom Zanussi - Nov. 10, 2011, 1:06 a.m.
On Wed, 2011-11-09 at 16:57 -0800, Cui, Dexuan wrote:
> Zanussi, Tom wrote on 2011-11-09:
> > On Wed, 2011-11-09 at 02:34 -0800, Cui, Dexuan wrote:
> >> How I wish I could notice the patch this morning so I could save 1 day!
> >> 
> > 
> > ??? Can you please explain what you mean?
> I meant I spent 1 day on debugging the same issue and got the cause but wasn't sure how the patch should be made, so I wanted to ask for suggestion in the ML and before that I tried to find out if anybody reported the same issue and I saw your patch that was sent 1.5 days ago... :-)
> 

Yeah, sure - I'm just kind of curious since I'd never seen it before
until I added the avx instruction support.

Just out of curiosity, what were your host and target arches?

Tom 

> >> Anyway, thanks TomZ for the patch!  :-)
> >> 
> > 
> > You're welcome, I guess!
> > 
> > Tom
> > 
> 
> Thanks,
> -- Dexuan
> 
>
Dexuan Cui - Nov. 10, 2011, 1:25 a.m.
Zanussi, Tom wrote on 2011-11-10:
> On Wed, 2011-11-09 at 16:57 -0800, Cui, Dexuan wrote:
>> Zanussi, Tom wrote on 2011-11-09:
>>> On Wed, 2011-11-09 at 02:34 -0800, Cui, Dexuan wrote:
>>>> How I wish I could notice the patch this morning so I could save 1 day!
>>>> 
>>> 
>>> ??? Can you please explain what you mean?
>> I meant I spent 1 day on debugging the same issue and got the cause
>> but wasn't sure how the patch should be made, so I wanted to ask for
>> suggestion in the ML and before that I tried to find out if anybody
>> reported the same issue and I saw your patch that was sent 1.5 days
>> ago... :-)
>> 
> 
> Yeah, sure - I'm just kind of curious since I'd never seen it before
> until I added the avx instruction support.
Actually I think the issue is always there(at least from when python was upgrade to 2.7.x).
e.g., in target, run "python" and try "import grp", we'll get an error saying the built-in module grp couldn't be found; in python's log.do_compile, we can see the obvious warnings.

> 
> Just out of curiosity, what were your host and target arches?
I'm trying the task "self hosted image"(https://wiki.yoctoproject.org/wiki/Build_Appliance_Design):
in target, I found bitbake's do_package failed because "import grp" failed, so I had to look into the issue.

Thanks,
-- Dexuan

Patch

diff --git a/meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch
new file mode 100644
index 0000000..6ccdb94
--- /dev/null
+++ b/meta/recipes-devtools/python/python/setup_py_skip_cross_import_check.patch
@@ -0,0 +1,27 @@ 
+This patch skips over the 'import check' setup.py does when building
+extensions.  This generally won't work when cross-compiling.
+
+Upstream-Status: Inappropriate [embedded-specific]
+
+Signed-off-by: Tom Zanussi <tom.zanussi@intel.com>
+
+Index: Python-2.7.2/setup.py
+===================================================================
+--- Python-2.7.2.orig/setup.py	2011-11-04 16:46:34.553796410 -0500
++++ Python-2.7.2/setup.py	2011-11-04 16:59:49.692802313 -0500
+@@ -287,6 +287,15 @@
+                           (ext.name, sys.exc_info()[1]))
+             self.failed.append(ext.name)
+             return
++
++        # If we're cross-compiling, we want to skip the import check
++        # i.e. we shouldn't be dynamically loading target shared libs
++        if os.environ.get('CROSS_COMPILE') is not None:
++            self.announce(
++                'WARNING: skipping import check for cross-compiled "%s"' %
++                ext.name)
++            return
++
+         # Workaround for Mac OS X: The Carbon-based modules cannot be
+         # reliably imported into a command-line Python
+         if 'Carbon' in ext.extra_link_args:
diff --git a/meta/recipes-devtools/python/python_2.7.2.bb b/meta/recipes-devtools/python/python_2.7.2.bb
index bb088a4..360e64e 100644
--- a/meta/recipes-devtools/python/python_2.7.2.bb
+++ b/meta/recipes-devtools/python/python_2.7.2.bb
@@ -18,6 +18,7 @@  SRC_URI += "\
   file://multilib.patch \
   file://cgi_py.patch \
   file://remove_sqlite_rpath.patch \
+  file://setup_py_skip_cross_import_check.patch \
 "
 
 S = "${WORKDIR}/Python-${PV}"
@@ -57,6 +58,8 @@  do_compile() {
 	# then call do_install twice we get Makefile.orig == Makefile.sysroot
 	install -m 0644 Makefile Makefile.sysroot
 
+	export CROSS_COMPILE="${TARGET_PREFIX}"
+
 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
 		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \
 		STAGING_LIBDIR=${STAGING_LIBDIR} \
@@ -78,6 +81,8 @@  do_install() {
 	# make install needs the original Makefile, or otherwise the inclues would
 	# go to ${D}${STAGING...}/...
 	install -m 0644 Makefile.orig Makefile
+
+	export CROSS_COMPILE="${TARGET_PREFIX}"
 	
 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/pgen \
 		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python \