Patchwork [RFC,v1,00/16] populate perl-native into its own directory

login
register
mail settings
Submitter Dexuan Cui
Date June 1, 2011, 1:18 p.m.
Message ID <cover.1306931844.git.dexuan.cui@intel.com>
Download mbox
Permalink /patch/5203/
State New, archived
Headers show

Pull-request

git://git.pokylinux.org/poky-contrib dcui/master_perl-native

Comments

Dexuan Cui - June 1, 2011, 1:18 p.m.
Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and
perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in
the PATH when bitbake is running.
This can cause some race conditions: many places detecting perl from PATH
can't make sure which path will be used as this depends on when perl-native's
populate_sysroot is finished, e.g., automake-native and autoconf-native could
use perl-native-runtime while gnu-config-native could use perl-native and
this inconsistent usages can cause trouble, e.g., bug 941.

And, as RP suggested, "the time to use perl-native instead of
perl-native-runtime is where any perl module is being built, perl itself is
being built or anything that has an explicit dependency on the perl version
present".

So I made the following changes to try to address the aboves issues:
1) populate perl-native into its own directory so it won't appear in PATH
by default, and add perlnative.bbclass for these recipes that really depend
on perl-native;
2) check all perl-native references and correct the ones that should be
perl-native-runtime;
3) fix any building issues due to the new location of perl-native,
especially cpan and cpan-base .bbclass.

With the changes, bug 941 doesn't appear.

Tests I did are:
I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and
x86_64 Ubuntu hosts and everything seems building fine.


Please review the changes and comment on them. Thanks!

--------------------------

The following changes since commit 0736b136efac283449c9218c01740eba4a7bc8cc:

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib dcui/master_perl-native
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master_perl-native

Dexuan Cui (16):
  native.bbclass: allow a native package to be populated into its own
    dir
  perl-native: populate into its own dir
  perlnative.bbclass: add the file
  gnu-config-native: should depend on perl-native-runtime rather than
    perl-native
  libcap: should depend on perl-native-runtime rather than perl-native
  openssl: should depend on perl-native-runtime rather than perl-native
  git: should depend on perl-native-runtime rather than perl-native
  coreutils: remove unnecessary dependency on perl
  dpkg: should depend on perl-native-runtime rather than perl-native
  webkit-gtk: should depend on perl-native-runtime rather than
    perl-native
  perl: inherit perlnative
  cpan.bbclass, cpan-base.bbclas: update them for the perlnative change
  libxml-parser-perl: inherit perlnative
  libconvert-asn1-perl: fix EXTRA_PERLFLAGS due to the perl-native
    change
  libxml-simple-perl: fix EXTRA_PERLFLAGS due the the perlnative change
  icon-naming-utils-native: inherit perlnative

 meta/classes/cpan-base.bbclass                     |   10 +++++++---
 meta/classes/cpan.bbclass                          |   12 ++++++------
 meta/classes/native.bbclass                        |    5 +++++
 meta/classes/perlnative.bbclass                    |    2 ++
 meta/recipes-connectivity/openssl/openssl.inc      |    2 +-
 .../recipes-connectivity/openssl/openssl_0.9.8r.bb |    2 +-
 meta/recipes-core/coreutils/coreutils_6.9.bb       |    6 +++---
 meta/recipes-core/coreutils/coreutils_8.9.bb       |    6 +++---
 meta/recipes-devtools/dpkg/dpkg.inc                |    2 +-
 meta/recipes-devtools/dpkg/dpkg_1.15.8.7.bb        |    2 +-
 meta/recipes-devtools/git/git.inc                  |    4 ++--
 meta/recipes-devtools/git/git_1.7.5.1.bb           |    2 +-
 .../gnu-config/gnu-config_20080123.bb              |    4 ++--
 .../icon-naming-utils-native_0.8.7.bb              |    4 ++--
 .../perl/libxml-parser-perl-native_2.40.bb         |    2 +-
 .../perl/libxml-parser-perl_2.40.bb                |    2 +-
 .../perl/libxml-simple-perl-native_2.18.bb         |    2 +-
 .../perl/libxml-simple-perl_2.18.bb                |    4 ++--
 meta/recipes-devtools/perl/perl-native_5.12.3.bb   |    5 ++++-
 meta/recipes-devtools/perl/perl_5.12.3.bb          |   10 +++++-----
 .../perl/libconvert-asn1-perl_0.22.bb              |    4 ++--
 meta/recipes-sato/webkit/webkit-gtk_svn.bb         |    6 +++---
 meta/recipes-support/libcap/libcap.inc             |    4 ++--
 meta/recipes-support/libcap/libcap_2.20.bb         |    2 +-
 24 files changed, 59 insertions(+), 45 deletions(-)
 create mode 100644 meta/classes/perlnative.bbclass
Richard Purdie - June 1, 2011, 1:38 p.m.
Hi Dexuan,

On Wed, 2011-06-01 at 21:18 +0800, Dexuan Cui wrote:
> Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and
> perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in
> the PATH when bitbake is running.
> This can cause some race conditions: many places detecting perl from PATH
> can't make sure which path will be used as this depends on when perl-native's
> populate_sysroot is finished, e.g., automake-native and autoconf-native could
> use perl-native-runtime while gnu-config-native could use perl-native and
> this inconsistent usages can cause trouble, e.g., bug 941.
> 
> And, as RP suggested, "the time to use perl-native instead of
> perl-native-runtime is where any perl module is being built, perl itself is
> being built or anything that has an explicit dependency on the perl version
> present".
> 
> So I made the following changes to try to address the aboves issues:
> 1) populate perl-native into its own directory so it won't appear in PATH
> by default, and add perlnative.bbclass for these recipes that really depend
> on perl-native;
> 2) check all perl-native references and correct the ones that should be
> perl-native-runtime;
> 3) fix any building issues due to the new location of perl-native,
> especially cpan and cpan-base .bbclass.
> 
> With the changes, bug 941 doesn't appear.
> 
> Tests I did are:
> I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and
> x86_64 Ubuntu hosts and everything seems building fine.
> 
> 
> Please review the changes and comment on them. Thanks!

I had a look through the series and it looks good to me. Hopefully this
addresses the perl issues people have been seeing once and for all (and
unclogs the dependencies a little) :)

I'm going to leave this on the mailing list for a while to give others a
chance to review it though.

Cheers,

Richard
Tom Rini - June 1, 2011, 5:17 p.m.
On 06/01/2011 06:18 AM, Dexuan Cui wrote:
> Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and
> perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in
> the PATH when bitbake is running.
> This can cause some race conditions: many places detecting perl from PATH
> can't make sure which path will be used as this depends on when perl-native's
> populate_sysroot is finished, e.g., automake-native and autoconf-native could
> use perl-native-runtime while gnu-config-native could use perl-native and
> this inconsistent usages can cause trouble, e.g., bug 941.
> 
> And, as RP suggested, "the time to use perl-native instead of
> perl-native-runtime is where any perl module is being built, perl itself is
> being built or anything that has an explicit dependency on the perl version
> present".
> 
> So I made the following changes to try to address the aboves issues:
> 1) populate perl-native into its own directory so it won't appear in PATH
> by default, and add perlnative.bbclass for these recipes that really depend
> on perl-native;
> 2) check all perl-native references and correct the ones that should be
> perl-native-runtime;
> 3) fix any building issues due to the new location of perl-native,
> especially cpan and cpan-base .bbclass.
> 
> With the changes, bug 941 doesn't appear.
> 
> Tests I did are:
> I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and
> x86_64 Ubuntu hosts and everything seems building fine.

How does this work (by which I mean, test some please :)) with meta-oe
where we have (or will have soon) a lot of perl module recipes?  My
concern is that we've just moved the race around a bit and we'll hit
this somewhere else where we both really need "perl-native" (since we
need some cpan stuff we've built) and also mangle in the perl we found
in PATH into some scripts...
Richard Purdie - June 1, 2011, 5:56 p.m.
On Wed, 2011-06-01 at 10:17 -0700, Tom Rini wrote:
> On 06/01/2011 06:18 AM, Dexuan Cui wrote:
> > Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and
> > perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in
> > the PATH when bitbake is running.
> > This can cause some race conditions: many places detecting perl from PATH
> > can't make sure which path will be used as this depends on when perl-native's
> > populate_sysroot is finished, e.g., automake-native and autoconf-native could
> > use perl-native-runtime while gnu-config-native could use perl-native and
> > this inconsistent usages can cause trouble, e.g., bug 941.
> > 
> > And, as RP suggested, "the time to use perl-native instead of
> > perl-native-runtime is where any perl module is being built, perl itself is
> > being built or anything that has an explicit dependency on the perl version
> > present".
> > 
> > So I made the following changes to try to address the aboves issues:
> > 1) populate perl-native into its own directory so it won't appear in PATH
> > by default, and add perlnative.bbclass for these recipes that really depend
> > on perl-native;
> > 2) check all perl-native references and correct the ones that should be
> > perl-native-runtime;
> > 3) fix any building issues due to the new location of perl-native,
> > especially cpan and cpan-base .bbclass.
> > 
> > With the changes, bug 941 doesn't appear.
> > 
> > Tests I did are:
> > I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and
> > x86_64 Ubuntu hosts and everything seems building fine.
> 
> How does this work (by which I mean, test some please :)) with meta-oe
> where we have (or will have soon) a lot of perl module recipes?  My
> concern is that we've just moved the race around a bit and we'll hit
> this somewhere else where we both really need "perl-native" (since we
> need some cpan stuff we've built) and also mangle in the perl we found
> in PATH into some scripts...

Anything building or using perl modules should be inheriting the
perlnative class. This adds the dependency on perl-native and the
appropriate PATH entry. Where is the race?

Cheers,

Richard
Tom Rini - June 1, 2011, 7:39 p.m.
On 06/01/2011 10:56 AM, Richard Purdie wrote:
> On Wed, 2011-06-01 at 10:17 -0700, Tom Rini wrote:
>> On 06/01/2011 06:18 AM, Dexuan Cui wrote:
>>> Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and
>>> perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in
>>> the PATH when bitbake is running.
>>> This can cause some race conditions: many places detecting perl from PATH
>>> can't make sure which path will be used as this depends on when perl-native's
>>> populate_sysroot is finished, e.g., automake-native and autoconf-native could
>>> use perl-native-runtime while gnu-config-native could use perl-native and
>>> this inconsistent usages can cause trouble, e.g., bug 941.
>>>
>>> And, as RP suggested, "the time to use perl-native instead of
>>> perl-native-runtime is where any perl module is being built, perl itself is
>>> being built or anything that has an explicit dependency on the perl version
>>> present".
>>>
>>> So I made the following changes to try to address the aboves issues:
>>> 1) populate perl-native into its own directory so it won't appear in PATH
>>> by default, and add perlnative.bbclass for these recipes that really depend
>>> on perl-native;
>>> 2) check all perl-native references and correct the ones that should be
>>> perl-native-runtime;
>>> 3) fix any building issues due to the new location of perl-native,
>>> especially cpan and cpan-base .bbclass.
>>>
>>> With the changes, bug 941 doesn't appear.
>>>
>>> Tests I did are:
>>> I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and
>>> x86_64 Ubuntu hosts and everything seems building fine.
>>
>> How does this work (by which I mean, test some please :)) with meta-oe
>> where we have (or will have soon) a lot of perl module recipes?  My
>> concern is that we've just moved the race around a bit and we'll hit
>> this somewhere else where we both really need "perl-native" (since we
>> need some cpan stuff we've built) and also mangle in the perl we found
>> in PATH into some scripts...
> 
> Anything building or using perl modules should be inheriting the
> perlnative class. This adds the dependency on perl-native and the
> appropriate PATH entry. Where is the race?

Maybe race isn't quite the right word.  But recipe A depends on
lib$something-perl-native, and brings in perl-native.  It also checks
for perl in its auto-foo and finds our perl.  It now also uses our perl
when it wants a host perl and all of the potential bad things happen, yes?
Phil Blundell - June 1, 2011, 8:05 p.m.
On Wed, 2011-06-01 at 12:39 -0700, Tom Rini wrote:
> Maybe race isn't quite the right word.  But recipe A depends on
> lib$something-perl-native, and brings in perl-native.  It also checks
> for perl in its auto-foo and finds our perl.  It now also uses our perl
> when it wants a host perl and all of the potential bad things happen, yes?

What are the circumstances in which it would want a host perl?  I don't
quite understand the issue here.  Surely anything that the host perl can
do, perl-native plus some combination of libfoo-perl-native can also do,
right?

p.
Tom Rini - June 1, 2011, 8:42 p.m.
On 06/01/2011 01:05 PM, Phil Blundell wrote:
> On Wed, 2011-06-01 at 12:39 -0700, Tom Rini wrote:
>> Maybe race isn't quite the right word.  But recipe A depends on
>> lib$something-perl-native, and brings in perl-native.  It also checks
>> for perl in its auto-foo and finds our perl.  It now also uses our perl
>> when it wants a host perl and all of the potential bad things happen, yes?
> 
> What are the circumstances in which it would want a host perl?  I don't
> quite understand the issue here.  Surely anything that the host perl can
> do, perl-native plus some combination of libfoo-perl-native can also do,
> right?

So, here's the two things going on:
- In some cases (libfoo-perl-native) we need to install various perl
modules in order to build other target apps.  This is the cpan case,
iirc.  We don't currently (and can't?) just play whatever games perl
would like to do to use system-wide perl and a local to TMPDIR cpan
directory.  We instead go down the path of having perl-native.
- In order to build perl for the target we need to have the same perl
version available for use.

What we do in oe-core (and oe.dev, historically) is provide perl-native
for both of these cases.  What falls down in this case is that  once
perl-native is built (and in our PATH), if it's a different version than
system-wide perl, stuff starts failing on version mis-match.  The patch
series here fixes that by saying stuff must opt-in to having perl-native
be in PATH rather than just system-wide perl.  What I'm saying is that
while in oe-core nothing falls down here (since nothing that gets
perl-native also tries to just run 'perl') meta-oe is or will have failures.

Unfortunately I don't have the test cases that drove me to make oe.dev
like it is handy but I have a feeling that if we go down this path it
won't be the last we see of the problem, but it might well pop up again.

There is another option here, which is to make perl-native be
perl-cross, live in its own special directory and figure out how to make
a TMPDIR-specific cpan repository available for use by system-wide perl.
Phil Blundell - June 1, 2011, 8:45 p.m.
On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote:
> What falls down in this case is that  once
> perl-native is built (and in our PATH), if it's a different version than
> system-wide perl, stuff starts failing on version mis-match.

I think that's the bit that I'm not properly understanding.  Which
versions are mismatching, exactly?  

Surely the local perl from the sysroot ought to be completely
self-contained and shouldn't be using any bits from the host perl
install at that point.

p.
Tom Rini - June 1, 2011, 8:59 p.m.
On 06/01/2011 01:45 PM, Phil Blundell wrote:
> On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote:
>> What falls down in this case is that  once
>> perl-native is built (and in our PATH), if it's a different version than
>> system-wide perl, stuff starts failing on version mis-match.
> 
> I think that's the bit that I'm not properly understanding.  Which
> versions are mismatching, exactly?  
> 
> Surely the local perl from the sysroot ought to be completely
> self-contained and shouldn't be using any bits from the host perl
> install at that point.

So this jogs my memory a bit!  It's not so much perl itself but stuff
that uses perl that can get dirty and then no, you have stuff thats
built for system perl and stuff that's built with perl-native clashing.

Relying even more on memory, I think help2man was one of the "easy"
culprits and since we also modify the env, we do things like have
help2man run with PERL5LIB and so on pointing system-wide perl at
perl-native's lib directory and so forth.
Dexuan Cui - June 2, 2011, 1:46 p.m.
Tom Rini wrote:
> On 06/01/2011 01:05 PM, Phil Blundell wrote:
>> On Wed, 2011-06-01 at 12:39 -0700, Tom Rini wrote:
>>> Maybe race isn't quite the right word.  But recipe A depends on
>>> lib$something-perl-native, and brings in perl-native.  It also
>>> checks 
>>> for perl in its auto-foo and finds our perl.  It now also uses our
>>> perl when it wants a host perl and all of the potential bad things
>>> happen, yes? 
>> 
>> What are the circumstances in which it would want a host perl?  I
>> don't quite understand the issue here.  Surely anything that the
>> host perl can do, perl-native plus some combination of
>> libfoo-perl-native can also do, right?
> 
> So, here's the two things going on:
> - In some cases (libfoo-perl-native) we need to install various perl
> modules in order to build other target apps.  This is the cpan case,
> iirc.  We don't currently (and can't?) just play whatever games perl
> would like to do to use system-wide perl and a local to TMPDIR cpan
> directory.  We instead go down the path of having perl-native.
> - In order to build perl for the target we need to have the same perl
> version available for use.
> 
> What we do in oe-core (and oe.dev, historically) is provide
> perl-native 
> for both of these cases.  What falls down in this case is that  once
> perl-native is built (and in our PATH), if it's a different version
> than system-wide perl, stuff starts failing on version mis-match. 
> The patch series here fixes that by saying stuff must opt-in to
> having perl-native 
> be in PATH rather than just system-wide perl.  What I'm saying is that
> while in oe-core nothing falls down here (since nothing that gets
> perl-native also tries to just run 'perl') meta-oe is or will have
> failures. 
> 
> Unfortunately I don't have the test cases that drove me to make oe.dev
> like it is handy but I have a feeling that if we go down this path it
> won't be the last we see of the problem, but it might well pop up
> again. 
> 
> There is another option here, which is to make perl-native be
> perl-cross, live in its own special directory and figure out how to
> make 
> a TMPDIR-specific cpan repository available for use by system-wide
> perl. 

Hi Tom,
Thank you very much for the detailed explanation!
However I'm actually new to cpan and know few about the history of that in oe.dev, so I'm not sure I know enough about the background to make a proper decision now.
I suppose Richard would comment here. :-)

Thanks,
-- Dexuan
Richard Purdie - June 2, 2011, 2:06 p.m.
On Wed, 2011-06-01 at 13:59 -0700, Tom Rini wrote:
> On 06/01/2011 01:45 PM, Phil Blundell wrote:
> > On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote:
> >> What falls down in this case is that  once
> >> perl-native is built (and in our PATH), if it's a different version than
> >> system-wide perl, stuff starts failing on version mis-match.
> > 
> > I think that's the bit that I'm not properly understanding.  Which
> > versions are mismatching, exactly?  
> > 
> > Surely the local perl from the sysroot ought to be completely
> > self-contained and shouldn't be using any bits from the host perl
> > install at that point.
> 
> So this jogs my memory a bit!  It's not so much perl itself but stuff
> that uses perl that can get dirty and then no, you have stuff thats
> built for system perl and stuff that's built with perl-native clashing.
> 
> Relying even more on memory, I think help2man was one of the "easy"
> culprits and since we also modify the env, we do things like have
> help2man run with PERL5LIB and so on pointing system-wide perl at
> perl-native's lib directory and so forth.

But with the proposed patch series either:

a) help2man depends perlnative.bbclass

In this case it can depend on perl-native being there, its in path and
things work as per OE.dev.

b) help2man doesn't depend in perlnative.bbclass

It only sees the system perl.

So I'm still not clear where the problem is?

Cheers,

Richard
Tom Rini - June 2, 2011, 2:25 p.m.
On 06/02/2011 07:06 AM, Richard Purdie wrote:
> On Wed, 2011-06-01 at 13:59 -0700, Tom Rini wrote:
>> On 06/01/2011 01:45 PM, Phil Blundell wrote:
>>> On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote:
>>>> What falls down in this case is that  once
>>>> perl-native is built (and in our PATH), if it's a different version than
>>>> system-wide perl, stuff starts failing on version mis-match.
>>>
>>> I think that's the bit that I'm not properly understanding.  Which
>>> versions are mismatching, exactly?  
>>>
>>> Surely the local perl from the sysroot ought to be completely
>>> self-contained and shouldn't be using any bits from the host perl
>>> install at that point.
>>
>> So this jogs my memory a bit!  It's not so much perl itself but stuff
>> that uses perl that can get dirty and then no, you have stuff thats
>> built for system perl and stuff that's built with perl-native clashing.
>>
>> Relying even more on memory, I think help2man was one of the "easy"
>> culprits and since we also modify the env, we do things like have
>> help2man run with PERL5LIB and so on pointing system-wide perl at
>> perl-native's lib directory and so forth.
> 
> But with the proposed patch series either:
> 
> a) help2man depends perlnative.bbclass
> 
> In this case it can depend on perl-native being there, its in path and
> things work as per OE.dev.
> 
> b) help2man doesn't depend in perlnative.bbclass
> 
> It only sees the system perl.
> 
> So I'm still not clear where the problem is?

Well, help2man-native is (or needs to be) a dep listed in
autotools.bbclass (since so many things need it, hence why it oe-core
still just has it as a required host utility instead of building it).

But help2man is just the easy/common case.  Heck, it _may_ blow up even
with the host help2man instead of help2man-native, if a recipe uses
system-wide help2man and perlnative.bbclass.  The root problem (again,
from memory) is that since we modify PERL5LIB and so on, when we do
that, we've opened ourselves up for system-wide perl trying to use
perl-native's stuff.
Phil Blundell - June 2, 2011, 2:37 p.m.
On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote:
> But help2man is just the easy/common case.  Heck, it _may_ blow up even
> with the host help2man instead of help2man-native, if a recipe uses
> system-wide help2man and perlnative.bbclass.  The root problem (again,
> from memory) is that since we modify PERL5LIB and so on, when we do
> that, we've opened ourselves up for system-wide perl trying to use
> perl-native's stuff.

Ah right, I think I see what you're getting at now.

If we've got a clean separation between perl-native and host perl,
though, can't we now just eliminate all of that futzing with PERL5LIB in
cpan.bbclass and such like places?  perl-native already knows how to
look in the right places within the sysroot for its modules so there
should be no need for anything else to be overriding it.

p.
Tom Rini - June 2, 2011, 4:28 p.m.
On 06/02/2011 07:37 AM, Phil Blundell wrote:
> On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote:
>> But help2man is just the easy/common case.  Heck, it _may_ blow up even
>> with the host help2man instead of help2man-native, if a recipe uses
>> system-wide help2man and perlnative.bbclass.  The root problem (again,
>> from memory) is that since we modify PERL5LIB and so on, when we do
>> that, we've opened ourselves up for system-wide perl trying to use
>> perl-native's stuff.
> 
> Ah right, I think I see what you're getting at now.
> 
> If we've got a clean separation between perl-native and host perl,
> though, can't we now just eliminate all of that futzing with PERL5LIB in
> cpan.bbclass and such like places?  perl-native already knows how to
> look in the right places within the sysroot for its modules so there
> should be no need for anything else to be overriding it.

Well, my question is, does it, really?  Even if we're using the sstate
cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp
is rm -rf'd) ?  Since we've got a create_wrapper around perl and
perl${PV} it should be I suppose (or is easily added there), but I'd
feel a lot better with some testing of the above case (and the updates
to cpan*bbclass).
Richard Purdie - June 2, 2011, 4:35 p.m.
On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote:
> On 06/02/2011 07:37 AM, Phil Blundell wrote:
> > On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote:
> >> But help2man is just the easy/common case.  Heck, it _may_ blow up even
> >> with the host help2man instead of help2man-native, if a recipe uses
> >> system-wide help2man and perlnative.bbclass.  The root problem (again,
> >> from memory) is that since we modify PERL5LIB and so on, when we do
> >> that, we've opened ourselves up for system-wide perl trying to use
> >> perl-native's stuff.
> > 
> > Ah right, I think I see what you're getting at now.
> > 
> > If we've got a clean separation between perl-native and host perl,
> > though, can't we now just eliminate all of that futzing with PERL5LIB in
> > cpan.bbclass and such like places?  perl-native already knows how to
> > look in the right places within the sysroot for its modules so there
> > should be no need for anything else to be overriding it.
> 
> Well, my question is, does it, really?

If it doesn't it really does need to get fixed. I've not seen any
evidence this has been the problem but it still could be.

>   Even if we're using the sstate
> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp
> is rm -rf'd) ?  Since we've got a create_wrapper around perl and
> perl${PV} it should be I suppose (or is easily added there), but I'd
> feel a lot better with some testing of the above case (and the updates
> to cpan*bbclass).

I only took the perl-native DEPENDS patch on the condition this gets
fixed properly. The patches that are there look to do that, at least for
OE-Core. If there are further issues we're going to have to take them as
they arise as I have an objection to crippling the build dependencies
because perl is broken. Really this could use some TLC from someone with
experience in the area...

Cheers,

Richard
Tom Rini - June 2, 2011, 4:55 p.m.
On 06/02/2011 09:35 AM, Richard Purdie wrote:
> On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote:
>> On 06/02/2011 07:37 AM, Phil Blundell wrote:
>>> On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote:
>>>> But help2man is just the easy/common case.  Heck, it _may_ blow up even
>>>> with the host help2man instead of help2man-native, if a recipe uses
>>>> system-wide help2man and perlnative.bbclass.  The root problem (again,
>>>> from memory) is that since we modify PERL5LIB and so on, when we do
>>>> that, we've opened ourselves up for system-wide perl trying to use
>>>> perl-native's stuff.
>>>
>>> Ah right, I think I see what you're getting at now.
>>>
>>> If we've got a clean separation between perl-native and host perl,
>>> though, can't we now just eliminate all of that futzing with PERL5LIB in
>>> cpan.bbclass and such like places?  perl-native already knows how to
>>> look in the right places within the sysroot for its modules so there
>>> should be no need for anything else to be overriding it.
>>
>> Well, my question is, does it, really?
> 
> If it doesn't it really does need to get fixed. I've not seen any
> evidence this has been the problem but it still could be.

I agree it needs to be fixed, yes.  One way or another...

>>   Even if we're using the sstate
>> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp
>> is rm -rf'd) ?  Since we've got a create_wrapper around perl and
>> perl${PV} it should be I suppose (or is easily added there), but I'd
>> feel a lot better with some testing of the above case (and the updates
>> to cpan*bbclass).
> 
> I only took the perl-native DEPENDS patch on the condition this gets
> fixed properly. The patches that are there look to do that, at least for
> OE-Core. If there are further issues we're going to have to take them as
> they arise as I have an objection to crippling the build dependencies
> because perl is broken. Really this could use some TLC from someone with
> experience in the area...

Well, I guess I'd boil down what I said above into a request like this
for v3:
- Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB /
PERLHOSTLIB.
- In /scratch/oecore/tmp0 build the images that were built for v1
- In /scratch/oecore/tmp1 build perl-native's full sstate cache.
- Keep the sstate cache from tmp1 and otherwise rm -rf tmp1.
- In /scratch/oecore/tmp2 using the sstate from tmp1, build the same
images that were built for tmp0.

If all of that works, I think we can call this safe enough for now and
possibly even really fixed, while we have someone actively kicking
around the recipes in question.
Richard Purdie - June 9, 2011, 8:04 a.m.
On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote:
> On 06/02/2011 09:35 AM, Richard Purdie wrote:
> > On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote:
> >>   Even if we're using the sstate
> >> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp
> >> is rm -rf'd) ?  Since we've got a create_wrapper around perl and
> >> perl${PV} it should be I suppose (or is easily added there), but I'd
> >> feel a lot better with some testing of the above case (and the updates
> >> to cpan*bbclass).
> > 
> > I only took the perl-native DEPENDS patch on the condition this gets
> > fixed properly. The patches that are there look to do that, at least for
> > OE-Core. If there are further issues we're going to have to take them as
> > they arise as I have an objection to crippling the build dependencies
> > because perl is broken. Really this could use some TLC from someone with
> > experience in the area...
> 
> Well, I guess I'd boil down what I said above into a request like this
> for v3:
> - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB /
> PERLHOSTLIB.

The first three of these are all about the *target* perl location and I
think we still need them due the mess that perl's build system is. With
the patch series in question they won't actually point at perl-native in
the target case and they are only really used for cross compiling
purposes.

PERLHOSTLIB is used by the target perl when cross compiling to find
native .so files. perl-native will always be present at this point and
again, it seems like a valid use case.

Summary is that I don't think perl-native is broken in any way but we do
need those variables.

> - In /scratch/oecore/tmp0 build the images that were built for v1
> - In /scratch/oecore/tmp1 build perl-native's full sstate cache.
> - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1.
> - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same
> images that were built for tmp0.

I'm confused by this test cycle. What do you mean by "build
perl-native's full sstate cache"?

I suspect you've asking for some partial sstate cache to be shared
between two builds?

Put simpler, you probably want:

in tmp0 "bitbake perl-native"
in tmp1, different location to tmp0, "bitbake core-image-sato" but
sharing the same sstate cache

Is that what you're thinking?

FWIW, I've been running locally with the patch series and I think I'd
like to merge it. If there are issues we can address them as and when
they're identified...

Cheers,

Richard
Richard Purdie - June 9, 2011, 8:08 a.m.
On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote:
> On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote:
> > On 06/02/2011 09:35 AM, Richard Purdie wrote:
> > > On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote:
> > >>   Even if we're using the sstate
> > >> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp
> > >> is rm -rf'd) ?  Since we've got a create_wrapper around perl and
> > >> perl${PV} it should be I suppose (or is easily added there), but I'd
> > >> feel a lot better with some testing of the above case (and the updates
> > >> to cpan*bbclass).
> > > 
> > > I only took the perl-native DEPENDS patch on the condition this gets
> > > fixed properly. The patches that are there look to do that, at least for
> > > OE-Core. If there are further issues we're going to have to take them as
> > > they arise as I have an objection to crippling the build dependencies
> > > because perl is broken. Really this could use some TLC from someone with
> > > experience in the area...
> > 
> > Well, I guess I'd boil down what I said above into a request like this
> > for v3:
> > - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB /
> > PERLHOSTLIB.
> 
> The first three of these are all about the *target* perl location and I
> think we still need them due the mess that perl's build system is. With
> the patch series in question they won't actually point at perl-native in
> the target case and they are only really used for cross compiling
> purposes.
> 
> PERLHOSTLIB is used by the target perl when cross compiling to find
> native .so files. perl-native will always be present at this point and
> again, it seems like a valid use case.
> 
> Summary is that I don't think perl-native is broken in any way but we do
> need those variables.
> 
> > - In /scratch/oecore/tmp0 build the images that were built for v1
> > - In /scratch/oecore/tmp1 build perl-native's full sstate cache.
> > - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1.
> > - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same
> > images that were built for tmp0.
> 
> I'm confused by this test cycle. What do you mean by "build
> perl-native's full sstate cache"?
> 
> I suspect you've asking for some partial sstate cache to be shared
> between two builds?
> 
> Put simpler, you probably want:
> 
> in tmp0 "bitbake perl-native"
> in tmp1, different location to tmp0, "bitbake core-image-sato" but
> sharing the same sstate cache

I meant to add that tmp0 should be renamed before this second step so if
there are hardcoded references in any of the sstate packages they
couldn't find anything in tmp0 as it would no longer exist.

> Is that what you're thinking?
> 
> FWIW, I've been running locally with the patch series and I think I'd
> like to merge it. If there are issues we can address them as and when
> they're identified...
> 
> Cheers,
> 
> Richard
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Dexuan Cui - June 9, 2011, 1:51 p.m.
Richard Purdie wrote:
> On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote:
>> On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote:
>>> On 06/02/2011 09:35 AM, Richard Purdie wrote:
>>>> On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote:
>>>>>   Even if we're using the sstate
>>>>> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and
>>>>> /foo/oecore/tmp is rm -rf'd) ?  Since we've got a create_wrapper
>>>>> around perl and perl${PV} it should be I suppose (or is easily
>>>>> added there), but I'd feel a lot better with some testing of the
>>>>> above case (and the updates to cpan*bbclass).
>>>> 
>>>> I only took the perl-native DEPENDS patch on the condition this
>>>> gets fixed properly. The patches that are there look to do that,
>>>> at least for OE-Core. If there are further issues we're going to
>>>> have to take them as they arise as I have an objection to
>>>> crippling the build dependencies because perl is broken. Really
>>>> this could use some TLC from someone with experience in the area...
>>> 
>>> Well, I guess I'd boil down what I said above into a request like
>>> this for v3: - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB /
>>> PERL_ARCHLIB / PERLHOSTLIB.
>> 
>> The first three of these are all about the *target* perl location
>> and I think we still need them due the mess that perl's build system
>> is. With the patch series in question they won't actually point at
>> perl-native in the target case and they are only really used for
>> cross compiling purposes. 
>> 
>> PERLHOSTLIB is used by the target perl when cross compiling to find
>> native .so files. perl-native will always be present at this point
>> and again, it seems like a valid use case.
>> 
>> Summary is that I don't think perl-native is broken in any way but
>> we do need those variables. 
>> 
>>> - In /scratch/oecore/tmp0 build the images that were built for v1
>>> - In /scratch/oecore/tmp1 build perl-native's full sstate cache.
>>> - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1.
>>> - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same
>>> images that were built for tmp0.
>> 
>> I'm confused by this test cycle. What do you mean by "build
>> perl-native's full sstate cache"?
>> 
>> I suspect you've asking for some partial sstate cache to be shared
>> between two builds? 
>> 
>> Put simpler, you probably want:
>> 
>> in tmp0 "bitbake perl-native"
>> in tmp1, different location to tmp0, "bitbake core-image-sato" but
>> sharing the same sstate cache
> 
> I meant to add that tmp0 should be renamed before this second step so
> if there are hardcoded references in any of the sstate packages they
> couldn't find anything in tmp0 as it would no longer exist.
Actually this doesn't work even without my patch series?
i.e., without my patch series,
1) I "bitbake perl-native" in /poky.git/build/;
2) mv /poky.git/build /poky.git/build.bak;
3) in /poky.git/build2/, modify the conf/local.conf to set SSTATE_MIRRORS to point to /poky.git/build.bak/sstate-cache/, and "bitbake sgmlspl-native" would fail in do_compile:
"Can't locate ExtUtils/Command.pm in @INC (@INC contains: /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-"linux/usr/lib/perl/5.12.3"
Is this an existing bug? 

Thanks,
-- Dexuan
Richard Purdie - June 9, 2011, 1:56 p.m.
On Thu, 2011-06-09 at 21:51 +0800, Cui, Dexuan wrote:
> Richard Purdie wrote:
> > On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote: 
> >> I suspect you've asking for some partial sstate cache to be shared
> >> between two builds? 
> >> 
> >> Put simpler, you probably want:
> >> 
> >> in tmp0 "bitbake perl-native"
> >> in tmp1, different location to tmp0, "bitbake core-image-sato" but
> >> sharing the same sstate cache
> > 
> > I meant to add that tmp0 should be renamed before this second step so
> > if there are hardcoded references in any of the sstate packages they
> > couldn't find anything in tmp0 as it would no longer exist.
>
> Actually this doesn't work even without my patch series?
> i.e., without my patch series,
> 1) I "bitbake perl-native" in /poky.git/build/;
> 2) mv /poky.git/build /poky.git/build.bak;
> 3) in /poky.git/build2/, modify the conf/local.conf to set SSTATE_MIRRORS to point to /poky.git/build.bak/sstate-cache/, and "bitbake sgmlspl-native" would fail in do_compile:
> "Can't locate ExtUtils/Command.pm in @INC (@INC contains: /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-"linux/usr/lib/perl/5.12.3"
> Is this an existing bug? 

Looks like it. This did work but it looks like its been broken
somehow :(

Cheers,

Richard
Dexuan Cui - June 10, 2011, 6:13 a.m.
Richard Purdie wrote:
> On Thu, 2011-06-09 at 21:51 +0800, Cui, Dexuan wrote:
>> Richard Purdie wrote:
>>> On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote:
>>>> I suspect you've asking for some partial sstate cache to be shared
>>>> between two builds? 
>>>> 
>>>> Put simpler, you probably want:
>>>> 
>>>> in tmp0 "bitbake perl-native"
>>>> in tmp1, different location to tmp0, "bitbake core-image-sato" but
>>>> sharing the same sstate cache
>>> 
>>> I meant to add that tmp0 should be renamed before this second step
>>> so if there are hardcoded references in any of the sstate packages
>>> they couldn't find anything in tmp0 as it would no longer exist.
>> 
>> Actually this doesn't work even without my patch series?
>> i.e., without my patch series,
>> 1) I "bitbake perl-native" in /poky.git/build/;
>> 2) mv /poky.git/build /poky.git/build.bak;
>> 3) in /poky.git/build2/, modify the conf/local.conf to set
>> SSTATE_MIRRORS to point to /poky.git/build.bak/sstate-cache/, and
>> "bitbake sgmlspl-native" would fail in do_compile: "Can't locate
>> ExtUtils/Command.pm in @INC (@INC contains:
>> /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3
>> /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3
>> /poky.git/build/tmp/sysroots/i686-"linux/usr/lib/perl/5.12.3" Is
>> this an existing bug?     
> 
> Looks like it. This did work but it looks like its been broken
> somehow :(
> 
Hi Richard,
I reported http://bugzilla.yoctoproject.org/show_bug.cgi?id=1157 to track this issue.

Thanks,
-- Dexuan
Tom Rini - June 10, 2011, 2:23 p.m.
On 06/09/2011 01:08 AM, Richard Purdie wrote:
> On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote:
>> On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote:
>>> On 06/02/2011 09:35 AM, Richard Purdie wrote:
>>>> On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote:
>>>>>   Even if we're using the sstate
>>>>> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp
>>>>> is rm -rf'd) ?  Since we've got a create_wrapper around perl and
>>>>> perl${PV} it should be I suppose (or is easily added there), but I'd
>>>>> feel a lot better with some testing of the above case (and the updates
>>>>> to cpan*bbclass).
>>>>
>>>> I only took the perl-native DEPENDS patch on the condition this gets
>>>> fixed properly. The patches that are there look to do that, at least for
>>>> OE-Core. If there are further issues we're going to have to take them as
>>>> they arise as I have an objection to crippling the build dependencies
>>>> because perl is broken. Really this could use some TLC from someone with
>>>> experience in the area...
>>>
>>> Well, I guess I'd boil down what I said above into a request like this
>>> for v3:
>>> - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB /
>>> PERLHOSTLIB.
>>
>> The first three of these are all about the *target* perl location and I
>> think we still need them due the mess that perl's build system is. With
>> the patch series in question they won't actually point at perl-native in
>> the target case and they are only really used for cross compiling
>> purposes.
>>
>> PERLHOSTLIB is used by the target perl when cross compiling to find
>> native .so files. perl-native will always be present at this point and
>> again, it seems like a valid use case.
>>
>> Summary is that I don't think perl-native is broken in any way but we do
>> need those variables.
>>
>>> - In /scratch/oecore/tmp0 build the images that were built for v1
>>> - In /scratch/oecore/tmp1 build perl-native's full sstate cache.
>>> - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1.
>>> - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same
>>> images that were built for tmp0.
>>
>> I'm confused by this test cycle. What do you mean by "build
>> perl-native's full sstate cache"?
>>
>> I suspect you've asking for some partial sstate cache to be shared
>> between two builds?
>>
>> Put simpler, you probably want:
>>
>> in tmp0 "bitbake perl-native"
>> in tmp1, different location to tmp0, "bitbake core-image-sato" but
>> sharing the same sstate cache
> 
> I meant to add that tmp0 should be renamed before this second step so if
> there are hardcoded references in any of the sstate packages they
> couldn't find anything in tmp0 as it would no longer exist.

Yes, this is what I'm asking for.
Tom Rini - June 10, 2011, 2:26 p.m.
On 06/09/2011 01:04 AM, Richard Purdie wrote:
> On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote:
>> On 06/02/2011 09:35 AM, Richard Purdie wrote:
>>> On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote:
>>>>   Even if we're using the sstate
>>>> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp
>>>> is rm -rf'd) ?  Since we've got a create_wrapper around perl and
>>>> perl${PV} it should be I suppose (or is easily added there), but I'd
>>>> feel a lot better with some testing of the above case (and the updates
>>>> to cpan*bbclass).
>>>
>>> I only took the perl-native DEPENDS patch on the condition this gets
>>> fixed properly. The patches that are there look to do that, at least for
>>> OE-Core. If there are further issues we're going to have to take them as
>>> they arise as I have an objection to crippling the build dependencies
>>> because perl is broken. Really this could use some TLC from someone with
>>> experience in the area...
>>
>> Well, I guess I'd boil down what I said above into a request like this
>> for v3:
>> - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB /
>> PERLHOSTLIB.
> 
> The first three of these are all about the *target* perl location and I
> think we still need them due the mess that perl's build system is. With
> the patch series in question they won't actually point at perl-native in
> the target case and they are only really used for cross compiling
> purposes.
> 
> PERLHOSTLIB is used by the target perl when cross compiling to find
> native .so files. perl-native will always be present at this point and
> again, it seems like a valid use case.
> 
> Summary is that I don't think perl-native is broken in any way but we do
> need those variables.

Maybe I'm having a dense morning here, but isn't that the point?  The
combination of perl's build system is messy and if we bring in cpan it
needs not only target perl locations (to dump things into) but may call
'perl' to do things and if we're on perl 5.14 and your host is only 5.10
or 5.12, bad things can happen.  And since cpan.bbclass isn't brought in
by target perl recipe, what now? :)