| Submitter | Dexuan Cui |
|---|---|
| Date | June 1, 2011, 1:18 p.m. |
| Message ID | <cover.1306931844.git.dexuan.cui@intel.com> |
| Download | mbox |
| Permalink | /patch/5203/ |
| State | New, archived |
| Headers | show |
Pull-request
git://git.pokylinux.org/poky-contrib dcui/master_perl-nativeComments
Hi Dexuan, On Wed, 2011-06-01 at 21:18 +0800, Dexuan Cui wrote: > Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and > perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in > the PATH when bitbake is running. > This can cause some race conditions: many places detecting perl from PATH > can't make sure which path will be used as this depends on when perl-native's > populate_sysroot is finished, e.g., automake-native and autoconf-native could > use perl-native-runtime while gnu-config-native could use perl-native and > this inconsistent usages can cause trouble, e.g., bug 941. > > And, as RP suggested, "the time to use perl-native instead of > perl-native-runtime is where any perl module is being built, perl itself is > being built or anything that has an explicit dependency on the perl version > present". > > So I made the following changes to try to address the aboves issues: > 1) populate perl-native into its own directory so it won't appear in PATH > by default, and add perlnative.bbclass for these recipes that really depend > on perl-native; > 2) check all perl-native references and correct the ones that should be > perl-native-runtime; > 3) fix any building issues due to the new location of perl-native, > especially cpan and cpan-base .bbclass. > > With the changes, bug 941 doesn't appear. > > Tests I did are: > I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and > x86_64 Ubuntu hosts and everything seems building fine. > > > Please review the changes and comment on them. Thanks! I had a look through the series and it looks good to me. Hopefully this addresses the perl issues people have been seeing once and for all (and unclogs the dependencies a little) :) I'm going to leave this on the mailing list for a while to give others a chance to review it though. Cheers, Richard
On 06/01/2011 06:18 AM, Dexuan Cui wrote: > Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and > perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in > the PATH when bitbake is running. > This can cause some race conditions: many places detecting perl from PATH > can't make sure which path will be used as this depends on when perl-native's > populate_sysroot is finished, e.g., automake-native and autoconf-native could > use perl-native-runtime while gnu-config-native could use perl-native and > this inconsistent usages can cause trouble, e.g., bug 941. > > And, as RP suggested, "the time to use perl-native instead of > perl-native-runtime is where any perl module is being built, perl itself is > being built or anything that has an explicit dependency on the perl version > present". > > So I made the following changes to try to address the aboves issues: > 1) populate perl-native into its own directory so it won't appear in PATH > by default, and add perlnative.bbclass for these recipes that really depend > on perl-native; > 2) check all perl-native references and correct the ones that should be > perl-native-runtime; > 3) fix any building issues due to the new location of perl-native, > especially cpan and cpan-base .bbclass. > > With the changes, bug 941 doesn't appear. > > Tests I did are: > I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and > x86_64 Ubuntu hosts and everything seems building fine. How does this work (by which I mean, test some please :)) with meta-oe where we have (or will have soon) a lot of perl module recipes? My concern is that we've just moved the race around a bit and we'll hit this somewhere else where we both really need "perl-native" (since we need some cpan stuff we've built) and also mangle in the perl we found in PATH into some scripts...
On Wed, 2011-06-01 at 10:17 -0700, Tom Rini wrote: > On 06/01/2011 06:18 AM, Dexuan Cui wrote: > > Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and > > perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in > > the PATH when bitbake is running. > > This can cause some race conditions: many places detecting perl from PATH > > can't make sure which path will be used as this depends on when perl-native's > > populate_sysroot is finished, e.g., automake-native and autoconf-native could > > use perl-native-runtime while gnu-config-native could use perl-native and > > this inconsistent usages can cause trouble, e.g., bug 941. > > > > And, as RP suggested, "the time to use perl-native instead of > > perl-native-runtime is where any perl module is being built, perl itself is > > being built or anything that has an explicit dependency on the perl version > > present". > > > > So I made the following changes to try to address the aboves issues: > > 1) populate perl-native into its own directory so it won't appear in PATH > > by default, and add perlnative.bbclass for these recipes that really depend > > on perl-native; > > 2) check all perl-native references and correct the ones that should be > > perl-native-runtime; > > 3) fix any building issues due to the new location of perl-native, > > especially cpan and cpan-base .bbclass. > > > > With the changes, bug 941 doesn't appear. > > > > Tests I did are: > > I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and > > x86_64 Ubuntu hosts and everything seems building fine. > > How does this work (by which I mean, test some please :)) with meta-oe > where we have (or will have soon) a lot of perl module recipes? My > concern is that we've just moved the race around a bit and we'll hit > this somewhere else where we both really need "perl-native" (since we > need some cpan stuff we've built) and also mangle in the perl we found > in PATH into some scripts... Anything building or using perl modules should be inheriting the perlnative class. This adds the dependency on perl-native and the appropriate PATH entry. Where is the race? Cheers, Richard
On 06/01/2011 10:56 AM, Richard Purdie wrote: > On Wed, 2011-06-01 at 10:17 -0700, Tom Rini wrote: >> On 06/01/2011 06:18 AM, Dexuan Cui wrote: >>> Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and >>> perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in >>> the PATH when bitbake is running. >>> This can cause some race conditions: many places detecting perl from PATH >>> can't make sure which path will be used as this depends on when perl-native's >>> populate_sysroot is finished, e.g., automake-native and autoconf-native could >>> use perl-native-runtime while gnu-config-native could use perl-native and >>> this inconsistent usages can cause trouble, e.g., bug 941. >>> >>> And, as RP suggested, "the time to use perl-native instead of >>> perl-native-runtime is where any perl module is being built, perl itself is >>> being built or anything that has an explicit dependency on the perl version >>> present". >>> >>> So I made the following changes to try to address the aboves issues: >>> 1) populate perl-native into its own directory so it won't appear in PATH >>> by default, and add perlnative.bbclass for these recipes that really depend >>> on perl-native; >>> 2) check all perl-native references and correct the ones that should be >>> perl-native-runtime; >>> 3) fix any building issues due to the new location of perl-native, >>> especially cpan and cpan-base .bbclass. >>> >>> With the changes, bug 941 doesn't appear. >>> >>> Tests I did are: >>> I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and >>> x86_64 Ubuntu hosts and everything seems building fine. >> >> How does this work (by which I mean, test some please :)) with meta-oe >> where we have (or will have soon) a lot of perl module recipes? My >> concern is that we've just moved the race around a bit and we'll hit >> this somewhere else where we both really need "perl-native" (since we >> need some cpan stuff we've built) and also mangle in the perl we found >> in PATH into some scripts... > > Anything building or using perl modules should be inheriting the > perlnative class. This adds the dependency on perl-native and the > appropriate PATH entry. Where is the race? Maybe race isn't quite the right word. But recipe A depends on lib$something-perl-native, and brings in perl-native. It also checks for perl in its auto-foo and finds our perl. It now also uses our perl when it wants a host perl and all of the potential bad things happen, yes?
On Wed, 2011-06-01 at 12:39 -0700, Tom Rini wrote: > Maybe race isn't quite the right word. But recipe A depends on > lib$something-perl-native, and brings in perl-native. It also checks > for perl in its auto-foo and finds our perl. It now also uses our perl > when it wants a host perl and all of the potential bad things happen, yes? What are the circumstances in which it would want a host perl? I don't quite understand the issue here. Surely anything that the host perl can do, perl-native plus some combination of libfoo-perl-native can also do, right? p.
On 06/01/2011 01:05 PM, Phil Blundell wrote: > On Wed, 2011-06-01 at 12:39 -0700, Tom Rini wrote: >> Maybe race isn't quite the right word. But recipe A depends on >> lib$something-perl-native, and brings in perl-native. It also checks >> for perl in its auto-foo and finds our perl. It now also uses our perl >> when it wants a host perl and all of the potential bad things happen, yes? > > What are the circumstances in which it would want a host perl? I don't > quite understand the issue here. Surely anything that the host perl can > do, perl-native plus some combination of libfoo-perl-native can also do, > right? So, here's the two things going on: - In some cases (libfoo-perl-native) we need to install various perl modules in order to build other target apps. This is the cpan case, iirc. We don't currently (and can't?) just play whatever games perl would like to do to use system-wide perl and a local to TMPDIR cpan directory. We instead go down the path of having perl-native. - In order to build perl for the target we need to have the same perl version available for use. What we do in oe-core (and oe.dev, historically) is provide perl-native for both of these cases. What falls down in this case is that once perl-native is built (and in our PATH), if it's a different version than system-wide perl, stuff starts failing on version mis-match. The patch series here fixes that by saying stuff must opt-in to having perl-native be in PATH rather than just system-wide perl. What I'm saying is that while in oe-core nothing falls down here (since nothing that gets perl-native also tries to just run 'perl') meta-oe is or will have failures. Unfortunately I don't have the test cases that drove me to make oe.dev like it is handy but I have a feeling that if we go down this path it won't be the last we see of the problem, but it might well pop up again. There is another option here, which is to make perl-native be perl-cross, live in its own special directory and figure out how to make a TMPDIR-specific cpan repository available for use by system-wide perl.
On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote: > What falls down in this case is that once > perl-native is built (and in our PATH), if it's a different version than > system-wide perl, stuff starts failing on version mis-match. I think that's the bit that I'm not properly understanding. Which versions are mismatching, exactly? Surely the local perl from the sysroot ought to be completely self-contained and shouldn't be using any bits from the host perl install at that point. p.
On 06/01/2011 01:45 PM, Phil Blundell wrote: > On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote: >> What falls down in this case is that once >> perl-native is built (and in our PATH), if it's a different version than >> system-wide perl, stuff starts failing on version mis-match. > > I think that's the bit that I'm not properly understanding. Which > versions are mismatching, exactly? > > Surely the local perl from the sysroot ought to be completely > self-contained and shouldn't be using any bits from the host perl > install at that point. So this jogs my memory a bit! It's not so much perl itself but stuff that uses perl that can get dirty and then no, you have stuff thats built for system perl and stuff that's built with perl-native clashing. Relying even more on memory, I think help2man was one of the "easy" culprits and since we also modify the env, we do things like have help2man run with PERL5LIB and so on pointing system-wide perl at perl-native's lib directory and so forth.
Tom Rini wrote: > On 06/01/2011 01:05 PM, Phil Blundell wrote: >> On Wed, 2011-06-01 at 12:39 -0700, Tom Rini wrote: >>> Maybe race isn't quite the right word. But recipe A depends on >>> lib$something-perl-native, and brings in perl-native. It also >>> checks >>> for perl in its auto-foo and finds our perl. It now also uses our >>> perl when it wants a host perl and all of the potential bad things >>> happen, yes? >> >> What are the circumstances in which it would want a host perl? I >> don't quite understand the issue here. Surely anything that the >> host perl can do, perl-native plus some combination of >> libfoo-perl-native can also do, right? > > So, here's the two things going on: > - In some cases (libfoo-perl-native) we need to install various perl > modules in order to build other target apps. This is the cpan case, > iirc. We don't currently (and can't?) just play whatever games perl > would like to do to use system-wide perl and a local to TMPDIR cpan > directory. We instead go down the path of having perl-native. > - In order to build perl for the target we need to have the same perl > version available for use. > > What we do in oe-core (and oe.dev, historically) is provide > perl-native > for both of these cases. What falls down in this case is that once > perl-native is built (and in our PATH), if it's a different version > than system-wide perl, stuff starts failing on version mis-match. > The patch series here fixes that by saying stuff must opt-in to > having perl-native > be in PATH rather than just system-wide perl. What I'm saying is that > while in oe-core nothing falls down here (since nothing that gets > perl-native also tries to just run 'perl') meta-oe is or will have > failures. > > Unfortunately I don't have the test cases that drove me to make oe.dev > like it is handy but I have a feeling that if we go down this path it > won't be the last we see of the problem, but it might well pop up > again. > > There is another option here, which is to make perl-native be > perl-cross, live in its own special directory and figure out how to > make > a TMPDIR-specific cpan repository available for use by system-wide > perl. Hi Tom, Thank you very much for the detailed explanation! However I'm actually new to cpan and know few about the history of that in oe.dev, so I'm not sure I know enough about the background to make a proper decision now. I suppose Richard would comment here. :-) Thanks, -- Dexuan
On Wed, 2011-06-01 at 13:59 -0700, Tom Rini wrote: > On 06/01/2011 01:45 PM, Phil Blundell wrote: > > On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote: > >> What falls down in this case is that once > >> perl-native is built (and in our PATH), if it's a different version than > >> system-wide perl, stuff starts failing on version mis-match. > > > > I think that's the bit that I'm not properly understanding. Which > > versions are mismatching, exactly? > > > > Surely the local perl from the sysroot ought to be completely > > self-contained and shouldn't be using any bits from the host perl > > install at that point. > > So this jogs my memory a bit! It's not so much perl itself but stuff > that uses perl that can get dirty and then no, you have stuff thats > built for system perl and stuff that's built with perl-native clashing. > > Relying even more on memory, I think help2man was one of the "easy" > culprits and since we also modify the env, we do things like have > help2man run with PERL5LIB and so on pointing system-wide perl at > perl-native's lib directory and so forth. But with the proposed patch series either: a) help2man depends perlnative.bbclass In this case it can depend on perl-native being there, its in path and things work as per OE.dev. b) help2man doesn't depend in perlnative.bbclass It only sees the system perl. So I'm still not clear where the problem is? Cheers, Richard
On 06/02/2011 07:06 AM, Richard Purdie wrote: > On Wed, 2011-06-01 at 13:59 -0700, Tom Rini wrote: >> On 06/01/2011 01:45 PM, Phil Blundell wrote: >>> On Wed, 2011-06-01 at 13:42 -0700, Tom Rini wrote: >>>> What falls down in this case is that once >>>> perl-native is built (and in our PATH), if it's a different version than >>>> system-wide perl, stuff starts failing on version mis-match. >>> >>> I think that's the bit that I'm not properly understanding. Which >>> versions are mismatching, exactly? >>> >>> Surely the local perl from the sysroot ought to be completely >>> self-contained and shouldn't be using any bits from the host perl >>> install at that point. >> >> So this jogs my memory a bit! It's not so much perl itself but stuff >> that uses perl that can get dirty and then no, you have stuff thats >> built for system perl and stuff that's built with perl-native clashing. >> >> Relying even more on memory, I think help2man was one of the "easy" >> culprits and since we also modify the env, we do things like have >> help2man run with PERL5LIB and so on pointing system-wide perl at >> perl-native's lib directory and so forth. > > But with the proposed patch series either: > > a) help2man depends perlnative.bbclass > > In this case it can depend on perl-native being there, its in path and > things work as per OE.dev. > > b) help2man doesn't depend in perlnative.bbclass > > It only sees the system perl. > > So I'm still not clear where the problem is? Well, help2man-native is (or needs to be) a dep listed in autotools.bbclass (since so many things need it, hence why it oe-core still just has it as a required host utility instead of building it). But help2man is just the easy/common case. Heck, it _may_ blow up even with the host help2man instead of help2man-native, if a recipe uses system-wide help2man and perlnative.bbclass. The root problem (again, from memory) is that since we modify PERL5LIB and so on, when we do that, we've opened ourselves up for system-wide perl trying to use perl-native's stuff.
On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote: > But help2man is just the easy/common case. Heck, it _may_ blow up even > with the host help2man instead of help2man-native, if a recipe uses > system-wide help2man and perlnative.bbclass. The root problem (again, > from memory) is that since we modify PERL5LIB and so on, when we do > that, we've opened ourselves up for system-wide perl trying to use > perl-native's stuff. Ah right, I think I see what you're getting at now. If we've got a clean separation between perl-native and host perl, though, can't we now just eliminate all of that futzing with PERL5LIB in cpan.bbclass and such like places? perl-native already knows how to look in the right places within the sysroot for its modules so there should be no need for anything else to be overriding it. p.
On 06/02/2011 07:37 AM, Phil Blundell wrote: > On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote: >> But help2man is just the easy/common case. Heck, it _may_ blow up even >> with the host help2man instead of help2man-native, if a recipe uses >> system-wide help2man and perlnative.bbclass. The root problem (again, >> from memory) is that since we modify PERL5LIB and so on, when we do >> that, we've opened ourselves up for system-wide perl trying to use >> perl-native's stuff. > > Ah right, I think I see what you're getting at now. > > If we've got a clean separation between perl-native and host perl, > though, can't we now just eliminate all of that futzing with PERL5LIB in > cpan.bbclass and such like places? perl-native already knows how to > look in the right places within the sysroot for its modules so there > should be no need for anything else to be overriding it. Well, my question is, does it, really? Even if we're using the sstate cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp is rm -rf'd) ? Since we've got a create_wrapper around perl and perl${PV} it should be I suppose (or is easily added there), but I'd feel a lot better with some testing of the above case (and the updates to cpan*bbclass).
On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote: > On 06/02/2011 07:37 AM, Phil Blundell wrote: > > On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote: > >> But help2man is just the easy/common case. Heck, it _may_ blow up even > >> with the host help2man instead of help2man-native, if a recipe uses > >> system-wide help2man and perlnative.bbclass. The root problem (again, > >> from memory) is that since we modify PERL5LIB and so on, when we do > >> that, we've opened ourselves up for system-wide perl trying to use > >> perl-native's stuff. > > > > Ah right, I think I see what you're getting at now. > > > > If we've got a clean separation between perl-native and host perl, > > though, can't we now just eliminate all of that futzing with PERL5LIB in > > cpan.bbclass and such like places? perl-native already knows how to > > look in the right places within the sysroot for its modules so there > > should be no need for anything else to be overriding it. > > Well, my question is, does it, really? If it doesn't it really does need to get fixed. I've not seen any evidence this has been the problem but it still could be. > Even if we're using the sstate > cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp > is rm -rf'd) ? Since we've got a create_wrapper around perl and > perl${PV} it should be I suppose (or is easily added there), but I'd > feel a lot better with some testing of the above case (and the updates > to cpan*bbclass). I only took the perl-native DEPENDS patch on the condition this gets fixed properly. The patches that are there look to do that, at least for OE-Core. If there are further issues we're going to have to take them as they arise as I have an objection to crippling the build dependencies because perl is broken. Really this could use some TLC from someone with experience in the area... Cheers, Richard
On 06/02/2011 09:35 AM, Richard Purdie wrote: > On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote: >> On 06/02/2011 07:37 AM, Phil Blundell wrote: >>> On Thu, 2011-06-02 at 07:25 -0700, Tom Rini wrote: >>>> But help2man is just the easy/common case. Heck, it _may_ blow up even >>>> with the host help2man instead of help2man-native, if a recipe uses >>>> system-wide help2man and perlnative.bbclass. The root problem (again, >>>> from memory) is that since we modify PERL5LIB and so on, when we do >>>> that, we've opened ourselves up for system-wide perl trying to use >>>> perl-native's stuff. >>> >>> Ah right, I think I see what you're getting at now. >>> >>> If we've got a clean separation between perl-native and host perl, >>> though, can't we now just eliminate all of that futzing with PERL5LIB in >>> cpan.bbclass and such like places? perl-native already knows how to >>> look in the right places within the sysroot for its modules so there >>> should be no need for anything else to be overriding it. >> >> Well, my question is, does it, really? > > If it doesn't it really does need to get fixed. I've not seen any > evidence this has been the problem but it still could be. I agree it needs to be fixed, yes. One way or another... >> Even if we're using the sstate >> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp >> is rm -rf'd) ? Since we've got a create_wrapper around perl and >> perl${PV} it should be I suppose (or is easily added there), but I'd >> feel a lot better with some testing of the above case (and the updates >> to cpan*bbclass). > > I only took the perl-native DEPENDS patch on the condition this gets > fixed properly. The patches that are there look to do that, at least for > OE-Core. If there are further issues we're going to have to take them as > they arise as I have an objection to crippling the build dependencies > because perl is broken. Really this could use some TLC from someone with > experience in the area... Well, I guess I'd boil down what I said above into a request like this for v3: - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB / PERLHOSTLIB. - In /scratch/oecore/tmp0 build the images that were built for v1 - In /scratch/oecore/tmp1 build perl-native's full sstate cache. - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1. - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same images that were built for tmp0. If all of that works, I think we can call this safe enough for now and possibly even really fixed, while we have someone actively kicking around the recipes in question.
On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote: > On 06/02/2011 09:35 AM, Richard Purdie wrote: > > On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote: > >> Even if we're using the sstate > >> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp > >> is rm -rf'd) ? Since we've got a create_wrapper around perl and > >> perl${PV} it should be I suppose (or is easily added there), but I'd > >> feel a lot better with some testing of the above case (and the updates > >> to cpan*bbclass). > > > > I only took the perl-native DEPENDS patch on the condition this gets > > fixed properly. The patches that are there look to do that, at least for > > OE-Core. If there are further issues we're going to have to take them as > > they arise as I have an objection to crippling the build dependencies > > because perl is broken. Really this could use some TLC from someone with > > experience in the area... > > Well, I guess I'd boil down what I said above into a request like this > for v3: > - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB / > PERLHOSTLIB. The first three of these are all about the *target* perl location and I think we still need them due the mess that perl's build system is. With the patch series in question they won't actually point at perl-native in the target case and they are only really used for cross compiling purposes. PERLHOSTLIB is used by the target perl when cross compiling to find native .so files. perl-native will always be present at this point and again, it seems like a valid use case. Summary is that I don't think perl-native is broken in any way but we do need those variables. > - In /scratch/oecore/tmp0 build the images that were built for v1 > - In /scratch/oecore/tmp1 build perl-native's full sstate cache. > - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1. > - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same > images that were built for tmp0. I'm confused by this test cycle. What do you mean by "build perl-native's full sstate cache"? I suspect you've asking for some partial sstate cache to be shared between two builds? Put simpler, you probably want: in tmp0 "bitbake perl-native" in tmp1, different location to tmp0, "bitbake core-image-sato" but sharing the same sstate cache Is that what you're thinking? FWIW, I've been running locally with the patch series and I think I'd like to merge it. If there are issues we can address them as and when they're identified... Cheers, Richard
On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote: > On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote: > > On 06/02/2011 09:35 AM, Richard Purdie wrote: > > > On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote: > > >> Even if we're using the sstate > > >> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp > > >> is rm -rf'd) ? Since we've got a create_wrapper around perl and > > >> perl${PV} it should be I suppose (or is easily added there), but I'd > > >> feel a lot better with some testing of the above case (and the updates > > >> to cpan*bbclass). > > > > > > I only took the perl-native DEPENDS patch on the condition this gets > > > fixed properly. The patches that are there look to do that, at least for > > > OE-Core. If there are further issues we're going to have to take them as > > > they arise as I have an objection to crippling the build dependencies > > > because perl is broken. Really this could use some TLC from someone with > > > experience in the area... > > > > Well, I guess I'd boil down what I said above into a request like this > > for v3: > > - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB / > > PERLHOSTLIB. > > The first three of these are all about the *target* perl location and I > think we still need them due the mess that perl's build system is. With > the patch series in question they won't actually point at perl-native in > the target case and they are only really used for cross compiling > purposes. > > PERLHOSTLIB is used by the target perl when cross compiling to find > native .so files. perl-native will always be present at this point and > again, it seems like a valid use case. > > Summary is that I don't think perl-native is broken in any way but we do > need those variables. > > > - In /scratch/oecore/tmp0 build the images that were built for v1 > > - In /scratch/oecore/tmp1 build perl-native's full sstate cache. > > - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1. > > - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same > > images that were built for tmp0. > > I'm confused by this test cycle. What do you mean by "build > perl-native's full sstate cache"? > > I suspect you've asking for some partial sstate cache to be shared > between two builds? > > Put simpler, you probably want: > > in tmp0 "bitbake perl-native" > in tmp1, different location to tmp0, "bitbake core-image-sato" but > sharing the same sstate cache I meant to add that tmp0 should be renamed before this second step so if there are hardcoded references in any of the sstate packages they couldn't find anything in tmp0 as it would no longer exist. > Is that what you're thinking? > > FWIW, I've been running locally with the patch series and I think I'd > like to merge it. If there are issues we can address them as and when > they're identified... > > Cheers, > > Richard > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Richard Purdie wrote: > On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote: >> On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote: >>> On 06/02/2011 09:35 AM, Richard Purdie wrote: >>>> On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote: >>>>> Even if we're using the sstate >>>>> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and >>>>> /foo/oecore/tmp is rm -rf'd) ? Since we've got a create_wrapper >>>>> around perl and perl${PV} it should be I suppose (or is easily >>>>> added there), but I'd feel a lot better with some testing of the >>>>> above case (and the updates to cpan*bbclass). >>>> >>>> I only took the perl-native DEPENDS patch on the condition this >>>> gets fixed properly. The patches that are there look to do that, >>>> at least for OE-Core. If there are further issues we're going to >>>> have to take them as they arise as I have an objection to >>>> crippling the build dependencies because perl is broken. Really >>>> this could use some TLC from someone with experience in the area... >>> >>> Well, I guess I'd boil down what I said above into a request like >>> this for v3: - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / >>> PERL_ARCHLIB / PERLHOSTLIB. >> >> The first three of these are all about the *target* perl location >> and I think we still need them due the mess that perl's build system >> is. With the patch series in question they won't actually point at >> perl-native in the target case and they are only really used for >> cross compiling purposes. >> >> PERLHOSTLIB is used by the target perl when cross compiling to find >> native .so files. perl-native will always be present at this point >> and again, it seems like a valid use case. >> >> Summary is that I don't think perl-native is broken in any way but >> we do need those variables. >> >>> - In /scratch/oecore/tmp0 build the images that were built for v1 >>> - In /scratch/oecore/tmp1 build perl-native's full sstate cache. >>> - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1. >>> - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same >>> images that were built for tmp0. >> >> I'm confused by this test cycle. What do you mean by "build >> perl-native's full sstate cache"? >> >> I suspect you've asking for some partial sstate cache to be shared >> between two builds? >> >> Put simpler, you probably want: >> >> in tmp0 "bitbake perl-native" >> in tmp1, different location to tmp0, "bitbake core-image-sato" but >> sharing the same sstate cache > > I meant to add that tmp0 should be renamed before this second step so > if there are hardcoded references in any of the sstate packages they > couldn't find anything in tmp0 as it would no longer exist. Actually this doesn't work even without my patch series? i.e., without my patch series, 1) I "bitbake perl-native" in /poky.git/build/; 2) mv /poky.git/build /poky.git/build.bak; 3) in /poky.git/build2/, modify the conf/local.conf to set SSTATE_MIRRORS to point to /poky.git/build.bak/sstate-cache/, and "bitbake sgmlspl-native" would fail in do_compile: "Can't locate ExtUtils/Command.pm in @INC (@INC contains: /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-"linux/usr/lib/perl/5.12.3" Is this an existing bug? Thanks, -- Dexuan
On Thu, 2011-06-09 at 21:51 +0800, Cui, Dexuan wrote: > Richard Purdie wrote: > > On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote: > >> I suspect you've asking for some partial sstate cache to be shared > >> between two builds? > >> > >> Put simpler, you probably want: > >> > >> in tmp0 "bitbake perl-native" > >> in tmp1, different location to tmp0, "bitbake core-image-sato" but > >> sharing the same sstate cache > > > > I meant to add that tmp0 should be renamed before this second step so > > if there are hardcoded references in any of the sstate packages they > > couldn't find anything in tmp0 as it would no longer exist. > > Actually this doesn't work even without my patch series? > i.e., without my patch series, > 1) I "bitbake perl-native" in /poky.git/build/; > 2) mv /poky.git/build /poky.git/build.bak; > 3) in /poky.git/build2/, modify the conf/local.conf to set SSTATE_MIRRORS to point to /poky.git/build.bak/sstate-cache/, and "bitbake sgmlspl-native" would fail in do_compile: > "Can't locate ExtUtils/Command.pm in @INC (@INC contains: /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 /poky.git/build/tmp/sysroots/i686-"linux/usr/lib/perl/5.12.3" > Is this an existing bug? Looks like it. This did work but it looks like its been broken somehow :( Cheers, Richard
Richard Purdie wrote: > On Thu, 2011-06-09 at 21:51 +0800, Cui, Dexuan wrote: >> Richard Purdie wrote: >>> On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote: >>>> I suspect you've asking for some partial sstate cache to be shared >>>> between two builds? >>>> >>>> Put simpler, you probably want: >>>> >>>> in tmp0 "bitbake perl-native" >>>> in tmp1, different location to tmp0, "bitbake core-image-sato" but >>>> sharing the same sstate cache >>> >>> I meant to add that tmp0 should be renamed before this second step >>> so if there are hardcoded references in any of the sstate packages >>> they couldn't find anything in tmp0 as it would no longer exist. >> >> Actually this doesn't work even without my patch series? >> i.e., without my patch series, >> 1) I "bitbake perl-native" in /poky.git/build/; >> 2) mv /poky.git/build /poky.git/build.bak; >> 3) in /poky.git/build2/, modify the conf/local.conf to set >> SSTATE_MIRRORS to point to /poky.git/build.bak/sstate-cache/, and >> "bitbake sgmlspl-native" would fail in do_compile: "Can't locate >> ExtUtils/Command.pm in @INC (@INC contains: >> /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 >> /poky.git/build/tmp/sysroots/i686-linux/usr/lib/perl/5.12.3 >> /poky.git/build/tmp/sysroots/i686-"linux/usr/lib/perl/5.12.3" Is >> this an existing bug? > > Looks like it. This did work but it looks like its been broken > somehow :( > Hi Richard, I reported http://bugzilla.yoctoproject.org/show_bug.cgi?id=1157 to track this issue. Thanks, -- Dexuan
On 06/09/2011 01:08 AM, Richard Purdie wrote: > On Thu, 2011-06-09 at 09:04 +0100, Richard Purdie wrote: >> On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote: >>> On 06/02/2011 09:35 AM, Richard Purdie wrote: >>>> On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote: >>>>> Even if we're using the sstate >>>>> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp >>>>> is rm -rf'd) ? Since we've got a create_wrapper around perl and >>>>> perl${PV} it should be I suppose (or is easily added there), but I'd >>>>> feel a lot better with some testing of the above case (and the updates >>>>> to cpan*bbclass). >>>> >>>> I only took the perl-native DEPENDS patch on the condition this gets >>>> fixed properly. The patches that are there look to do that, at least for >>>> OE-Core. If there are further issues we're going to have to take them as >>>> they arise as I have an objection to crippling the build dependencies >>>> because perl is broken. Really this could use some TLC from someone with >>>> experience in the area... >>> >>> Well, I guess I'd boil down what I said above into a request like this >>> for v3: >>> - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB / >>> PERLHOSTLIB. >> >> The first three of these are all about the *target* perl location and I >> think we still need them due the mess that perl's build system is. With >> the patch series in question they won't actually point at perl-native in >> the target case and they are only really used for cross compiling >> purposes. >> >> PERLHOSTLIB is used by the target perl when cross compiling to find >> native .so files. perl-native will always be present at this point and >> again, it seems like a valid use case. >> >> Summary is that I don't think perl-native is broken in any way but we do >> need those variables. >> >>> - In /scratch/oecore/tmp0 build the images that were built for v1 >>> - In /scratch/oecore/tmp1 build perl-native's full sstate cache. >>> - Keep the sstate cache from tmp1 and otherwise rm -rf tmp1. >>> - In /scratch/oecore/tmp2 using the sstate from tmp1, build the same >>> images that were built for tmp0. >> >> I'm confused by this test cycle. What do you mean by "build >> perl-native's full sstate cache"? >> >> I suspect you've asking for some partial sstate cache to be shared >> between two builds? >> >> Put simpler, you probably want: >> >> in tmp0 "bitbake perl-native" >> in tmp1, different location to tmp0, "bitbake core-image-sato" but >> sharing the same sstate cache > > I meant to add that tmp0 should be renamed before this second step so if > there are hardcoded references in any of the sstate packages they > couldn't find anything in tmp0 as it would no longer exist. Yes, this is what I'm asking for.
On 06/09/2011 01:04 AM, Richard Purdie wrote: > On Thu, 2011-06-02 at 09:55 -0700, Tom Rini wrote: >> On 06/02/2011 09:35 AM, Richard Purdie wrote: >>> On Thu, 2011-06-02 at 09:28 -0700, Tom Rini wrote: >>>> Even if we're using the sstate >>>> cache from /foo/oecore/tmp over in /bar/oecore/tmp (and /foo/oecore/tmp >>>> is rm -rf'd) ? Since we've got a create_wrapper around perl and >>>> perl${PV} it should be I suppose (or is easily added there), but I'd >>>> feel a lot better with some testing of the above case (and the updates >>>> to cpan*bbclass). >>> >>> I only took the perl-native DEPENDS patch on the condition this gets >>> fixed properly. The patches that are there look to do that, at least for >>> OE-Core. If there are further issues we're going to have to take them as >>> they arise as I have an objection to crippling the build dependencies >>> because perl is broken. Really this could use some TLC from someone with >>> experience in the area... >> >> Well, I guess I'd boil down what I said above into a request like this >> for v3: >> - Modify cpan.bbclass to NOT set PERL_INC / PERL_LIB / PERL_ARCHLIB / >> PERLHOSTLIB. > > The first three of these are all about the *target* perl location and I > think we still need them due the mess that perl's build system is. With > the patch series in question they won't actually point at perl-native in > the target case and they are only really used for cross compiling > purposes. > > PERLHOSTLIB is used by the target perl when cross compiling to find > native .so files. perl-native will always be present at this point and > again, it seems like a valid use case. > > Summary is that I don't think perl-native is broken in any way but we do > need those variables. Maybe I'm having a dense morning here, but isn't that the point? The combination of perl's build system is messy and if we bring in cpan it needs not only target perl locations (to dump things into) but may call 'perl' to do things and if we're on perl 5.14 and your host is only 5.10 or 5.12, bad things can happen. And since cpan.bbclass isn't brought in by target perl recipe, what now? :)
Currently both perl-native (a.k.a. ${STAGING_BINDIR_NATIVE}/perl )and perl-native-runtime(a.k.a. the system perl, or /usr/bin/perl) appear in the PATH when bitbake is running. This can cause some race conditions: many places detecting perl from PATH can't make sure which path will be used as this depends on when perl-native's populate_sysroot is finished, e.g., automake-native and autoconf-native could use perl-native-runtime while gnu-config-native could use perl-native and this inconsistent usages can cause trouble, e.g., bug 941. And, as RP suggested, "the time to use perl-native instead of perl-native-runtime is where any perl module is being built, perl itself is being built or anything that has an explicit dependency on the perl version present". So I made the following changes to try to address the aboves issues: 1) populate perl-native into its own directory so it won't appear in PATH by default, and add perlnative.bbclass for these recipes that really depend on perl-native; 2) check all perl-native references and correct the ones that should be perl-native-runtime; 3) fix any building issues due to the new location of perl-native, especially cpan and cpan-base .bbclass. With the changes, bug 941 doesn't appear. Tests I did are: I tried "bitbale core-image-sato-sdk and meta-toolchain-gmae" in x86_32 and x86_64 Ubuntu hosts and everything seems building fine. Please review the changes and comment on them. Thanks! -------------------------- The following changes since commit 0736b136efac283449c9218c01740eba4a7bc8cc: are available in the git repository at: git://git.pokylinux.org/poky-contrib dcui/master_perl-native http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master_perl-native Dexuan Cui (16): native.bbclass: allow a native package to be populated into its own dir perl-native: populate into its own dir perlnative.bbclass: add the file gnu-config-native: should depend on perl-native-runtime rather than perl-native libcap: should depend on perl-native-runtime rather than perl-native openssl: should depend on perl-native-runtime rather than perl-native git: should depend on perl-native-runtime rather than perl-native coreutils: remove unnecessary dependency on perl dpkg: should depend on perl-native-runtime rather than perl-native webkit-gtk: should depend on perl-native-runtime rather than perl-native perl: inherit perlnative cpan.bbclass, cpan-base.bbclas: update them for the perlnative change libxml-parser-perl: inherit perlnative libconvert-asn1-perl: fix EXTRA_PERLFLAGS due to the perl-native change libxml-simple-perl: fix EXTRA_PERLFLAGS due the the perlnative change icon-naming-utils-native: inherit perlnative meta/classes/cpan-base.bbclass | 10 +++++++--- meta/classes/cpan.bbclass | 12 ++++++------ meta/classes/native.bbclass | 5 +++++ meta/classes/perlnative.bbclass | 2 ++ meta/recipes-connectivity/openssl/openssl.inc | 2 +- .../recipes-connectivity/openssl/openssl_0.9.8r.bb | 2 +- meta/recipes-core/coreutils/coreutils_6.9.bb | 6 +++--- meta/recipes-core/coreutils/coreutils_8.9.bb | 6 +++--- meta/recipes-devtools/dpkg/dpkg.inc | 2 +- meta/recipes-devtools/dpkg/dpkg_1.15.8.7.bb | 2 +- meta/recipes-devtools/git/git.inc | 4 ++-- meta/recipes-devtools/git/git_1.7.5.1.bb | 2 +- .../gnu-config/gnu-config_20080123.bb | 4 ++-- .../icon-naming-utils-native_0.8.7.bb | 4 ++-- .../perl/libxml-parser-perl-native_2.40.bb | 2 +- .../perl/libxml-parser-perl_2.40.bb | 2 +- .../perl/libxml-simple-perl-native_2.18.bb | 2 +- .../perl/libxml-simple-perl_2.18.bb | 4 ++-- meta/recipes-devtools/perl/perl-native_5.12.3.bb | 5 ++++- meta/recipes-devtools/perl/perl_5.12.3.bb | 10 +++++----- .../perl/libconvert-asn1-perl_0.22.bb | 4 ++-- meta/recipes-sato/webkit/webkit-gtk_svn.bb | 6 +++--- meta/recipes-support/libcap/libcap.inc | 4 ++-- meta/recipes-support/libcap/libcap_2.20.bb | 2 +- 24 files changed, 59 insertions(+), 45 deletions(-) create mode 100644 meta/classes/perlnative.bbclass