| Submitter | Richard Purdie |
|---|---|
| Date | Sept. 7, 2012, 4:05 p.m. |
| Message ID | <1347033943.8619.0.camel@ted> |
| Download | mbox | patch |
| Permalink | /patch/36135/ |
| State | New |
| Headers | show |
Comments
I'm curious, is there any [easy] way we can force a rerun of configure as a test pass over the system? I'd like a way to verify that both this patch works as expected, and future recipes work as expected. (It would also be nice to test the things that don't use the autotools.bbclass..) --Mark On 9/7/12 11:05 AM, Richard Purdie wrote: > Unfortunately whilst reruning configure and make against a project will mostly > work there are situations where it does not correctly do the right thing. > > In particular, eglibc and gcc will fail out with errors where settings > do not match a previously built configuration. It could be argued they are > broken but the situation is what it is. There is the possibility of more subtle > errors too. > > This patch adds a "make distclean" call to recipes where configure is > rerunning and the sstate checksum for do_configure has changed. We could > simply use a stamp but saving out the previous configuration checksum > adds some data at no real overhead. > > If we find there are things out there which don't have a "distclean" target, > we can disable this behaviour with CONFIGURESTAMPFILE = "" in the recipe, > or users could disable it globally. > > [YOCTO #2774] > [YOCTO #2848] > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > diff --git a/meta/classes/autotools.bbclass b/meta/classes/autotools.bbclass > index 4c4bf87..1ab2e0c 100644 > --- a/meta/classes/autotools.bbclass > +++ b/meta/classes/autotools.bbclass > @@ -89,7 +89,16 @@ oe_runconf () { > > AUTOTOOLS_AUXDIR ?= "${S}" > > +CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate" > + > autotools_do_configure() { > + if [ -n "${CONFIGURESTAMPFILE}" -a -e "${CONFIGURESTAMPFILE}" ]; then > + if [ "`cat ${CONFIGURESTAMPFILE}`" != "${BB_TASKHASH}" ]; then > + echo "Previously configured build detected, running make distclean" > + oe_runmake distclean > + fi > + fi > + > case ${PN} in > autoconf*) > ;; > @@ -180,6 +189,9 @@ autotools_do_configure() { > else > bbnote "nothing to configure" > fi > + if [ -n "${CONFIGURESTAMPFILE}" ]; then > + echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE} > + fi > } > > autotools_do_install() { > > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core >
On Fri, 2012-09-07 at 11:32 -0500, Mark Hatle wrote: > I'm curious, is there any [easy] way we can force a rerun of configure as a test > pass over the system? rm tmp/stamps/*/*do_configure* along with a rename of the sstate cache so it doesn't get used. > I'd like a way to verify that both this patch works as expected, and future > recipes work as expected. (It would also be nice to test the things that don't > use the autotools.bbclass..) Cheers, Richard
On 7 September 2012 17:32, Mark Hatle <mark.hatle@windriver.com> wrote: > I'm curious, is there any [easy] way we can force a rerun of configure as a > test pass over the system? > > I'd like a way to verify that both this patch works as expected, and future > recipes work as expected. (It would also be nice to test the things that > don't use the autotools.bbclass..) Yeah, I expect we'll discover some cases when upstream just don't expect a distclean. By generally taking from git and re-running the entire autotools we *should* be okay, but... Ross
On Fri, 2012-09-07 at 17:42 +0100, Burton, Ross wrote: > On 7 September 2012 17:32, Mark Hatle <mark.hatle@windriver.com> wrote: > > I'm curious, is there any [easy] way we can force a rerun of configure as a > > test pass over the system? > > > > I'd like a way to verify that both this patch works as expected, and future > > recipes work as expected. (It would also be nice to test the things that > > don't use the autotools.bbclass..) > > Yeah, I expect we'll discover some cases when upstream just don't > expect a distclean. By generally taking from git and re-running the > entire autotools we *should* be okay, but... Further testing suggests we either going to need a whitelist or a blacklist for this :/ The key places people get bitten are eglibc and gcc so those should be straight forward to test, the question is how widely to deploy this initially. I think the mechanism is good, its now just a question of the implementation detail. FWIW, libgpg-error fails with checksum issues (checksumming a generated file?!) and libtool has issues about cleaning directories that have makefiles that were never generated... Cheers, Richard
On Fri, 2012-09-07 at 18:09 +0100, Richard Purdie wrote: > The key places people get bitten are eglibc and gcc so those should be > straight forward to test, the question is how widely to deploy this > initially. I think the mechanism is good, its now just a question of the > implementation detail. Eglibc and gcc, at least, support building with ${B} != ${S} so it might be easier/quicker to just blow away the whole ${B} tree rather than trying to distclean it. I imagine that most modern-ish autotools-based packages will also build fine in that configuration, though there are bound to be some that don't. It's hard to say whether there are likely to be more or fewer of those than there are ones where "make distclean" fails. p.
On (07/09/12 18:15), Phil Blundell wrote: > On Fri, 2012-09-07 at 18:09 +0100, Richard Purdie wrote: > > The key places people get bitten are eglibc and gcc so those should be > > straight forward to test, the question is how widely to deploy this > > initially. I think the mechanism is good, its now just a question of the > > implementation detail. > > Eglibc and gcc, at least, support building with ${B} != ${S} so it might > be easier/quicker to just blow away the whole ${B} tree rather than > trying to distclean it. yeah and I think separating B from S in general has merits in long run > > I imagine that most modern-ish autotools-based packages will also build > fine in that configuration, though there are bound to be some that > don't. It's hard to say whether there are likely to be more or fewer of > those than there are ones where "make distclean" fails. > either way there seems to be same amount of uncertainity. > p. > > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
On 9/8/12 3:05 AM, Khem Raj wrote: > On (07/09/12 18:15), Phil Blundell wrote: >> On Fri, 2012-09-07 at 18:09 +0100, Richard Purdie wrote: >>> The key places people get bitten are eglibc and gcc so those should be >>> straight forward to test, the question is how widely to deploy this >>> initially. I think the mechanism is good, its now just a question of the >>> implementation detail. >> >> Eglibc and gcc, at least, support building with ${B} != ${S} so it might >> be easier/quicker to just blow away the whole ${B} tree rather than >> trying to distclean it. > > yeah and I think separating B from S in general has merits in long run In principal I agree completely, however in practice at least 20-30% of the packages I try won't deal w/ the B/S separation properly. A good optimization is likely when B != S, blow away B and start over... if they are the same, the distclean is likely the best approach.. >> >> I imagine that most modern-ish autotools-based packages will also build >> fine in that configuration, though there are bound to be some that >> don't. It's hard to say whether there are likely to be more or fewer of >> those than there are ones where "make distclean" fails. >> > > either way there seems to be same amount of uncertainity. > >> p. >> >> >> >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core >
On Sat, 2012-09-08 at 08:30 -0500, Mark Hatle wrote: > In principal I agree completely, however in practice at least 20-30% of the > packages I try won't deal w/ the B/S separation properly. So what I did in my most recent build system[1] is: 1) Default to srcdir != builddir 2) Tag modules which don't support that See: http://people.gnome.org/~walters/docs/build-api.txt If a module doesn't support srcdir != builddir, I simply copy the source tree to the builddir. This is can be a confusing trap from a developer perspective; I've caught myself editing the copied source files, and then had them blown away during a build. But in general we should pressure upstreams to support srcdir != builddir, because the benefits are great and the drawbacks small. It'd be awesome if you guys considered having the bitbake autotools class implement the same semantics. [1] http://git.gnome.org/browse/gnome-ostree/tree/src/ostbuild/pyostbuild/builtin_compile_one.py#n132
On Fri, 2012-09-07 at 17:05 +0100, Richard Purdie wrote: > Unfortunately whilst reruning configure and make against a project will mostly > work there are situations where it does not correctly do the right thing. > > In particular, eglibc and gcc will fail out with errors where settings > do not match a previously built configuration. It could be argued they are > broken but the situation is what it is. There is the possibility of more subtle > errors too. > > This patch adds a "make distclean" call to recipes where configure is > rerunning and the sstate checksum for do_configure has changed. We could > simply use a stamp but saving out the previous configuration checksum > adds some data at no real overhead. The major problem with distclean is that it completely falls over in the scenario where the source code has changed; the new distclean won't know about *old* object files it no longer builds, files renamed, etc. In GNOME we just this cycle landed this patch to use "git clean -dfx" instead of "make distclean" if possible: https://bugzilla.gnome.org/show_bug.cgi?id=656081 It might be interesting to have an option to run: "git init; git add .; git commit -a -m auto-import" on each tarball build in OE. I've been considering doing this for jhbuild. (My latest build system *only* builds from git repositories, so it works there =) )
On 9/8/12 10:44 AM, Colin Walters wrote: > On Fri, 2012-09-07 at 17:05 +0100, Richard Purdie wrote: >> Unfortunately whilst reruning configure and make against a project will mostly >> work there are situations where it does not correctly do the right thing. >> >> In particular, eglibc and gcc will fail out with errors where settings >> do not match a previously built configuration. It could be argued they are >> broken but the situation is what it is. There is the possibility of more subtle >> errors too. >> >> This patch adds a "make distclean" call to recipes where configure is >> rerunning and the sstate checksum for do_configure has changed. We could >> simply use a stamp but saving out the previous configuration checksum >> adds some data at no real overhead. > > The major problem with distclean is that it completely falls over in the > scenario where the source code has changed; the new distclean won't know > about *old* object files it no longer builds, files renamed, etc. > > In GNOME we just this cycle landed this patch to use "git clean -dfx" > instead of "make distclean" if possible: > https://bugzilla.gnome.org/show_bug.cgi?id=656081 > > It might be interesting to have an option to run: > "git init; git add .; git commit -a -m auto-import" on each tarball > build in OE. I've been considering doing this for jhbuild. > > (My latest build system *only* builds from git repositories, so > it works there =) ) While we're not managing the patching w/ git.. we are with quilt. Does quilt have any facilities for restoring to the pristine last copy vs whatever is in the current tree? (I don't think it does BTW.) --Mark > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core >
Patch
diff --git a/meta/classes/autotools.bbclass b/meta/classes/autotools.bbclass index 4c4bf87..1ab2e0c 100644 --- a/meta/classes/autotools.bbclass +++ b/meta/classes/autotools.bbclass @@ -89,7 +89,16 @@ oe_runconf () { AUTOTOOLS_AUXDIR ?= "${S}" +CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate" + autotools_do_configure() { + if [ -n "${CONFIGURESTAMPFILE}" -a -e "${CONFIGURESTAMPFILE}" ]; then + if [ "`cat ${CONFIGURESTAMPFILE}`" != "${BB_TASKHASH}" ]; then + echo "Previously configured build detected, running make distclean" + oe_runmake distclean + fi + fi + case ${PN} in autoconf*) ;; @@ -180,6 +189,9 @@ autotools_do_configure() { else bbnote "nothing to configure" fi + if [ -n "${CONFIGURESTAMPFILE}" ]; then + echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE} + fi } autotools_do_install() {
Unfortunately whilst reruning configure and make against a project will mostly work there are situations where it does not correctly do the right thing. In particular, eglibc and gcc will fail out with errors where settings do not match a previously built configuration. It could be argued they are broken but the situation is what it is. There is the possibility of more subtle errors too. This patch adds a "make distclean" call to recipes where configure is rerunning and the sstate checksum for do_configure has changed. We could simply use a stamp but saving out the previous configuration checksum adds some data at no real overhead. If we find there are things out there which don't have a "distclean" target, we can disable this behaviour with CONFIGURESTAMPFILE = "" in the recipe, or users could disable it globally. [YOCTO #2774] [YOCTO #2848] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> ---