[v2] classes/waf: Fix builds when B != S

Submitted by Joshua Watt on Dec. 1, 2018, 3:01 a.m. | Patch ID: 156773

Details

Message ID 20181201030142.9385-1-JPEWhacker@gmail.com
State Master Next
Commit 62dffb71ce22222c635bd90eaa47dd01f70f9c0f
Headers show

Commit Message

Joshua Watt Dec. 1, 2018, 3:01 a.m.
Waf requires that the current working directory be ${S} (the location of
the wscript) when building. Most of the time, this was true only because
B defaults to S. However, anything that changed that behavior (notably,
using externalsrc) would break the recipe. Remedy this by explicitly
changing cwd to ${S} when running waf commands. As a happy side effect,
B can be set up for "out of tree" builds to keep the source directory
clean.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes/waf.bbclass | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

Patch hide | download patch | download mbox

diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
index 19e93761b39..8e6d754c299 100644
--- a/meta/classes/waf.bbclass
+++ b/meta/classes/waf.bbclass
@@ -1,6 +1,8 @@ 
 # avoids build breaks when using no-static-libs.inc
 DISABLE_STATIC = ""
 
+B = "${WORKDIR}/build"
+
 EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
 
 python waf_preconfigure() {
@@ -22,16 +24,16 @@  python waf_preconfigure() {
 do_configure[prefuncs] += "waf_preconfigure"
 
 waf_do_configure() {
-	${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
+	(cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
 }
 
 do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
 waf_do_compile()  {
-	${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
+	(cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
 }
 
 waf_do_install() {
-	${S}/waf install --destdir=${D}
+	(cd ${S} && ./waf install --destdir=${D})
 }
 
 EXPORT_FUNCTIONS do_configure do_compile do_install

Comments

Khem Raj Dec. 2, 2018, 12:09 a.m.
Hi Joshua

There is a build failure for a2jmidid see

http://errors.yoctoproject.org/Errors/Details/202833/

Seem to be related can you validate

On 11/30/18 7:01 PM, Joshua Watt wrote:
> Waf requires that the current working directory be ${S} (the location of
> the wscript) when building. Most of the time, this was true only because
> B defaults to S. However, anything that changed that behavior (notably,
> using externalsrc) would break the recipe. Remedy this by explicitly
> changing cwd to ${S} when running waf commands. As a happy side effect,
> B can be set up for "out of tree" builds to keep the source directory
> clean.
> 
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>   meta/classes/waf.bbclass | 8 +++++---
>   1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> index 19e93761b39..8e6d754c299 100644
> --- a/meta/classes/waf.bbclass
> +++ b/meta/classes/waf.bbclass
> @@ -1,6 +1,8 @@
>   # avoids build breaks when using no-static-libs.inc
>   DISABLE_STATIC = ""
>   
> +B = "${WORKDIR}/build"
> +
>   EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
>   
>   python waf_preconfigure() {
> @@ -22,16 +24,16 @@ python waf_preconfigure() {
>   do_configure[prefuncs] += "waf_preconfigure"
>   
>   waf_do_configure() {
> -	${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
> +	(cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
>   }
>   
>   do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
>   waf_do_compile()  {
> -	${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
> +	(cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
>   }
>   
>   waf_do_install() {
> -	${S}/waf install --destdir=${D}
> +	(cd ${S} && ./waf install --destdir=${D})
>   }
>   
>   EXPORT_FUNCTIONS do_configure do_compile do_install
>
Khem Raj Dec. 3, 2018, 8:14 p.m.
On Sat, Dec 1, 2018 at 4:09 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> Hi Joshua
>
> There is a build failure for a2jmidid see
>
> http://errors.yoctoproject.org/Errors/Details/202833/
>
> Seem to be related can you validate
>
> On 11/30/18 7:01 PM, Joshua Watt wrote:
> > Waf requires that the current working directory be ${S} (the location of
> > the wscript) when building. Most of the time, this was true only because
> > B defaults to S. However, anything that changed that behavior (notably,
> > using externalsrc) would break the recipe. Remedy this by explicitly
> > changing cwd to ${S} when running waf commands. As a happy side effect,
> > B can be set up for "out of tree" builds to keep the source directory
> > clean.

I have sent a workaround for a2jmidid to ml which means we can go
ahead with this change in OE-Core

> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> >   meta/classes/waf.bbclass | 8 +++++---
> >   1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> > index 19e93761b39..8e6d754c299 100644
> > --- a/meta/classes/waf.bbclass
> > +++ b/meta/classes/waf.bbclass
> > @@ -1,6 +1,8 @@
> >   # avoids build breaks when using no-static-libs.inc
> >   DISABLE_STATIC = ""
> >
> > +B = "${WORKDIR}/build"
> > +
> >   EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
> >
> >   python waf_preconfigure() {
> > @@ -22,16 +24,16 @@ python waf_preconfigure() {
> >   do_configure[prefuncs] += "waf_preconfigure"
> >
> >   waf_do_configure() {
> > -     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
> > +     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
> >   }
> >
> >   do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
> >   waf_do_compile()  {
> > -     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
> > +     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
> >   }
> >
> >   waf_do_install() {
> > -     ${S}/waf install --destdir=${D}
> > +     (cd ${S} && ./waf install --destdir=${D})
> >   }
> >
> >   EXPORT_FUNCTIONS do_configure do_compile do_install
> >
Andreas Müller Dec. 3, 2018, 9:50 p.m.
On Mon, Dec 3, 2018 at 9:14 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Sat, Dec 1, 2018 at 4:09 PM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > Hi Joshua
> >
> > There is a build failure for a2jmidid see
> >
> > http://errors.yoctoproject.org/Errors/Details/202833/
> >
> > Seem to be related can you validate
> >
> > On 11/30/18 7:01 PM, Joshua Watt wrote:
> > > Waf requires that the current working directory be ${S} (the location of
> > > the wscript) when building. Most of the time, this was true only because
> > > B defaults to S. However, anything that changed that behavior (notably,
> > > using externalsrc) would break the recipe. Remedy this by explicitly
> > > changing cwd to ${S} when running waf commands. As a happy side effect,
> > > B can be set up for "out of tree" builds to keep the source directory
> > > clean.
>
> I have sent a workaround for a2jmidid to ml which means we can go
> ahead with this change in OE-Core
>
> > >
> > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > > ---
> > >   meta/classes/waf.bbclass | 8 +++++---
> > >   1 file changed, 5 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> > > index 19e93761b39..8e6d754c299 100644
> > > --- a/meta/classes/waf.bbclass
> > > +++ b/meta/classes/waf.bbclass
> > > @@ -1,6 +1,8 @@
> > >   # avoids build breaks when using no-static-libs.inc
> > >   DISABLE_STATIC = ""
> > >
> > > +B = "${WORKDIR}/build"
> > > +
> > >   EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
> > >
> > >   python waf_preconfigure() {
> > > @@ -22,16 +24,16 @@ python waf_preconfigure() {
> > >   do_configure[prefuncs] += "waf_preconfigure"
> > >
> > >   waf_do_configure() {
> > > -     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
> > > +     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
> > >   }
> > >
> > >   do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
> > >   waf_do_compile()  {
> > > -     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
> > > +     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
> > >   }
> > >
> > >   waf_do_install() {
> > > -     ${S}/waf install --destdir=${D}
> > > +     (cd ${S} && ./waf install --destdir=${D})
> > >   }
> > >
> > >   EXPORT_FUNCTIONS do_configure do_compile do_install
> > >
> --
Hmm - the a2jmidid shows what's going to happen: All packets shipping
older waf need workaround hacks. This should not be the way to go
because then we can remove waf.bbclass entirely.

Andreas
Khem Raj Dec. 3, 2018, 10:04 p.m.
On Mon, Dec 3, 2018 at 1:50 PM Andreas Müller <schnitzeltony@gmail.com> wrote:
>
> On Mon, Dec 3, 2018 at 9:14 PM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > On Sat, Dec 1, 2018 at 4:09 PM Khem Raj <raj.khem@gmail.com> wrote:
> > >
> > > Hi Joshua
> > >
> > > There is a build failure for a2jmidid see
> > >
> > > http://errors.yoctoproject.org/Errors/Details/202833/
> > >
> > > Seem to be related can you validate
> > >
> > > On 11/30/18 7:01 PM, Joshua Watt wrote:
> > > > Waf requires that the current working directory be ${S} (the location of
> > > > the wscript) when building. Most of the time, this was true only because
> > > > B defaults to S. However, anything that changed that behavior (notably,
> > > > using externalsrc) would break the recipe. Remedy this by explicitly
> > > > changing cwd to ${S} when running waf commands. As a happy side effect,
> > > > B can be set up for "out of tree" builds to keep the source directory
> > > > clean.
> >
> > I have sent a workaround for a2jmidid to ml which means we can go
> > ahead with this change in OE-Core
> >
> > > >
> > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > > > ---
> > > >   meta/classes/waf.bbclass | 8 +++++---
> > > >   1 file changed, 5 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> > > > index 19e93761b39..8e6d754c299 100644
> > > > --- a/meta/classes/waf.bbclass
> > > > +++ b/meta/classes/waf.bbclass
> > > > @@ -1,6 +1,8 @@
> > > >   # avoids build breaks when using no-static-libs.inc
> > > >   DISABLE_STATIC = ""
> > > >
> > > > +B = "${WORKDIR}/build"
> > > > +
> > > >   EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
> > > >
> > > >   python waf_preconfigure() {
> > > > @@ -22,16 +24,16 @@ python waf_preconfigure() {
> > > >   do_configure[prefuncs] += "waf_preconfigure"
> > > >
> > > >   waf_do_configure() {
> > > > -     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF}
> > > > +     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
> > > >   }
> > > >
> > > >   do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
> > > >   waf_do_compile()  {
> > > > -     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
> > > > +     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)})
> > > >   }
> > > >
> > > >   waf_do_install() {
> > > > -     ${S}/waf install --destdir=${D}
> > > > +     (cd ${S} && ./waf install --destdir=${D})
> > > >   }
> > > >
> > > >   EXPORT_FUNCTIONS do_configure do_compile do_install
> > > >
> > --
> Hmm - the a2jmidid shows what's going to happen: All packets shipping
> older waf need workaround hacks. This should not be the way to go
> because then we can remove waf.bbclass entirely.
>

FWIW out of all meta-openembedded layers only 1 recipe was having
this issue, while the issue you mention is real, we might have to weigh
in if we should not do this just for a handful of recipes. My
suggestion would be to change the recipes if its not a cascade

> Andreas
Joshua Watt Dec. 3, 2018, 10:22 p.m.
On Mon, 2018-12-03 at 22:50 +0100, Andreas Müller wrote:
> On Mon, Dec 3, 2018 at 9:14 PM Khem Raj <raj.khem@gmail.com> wrote:
> > On Sat, Dec 1, 2018 at 4:09 PM Khem Raj <raj.khem@gmail.com> wrote:
> > > Hi Joshua
> > > 
> > > There is a build failure for a2jmidid see
> > > 
> > > http://errors.yoctoproject.org/Errors/Details/202833/
> > > 
> > > Seem to be related can you validate
> > > 
> > > On 11/30/18 7:01 PM, Joshua Watt wrote:
> > > > Waf requires that the current working directory be ${S} (the
> > > > location of
> > > > the wscript) when building. Most of the time, this was true
> > > > only because
> > > > B defaults to S. However, anything that changed that behavior
> > > > (notably,
> > > > using externalsrc) would break the recipe. Remedy this by
> > > > explicitly
> > > > changing cwd to ${S} when running waf commands. As a happy side
> > > > effect,
> > > > B can be set up for "out of tree" builds to keep the source
> > > > directory
> > > > clean.
> > 
> > I have sent a workaround for a2jmidid to ml which means we can go
> > ahead with this change in OE-Core
> > 
> > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > > > ---
> > > >   meta/classes/waf.bbclass | 8 +++++---
> > > >   1 file changed, 5 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/meta/classes/waf.bbclass
> > > > b/meta/classes/waf.bbclass
> > > > index 19e93761b39..8e6d754c299 100644
> > > > --- a/meta/classes/waf.bbclass
> > > > +++ b/meta/classes/waf.bbclass
> > > > @@ -1,6 +1,8 @@
> > > >   # avoids build breaks when using no-static-libs.inc
> > > >   DISABLE_STATIC = ""
> > > > 
> > > > +B = "${WORKDIR}/build"
> > > > +
> > > >   EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"
> > > > 
> > > >   python waf_preconfigure() {
> > > > @@ -22,16 +24,16 @@ python waf_preconfigure() {
> > > >   do_configure[prefuncs] += "waf_preconfigure"
> > > > 
> > > >   waf_do_configure() {
> > > > -     ${S}/waf configure --prefix=${prefix} ${WAF_EXTRA_CONF}
> > > > ${EXTRA_OECONF}
> > > > +     (cd ${S} && ./waf configure -o ${B} --prefix=${prefix}
> > > > ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
> > > >   }
> > > > 
> > > >   do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
> > > >   waf_do_compile()  {
> > > > -     ${S}/waf build ${@oe.utils.parallel_make_argument(d, '-
> > > > j%d', limit=64)}
> > > > +     (cd ${S} && ./waf build ${@oe.utils.parallel_make_argumen
> > > > t(d, '-j%d', limit=64)})
> > > >   }
> > > > 
> > > >   waf_do_install() {
> > > > -     ${S}/waf install --destdir=${D}
> > > > +     (cd ${S} && ./waf install --destdir=${D})
> > > >   }
> > > > 
> > > >   EXPORT_FUNCTIONS do_configure do_compile do_install
> > > > 
> > --
> Hmm - the a2jmidid shows what's going to happen: All packets shipping
> older waf need workaround hacks. This should not be the way to go
> because then we can remove waf.bbclass entirely.

Yes, that might actually be the most "waf" answer to that problem...
for better or worse waf is designed to be fully self contained, so
having a "standard" class that builds it is somewhat problematic. Waf
itself could change at any time and render the canned configuration
useless. This doesn't really affect any of the projects using waf
because they usually just download (or build) the latest when they
start development and don't upgrade (or do so infrequently).

The other option that I've though about (not that I really like it) is
to have some sort of WAF_VERSION variable that waf.bbclass keys off of
to detect what support is available. However, even this isn't fool
proof because of add-in modules that you can choose to build into your
project's version of waf... these can basically do whatever you want
including adding new command line arguments, and adding or modifying
existing behaviors. I'm not in favor of going down the road of making
variables like WAF_HAS_GNU_DIRS_PLUGIN that each project has to
configure.

OTOH, as long as the projects we are building with waf currently use a
new enough version (I know 1.7 works and 1.4 is broken), our current
bbclass works and maybe that's "good enough" for now?

I'm open to suggestions.


> 
> Andreas
Andreas Müller Dec. 3, 2018, 11:18 p.m.
On Mon, Dec 3, 2018 at 11:22 PM Joshua Watt <jpewhacker@gmail.com> wrote:
> > Hmm - the a2jmidid shows what's going to happen: All packets shipping
> > older waf need workaround hacks. This should not be the way to go
> > because then we can remove waf.bbclass entirely.
>
> Yes, that might actually be the most "waf" answer to that problem...
> for better or worse waf is designed to be fully self contained, so
> having a "standard" class that builds it is somewhat problematic. Waf
> itself could change at any time and render the canned configuration
> useless. This doesn't really affect any of the projects using waf
> because they usually just download (or build) the latest when they
> start development and don't upgrade (or do so infrequently).
>
> The other option that I've though about (not that I really like it) is
> to have some sort of WAF_VERSION variable that waf.bbclass keys off of
> to detect what support is available. However, even this isn't fool
> proof because of add-in modules that you can choose to build into your
> project's version of waf... these can basically do whatever you want
> including adding new command line arguments, and adding or modifying
> existing behaviors. I'm not in favor of going down the road of making
> variables like WAF_HAS_GNU_DIRS_PLUGIN that each project has to
> configure.
>
> OTOH, as long as the projects we are building with waf currently use a
> new enough version (I know 1.7 works and 1.4 is broken), our current
> bbclass works and maybe that's "good enough" for now?
>
> I'm open to suggestions.
Have meta-qt5-extra with many recipes on old sources which potentially
fail for this change. Will build with your's and Khem's patch and fix
affected recipes. Think version magic is not necessary.

Andreas