Patchwork [1/6] scripts/bitbake: ensure user is in build directory

login
register
mail settings
Submitter Paul Eggleton
Date March 14, 2012, 12:36 a.m.
Message ID <e4d001f8c6f8945c8c70c2558ad5f9e1fd85dfad.1331685330.git.paul.eggleton@linux.intel.com>
Download mbox | patch
Permalink /patch/23217/
State Accepted
Commit b4df1c7c79b5c801658bcf890ba3a8eab3d83189
Headers show

Comments

Paul Eggleton - March 14, 2012, 12:36 a.m.
If the user is in any directory other than $BUILDDIR when the bitbake
wrapper script is run, then show an error an exit.

Fixes [YOCTO #2071].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/bitbake |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
Darren Hart - March 14, 2012, 2:52 p.m.
On 03/13/2012 05:36 PM, Paul Eggleton wrote:
> If the user is in any directory other than $BUILDDIR when the bitbake
> wrapper script is run, then show an error an exit.
> 
> Fixes [YOCTO #2071].
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  scripts/bitbake |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/scripts/bitbake b/scripts/bitbake
> index dda3b26..45c8697 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -47,6 +47,11 @@ float_test() {
>  # but earlier versions do not
>  float_test "$TARVERSION > 1.23" && needtar="0"
>  
> +if [ "`pwd`" != "$BUILDDIR" ] ; then
> +    echo "BitBake must be run from your build directory: $BUILDDIR"
> +    exit 1

Should this have some kind of prefix? "ERROR: " or something along those
lines for consistency with other output?

Otherwise, YAY! I've tripped over this frequently and keep doing the cd
dance after a short "What the heck" session. Such a simple fix to -
Thanks Paul!

> +fi
> +
>  buildpseudo="1"
>  if [ $needpseudo = "1" ] && [ -e "$BUILDDIR/pseudodone" ]; then
>      PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
Paul Eggleton - March 14, 2012, 3:09 p.m.
On Wednesday 14 March 2012 07:52:01 Darren Hart wrote:
> Should this have some kind of prefix? "ERROR: " or something along those
> lines for consistency with other output?

I did wonder about that; in the end I elected to be consistent with the python 
version errors in the same script (although I was the one who added those 
IIRC... heh). Perhaps we ought to add the "ERROR: " prefix to all of the errors 
in the wrapper script.

Cheers,
Paul
Darren Hart - March 14, 2012, 3:19 p.m.
On 03/14/2012 08:09 AM, Paul Eggleton wrote:
> On Wednesday 14 March 2012 07:52:01 Darren Hart wrote:
>> Should this have some kind of prefix? "ERROR: " or something along those
>> lines for consistency with other output?
> 
> I did wonder about that; in the end I elected to be consistent with the python 
> version errors in the same script (although I was the one who added those 
> IIRC... heh). Perhaps we ought to add the "ERROR: " prefix to all of the errors 
> in the wrapper script.
> 

Consistency is king here IMO. I don't care much one way or the other, so
long as it's consistent.
Andreas Oberritter - March 15, 2012, 12:30 a.m.
Hello Paul,

On 14.03.2012 01:36, Paul Eggleton wrote:
> If the user is in any directory other than $BUILDDIR when the bitbake
> wrapper script is run, then show an error an exit.

this patch broke my setup.

My $BUILDDIR points to tmp, so that pseudo doesn't get rebuilt for every
machine. I have a shared tmp for many machines.

BUILDDIR doesn't seem to have any other use than pointing to the
'pseudodone' file. I don't understand why it's required to run bitbake
from there.

I'm using a tiny shell fragment to set up my build environment. This is
how it looks like:

export BUILDDIR=${PREFIX}/tmp
export
PATH=${PREFIX}/openembedded-core/scripts:${PREFIX}/bitbake/bin:${PATH}

local.conf and bblayers.conf for individual machines are stored in
${PREFIX}/build/${MACHINE}/conf.

Regards,
Andreas

> Fixes [YOCTO #2071].
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  scripts/bitbake |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/scripts/bitbake b/scripts/bitbake
> index dda3b26..45c8697 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -47,6 +47,11 @@ float_test() {
>  # but earlier versions do not
>  float_test "$TARVERSION > 1.23" && needtar="0"
>  
> +if [ "`pwd`" != "$BUILDDIR" ] ; then
> +    echo "BitBake must be run from your build directory: $BUILDDIR"
> +    exit 1
> +fi
> +
>  buildpseudo="1"
>  if [ $needpseudo = "1" ] && [ -e "$BUILDDIR/pseudodone" ]; then
>      PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
Paul Eggleton - March 15, 2012, 11:09 p.m.
On Thursday 15 March 2012 01:30:31 Andreas Oberritter wrote:
> On 14.03.2012 01:36, Paul Eggleton wrote:
> > If the user is in any directory other than $BUILDDIR when the bitbake
> > wrapper script is run, then show an error an exit.
> 
> this patch broke my setup.

Ah, sorry about that.
 
> My $BUILDDIR points to tmp, so that pseudo doesn't get rebuilt for every
> machine. I have a shared tmp for many machines.

So a couple of things:

1) Unless I'm missing something you can share the same TMPDIR between multiple 
build directories to get the same result.
 
2) pseudo is a native package. It shouldn't be rebuilt when changing MACHINE. 
In fact I just verified by creating a different build directory with only 
MACHINE changed in the config, it is not rebuilt.

> BUILDDIR doesn't seem to have any other use than pointing to the
> 'pseudodone' file. I don't understand why it's required to run bitbake
> from there.

Well, it's required that bitbake is run from the build directory and when you 
use the setup script as intended that's where BUILDDIR points to. I hadn't 
anticipated that anyone would be changing BUILDDIR to point to anything other 
than the build dir, however I don't really think it's a good idea to support 
that.

Cheers,
Paul
Andreas Oberritter - March 16, 2012, 12:20 a.m.
On 16.03.2012 00:09, Paul Eggleton wrote:
> On Thursday 15 March 2012 01:30:31 Andreas Oberritter wrote:
>> On 14.03.2012 01:36, Paul Eggleton wrote:
>>> If the user is in any directory other than $BUILDDIR when the bitbake
>>> wrapper script is run, then show an error an exit.
>>
>> this patch broke my setup.
> 
> Ah, sorry about that.
>  
>> My $BUILDDIR points to tmp, so that pseudo doesn't get rebuilt for every
>> machine. I have a shared tmp for many machines.
> 
> So a couple of things:
> 
> 1) Unless I'm missing something you can share the same TMPDIR between multiple 
> build directories to get the same result.
>  
> 2) pseudo is a native package. It shouldn't be rebuilt when changing MACHINE. 
> In fact I just verified by creating a different build directory with only 
> MACHINE changed in the config, it is not rebuilt.

Sorry, I messed up some details. In fact, pseudo-native doesn't get
rebuilt, but bitbake pseudo-native still gets executed for every
machine, unless $BUILDDIR/pseudodone is present and contains
PSEUDOBINDIR. This just wastes time and confuses users who receive a
message saying "Pseudo is not present but is required, building this
first before the main build".

>> BUILDDIR doesn't seem to have any other use than pointing to the
>> 'pseudodone' file. I don't understand why it's required to run bitbake
>> from there.
> 
> Well, it's required that bitbake is run from the build directory and when you 
> use the setup script as intended that's where BUILDDIR points to. I hadn't 
> anticipated that anyone would be changing BUILDDIR to point to anything other 
> than the build dir, however I don't really think it's a good idea to support 
> that.

That's because BUILDDIR has no meaning outside oe-core's setup scripts.
In scripts/bitbake, BUILDDIR would better be called PSEUDODONEDIR or
similar, because that's what the variable really means in this context
[1]. If I had a choice to override a more suitably named variable I
certainly would do that. ;-)

In order to verify that scripts/bitbake is called from the build
directory, you could as well just check whether $PWD/conf/local.conf exists.

I'm not using oe-core's scripts, because they make assumptions about the
directory layout that don't fit my project's needs. I think they are
overly complex and they display texts that seem to suit yocto's needs
but at least not mine.

Regards,
Andreas

[1]
$ grep BUILDDIR scripts/bitbake
if [ $needpseudo = "1" ] && [ -e "$BUILDDIR/pseudodone" ]; then
    PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
    echo $PSEUDOBINDIR > $BUILDDIR/pseudodone
    PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
Paul Eggleton - March 16, 2012, 10:56 a.m.
On Friday 16 March 2012 01:20:44 Andreas Oberritter wrote:
> Sorry, I messed up some details. In fact, pseudo-native doesn't get
> rebuilt, but bitbake pseudo-native still gets executed for every
> machine, unless $BUILDDIR/pseudodone is present and contains
> PSEUDOBINDIR. This just wastes time and confuses users who receive a
> message saying "Pseudo is not present but is required, building this
> first before the main build".

It takes a small amount of time just once. However if this is really an issue 
perhaps we could address it directly rather than hacking around it?

> >> BUILDDIR doesn't seem to have any other use than pointing to the
> >> 'pseudodone' file. I don't understand why it's required to run bitbake
> >> from there.
> > 
> > Well, it's required that bitbake is run from the build directory and when
> > you use the setup script as intended that's where BUILDDIR points to. I
> > hadn't anticipated that anyone would be changing BUILDDIR to point to
> > anything other than the build dir, however I don't really think it's a
> > good idea to support that.
> 
> That's because BUILDDIR has no meaning outside oe-core's setup scripts.
> In scripts/bitbake, BUILDDIR would better be called PSEUDODONEDIR or
> similar, because that's what the variable really means in this context

The naming is such that it should point to the build directory because that's 
where pseudodone is meant to be written. I think when it was introduced there 
was an idea that it might be useful in other contexts, thus the naming.

> In order to verify that scripts/bitbake is called from the build
> directory, you could as well just check whether $PWD/conf/local.conf exists.

Unfortunately it's not that simple. bitbake.conf pulls in local.conf using 
"include" and not "require", thus it doesn't actually have to exist anywhere.

> I'm not using oe-core's scripts, because they make assumptions about the
> directory layout that don't fit my project's needs. I think they are
> overly complex

Could you perhaps elaborate on what needs those scripts are not fulfilling?

> and they display texts that seem to suit yocto's needs but at least not
> mine.

We print these messages so that people know what to do and where to find 
further information. I don't think that's unreasonable. However if you have 
suggestions on how we might improve those messages or allow them to be 
customised if necessary, we could look at changing them.

Cheers,
Paul
VIJAY KUMAR - March 16, 2012, 11:55 a.m.
Hi paul,

   Iam not specialist in this. I am just following the steps mentioned here

     http://icedtea.classpath.org/wiki/CrossCompileOECoreTutorial

I am building from this path  vj@ubuntu:/host/coreembedded/oe-core/build$
and it contains pseudodone file.

and in vj@ubuntu:/host/coreembedded/oe-core/build/conf local.conf exits..

and the same is calling bitbake which is present in scripts..

Regards,
VJ

On Fri, Mar 16, 2012 at 4:26 PM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> On Friday 16 March 2012 01:20:44 Andreas Oberritter wrote:
> > Sorry, I messed up some details. In fact, pseudo-native doesn't get
> > rebuilt, but bitbake pseudo-native still gets executed for every
> > machine, unless $BUILDDIR/pseudodone is present and contains
> > PSEUDOBINDIR. This just wastes time and confuses users who receive a
> > message saying "Pseudo is not present but is required, building this
> > first before the main build".
>
> It takes a small amount of time just once. However if this is really an
> issue
> perhaps we could address it directly rather than hacking around it?
>
> > >> BUILDDIR doesn't seem to have any other use than pointing to the
> > >> 'pseudodone' file. I don't understand why it's required to run bitbake
> > >> from there.
> > >
> > > Well, it's required that bitbake is run from the build directory and
> when
> > > you use the setup script as intended that's where BUILDDIR points to. I
> > > hadn't anticipated that anyone would be changing BUILDDIR to point to
> > > anything other than the build dir, however I don't really think it's a
> > > good idea to support that.
> >
> > That's because BUILDDIR has no meaning outside oe-core's setup scripts.
> > In scripts/bitbake, BUILDDIR would better be called PSEUDODONEDIR or
> > similar, because that's what the variable really means in this context
>
> The naming is such that it should point to the build directory because
> that's
> where pseudodone is meant to be written. I think when it was introduced
> there
> was an idea that it might be useful in other contexts, thus the naming.
>
> > In order to verify that scripts/bitbake is called from the build
> > directory, you could as well just check whether $PWD/conf/local.conf
> exists.
>
> Unfortunately it's not that simple. bitbake.conf pulls in local.conf using
> "include" and not "require", thus it doesn't actually have to exist
> anywhere.
>
> > I'm not using oe-core's scripts, because they make assumptions about the
> > directory layout that don't fit my project's needs. I think they are
> > overly complex
>
> Could you perhaps elaborate on what needs those scripts are not fulfilling?
>
> > and they display texts that seem to suit yocto's needs but at least not
> > mine.
>
> We print these messages so that people know what to do and where to find
> further information. I don't think that's unreasonable. However if you have
> suggestions on how we might improve those messages or allow them to be
> customised if necessary, we could look at changing them.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
Paul Eggleton - March 16, 2012, 12:19 p.m.
Hi Vijay,

On Friday 16 March 2012 17:25:33 VIJAY KUMAR wrote:
>    Iam not specialist in this. I am just following the steps mentioned here
> 
>      http://icedtea.classpath.org/wiki/CrossCompileOECoreTutorial
> 
> I am building from this path  vj@ubuntu:/host/coreembedded/oe-core/build$
> and it contains pseudodone file.
> 
> and in vj@ubuntu:/host/coreembedded/oe-core/build/conf local.conf exits..
> 
> and the same is calling bitbake which is present in scripts..

FYI when you run bitbake after using . ./oe-init-build-env it sets up PATH so 
that "bitbake" runs the bitbake wrapper script in scripts/, that is intended.

Are you experiencing a problem relating to this?

Cheers,
Paul
Andreas Oberritter - March 18, 2012, 3:07 p.m.
On 16.03.2012 11:56, Paul Eggleton wrote:
> On Friday 16 March 2012 01:20:44 Andreas Oberritter wrote:
>> Sorry, I messed up some details. In fact, pseudo-native doesn't get
>> rebuilt, but bitbake pseudo-native still gets executed for every
>> machine, unless $BUILDDIR/pseudodone is present and contains
>> PSEUDOBINDIR. This just wastes time and confuses users who receive a
>> message saying "Pseudo is not present but is required, building this
>> first before the main build".
> 
> It takes a small amount of time just once. However if this is really an issue 
> perhaps we could address it directly rather than hacking around it?
> 
>>>> BUILDDIR doesn't seem to have any other use than pointing to the
>>>> 'pseudodone' file. I don't understand why it's required to run bitbake
>>>> from there.
>>>
>>> Well, it's required that bitbake is run from the build directory and when
>>> you use the setup script as intended that's where BUILDDIR points to. I
>>> hadn't anticipated that anyone would be changing BUILDDIR to point to
>>> anything other than the build dir, however I don't really think it's a
>>> good idea to support that.
>>
>> That's because BUILDDIR has no meaning outside oe-core's setup scripts.
>> In scripts/bitbake, BUILDDIR would better be called PSEUDODONEDIR or
>> similar, because that's what the variable really means in this context
> 
> The naming is such that it should point to the build directory because that's 
> where pseudodone is meant to be written. I think when it was introduced there 
> was an idea that it might be useful in other contexts, thus the naming.

What is the reason for storing 'pseudodone' inside BUILDDIR? Wouldn't it
better be stored inside TMPDIR by default? Isn't there any way to get
rid of pseudodone completely?

>> In order to verify that scripts/bitbake is called from the build
>> directory, you could as well just check whether $PWD/conf/local.conf exists.
> 
> Unfortunately it's not that simple. bitbake.conf pulls in local.conf using 
> "include" and not "require", thus it doesn't actually have to exist anywhere.

In fact it is that simple, because If local.conf doesn't exist,
oe-setup-builddir creates it.

>> I'm not using oe-core's scripts, because they make assumptions about the
>> directory layout that don't fit my project's needs. I think they are
>> overly complex
> 
> Could you perhaps elaborate on what needs those scripts are not fulfilling?

1.) OE-core is not my top-level directory. It's included as a git
submodule. it must not contain any data not fetched from the OE-core
repository (i.e. no bitbake tree, no build dirs).

2.) Switching between build directories must not involve changing the
environment.

>> and they display texts that seem to suit yocto's needs but at least not
>> mine.
> 
> We print these messages so that people know what to do and where to find 
> further information. I don't think that's unreasonable. However if you have 
> suggestions on how we might improve those messages or allow them to be 
> customised if necessary, we could look at changing them.

I'd like to display none of those messages to my users. We have our own
help texts that are specific to our distribution and configuration.

Our setup script is based on GNU make and uses make's dependency
tracking. It creates local.conf on its own. I don't want any other
script to do that.

I really don't want to be forced to use those OE-core scripts, if a
two-line script is sufficient to fulfil my needs. This two-line script
just sets up the path to bitbake and sets the location of 'pseudodone'
(which already sucks - the path to bitbake should be sufficient alone).

Regards,
Andreas
Khem Raj - April 2, 2012, 8 p.m.
On (14/03/12 00:36), Paul Eggleton wrote:
> If the user is in any directory other than $BUILDDIR when the bitbake
> wrapper script is run, then show an error an exit.
> 
> Fixes [YOCTO #2071].


angstrom e.g. does not have this restriction but uses script/bitbake
so this fix seems to be wrong here since its specific to OE-Core's
default build env.

> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  scripts/bitbake |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/scripts/bitbake b/scripts/bitbake
> index dda3b26..45c8697 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -47,6 +47,11 @@ float_test() {
>  # but earlier versions do not
>  float_test "$TARVERSION > 1.23" && needtar="0"
>  
> +if [ "`pwd`" != "$BUILDDIR" ] ; then
> +    echo "BitBake must be run from your build directory: $BUILDDIR"
> +    exit 1
> +fi
> +
>  buildpseudo="1"
>  if [ $needpseudo = "1" ] && [ -e "$BUILDDIR/pseudodone" ]; then
>      PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
> -- 
> 1.7.5.4
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Paul Eggleton - April 2, 2012, 8:12 p.m.
On Monday 02 April 2012 13:00:15 Khem Raj wrote:
> On (14/03/12 00:36), Paul Eggleton wrote:
> > If the user is in any directory other than $BUILDDIR when the bitbake
> > wrapper script is run, then show an error an exit.
> > 
> > Fixes [YOCTO #2071].
> 
> angstrom e.g. does not have this restriction but uses script/bitbake
> so this fix seems to be wrong here since its specific to OE-Core's
> default build env.

This has been implemented differently now:

http://cgit.openembedded.org/openembedded-core/commit/?id=769384decb095fb3c49eb13b8f7f69c978d0bcba

Cheers,
Paul
Chris Larson - April 2, 2012, 8:32 p.m.
On Mon, Apr 2, 2012 at 1:12 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Monday 02 April 2012 13:00:15 Khem Raj wrote:
>> On (14/03/12 00:36), Paul Eggleton wrote:
>> > If the user is in any directory other than $BUILDDIR when the bitbake
>> > wrapper script is run, then show an error an exit.
>> >
>> > Fixes [YOCTO #2071].
>>
>> angstrom e.g. does not have this restriction but uses script/bitbake
>> so this fix seems to be wrong here since its specific to OE-Core's
>> default build env.
>
> This has been implemented differently now:
>
> http://cgit.openembedded.org/openembedded-core/commit/?id=769384decb095fb3c49eb13b8f7f69c978d0bcba

This'll break the ability to run bitbake from subdirs underneath
BUILDDIR, which is perfectly valid, and supported by bitbake's code
which traverses up the current working path to find
conf/bblayers.conf.
Paul Eggleton - April 2, 2012, 8:56 p.m.
On Monday 02 April 2012 13:32:15 Chris Larson wrote:
> On Mon, Apr 2, 2012 at 1:12 PM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
> > This has been implemented differently now:
> > 
> > http://cgit.openembedded.org/openembedded-core/commit/?id=769384decb095fb3
> > c49eb13b8f7f69c978d0bcba
> This'll break the ability to run bitbake from subdirs underneath
> BUILDDIR, which is perfectly valid, and supported by bitbake's code
> which traverses up the current working path to find
> conf/bblayers.conf.

I don't know if you've tried this recently but it no longer works with current 
versions of OE-Core and bitbake (can't find local.conf). I'm not exactly sure 
why not and it's likely we should fix it, but right now it's broken anyway.

Cheers,
Paul

Patch

diff --git a/scripts/bitbake b/scripts/bitbake
index dda3b26..45c8697 100755
--- a/scripts/bitbake
+++ b/scripts/bitbake
@@ -47,6 +47,11 @@  float_test() {
 # but earlier versions do not
 float_test "$TARVERSION > 1.23" && needtar="0"
 
+if [ "`pwd`" != "$BUILDDIR" ] ; then
+    echo "BitBake must be run from your build directory: $BUILDDIR"
+    exit 1
+fi
+
 buildpseudo="1"
 if [ $needpseudo = "1" ] && [ -e "$BUILDDIR/pseudodone" ]; then
     PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`