Patchwork [bitbake-devel,0/1] bitbake-whatchanged: print what is about to happen

The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:

  subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/whatchanged
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged

Robert Yang (1):
  bitbake-whatchanged: print what is about to happen

 bitbake/bin/bitbake-whatchanged | 334 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 334 insertions(+)
 create mode 100755 bitbake/bin/bitbake-whatchanged

On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
> The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
> 
>   subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
> +0100)
> 
> are available in the git repository at:
> 
>   git://git.pokylinux.org/poky-contrib robert/whatchanged
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
> 
> Robert Yang (1):
>   bitbake-whatchanged: print what is about to happen
> 
>  bitbake/bin/bitbake-whatchanged | 334
> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
>  create mode 100755 bitbake/bin/bitbake-whatchanged

I tried this on top of latest master, unfortunately what I got was the
following:

-------------- snip ----------------
Figuring out the TMPDIR ...
Moving the stamps to stamps.old ...
Generating the new stamps ... (need several minutes)
ERROR occurred!!! Recovering the stamps dir ...
Removing the newly generated stamps ...
Moving the stamps.old back to stamps ...
Traceback (most recent call last):
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>
    sys.exit(main())
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
    old_recon = recon_dict(old_dict)
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
    full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
NameError: global name 'fn' is not defined
-------------- snip ----------------

A couple of other things:

1) We ought to be able to assume that TMPDIR is the same regardless of 
the recipe specified; this avoids having to parse all of the recipes just to
get the value of this variable.

2) I'm a little concerned with the general approach - is there no way of
avoiding having to copy and move around the stamps directory? It seems
a little risky if nothing else.

Cheers,
Paul

On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
> On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
> > The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
> > 
> >   subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
> > +0100)
> > 
> > are available in the git repository at:
> > 
> >   git://git.pokylinux.org/poky-contrib robert/whatchanged
> >   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
> > 
> > Robert Yang (1):
> >   bitbake-whatchanged: print what is about to happen
> > 
> >  bitbake/bin/bitbake-whatchanged | 334
> > ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
> >  create mode 100755 bitbake/bin/bitbake-whatchanged
> 
> I tried this on top of latest master, unfortunately what I got was the
> following:
> 
> -------------- snip ----------------
> Figuring out the TMPDIR ...
> Moving the stamps to stamps.old ...
> Generating the new stamps ... (need several minutes)
> ERROR occurred!!! Recovering the stamps dir ...
> Removing the newly generated stamps ...
> Moving the stamps.old back to stamps ...
> Traceback (most recent call last):
>   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>
>     sys.exit(main())
>   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
>     old_recon = recon_dict(old_dict)
>   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
>     full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
> NameError: global name 'fn' is not defined
> -------------- snip ----------------
> 
> A couple of other things:
> 
> 1) We ought to be able to assume that TMPDIR is the same regardless of 
> the recipe specified; this avoids having to parse all of the recipes just to
> get the value of this variable.
> 
> 2) I'm a little concerned with the general approach - is there no way of
> avoiding having to copy and move around the stamps directory? It seems
> a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its
something people likely want in conjunction with that.

Cheers,

Richard

On 08/08/2012 05:01 AM, Richard Purdie wrote:
> On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
>> On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
>>> The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
>>>
>>>    subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
>>> +0100)
>>>
>>> are available in the git repository at:
>>>
>>>    git://git.pokylinux.org/poky-contrib robert/whatchanged
>>>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
>>>
>>> Robert Yang (1):
>>>    bitbake-whatchanged: print what is about to happen
>>>
>>>   bitbake/bin/bitbake-whatchanged | 334
>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
>>>   create mode 100755 bitbake/bin/bitbake-whatchanged
>>
>> I tried this on top of latest master, unfortunately what I got was the
>> following:
>>
>> -------------- snip ----------------
>> Figuring out the TMPDIR ...
>> Moving the stamps to stamps.old ...
>> Generating the new stamps ... (need several minutes)
>> ERROR occurred!!! Recovering the stamps dir ...
>> Removing the newly generated stamps ...
>> Moving the stamps.old back to stamps ...
>> Traceback (most recent call last):
>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>
>>      sys.exit(main())
>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
>>      old_recon = recon_dict(old_dict)
>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
>>      full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
>> NameError: global name 'fn' is not defined

Thanks Paul, I've fixed this one, my testing didn't get here before, it should
be the string 'fn', not the variable fn.

>> -------------- snip ----------------
>>
>> A couple of other things:
>>
>> 1) We ought to be able to assume that TMPDIR is the same regardless of
>> the recipe specified; this avoids having to parse all of the recipes just to
>> get the value of this variable.
>>

I'm sorry, I don't understand what did you mean here. it seems that what
I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
the beginning, then use it elsewhere.

>> 2) I'm a little concerned with the general approach - is there no way of
>> avoiding having to copy and move around the stamps directory? It seems
>> a little risky if nothing else.
>
> I think adding a parameter to -S would be a good move for this, its
> something people likely want in conjunction with that.
>

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
as far as I know, the "-S" is a bool option currently, it doesn't accept
an argument, I think that we have the following 2 solutions:

1) Modify the "-S" to accept an argument, but this may break the the usage
    of the "bitbake -S", the currently usage is:

    bitbake -S <recipe>

    We may change it to:

    bitbake -S <tmpdir>(or stampsdir) <recipe>

    But it seems that it's not easy differentiate the argument behind "-S".

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
    currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
    the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
    cause other problems.

I'd like to send a patch for 2) if you are OK with it.

// Robert


> Cheers,
>
> Richard
>
>
>
>
>
>

On Wednesday 08 August 2012 11:40:14 Robert Yang wrote:
> On 08/08/2012 05:01 AM, Richard Purdie wrote:
> > On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
> >> A couple of other things:
> >> 
> >> 1) We ought to be able to assume that TMPDIR is the same regardless of
> >> the recipe specified; this avoids having to parse all of the recipes just
> >> to get the value of this variable.
> 
> I'm sorry, I don't understand what did you mean here. it seems that what
> I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
> the beginning, then use it elsewhere.

Sorry, I wasn't being very clear. If you specify the recipe with bitbake -e, 
bitbake has to go through a parse of the recipes (retrieving from the cache if 
available of course, but even that still takes a few seconds). I'm suggesting 
you don't specify the recipe as TMPDIR shouldn't be recipe-specific, and save 
quite a bit of time.

> >> 2) I'm a little concerned with the general approach - is there no way of
> >> avoiding having to copy and move around the stamps directory? It seems
> >> a little risky if nothing else.
> > 
> > I think adding a parameter to -S would be a good move for this, its
> > something people likely want in conjunction with that.
> 
> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
> as far as I know, the "-S" is a bool option currently, it doesn't accept
> an argument, I think that we have the following 2 solutions:
> 
> 1) Modify the "-S" to accept an argument, but this may break the the usage
>     of the "bitbake -S", the currently usage is:
> 
>     bitbake -S <recipe>
> 
>     We may change it to:
> 
>     bitbake -S <tmpdir>(or stampsdir) <recipe>
> 
>     But it seems that it's not easy differentiate the argument behind "-S".

I have to admit I'm not sure what -S is currently being used for; so it's hard 
for me to comment on what might get broken if we change this. I suspect it's 
not being used very much at all though.
 
> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>     currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve
> it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it
> will cause other problems.
> 
> I'd like to send a patch for 2) if you are OK with it.

We already have a mechanism to allow through variables from the environment - 
BB_ENV_WHITELIST / BB_ENV_EXTRAWHITE, and we should make use of that in 
preference to os.getenv(). I think we would not want this change in stamp 
writing behaviour to take effect unless -S is being used though, which suggests 
it ought to be a new variable that is not checked unless -S has been specified.

Cheers,
Paul

On 08/08/2012 11:40 AM, Robert Yang wrote:
>
> On 08/08/2012 05:01 AM, Richard Purdie wrote:
>> On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
>>> On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
>>>> The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
>>>>
>>>>    subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
>>>> +0100)
>>>>
>>>> are available in the git repository at:
>>>>
>>>>    git://git.pokylinux.org/poky-contrib robert/whatchanged
>>>>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
>>>>
>>>> Robert Yang (1):
>>>>    bitbake-whatchanged: print what is about to happen
>>>>
>>>>   bitbake/bin/bitbake-whatchanged | 334
>>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
>>>>   create mode 100755 bitbake/bin/bitbake-whatchanged
>>>
>>> I tried this on top of latest master, unfortunately what I got was the
>>> following:
>>>
>>> -------------- snip ----------------
>>> Figuring out the TMPDIR ...
>>> Moving the stamps to stamps.old ...
>>> Generating the new stamps ... (need several minutes)
>>> ERROR occurred!!! Recovering the stamps dir ...
>>> Removing the newly generated stamps ...
>>> Moving the stamps.old back to stamps ...
>>> Traceback (most recent call last):
>>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in
>>> <module>
>>>      sys.exit(main())
>>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in
>>> main
>>>      old_recon = recon_dict(old_dict)
>>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in
>>> recon_dict
>>>      full_path_pre = "%s/%s" %
>>> (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
>>> NameError: global name 'fn' is not defined
>
> Thanks Paul, I've fixed this one, my testing didn't get here before, it should
> be the string 'fn', not the variable fn.
>
>>> -------------- snip ----------------
>>>
>>> A couple of other things:
>>>
>>> 1) We ought to be able to assume that TMPDIR is the same regardless of
>>> the recipe specified; this avoids having to parse all of the recipes just to
>>> get the value of this variable.
>>>
>
> I'm sorry, I don't understand what did you mean here. it seems that what
> I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
> the beginning, then use it elsewhere.
>
>>> 2) I'm a little concerned with the general approach - is there no way of
>>> avoiding having to copy and move around the stamps directory? It seems
>>> a little risky if nothing else.
>>
>> I think adding a parameter to -S would be a good move for this, its
>> something people likely want in conjunction with that.
>>

I just notices that added the variable STAMP to BB_ENV_EXTRAWHITE would make
it work, thanks, I missed this before, I will send an updated sooner.

// Robert

>
> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
> as far as I know, the "-S" is a bool option currently, it doesn't accept
> an argument, I think that we have the following 2 solutions:
>
> 1) Modify the "-S" to accept an argument, but this may break the the usage
>     of the "bitbake -S", the currently usage is:
>
>     bitbake -S <recipe>
>
>     We may change it to:
>
>     bitbake -S <tmpdir>(or stampsdir) <recipe>
>
>     But it seems that it's not easy differentiate the argument behind "-S".
>
> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>     currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
>     the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
>     cause other problems.
>
> I'd like to send a patch for 2) if you are OK with it.
>
> // Robert
>
>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>>
>>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>

On Wed, 2012-08-08 at 11:40 +0800, Robert Yang wrote:
> >> 2) I'm a little concerned with the general approach - is there no way of
> >> avoiding having to copy and move around the stamps directory? It seems
> >> a little risky if nothing else.
> >
> > I think adding a parameter to -S would be a good move for this, its
> > something people likely want in conjunction with that.
> >
> 
> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
> as far as I know, the "-S" is a bool option currently, it doesn't accept
> an argument, I think that we have the following 2 solutions:
> 
> 1) Modify the "-S" to accept an argument, but this may break the the usage
>     of the "bitbake -S", the currently usage is:
> 
>     bitbake -S <recipe>
> 
>     We may change it to:
> 
>     bitbake -S <tmpdir>(or stampsdir) <recipe>
> 
>     But it seems that it's not easy differentiate the argument behind "-S".

Hmm, we probably could change this option as long as we update the
manuals too. Ideally I would like some way to say "use the default
stamps directory" without having to put a full path in.

> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>     currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
>     the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
>     cause other problems.
> 
> I'd like to send a patch for 2) if you are OK with it.

Please don't use TMPDIR, bitbake has no knowledge of that variable, nor
should it have and it will change the cache directory. I'd suggest using
STAMP, that is what the variable is designed for. We just need to allow
it from the environment which Paul mentions and this is what I
originally proposed.

I'm leaning towards 2) and STAMP with BB_ENV_WHITELIST (or whatever the
variable is called) updated accordingly.

Cheers,

Richard

On 08/08/2012 05:12 PM, Paul Eggleton wrote:
> On Wednesday 08 August 2012 11:40:14 Robert Yang wrote:
>> On 08/08/2012 05:01 AM, Richard Purdie wrote:
>>> On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
>>>> A couple of other things:
>>>>
>>>> 1) We ought to be able to assume that TMPDIR is the same regardless of
>>>> the recipe specified; this avoids having to parse all of the recipes just
>>>> to get the value of this variable.
>>
>> I'm sorry, I don't understand what did you mean here. it seems that what
>> I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
>> the beginning, then use it elsewhere.
>
> Sorry, I wasn't being very clear. If you specify the recipe with bitbake -e,
> bitbake has to go through a parse of the recipes (retrieving from the cache if
> available of course, but even that still takes a few seconds). I'm suggesting
> you don't specify the recipe as TMPDIR shouldn't be recipe-specific, and save
> quite a bit of time.
>
>>>> 2) I'm a little concerned with the general approach - is there no way of
>>>> avoiding having to copy and move around the stamps directory? It seems
>>>> a little risky if nothing else.
>>>
>>> I think adding a parameter to -S would be a good move for this, its
>>> something people likely want in conjunction with that.
>>
>> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
>> as far as I know, the "-S" is a bool option currently, it doesn't accept
>> an argument, I think that we have the following 2 solutions:
>>

Got it, thanks.

>> 1) Modify the "-S" to accept an argument, but this may break the the usage
>>      of the "bitbake -S", the currently usage is:
>>
>>      bitbake -S <recipe>
>>
>>      We may change it to:
>>
>>      bitbake -S <tmpdir>(or stampsdir) <recipe>
>>
>>      But it seems that it's not easy differentiate the argument behind "-S".
>
> I have to admit I'm not sure what -S is currently being used for; so it's hard
> for me to comment on what might get broken if we change this. I suspect it's
> not being used very much at all though.
>
>> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>>      currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve
>> it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it
>> will cause other problems.
>>
>> I'd like to send a patch for 2) if you are OK with it.
>
> We already have a mechanism to allow through variables from the environment -
> BB_ENV_WHITELIST / BB_ENV_EXTRAWHITE, and we should make use of that in
> preference to os.getenv(). I think we would not want this change in stamp
> writing behaviour to take effect unless -S is being used though, which suggests
> it ought to be a new variable that is not checked unless -S has been specified.
>

Thank you very much, I will add a BB_DUMP_STAMP.

// Robert

> Cheers,
> Paul
>

On 08/08/2012 05:21 PM, Richard Purdie wrote:
> On Wed, 2012-08-08 at 11:40 +0800, Robert Yang wrote:
>>>> 2) I'm a little concerned with the general approach - is there no way of
>>>> avoiding having to copy and move around the stamps directory? It seems
>>>> a little risky if nothing else.
>>>
>>> I think adding a parameter to -S would be a good move for this, its
>>> something people likely want in conjunction with that.
>>>
>>
>> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
>> as far as I know, the "-S" is a bool option currently, it doesn't accept
>> an argument, I think that we have the following 2 solutions:
>>
>> 1) Modify the "-S" to accept an argument, but this may break the the usage
>>      of the "bitbake -S", the currently usage is:
>>
>>      bitbake -S <recipe>
>>
>>      We may change it to:
>>
>>      bitbake -S <tmpdir>(or stampsdir) <recipe>
>>
>>      But it seems that it's not easy differentiate the argument behind "-S".
>
> Hmm, we probably could change this option as long as we update the
> manuals too. Ideally I would like some way to say "use the default
> stamps directory" without having to put a full path in.
>
>> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>>      currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
>>      the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
>>      cause other problems.
>>
>> I'd like to send a patch for 2) if you are OK with it.
>
> Please don't use TMPDIR, bitbake has no knowledge of that variable, nor
> should it have and it will change the cache directory. I'd suggest using
> STAMP, that is what the variable is designed for. We just need to allow
> it from the environment which Paul mentions and this is what I
> originally proposed.
>
> I'm leaning towards 2) and STAMP with BB_ENV_WHITELIST (or whatever the
> variable is called) updated accordingly.
>

Thank you very much, the STAMP is:

STAMP = "${TMPDIR}/stamps/${MULTIMACH_TARGET_SYS}/${PF}"

How about add a variable STAMP_DIR, it would be:

STAMP_DIR ?= "${TMPDIR}/stamps/"
STAMP = "${STAMP_DIR}/${MULTIMACH_TARGET_SYS}/${PF}"

Then add STAMP_DIR to BB_ENV_WHITELIST and we can run bitbake -S like:
STAMP_DIR=/what/we/want bitbake -S

Thanks

// Robert


> Cheers,
>
> Richard
>
>
>