Patchwork [2/21] Add directory information to the pkgdata files

login
register
mail settings
Submitter Mark Hatle
Date May 29, 2013, 3:09 p.m.
Message ID <1369840203-21898-3-git-send-email-mark.hatle@windriver.com>
Download mbox | patch
Permalink /patch/50707/
State New
Headers show

Comments

Mark Hatle - May 29, 2013, 3:09 p.m.
Add S(ource) and B(uild) directory information to the recipe pkgdata files.
This allows external tools to find the appropriate information, and be able
to easily access the corresponding sources and build directories.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/package.bbclass | 2 ++
 1 file changed, 2 insertions(+)
Paul Eggleton - May 29, 2013, 3:11 p.m.
On Wednesday 29 May 2013 10:09:44 Mark Hatle wrote:
> Add S(ource) and B(uild) directory information to the recipe pkgdata files.
> This allows external tools to find the appropriate information, and be able
> to easily access the corresponding sources and build directories.
> 
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
>  meta/classes/package.bbclass | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 02a1460..19b2b4c 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1124,6 +1124,8 @@ python emit_pkgdata() {
> 
>      data_file = pkgdatadir + d.expand("/${PN}" )
>      f = open(data_file, 'w')
> +    f.write("S: %s\n" % d.expand("${S}"))
> +    f.write("B: %s\n" % d.expand("${B}"))
>      f.write("PACKAGES: %s\n" % packages)
>      f.close()

I'm not sure I'm totally comfortable with this idea. External tools shouldn't 
necessarily expect to be able to poke into the source after packaging occurs - 
rm_work (if enabled) will remove the work directory, and separately sstate may 
restore the pkgdata and not the workdir. In both situations these values will 
be invalid.

If tools need to be able to find out the values of S and B then I think they 
ought to be querying them via bitbake.

Cheers,
Paul
Mark Hatle - May 29, 2013, 3:23 p.m.
On 5/29/13 10:11 AM, Paul Eggleton wrote:
> On Wednesday 29 May 2013 10:09:44 Mark Hatle wrote:
>> Add S(ource) and B(uild) directory information to the recipe pkgdata files.
>> This allows external tools to find the appropriate information, and be able
>> to easily access the corresponding sources and build directories.
>>
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>> ---
>>   meta/classes/package.bbclass | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
>> index 02a1460..19b2b4c 100644
>> --- a/meta/classes/package.bbclass
>> +++ b/meta/classes/package.bbclass
>> @@ -1124,6 +1124,8 @@ python emit_pkgdata() {
>>
>>       data_file = pkgdatadir + d.expand("/${PN}" )
>>       f = open(data_file, 'w')
>> +    f.write("S: %s\n" % d.expand("${S}"))
>> +    f.write("B: %s\n" % d.expand("${B}"))
>>       f.write("PACKAGES: %s\n" % packages)
>>       f.close()
>
> I'm not sure I'm totally comfortable with this idea. External tools shouldn't
> necessarily expect to be able to poke into the source after packaging occurs -
> rm_work (if enabled) will remove the work directory, and separately sstate may
> restore the pkgdata and not the workdir. In both situations these values will
> be invalid.
>
> If tools need to be able to find out the values of S and B then I think they
> ought to be querying them via bitbake.

When discussing this in the past, via IRC, it was mentioned that the pkgdata has 
all of the necessary data to figure out what has been built and much of the data 
that external (think eclipse based UI tools) need to figure out.

This particular case allows our tooling to not only list what was built, but 
allows the tools to go into the directory (B or S) and explore the sources, 
logs, etc.

If this is rejected, I'm fine, we'll just carry it forward on our own.. but I 
think it does have some generic value.

(Actually querying bitbake is what we started with, but when you query it 1000 
times.. it's pretty obvious this doesn't work, the time alone to query it for 
the 'B' and 'S' of each recipe is significant to the point of being unusable for 
external tools.)

--Mark

> Cheers,
> Paul
>
Martin Jansa - May 29, 2013, 3:59 p.m.
On Wed, May 29, 2013 at 10:09:44AM -0500, Mark Hatle wrote:
> Add S(ource) and B(uild) directory information to the recipe pkgdata files.
> This allows external tools to find the appropriate information, and be able
> to easily access the corresponding sources and build directories.
> 
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
>  meta/classes/package.bbclass | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 02a1460..19b2b4c 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1124,6 +1124,8 @@ python emit_pkgdata() {
>  
>      data_file = pkgdatadir + d.expand("/${PN}" )
>      f = open(data_file, 'w')
> +    f.write("S: %s\n" % d.expand("${S}"))
> +    f.write("B: %s\n" % d.expand("${B}"))
>      f.write("PACKAGES: %s\n" % packages)
>      f.close()

how does this interact with sstate? does setscene task create
pkgdata with right paths?
Mark Hatle - May 29, 2013, 4:30 p.m.
On 5/29/13 10:59 AM, Martin Jansa wrote:
> On Wed, May 29, 2013 at 10:09:44AM -0500, Mark Hatle wrote:
>> Add S(ource) and B(uild) directory information to the recipe pkgdata files.
>> This allows external tools to find the appropriate information, and be able
>> to easily access the corresponding sources and build directories.
>>
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>> ---
>>   meta/classes/package.bbclass | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
>> index 02a1460..19b2b4c 100644
>> --- a/meta/classes/package.bbclass
>> +++ b/meta/classes/package.bbclass
>> @@ -1124,6 +1124,8 @@ python emit_pkgdata() {
>>
>>       data_file = pkgdatadir + d.expand("/${PN}" )
>>       f = open(data_file, 'w')
>> +    f.write("S: %s\n" % d.expand("${S}"))
>> +    f.write("B: %s\n" % d.expand("${B}"))
>>       f.write("PACKAGES: %s\n" % packages)
>>       f.close()
>
> how does this interact with sstate? does setscene task create
> pkgdata with right paths?
>

The pkgdata is always current to the current build/work directory.  When the 
sstate-cache runs, and the associated pkgdata is dumped to the disk the S and B 
for the current directory is written.  But when sstate-cache is used, there is 
no extracted source (or build) components so those directories remain unresolved.

It's up to the tooling that are using these files to check if the directory 
exists, if it does not -- then using bitbake -c patch <recipe> will create it. 
(even in the sstate-cache case.)

--Mark
Khem Raj - May 29, 2013, 5:36 p.m.
On Wednesday, May 29, 2013, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 5/29/13 10:59 AM, Martin Jansa wrote:
>>
>> On Wed, May 29, 2013 at 10:09:44AM -0500, Mark Hatle wrote:
>>>
>>> Add S(ource) and B(uild) directory information to the recipe pkgdata
files.
>>> This allows external tools to find the appropriate information, and be
able
>>> to easily access the corresponding sources and build directories.
>>>
>>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>

Hi Mark

This won't work well when package is populated from sstate is there a way
for it to work seamlessly across sstate it might be useful


>>> ---
>>>   meta/classes/package.bbclass | 2 ++
>>>   1 file changed, 2 insertions(+)
>>>
>>> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
>>> index 02a1460..19b2b4c 100644
>>> --- a/meta/classes/package.bbclass
>>> +++ b/meta/classes/package.bbclass
>>> @@ -1124,6 +1124,8 @@ python emit_pkgdata() {
>>>
>>>       data_file = pkgdatadir + d.expand("/${PN}" )
>>>       f = open(data_file, 'w')
>>> +    f.write("S: %s\n" % d.expand("${S}"))
>>> +    f.write("B: %s\n" % d.expand("${B}"))
>>>       f.write("PACKAGES: %s\n" % packages)
>>>       f.close()
>>
>> how does this interact with sstate? does setscene task create
>> pkgdata with right paths?
>>
>
> The pkgdata is always current to the current build/work directory.  When
the sstate-cache runs, and the associated pkgdata is dumped to the disk the
S and B for the current directory is written.  But when sstate-cache is
used, there is no extracted source (or build) components so those
directories remain unresolved.
>
> It's up to the tooling that are using these files to check if the
directory exists, if it does not -- then using bitbake -c patch <recipe>
will create it. (even in the sstate-cache case.)
>
> --Mark
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
Phil Blundell - May 29, 2013, 7:59 p.m.
On Wed, 2013-05-29 at 11:30 -0500, Mark Hatle wrote:
> It's up to the tooling that are using these files to check if the directory 
> exists, if it does not -- then using bitbake -c patch <recipe> will create it. 
> (even in the sstate-cache case.)

I'm not sure whether checking that the directory exists is all that
safe; in the shared-sstate scenario I think it's conceivable that the
directory might exist but contain something else, or be unreadable due
to permissions, or disappear underneath you while you're using it.

And, of course, "bitbake -c patch ..." won't necessarily create the same
${S} that you got from pkgdata in that case, it will create it in your
local TMPDIR.

Also, the subject line for this patch (and several others from the batch
you just sent) is not in the right format.  Please see the guidelines on
the wiki.

p.
Mark Hatle - May 29, 2013, 8:44 p.m.
On 5/29/13 2:59 PM, Phil Blundell wrote:
> On Wed, 2013-05-29 at 11:30 -0500, Mark Hatle wrote:
>> It's up to the tooling that are using these files to check if the directory
>> exists, if it does not -- then using bitbake -c patch <recipe> will create it.
>> (even in the sstate-cache case.)
>
> I'm not sure whether checking that the directory exists is all that
> safe; in the shared-sstate scenario I think it's conceivable that the
> directory might exist but contain something else, or be unreadable due
> to permissions, or disappear underneath you while you're using it.
>
> And, of course, "bitbake -c patch ..." won't necessarily create the same
> ${S} that you got from pkgdata in that case, it will create it in your
> local TMPDIR.

On 5/29/13 12:36 PM, Khem Raj wrote:
 > Hi Mark
 >
 > This won't work well when package is populated from sstate is there a way for it
 > to work seamlessly across sstate it might be useful

I was looking at this again.  I thought the pkgdata was reconstructed each time 
in the current TMPDIR.  I didn't realize it was just restored from the 
sstate-cache.  (We've got other patches in layers that trigger pkgdata-like 
files to be generated as well.. and I must have gotten the two confused.)

So yes, I understand now that this won't work properly as the embedded path is 
outside of the TMPDIR.  The intention was that the path would change to match 
the current build directories TMPDIR.  And due to the magic of the sstate-cache 
checksumming, running the do_patch should result in the same sources...


So -if- others thought this was useful, I'll fix the patch to change the S/B on 
the sstate setup to the proper TMPDIR locations.  But so far I'm not hearing 
anyone who wants this -- so I'll just keep it local.

> Also, the subject line for this patch (and several others from the batch
> you just sent) is not in the right format.  Please see the guidelines on
> the wiki.

I missed that, I'll correct the commits summary lines and resubmit them.

Drop 2/21
Fixup 9, 15, & 18

Did I miss any?

--Mark

> p.
>
>
Paul Eggleton - May 30, 2013, 8:29 a.m.
On Wednesday 29 May 2013 15:44:58 Mark Hatle wrote:
> On 5/29/13 2:59 PM, Phil Blundell wrote:
> > On Wed, 2013-05-29 at 11:30 -0500, Mark Hatle wrote:
> >> It's up to the tooling that are using these files to check if the
> >> directory
> >> exists, if it does not -- then using bitbake -c patch <recipe> will
> >> create it. (even in the sstate-cache case.)
> > 
> > I'm not sure whether checking that the directory exists is all that
> > safe; in the shared-sstate scenario I think it's conceivable that the
> > directory might exist but contain something else, or be unreadable due
> > to permissions, or disappear underneath you while you're using it.
> > 
> > And, of course, "bitbake -c patch ..." won't necessarily create the same
> > ${S} that you got from pkgdata in that case, it will create it in your
> > local TMPDIR.
> 
> On 5/29/13 12:36 PM, Khem Raj wrote:
>  > Hi Mark
>  > 
>  > This won't work well when package is populated from sstate is there a way
>  > for it to work seamlessly across sstate it might be useful
> 
> I was looking at this again.  I thought the pkgdata was reconstructed each
> time in the current TMPDIR.  I didn't realize it was just restored from the
> sstate-cache.  (We've got other patches in layers that trigger pkgdata-like
> files to be generated as well.. and I must have gotten the two confused.)

I think this is new behaviour for 1.4, so that might explain some of the 
confusion. Previously the pkgdata generation happened in do_package, but now 
that do_rootfs and other places rely upon the presence of pkgdata, it is now a 
separate task so it can be accelerated by sstate and do_package doesn't need 
to be re-run.

Cheers,
Paul
Richard Purdie - May 30, 2013, 8:33 a.m.
On Thu, 2013-05-30 at 09:29 +0100, Paul Eggleton wrote:
> On Wednesday 29 May 2013 15:44:58 Mark Hatle wrote:
> > On 5/29/13 2:59 PM, Phil Blundell wrote:
> > > On Wed, 2013-05-29 at 11:30 -0500, Mark Hatle wrote:
> > >> It's up to the tooling that are using these files to check if the
> > >> directory
> > >> exists, if it does not -- then using bitbake -c patch <recipe> will
> > >> create it. (even in the sstate-cache case.)
> > > 
> > > I'm not sure whether checking that the directory exists is all that
> > > safe; in the shared-sstate scenario I think it's conceivable that the
> > > directory might exist but contain something else, or be unreadable due
> > > to permissions, or disappear underneath you while you're using it.
> > > 
> > > And, of course, "bitbake -c patch ..." won't necessarily create the same
> > > ${S} that you got from pkgdata in that case, it will create it in your
> > > local TMPDIR.
> > 
> > On 5/29/13 12:36 PM, Khem Raj wrote:
> >  > Hi Mark
> >  > 
> >  > This won't work well when package is populated from sstate is there a way
> >  > for it to work seamlessly across sstate it might be useful
> > 
> > I was looking at this again.  I thought the pkgdata was reconstructed each
> > time in the current TMPDIR.  I didn't realize it was just restored from the
> > sstate-cache.  (We've got other patches in layers that trigger pkgdata-like
> > files to be generated as well.. and I must have gotten the two confused.)
> 
> I think this is new behaviour for 1.4, so that might explain some of the 
> confusion. Previously the pkgdata generation happened in do_package, but now 
> that do_rootfs and other places rely upon the presence of pkgdata, it is now a 
> separate task so it can be accelerated by sstate and do_package doesn't need 
> to be re-run.

It moved from do_package to do_packagedata (or whatever its called) in
1.4 for performance reasons but it has always been installed from the
sstate cache since the beginning of sstate.

Cheers,

Richard

Patch

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 02a1460..19b2b4c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1124,6 +1124,8 @@  python emit_pkgdata() {
 
     data_file = pkgdatadir + d.expand("/${PN}" )
     f = open(data_file, 'w')
+    f.write("S: %s\n" % d.expand("${S}"))
+    f.write("B: %s\n" % d.expand("${B}"))
     f.write("PACKAGES: %s\n" % packages)
     f.close()