Patchwork [RFC] license: Ensure we find multilib packages also

login
register
mail settings
Submitter Saul Wold
Date Sept. 13, 2012, 7:26 p.m.
Message ID <1347564379-14252-1-git-send-email-sgw@linux.intel.com>
Download mbox | patch
Permalink /patch/36497/
State New
Headers show

Comments

Saul Wold - Sept. 13, 2012, 7:26 p.m.
Make sure to find -package, this was causing a failure
in the multi-lib build license generation during rootfs.

Signed-off-by: Saul Wold <sgw@linux.intel.com>
---
 meta/classes/license.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Paul Eggleton - Sept. 13, 2012, 10:31 p.m.
On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
> Make sure to find -package, this was causing a failure
> in the multi-lib build license generation during rootfs.
> 
> Signed-off-by: Saul Wold <sgw@linux.intel.com>
> ---
>  meta/classes/license.bbclass |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> index 29fe938..b29067c 100644
> --- a/meta/classes/license.bbclass
> +++ b/meta/classes/license.bbclass
> @@ -88,7 +88,7 @@ license_create_manifest() {
>  	# list of installed packages is broken for deb
>  	for pkg in ${INSTALLED_PKGS}; do
>  		# not the best way to do this but licenses are not arch dependant iirc
> -		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
> +		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
>  		pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
> 
>  		# exclude locale recipes

Surely this could end up matching a the wrong file when one package name is a 
substring of another?

Cheers,
Paul
Saul Wold - Sept. 13, 2012, 10:43 p.m.
On 09/13/2012 03:31 PM, Paul Eggleton wrote:
> On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
>> Make sure to find -package, this was causing a failure
>> in the multi-lib build license generation during rootfs.
>>
>> Signed-off-by: Saul Wold <sgw@linux.intel.com>
>> ---
>>   meta/classes/license.bbclass |    2 +-
>>   1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
>> index 29fe938..b29067c 100644
>> --- a/meta/classes/license.bbclass
>> +++ b/meta/classes/license.bbclass
>> @@ -88,7 +88,7 @@ license_create_manifest() {
>>   	# list of installed packages is broken for deb
>>   	for pkg in ${INSTALLED_PKGS}; do
>>   		# not the best way to do this but licenses are not arch dependant iirc
>> -		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
>> +		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
>>   		pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
>>
>>   		# exclude locale recipes
>
> Surely this could end up matching a the wrong file when one package name is a
> substring of another?
>
I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be 
empty when rootfs runs, that did not help, I thought about adding the 
'-', but that would fail in the non-multilib case, how can I determine 
what the prefix will be to get a more accurate match?

Sau!

> Cheers,
> Paul
>
Richard Purdie - Sept. 14, 2012, 4:01 p.m.
On Thu, 2012-09-13 at 15:43 -0700, Saul Wold wrote:
> On 09/13/2012 03:31 PM, Paul Eggleton wrote:
> > On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
> >> Make sure to find -package, this was causing a failure
> >> in the multi-lib build license generation during rootfs.
> >>
> >> Signed-off-by: Saul Wold <sgw@linux.intel.com>
> >> ---
> >>   meta/classes/license.bbclass |    2 +-
> >>   1 files changed, 1 insertions(+), 1 deletions(-)
> >>
> >> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> >> index 29fe938..b29067c 100644
> >> --- a/meta/classes/license.bbclass
> >> +++ b/meta/classes/license.bbclass
> >> @@ -88,7 +88,7 @@ license_create_manifest() {
> >>   	# list of installed packages is broken for deb
> >>   	for pkg in ${INSTALLED_PKGS}; do
> >>   		# not the best way to do this but licenses are not arch dependant iirc
> >> -		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
> >> +		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
> >>   		pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
> >>
> >>   		# exclude locale recipes
> >
> > Surely this could end up matching a the wrong file when one package name is a
> > substring of another?
> >
> I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be 
> empty when rootfs runs, that did not help, I thought about adding the 
> '-', but that would fail in the non-multilib case, how can I determine 
> what the prefix will be to get a more accurate match?

I would observe here that this probably works for ipk and multilib and
only breaks for multilib+rpm. The reason is that rpm collapses the
namespace when it creates the package list so "lib32-bash" becomes bash.
I think this might be an error in however we generate the INSTALLED_PKGS
list and we might need to revisit the rpm mechanism and ensure the
multilib prefixes get added.

Cheers,

Richard
Mark Hatle - Sept. 14, 2012, 4:48 p.m.
On 9/14/12 11:01 AM, Richard Purdie wrote:
> On Thu, 2012-09-13 at 15:43 -0700, Saul Wold wrote:
>> On 09/13/2012 03:31 PM, Paul Eggleton wrote:
>>> On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
>>>> Make sure to find -package, this was causing a failure
>>>> in the multi-lib build license generation during rootfs.
>>>>
>>>> Signed-off-by: Saul Wold <sgw@linux.intel.com>
>>>> ---
>>>>    meta/classes/license.bbclass |    2 +-
>>>>    1 files changed, 1 insertions(+), 1 deletions(-)
>>>>
>>>> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
>>>> index 29fe938..b29067c 100644
>>>> --- a/meta/classes/license.bbclass
>>>> +++ b/meta/classes/license.bbclass
>>>> @@ -88,7 +88,7 @@ license_create_manifest() {
>>>>    	# list of installed packages is broken for deb
>>>>    	for pkg in ${INSTALLED_PKGS}; do
>>>>    		# not the best way to do this but licenses are not arch dependant iirc
>>>> -		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
>>>> +		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
>>>>    		pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
>>>>
>>>>    		# exclude locale recipes
>>>
>>> Surely this could end up matching a the wrong file when one package name is a
>>> substring of another?
>>>
>> I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be
>> empty when rootfs runs, that did not help, I thought about adding the
>> '-', but that would fail in the non-multilib case, how can I determine
>> what the prefix will be to get a more accurate match?
>
> I would observe here that this probably works for ipk and multilib and
> only breaks for multilib+rpm. The reason is that rpm collapses the
> namespace when it creates the package list so "lib32-bash" becomes bash.
> I think this might be an error in however we generate the INSTALLED_PKGS
> list and we might need to revisit the rpm mechanism and ensure the
> multilib prefixes get added.

RPM has internal mechanisms to tag both the filetypes and architectures as being 
different, unlike most of the other packaging formats which assume a package 
with the same name is the same.

The way to deal with this is via the mapping actions.  You should be able to map 
from ML_PREFIX-${PN} to ${PN}.${ARCH} and of course the reverse...

How it works....

So during package generation we know the package arch is unique and just strip 
off the mlprefix.  To do the reverse iterate over all of the available 
MULTILIBS, there is code in rootfs_rpm.bbclass already to do this.  Then 
reconstruct the package name based on the multilib configuration.

--Mark


> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
Elizabeth Flanagan - Sept. 14, 2012, 8:50 p.m.
On Fri, Sep 14, 2012 at 9:48 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 9/14/12 11:01 AM, Richard Purdie wrote:
>>
>> On Thu, 2012-09-13 at 15:43 -0700, Saul Wold wrote:
>>>
>>> On 09/13/2012 03:31 PM, Paul Eggleton wrote:
>>>>
>>>> On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
>>>>>
>>>>> Make sure to find -package, this was causing a failure
>>>>> in the multi-lib build license generation during rootfs.
>>>>>
>>>>> Signed-off-by: Saul Wold <sgw@linux.intel.com>
>>>>> ---
>>>>>    meta/classes/license.bbclass |    2 +-
>>>>>    1 files changed, 1 insertions(+), 1 deletions(-)
>>>>>
>>>>> diff --git a/meta/classes/license.bbclass
>>>>> b/meta/classes/license.bbclass
>>>>> index 29fe938..b29067c 100644
>>>>> --- a/meta/classes/license.bbclass
>>>>> +++ b/meta/classes/license.bbclass
>>>>> @@ -88,7 +88,7 @@ license_create_manifest() {
>>>>>         # list of installed packages is broken for deb
>>>>>         for pkg in ${INSTALLED_PKGS}; do
>>>>>                 # not the best way to do this but licenses are not arch
>>>>> dependant iirc
>>>>> -               filename=`ls
>>>>> ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
>>>>> +               filename=`ls
>>>>> ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
>>>>>                 pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
>>>>>
>>>>>                 # exclude locale recipes
>>>>
>>>>
>>>> Surely this could end up matching a the wrong file when one package name
>>>> is a
>>>> substring of another?
>>>>
>>> I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be
>>> empty when rootfs runs, that did not help, I thought about adding the
>>> '-', but that would fail in the non-multilib case, how can I determine
>>> what the prefix will be to get a more accurate match?

I'm not particularly happy with how we've been doing manifests for a
while now and have been hacking at a revamp of it for the past 2
months on and off.

We really shouldn't be relying on package data at all for license
manifests. As the manifests are relying on list_installed_packages,
they end up inaccurate anyway as list_installed_packages does exactly
that. List software installed via package. Not installed via the
package manager? Then you aren't in the manifest. This ends up missing
quite a few things necessary from a release engineering standpoint
(like, hey, modutils? The kernel? Not listed in the manifest. IMHO
this is wrong.)

As a release engineer, I kind of don't really care if the software on
my image was installed via the packaging system. I just want a list of
what is on the images so I can audit my images to ensure GPL
commitments.

One way I've been looking at fixing this is via event handlers. I'll
have some code for RFC on this hopefully soon, but a preview (which
may or may not work it's still a WIP) is at:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=eflanagan/license_manifest

One thing the patch I've been working on does other than creates
accurate manifests is it also tries to create a semi-compliant SPDX
1.0 manifest.

I know this is late in the cycle so it might not make 1.3, but this
may be something we want to look at to end up solving this issue once
and for all.

-b

>>
>>
>> I would observe here that this probably works for ipk and multilib and
>> only breaks for multilib+rpm. The reason is that rpm collapses the
>> namespace when it creates the package list so "lib32-bash" becomes bash.
>> I think this might be an error in however we generate the INSTALLED_PKGS
>> list and we might need to revisit the rpm mechanism and ensure the
>> multilib prefixes get added.
>
>
> RPM has internal mechanisms to tag both the filetypes and architectures as
> being different, unlike most of the other packaging formats which assume a
> package with the same name is the same.
>
> The way to deal with this is via the mapping actions.  You should be able to
> map from ML_PREFIX-${PN} to ${PN}.${ARCH} and of course the reverse...
>
> How it works....
>
> So during package generation we know the package arch is unique and just
> strip off the mlprefix.  To do the reverse iterate over all of the available
> MULTILIBS, there is code in rootfs_rpm.bbclass already to do this.  Then
> reconstruct the package name based on the multilib configuration.
>
> --Mark
>
>
>
>> Cheers,
>>
>> Richard
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Richard Purdie - Sept. 14, 2012, 10:09 p.m.
On Fri, 2012-09-14 at 13:50 -0700, Flanagan, Elizabeth wrote:
> On Fri, Sep 14, 2012 at 9:48 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> > On 9/14/12 11:01 AM, Richard Purdie wrote:
> >>
> >> On Thu, 2012-09-13 at 15:43 -0700, Saul Wold wrote:
> >>>
> >>> On 09/13/2012 03:31 PM, Paul Eggleton wrote:
> >>>>
> >>>> On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
> >>>>>
> >>>>> Make sure to find -package, this was causing a failure
> >>>>> in the multi-lib build license generation during rootfs.
> >>>>>
> >>>>> Signed-off-by: Saul Wold <sgw@linux.intel.com>
> >>>>> ---
> >>>>>    meta/classes/license.bbclass |    2 +-
> >>>>>    1 files changed, 1 insertions(+), 1 deletions(-)
> >>>>>
> >>>>> diff --git a/meta/classes/license.bbclass
> >>>>> b/meta/classes/license.bbclass
> >>>>> index 29fe938..b29067c 100644
> >>>>> --- a/meta/classes/license.bbclass
> >>>>> +++ b/meta/classes/license.bbclass
> >>>>> @@ -88,7 +88,7 @@ license_create_manifest() {
> >>>>>         # list of installed packages is broken for deb
> >>>>>         for pkg in ${INSTALLED_PKGS}; do
> >>>>>                 # not the best way to do this but licenses are not arch
> >>>>> dependant iirc
> >>>>> -               filename=`ls
> >>>>> ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
> >>>>> +               filename=`ls
> >>>>> ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
> >>>>>                 pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
> >>>>>
> >>>>>                 # exclude locale recipes
> >>>>
> >>>>
> >>>> Surely this could end up matching a the wrong file when one package name
> >>>> is a
> >>>> substring of another?
> >>>>
> >>> I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be
> >>> empty when rootfs runs, that did not help, I thought about adding the
> >>> '-', but that would fail in the non-multilib case, how can I determine
> >>> what the prefix will be to get a more accurate match?
> 
> I'm not particularly happy with how we've been doing manifests for a
> while now and have been hacking at a revamp of it for the past 2
> months on and off.
> 
> We really shouldn't be relying on package data at all for license
> manifests. As the manifests are relying on list_installed_packages,
> they end up inaccurate anyway as list_installed_packages does exactly
> that. List software installed via package. Not installed via the
> package manager? Then you aren't in the manifest. This ends up missing
> quite a few things necessary from a release engineering standpoint
> (like, hey, modutils? The kernel? Not listed in the manifest. IMHO
> this is wrong.)
> 
> As a release engineer, I kind of don't really care if the software on
> my image was installed via the packaging system. I just want a list of
> what is on the images so I can audit my images to ensure GPL
> commitments.
> 
> One way I've been looking at fixing this is via event handlers. I'll
> have some code for RFC on this hopefully soon, but a preview (which
> may or may not work it's still a WIP) is at:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=eflanagan/license_manifest
> 
> One thing the patch I've been working on does other than creates
> accurate manifests is it also tries to create a semi-compliant SPDX
> 1.0 manifest.
> 
> I know this is late in the cycle so it might not make 1.3, but this
> may be something we want to look at to end up solving this issue once
> and for all.

This depends whether this is a release manifest or an image manifest.
You likely want *both* under different circumstances. If the kernel is
installed in the image, it will be listed in the image manifest. We just
need to be really clear about what the manifest is for.

As for modutils, we use kmod now. Basically anything doing deployment
should have a manifest associated with it. That is do_deploy (kernel,
uboot, some small list of others) and do_rootfs. do_package_write_* are
also arguably deployment in another form, as are sstate feeds.

Cheers,

Richard
Paul Eggleton - Sept. 14, 2012, 10:11 p.m.
On Friday 14 September 2012 13:50:22 Flanagan, Elizabeth wrote:
> We really shouldn't be relying on package data at all for license
> manifests. As the manifests are relying on list_installed_packages,
> they end up inaccurate anyway as list_installed_packages does exactly
> that. List software installed via package. Not installed via the
> package manager? Then you aren't in the manifest. This ends up missing
> quite a few things necessary from a release engineering standpoint
> (like, hey, modutils? The kernel? Not listed in the manifest. IMHO
> this is wrong.)

What do you mean by modutils? If you mean the earlier suggestion that module-
init-tools was missing from the manifest, I'm fairly certain that was a 
misunderstanding because of it being replaced by kmod.

The thing is, we have no other means of accurately determining the contents of 
the image than the installed package list, given that the package manager is 
ultimately in charge of resolving and installing dependencies during image 
creation. The list may not cover additional files copied into tmp/deploy as 
part of building the image (kernel, bootloader, etc.) - however, that does not 
make it inaccurate as to the contents of the image, let's be clear about that. 

I agree we do need to write out the license information for these additional 
files; however, since they aren't actually *in* the image itself, I think we 
need to list these in a separate file. What happens for example if the system 
builds u-boot as a result of building my image, but I don't ever end up 
distributing that with the image?

As to the mechanism for picking these up, the only real possibility I can see 
is to hook into do_deploy; this is not currently straightforward though since 
that's not something that is implemented generically at the moment.

Cheers,
Paul
Elizabeth Flanagan - Sept. 14, 2012, 10:26 p.m.
On Fri, Sep 14, 2012 at 3:11 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Friday 14 September 2012 13:50:22 Flanagan, Elizabeth wrote:
>> We really shouldn't be relying on package data at all for license
>> manifests. As the manifests are relying on list_installed_packages,
>> they end up inaccurate anyway as list_installed_packages does exactly
>> that. List software installed via package. Not installed via the
>> package manager? Then you aren't in the manifest. This ends up missing
>> quite a few things necessary from a release engineering standpoint
>> (like, hey, modutils? The kernel? Not listed in the manifest. IMHO
>> this is wrong.)
>
> What do you mean by modutils? If you mean the earlier suggestion that module-
> init-tools was missing from the manifest, I'm fairly certain that was a
> misunderstanding because of it being replaced by kmod.
>

Correct. Even so, there are things obviously missing from the
manifests. Kernel is the most obvious.

> The thing is, we have no other means of accurately determining the contents of
> the image than the installed package list, given that the package manager is
> ultimately in charge of resolving and installing dependencies during image
> creation. The list may not cover additional files copied into tmp/deploy as
> part of building the image (kernel, bootloader, etc.) - however, that does not
> make it inaccurate as to the contents of the image, let's be clear about that.
>

No, but it's like the joke about the two software managers in a hot
air balloon asking the engineer on the group where they are only to be
told "In a hot air balloon". It's technically correct but the user
case I keep running into here is that people want to know what they're
deploying with a product. The expectation people have (and right or
wrong, it's what I keep running into) is that everything that's on the
hddimg is what is on the manifest.

> I agree we do need to write out the license information for these additional
> files; however, since they aren't actually *in* the image itself, I think we
> need to list these in a separate file. What happens for example if the system
> builds u-boot as a result of building my image, but I don't ever end up
> distributing that with the image?

Then you have a manifest that has software that you end up complying
with the GPL (when you don't really need to) verses having a manifest
that doesn't list software which you do need to comply with the GPL.
Both are wrong, but one is more wrong.

RP is right here. We probably need to list a few different use cases
here and support the most common ones.

>
> As to the mechanism for picking these up, the only real possibility I can see
> is to hook into do_deploy; this is not currently straightforward though since
> that's not something that is implemented generically at the moment.

I'm currently hooking into do_package because, as best I can figure it
out, everything ends up getting packaged, even if it's not installed
via package. I'm happy for other suggestions though.

-b

>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
Mark Hatle - Sept. 14, 2012, 10:31 p.m.
On 9/14/12 5:26 PM, Flanagan, Elizabeth wrote:
> On Fri, Sep 14, 2012 at 3:11 PM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
>> On Friday 14 September 2012 13:50:22 Flanagan, Elizabeth wrote:
>>> We really shouldn't be relying on package data at all for license
>>> manifests. As the manifests are relying on list_installed_packages,
>>> they end up inaccurate anyway as list_installed_packages does exactly
>>> that. List software installed via package. Not installed via the
>>> package manager? Then you aren't in the manifest. This ends up missing
>>> quite a few things necessary from a release engineering standpoint
>>> (like, hey, modutils? The kernel? Not listed in the manifest. IMHO
>>> this is wrong.)
>>
>> What do you mean by modutils? If you mean the earlier suggestion that module-
>> init-tools was missing from the manifest, I'm fairly certain that was a
>> misunderstanding because of it being replaced by kmod.
>>
>
> Correct. Even so, there are things obviously missing from the
> manifests. Kernel is the most obvious.
>
>> The thing is, we have no other means of accurately determining the contents of
>> the image than the installed package list, given that the package manager is
>> ultimately in charge of resolving and installing dependencies during image
>> creation. The list may not cover additional files copied into tmp/deploy as
>> part of building the image (kernel, bootloader, etc.) - however, that does not
>> make it inaccurate as to the contents of the image, let's be clear about that.
>>
>
> No, but it's like the joke about the two software managers in a hot
> air balloon asking the engineer on the group where they are only to be
> told "In a hot air balloon". It's technically correct but the user
> case I keep running into here is that people want to know what they're
> deploying with a product. The expectation people have (and right or
> wrong, it's what I keep running into) is that everything that's on the
> hddimg is what is on the manifest.

 From thinking about the deployment of the rootfs through images, it seems to me 
it should be the responsibility of the components populating the items to 
produce a manifest of what they've done.  Be it install a package, or copy a 
file.  Then at the end collect together these manifest fragments to describe 
exactly what was done (and where it was done to!).

>> I agree we do need to write out the license information for these additional
>> files; however, since they aren't actually *in* the image itself, I think we
>> need to list these in a separate file. What happens for example if the system
>> builds u-boot as a result of building my image, but I don't ever end up
>> distributing that with the image?
>
> Then you have a manifest that has software that you end up complying
> with the GPL (when you don't really need to) verses having a manifest
> that doesn't list software which you do need to comply with the GPL.
> Both are wrong, but one is more wrong.
>
> RP is right here. We probably need to list a few different use cases
> here and support the most common ones.
>
>>
>> As to the mechanism for picking these up, the only real possibility I can see
>> is to hook into do_deploy; this is not currently straightforward though since
>> that's not something that is implemented generically at the moment.
>
> I'm currently hooking into do_package because, as best I can figure it
> out, everything ends up getting packaged, even if it's not installed
> via package. I'm happy for other suggestions though.
>
> -b
>
>>
>> Cheers,
>> Paul
>>
>> --
>>
>> Paul Eggleton
>> Intel Open Source Technology Centre
>
>
>
Paul Eggleton - Sept. 14, 2012, 11:09 p.m.
On Friday 14 September 2012 15:26:14 Flanagan, Elizabeth wrote:
> > The thing is, we have no other means of accurately determining the
> > contents of the image than the installed package list, given that the
> > package manager is ultimately in charge of resolving and installing
> > dependencies during image creation. The list may not cover additional
> > files copied into tmp/deploy as part of building the image (kernel,
> > bootloader, etc.) - however, that does not make it inaccurate as to the
> > contents of the image, let's be clear about that.
>
> No, but it's like the joke about the two software managers in a hot
> air balloon asking the engineer on the group where they are only to be
> told "In a hot air balloon". It's technically correct but the user
> case I keep running into here is that people want to know what they're
> deploying with a product.

Again, I absolutely agree we need to report the licenses for these files. I'm 
just advocating not commingling two separate things.

> The expectation people have (and right or
> wrong, it's what I keep running into) is that everything that's on the
> hddimg is what is on the manifest.

hddimg files are a special situation anyway though. To handle those properly we 
would have to use some special license reporting in the same place where the 
hddimg is composed, because we cannot really tell for certain anywhere else in 
the code as to what is in it.

> > I agree we do need to write out the license information for these
> > additional files; however, since they aren't actually *in* the image
> > itself, I think we need to list these in a separate file. What happens
> > for example if the system builds u-boot as a result of building my image,
> > but I don't ever end up distributing that with the image?
> 
> Then you have a manifest that has software that you end up complying
> with the GPL (when you don't really need to) verses having a manifest
> that doesn't list software which you do need to comply with the GPL.
> Both are wrong, but one is more wrong.
>
> RP is right here. We probably need to list a few different use cases
> here and support the most common ones.

I think we can satisfy all of these cases in the manner I described - a 
manifest for the image (what we already have) and an additional manifest 
beside it that lists all of the additionally deployed files along with their 
license information. If the files are right next to eachother with the same 
base name there can't be any confusion.

> > As to the mechanism for picking these up, the only real possibility I can
> > see is to hook into do_deploy; this is not currently straightforward
> > though since that's not something that is implemented generically at the
> > moment.
>
> I'm currently hooking into do_package because, as best I can figure it
> out, everything ends up getting packaged, even if it's not installed
> via package. I'm happy for other suggestions though.

pkgdata (which writes files into TMPDIR/pkgdata) already hooks into do_package 
and already lists LICENSE on a per-package basis for every package written 
out. Clearly we can't just take absolutely every package produced as the list 
for the manifest though, because not all of that gets installed into the 
image. Another more serious issue is that it's not guaranteed that everything 
that gets deployed is packaged, so this probably wouldn't satisfactorily solve 
the issue anyway.

As far as I can see, do_deploy is the only way, since we want to know what is 
deployed and which recipe deployed it. We'll just have to adjust things so 
that there is a central point we can tap into for do_deploy in order to 
collect the desired information - I haven't looked into how yet but I don't 
anticipate it being too hard.

I think we're also going to have to add a specific call that can be made from 
anywhere during do_rootfs that adds a recipe to be reported in the manifest. 
We can use this for custom cases such as hddimg and provide it for anyone 
doing anything similar to call manually (e.g. adding non-packaged files into 
their image in a post-processing function - not really recommended but I know 
people do do it).

Cheers,
Paul

Patch

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 29fe938..b29067c 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -88,7 +88,7 @@  license_create_manifest() {
 	# list of installed packages is broken for deb
 	for pkg in ${INSTALLED_PKGS}; do
 		# not the best way to do this but licenses are not arch dependant iirc
-		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
+		filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
 		pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
 
 		# exclude locale recipes