Patchwork [0/1,v3] license.bbclass parsing changes rebased sans debug

login
register
mail settings
Submitter Elizabeth Flanagan
Date May 27, 2011, 10:13 p.m.
Message ID <4DE02216.1010003@intel.com>
Download mbox
Permalink /patch/4939/
State New, archived
Headers show

Pull-request

git://git.pokylinux.org/poky-contrib eflanagan/common-licenses

Comments

Elizabeth Flanagan - May 27, 2011, 10:13 p.m.
Using python abstract syntax trees as a parser and an ast visitor class, I've
begun the first steps to being able to put some sanity to license selection.

One note. License modifiers have traditionally been limited to the "or greater"
modifier "+". I've added another modifier "with exceptions" for licenses that
have no generic ie common-licenses/GPL-2,0-with-GCC-exception

The following changes since commit 62e3f0ff7b27d8beb21b4b226051c3db654ea4e4:

   License Field Cleanup: Non-standard field names (2011-05-27 18:27:22 +0100)

are available in the git repository at:
   git://git.pokylinux.org/poky-contrib eflanagan/common-licenses
   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/common-licenses

Beth Flanagan (1):
   license.bbclass: Sane Parsing of licenses

  meta/classes/license.bbclass |  152 +++++++++++++++++++++++++++++++-----------
  1 files changed, 114 insertions(+), 38 deletions(-)
Richard Purdie - May 27, 2011, 10:36 p.m.
On Fri, 2011-05-27 at 15:13 -0700, Elizabeth Flanagan wrote:
> Using python abstract syntax trees as a parser and an ast visitor class, I've
> begun the first steps to being able to put some sanity to license selection.
> 
> One note. License modifiers have traditionally been limited to the "or greater"
> modifier "+". I've added another modifier "with exceptions" for licenses that
> have no generic ie common-licenses/GPL-2,0-with-GCC-exception
> 
> The following changes since commit 62e3f0ff7b27d8beb21b4b226051c3db654ea4e4:
> 
>    License Field Cleanup: Non-standard field names (2011-05-27 18:27:22 +0100)
> 
> are available in the git repository at:
>    git://git.pokylinux.org/poky-contrib eflanagan/common-licenses
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/common-licenses
> 
> Beth Flanagan (1):
>    license.bbclass: Sane Parsing of licenses

Merged to master, thanks.

Richard
Koen Kooi - May 28, 2011, 7:45 a.m.
Op 28 mei 2011, om 00:13 heeft Elizabeth Flanagan het volgende geschreven:

> 
> Using python abstract syntax trees as a parser and an ast visitor class, I've
> begun the first steps to being able to put some sanity to license selection.
> 
> One note. License modifiers have traditionally been limited to the "or greater"
> modifier "+". I've added another modifier "with exceptions" for licenses that
> have no generic ie common-licenses/GPL-2,0-with-GCC-exception

I get this now:

WARNING: /OE/tentacle/sources/openembedded-core/meta/files/common-licenses
WARNING: We need to use a SPDXLICENSEMAP for GPLv2

Is that really a warning, should that be a note or even complete hidden from the console?

regards,

Koen
Koen Kooi - May 28, 2011, 9:16 a.m.
Op 28 mei 2011, om 00:13 heeft Elizabeth Flanagan het volgende geschreven:

> 
> Using python abstract syntax trees as a parser and an ast visitor class, I've
> begun the first steps to being able to put some sanity to license selection.
> 
> One note. License modifiers have traditionally been limited to the "or greater"
> modifier "+". I've added another modifier "with exceptions" for licenses that
> have no generic ie common-licenses/GPL-2,0-with-GCC-exception

When building shadow I get:

| WARNING: /home/koen/angstrom-core/sources/openembedded-core/meta/files/common-licenses
| ERROR: Error executing a python function in /home/koen/angstrom-core/sources/openembedded-core/meta/recipes-extended/shadow/shadow_4.1.4.3.bb:
| RuntimeError: maximum recursion depth exceeded while calling a Python object
| 
| ERROR: The stack trace of python calls that resulted in this exception/failure was:
| ERROR:   File "do_populate_lic", line 121, in <module>
| ERROR:
| ERROR:   File "do_populate_lic", line 118, in do_populate_lic
| ERROR:
| ERROR:   File "/usr/lib64/python2.7/ast.py", line 243, in visit
| ERROR:     return visitor(node)
| ERROR:
| ERROR:   File "do_populate_lic", line 13, in generic_visit

and a few hundred more lines, I had to change the scrollback buffer from 512 lines to unlimited to catch this :) I can send the full log on request.
Elizabeth Flanagan - May 28, 2011, 1:21 p.m.
Could you send me the log. That's odd and shouldn't have happened (the license is only two licenses...)

-b

On 05/28/2011 02:16 AM, Koen Kooi wrote:
>
> Op 28 mei 2011, om 00:13 heeft Elizabeth Flanagan het volgende geschreven:
>
>>
>> Using python abstract syntax trees as a parser and an ast visitor class, I've
>> begun the first steps to being able to put some sanity to license selection.
>>
>> One note. License modifiers have traditionally been limited to the "or greater"
>> modifier "+". I've added another modifier "with exceptions" for licenses that
>> have no generic ie common-licenses/GPL-2,0-with-GCC-exception
>
> When building shadow I get:
>
> | WARNING: /home/koen/angstrom-core/sources/openembedded-core/meta/files/common-licenses
> | ERROR: Error executing a python function in /home/koen/angstrom-core/sources/openembedded-core/meta/recipes-extended/shadow/shadow_4.1.4.3.bb:
> | RuntimeError: maximum recursion depth exceeded while calling a Python object
> |
> | ERROR: The stack trace of python calls that resulted in this exception/failure was:
> | ERROR:   File "do_populate_lic", line 121, in<module>
> | ERROR:
> | ERROR:   File "do_populate_lic", line 118, in do_populate_lic
> | ERROR:
> | ERROR:   File "/usr/lib64/python2.7/ast.py", line 243, in visit
> | ERROR:     return visitor(node)
> | ERROR:
> | ERROR:   File "do_populate_lic", line 13, in generic_visit
>
> and a few hundred more lines, I had to change the scrollback buffer from 512 lines to unlimited to catch this :) I can send the full log on request.
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
Koen Kooi - May 28, 2011, 1:41 p.m.
Here you go :)
Op 28 mei 2011, om 15:21 heeft Elizabeth Flanagan het volgende geschreven:

> Could you send me the log. That's odd and shouldn't have happened (the license is only two licenses...)
> 
> -b
> 
> On 05/28/2011 02:16 AM, Koen Kooi wrote:
>> 
>> Op 28 mei 2011, om 00:13 heeft Elizabeth Flanagan het volgende geschreven:
>> 
>>> 
>>> Using python abstract syntax trees as a parser and an ast visitor class, I've
>>> begun the first steps to being able to put some sanity to license selection.
>>> 
>>> One note. License modifiers have traditionally been limited to the "or greater"
>>> modifier "+". I've added another modifier "with exceptions" for licenses that
>>> have no generic ie common-licenses/GPL-2,0-with-GCC-exception
>> 
>> When building shadow I get:
>> 
>> | WARNING: /home/koen/angstrom-core/sources/openembedded-core/meta/files/common-licenses
>> | ERROR: Error executing a python function in /home/koen/angstrom-core/sources/openembedded-core/meta/recipes-extended/shadow/shadow_4.1.4.3.bb:
>> | RuntimeError: maximum recursion depth exceeded while calling a Python object
>> |
>> | ERROR: The stack trace of python calls that resulted in this exception/failure was:
>> | ERROR:   File "do_populate_lic", line 121, in<module>
>> | ERROR:
>> | ERROR:   File "do_populate_lic", line 118, in do_populate_lic
>> | ERROR:
>> | ERROR:   File "/usr/lib64/python2.7/ast.py", line 243, in visit
>> | ERROR:     return visitor(node)
>> | ERROR:
>> | ERROR:   File "do_populate_lic", line 13, in generic_visit
>> 
>> and a few hundred more lines, I had to change the scrollback buffer from 512 lines to unlimited to catch this :) I can send the full log on request.
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>> 
> 
> -- 
> ---------------
> Elizabeth Flanagan
> Yocto Project
> Release Engineer
Khem Raj - May 28, 2011, 4:55 p.m.
On 5/28/2011 2:16 AM, Koen Kooi wrote:
>
> Op 28 mei 2011, om 00:13 heeft Elizabeth Flanagan het volgende geschreven:
>
>>
>> Using python abstract syntax trees as a parser and an ast visitor class, I've
>> begun the first steps to being able to put some sanity to license selection.
>>
>> One note. License modifiers have traditionally been limited to the "or greater"
>> modifier "+". I've added another modifier "with exceptions" for licenses that
>> have no generic ie common-licenses/GPL-2,0-with-GCC-exception
>
> When building shadow I get:


and when building elfutils I get

ERROR: (file: 'do_populate_lic', lineno: 116, function: do_populate_lic)
ERROR: Function 'do_populate_lic' failed
ERROR: Logfile of failure stored in: 
/home/kraj/work/slugos/build/tmp-slugos-eglibc/work/armv5te-oe-linux-gnueabi/elfutils-0.148-r2/temp/log.do_populate_lic.31784
Log data follows:
| WARNING: 
/home/kraj/work/slugos/openembedded-core/meta/files/common-licenses
| ERROR: Error executing a python function in 
/home/kraj/work/slugos/openembedded-core/meta/recipes-devtools/elfutils/elfutils_0.148.bb:
|   File "<unknown>", line 1
|     ( 'GPLv2' & 'Exception' )
|    ^
| IndentationError: unexpected indent
|
| ERROR: The stack trace of python calls that resulted in this 
exception/failure was:
| ERROR:   File "do_populate_lic", line 121, in <module>
| ERROR:
| ERROR:   File "do_populate_lic", line 116, in do_populate_lic
| ERROR:
| ERROR:   File "/usr/lib/python2.7/ast.py", line 37, in parse
| ERROR:     return compile(source, filename, mode, PyCF_ONLY_AST)
| ERROR:
| ERROR: The code that was being executed was:
| ERROR:      0117:    v = LicenseVisitor()
| ERROR:      0118:    v.visit(node)
| ERROR:      0119:
| ERROR:      0120:
| ERROR:  *** 0121:do_populate_lic(d)
| ERROR:      0122:
| ERROR: (file: 'do_populate_lic', lineno: 121, function: <module>)
| ERROR:      0112:            clean_licenses += "'" + x + "'"
| ERROR:      0113:        else:
| ERROR:      0114:            clean_licenses += " " + x + " "
| ERROR:      0115:
| ERROR:  *** 0116:    node = ast.parse(clean_licenses)
| ERROR:      0117:    v = LicenseVisitor()
| ERROR:      0118:    v.visit(node)
| ERROR:      0119:
| ERROR:      0120:
| ERROR: (file: 'do_populate_lic', lineno: 116, function: do_populate_lic)
| ERROR: Function 'do_populate_lic' failed
NOTE: package elfutils-0.148-r2: task do_populate_lic: Failed


>
> | WARNING: /home/koen/angstrom-core/sources/openembedded-core/meta/files/common-licenses
> | ERROR: Error executing a python function in /home/koen/angstrom-core/sources/openembedded-core/meta/recipes-extended/shadow/shadow_4.1.4.3.bb:
> | RuntimeError: maximum recursion depth exceeded while calling a Python object
> |
> | ERROR: The stack trace of python calls that resulted in this exception/failure was:
> | ERROR:   File "do_populate_lic", line 121, in<module>
> | ERROR:
> | ERROR:   File "do_populate_lic", line 118, in do_populate_lic
> | ERROR:
> | ERROR:   File "/usr/lib64/python2.7/ast.py", line 243, in visit
> | ERROR:     return visitor(node)
> | ERROR:
> | ERROR:   File "do_populate_lic", line 13, in generic_visit
>
> and a few hundred more lines, I had to change the scrollback buffer from 512 lines to unlimited to catch this :) I can send the full log on request.
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Elizabeth Flanagan - May 28, 2011, 5:54 p.m.
In this case, elfutils should probably be changes as well as license.bbclass. As do_populate_lic is looking for valid 
python to make a syntax tree from, this will fail. Solution:

1. Fix license.bbclass to check LICENSE and massage it a bit more.
2. Fix elfutils to be GPLv2* (unless it's the GCC exception. I haven't read the license yet).

I'll poke around this today.

-b

On 05/28/2011 09:55 AM, Khem Raj wrote:
> On 5/28/2011 2:16 AM, Koen Kooi wrote:
>>
>> Op 28 mei 2011, om 00:13 heeft Elizabeth Flanagan het volgende geschreven:
>>
>>>
>>> Using python abstract syntax trees as a parser and an ast visitor class, I've
>>> begun the first steps to being able to put some sanity to license selection.
>>>
>>> One note. License modifiers have traditionally been limited to the "or greater"
>>> modifier "+". I've added another modifier "with exceptions" for licenses that
>>> have no generic ie common-licenses/GPL-2,0-with-GCC-exception
>>
>> When building shadow I get:
>
>
> and when building elfutils I get
>
> ERROR: (file: 'do_populate_lic', lineno: 116, function: do_populate_lic)
> ERROR: Function 'do_populate_lic' failed
> ERROR: Logfile of failure stored in:
> /home/kraj/work/slugos/build/tmp-slugos-eglibc/work/armv5te-oe-linux-gnueabi/elfutils-0.148-r2/temp/log.do_populate_lic.31784
> Log data follows:
> | WARNING:
> /home/kraj/work/slugos/openembedded-core/meta/files/common-licenses
> | ERROR: Error executing a python function in
> /home/kraj/work/slugos/openembedded-core/meta/recipes-devtools/elfutils/elfutils_0.148.bb:
> |   File "<unknown>", line 1
> |     ( 'GPLv2'&  'Exception' )
> |    ^
> | IndentationError: unexpected indent
> |
> | ERROR: The stack trace of python calls that resulted in this
> exception/failure was:
> | ERROR:   File "do_populate_lic", line 121, in<module>
> | ERROR:
> | ERROR:   File "do_populate_lic", line 116, in do_populate_lic
> | ERROR:
> | ERROR:   File "/usr/lib/python2.7/ast.py", line 37, in parse
> | ERROR:     return compile(source, filename, mode, PyCF_ONLY_AST)
> | ERROR:
> | ERROR: The code that was being executed was:
> | ERROR:      0117:    v = LicenseVisitor()
> | ERROR:      0118:    v.visit(node)
> | ERROR:      0119:
> | ERROR:      0120:
> | ERROR:  *** 0121:do_populate_lic(d)
> | ERROR:      0122:
> | ERROR: (file: 'do_populate_lic', lineno: 121, function:<module>)
> | ERROR:      0112:            clean_licenses += "'" + x + "'"
> | ERROR:      0113:        else:
> | ERROR:      0114:            clean_licenses += " " + x + " "
> | ERROR:      0115:
> | ERROR:  *** 0116:    node = ast.parse(clean_licenses)
> | ERROR:      0117:    v = LicenseVisitor()
> | ERROR:      0118:    v.visit(node)
> | ERROR:      0119:
> | ERROR:      0120:
> | ERROR: (file: 'do_populate_lic', lineno: 116, function: do_populate_lic)
> | ERROR: Function 'do_populate_lic' failed
> NOTE: package elfutils-0.148-r2: task do_populate_lic: Failed
>
>
>>
>> | WARNING: /home/koen/angstrom-core/sources/openembedded-core/meta/files/common-licenses
>> | ERROR: Error executing a python function in /home/koen/angstrom-core/sources/openembedded-core/meta/recipes-extended/shadow/shadow_4.1.4.3.bb:
>> | RuntimeError: maximum recursion depth exceeded while calling a Python object
>> |
>> | ERROR: The stack trace of python calls that resulted in this exception/failure was:
>> | ERROR:   File "do_populate_lic", line 121, in<module>
>> | ERROR:
>> | ERROR:   File "do_populate_lic", line 118, in do_populate_lic
>> | ERROR:
>> | ERROR:   File "/usr/lib64/python2.7/ast.py", line 243, in visit
>> | ERROR:     return visitor(node)
>> | ERROR:
>> | ERROR:   File "do_populate_lic", line 13, in generic_visit
>>
>> and a few hundred more lines, I had to change the scrollback buffer from 512 lines to unlimited to catch this :) I can send the full log on request.
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
Otavio Salvador - May 28, 2011, 6:19 p.m.
On Fri, May 27, 2011 at 22:13, Elizabeth Flanagan
<elizabeth.flanagan@intel.com> wrote:
...
> Beth Flanagan (1):
>  license.bbclass: Sane Parsing of licenses
...

The change itself was OK and really an improvement but please pay some
attention to whitespaces. It is nice to avoid useless whitespaces into
new code.
Khem Raj - May 28, 2011, 8:57 p.m.
On 5/28/2011 10:54 AM, Elizabeth Flanagan wrote:
> In this case, elfutils should probably be changes as well as
> license.bbclass. As do_populate_lic is looking for valid python to make
> a syntax tree from, this will fail. Solution:
>
> 1. Fix license.bbclass to check LICENSE and massage it a bit more.
> 2. Fix elfutils to be GPLv2* (unless it's the GCC exception. I haven't
> read the license yet).

It does not like () around license values so simple solution is to replace

-LICENSE = "(GPLv2 & Exception)"
+LICENSE = "GPLv2 & Exception"

Generally its preferable to fix eflutils first before license class in 
this case to keep the tree bisect'able.

>
> I'll poke around this today.
>
> -b
>
> On 05/28/2011 09:55 AM, Khem Raj wrote:
>> On 5/28/2011 2:16 AM, Koen Kooi wrote:
>>>
>>> Op 28 mei 2011, om 00:13 heeft Elizabeth Flanagan het volgende
>>> geschreven:
>>>
>>>>
>>>> Using python abstract syntax trees as a parser and an ast visitor
>>>> class, I've
>>>> begun the first steps to being able to put some sanity to license
>>>> selection.
>>>>
>>>> One note. License modifiers have traditionally been limited to the
>>>> "or greater"
>>>> modifier "+". I've added another modifier "with exceptions" for
>>>> licenses that
>>>> have no generic ie common-licenses/GPL-2,0-with-GCC-exception
>>>
>>> When building shadow I get:
>>
>>
>> and when building elfutils I get
>>
>> ERROR: (file: 'do_populate_lic', lineno: 116, function: do_populate_lic)
>> ERROR: Function 'do_populate_lic' failed
>> ERROR: Logfile of failure stored in:
>> /home/kraj/work/slugos/build/tmp-slugos-eglibc/work/armv5te-oe-linux-gnueabi/elfutils-0.148-r2/temp/log.do_populate_lic.31784
>>
>> Log data follows:
>> | WARNING:
>> /home/kraj/work/slugos/openembedded-core/meta/files/common-licenses
>> | ERROR: Error executing a python function in
>> /home/kraj/work/slugos/openembedded-core/meta/recipes-devtools/elfutils/elfutils_0.148.bb:
>>
>> | File "<unknown>", line 1
>> | ( 'GPLv2'& 'Exception' )
>> | ^
>> | IndentationError: unexpected indent
>> |
>> | ERROR: The stack trace of python calls that resulted in this
>> exception/failure was:
>> | ERROR: File "do_populate_lic", line 121, in<module>
>> | ERROR:
>> | ERROR: File "do_populate_lic", line 116, in do_populate_lic
>> | ERROR:
>> | ERROR: File "/usr/lib/python2.7/ast.py", line 37, in parse
>> | ERROR: return compile(source, filename, mode, PyCF_ONLY_AST)
>> | ERROR:
>> | ERROR: The code that was being executed was:
>> | ERROR: 0117: v = LicenseVisitor()
>> | ERROR: 0118: v.visit(node)
>> | ERROR: 0119:
>> | ERROR: 0120:
>> | ERROR: *** 0121:do_populate_lic(d)
>> | ERROR: 0122:
>> | ERROR: (file: 'do_populate_lic', lineno: 121, function:<module>)
>> | ERROR: 0112: clean_licenses += "'" + x + "'"
>> | ERROR: 0113: else:
>> | ERROR: 0114: clean_licenses += " " + x + " "
>> | ERROR: 0115:
>> | ERROR: *** 0116: node = ast.parse(clean_licenses)
>> | ERROR: 0117: v = LicenseVisitor()
>> | ERROR: 0118: v.visit(node)
>> | ERROR: 0119:
>> | ERROR: 0120:
>> | ERROR: (file: 'do_populate_lic', lineno: 116, function:
>> do_populate_lic)
>> | ERROR: Function 'do_populate_lic' failed
>> NOTE: package elfutils-0.148-r2: task do_populate_lic: Failed
>>
>>
>>>
>>> | WARNING:
>>> /home/koen/angstrom-core/sources/openembedded-core/meta/files/common-licenses
>>>
>>> | ERROR: Error executing a python function in
>>> /home/koen/angstrom-core/sources/openembedded-core/meta/recipes-extended/shadow/shadow_4.1.4.3.bb:
>>>
>>> | RuntimeError: maximum recursion depth exceeded while calling a
>>> Python object
>>> |
>>> | ERROR: The stack trace of python calls that resulted in this
>>> exception/failure was:
>>> | ERROR: File "do_populate_lic", line 121, in<module>
>>> | ERROR:
>>> | ERROR: File "do_populate_lic", line 118, in do_populate_lic
>>> | ERROR:
>>> | ERROR: File "/usr/lib64/python2.7/ast.py", line 243, in visit
>>> | ERROR: return visitor(node)
>>> | ERROR:
>>> | ERROR: File "do_populate_lic", line 13, in generic_visit
>>>
>>> and a few hundred more lines, I had to change the scrollback buffer
>>> from 512 lines to unlimited to catch this :) I can send the full log
>>> on request.
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>
Elizabeth Flanagan - May 28, 2011, 11:31 p.m.
On 05/28/2011 01:57 PM, Khem Raj wrote:
> On 5/28/2011 10:54 AM, Elizabeth Flanagan wrote:
>> In this case, elfutils should probably be changes as well as
>> license.bbclass. As do_populate_lic is looking for valid python to make
>> a syntax tree from, this will fail. Solution:
>>
>> 1. Fix license.bbclass to check LICENSE and massage it a bit more.
>> 2. Fix elfutils to be GPLv2* (unless it's the GCC exception. I haven't
>> read the license yet).
>
> It does not like () around license values so simple solution is to replace
>
> -LICENSE = "(GPLv2&  Exception)"
> +LICENSE = "GPLv2&  Exception"
>
> Generally its preferable to fix eflutils first before license class in
> this case to keep the tree bisect'able.
>

Actually, the issue is my .replace("(", " ( "). It makes the string " ( 'GPLv2' & 'Exception' ) " which tosses an error 
because of the indented string. I have a fix I'm putting a pull request in (in about 10 minutes) for both this and the 
issue Koen found.

-b

---------------
Elizabeth Flanagan
Yocto Project
Release Engineer