Patchwork Toolchain library whitelisting: A first pass (preliminary patch/RFC)

login
register
mail settings
Submitter Peter Seebach
Date April 26, 2012, 9:08 p.m.
Message ID <20120426160844.2d2a0b9f@wrlaptop>
Download mbox | patch
Permalink /patch/26473/
State New
Headers show

Comments

Peter Seebach - April 26, 2012, 9:08 p.m.
On Wed, 25 Apr 2012 20:38:05 -0500
Peter Seebach <peter.seebach@windriver.com> wrote:
> This is a followup from some chat in #yocto and elsewhere.

Okay, some more followup.

While testing this, I kept burning myself on perfectly reasonable
things to get wrong while defining and using multilibs, so I wrote a
bunch of sanity checks for that.

The intent of this is to validate that tunings (including multilibs)
are configured in a reasonable way that we would expect to work.  This
includes:

1.  Verifying that no multilib's tuning is the same as DEFAULTTUNE.
2.  Verifying that no multilib's library name is 'lib', because that
causes really cryptic error messages parsing recipes.
3.  For each tuning, verify:
3a.  The tuning has features.
3b.  Every feature has a TUNEVALID[x] entry.
3c.  If the feature has a TUNECONFLICTS[x] entry, no feature listed in
it is included in the feature list.
3d.  If the value TUNEABI_WHITELIST exists, the tuning's
TUNEABI_tune-foo value, or the tuning's name if that doesn't exist, is
in TUNEABI_WHITELIST, or alternatively, TUNEABI_OVERRIDE is defined.

The whitelist feature is optional, and my intent would be not to define
any TUNEABI_tune values in oe-core, but just to maintain the hooks so
that people with custom (and often prebuilt-binary) toolchains can use
it without all of us writing our own.

I am totally aware that my Python is a little rough, and would be happy
to improve the legibility or idiom.

Separately, I propose also the following fix:

     # Check TARGET_ARCH is set correctly
-    if data.getVar('TARGE_ARCH', e.data, False) == '${TUNE_ARCH}':
+    if data.getVar('TARGET_ARCH', e.data, False) == '${TUNE_ARCH}':

Anyway, the patch:
Mark Hatle - April 26, 2012, 10:01 p.m.
In general I like this.. see a few critiques below:

On 4/26/12 4:08 PM, Peter Seebach wrote:
> On Wed, 25 Apr 2012 20:38:05 -0500
> Peter Seebach<peter.seebach@windriver.com>  wrote:
>> This is a followup from some chat in #yocto and elsewhere.
>
> Okay, some more followup.
>
> While testing this, I kept burning myself on perfectly reasonable
> things to get wrong while defining and using multilibs, so I wrote a
> bunch of sanity checks for that.
>
> The intent of this is to validate that tunings (including multilibs)
> are configured in a reasonable way that we would expect to work.  This
> includes:
>
> 1.  Verifying that no multilib's tuning is the same as DEFAULTTUNE.
> 2.  Verifying that no multilib's library name is 'lib', because that
> causes really cryptic error messages parsing recipes.
> 3.  For each tuning, verify:
> 3a.  The tuning has features.
> 3b.  Every feature has a TUNEVALID[x] entry.
> 3c.  If the feature has a TUNECONFLICTS[x] entry, no feature listed in
> it is included in the feature list.
> 3d.  If the value TUNEABI_WHITELIST exists, the tuning's
> TUNEABI_tune-foo value, or the tuning's name if that doesn't exist, is
> in TUNEABI_WHITELIST, or alternatively, TUNEABI_OVERRIDE is defined.
>
> The whitelist feature is optional, and my intent would be not to define
> any TUNEABI_tune values in oe-core, but just to maintain the hooks so
> that people with custom (and often prebuilt-binary) toolchains can use
> it without all of us writing our own.
>
> I am totally aware that my Python is a little rough, and would be happy
> to improve the legibility or idiom.
>
> Separately, I propose also the following fix:
>
>       # Check TARGET_ARCH is set correctly
> -    if data.getVar('TARGE_ARCH', e.data, False) == '${TUNE_ARCH}':
> +    if data.getVar('TARGET_ARCH', e.data, False) == '${TUNE_ARCH}':
>
> Anyway, the patch:
>
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index 687ddeb..b7f93b5 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -11,6 +11,70 @@ def raise_sanity_error(msg):
>
>       %s""" % msg)
>
> +# Check a single tune for validity.
> +def check_toolchain_tune(data, tune, multilib):
> +    tune_errors = []
> +    bb.note("Sanity-checking tuning '%s' (%s) features:" % (tune, multilib))
> +    features = data.getVar("TUNE_FEATURES_tune-%s" % tune, True) or ""
> +    if features == '':
> +        return "Tuning '%s' has no defined features, and cannot be used." % tune
> +    features = features.split(' ')

split does a whitespace based split automatically, I'm used to seeing .split() 
everywhere.  (I won't comment on the other similar split items)

> +    validities = data.getVarFlags('TUNEVALID') or ""

"validities"?  that a new word?  ;)

> +    conflicts = data.getVarFlags('TUNECONFLICTS') or ""
> +    split_conflicts = {}
> +    for feature in features:
> +       if feature in conflicts:
> +	   if feature not in split_conflicts:
> +	       split_conflicts[feature] = conflicts[feature].split(' ')
> +	   for other in features:
> +	       if other in split_conflicts[feature]:
> +		   tune_errors.append("Feature '%s' conflicts with '%s'." %
> +		       ( feature, other ))
> +       if feature in validities:
> +	   bb.note("  %s: %s" % (feature, validities[feature]))
> +       else:
> +	   tune_errors.append("Feature '%s' is not defined." % feature)
> +    whitelist = data.getVar("TUNEABI_WHITELIST", True) or ''
> +    if whitelist != '':
> +	override = data.getVar("TUNEABI_OVERRIDE", True) or ''
> +	if not override:
> +	    tuneabi = data.getVar("TUNEABI", True) or ""
> +	    if tuneabi == "" or tuneabi.startswith('$'):
> +		tuneabi = tune
> +	    if True not in [x in whitelist.split(' ') for x in tuneabi.split(' ')]:
> +		tune_errors.append("Tuning '%s' (%s) cannot be used with any supported tuning/ABI." %
> +		    (tune, tuneabi))
> +    if tune_errors:
> +        return "Tuning '%s' has the following errors:\n" + '\n'.join(tune_errors)
> +
> +def check_toolchain(data):
> +    tune_error_set = []
> +    deftune = data.getVar("DEFAULTTUNE", True)
> +    tune_errors = check_toolchain_tune(data, deftune, 'default')
> +    if tune_errors:
> +        tune_error_set.append(tune_errors)
> +
> +    multilibs = data.getVar("MULTILIBS", True) or ""
> +    if multilibs != "":

Change the above to:
     multilibs = data.getVar("MULTILIBS", True)
     if multilibs:

> +        for pairs in [x.split(':') for x in multilibs.split(' ')]:
> +	    if pairs[0] != 'multilib':
> +	        bb.warn("Got an unexpected '%s' in MULTILIBS." % pairs[0])
> +	    else:
> +		if pairs[1] == 'lib':
> +		    tune_error_set.append("The multilib 'lib' was specified, but that doesn't work. You need lib32 or lib64.")

I'm surprised, why doesn't 'lib' work?  I was under the impression the naming 
was completely arbitrary.

Also we can definitely have more then just lib32 or lib64.  We already often use 
libx32 in testing the x32 layer.

> +		else:
> +		    tune = data.getVar("DEFAULTTUNE_virtclass-multilib-%s" % pairs[1], True)

If the tune isn't defined, then the multilib configuration is invalid.  I 
thought we already had a check for that somewhere else.. but if not.. it 
wouldn't be a bad idea to mention that here for the user.

A simple if not tune:  would do it..

> +		    if tune == deftune:
> +		        tune_error_set.append("Multilib '%s' (%s) is also the default tuning." % (pairs[1], deftune))

I wonder if this is an error or a warning.. I suspect it would be unintentional, 
but I'm not sure it's a failure?

> +		    else:
> +		        tune_errors = check_toolchain_tune(data, tune, pairs[1])
> +		    if tune_errors:
> +		        tune_error_set.append(tune_errors)
> +    if tune_error_set:
> +        return "Toolchain tunings invalid:\n" + '\n'.join(tune_error_set)
> +
> +    return ""
> +
>   def check_conf_exists(fn, data):
>       bbpath = []
>       fn = data.expand(fn)
> @@ -327,6 +391,9 @@ def check_sanity(e):
>           messages = messages + pseudo_msg + '\n'
>
>       check_supported_distro(e)
> +    toolchain_msg = check_toolchain(e.data)
> +    if toolchain_msg != "":
> +        messages = messages + toolchain_msg + '\n'

Your whitespace usage looks different.. perhaps that is just my mailer.

--Mark

>
>       # Check if DISPLAY is set if IMAGETEST is set
>       if not data.getVar( 'DISPLAY', e.data, True ) and data.getVar( 'IMAGETEST', e.data, True ) == 'qemu':
Peter Seebach - April 26, 2012, 10:42 p.m.
On Thu, 26 Apr 2012 17:01:41 -0500
Mark Hatle <mark.hatle@windriver.com> wrote:

> split does a whitespace based split automatically, I'm used to
> seeing .split() everywhere.  (I won't comment on the other similar
> split items)

Okay.
 
> > +    validities = data.getVarFlags('TUNEVALID') or ""
 
> "validities"?  that a new word?  ;)

Pretty much.  I was trying to think of a name for "the set of valid
things".  :)

> Change the above to:
>      multilibs = data.getVar("MULTILIBS", True)
>      if multilibs:

If someone has done:

MULTILIBS = ""

this then ends up being confused because pairs[0] of the single
returned item is still "", and that's not valid.
 
> > +		if pairs[1] == 'lib':
> > +		    tune_error_set.append("The multilib 'lib' was
> > specified, but that doesn't work. You need lib32 or lib64.")
 
> I'm surprised, why doesn't 'lib' work?  I was under the impression
> the naming was completely arbitrary.

I am surprised too, but if I use that name, I get a cryptic parse
error from libxcb's recipe which I couldn't understand.
 
> Also we can definitely have more then just lib32 or lib64.  We
> already often use libx32 in testing the x32 layer.

Okay.  I can improve the message, for sure.
 
> If the tune isn't defined, then the multilib configuration is
> invalid.  I thought we already had a check for that somewhere else..
> but if not.. it wouldn't be a bad idea to mention that here for the
> user.

That probably wants to be tested too.

> I wonder if this is an error or a warning.. I suspect it would be
> unintentional, but I'm not sure it's a failure?

I'm not totally sure.  I blamed that for my problem where I kept
getting a TUNE_ARCH of "x86_64x86_64" or "i586i586", but it turned out
to be a red herring.

I could change that to a warning.

> Your whitespace usage looks different.. perhaps that is just my
> mailer.

I almost certainly have tabs.

-s
Chris Larson - April 27, 2012, 5:15 a.m.
On Thu, Apr 26, 2012 at 3:42 PM, Peter Seebach
<peter.seebach@windriver.com> wrote:
>> Change the above to:
>>      multilibs = data.getVar("MULTILIBS", True)
>>      if multilibs:
>
> If someone has done:
>
> MULTILIBS = ""
>
> this then ends up being confused because pairs[0] of the single
> returned item is still "", and that's not valid.


This doesn't make sense. If you do "if multilibs:" and multilibs is
the empty string, it'll evaluate as false, and that block won't be
entered.

Patch

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 687ddeb..b7f93b5 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -11,6 +11,70 @@  def raise_sanity_error(msg):
     
     %s""" % msg)
 
+# Check a single tune for validity.
+def check_toolchain_tune(data, tune, multilib):
+    tune_errors = []
+    bb.note("Sanity-checking tuning '%s' (%s) features:" % (tune, multilib))
+    features = data.getVar("TUNE_FEATURES_tune-%s" % tune, True) or ""
+    if features == '':
+        return "Tuning '%s' has no defined features, and cannot be used." % tune
+    features = features.split(' ')
+    validities = data.getVarFlags('TUNEVALID') or ""
+    conflicts = data.getVarFlags('TUNECONFLICTS') or ""
+    split_conflicts = {}
+    for feature in features:
+       if feature in conflicts:
+	   if feature not in split_conflicts:
+	       split_conflicts[feature] = conflicts[feature].split(' ')
+	   for other in features:
+	       if other in split_conflicts[feature]:
+		   tune_errors.append("Feature '%s' conflicts with '%s'." %
+		       ( feature, other ))
+       if feature in validities:
+	   bb.note("  %s: %s" % (feature, validities[feature]))
+       else:
+	   tune_errors.append("Feature '%s' is not defined." % feature)
+    whitelist = data.getVar("TUNEABI_WHITELIST", True) or ''
+    if whitelist != '':
+	override = data.getVar("TUNEABI_OVERRIDE", True) or ''
+	if not override:
+	    tuneabi = data.getVar("TUNEABI", True) or ""
+	    if tuneabi == "" or tuneabi.startswith('$'):
+		tuneabi = tune
+	    if True not in [x in whitelist.split(' ') for x in tuneabi.split(' ')]:
+		tune_errors.append("Tuning '%s' (%s) cannot be used with any supported tuning/ABI." %
+		    (tune, tuneabi))
+    if tune_errors:
+        return "Tuning '%s' has the following errors:\n" + '\n'.join(tune_errors)
+
+def check_toolchain(data):
+    tune_error_set = []
+    deftune = data.getVar("DEFAULTTUNE", True)
+    tune_errors = check_toolchain_tune(data, deftune, 'default')
+    if tune_errors:
+        tune_error_set.append(tune_errors)
+
+    multilibs = data.getVar("MULTILIBS", True) or ""
+    if multilibs != "":
+        for pairs in [x.split(':') for x in multilibs.split(' ')]:
+	    if pairs[0] != 'multilib':
+	        bb.warn("Got an unexpected '%s' in MULTILIBS." % pairs[0])
+	    else:
+		if pairs[1] == 'lib':
+		    tune_error_set.append("The multilib 'lib' was specified, but that doesn't work. You need lib32 or lib64.")
+		else:
+		    tune = data.getVar("DEFAULTTUNE_virtclass-multilib-%s" % pairs[1], True)
+		    if tune == deftune:
+		        tune_error_set.append("Multilib '%s' (%s) is also the default tuning." % (pairs[1], deftune))
+		    else:
+		        tune_errors = check_toolchain_tune(data, tune, pairs[1])
+		    if tune_errors:
+		        tune_error_set.append(tune_errors)
+    if tune_error_set:
+        return "Toolchain tunings invalid:\n" + '\n'.join(tune_error_set)
+
+    return ""
+
 def check_conf_exists(fn, data):
     bbpath = []
     fn = data.expand(fn)
@@ -327,6 +391,9 @@  def check_sanity(e):
         messages = messages + pseudo_msg + '\n'
 
     check_supported_distro(e)
+    toolchain_msg = check_toolchain(e.data)
+    if toolchain_msg != "":
+        messages = messages + toolchain_msg + '\n'
 
     # Check if DISPLAY is set if IMAGETEST is set
     if not data.getVar( 'DISPLAY', e.data, True ) and data.getVar( 'IMAGETEST', e.data, True ) == 'qemu':