Patchwork [1/1] sanity.bbclass: check for validity of TMPDIR

login
register
mail settings
Submitter Qi.Chen@windriver.com
Date Nov. 14, 2013, 1:07 p.m.
Message ID <a86f7984cd1b08978d4358f862ff97d3d5cd3fb7.1384433183.git.Qi.Chen@windriver.com>
Download mbox | patch
Permalink /patch/61707/
State New
Headers show

Comments

Qi.Chen@windriver.com - Nov. 14, 2013, 1:07 p.m.
From: Chen Qi <Qi.Chen@windriver.com>

TMPDIR must be an absolute path, otherwise, the build will fail.
Special characters in TMPDIR will also cause build failures.

This patch enables checking for the validity of TMPDIR.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/sanity.bbclass |    7 +++++++
 1 file changed, 7 insertions(+)
Richard Purdie - Nov. 14, 2013, 1:12 p.m.
On Thu, 2013-11-14 at 21:07 +0800, Qi.Chen@windriver.com wrote:
> From: Chen Qi <Qi.Chen@windriver.com>
> 
> TMPDIR must be an absolute path, otherwise, the build will fail.
> Special characters in TMPDIR will also cause build failures.
> 
> This patch enables checking for the validity of TMPDIR.
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/classes/sanity.bbclass |    7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index 83378b0..e45906e 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
>          with open(checkfile, "w") as f:
>              f.write(tmpdir)
>  
> +    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
> +    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
> +    import re
> +    valid_pattern = re.compile(valid_tmpdir_regexp)
> +    if not valid_pattern.match(tmpdir):
> +        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
> +

I'm ok with the check itself however this does not need to be run every
build. It needs to run when tmpdir changes only.

Cheers,

Richard
Gary Thomas - Nov. 14, 2013, 1:27 p.m.
On 2013-11-14 06:07, Qi.Chen@windriver.com wrote:
> From: Chen Qi <Qi.Chen@windriver.com>
>
> TMPDIR must be an absolute path, otherwise, the build will fail.
> Special characters in TMPDIR will also cause build failures.
>
> This patch enables checking for the validity of TMPDIR.
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>   meta/classes/sanity.bbclass |    7 +++++++
>   1 file changed, 7 insertions(+)
>
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index 83378b0..e45906e 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
>           with open(checkfile, "w") as f:
>               f.write(tmpdir)
>
> +    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
> +    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
> +    import re
> +    valid_pattern = re.compile(valid_tmpdir_regexp)
> +    if not valid_pattern.match(tmpdir):
> +        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
> +
>   def check_sanity(sanity_data):
>       import subprocess

Check the wording of the warning "The TMPDIR should match the this regexp"
should probably be "The TMPDIR should match this regexp" or maybe use a simpler
version like "The TMPDIR name should must be an absolute path and contain only letters,
digits and the characters -_"

Also, is "-" actually valid?  I seem to recall having problems when my build
tree had the hyphen ("-") in the path.
Phil Blundell - Nov. 14, 2013, 1:42 p.m.
On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
> Also, is "-" actually valid?  I seem to recall having problems when my build
> tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me.  If there are any recipes which
break in that situation then we should just fix them. 

OE already has quite a range of baroque restrictions on what sort of
TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
spaces in the name) and every new prohibition represents a loss in
usability.  This patch as proposed already forbids a whole range of
characters, including things like "+", and I think that disallowing "-"
as well would be a step too far.

p.
Gary Thomas - Nov. 14, 2013, 1:50 p.m.
On 2013-11-14 06:42, Phil Blundell wrote:
> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>> Also, is "-" actually valid?  I seem to recall having problems when my build
>> tree had the hyphen ("-") in the path.
>
> Paths with "-" in certainly work for me.  If there are any recipes which
> break in that situation then we should just fix them.
>
> OE already has quite a range of baroque restrictions on what sort of
> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
> spaces in the name) and every new prohibition represents a loss in
> usability.  This patch as proposed already forbids a whole range of
> characters, including things like "+", and I think that disallowing "-"
> as well would be a step too far.

Fair enough.  I don't recall exactly what problems I had (in the dark
past), but I've just avoided trees with "-" in them.  I'll give it
another look if/when an opportunity comes by.
Mark Hatle - Nov. 14, 2013, 3:49 p.m.
On 11/14/13, 7:42 AM, Phil Blundell wrote:
> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>> Also, is "-" actually valid?  I seem to recall having problems when my build
>> tree had the hyphen ("-") in the path.
>
> Paths with "-" in certainly work for me.  If there are any recipes which
> break in that situation then we should just fix them.

We found an issue with paths that -start- with a '-', there are apparently many 
places where paths are passed into various shell, and the initial '-' can be 
read as an argument identifier.

> OE already has quite a range of baroque restrictions on what sort of
> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
> spaces in the name) and every new prohibition represents a loss in
> usability.  This patch as proposed already forbids a whole range of
> characters, including things like "+", and I think that disallowing "-"
> as well would be a step too far.

I don't believe Qi Chen sent it to the list, but we built path names with all of 
the special characters and tried to run builds.  The specific list is based on 
the results of those tests.  The other items are already broken, and we're 
trying to be explicit with this for end users.  (Note, it's really the TMPDIR 
that matters.. the 'build' directory is a lot more flexible.)

--Mark

> p.
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
Phil Blundell - Nov. 14, 2013, 4:04 p.m.
On Thu, 2013-11-14 at 09:49 -0600, Mark Hatle wrote:
> On 11/14/13, 7:42 AM, Phil Blundell wrote:
> > On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
> >> Also, is "-" actually valid?  I seem to recall having problems when my build
> >> tree had the hyphen ("-") in the path.
> >
> > Paths with "-" in certainly work for me.  If there are any recipes which
> > break in that situation then we should just fix them.
> 
> We found an issue with paths that -start- with a '-', there are apparently many 
> places where paths are passed into various shell, and the initial '-' can be 
> read as an argument identifier.

Right, I can imagine that this would break fairly badly.  But if we're
going to require TMPDIR to be an absolute path (which seems perfectly
reasonable) then by definition it can't start with - so this will
presumably become a non-issue.

> I don't believe Qi Chen sent it to the list, but we built path names with all of 
> the special characters and tried to run builds.  The specific list is based on 
> the results of those tests.  The other items are already broken, and we're 
> trying to be explicit with this for end users.  (Note, it's really the TMPDIR 
> that matters.. the 'build' directory is a lot more flexible.)

Yeah, I saw the results of those tests.  My slight reservation with that
methodology is that, if any single recipe fails with a given character
in TMPDIR, no matter how obscure that recipe might be, it seems that
this is enough to make that character verboten.  For example, the
rationale for excluding "+" seems to be that it breaks tcl; if this is
the only recipe that fails then it seems like it might be worth at least
trying to fix tcl rather than excluding "+" from TMPDIRs for ever more.

But, clearly, there are some characters in the list that are "just
silly" to have in pathnames.  I don't think anybody would be surprised
to learn that "*" and "$" cause problems and blacklisting those ones
does seem quite reasonable.

p.
Saul Wold - Nov. 14, 2013, 6:01 p.m.
On 11/14/2013 07:49 AM, Mark Hatle wrote:
> On 11/14/13, 7:42 AM, Phil Blundell wrote:
>> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>>> Also, is "-" actually valid?  I seem to recall having problems when
>>> my build
>>> tree had the hyphen ("-") in the path.
>>
>> Paths with "-" in certainly work for me.  If there are any recipes which
>> break in that situation then we should just fix them.
>
> We found an issue with paths that -start- with a '-', there are
> apparently many places where paths are passed into various shell, and
> the initial '-' can be read as an argument identifier.
>

I think that I saw an email talking about "-D" in the TMPDIR caused some 
some recipes to pick that up as a CFLAG define and cause problems

Sau!

>> OE already has quite a range of baroque restrictions on what sort of
>> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
>> spaces in the name) and every new prohibition represents a loss in
>> usability.  This patch as proposed already forbids a whole range of
>> characters, including things like "+", and I think that disallowing "-"
>> as well would be a step too far.
>
> I don't believe Qi Chen sent it to the list, but we built path names
> with all of the special characters and tried to run builds.  The
> specific list is based on the results of those tests.  The other items
> are already broken, and we're trying to be explicit with this for end
> users.  (Note, it's really the TMPDIR that matters.. the 'build'
> directory is a lot more flexible.)
>
> --Mark
>
>> p.
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
Mark Hatle - Nov. 14, 2013, 6:02 p.m.
On 11/14/13, 10:04 AM, Phil Blundell wrote:
> On Thu, 2013-11-14 at 09:49 -0600, Mark Hatle wrote:
>> On 11/14/13, 7:42 AM, Phil Blundell wrote:
>>> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>>>> Also, is "-" actually valid?  I seem to recall having problems when my build
>>>> tree had the hyphen ("-") in the path.
>>>
>>> Paths with "-" in certainly work for me.  If there are any recipes which
>>> break in that situation then we should just fix them.
>>
>> We found an issue with paths that -start- with a '-', there are apparently many
>> places where paths are passed into various shell, and the initial '-' can be
>> read as an argument identifier.
>
> Right, I can imagine that this would break fairly badly.  But if we're
> going to require TMPDIR to be an absolute path (which seems perfectly
> reasonable) then by definition it can't start with - so this will
> presumably become a non-issue.

Except there are a number of packages that try to parse pieces of strings 
derived from TMPDIR.

Set your TMPDIR path to include "-Dmypath" some time.  SVN will blow up.. :P

>> I don't believe Qi Chen sent it to the list, but we built path names with all of
>> the special characters and tried to run builds.  The specific list is based on
>> the results of those tests.  The other items are already broken, and we're
>> trying to be explicit with this for end users.  (Note, it's really the TMPDIR
>> that matters.. the 'build' directory is a lot more flexible.)
>
> Yeah, I saw the results of those tests.  My slight reservation with that
> methodology is that, if any single recipe fails with a given character
> in TMPDIR, no matter how obscure that recipe might be, it seems that
> this is enough to make that character verboten.  For example, the
> rationale for excluding "+" seems to be that it breaks tcl; if this is
> the only recipe that fails then it seems like it might be worth at least
> trying to fix tcl rather than excluding "+" from TMPDIRs for ever more.

I agree, in the case above, the SVN example I've asked our engineers to look 
into trying to fix that case.  But what they're finding is that once you fix one 
case, then the next breaks.. and the next breaks.

> But, clearly, there are some characters in the list that are "just
> silly" to have in pathnames.  I don't think anybody would be surprised
> to learn that "*" and "$" cause problems and blacklisting those ones
> does seem quite reasonable.

:)

I'd like to see us blacklist the bunch, and then turn around and start opening 
the set backup.  But I don't know how practical this actually is -- unless we 
can get automated test machines running each odd-ball combination on a nightly 
basis.

--Mark

> p.
>
>
Qi.Chen@windriver.com - Nov. 15, 2013, 2:41 a.m.
On 11/14/2013 09:12 PM, Richard Purdie wrote:
> On Thu, 2013-11-14 at 21:07 +0800, Qi.Chen@windriver.com wrote:
>> From: Chen Qi <Qi.Chen@windriver.com>
>>
>> TMPDIR must be an absolute path, otherwise, the build will fail.
>> Special characters in TMPDIR will also cause build failures.
>>
>> This patch enables checking for the validity of TMPDIR.
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   meta/classes/sanity.bbclass |    7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
>> index 83378b0..e45906e 100644
>> --- a/meta/classes/sanity.bbclass
>> +++ b/meta/classes/sanity.bbclass
>> @@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
>>           with open(checkfile, "w") as f:
>>               f.write(tmpdir)
>>   
>> +    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
>> +    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
>> +    import re
>> +    valid_pattern = re.compile(valid_tmpdir_regexp)
>> +    if not valid_pattern.match(tmpdir):
>> +        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
>> +
> I'm ok with the check itself however this does not need to be run every
> build. It needs to run when tmpdir changes only.
>
> Cheers,
>
> Richard
>
>
>
>

Thanks for pointing it out.

I've changed the logic a little bit.
1. Create the directory hierarchy for TMPDIR only if TMPDIR is valid
2. Handle ABI version change only if TMPDIR has been created.

So only when we had a valid TMPDIR for last build and this time the 
TMPDIR hasn't changed location, we don't check the validity for TMPDIR.

I've sent out  a V2.

Best Regards,
Chen Qi
Qi.Chen@windriver.com - Nov. 15, 2013, 2:49 a.m.
On 11/15/2013 02:01 AM, Saul Wold wrote:
> On 11/14/2013 07:49 AM, Mark Hatle wrote:
>> On 11/14/13, 7:42 AM, Phil Blundell wrote:
>>> On Thu, 2013-11-14 at 06:27 -0700, Gary Thomas wrote:
>>>> Also, is "-" actually valid?  I seem to recall having problems when
>>>> my build
>>>> tree had the hyphen ("-") in the path.
>>>
>>> Paths with "-" in certainly work for me.  If there are any recipes 
>>> which
>>> break in that situation then we should just fix them.
>>
>> We found an issue with paths that -start- with a '-', there are
>> apparently many places where paths are passed into various shell, and
>> the initial '-' can be read as an argument identifier.
>>
>
> I think that I saw an email talking about "-D" in the TMPDIR caused 
> some some recipes to pick that up as a CFLAG define and cause problems
>
> Sau!
>

Yes. It's actually caused by the following statement in subversion-native.
./build/ac-macros/neon.m4: SVN_NEON_INCLUDES=[`$PKG_CONFIG neon --cflags 
| $SED -e 's/-D[^ ]*//g'`]
./build/ac-macros/neon.m4: SVN_NEON_INCLUDES=[`$neon_config --cflags | 
$SED -e 's/-D[^ ]*//g'`]

And there's a bug filed for it. 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5458

Best Regards,
Chen Qi

>>> OE already has quite a range of baroque restrictions on what sort of
>>> TMPDIR you are allowed to use (no nfs, no symlinks in the path, no
>>> spaces in the name) and every new prohibition represents a loss in
>>> usability.  This patch as proposed already forbids a whole range of
>>> characters, including things like "+", and I think that disallowing "-"
>>> as well would be a step too far.
>>
>> I don't believe Qi Chen sent it to the list, but we built path names
>> with all of the special characters and tried to run builds.  The
>> specific list is based on the results of those tests.  The other items
>> are already broken, and we're trying to be explicit with this for end
>> users.  (Note, it's really the TMPDIR that matters.. the 'build'
>> directory is a lot more flexible.)
>>
>> --Mark
>>
>>> p.
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

Patch

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 83378b0..e45906e 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -672,6 +672,13 @@  def check_sanity_everybuild(status, d):
         with open(checkfile, "w") as f:
             f.write(tmpdir)
 
+    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
+    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
+    import re
+    valid_pattern = re.compile(valid_tmpdir_regexp)
+    if not valid_pattern.match(tmpdir):
+        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
+
 def check_sanity(sanity_data):
     import subprocess