[bitbake-devel,1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

Submitted by Robert Yang on May 9, 2019, 8:03 a.m. | Patch ID: 161084

Details

Message ID ac4edd962cb6b4831cb85cd731d2d2a810c62503.1557388944.git.liezhi.yang@windriver.com
State New
Headers show

Commit Message

Robert Yang May 9, 2019, 8:03 a.m.
The bb.fatal() raises BBHandledException() which causes double exceptions,
e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:
$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.

[YOCTO #13267]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cookerdata.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Patch hide | download patch | download mbox

diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index f8ae410..585edc5 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -301,7 +301,9 @@  class CookerDataBuilder(object):
             if multiconfig:
                 bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data)
 
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
             raise bb.BBHandledException
         except bb.data_smart.ExpansionError as e:
             logger.error(str(e))

Comments

Richard Purdie May 12, 2019, 8:28 a.m.
On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
> The bb.fatal() raises BBHandledException() which causes double exceptions,
> e.g.:
> 
> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
> $ bitbake -p
> [snip]
> During handling of the above exception, another exception occurred:
> [snip]
> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
>   hello
> 
> Use "raise" rather than "raise bb.BBHandledException" to fix the double
> exceptions.
> 
> [YOCTO #13267]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/cookerdata.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/cookerdata.py
> b/bitbake/lib/bb/cookerdata.py
> index f8ae410..585edc5 100644
> --- a/bitbake/lib/bb/cookerdata.py
> +++ b/bitbake/lib/bb/cookerdata.py
> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>              if multiconfig:
>                  bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
> ), self.data)
>  
> -        except (SyntaxError, bb.BBHandledException):
> +        except bb.BBHandledException:
> +            raise
> +        except SyntaxError:
>              raise bb.BBHandledException
>          except bb.data_smart.ExpansionError as e:
>              logger.error(str(e))

Hi Robert,

This doesn't sound right, where is this exception being printed a
second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it
should pass through BBHandledException?

Cheers,

Richard
Robert Yang May 14, 2019, 11:02 a.m.
On 5/12/19 4:28 PM, Richard Purdie wrote:
> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>> The bb.fatal() raises BBHandledException() which causes double exceptions,
>> e.g.:
>>
>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>> $ bitbake -p
>> [snip]
>> During handling of the above exception, another exception occurred:
>> [snip]
>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
>>    hello
>>
>> Use "raise" rather than "raise bb.BBHandledException" to fix the double
>> exceptions.
>>
>> [YOCTO #13267]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/cookerdata.py | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/cookerdata.py
>> b/bitbake/lib/bb/cookerdata.py
>> index f8ae410..585edc5 100644
>> --- a/bitbake/lib/bb/cookerdata.py
>> +++ b/bitbake/lib/bb/cookerdata.py
>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>               if multiconfig:
>>                   bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>> ), self.data)
>>   
>> -        except (SyntaxError, bb.BBHandledException):
>> +        except bb.BBHandledException:
>> +            raise
>> +        except SyntaxError:
>>               raise bb.BBHandledException
>>           except bb.data_smart.ExpansionError as e:
>>               logger.error(str(e))
> 
> Hi Robert,
> 
> This doesn't sound right, where is this exception being printed a
> second time? The point of "BBHandledException" is to say "don't print
> any further traces for this exception". If this fixes the bug, it means
> something somewhere is printing a trace for a second time when it
> should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
[hangs]

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.

I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes
    one of result_queue or parser_quit have a deadlock, then
    causes the shutdown():process.join() hanged.

2) The sub processes, parser_quit and result_queue are running dependently,
    we can't make sure that cooker.py:Parser():realrun() can get data from
    parser_quit, so there is a race issue, the Parser's subprocess maybe already
    exited before parser_quit sends data to it.

I think that we can move part of parse_next()'s code into parse_generator()
to fix the deadlock problem, and we may need change the working way of
parser_quit. But I'm not sure, I need your suggestions before work on it.

// Robert

> 
> Cheers,
> 
> Richard
> 
>
Robert Yang May 20, 2019, 8:55 a.m.
On 5/14/19 7:02 PM, Robert Yang wrote:
> 
> 
> On 5/12/19 4:28 PM, Richard Purdie wrote:
>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>> The bb.fatal() raises BBHandledException() which causes double exceptions,
>>> e.g.:
>>>
>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>> $ bitbake -p
>>> [snip]
>>> During handling of the above exception, another exception occurred:
>>> [snip]
>>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be 
>>> unavailable in PATH, please install them in order to proceed:
>>>    hello
>>>
>>> Use "raise" rather than "raise bb.BBHandledException" to fix the double
>>> exceptions.
>>>
>>> [YOCTO #13267]
>>>
>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>> ---
>>>   bitbake/lib/bb/cookerdata.py | 4 +++-
>>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>> b/bitbake/lib/bb/cookerdata.py
>>> index f8ae410..585edc5 100644
>>> --- a/bitbake/lib/bb/cookerdata.py
>>> +++ b/bitbake/lib/bb/cookerdata.py
>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>               if multiconfig:
>>>                   bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>> ), self.data)
>>> -        except (SyntaxError, bb.BBHandledException):
>>> +        except bb.BBHandledException:
>>> +            raise
>>> +        except SyntaxError:
>>>               raise bb.BBHandledException
>>>           except bb.data_smart.ExpansionError as e:
>>>               logger.error(str(e))
>>
>> Hi Robert,
>>
>> This doesn't sound right, where is this exception being printed a
>> second time? The point of "BBHandledException" is to say "don't print
>> any further traces for this exception". If this fixes the bug, it means
>> something somewhere is printing a trace for a second time when it
>> should pass through BBHandledException?
> 
> Hi RP,
> 
> I found another serious problem when tried to raise BBHandledException. There
> is a deadlock when a recipe is failed to parse, e.g.:
> 
> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
> $ bitbake -p
> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
> [hangs]
> 
> Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
> are still existed, and we need kill them manually, otherwise we can't start
> bitbake again.
> 
> I think that there are two problems:
> 1) The cooker.py:parse_generator() raises an exception which makes
>     one of result_queue or parser_quit have a deadlock, then
>     causes the shutdown():process.join() hanged.
> 
> 2) The sub processes, parser_quit and result_queue are running dependently,
>     we can't make sure that cooker.py:Parser():realrun() can get data from
>     parser_quit, so there is a race issue, the Parser's subprocess maybe already
>     exited before parser_quit sends data to it.
> 
> I think that we can move part of parse_next()'s code into parse_generator()
> to fix the deadlock problem, and we may need change the working way of
> parser_quit. But I'm not sure, I need your suggestions before work on it.

Hi RP,

Do you have any suggestions, please?

// Robert

> 
> // Robert
> 
>>
>> Cheers,
>>
>> Richard
>>
>>