Patchwork [bitbake-devel,1/1] cooker.py: terminate the Parser processes

login
register
mail settings
Submitter Kang Kai
Date May 23, 2012, 9:40 a.m.
Message ID <dac2367a09e77d0230e8c2e625484336533ea50e.1337764882.git.kai.kang@windriver.com>
Download mbox | patch
Permalink /patch/28417/
State New
Headers show

Comments

Kang Kai - May 23, 2012, 9:40 a.m.
[Yocto 2142]

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
It hangs on function BitBakeServerConnection::terminate in file
server/process.py:
    else:
        self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as
cpu numbers. When quit the Parser processes they make their internal
Queue to call cancel_join_thread() to avoid block but don't work at
this time.
So force to terminate the Parser processes.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 bitbake/lib/bb/cooker.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
Chris Larson - May 23, 2012, 3:34 p.m.
terminate()'ing processes can corrupt any queues those processes have
open, and thereby disrupt any pending log messages in the queues
heading to the ui as a result. If that's not an issue here, so be it,
but be aware of the issues -- we had to remove terminate() from the
shutdown in the past due to this issue.

On Wed, May 23, 2012 at 2:40 AM, Kang Kai <kai.kang@windriver.com> wrote:
> [Yocto 2142]
>
> Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
> It hangs on function BitBakeServerConnection::terminate in file
> server/process.py:
>    else:
>        self.procserver.join()
> It is waiting for the children process quit.
>
> In stage of parse recipes BBCooker spawns Parser processes as many as
> cpu numbers. When quit the Parser processes they make their internal
> Queue to call cancel_join_thread() to avoid block but don't work at
> this time.
> So force to terminate the Parser processes.
>
> Signed-off-by: Kang Kai <kai.kang@windriver.com>
> ---
>  bitbake/lib/bb/cooker.py |    9 ++++++---
>  1 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index dea0aad..4a4dc38 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1175,7 +1175,7 @@ class BBCooker:
>             return
>
>         if self.state in (state.shutdown, state.stop):
> -            self.parser.shutdown(clean=False)
> +            self.parser.shutdown(clean=False, force = True)
>             sys.exit(1)
>
>         if self.state != state.parsing:
> @@ -1608,10 +1608,13 @@ class CookerParser(object):
>                 self.parser_quit.put(None)
>
>             self.jobs.cancel_join_thread()
> -            sys.exit(1)
>
>         for process in self.processes:
> -            process.join()
> +            if force:
> +                process.join(.1)
> +                process.terminate()
> +            else:
> +                process.join()
>         self.feeder.join()
>
>         sync = threading.Thread(target=self.bb_cache.sync)
> --
> 1.7.5.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Kang Kai - May 24, 2012, 8:29 a.m.
On 2012?05?23? 23:34, Chris Larson wrote:
> terminate()'ing processes can corrupt any queues those processes have
> open, and thereby disrupt any pending log messages in the queues
> heading to the ui as a result. If that's not an issue here, so be it,
> but be aware of the issues -- we had to remove terminate() from the
> shutdown in the past due to this issue.
Hi Chris,

Call terminate() directly seems not be elegant. But I didn't figure out 
why after the queue belongs to Parse process
calls  cancel_join_thread(), so ..
Before the terminate() the every Parse process, it is waited for 0.1 
second to quit.
If the process will not be blocked, I suppose that 0.1 second is enough 
for them to quit.

What's your opinion?

Thanks,
Kai
>
> On Wed, May 23, 2012 at 2:40 AM, Kang Kai<kai.kang@windriver.com>  wrote:
>> [Yocto 2142]
>>
>> Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
>> It hangs on function BitBakeServerConnection::terminate in file
>> server/process.py:
>>     else:
>>         self.procserver.join()
>> It is waiting for the children process quit.
>>
>> In stage of parse recipes BBCooker spawns Parser processes as many as
>> cpu numbers. When quit the Parser processes they make their internal
>> Queue to call cancel_join_thread() to avoid block but don't work at
>> this time.
>> So force to terminate the Parser processes.
>>
>> Signed-off-by: Kang Kai<kai.kang@windriver.com>
>> ---
>>   bitbake/lib/bb/cooker.py |    9 ++++++---
>>   1 files changed, 6 insertions(+), 3 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index dea0aad..4a4dc38 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1175,7 +1175,7 @@ class BBCooker:
>>              return
>>
>>          if self.state in (state.shutdown, state.stop):
>> -            self.parser.shutdown(clean=False)
>> +            self.parser.shutdown(clean=False, force = True)
>>              sys.exit(1)
>>
>>          if self.state != state.parsing:
>> @@ -1608,10 +1608,13 @@ class CookerParser(object):
>>                  self.parser_quit.put(None)
>>
>>              self.jobs.cancel_join_thread()
>> -            sys.exit(1)
>>
>>          for process in self.processes:
>> -            process.join()
>> +            if force:
>> +                process.join(.1)
>> +                process.terminate()
>> +            else:
>> +                process.join()
>>          self.feeder.join()
>>
>>          sync = threading.Thread(target=self.bb_cache.sync)
>> --
>> 1.7.5.4
>>
>>
>> _______________________________________________
>> bitbake-devel mailing list
>> bitbake-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>

Patch

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index dea0aad..4a4dc38 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1175,7 +1175,7 @@  class BBCooker:
             return
 
         if self.state in (state.shutdown, state.stop):
-            self.parser.shutdown(clean=False)
+            self.parser.shutdown(clean=False, force = True)
             sys.exit(1)
 
         if self.state != state.parsing:
@@ -1608,10 +1608,13 @@  class CookerParser(object):
                 self.parser_quit.put(None)
 
             self.jobs.cancel_join_thread()
-            sys.exit(1)
 
         for process in self.processes:
-            process.join()
+            if force:
+                process.join(.1)
+                process.terminate()
+            else:
+                process.join()
         self.feeder.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)