[bitbake-devel,1/1] cooker.py: terminate the Parser processes

Submitted by Kang Kai on May 23, 2012, 9:40 a.m.

Details

Message ID dac2367a09e77d0230e8c2e625484336533ea50e.1337764882.git.kai.kang@windriver.com
State New
Headers show

Commit Message

Kang Kai May 23, 2012, 9:40 a.m.
[Yocto 2142]

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
It hangs on function BitBakeServerConnection::terminate in file
server/process.py:
    else:
        self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as
cpu numbers. When quit the Parser processes they make their internal
Queue to call cancel_join_thread() to avoid block but don't work at
this time.
So force to terminate the Parser processes.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 bitbake/lib/bb/cooker.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)

Patch hide | download patch | download mbox

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index dea0aad..4a4dc38 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1175,7 +1175,7 @@  class BBCooker:
             return
 
         if self.state in (state.shutdown, state.stop):
-            self.parser.shutdown(clean=False)
+            self.parser.shutdown(clean=False, force = True)
             sys.exit(1)
 
         if self.state != state.parsing:
@@ -1608,10 +1608,13 @@  class CookerParser(object):
                 self.parser_quit.put(None)
 
             self.jobs.cancel_join_thread()
-            sys.exit(1)
 
         for process in self.processes:
-            process.join()
+            if force:
+                process.join(.1)
+                process.terminate()
+            else:
+                process.join()
         self.feeder.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)

Comments

Chris Larson May 23, 2012, 3:34 p.m.
terminate()'ing processes can corrupt any queues those processes have
open, and thereby disrupt any pending log messages in the queues
heading to the ui as a result. If that's not an issue here, so be it,
but be aware of the issues -- we had to remove terminate() from the
shutdown in the past due to this issue.

On Wed, May 23, 2012 at 2:40 AM, Kang Kai <kai.kang@windriver.com> wrote:
> [Yocto 2142]
>
> Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
> It hangs on function BitBakeServerConnection::terminate in file
> server/process.py:
>    else:
>        self.procserver.join()
> It is waiting for the children process quit.
>
> In stage of parse recipes BBCooker spawns Parser processes as many as
> cpu numbers. When quit the Parser processes they make their internal
> Queue to call cancel_join_thread() to avoid block but don't work at
> this time.
> So force to terminate the Parser processes.
>
> Signed-off-by: Kang Kai <kai.kang@windriver.com>
> ---
>  bitbake/lib/bb/cooker.py |    9 ++++++---
>  1 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index dea0aad..4a4dc38 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1175,7 +1175,7 @@ class BBCooker:
>             return
>
>         if self.state in (state.shutdown, state.stop):
> -            self.parser.shutdown(clean=False)
> +            self.parser.shutdown(clean=False, force = True)
>             sys.exit(1)
>
>         if self.state != state.parsing:
> @@ -1608,10 +1608,13 @@ class CookerParser(object):
>                 self.parser_quit.put(None)
>
>             self.jobs.cancel_join_thread()
> -            sys.exit(1)
>
>         for process in self.processes:
> -            process.join()
> +            if force:
> +                process.join(.1)
> +                process.terminate()
> +            else:
> +                process.join()
>         self.feeder.join()
>
>         sync = threading.Thread(target=self.bb_cache.sync)
> --
> 1.7.5.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Kang Kai May 24, 2012, 8:29 a.m.
On 2012?05?23? 23:34, Chris Larson wrote:
> terminate()'ing processes can corrupt any queues those processes have
> open, and thereby disrupt any pending log messages in the queues
> heading to the ui as a result. If that's not an issue here, so be it,
> but be aware of the issues -- we had to remove terminate() from the
> shutdown in the past due to this issue.
Hi Chris,

Call terminate() directly seems not be elegant. But I didn't figure out 
why after the queue belongs to Parse process
calls  cancel_join_thread(), so ..
Before the terminate() the every Parse process, it is waited for 0.1 
second to quit.
If the process will not be blocked, I suppose that 0.1 second is enough 
for them to quit.

What's your opinion?

Thanks,
Kai
>
> On Wed, May 23, 2012 at 2:40 AM, Kang Kai<kai.kang@windriver.com>  wrote:
>> [Yocto 2142]
>>
>> Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
>> It hangs on function BitBakeServerConnection::terminate in file
>> server/process.py:
>>     else:
>>         self.procserver.join()
>> It is waiting for the children process quit.
>>
>> In stage of parse recipes BBCooker spawns Parser processes as many as
>> cpu numbers. When quit the Parser processes they make their internal
>> Queue to call cancel_join_thread() to avoid block but don't work at
>> this time.
>> So force to terminate the Parser processes.
>>
>> Signed-off-by: Kang Kai<kai.kang@windriver.com>
>> ---
>>   bitbake/lib/bb/cooker.py |    9 ++++++---
>>   1 files changed, 6 insertions(+), 3 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index dea0aad..4a4dc38 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1175,7 +1175,7 @@ class BBCooker:
>>              return
>>
>>          if self.state in (state.shutdown, state.stop):
>> -            self.parser.shutdown(clean=False)
>> +            self.parser.shutdown(clean=False, force = True)
>>              sys.exit(1)
>>
>>          if self.state != state.parsing:
>> @@ -1608,10 +1608,13 @@ class CookerParser(object):
>>                  self.parser_quit.put(None)
>>
>>              self.jobs.cancel_join_thread()
>> -            sys.exit(1)
>>
>>          for process in self.processes:
>> -            process.join()
>> +            if force:
>> +                process.join(.1)
>> +                process.terminate()
>> +            else:
>> +                process.join()
>>          self.feeder.join()
>>
>>          sync = threading.Thread(target=self.bb_cache.sync)
>> --
>> 1.7.5.4
>>
>>
>> _______________________________________________
>> bitbake-devel mailing list
>> bitbake-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>