mbox series

[00/11,v2] Bitbake server thread enabling

Message ID 20221231165849.964169-1-richard.purdie@linuxfoundation.org
Headers show
Series Bitbake server thread enabling | expand

Message

Richard Purdie Dec. 31, 2022, 4:58 p.m. UTC 
This series adds 'idle' thread support to bitbake. These changes have
struggled a bit in testing as we're trying to add threading support to
code which never had it originally. We've been aiming at this for
some time (over a decade) but have reached the point where we need to 
do it and deal with any issues arrising.

The 'idle' thread is badly named as it is where the "real work"
of bitbake gets done, e.g. runqueue (task execution) is run as an 
idle handler. We can worry about renaming things later, getting the
basic changes working is my concern right now.

The problem these changes solve:
  * allows the UI to tell if the server is alive and still there
    even when async commands are running
  * allows the UI to interrupt the server and tell it to shutdown
    (i.e. make Ctrl+C repsonsive from the UI)
  * allow us to make progress on the various "bitbake hanging" bugs
    we have where the UI can't connect

There is much more work needed here and the code can be considerably
cleaned up to make it clearer and most accessible, those changes
can follow and build from this though.

Thanks to Joshua Watt for some tweaks to improve the code.

v2 changes the threading event into a condition and cleans up
some locking on the idle functions. As part of the changes it made
sense to reorderand clean up the series. One part of the previous
series was merged to try and fix issues being seen with master.

Richard Purdie (11):
  knotty: Avoid looping with tracebacks
  event: Add enable/disable heartbeat code
  cooker/cookerdata: Rework the way the datastores are reset
  server/process: Improve exception and idle function logging
  command: Tweak finishAsyncCommand ordering to avoid races
  cooker: Ensure commands clean up any parser processes
  server/process: Improve idle loop exit code
  event: Always use threadlock
  server/process: Add locking around idle functions accesses
  server/process: Run idle commands in a separate idle thread
  knotty: Ping the server/cooker periodically

 lib/bb/command.py             |  26 ++++----
 lib/bb/cooker.py              |  58 +++++++++++-------
 lib/bb/cookerdata.py          |  31 ++++++----
 lib/bb/event.py               |  82 +++++++++++++------------
 lib/bb/server/process.py      | 111 +++++++++++++++++++++++++---------
 lib/bb/server/xmlrpcserver.py |   2 +-
 lib/bb/tests/event.py         |  17 +-----
 lib/bb/ui/knotty.py           |  16 +++--
 8 files changed, 207 insertions(+), 136 deletions(-)