| Message ID | 20220419094616.433632-2-quaresma.jose@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] sstate: add a LockedSet class to be used on the mirror thread pool | expand |
On Tue, 2022-04-19 at 10:46 +0100, Jose Quaresma wrote: > We don't need a new copy of the bitbake data [localdata2] in each running > thread of the pool. > > We can do the copy on the startup of the thread pool in each thread worker > and reuse it in each running thread, this is the same we did for the > connection_cache which is reused by fetcher. > > May be related with [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > --- > meta/classes/sstate.bbclass | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) It is a standard coding practise where you're changing the data store and don't want changes to persist, to use createCopy() on the datastore. It is fast and cheap and I'm not sure we need to make this thread specific? Was there a reason you thought we should change that to fix some specific issue? Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia terça, 19/04/2022 à(s) 11:57: > On Tue, 2022-04-19 at 10:46 +0100, Jose Quaresma wrote: > > We don't need a new copy of the bitbake data [localdata2] in each running > > thread of the pool. > > > > We can do the copy on the startup of the thread pool in each thread > worker > > and reuse it in each running thread, this is the same we did for the > > connection_cache which is reused by fetcher. > > > > May be related with [YOCTO #14775] -- > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 > > > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > > --- > > meta/classes/sstate.bbclass | 6 +++--- > > 1 file changed, 3 insertions(+), 3 deletions(-) > > It is a standard coding practise where you're changing the data store and > don't > want changes to persist, to use createCopy() on the datastore. It is fast > and > cheap and I'm not sure we need to make this thread specific? > My interpretation of the use of the createCopy was so that the existing one localdata, would not be used at the same time in the thread pool. i.e: each thread has its own localdata2 copied from localdata. In my patch the behavior is the same, each thread has its own localdata2 but the call of createCopy is done in a safe place. > > Was there a reason you thought we should change that to fix some specific > issue? > I suspect that there may be some race condition when calling the createCopy on the thread pool. If I understand well, the createCopy on the bitbake/lib/bb/data_smart.py copies a couple of structures. So I don't know if the call createCopy is safe to do inside the tread pool, if you think so please drop this patch. Jose > Cheers, > > Richard > >
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index ebc336b497..7526ec7562 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -979,6 +979,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, from bb.fetch2 import FetchConnectionCache def checkstatus_init(thread_worker): thread_worker.connection_cache = FetchConnectionCache() + thread_worker.localdata2 = bb.data.createCopy(localdata) def checkstatus_end(thread_worker): thread_worker.connection_cache.close_connections() @@ -986,15 +987,14 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, def checkstatus(thread_worker, arg): (tid, sstatefile) = arg - localdata2 = bb.data.createCopy(localdata) srcuri = "file://" + sstatefile - localdata2.setVar('SRC_URI', srcuri) + thread_worker.localdata2.setVar('SRC_URI', srcuri) bb.debug(2, "SState: Attempting to fetch %s" % srcuri) import traceback try: - fetcher = bb.fetch2.Fetch(srcuri.split(), localdata2, + fetcher = bb.fetch2.Fetch(srcuri.split(), thread_worker.localdata2, connection_cache=thread_worker.connection_cache) fetcher.checkstatus() bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
We don't need a new copy of the bitbake data [localdata2] in each running thread of the pool. We can do the copy on the startup of the thread pool in each thread worker and reuse it in each running thread, this is the same we did for the connection_cache which is reused by fetcher. May be related with [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> --- meta/classes/sstate.bbclass | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)