Message ID | 20211202220517.2106590-1-quaresma.jose@gmail.com |
---|---|
State | New |
Headers | show |
Series | [1/2] sstate: don't limit the thread pool size when checking mirrors | expand |
On Thu, 2021-12-02 at 22:05 +0000, Jose Quaresma wrote: > This improves the time needed to check the mirrors, > reducing it from 50s to 8s when building the core-image-minimal > using the yocto upstream sstate and hashequivlance servers > on a machine with 8 cpu cores. > > Tested with: > > SSTATE_MIRRORS = "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH" > BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687" > bitbake core-image-minimal > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > --- > meta/classes/sstate.bbclass | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > index 0326d27c74..5e404d7cd8 100644 > --- a/meta/classes/sstate.bbclass > +++ b/meta/classes/sstate.bbclass > @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, > tasklist.append((tid, sstatefile)) > > if tasklist: > - nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist)) > + nproc = len(tasklist) > > progress = len(tasklist) >= 100 > if progress: No, we're not doing that. That opens a ton (thousands) of parallel connections to the servers and isn't fair on the systems hosting the data. In many cases that would look like a DoS attack. Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quinta, 2/12/2021 à(s) 23:16: > On Thu, 2021-12-02 at 22:05 +0000, Jose Quaresma wrote: > > This improves the time needed to check the mirrors, > > reducing it from 50s to 8s when building the core-image-minimal > > using the yocto upstream sstate and hashequivlance servers > > on a machine with 8 cpu cores. > > > > Tested with: > > > > SSTATE_MIRRORS = "file://.* > http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH" > > BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687" > > bitbake core-image-minimal > > > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > > --- > > meta/classes/sstate.bbclass | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > > index 0326d27c74..5e404d7cd8 100644 > > --- a/meta/classes/sstate.bbclass > > +++ b/meta/classes/sstate.bbclass > > @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, > currentcount=0, summary=True, > > tasklist.append((tid, sstatefile)) > > > > if tasklist: > > - nproc = min(int(d.getVar("BB_NUMBER_THREADS")), > len(tasklist)) > > + nproc = len(tasklist) > > > > progress = len(tasklist) >= 100 > > if progress: > > No, we're not doing that. That opens a ton (thousands) of parallel > connections > to the servers and isn't fair on the systems hosting the data. In many > cases > that would look like a DoS attack. > > Cheers, > > Richard > > > > Got it and I have to agree that it will launch a DoS to the server. However there are some cases around that have the mirror hosted locally on the file system. I'll try to find a solution that doesn't affect the http mirror.
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 0326d27c74..5e404d7cd8 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, tasklist.append((tid, sstatefile)) if tasklist: - nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist)) + nproc = len(tasklist) progress = len(tasklist) >= 100 if progress:
This improves the time needed to check the mirrors, reducing it from 50s to 8s when building the core-image-minimal using the yocto upstream sstate and hashequivlance servers on a machine with 8 cpu cores. Tested with: SSTATE_MIRRORS = "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH" BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687" bitbake core-image-minimal Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> --- meta/classes/sstate.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)