| Message ID | 20240208152707.2849716-1-richard.purdie@linuxfoundation.org |
|---|---|
| State | New |
| Headers | show |
| Series | [yocto-autobuilder-helper] config.json: Try allowing a bit more parallelism | expand |
On 2024-02-08 10:27 a.m., Richard Purdie wrote: > Currently the autobuilders seem to run at 25% idle at times. We see a lof > of bouncing on IO load and it appears the systems could cope with higher > values. > > Increase the make load average, cpu and IO pressure limits a bit to > see if build times improve. > > Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org> > --- > config.json | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/config.json b/config.json > index e8b51cd..7539aeb 100644 > --- a/config.json > +++ b/config.json > @@ -49,9 +49,9 @@ > "BB_GENERATE_MIRROR_TARBALLS = '1'", > "BB_NUMBER_THREADS = '16'", > "BB_NUMBER_PARSE_THREADS = '16'", > - "PARALLEL_MAKE = '-j 16 -l 52'", > - "BB_PRESSURE_MAX_CPU = '10000'", > - "BB_PRESSURE_MAX_IO = '10000'", > + "PARALLEL_MAKE = '-j 16 -l 75'", A value of 75 for the load average limit seems a bit high given that there are only 56 v-cores on these machines but maybe that limit won't kick in if the pressure regulation works. I really need to fix and write tests for the pressure regulation but I'm just not able to make the time these days. Sorry. I guess this change is worth a try to get more data. ../Randy > + "BB_PRESSURE_MAX_CPU = '20000'", > + "BB_PRESSURE_MAX_IO = '20000'", > "XZ_MEMLIMIT = '5%'", > "XZ_THREADS = '8'", > "ZSTD_THREADS = '8'",
On Thu, 2024-02-08 at 18:13 -0500, Randy MacLeod wrote: > On 2024-02-08 10:27 a.m., Richard Purdie wrote: > > Currently the autobuilders seem to run at 25% idle at times. We see a lof > > of bouncing on IO load and it appears the systems could cope with higher > > values. > > > > Increase the make load average, cpu and IO pressure limits a bit to > > see if build times improve. > > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > --- > > config.json | 6 +++--- > > 1 file changed, 3 insertions(+), 3 deletions(-) > > > > diff --git a/config.json b/config.json > > index e8b51cd..7539aeb 100644 > > --- a/config.json > > +++ b/config.json > > @@ -49,9 +49,9 @@ > > "BB_GENERATE_MIRROR_TARBALLS = '1'", > > "BB_NUMBER_THREADS = '16'", > > "BB_NUMBER_PARSE_THREADS = '16'", > > - "PARALLEL_MAKE = '-j 16 -l 52'", > > - "BB_PRESSURE_MAX_CPU = '10000'", > > - "BB_PRESSURE_MAX_IO = '10000'", > > + "PARALLEL_MAKE = '-j 16 -l 75'", > A value of 75 for the load average limit seems a bit high given that > > there are only 56 v-cores on these machines but maybe that limit won't > kick in if the pressure regulation works. I'm working on the principle that 52 appears to be leaving the machines too underloaded. The scheduler is designed to handle load so having some new work always ready is probably better overall... > I really need to fix and write tests for the pressure regulation but > I'm just not able to make the time these days. Sorry. > I guess this change is worth a try to get more data. > I think that is where my thinking was at, we need to try something different as the performance at the moment just feels wrong somehow. Cheers, Richard
On 08/02/2024 23:31:49+0000, Richard Purdie wrote: > On Thu, 2024-02-08 at 18:13 -0500, Randy MacLeod wrote: > > On 2024-02-08 10:27 a.m., Richard Purdie wrote: > > > Currently the autobuilders seem to run at 25% idle at times. We see a lof > > > of bouncing on IO load and it appears the systems could cope with higher > > > values. > > > > > > Increase the make load average, cpu and IO pressure limits a bit to > > > see if build times improve. > > > > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > > --- > > > �config.json | 6 +++--- > > > �1 file changed, 3 insertions(+), 3 deletions(-) > > > > > > diff --git a/config.json b/config.json > > > index e8b51cd..7539aeb 100644 > > > --- a/config.json > > > +++ b/config.json > > > @@ -49,9 +49,9 @@ > > > �������������"BB_GENERATE_MIRROR_TARBALLS = '1'", > > > �������������"BB_NUMBER_THREADS = '16'", > > > �������������"BB_NUMBER_PARSE_THREADS = '16'", > > > - "PARALLEL_MAKE = '-j 16 -l 52'", > > > - "BB_PRESSURE_MAX_CPU = '10000'", > > > - "BB_PRESSURE_MAX_IO = '10000'", > > > + "PARALLEL_MAKE = '-j 16 -l 75'", > > >��A value of 75 for the load average limit seems a bit high given that > > > > there are only 56 v-cores on these machines but maybe that limit won't > > kick in if the pressure regulation works. > > I'm working on the principle that 52 appears to be leaving the machines > too underloaded. The scheduler is designed to handle load so having > some new work always ready is probably better overall... > > > > I really need to fix and write tests for the pressure regulation but� > > I'm just not able to make the time these days. Sorry. > > > I guess this change is worth a try to get more data. > > > > I think that is where my thinking was at, we need to try something > different as the performance at the moment just feels wrong somehow. The centos workers don't have pressure regulation and I have recently seen bitbake timeout on those s I'm a bit concerned this will exacerbate the issue.
On Fri, 2024-02-09 at 00:40 +0100, Alexandre Belloni wrote: > On 08/02/2024 23:31:49+0000, Richard Purdie wrote: > > On Thu, 2024-02-08 at 18:13 -0500, Randy MacLeod wrote: > > > On 2024-02-08 10:27 a.m., Richard Purdie wrote: > > > > Currently the autobuilders seem to run at 25% idle at times. We see a lof > > > > of bouncing on IO load and it appears the systems could cope with higher > > > > values. > > > > > > > > Increase the make load average, cpu and IO pressure limits a bit to > > > > see if build times improve. > > > > > > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > > > --- > > > > config.json | 6 +++--- > > > > 1 file changed, 3 insertions(+), 3 deletions(-) > > > > > > > > diff --git a/config.json b/config.json > > > > index e8b51cd..7539aeb 100644 > > > > --- a/config.json > > > > +++ b/config.json > > > > @@ -49,9 +49,9 @@ > > > > "BB_GENERATE_MIRROR_TARBALLS = '1'", > > > > "BB_NUMBER_THREADS = '16'", > > > > "BB_NUMBER_PARSE_THREADS = '16'", > > > > - "PARALLEL_MAKE = '-j 16 -l 52'", > > > > - "BB_PRESSURE_MAX_CPU = '10000'", > > > > - "BB_PRESSURE_MAX_IO = '10000'", > > > > + "PARALLEL_MAKE = '-j 16 -l 75'", > > > > > A value of 75 for the load average limit seems a bit high given that > > > > > > there are only 56 v-cores on these machines but maybe that limit won't > > > kick in if the pressure regulation works. > > > > I'm working on the principle that 52 appears to be leaving the machines > > too underloaded. The scheduler is designed to handle load so having > > some new work always ready is probably better overall... > > > > > > > I really need to fix and write tests for the pressure regulation but > > > I'm just not able to make the time these days. Sorry. > > > > > I guess this change is worth a try to get more data. > > > > > > > I think that is where my thinking was at, we need to try something > > different as the performance at the moment just feels wrong somehow. > > The centos workers don't have pressure regulation and I have recently > seen bitbake timeout on those s I'm a bit concerned this will > exacerbate the issue. It is a valid concern. My last build did feel "faster" but whether that was a fair comparison, not sure. Lets keep an eye on things and keep this in mind... Cheers, Richard
diff --git a/config.json b/config.json index e8b51cd..7539aeb 100644 --- a/config.json +++ b/config.json @@ -49,9 +49,9 @@ "BB_GENERATE_MIRROR_TARBALLS = '1'", "BB_NUMBER_THREADS = '16'", "BB_NUMBER_PARSE_THREADS = '16'", - "PARALLEL_MAKE = '-j 16 -l 52'", - "BB_PRESSURE_MAX_CPU = '10000'", - "BB_PRESSURE_MAX_IO = '10000'", + "PARALLEL_MAKE = '-j 16 -l 75'", + "BB_PRESSURE_MAX_CPU = '20000'", + "BB_PRESSURE_MAX_IO = '20000'", "XZ_MEMLIMIT = '5%'", "XZ_THREADS = '8'", "ZSTD_THREADS = '8'",
Currently the autobuilders seem to run at 25% idle at times. We see a lof of bouncing on IO load and it appears the systems could cope with higher values. Increase the make load average, cpu and IO pressure limits a bit to see if build times improve. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- config.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)