From patchwork Fri Nov 10 13:54:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin Jansa X-Patchwork-Id: 34257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E949C4332F for ; Fri, 10 Nov 2023 13:54:55 +0000 (UTC) Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by mx.groups.io with SMTP id smtpd.web11.27840.1699624489014687284 for ; Fri, 10 Nov 2023 05:54:49 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=gD/7XR+O; spf=pass (domain: gmail.com, ip: 209.85.218.44, mailfrom: martin.jansa@gmail.com) Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-9e62f903e88so102885966b.2 for ; Fri, 10 Nov 2023 05:54:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699624487; x=1700229287; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Kz+QMGfoYENFiTP4A2/snXyk8OjWOYzYu3MEMUiaKqQ=; b=gD/7XR+OCoDl9O12hyjyl69QlNxJtc6WCrsLIE2eG93w0VhGfbrUtlITtPVck/oSlD nUe5N4jr4BosqgSwxKG+/8ydUdQQqeHxlUaKeFkQjs9tQJc2g9sUi4W6hgx9OtiAfjun GbHCjM0mHoTuhaL4Ov0X6UWmDO/obTzCTxuYkhRZmaokeoaGenn6ANDgfL8+E8xMeGOy d/ZJfzru8KpO5lKu2jf0JFN6u5w53ePDlwcT7+xd+EV9EJnt8LsP9JXZjSSNWL1KkPpB wb0UetiHAXVOs+PhS+Um8rdbbVA9IUInW6mbGsnmNaoRSge2lipmWq4NrPfBFXmeEtJ1 LTgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699624487; x=1700229287; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Kz+QMGfoYENFiTP4A2/snXyk8OjWOYzYu3MEMUiaKqQ=; b=LAGJMOjR+lX6HLkHRG7+9RxFtqCupeLxFM0akbNbwNcvph5c+cPeLTgOpFAY2iseIT RPLIySly60G5qVlYd/CRV6m2ub9B7Baz5nE6RhQyDLwbt8XI2JxFq9wCQ91csBfBwhoJ OJ6bWkPwZZ6jImfQEsWoK8wcMtHwca0S7bDfKw7xfXLGhyawO595/CFLdDMuPrKeNVUN 0bV0Iv1sQ8zEmuiw3KfDYZDVq76zT1bnR5i4LnMwG1QxmmIFZHBffL+ZW+YaXAjjUU3d eza9vwbsrDqqIXDmjWI4CJBK8dZ4sZx5RqjpwMBeOwC46TV4fd9vT++dMgooPDT55ImD eOcg== X-Gm-Message-State: AOJu0YzxnWnWnCUtuQmqodQ0vqh7nUgrAagTCiHBLTgjWaSj5tIutfmL 6LMRFyPUR/WsMJAbcu5f4i/bsPRv63E= X-Google-Smtp-Source: AGHT+IEDWNILRwLg9IzYLpjA+TBbhvVF+YYVYcGzrgKJoVtgGMBgs3b5C+TZUpouPPOJMB0hcG6UTQ== X-Received: by 2002:a17:906:ef0c:b0:9c4:344e:b496 with SMTP id f12-20020a170906ef0c00b009c4344eb496mr7196337ejs.11.1699624487380; Fri, 10 Nov 2023 05:54:47 -0800 (PST) Received: from localhost (ip-109-238-218-228.aim-net.cz. [109.238.218.228]) by smtp.gmail.com with ESMTPSA id p27-20020a170906141b00b009adc5802d08sm3953133ejc.190.2023.11.10.05.54.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 05:54:46 -0800 (PST) From: Martin Jansa To: bitbake-devel@lists.openembedded.org Cc: steve@sakoman.com, Chen Qi , Richard Purdie Subject: [2.0][PATCH 1/5] runqueue: fix PSI check calculation Date: Fri, 10 Nov 2023 14:54:33 +0100 Message-ID: <20231110135437.2902924-1-martin.jansa@gmail.com> X-Mailer: git-send-email 2.42.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 10 Nov 2023 13:54:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15494 From: Chen Qi The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. Signed-off-by: Chen Qi Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 8605c46c8..3e89c38bc 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -198,15 +198,20 @@ class RunQueueScheduler(object): curr_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1] curr_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1] curr_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1] - exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) > self.rq.max_cpu_pressure - exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) > self.rq.max_io_pressure - exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) > self.rq.max_memory_pressure now = time.time() - if now - self.prev_pressure_time > 1.0: + tdiff = now - self.prev_pressure_time + if tdiff > 1.0: + exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff > self.rq.max_cpu_pressure + exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff > self.rq.max_io_pressure + exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff > self.rq.max_memory_pressure self.prev_cpu_pressure = curr_cpu_pressure self.prev_io_pressure = curr_io_pressure self.prev_memory_pressure = curr_memory_pressure self.prev_pressure_time = now + else: + exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) > self.rq.max_cpu_pressure + exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) > self.rq.max_io_pressure + exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) > self.rq.max_memory_pressure return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure) return False From patchwork Fri Nov 10 13:54:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin Jansa X-Patchwork-Id: 34260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADDCEC04E83 for ; Fri, 10 Nov 2023 13:54:55 +0000 (UTC) Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by mx.groups.io with SMTP id smtpd.web11.27841.1699624490132443519 for ; Fri, 10 Nov 2023 05:54:50 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=faSgUVY+; spf=pass (domain: gmail.com, ip: 209.85.218.46, mailfrom: martin.jansa@gmail.com) Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-9e28724ac88so341238666b.2 for ; Fri, 10 Nov 2023 05:54:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699624488; x=1700229288; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bE4iSbRE95DYeQJn9m9ODF0mx/nr33IC05l3IylHnVg=; b=faSgUVY+oOPmGD9W8lMZ7P/cXnd5IcaMXDaBSd5fOcSp8BwTaXwsNniXkcde40oT8K q1YWfScGnw7B1gixHO5esUJuAgXcsQNi2nAipRoaUez0hyBBLZhPObgsbVrt0vY0NTy9 iZt0r4jpLk5asubAu88bpPlPqLFr6WKbFd2+36Ak2/TvHkmNJEih40ce9Iy8p0jCycDe 8jXeWDKJ1l6WyQ55lgHy4W6eZwgFgcImN1RTVahqqm8gQhq2IHQY58cF09Fxv51R8Na9 r4zRlckU0MXKjsbTZX3NZoXeLaawu1ABLdy6+EW6TVeIqA1r1bNwXDAAim5doUHShaFf Gjig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699624488; x=1700229288; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bE4iSbRE95DYeQJn9m9ODF0mx/nr33IC05l3IylHnVg=; b=RuLuPDIi7RUjdBvvRg+AFh7sU+aEBGEHDoN5AqLSOv3Tj4JSgTsnmrpmHOjCVtWXfb OdkXxXcgRZEf0Qt3bHJNvMqpCypv9rGKIqUBnADQMSs44vuqlKJD4WsuNjAzra7ldR4m /gUBFxe9zVwZjgnoSJE0M9Fvl/J+f3FjL8S9Z3N+RpsIrAXSQprVB+qhWphpYtgaBBDN /RIsN8pzzU4MunmF1M/rvPTp7JEWhl1EDvU777bYG+tgvUKmFJeJMuZYT695wzfcSK2/ TrGnZaiHEKL1BqU+j+z+PtaaGxonK1Gu32KUIvWUZjkppQLfEgfLtv/uInEUoneT5RWX VH8Q== X-Gm-Message-State: AOJu0Yx2oVXYFVCcv6o5ACXiB40vngYJfD93IjJTTPSLt3wQoS2s7HzT p+MhYGEqcG9cnM97gzb4dfgaJ4xBwXI= X-Google-Smtp-Source: AGHT+IGBV4Mt8nIT/m+VCC2SqmUzCYM0bJDbmz64CmPIbicyme1L4+cYgx0mgkIWhr9Jutg/mLdZJA== X-Received: by 2002:a17:907:7255:b0:9de:dc3d:3b2 with SMTP id ds21-20020a170907725500b009dedc3d03b2mr7228319ejc.16.1699624488384; Fri, 10 Nov 2023 05:54:48 -0800 (PST) Received: from localhost (ip-109-238-218-228.aim-net.cz. [109.238.218.228]) by smtp.gmail.com with ESMTPSA id h13-20020a17090791cd00b009e5f5efac10sm1070923ejz.208.2023.11.10.05.54.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 05:54:47 -0800 (PST) From: Martin Jansa To: bitbake-devel@lists.openembedded.org Cc: steve@sakoman.com, Richard Purdie Subject: [2.0][PATCH 2/5] runqueue: Add pressure change logging Date: Fri, 10 Nov 2023 14:54:34 +0100 Message-ID: <20231110135437.2902924-2-martin.jansa@gmail.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231110135437.2902924-1-martin.jansa@gmail.com> References: <20231110135437.2902924-1-martin.jansa@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 10 Nov 2023 13:54:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15495 From: Richard Purdie It is currently hard to tell when bitbake is throttling task execution due to system pressure changes. Add notes to the console output to make this clearer, only generating output when the values change. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 3e89c38bc..ab9e3fbe8 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -212,6 +212,10 @@ class RunQueueScheduler(object): exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) > self.rq.max_cpu_pressure exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) > self.rq.max_io_pressure exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) > self.rq.max_memory_pressure + pressure_state = (exceeds_cpu_pressure, exceeds_io_pressure, exceeds_memory_pressure) + if hasattr(self, "pressure_state") and pressure_state != self.pressure_state: + bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s" % pressure_state) + self.pressure_state = pressure_state return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure) return False From patchwork Fri Nov 10 13:54:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin Jansa X-Patchwork-Id: 34261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FAE1C4167D for ; Fri, 10 Nov 2023 13:54:55 +0000 (UTC) Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by mx.groups.io with SMTP id smtpd.web10.27573.1699624490691717507 for ; Fri, 10 Nov 2023 05:54:51 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=lQiJz5Cz; spf=pass (domain: gmail.com, ip: 209.85.208.52, mailfrom: martin.jansa@gmail.com) Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-53e04b17132so3361242a12.0 for ; Fri, 10 Nov 2023 05:54:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699624489; x=1700229289; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gI0KOUJltVijVagHWsEo8zxgJqvTT1b1HDhLCFNnixQ=; b=lQiJz5CzK/uRNxleTtA9MM8OQk+WpABq4Bz/FyTedVuEcPPCzQKyC0/PJ5/E+eZGRg bF6BmGfHOiWz50TqkweYrFb2gusjBjQV3XxZMYEhE28tJJCuS5X8/4pA3QNVhM6xSSz5 utloQO2h1Dj7I2uqTugYdrpkZRTqPnLOaIFGfVgn2J/9eD3ZlrgQJGthFZ5FaolywoTl GsBpyjP4A9/FYKN+7qPI29kGpvJ1uQsYcb/4T/nLnagr2xwITJMn62Bco2iJib9bqTIp 3xFIFzjqjj6+j4nJgApojFDiUE8OgRgpIYl67hw+RAnno2LYEWd6BfADqOqXrDR7Ju+x 6gPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699624489; x=1700229289; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gI0KOUJltVijVagHWsEo8zxgJqvTT1b1HDhLCFNnixQ=; b=JP030jagfXFP+VoaOlqnkOc3nAWqoUKYxmO4kKd2DcR1YMsHQ09HCQ9oKymEx7P0QP pl2sJFK9M7Sk+Qn5/3pP9NJUx+HWrtuVSpdzojq6BrntVnoL18JUByHW/y5loM5PdrW3 MNss1JHVcU6e8zJ+1nrXEffDa33BKBDcY+05+Hf+gi+6723DUe+oL/cON52Sd+7evlul JQJzfGiUdRm/9Ts+IVcCOs51WeA+YcFegcNYFVkMfJ2wvuPd9FSq7I8kYT0tzSeIxINH ERRKNfJMabe8FmCRIo4ZWkBUqYsXrHGBmCNZ0uIprYLZr/5yPkogsQUtLDnjsDhvN5qc XcDg== X-Gm-Message-State: AOJu0YxoiFIaWoQCFcqfVmUa6jdtXremskeRrh4dgnKRh74iYbXvrF8/ ZiUTVRWDXbanS9bHRpbC6YPYbON0Zzw= X-Google-Smtp-Source: AGHT+IHkxpZ8oJK0tpbClQqWTGJP6Zf9VmKTfuXgUZxEiTMIltjAR0G6uI0Y5UWfw0GqDj9dXLkUtQ== X-Received: by 2002:a50:9fc8:0:b0:543:6444:ef2a with SMTP id c66-20020a509fc8000000b005436444ef2amr6033449edf.32.1699624489183; Fri, 10 Nov 2023 05:54:49 -0800 (PST) Received: from localhost (ip-109-238-218-228.aim-net.cz. [109.238.218.228]) by smtp.gmail.com with ESMTPSA id s20-20020a50d494000000b005434e3d8e7bsm1195419edi.1.2023.11.10.05.54.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 05:54:48 -0800 (PST) From: Martin Jansa To: bitbake-devel@lists.openembedded.org Cc: steve@sakoman.com, Chen Qi , Richard Purdie Subject: [2.0][PATCH 3/5] runqueue.py: fix PSI check logic Date: Fri, 10 Nov 2023 14:54:35 +0100 Message-ID: <20231110135437.2902924-3-martin.jansa@gmail.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231110135437.2902924-1-martin.jansa@gmail.com> References: <20231110135437.2902924-1-martin.jansa@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 10 Nov 2023 13:54:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15496 From: Chen Qi The current calculation is not correct because if tdiff is less than 1.0, it's not taken into consideration when calculating the current pressure. Also, make it clear that the 1.0s is the psi accumulation cycle, which might be changed in the future. We have this cycle because it could largely avoid the 0 result issue, that is, if the interval between checks are too small, the result might be 0. With this accumulation logic, which has been there but let's make it clear, this 0 result problem could be mitigated. Signed-off-by: Chen Qi Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index ab9e3fbe8..830af8eef 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -200,7 +200,8 @@ class RunQueueScheduler(object): curr_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1] now = time.time() tdiff = now - self.prev_pressure_time - if tdiff > 1.0: + psi_accumulation_interval = 1.0 + if tdiff > psi_accumulation_interval: exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff > self.rq.max_cpu_pressure exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff > self.rq.max_io_pressure exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff > self.rq.max_memory_pressure @@ -209,9 +210,9 @@ class RunQueueScheduler(object): self.prev_memory_pressure = curr_memory_pressure self.prev_pressure_time = now else: - exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) > self.rq.max_cpu_pressure - exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) > self.rq.max_io_pressure - exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) > self.rq.max_memory_pressure + exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff > self.rq.max_cpu_pressure + exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff > self.rq.max_io_pressure + exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff > self.rq.max_memory_pressure pressure_state = (exceeds_cpu_pressure, exceeds_io_pressure, exceeds_memory_pressure) if hasattr(self, "pressure_state") and pressure_state != self.pressure_state: bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s" % pressure_state) From patchwork Fri Nov 10 13:54:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin Jansa X-Patchwork-Id: 34258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D678C4167B for ; Fri, 10 Nov 2023 13:54:55 +0000 (UTC) Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by mx.groups.io with SMTP id smtpd.web10.27574.1699624491601403220 for ; Fri, 10 Nov 2023 05:54:51 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=jt0zmJ1E; spf=pass (domain: gmail.com, ip: 209.85.218.49, mailfrom: martin.jansa@gmail.com) Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-991c786369cso349506866b.1 for ; Fri, 10 Nov 2023 05:54:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699624490; x=1700229290; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X0CC3wgvi5i52KcqgzGTU3OvQVAPBOJAc1whAOehGbU=; b=jt0zmJ1EDyHMi56JHP2dEU1vROmqGtCUfNOSeNlBFoTAKopPuLs8WJmDr5yDbDsMDu pq2+LKXiWUGuxEJXE4SumQjgW01oHO8Z4vGKoMQk8qsMpJh9qafAhNOZYF+ZUsOXR374 NtnNkqvaLqXWunKV6XKcMJSZ7VeIPNvR4uiKL+v8wHPeaiUuRbyxlhiyBVEPw9dCo84p v4CZlnpRFSDNmAjWPSi42JFEt5hdvrqBd01+8CVEbZl+ylaQDHuxMHg9yN65UhJxKV6E rGPl2qHFA23pZEuljlPvKuvJoUC7mcv93D0E36tK38mL+fvurIWQzPOfA/ae7UmP4Av0 m3fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699624490; x=1700229290; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X0CC3wgvi5i52KcqgzGTU3OvQVAPBOJAc1whAOehGbU=; b=WYQpwgduUZHNlRNCNddlCcb/u/8n8011PuNPO9lgDMWe5gjF+defz3Ug7u7JxlJSJk dQHQBplUusr7drq7OXt/doH6n4Y7XsLLvnnrksgRZxsdrZXQKgl3bXJ6WIhL0TMbKlZt UBGQwWRwjN/QsByKKyo/+RjPYphug7b1cWuRf0OqgTc7l2bKgWsPKc1aVW+83+KPa8Or DtEUwzgVY8dz2E0whGctn1a5ru5M5olt2GhPAImjMKcdCLAIJC9eaQ+ka0NDa+30uhVI jPaumjtLYI0PXVUQOMAhpPWOhgMeHjDzUFqA9NWlxHWq7aq2ugmFMt9Q7B9/Yxl4Nyr/ 0jgQ== X-Gm-Message-State: AOJu0YwSpA0gz19viybEFW0zDoLI8WhK9loNADgPVwrsj9kFmuVdiTtj gYK2i3fhVY4V4hQKA/mIbSVvDFMZwbY= X-Google-Smtp-Source: AGHT+IG09bz+yJL8wIylN5CRInDndUabq4hhODxlScu62Ccab3EbP1okC47d3kLtkkjV8ceZCa851Q== X-Received: by 2002:a17:906:fe4c:b0:9d2:5cf8:e61 with SMTP id wz12-20020a170906fe4c00b009d25cf80e61mr6176482ejb.35.1699624489940; Fri, 10 Nov 2023 05:54:49 -0800 (PST) Received: from localhost (ip-109-238-218-228.aim-net.cz. [109.238.218.228]) by smtp.gmail.com with ESMTPSA id f13-20020a170906c08d00b009b928eb8dd3sm3936724ejz.163.2023.11.10.05.54.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 05:54:49 -0800 (PST) From: Martin Jansa To: bitbake-devel@lists.openembedded.org Cc: steve@sakoman.com, Martin Jansa Subject: [2.0][PATCH 4/5] runqueue: show more pressure data Date: Fri, 10 Nov 2023 14:54:36 +0100 Message-ID: <20231110135437.2902924-4-martin.jansa@gmail.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231110135437.2902924-1-martin.jansa@gmail.com> References: <20231110135437.2902924-1-martin.jansa@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 10 Nov 2023 13:54:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15497 From: Martin Jansa * with latest bitbake I'm seeing very low number of bitbake tasks executed in parallel, probably due to pressure regulation show the values this is based on in the note * also simplify a bit by counting the pressure and exceeds signs only once Signed-off-by: Martin Jansa --- lib/bb/runqueue.py | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 830af8eef..e01c496c4 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -201,21 +201,23 @@ class RunQueueScheduler(object): now = time.time() tdiff = now - self.prev_pressure_time psi_accumulation_interval = 1.0 + cpu_pressure = (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff + io_pressure = (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff + memory_pressure = (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff + exceeds_cpu_pressure = self.rq.max_cpu_pressure and cpu_pressure > self.rq.max_cpu_pressure + exceeds_io_pressure = self.rq.max_io_pressure and io_pressure > self.rq.max_io_pressure + exceeds_memory_pressure = self.rq.max_memory_pressure and memory_pressure > self.rq.max_memory_pressure + if tdiff > psi_accumulation_interval: - exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff > self.rq.max_cpu_pressure - exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff > self.rq.max_io_pressure - exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff > self.rq.max_memory_pressure self.prev_cpu_pressure = curr_cpu_pressure self.prev_io_pressure = curr_io_pressure self.prev_memory_pressure = curr_memory_pressure self.prev_pressure_time = now - else: - exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff > self.rq.max_cpu_pressure - exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff > self.rq.max_io_pressure - exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff > self.rq.max_memory_pressure + pressure_state = (exceeds_cpu_pressure, exceeds_io_pressure, exceeds_memory_pressure) + pressure_values = (round(cpu_pressure,1), self.rq.max_cpu_pressure, round(io_pressure,1), self.rq.max_io_pressure, round(memory_pressure,1), self.rq.max_memory_pressure) if hasattr(self, "pressure_state") and pressure_state != self.pressure_state: - bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s" % pressure_state) + bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s (CPU: %s/%s, IO: %s/%s, Mem: %s/%s)" % (pressure_state + pressure_values)) self.pressure_state = pressure_state return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure) return False From patchwork Fri Nov 10 13:54:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin Jansa X-Patchwork-Id: 34259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2802C04A95 for ; Fri, 10 Nov 2023 13:54:55 +0000 (UTC) Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by mx.groups.io with SMTP id smtpd.web10.27575.1699624492287621914 for ; Fri, 10 Nov 2023 05:54:52 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=FHSREFiT; spf=pass (domain: gmail.com, ip: 209.85.218.44, mailfrom: martin.jansa@gmail.com) Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-9dbb3e0ff65so326940466b.1 for ; Fri, 10 Nov 2023 05:54:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699624491; x=1700229291; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C/Ssk5ZljAlHZY1sVbMulbCihTS3HypldkckMP8jM9A=; b=FHSREFiTTC3no9xG/FaXHaQuBKb4Bc4v4pVlvMpdUvV9HktdZvftR1hrWoBw5p5UKd 28QoFYE+F72qhvtKA8+3umxo9s1O0ZPhVN35Rw/nezux1HJUlgjhY6t5TNx7lx6NOmyb AVoLZhOI+tb1ix1ZyRnJrShSJgJ/9m2OmwjbEIKUuq4jtUQPXbPYgSZwnJS2H6vLoa70 bb8O9niEk8tws6vsuwRuu4teBTeScFjPoOquIwXNiLals99NMdbEOs0/HY69fKij079W jFjUCSmzn5Uml1Dy9JPjp3H6SAD31jBNLmLGfdODp9yfBUvevsDnpPRKwpjO/vUr5s58 73ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699624491; x=1700229291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C/Ssk5ZljAlHZY1sVbMulbCihTS3HypldkckMP8jM9A=; b=LbHAiNAZaPQhiLLdE++trd9OzMZzhKXUIQMmmBwqFh7KvWSGweVnM5AfAui4ztPpEo +RKuMdt4DekU++lrgMWWKv+RhLTxkM3KkVabyPrsJloa7nSraAur90h5gOJsMmdTEFCg l+hmUQMwqgzPxXh5/LUpJXNQMnxvYEMol+uYr0Zfw4/VSlTNNKTTEASgiF4qoZzJPhzz XRqPCsRYaGY+2qC/9M5RDejae/Sdsc7ACZd9AaA6hqamu+JzQTVnpNB5arTe74Q63BeC q1OfZCqgnhpTgCvkuCexISl/kt1JwfdS62wbA0Jb/lAFQTY2p1SDnEXwWDf/84M2P+O2 gjcw== X-Gm-Message-State: AOJu0YxZIAq5+3deGuaIZTjiPH4Z6ecGQqcp8uqLJSJSDEP41e1q0xrP ZOb2f9XlI8CrVih+QAz1qAK62R3++8U= X-Google-Smtp-Source: AGHT+IG/J3517KKxZRqOCKFpUkCWgvCsCgLJui71S1Ti7Ot04oc0FdWcIZ3tJApxtRJUTeeWODiJrA== X-Received: by 2002:a17:906:4a95:b0:9e5:fdd7:f521 with SMTP id x21-20020a1709064a9500b009e5fdd7f521mr1469820eju.41.1699624490766; Fri, 10 Nov 2023 05:54:50 -0800 (PST) Received: from localhost (ip-109-238-218-228.aim-net.cz. [109.238.218.228]) by smtp.gmail.com with ESMTPSA id a17-20020a170906685100b009aa292a2df2sm3973000ejs.217.2023.11.10.05.54.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 05:54:50 -0800 (PST) From: Martin Jansa To: bitbake-devel@lists.openembedded.org Cc: steve@sakoman.com, Martin Jansa Subject: [2.0][PATCH 5/5] runqueue: show number of currently running bitbake threads when pressure changes Date: Fri, 10 Nov 2023 14:54:37 +0100 Message-ID: <20231110135437.2902924-5-martin.jansa@gmail.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231110135437.2902924-1-martin.jansa@gmail.com> References: <20231110135437.2902924-1-martin.jansa@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 10 Nov 2023 13:54:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15498 From: Martin Jansa * it might be a bit confusing as it shows number of threads before making the decision to start more tasks and also it can show only a few tasks running, but not because of pressure when there just aren't many tasks left or wait for their dependencies to be finished first * example output: NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads Signed-off-by: Martin Jansa --- lib/bb/runqueue.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index e01c496c4..0b6eddd45 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -217,7 +217,7 @@ class RunQueueScheduler(object): pressure_state = (exceeds_cpu_pressure, exceeds_io_pressure, exceeds_memory_pressure) pressure_values = (round(cpu_pressure,1), self.rq.max_cpu_pressure, round(io_pressure,1), self.rq.max_io_pressure, round(memory_pressure,1), self.rq.max_memory_pressure) if hasattr(self, "pressure_state") and pressure_state != self.pressure_state: - bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s (CPU: %s/%s, IO: %s/%s, Mem: %s/%s)" % (pressure_state + pressure_values)) + bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s (CPU: %s/%s, IO: %s/%s, Mem: %s/%s) - using %s/%s bitbake threads" % (pressure_state + pressure_values + (len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks))) self.pressure_state = pressure_state return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure) return False