From patchwork Tue Jun 21 16:14:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aryaman Gupta X-Patchwork-Id: 9420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BDC9C43334 for ; Tue, 21 Jun 2022 16:14:35 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.35000.1655828071093705997 for ; Tue, 21 Jun 2022 09:14:32 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriver.com header.s=pps06212021 header.b=nWzBYpGK; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=3171c08643=aryaman.gupta@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25LDTFrY004368 for ; Tue, 21 Jun 2022 09:14:30 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : subject : date : message-id : content-transfer-encoding : content-type : mime-version; s=PPS06212021; bh=SHqpb6CSW8CRPBLL2I8Ydj+EO+dRgwiD1oEXjo9eIzE=; b=nWzBYpGKD7i1ByElxa1xvCiI2h6cOmSAXHkhJXA1LxbFBGKNBy/wJkBc/wUHVf+kDKwk zAWiwnWCEEaUMvgHO3RO0kEiIV3MCnQcTQjPKcNvfsZ/5m6HBuSC69xU4bnuqDnlIMUg sWxC1Q1WMOmnj4yvEWcb4o5Ux5HlaJ22jxlp6UaeHxCheOwbXR2fClencSZ+rPLR8C04 scclrRJuIJFsZFSkHQKRMW9KGU+wUlyU8edD3HreHj6QV9Efk9Dn9Pf7JYrqsdnpv66/ O5CPz2F4t2Mxrv32xZFZDv6i7WD7dLo467h4tsP0mp2snSPC6sSMrJrYeCEixisL+Eg/ UA== Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3gsa06a7ud-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 21 Jun 2022 09:14:30 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mmSEGPcuAaSkt8WnmorKhYCChOrou2LGBuw/dIdk/7JP+SbRM+Mtrvb5I5eBkPpK3VP187V4Ea9CqHmqKtkFl9g832u+KCv+fjUOGJs13J9G9TC/7Lb7IGgd+gV7scAs+eaeGX0+9BQPiPccjkaatAANifrCv1SR5EeygXYx+D9NjPU7gj5Ue2ZBdESwNXZDXvBm89wu2GSPbbt1ZamHsb5hpPNVbCifEINluRibe3yj+e2I7ciGaNxxSktc8w3iBpBU76qQlBCqy+Wd4AoghQVAFpB3ArvuAEV4vYRdaqDqmoaVqbqUjlivNxs3aH7OSb4nSzr0sPIF/l50mXUlxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SHqpb6CSW8CRPBLL2I8Ydj+EO+dRgwiD1oEXjo9eIzE=; b=iOp9gre+D0Akj/8sZuOQO4b6jQyhGB779hfuSIuAxQRvM/e2biGHd8iz9ZkJlN7BjkLgwzHOqTG9F9BxKGdjZdg1eES8RMOk5+UKJtIVkjwnhR6UCs+9Q/SwidvZ3goS5BxojSe677lfu2cWZSyIBpcKkrx16V3Pb3QANCnolmptkwnR+Quvb6KODNnlrL/k8Y9gEgo+aJnHYhkoVf4ZAl9h8zsWoM90+lwnvkRDm/jGoQTML/KwoQLbruLxi3pysDrbvQb+dS0Oi48mZgu1m8rGKLmiPBHmxjEKHcv2Nfw1j08oIs/uQW0j34NE18QO3uz5aNLP76XnJIshY//eZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from PH7PR11MB6030.namprd11.prod.outlook.com (2603:10b6:510:1d1::19) by BN7PR11MB2865.namprd11.prod.outlook.com (2603:10b6:406:ac::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.16; Tue, 21 Jun 2022 16:14:26 +0000 Received: from PH7PR11MB6030.namprd11.prod.outlook.com ([fe80::14a3:342b:5f14:eda6]) by PH7PR11MB6030.namprd11.prod.outlook.com ([fe80::14a3:342b:5f14:eda6%7]) with mapi id 15.20.5353.018; Tue, 21 Jun 2022 16:14:26 +0000 From: Aryaman Gupta To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 1/3] buildstats.py: enable collection of /proc/pressure data Date: Tue, 21 Jun 2022 12:14:11 -0400 Message-Id: <20220621161413.2779759-1-aryaman.gupta@windriver.com> X-Mailer: git-send-email 2.35.1 X-ClientProxiedBy: YQBPR0101CA0328.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:6c::20) To PH7PR11MB6030.namprd11.prod.outlook.com (2603:10b6:510:1d1::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d9aea868-46ce-4f34-aa41-08da53a11c98 X-MS-TrafficTypeDiagnostic: BN7PR11MB2865:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pec/uykh6q5F+y8Ln2vrYclh2GE1jprh8BKCuniGEwcW9Zd0k/DqGU/SEbVI7EM0e/Q7z2BGQI+RLTfh7qfTYi8Tye00RZ2T6in3w3AD+0n4z9tXVjEt6fSYeL1nf1GuWA5Y4savDvThRmex7FZEuFoi8d1qsqxaLaZ+xcQfcFuvJhQpkYWT5RAlC6SNZEaOutwRKAuD4ym+1NBvaXBKzrj14MuLYbTcCxplIs96LOD2MSwXSC3k9aBAxKtwDBoVAOAtpy0ey4/KA+zv/sLXgIWvnUZvvtlg+nTjt4HtFpBjb1ffTaygVbx9yFA2MfTBjhnr1SGk/5RltWgfcTP4Pmxh98jsGUquJ6pODUFyOuqMe3HuwVuEqeldb0eoDi3vs7Jy8Q1epfzmRran6zjEWwbZObz5W19v5MR0xIIq93rZBBXD6VC7bDZGJbKxDVJs3G6v8m9Dj+cnkCyEuE0lRQ5EiIKJmIGOdIEg5ApUQ60qCJSxcRQkq8qLx6oO72F4PdZBpP9/vY1/XAy5m5j9i2WGhesD1BVptJLm4Eb1nGvuokx4zt77P8WB5XqK4jtMdRg/3dz0kyCBXFn4/D5SIS89cKLsmDx+OZl+MiiM11zJmmQbf5RY1xboBuKlZhJMSvtq+fJkp9dhDBVwjj1tAsyHde5nFFm14vK3xsUhRG9fO7e73zhMbpqZxf+d3LZypJppO+oQ4gYDeabJZaaRc06LUvTtehK2ahcvTk7uz0rpZdwiaOvj26b+NE7ZIthOhjElKgeyDuPi/F5rWLjmszmQ5I5mx9UldLuAU43pZ2s= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH7PR11MB6030.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(376002)(346002)(366004)(396003)(136003)(39850400004)(52116002)(6506007)(1076003)(186003)(66556008)(41300700001)(44832011)(6666004)(2616005)(66476007)(26005)(83380400001)(38350700002)(6512007)(38100700002)(6916009)(36756003)(5660300002)(86362001)(2906002)(66946007)(316002)(8936002)(8676002)(6486002)(478600001)(966005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Nq8FKg9E25RA/7mi+JDs5sxBCBsOZmtkXZdJf21heWtU0FQJQYBiQzzfXKKtqXJBP4U5ZaavpUBaG8s3pwdpde6KFInaeV/QtGTKNZCvtLFNWwMv4BoE/Q7PEHHnLQNBO6EyN5VQJWqETkn7GjJZJL+3m5ty4yoJ3toKonS6oeRtNCdrS3/h7WD55UeZ7eCzRxikmcicqBhJUkeIMfdNwKjc2J5f6NOjG3YLfCH0ygJVmYYPGJ/MPfA+TGZZGqjsrBvql9LPIhDiIi8nkMzFTJJZJRRraCNuWhCjZ+bm+PkH1jEDaZr1LQpT1qOX6QOTx1QHvKGXRKeT3JAmGcL8mSFicNdkV9O7r6yKjMKYszyAw4LcBc3+ArxIDABj72QJBs55loqt5hjc7I20OcFMqcTQIc/ykeIG/2BH3170wxitr7RR8lsCuLs0nV7tqBrF3x+C5oeOKPN5AH1/kTtQI69VJw1073B//TbnbcxOlMcwk5OtLiQ3dS8ATNM4keT7f0yOopfUZ8iByyyix5N2PD47Y14WWKdbRFVTqgkahAB7mqIGX9Q/u6vDSZJgKmZnELXkjmR2D5W/lFPNtIxSqoz76X/DIzvsx5mtDxbeuzjdF8rsmPN+Oy1dzZd6pRJapGvVMuMph7phjQUfAEzedWPggAP5tcRWCSQz2ZocmkI4qeohBCNmLY0nup5zM/LPfkR8ZUtabdimav/1NobFE2+CHfw+m9GvsF5tmeA8Dics0juUSpsWl2VL2G4h6c1/XfhpSCkebkquplDUUrhwOWcwQNOd2KR6qzu9jn/IMGzHkXk3i7RPmzVY1VPbNchNuiOgX0PJQLG3/YjE3eq33Ez9QE1uh0wjCLZPeC08gz+5ARC63kb4S5XwLHDn32faY73bHcmedsK6InTRU4l+TxgxuIAybhhbmSpfaygotc4vIDQPlC+ygPB1wMIzQF8rt9oxU1fhfR4Z0q4oj/c1DOC15XbuRwyjEaaR6P6Sdw0XOkkw2nt2pktpCN6373gZgRPbK9P+gfrUoeQw3D3SnRA78q+41vws+z/p1M+isgWS2FYoMLL6tpanBEPz7JoAvM6pUuh8qToZk9BkL3AJ6Pygg56Pa1c6bKVceR6qlKvOdrNV7BGligJkFIItMYLCvZzrlASATTVDbJr/VEMcdNGjLNCrxt1bP872lFvO5Nk83u6lkDjYl5ubeeP+p4SNOyBZYQ8cRAhaVlVimslN4wEhjgpL9ObDPwncaEam6lDB8qvbJv4PgepKK4OZAvkqEseyxtYH5UTd1T6tCfRIO9MCfiSg7l7B4ZxFUMC/TPp8X8ZswYEm4gpF48NvQ0LhnQkvUr5nxs7QB6vv4/FBSuxmH+g4TrcxshrfYvNpQDaOe4jnP2MZWtd9PKTMjhfJtD/JNIIiJcYVG+VLERBlT2UCP+uB6/dFMp5TIdrYZcKFTFhSwmDVTYwvunsGwOZ9faJLiQf8Ui3Y/DN+O31YPJVAp9w1g48BNj97AFbpAGdrExXD7uTBKvfSWn7M1JtgKIg57yyL40LXjLUKSfmbjwvxbvM3xp9cnbRaO0Tr66QYM/hRNZCbY1MsLh0MWw7N7N8f9T1TrWQJ7lNCO7eb0F7NYsPpr5rTpC/VG12qRcjcfdZdUVvvQMGBAv7WVos3HrXGvO1pIsZPr1RNMEERwaGAklyhnYwTW7Jba/TXiH/Iu5At8JVSIf5iXECvEdBM7oNGUzDHmOnXkSi1ZJ12sbU60sHgBH9/Env9XvdNKLg= X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: d9aea868-46ce-4f34-aa41-08da53a11c98 X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB6030.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 16:14:26.2984 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zgfEpjLpoRsT3z2JrEuDvhvDw/Mc99w9YGa6MgMZasZrf6AnIRQ5zBzPQUaOdiSvj09yABhAFj8tVUv1pJ6UrsWOVpWXcm8P5I3J7y6tbAE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2865 X-Proofpoint-GUID: 4ZohRlxWsdiseYoTezOFkh5q-jFrLrw5 X-Proofpoint-ORIG-GUID: 4ZohRlxWsdiseYoTezOFkh5q-jFrLrw5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-21_08,2022-06-21_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 priorityscore=1501 mlxlogscore=625 clxscore=1015 mlxscore=0 suspectscore=0 spamscore=0 phishscore=0 bulkscore=0 malwarescore=0 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206210067 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 16:14:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/167164 The Linux pressure monitoring system helps determine when system resources are being overutilized by measuring how contended the CPU, IO and memory are. This information can be found under /proc/pressure/ which contains 3 files - cpu, memory and io. In each of the files, the format is as follows: some avg10=70.24 avg60=68.52 avg300=69.91 total=3559632828 full avg10=57.59 avg60=58.06 avg300=60.38 total=3300487258 The "some" state of a given resource represents when one or more tasks are delayed on that resource whereas the "full" state represents when all the tasks are delayed. Currently, we only collect data from the "some" state but the "full" data can simply be appended to the log files if neccessary. The "avg10", "avg60" and "avg300" fields represent the average percentage of time runnable tasks were delayed in the last 10, 60 or 300 seconds respectively. The "total" field represents the total time, in microseconds, that some runnable task was delayed on a resource. More information can be found at: https://www.kernel.org/doc/html/latest/accounting/psi.html and in the source code under kernel/sched/psi.c This commit adds functionality to collect and log the "some" CPU, memory and IO pressure. The "avg10", "avg60" and "avg300" fields are logged without change. In place of the "total" field, the difference between the current "total" and the previous sample's "total" is logged, allowing the measurement of pressure in between each polling interval, as was done for /proc/stat data. The log files are stored in: /tmp/buildstats//reduced_proc_pressure/{cpu,io,memory}.log mirroring the directory structure of /proc/pressure. If the /proc/pressure directory does not exist or the resource files can't be read/opened, the reduced_proc_pressure directory is not created. Signed-off-by: Aryaman Gupta Signed-off-by: Randy MacLeod --- meta/lib/buildstats.py | 57 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 10 deletions(-) diff --git a/meta/lib/buildstats.py b/meta/lib/buildstats.py index c52b6c3b72..64ad3ef40e 100644 --- a/meta/lib/buildstats.py +++ b/meta/lib/buildstats.py @@ -14,13 +14,27 @@ class SystemStats: bn = d.getVar('BUILDNAME') bsdir = os.path.join(d.getVar('BUILDSTATS_BASE'), bn) bb.utils.mkdirhier(bsdir) + file_handlers = [('diskstats', self._reduce_diskstats), + ('meminfo', self._reduce_meminfo), + ('stat', self._reduce_stat)] + + # Some hosts like openSUSE have readable /proc/pressure files + # but throw errors when these files are opened. Catch these error + # and ensure that the reduce_proc_pressure directory is not created. + if os.path.exists("/proc/pressure"): + try: + source = open('/proc/pressure/cpu', 'rb') + source.read() + pressuredir = os.path.join(bsdir, 'reduced_proc_pressure') + bb.utils.mkdirhier(pressuredir) + file_handlers.extend([('pressure/cpu', self._reduce_pressure), + ('pressure/io', self._reduce_pressure), + ('pressure/memory', self._reduce_pressure)]) + except Exception: + pass self.proc_files = [] - for filename, handler in ( - ('diskstats', self._reduce_diskstats), - ('meminfo', self._reduce_meminfo), - ('stat', self._reduce_stat), - ): + for filename, handler in (file_handlers): # The corresponding /proc files might not exist on the host. # For example, /proc/diskstats is not available in virtualized # environments like Linux-VServer. Silently skip collecting @@ -48,13 +62,15 @@ class SystemStats: self.diskstats_ltime = None self.diskstats_data = None self.stat_ltimes = None + # Last time we sampled /proc/pressure. All resources stored in a single dict with the key as filename + self.last_pressure = {"pressure/cpu": None, "pressure/io": None, "pressure/memory": None} def close(self): self.monitor_disk.close() for _, output, _ in self.proc_files: output.close() - def _reduce_meminfo(self, time, data): + def _reduce_meminfo(self, time, data, filename): """ Extracts 'MemTotal', 'MemFree', 'Buffers', 'Cached', 'SwapTotal', 'SwapFree' and writes their values into a single line, in that order. @@ -75,7 +91,7 @@ class SystemStats: disk = linetokens[2] return self.diskstats_regex.match(disk) - def _reduce_diskstats(self, time, data): + def _reduce_diskstats(self, time, data, filename): relevant_tokens = filter(self._diskstats_is_relevant_line, map(lambda x: x.split(), data.split(b'\n'))) diskdata = [0] * 3 reduced = None @@ -104,10 +120,10 @@ class SystemStats: return reduced - def _reduce_nop(self, time, data): + def _reduce_nop(self, time, data, filename): return (time, data) - def _reduce_stat(self, time, data): + def _reduce_stat(self, time, data, filename): if not data: return None # CPU times {user, nice, system, idle, io_wait, irq, softirq} from first line @@ -126,6 +142,27 @@ class SystemStats: self.stat_ltimes = times return reduced + def _reduce_pressure(self, time, data, filename): + """ + Return reduced pressure: {avg10, avg60, avg300} and delta total compared to the previous sample + for the cpu, io and memory resources. A common function is used for all 3 resources since the + format of the /proc/pressure file is the same in each case. + """ + if not data: + return None + tokens = data.split(b'\n', 1)[0].split() + avg10 = float(tokens[1].split(b'=')[1]) + avg60 = float(tokens[2].split(b'=')[1]) + avg300 = float(tokens[3].split(b'=')[1]) + total = int(tokens[4].split(b'=')[1]) + + reduced = None + if self.last_pressure[filename]: + delta = total - self.last_pressure[filename] + reduced = (time, (avg10, avg60, avg300, delta)) + self.last_pressure[filename] = total + return reduced + def sample(self, event, force): now = time.time() if (now - self.last_proc > self.min_seconds) or force: @@ -133,7 +170,7 @@ class SystemStats: with open(os.path.join('/proc', filename), 'rb') as input: data = input.read() if handler: - reduced = handler(now, data) + reduced = handler(now, data, filename) else: reduced = (now, data) if reduced: From patchwork Tue Jun 21 16:14:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aryaman Gupta X-Patchwork-Id: 9419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FAC1CCA481 for ; Tue, 21 Jun 2022 16:14:35 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web08.45564.1655828072089759434 for ; Tue, 21 Jun 2022 09:14:32 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriver.com header.s=pps06212021 header.b=BCzEDrbD; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=3171c08643=aryaman.gupta@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25LDTFra004368 for ; Tue, 21 Jun 2022 09:14:31 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=PPS06212021; bh=6uQAcDNP1cNXHStFkD39KncmypH7YPv13wFYpbauEok=; b=BCzEDrbD5386YNeJNCiNMECVaIbvlrPJCClx/SGgLs1z9+PBQmAwZhxfhdsQ9ot0IUnz /RtL3aGO9pzLZF1UGlp8FzTnV2FuEoPh5m5hA/uJ2Wqh5W89M0XVI8pV3+ygL1qW9Tn1 hHqjm3lyLCfcnydCB/jR5FoKKsFNayclkvIRjQIJIfkaEaqCG+iv1z4WOgjsccnsRBRL yX0alXLREwOSO4DRTizIcdIqtAH8Gp+De7sAcuCVu1HWM2NwJfR7mWlXtvP5enIbdOmD PnKwPo5e0yK8CHfoZbPSPHVKG3UK8S0M0DmYNU7StkKs/iqR17hsYzomDffLCWAaj2U3 SQ== Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3gsa06a7ud-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 21 Jun 2022 09:14:31 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AdOoNs4bpOAjHjek/uSlMBdNCLrtYOJKFjUSxwjxEe0WW4isfr/3KYlubmkl59PvfAssevXcjxqN4e9UgdPshUuXG7u3ZT5FOh2iwQqpIHHFEW868rrD6CyrZov1lfX55AqeCIEbDPWg4MFocvAgkH7BFF1zXFlK5Y5m6zlXBzodBjAuCoET/WRwjYZOIg/MpEIF/IWCCMt5f/0HDaDELkqGNsiO8vfsfnUa3mTjfrNSoVYWdLlIwLhDv/G0YNA9jP61jcXIOSzO58UBmegBCmKOGwyySXG9t5uMT2UGD4SVEeRF3JIXJKHCBXrRBj2ahCDDs0Wx24Ao4Md8O+6JwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6uQAcDNP1cNXHStFkD39KncmypH7YPv13wFYpbauEok=; b=f3W5oo/ITJlzfwMgpIxLqoS2E4GcPTCp/vO+hrWqyMiDXsvMMVMsXeWXxYy7yljUPEip1BflHgglXjS8qcryY3f3yPltAE99hR6onF2A9MPLtHcinEsXM9h32e8UetxLGwomRqgd76U3okO5pyHoOF5X8BN5Q0lGA9L2+H2gmC4CYDisOxyNk3PzzwOZ1C4z8unqzVceRTTyRRMGhhCRaw2qQ3KEWTQvJxU9j2c134BHj4GkRLEt2Zt6uM0lxhCB23QeqzRPolcog7syC7oUePbD9UjHiWKBpT6QYzntr1k19Jr0UH5sBMrovSvJasRFVk95mAJvQ1pWnQQYs9p09A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from PH7PR11MB6030.namprd11.prod.outlook.com (2603:10b6:510:1d1::19) by BN7PR11MB2865.namprd11.prod.outlook.com (2603:10b6:406:ac::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.16; Tue, 21 Jun 2022 16:14:27 +0000 Received: from PH7PR11MB6030.namprd11.prod.outlook.com ([fe80::14a3:342b:5f14:eda6]) by PH7PR11MB6030.namprd11.prod.outlook.com ([fe80::14a3:342b:5f14:eda6%7]) with mapi id 15.20.5353.018; Tue, 21 Jun 2022 16:14:27 +0000 From: Aryaman Gupta To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 2/3] pybootchartgui: render cpu and io pressure Date: Tue, 21 Jun 2022 12:14:12 -0400 Message-Id: <20220621161413.2779759-2-aryaman.gupta@windriver.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220621161413.2779759-1-aryaman.gupta@windriver.com> References: <20220621161413.2779759-1-aryaman.gupta@windriver.com> X-ClientProxiedBy: YQBPR0101CA0328.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:6c::20) To PH7PR11MB6030.namprd11.prod.outlook.com (2603:10b6:510:1d1::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8dd2024a-8555-45bf-8c93-08da53a11d01 X-MS-TrafficTypeDiagnostic: BN7PR11MB2865:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: suD6zmINHa07OLIaxJvP3Ejguqe4sEbvYQyUuEV6focIH4JE5lFifsNIOfRkEdFLQ4kzwn6yFVUlh0C+bKil7msXfcAgSdHUKF1t2xOEQZ9id3IaNplwoB13TEUJwgVnhDQ7mUiQlvYEXnmvrutyxt4ywqIKz8kdo7Ackz6fWvwJ9CrVlRYxXFtpBSibqVJH8Nq5L0LxPgBaP1ZrJ1KvM1tK/p6iXs7pIYta1fm+yIKy7LFfsko4dzhV2vqDNC65N5KV6X0s7Elro+X1Ik9MHoyaXvEbuNrJnl4aC5INv4KNLZ8bjnN6m3e/N++gBxqF3f5P9IEr8N4NmAQR4M0klpB+1//XTIgNvN7dUyUKcmMi06O8hJdQGglMyQjY5Z2p8pqm9ePlViFp/0SK80hC/Q/ZcWJ/01nd3jwz8+Iflmxjgx5BCFYkn5nDZh2miRQP5SgbZvEdCuUH2NxURifQ19cUHS6b79XY7j+CJ2yk4raVCt3x9uJHMhZ+slaxFJ5tnOL6wOU7mVCSpG8MGcpTo68zYaz4IvCezQKRpkjNfyr1XghzhVk/2ecScp5xOB1+bDgeXbch8UHNlxwqRhzLsRMa0LUlHrFi+lBh1QKNyZ7FmeErmV39siQveAqHuaAzdfjBltmHA4+f+eOqZuOsaw+0tlYMHh58pUiUT1PBUa8bZQgYPa5XpUgcVWXMqB/1PfIP9n251/9BFNcZwfJTFQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH7PR11MB6030.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(376002)(346002)(366004)(396003)(136003)(39850400004)(52116002)(6506007)(1076003)(186003)(66556008)(41300700001)(44832011)(6666004)(2616005)(66476007)(26005)(83380400001)(38350700002)(6512007)(38100700002)(6916009)(36756003)(5660300002)(86362001)(2906002)(66946007)(316002)(8936002)(8676002)(6486002)(478600001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PYKrHme3JB/OVKWMXclgGKNLnbSAryK3eH0zzm0LWJjmOmioTGFvbxrAzUJanYu8ehAzifdWtaTsaZQ13L6V9lddMYY9xVCC7iGUimmN7b9NYS/ylSnqBbYReMVry3QZQopIy1mpkl5hD4bhVolP0U89Ug6UPtRZeSL6o6Z+pWJj5waX2jWfkzgHr/chAltF3ShSCU8PgcP+Vw2OkBsB49zFgeFKkZBiirZY9ydjXFtVTZjKEKBpVUiHeXSJjlPMqb3ahTJRy+Ewcn9Ql30uBU2dBg7L4j3MkiUmWAmZKMfrGxj5JDeKLQT0K1pOBcZVi6sZboRL2WD2yYrYu47RgI/KkaYc0/MHVtsrQYfbC2+/dXCECuPwLYIB+SFQ6NCLQfx/HMXwLtO3DTN2jitsrpTFoRNXDbm0YSTaiuTONBqQywhyFN9Uege+GeC1k9oODd3slCCRVV2noW5O1t+p/t4i8KkuN4qaGr1gojjv+Qq1ecicmn/eqp8KdKuYWHf4vGab2nZzp+7hI4hL/yUjImr+2IzYts9EdRhGtoHtrD+kqVGhth2dfTLNdEk1kXhPFU02HCV5ibhsNNmDYRlA3iIsT7/9i8GdjBP0HBYrhIQaXCPnGpYAbCQwTJmo/VNTvS7cfsDUM+KizAKLNceHwzZ+64Yjz9ieF/Nryu5z7ZXwM8IAxkfjkqOqoYMbqmyT7Pftuyd5bMnHCijW3jwzXc95wkHl+MoVwzoLtFYyGx+7FrozC8teJUAfIsbCmgJxo23GU+vyZMFzWleJJxNC1mwbnSrIFH+Oq6t/xI7vZdi0KiSwhFO5QPCI5PliHqNURsW08JMXIfetjcNwJd9rJC09QPmw2rIVus7l2YXSTJniwOoy8CRtIpsvbuIUlU2qw3U24+M1Ns3jOD8T4iMiArNYhiGzbj6Ehpn/acaopf12tsUpro/r/hRndLOo+grtw4MsP0K3/KPKweC443e5N7HXZdBc4fRmYr+Sdk7EAw1Bup2W6YbCdYtUFHUY+N73g5E5Xizob2s6deLuZtOZW4UFyttXfs1gcAlNiM3drBn2p2IouJZ10R3AtHHh4Rj26vd1fIPE0hl5tsyojB+Q38Zrz9/zos8IEGSp1Je9hvAbuxlAvYJKjBAPewopr310jNckGOIOwk3KRoGVFziPkwcVZQghH1MUlEBbrZ6sxtwFJrRaMoZd0iHG0npqYdGDg7kxb3OAoMzMDDJTMxFdEjJwzXj2PAMgpjX+BindDNAGiVkF8Eso3LZpWSWkDRg9SoIdPqGsA1sEODK0NZpGq95THy+j2lX0CvAdjsn+jZvGs7gBvhPCxp0k0EEHoTuDiMVTakrEPgE6ihhXaNlSaFeldTEHE49Gq9/Q8hKmtWMJVfnna5li7i8Ybx2IA60JrJOFLx0pLfO3cpPMvTomOE608M9HzWvUZX3P2fLzKd7PtaVtMxobyRlv6lRKPWGTh7iOOYpRdMd5YZvR6y54Fp94yvEWmGz36Bl+shFCIjPtAETVSo8kkIQWehm/fpaxQWTAezG/7lmd/N5IpYS1RriZbrWgPeFDBf157WeXOhwqo8Y8L5M1Y1cwir0B+fSagWHT4eDgyKiNdKD9YDP/RN99Vq3X52HuUsiIlkNSAeQD63YYbrYVWdXLrMr/xxAUjeuokRLQqo+JwVfukUk7sbFjjYLJF4RkK5RMUP4MWRv5nXF8QfRMZXxH1Iv6yuYdF3iFf/MUAykwh7EyKXn9czY0kd9o0ZC1tGja31a8wDw= X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8dd2024a-8555-45bf-8c93-08da53a11d01 X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB6030.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 16:14:26.9545 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pZwPFrElvi3UK62q/00HY36Yl+GJeawpnW4J8hlyqT61vhLRSsLILVolrjYVjHYqSD9K4VDxXqztiZb06adhgklD5iQ+dARwnc+YoWABcCU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2865 X-Proofpoint-GUID: 9sxaCVYKUBO7dVmmwonYrILJEp-eFj4G X-Proofpoint-ORIG-GUID: 9sxaCVYKUBO7dVmmwonYrILJEp-eFj4G X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-21_08,2022-06-21_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 priorityscore=1501 mlxlogscore=799 clxscore=1015 mlxscore=0 suspectscore=0 spamscore=0 phishscore=0 bulkscore=0 malwarescore=0 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206210067 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 16:14:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/167163 Add two new, separate charts showing the avg10 and delta total pressure over time for the CPU and IO resources. The height of the avg10 data in each chart represents the percentage of time "some" task was delayed over the specific resource during the last 10 seconds of the build. The height of the delta total data in each chart represents the total time "some" task was delayed since the last sample was collected. If the reduced_proc_pressure data is not present in the buildstats log, then the new charts are not shown at all rather than being present but unpopulated. Note that the delta total graphs may appear "spikey", oscillating from high values to low. This behaviour is fixed in a subsequent commit. Signed-off-by: Aryaman Gupta Signed-off-by: Randy MacLeod --- scripts/pybootchartgui/pybootchartgui/draw.py | 77 +++++++++++++++++++ .../pybootchartgui/pybootchartgui/parsing.py | 28 +++++++ .../pybootchartgui/pybootchartgui/samples.py | 17 ++++ 3 files changed, 122 insertions(+) diff --git a/scripts/pybootchartgui/pybootchartgui/draw.py b/scripts/pybootchartgui/pybootchartgui/draw.py index fc708b55c3..a13df3a3fa 100644 --- a/scripts/pybootchartgui/pybootchartgui/draw.py +++ b/scripts/pybootchartgui/pybootchartgui/draw.py @@ -80,6 +80,18 @@ MEM_BUFFERS_COLOR = (0.4, 0.4, 0.4, 0.3) # Swap color MEM_SWAP_COLOR = DISK_TPUT_COLOR +# avg10 CPU pressure color +CPU_PRESSURE_AVG10_COLOR = (0.0, 0.0, 0.0, 1.0) +# delta total CPU pressure color +CPU_PRESSURE_TOTAL_COLOR = CPU_COLOR +# avg10 IO pressure color +IO_PRESSURE_AVG10_COLOR = (0.0, 0.0, 0.0, 1.0) +# delta total IO pressure color +IO_PRESSURE_TOTAL_COLOR = IO_COLOR + + + + # Process border color. PROC_BORDER_COLOR = (0.71, 0.71, 0.71, 1.0) # Waiting process color. @@ -415,6 +427,71 @@ def render_charts(ctx, options, clip, trace, curr_y, w, h, sec_w): curr_y = curr_y + 30 + bar_h + # render CPU pressure chart + if trace.cpu_pressure: + draw_legend_line(ctx, "avg10 CPU Pressure", CPU_PRESSURE_AVG10_COLOR, off_x, curr_y+20, leg_s) + draw_legend_box(ctx, "delta total CPU Pressure", CPU_PRESSURE_TOTAL_COLOR, off_x + 140, curr_y+20, leg_s) + + # render delta total cpu + chart_rect = (off_x, curr_y+30, w, bar_h) + if clip_visible (clip, chart_rect): + draw_box_ticks (ctx, chart_rect, sec_w) + draw_annotations (ctx, proc_tree, trace.times, chart_rect) + draw_chart (ctx, CPU_PRESSURE_TOTAL_COLOR, True, chart_rect, \ + [(sample.time, sample.deltaTotal) for sample in trace.cpu_pressure], \ + proc_tree, None) + + # render avg10 cpu + max_sample = max (trace.cpu_pressure, key = lambda s: s.avg10) + if clip_visible (clip, chart_rect): + draw_chart (ctx, CPU_PRESSURE_AVG10_COLOR, False, chart_rect, \ + [(sample.time, sample.avg10) for sample in trace.cpu_pressure], \ + proc_tree, None) + + pos_x = off_x + ((max_sample.time - proc_tree.start_time) * w / proc_tree.duration) + + shift_x, shift_y = -20, 20 + if (pos_x < off_x + 245): + shift_x, shift_y = 5, 40 + + + label = "%d%%" % (max_sample.avg10) + draw_text (ctx, label, CPU_PRESSURE_AVG10_COLOR, pos_x + shift_x, curr_y + shift_y) + + curr_y = curr_y + 30 + bar_h + + # render delta total io + if trace.io_pressure: + draw_legend_line(ctx, "avg10 I/O Pressure", IO_PRESSURE_AVG10_COLOR, off_x, curr_y+20, leg_s) + draw_legend_box(ctx, "delta total I/O Pressure", IO_PRESSURE_TOTAL_COLOR, off_x + 140, curr_y+20, leg_s) + + # render avg10 io + chart_rect = (off_x, curr_y+30, w, bar_h) + if clip_visible (clip, chart_rect): + draw_box_ticks (ctx, chart_rect, sec_w) + draw_annotations (ctx, proc_tree, trace.times, chart_rect) + draw_chart (ctx, IO_PRESSURE_TOTAL_COLOR, True, chart_rect, \ + [(sample.time, sample.deltaTotal) for sample in trace.io_pressure], \ + proc_tree, None) + + # render io pressure + max_sample = max (trace.io_pressure, key = lambda s: s.avg10) + if clip_visible (clip, chart_rect): + draw_chart (ctx, IO_PRESSURE_AVG10_COLOR, False, chart_rect, \ + [(sample.time, sample.avg10) for sample in trace.io_pressure], \ + proc_tree, None) + + pos_x = off_x + ((max_sample.time - proc_tree.start_time) * w / proc_tree.duration) + + shift_x, shift_y = -20, 20 + if (pos_x < off_x + 245): + shift_x, shift_y = 5, 40 + + label = "%d%%" % (max_sample.avg10) + draw_text (ctx, label, IO_PRESSURE_AVG10_COLOR, pos_x + shift_x, curr_y + shift_y) + + curr_y = curr_y + 30 + bar_h + # render disk space usage # # Draws the amount of disk space used on each volume relative to the diff --git a/scripts/pybootchartgui/pybootchartgui/parsing.py b/scripts/pybootchartgui/pybootchartgui/parsing.py index b42dac6b88..004d6fb953 100644 --- a/scripts/pybootchartgui/pybootchartgui/parsing.py +++ b/scripts/pybootchartgui/pybootchartgui/parsing.py @@ -49,6 +49,8 @@ class Trace: self.parent_map = None self.mem_stats = [] self.monitor_disk = None + self.cpu_pressure = [] + self.io_pressure = [] self.times = [] # Always empty, but expected by draw.py when drawing system charts. if len(paths): @@ -554,6 +556,27 @@ def _parse_monitor_disk_log(file): return disk_stats +def _parse_pressure_logs(file, filename): + """ + Parse file for "some" pressure with 'avg10', 'avg60' 'avg300' and delta total values + (in that order) directly stored on one line for both CPU and IO, based on filename. + """ + pressure_stats = [] + if filename == "cpu.log": + SamplingClass = CPUPressureSample + else: + SamplingClass = IOPressureSample + for time, lines in _parse_timed_blocks(file): + for line in lines: + if not line: continue + tokens = line.split() + avg10 = float(tokens[0]) + avg60 = float(tokens[1]) + avg300 = float(tokens[2]) + delta = float(tokens[3]) + pressure_stats.append(SamplingClass(time, avg10, avg60, avg300, delta)) + + return pressure_stats # if we boot the kernel with: initcall_debug printk.time=1 we can # get all manner of interesting data from the dmesg output @@ -741,6 +764,11 @@ def _do_parse(writer, state, filename, file): state.cmdline = _parse_cmdline_log(writer, file) elif name == "monitor_disk.log": state.monitor_disk = _parse_monitor_disk_log(file) + #pressure logs are in a subdirectory + elif name == "cpu.log": + state.cpu_pressure = _parse_pressure_logs(file, name) + elif name == "io.log": + state.io_pressure = _parse_pressure_logs(file, name) elif not filename.endswith('.log'): _parse_bitbake_buildstats(writer, state, filename, file) t2 = time.process_time() diff --git a/scripts/pybootchartgui/pybootchartgui/samples.py b/scripts/pybootchartgui/pybootchartgui/samples.py index 9fc309b3ab..472dc27be0 100644 --- a/scripts/pybootchartgui/pybootchartgui/samples.py +++ b/scripts/pybootchartgui/pybootchartgui/samples.py @@ -37,6 +37,23 @@ class CPUSample: return str(self.time) + "\t" + str(self.user) + "\t" + \ str(self.sys) + "\t" + str(self.io) + "\t" + str (self.swap) +class CPUPressureSample: + def __init__(self, time, avg10, avg60, avg300, deltaTotal): + self.time = time + self.avg10 = avg10 + self.avg60 = avg60 + self.avg300 = avg300 + self.deltaTotal = deltaTotal + +class IOPressureSample: + def __init__(self, time, avg10, avg60, avg300, deltaTotal): + self.time = time + self.avg10 = avg10 + self.avg60 = avg60 + self.avg300 = avg300 + self.deltaTotal = deltaTotal + + class MemSample: used_values = ('MemTotal', 'MemFree', 'Buffers', 'Cached', 'SwapTotal', 'SwapFree',) From patchwork Tue Jun 21 16:14:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aryaman Gupta X-Patchwork-Id: 9418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DB6FC433EF for ; Tue, 21 Jun 2022 16:14:35 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web09.45574.1655828071093206205 for ; Tue, 21 Jun 2022 09:14:31 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriver.com header.s=pps06212021 header.b=fnb2R3ok; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=3171c08643=aryaman.gupta@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25LDTFrZ004368 for ; Tue, 21 Jun 2022 09:14:30 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=PPS06212021; bh=vUTks1R71j2uDaemX2blApmAID8TEPC6TY0wqGWIgTY=; b=fnb2R3okWTSLqm4A66mm0wsUQzKJe3R7FVWTTxms011UXP2KOWcTN6dE2ukLXcOYiAM0 lxdxvwE73FV1aHzjk3TzSWEmup426NbYEa/o1UTHFCnKu5r1ZMOruswgGwp9BLMFtfcE QDcWgE7labX/ryCHcixIRTsVsNVSlYVzz/xQBATjkppUsJ2SEVZo1zh0rteg7sXqsXcR 1cSlVeKqfqSVzWf8Cq/pWiCwQRbMRzGNAD/uG0OErFlYjZywrII4wqQooU6xtB9L4rVe vynSmy3ZfvRZAJvnVVeuTaLq6hT2NTiU2HBngIYZSMzBO380HsgN+S1fPmdzxygm/MCN uQ== Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3gsa06a7ud-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 21 Jun 2022 09:14:30 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Es3hBYWLqERQCfdVPUyZ+tqvqavX3tg7+dnU/l1MLdXjsMqvBg2kEVgN/QTyO94APigmdFkU0VCrUO7/tWOEEhO36hfuMlkd2ZuxpVV5r/wKos+6EPrdxijmnBsXNFZ+JGNIyGtx7l7emn5Hlf2YqK+8NKW9MtGmTlOBK9zmLWisYAuZ4ySkwi4qerqe9x4TY5/FsweW0IdXSNaefZkoDyT2dS4wk/VisvIXTFU7NxKeIV0mgHZTjKu2UxXZE1NCihizSEfszRS4BFMH2imHCbtkbCa3dOL2NT9PTlQ1w+B6qlkDxK7bAFfBk3KBOdMuLU87uJXNTc+8LfY2yFRNWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vUTks1R71j2uDaemX2blApmAID8TEPC6TY0wqGWIgTY=; b=gw0aAYmUqbr6x/uJImuIuzKt8ggTBhc/AJdslX2vXpR7u+dvTKaxMOR7R9RitvMq9YMHMKSvdyTjjXkKFpvYZ1NMU4T66JQRd1vCb2CZP+VO20zLtAzD7v/nvBgn71q4qUtNmNXDmGX542LV3TCykZJ67aNx3ECzF/LatECxFFlBUwiu9kULzt6BmW3vYP0/ogTZVm3Bz/P1RV8L2ZdcEI3epbk7hIO7PmYhgsx9w9WKe60KDwzW8FSV+RRiq67Jcw+3/r/EUt5/H5RA1GMpzvjx2s5WLcQO9lPCbmDfKOQLKR8bsYljb+qy42aPd4IkKjepBsqjXQ6qxPg1H7T1KQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from PH7PR11MB6030.namprd11.prod.outlook.com (2603:10b6:510:1d1::19) by BN7PR11MB2865.namprd11.prod.outlook.com (2603:10b6:406:ac::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.16; Tue, 21 Jun 2022 16:14:28 +0000 Received: from PH7PR11MB6030.namprd11.prod.outlook.com ([fe80::14a3:342b:5f14:eda6]) by PH7PR11MB6030.namprd11.prod.outlook.com ([fe80::14a3:342b:5f14:eda6%7]) with mapi id 15.20.5353.018; Tue, 21 Jun 2022 16:14:28 +0000 From: Aryaman Gupta To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 3/3] buildstats.bbclass: correct sampling of system stats Date: Tue, 21 Jun 2022 12:14:13 -0400 Message-Id: <20220621161413.2779759-3-aryaman.gupta@windriver.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220621161413.2779759-1-aryaman.gupta@windriver.com> References: <20220621161413.2779759-1-aryaman.gupta@windriver.com> X-ClientProxiedBy: YQBPR0101CA0328.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:6c::20) To PH7PR11MB6030.namprd11.prod.outlook.com (2603:10b6:510:1d1::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4e1b4fa5-14b0-4881-aa64-08da53a11d65 X-MS-TrafficTypeDiagnostic: BN7PR11MB2865:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hB1jbmZDvvZ7lFfItAyMa7e+fEzVsaGZtEoohI/gQ3l37GNW6Ygme82X1PzM0/CCDaXDXavYeSCEYVljk4DfgTW47qPVUWJXzow7UjYihpKrvnJHVhI8WMBMSgzBwFoNek4Xfh/7x5zvdxjAR4hehvOPNeXaU5DOpQzZW0Bd3/n+fXFOtWaFNTLpbuMGuf09kBBDobEMnbrdk/LZK1GwBY6V0phn82q5oeoOdnXBgCXqW3qBgn4KrHx+e0azK6vHsO6ukkUK5fj2wOt7x3laL4K+QVLzS51YaaybSi/Pc+9M0QNnycr6PPO8KnGkbjJajxq2KpdB+04kSNyg4nY83jGVjFjWkMuA+Aq0cy4x2KVOVL1E/ESevopCS6lc+Hg8bb2KIhrFklfXZk57SyuvWqZEVmykNjX+kuAHWvpI+FKZfiSVJ4LeuSAuIDZtKZIHgrg0+tj/lJLhYWT1YuRsjXQAvmvn4i7Q7Qg2n19aGTzNjYSIerHDBqPm2wjQQEv2jYqHnKoOUZotj5y7jr781VpOIi35RcxsWKdncan3zmiurnRae3VYvPIMKV2+Dy6RnGr+MKZKZMsB1as1dCss1xOjZDFdeDXkkohqEsdDWwe/aRaFIuyuirTrbEHVWON/jQWOq+NXW+uqPLggwK333lmj+4+RrLgHacncImAfIkR4jvvEODcM6KMNSe5LC5y3Y8hjf5qmrg7GsGafUCZARA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH7PR11MB6030.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(376002)(346002)(366004)(396003)(136003)(39850400004)(52116002)(6506007)(1076003)(186003)(66556008)(41300700001)(44832011)(6666004)(2616005)(66476007)(26005)(83380400001)(38350700002)(6512007)(38100700002)(6916009)(36756003)(5660300002)(86362001)(2906002)(66946007)(316002)(8936002)(8676002)(6486002)(478600001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Sn1k2D4MdrFv+lTtpwAMPMwH6AOdI6VsT/GuwlGLH3g++ikRYs2toZWJ21AA4VFcTTDE7d703P/GVmWcdVJRpqcdMGUDkNaMYzX2/6CGQN8/FIFQ3pjinjFo1BWCM2bhVO8pCUh62k5IbO5/p70kxlvO6a7vNXtX7hDS6BFkk5LFoBCp+7Ik2L2rmPDdi7od1nV8EoS74KfZpH732P3GHmqKQdneZfSyaij/WWeGuP/nrMcry3bTJ19fQqDGD7eoTB7aUNtDucN17JjrMkNjaVaDcmcacAP0PQlwBQzCT0gFVtluxLwgg2+H3k6cPXgvT8p1CEnh+xfWK6vg+WU7Iye4VFa363TXTBvFLLHCM8aoq1FYoqOFLXjkk6ZunrWf9Xdo1b1TsdFsGQBLpMKcit/rh+gBR8fkFgBSMwCB+zJo/JO8DeDQr1BMXpEPV1a4VUXFShYIZQu1NK9b7zCtq+EnwAbYflVy+9ThluJ+VLogGx7Tz5rG3FvbJXLZRa2LqTFvpAP3JXl8dLJ7gD4L3eHvpIdJVw6+qfn37b3yy3W95EI1+/RbHT+hx2PasnINWmnjUj4q1UC2tgdUQaUB+0Wd7eIsi9KxIuiqWWiN4nO8N0qTEBfCRulznaQgift6+hUX6m2Xro/CTW4WkumsWi2je9cX/4l3WisdyrkM2nmFJ8IkknO8vgD+PuanHAaGrUOE/1IGf70gMd7XzlkgHaj3dWcC07eZqc3D3IczYa6kaMF6Q/ywRlwPZMgkA3ZUlL8sgw9kZqtHCRkL3TGtWPd2wwLJ2w1vMq40PzNbA7nb1YaHep3j3U/HfHBB/69EZQxE8mxThbwB6riUAaECY2kseqIqYWIBZi6wl9krNnNHszSJPMiN3FZb1kYk7DS9Syem9n3+8vSwvwjtVZAUQZwzVm3nm+08IL/a3Rr3E1k1Jy7YItU6nfpPU7GWWYXWbjw5jMmCTJbvIJmOGjOgTxDyX7kgrI7OBv73rRaXek2gtSfdA2RRdvH4BS0gQ6ziOMQIPhhC392shVa8FvE1UvfDVp6JfAaL5iBDxwF35DSPdmQsmQIpLUskkQzJC3OtuUhTCG44uxjaj1c7Olt7x8+BoBQSNjgZgjWm2H5wv4bHPtPhqMnXYOzutJBquaqwZ6iRuNIsQNUOiBVtGrqr0c0gpQgzN/qujuYTmpV7v9GATksdvUsOAWyrM1IWuHoJenZUZaKS0jG65PKlAzY5hZfm2uE6YU+h3BmEVhopL1ehO0KwBZ1UYxzR7m6rb9M6yxWxGiPy//VJL1tNzK9JOf7zyMPNofNsF2Kdolb0YNEKgupb8+uhHSU1O1peVr8pole9bPM1X0j+RkAfBeaYrT/TK+Rx6bJZ310SHwq6qSety0QQr2NaxwB7nSqO/CZdUvsKqhjUnVGSNBYm7xUxqoQuQf6rYMNOuzs6eB6PlyDVhSE1fLKeFUloHXNekQzFrcPUQOIHDOLKgVTqLW0a0XY4BUxBwhZeoLSTd2/RZ+gDHdTsSHc7Ihnl3NHcYbeKqFFRgLGObp69spfrleb/EUSzGTUf9LzLiflcSKBKt6F0XY+aAhahajZXh51PyQDbQP5+LwlFY9tENIB9L4x/jsa5XMCx04R7mjNz9qjAt/jgP24BRFmo1kOUc4MkexQokbc7Y7Ne8r5+jRBd9RnlQElpu2k0F6HuldF+u43hrDQ1g0wFKyKJ5Y8BVwZnllLUo9AB6pwPYGagMoJ5XbEU+ZAikSwqtlLVK0hojpzTYk0= X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e1b4fa5-14b0-4881-aa64-08da53a11d65 X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB6030.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 16:14:27.5638 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TMFcKhjxiwuRbBUHmrJh1Zjmln5+QOiZXYSndzI+bBCthTrstY2zn0NxqrEizBw1GeioMlKPXgkxKvHo9C0/BQ3WzQFIUVCwU0m/69EO0PY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2865 X-Proofpoint-GUID: iAUrWEZGW_-Ffjhm0w9CqvrYWxCtq-MA X-Proofpoint-ORIG-GUID: iAUrWEZGW_-Ffjhm0w9CqvrYWxCtq-MA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-21_08,2022-06-21_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 priorityscore=1501 mlxlogscore=794 clxscore=1015 mlxscore=0 suspectscore=0 spamscore=0 phishscore=0 bulkscore=0 malwarescore=0 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206210067 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 21 Jun 2022 16:14:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/167162 The last time of sampling would be updated within the SystemStats class but not re-recorded into the datastore, leading to multiple samples being collected in the same second in the sample function of buildstats.py due to the runQueueTaskStarted events. Fix this to collect and store only one sample per second, as originally intended, by only recording samples on instances of the Heartbeat and BuildCompleted events. This fix elimates the spikiness of sampled data, in cases where the difference between the current and the last sample is taken. Previously, since many samples per second were recorded, certain types of data would result in a very small elapsed time and hence a small numerical difference. For example, the CPU usage from /proc/stat is a running total of usage and taking the difference between data collected 0.1 seconds apart would result in usage appearing lower than it actually was. Signed-off-by: Aryaman Gupta Signed-off-by: Randy MacLeod --- meta/classes/buildstats.bbclass | 3 ++- meta/lib/buildstats.py | 7 ++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass index 0de605200a..9e25a42a5c 100644 --- a/meta/classes/buildstats.bbclass +++ b/meta/classes/buildstats.bbclass @@ -282,10 +282,11 @@ python runqueue_stats () { if not system_stats and isinstance(e, (bb.runqueue.sceneQueueTaskStarted, bb.runqueue.runQueueTaskStarted)): system_stats = buildstats.SystemStats(d) d.setVar('_buildstats_system_stats', system_stats) - if system_stats: + if system_stats and isinstance(e, (bb.event.HeartbeatEvent, bb.event.BuildCompleted)): # Ensure that we sample at important events. done = isinstance(e, bb.event.BuildCompleted) system_stats.sample(e, force=done) + d.setVar('_buildstats_system_stats', system_stats) if done: system_stats.close() d.delVar('_buildstats_system_stats') diff --git a/meta/lib/buildstats.py b/meta/lib/buildstats.py index 64ad3ef40e..9829a0ff65 100644 --- a/meta/lib/buildstats.py +++ b/meta/lib/buildstats.py @@ -55,7 +55,12 @@ class SystemStats: # becames relevant when we get called very often while many # short tasks get started. Sampling during quiet periods # depends on the heartbeat event, which fires less often. - self.min_seconds = 1 + # The Heartbeat events occur roughly once every second but the actual time + # between these events deviates by a few milliseconds, in most cases. Hence + # pick a somewhat arbitary threshold such that we sample a large majority + # of the Heartbeat events. This ignores rare events that fall outside the minimum + # but allows for fairly consistent intervals between samples. + self.min_seconds = 0.990 self.meminfo_regex = re.compile(rb'^(MemTotal|MemFree|Buffers|Cached|SwapTotal|SwapFree):\s*(\d+)') self.diskstats_regex = re.compile(rb'^([hsv]d.|mtdblock\d|mmcblk\d|cciss/c\d+d\d+.*)$')