From patchwork Fri Apr 7 03:07:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: ChenQi X-Patchwork-Id: 22362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ED82C76196 for ; Fri, 7 Apr 2023 03:07:38 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.69.1680836848864336673 for ; Thu, 06 Apr 2023 20:07:28 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriver.com header.s=pps06212021 header.b=KbZeCVMS; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=3461ef5591=qi.chen@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3372iTZi004635 for ; Thu, 6 Apr 2023 20:07:28 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : subject : date : message-id : content-transfer-encoding : content-type : mime-version; s=PPS06212021; bh=mAiS3YvkXC5rKt7c8KFvpBBCUnIOoarFZF7cQbIfHHk=; b=KbZeCVMSHC5Et4hH46FeLVuA9MjPlODyBPqul4Y28ZYfRXXlAuDvDv3xMywqxlLpAxFa gV+DNX/5aEXXHm4p5Va7e2sjJo1z0yR0V+ZjLtbZ6ptcfgrnrSisemAYy/IrC1Y3NcXR j8OMLrr5fsynaEI1PPdr+R7Zq6uhb6grPiRSlufQToyIbJ+OYEdwWz1Fv7bxTBQPNQDI qsdf7WAYSUPzLF/z2ecih5C6jLDTGKfN1vBd9+2WTFFy8i/Q0lawMwGCzEOSzQUaDPNm GhrP9bI06FxqzMG9oIeZBFlh5IqWpmylE6KxkE+g2SWqucL1oJcg1T33R3A7o9hzPGbu vg== Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2108.outbound.protection.outlook.com [104.47.58.108]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3ppfw6xdaf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 06 Apr 2023 20:07:28 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SpnVPYsU6nf3ld3Goav5m6z+PuWnf+9vkIgnOj6jFEt41rVVhmIC2Wlb2SrfFV0rtwGZ4G48Y/bWgDzkm0Qeh5vFQZdFRUCEHRZ/Cb3SLKJDHH8AK/2kqVn4PiWHxznsgBlNlRXq6XHVX8RDQk3rWfZsPkjdHGx8ByRymFHT+UXmwqMCpgoH4mRj7+lg7gG0d41oYfguZeBamfzE/js5yw2nx0B6nq9vS6LlVWwQek3f8TDQH9ZrLts2WteK1UTNqRVQHVBy0WwBPtM5OmpauaeiZPxKcXIYc1y4aNad9bXOr4Vv7sz1a6qcax7w1Ydq+j4E7w+MyYmrUTrgRR94rA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mAiS3YvkXC5rKt7c8KFvpBBCUnIOoarFZF7cQbIfHHk=; b=Bj5CCqp4IWHa0X0/KZjMBDx3b7xBW9+dc0p5IIruCJB4c+t+KAYpQQDQO7B3aMJCBwEF6kkbcrAG8tawAaIM3mXmROlLy/i0Q1UaAwooFgNEK66s+Jwbeg4jx1adymffyHyUGkD1WG4dTqXhzjTeZ3JJBq270BafAMQjPaK/zR90Yu5QLXf/jdx8Bm8WgJiXI2Y/rwYmGwT8pLRhokL+18mdQ/N9rJMM7tqCnTF1KJM3m1Jn0T4pG8mBQxjF7et/HIf4UB5kvFuSsOC0F08ZUxFtJn5Hz9az5ary96zh7fqZy70tFu/ATaHtqHB4VmHJvi4dl3zjsEsoryOPdoXVxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) by DS0PR11MB7829.namprd11.prod.outlook.com (2603:10b6:8:f1::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Fri, 7 Apr 2023 03:07:26 +0000 Received: from CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::ab37:4985:7b95:2b95]) by CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::ab37:4985:7b95:2b95%6]) with mapi id 15.20.6254.035; Fri, 7 Apr 2023 03:07:26 +0000 From: Chen Qi To: bitbake-devel@lists.openembedded.org Subject: [bitbake-devel][PATCH 1/2] runqueue: fix PSI check calculation Date: Thu, 6 Apr 2023 20:07:14 -0700 Message-Id: <20230407030715.4394-1-Qi.Chen@windriver.com> X-Mailer: git-send-email 2.37.1 X-ClientProxiedBy: SJ0PR05CA0095.namprd05.prod.outlook.com (2603:10b6:a03:334::10) To CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR11MB5602:EE_|DS0PR11MB7829:EE_ X-MS-Office365-Filtering-Correlation-Id: accbe6e2-a038-4aa8-4af2-08db371536da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: d04r9+OAQEiKUX0wfcILqrA2stILOtJlHXv/rNvsh0HJ4Lp8+XWh+Dl/FmbdjDACNhr4QgWm67uGkowpFgzGsz2BioaTq5aBPkUxe2sJU5u7O70dO955e9xwXqSWXAqnAQnmv8N9mCK2TCzwuBGS+TdA8084Pdnr6xsfbQMb2fKtnjReMj4wSpFuQsJH4YWOpNnEpZC1fOvAHqP0CCMwIuTtyfIfMd6UESReAW1rzejQ0wg1H7bEeS4e1k+zb2oT+JAwQoKZGkhgaHv6+ekF3M/jBSGng5KP1KD9yTeF5yM09Q/N7SEXzK8M+dYIlTSVrdM1AL9FOZDXUkSwxuwPNKN8cgZHdhghf6eiL5hAXlnvR8UUfUQaviYGmNYy0dHALOEF20p+nm9eZMC/Lx3YrTpSjYeRE+kswHr8at2t6xIvOtMjq0rRkWi7/ALek93iZkGYyGA1Q99GT4rpk8aqNPb6j4z2td1sZk5lLJmYF9UPe3O1JgvqmWvXpjHoYkeOyNRRzaAQwQJomnrGgycTXoalTpBJuaXwsOvUluLISgtLVYLFVP5eHzhg59Tr1sgc4pNx3S+X9DPcLk2xvC49lYR+EYhW/KIcZXuYrqy2uPlsSr5Yc904eocOxGxTc3goNWN/8AV4/o9o9P7hySpkzA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR11MB5602.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(39850400004)(136003)(396003)(376002)(346002)(366004)(451199021)(2906002)(86362001)(36756003)(2616005)(186003)(6486002)(6512007)(52116002)(1076003)(6506007)(83380400001)(26005)(6666004)(316002)(66946007)(38100700002)(66556008)(478600001)(66476007)(6916009)(8676002)(41300700001)(5660300002)(38350700002)(8936002)(21314003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: z7brm/RUnBI+fB7UBsUtiELlGYE1ER6ApZdqQMA2pwEKiAN+CMOrTN/Xm5Efoo+EDs8/M0TfFVu2lufaR5dQLPJ8s5ZxlXdn2PVwnNrDFXbiV4e4kBsYh+yiqPALQ3jhUGCem+coMlkHeX3oSqu5TjxSxbCjeQca3r04UK2QmtIjWFtZ+wUmV0/DLsjbToQl77UNObDwuXH9SAxS3LN3vbzqkP2sw75abupKoiWuTTEviMiX4OdkKxTWGw5UtHN2xuviT21gxSo8JwqJgh3VtlyK8GKgvOj95AMNI0Vd7VbG+w60AK2PoIGFbugYF+7LsBRx0ZVeqVY4w6lFNrXU7jom8RDNhugqFci9mdrjzdeSe7J3HKDi7MjOjTbDpK0knXp3AFaTTK7+sjFhGzoYmiRQj0U7D7Uxw0PHlarjGgNY0Cl1E8/GxUUxymgv1v2G/N0ljEwnpc19+ZSgfjklWbNWRA/TKnCXh79xyW1yeqY/Sd3JImX0WlErH2aHBK6Vk0fLSQheoa+jkdK84t/CDhJcQBySiR7I00YIpaCdJCmN7fN5EWTVn7duBWTTSrOq9EQklOd12aLA1HJlkDmUumpTo7Z/vJI5QwZSzf3uDH2aW0Hnt2QLOJPrNUwAwuoRrEV0k9pb24hi5omNNERt2touZhQYVCpUGd//ftwM+Z590cN7exweKhxRz+Jw66essQvogHKc3NpRLrd0ud938yDoTfIOYVPh0WkbujvZPWvUtGr0uMhlMA6JIum0ZbyzfNnNVFn1XHaPXyU2ja+E5zK1W4bqPbZ6Ca57GCoJzhaQ008kB5XzDXmDkM1nom4d4zSN0awP7mSA7lMrIXkayl/WmE5VP7QN/wA6uoRbaDeXCehJnesc8tWYmpJKpuLZ/7veaiOArE++53kK8/ja0QeriPuWiZqICsyML7cNJefkGraDA9O9wxrG9WM2TYTTlWo2XRt3j3e4Sfj40dyOGO47hf34g6+POpjw1p0+gOgsKCH6NgaMm8XtElXdWGkTRWgiFIaSW6aSn4IV/91qPYvooOTh/9TjAWpUnITBZf6Zkpfi44XXXqmMU316sJMtP73qefKB4AS78/3Hcw/vcvSKNrXEJle+AOEPegEh3Spgjqv1wqgKaIHc+hqL/THHuxdES/Ae1sWWyGrf/IF3wAwrVI/RXcolAtqMfA33sHUtPkT9RYCkEz/U36C8j3AmyP8AoJqpKDAfTUXPPHF+zQHMASInOg+wS/AEATb34fjCibgbF9QQ+xNy54riVifD2IVihMAgYgY7tmusQ4U3CGGFEoMoP/C3ebwxPcZPVZGgLgNfhH5BHYr2y+Qik8ICT0MByZH7NqNbIs2qqv0AYjytedKfUfTyQLLNrhWb2ls1BVeKdJ11jEN5Vgs2wCioPwfxdSkil8xoynahl9o0PnjH4DDNDLZHSm92CBZ8K2pm8mKUSz5/CSTonvn2qtu8SVqlbXvgK3H2pG/hCmHj+eREBYDTLazFZVp+h6MkbU4lKRTtan8vWCKeVWi3DkCo6cSY2778vcQXcTuSXld89tbFQNTU68FuNVzMNPSEe8N1HVobcnBbi1UNi/6F4IB8 X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: accbe6e2-a038-4aa8-4af2-08db371536da X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5602.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Apr 2023 03:07:25.8753 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /GoyBLTIDueFmjPV0BZS7Nxvt5eff3oZ5x9a3bKKDsD/wR3jtuUpDA9m54bu+G6eVvjfpqKZ+lxpqLrr5WooQw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7829 X-Proofpoint-ORIG-GUID: B5NMD1lA70RTnrlzggYWmxdZNNFDsNPt X-Proofpoint-GUID: B5NMD1lA70RTnrlzggYWmxdZNNFDsNPt X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-06_14,2023-04-06_03,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 spamscore=0 clxscore=1011 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 adultscore=0 phishscore=0 mlxscore=0 priorityscore=1501 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304070028 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 07 Apr 2023 03:07:38 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14682 The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. Signed-off-by: Chen Qi --- bitbake/lib/bb/runqueue.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index e629ab7e7b..02f1474540 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -198,15 +198,20 @@ class RunQueueScheduler(object): curr_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1] curr_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1] curr_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1] - exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) > self.rq.max_cpu_pressure - exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) > self.rq.max_io_pressure - exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) > self.rq.max_memory_pressure now = time.time() - if now - self.prev_pressure_time > 1.0: + tdiff = now - self.prev_pressure_time + if tdiff > 1.0: + exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff > self.rq.max_cpu_pressure + exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff > self.rq.max_io_pressure + exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff > self.rq.max_memory_pressure self.prev_cpu_pressure = curr_cpu_pressure self.prev_io_pressure = curr_io_pressure self.prev_memory_pressure = curr_memory_pressure self.prev_pressure_time = now + else: + exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) > self.rq.max_cpu_pressure + exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) > self.rq.max_io_pressure + exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) > self.rq.max_memory_pressure return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure) return False From patchwork Fri Apr 7 03:07:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: ChenQi X-Patchwork-Id: 22361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6182BC77B6C for ; Fri, 7 Apr 2023 03:07:38 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.70.1680836849232696229 for ; Thu, 06 Apr 2023 20:07:29 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@windriver.com header.s=pps06212021 header.b=J72a1f4s; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=3461ef5591=qi.chen@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3372iTZj004635 for ; Thu, 6 Apr 2023 20:07:29 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=PPS06212021; bh=sJlbIir+dgUCJS3A7QC3N1oMGwsBMT48YvNL3LIquwU=; b=J72a1f4sjhyBLb2wGbzczdNUJYC+SGfyQS4YoAvzEfJn/1zmLnmLTBO5wrYXzJsqxIkG JN6aMbkH5kgzXkrl2vJoKOoJ9M6mxlfvHRPd8+wPxhx+X1Du11fTAoUUtXZ1ZH6CN3LK gzPK3DaNuaxX7NLFGovdkAB0cNtg5/JUpGlw4176zx0Znjc4tJBohPBlNn5mWcxAmEfH o1LPVbyu/rJZq7r80YZsxU2Qb24ZkE/2RI2lkcxuYACpb+dYagT++8+paUTQWv/Ie1Cr TIoPc48HuRxzJ2/8BCbTDpKMgMHMCs0/1FJap2a5Hb32zpqWxz5j7wGl5NOZIYRghFo4 dQ== Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2108.outbound.protection.outlook.com [104.47.58.108]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3ppfw6xdaf-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 06 Apr 2023 20:07:28 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FaAk0o1T1ISOxuiG5uJQltyvm6XoO4GcUBQ5VkibouQoJgE32Ux62dlntlxxBoHtXXJlaHnrIfWAAxRporTJbCJAjZ7CNmaHop8bsPvUNhp0rL8q3B/YEU+sdCBuP9aVJsvMlUWQnH3M+rbixgDBlHcQzaGUR+nrvio0YCe7Aq2m47dB227VfDsxSzNKKD6jDHw4lMt13FQXpMm/b7Ntjrxu48+jgCQVcXCaP1AqAeTySiX+gOEQI3POuQcO2dReMIu0IxN1zyPOYVB6AWojUQ2HofgKMIQM7P83nzuXh4k85sBy/IoN/Bi+fbo5JBKgMqXbFXFNEQhc4pGGN7Uesg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sJlbIir+dgUCJS3A7QC3N1oMGwsBMT48YvNL3LIquwU=; b=L9UTub1iBVnmtrWjYK7Uz6yRh2sgpyqVnf5KD32YQ1S1TwCc4tKzhhOqA1vcTn0JMQiVOwBsF6i+MsRZtMcgcHD9Wb6wIJgd/JtBBy3/YVeB28AP6TcAkIoucvkeafHp9dEiR9vJJDM/rOJdHTOfxtRxvP9Xiss2vVrL9XUpP/igcG4FARGPY/oi6v1SR3W3dBZiP1jN6GpAhTFayZ5xVKibox8r7vPjHs119NYUzO9A5zDPF9/e+CuA/mnl860nzVNnOUxfFqFcbqhfphp9HUppSsnWPJbbAc0dsLkhasx+PQDQ0ghkjNxqmnNoJuTgxDF17MFqk7NAY+HZIxCfwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) by DS0PR11MB7829.namprd11.prod.outlook.com (2603:10b6:8:f1::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Fri, 7 Apr 2023 03:07:27 +0000 Received: from CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::ab37:4985:7b95:2b95]) by CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::ab37:4985:7b95:2b95%6]) with mapi id 15.20.6254.035; Fri, 7 Apr 2023 03:07:26 +0000 From: Chen Qi To: bitbake-devel@lists.openembedded.org Subject: [bitbake-devel][PATCH 2/2] runqueue.py: use 'full' line for PSI check Date: Thu, 6 Apr 2023 20:07:15 -0700 Message-Id: <20230407030715.4394-2-Qi.Chen@windriver.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230407030715.4394-1-Qi.Chen@windriver.com> References: <20230407030715.4394-1-Qi.Chen@windriver.com> X-ClientProxiedBy: SJ0PR05CA0095.namprd05.prod.outlook.com (2603:10b6:a03:334::10) To CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR11MB5602:EE_|DS0PR11MB7829:EE_ X-MS-Office365-Filtering-Correlation-Id: 26da423c-805d-40e8-25c8-08db3715370a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pgHuZXWSLrVfNdujIuwrO0hGdhOmudTKPA3psgjWXtH3qV4QkmJUQJeYEir3UHP6mZHgfbTjsGHVytXcKO1Kqjx58mBly/2Tk4QBAygBP2kHGE7qmoJTb61EBYXyMrgY+3AZu62G69emVNcjxRaoJlwhJj2iNMBFxtl55ZFBYTBFPj3o6j9gBVDaf5m3U4CksVUXFWHtBGoTS4fwvdGYEJdLcGaVO2PuFcVqHrSVFXVhGYEesXDcZ1o1ILTAkkprXow1hxrKOUiI1ypPjSED5LY4vKHNLFJN6SH9te18AUlVwlwyVOO69hqWBI7LKiccwgDAFxxmw10coR7i7L6EoCgUe20JlKbDzWoxt8M7r1u0N3Q5skxR6fGbMFIEcOwbUnkX3yAQ8BjV63rb0VpXbRk954EisQCdlEh+cqMdFVkC6A1b043DD7i0ARkdbX6JPTuYtoUL8Sx+ULn31xTlLp+oUYAULc8oElD0q8V2Bx1NPn3kCHRADMVoji7Wb9HFfzBKddXavFVk0Aknx3BoQTdDvsLxUK+CWsf1yBf3rLTi1zXWokeBQ0HX3sK8huD9SmVx+EDiwxxlKVJsXfW32N9o/IdrN1zNBXXB1dVumcg= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR11MB5602.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(39850400004)(136003)(396003)(376002)(346002)(366004)(451199021)(2906002)(86362001)(36756003)(966005)(2616005)(186003)(6486002)(6512007)(52116002)(1076003)(6506007)(83380400001)(26005)(6666004)(316002)(66946007)(38100700002)(66556008)(478600001)(66476007)(6916009)(8676002)(41300700001)(5660300002)(38350700002)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?H0Xy6CiDIyTaQvoczX/nX92Wyg5E?= =?utf-8?q?N1qFSbwfmGGcIGucslMLJFbo6DpH1jTsLgRxiwclqfzpXSJuu+GhK8FuTEH9Z+T1y?= =?utf-8?q?XYqnkTFOxUUSprIqmnWtRXdor9gMkc1H8GOVokBppZThCqk5MrSZY3Kj1Jba7Nh9Q?= =?utf-8?q?7/RYesL3t+JBp7txbgkD3ykoiA9o8Q5WnDt8GvRTrwrwssCMDfhIuTGbOyYhkulkc?= =?utf-8?q?2NJRfFfBG/DpP0wLaBa9CNEQtlY+y7iw4dvyo2wOW/jAOa8g/IwKg8//ViU87lKyf?= =?utf-8?q?tVGjtM261jSnHWT0wdZHoPpJlRiRTW8NAmjZCAa7gsGaHzMO9mMZuBJljgT2z+7sE?= =?utf-8?q?UUEIXi4q7l76ypdo/ZG7RgmnAHaluq8CzlCPh9qwgnduVBf5OCm3pQ13un6hI7frd?= =?utf-8?q?+1V2WDRDNS7Ldu4Zdo3bPBaBMc8Rd8DXhsc7iZUBTp0601fF4d8hP0FamO+gDMFuV?= =?utf-8?q?5z53qs2VumCK+D5ZB33qb+6ecBpCXzAwzkxspcO+l7xVtoXcwcWTQ67jTXeO02Hap?= =?utf-8?q?aXtfW4HQ00uls5M0qvR6AisZGt5tacc9YcwGGxluYV/3qD2aj2160jN6Qjk64JWhL?= =?utf-8?q?J929omhvdupZ7rD3hGfJ9krcJOsPP2FP/igjWCbSsudWXs6aU9Y+U87OO/kAsI1oO?= =?utf-8?q?SfbBYamq0r9SnNcepGLetROz1vHH5GkxeU20CUo8TbtQECmgmO3Xx4WqPidq5W1OZ?= =?utf-8?q?hEJ1fHRzuBY6uUwHW9DQBRMordGzjlvLr3Vza5tduCTI3YN7/IGHD8OjqOhVoxmX9?= =?utf-8?q?/w2E1VO07KgYpO+GLN6ZwaUGwgYkn/E9JPZJkdf9IJSXxXefxTOZf8ll/opsIyGfN?= =?utf-8?q?56iy0IIy0TdnWJNIfmcFHUm2lYOJqN+ngU3ngW5LapXmmnqHYjX5LYc3YLqvyUVst?= =?utf-8?q?xDv+3JSm1ABqVPmvAG/FBqxmkUSiYeZ80/zcHD2u3geYGWz/JMR6JHyoGXMlPQyZ3?= =?utf-8?q?NNGq5fUAeadEsB3FA4LWV6KE/MS2Rzvd59TIIQ44Dpg/Jk4s+TQJ8uXOCoH+5WDlo?= =?utf-8?q?cqIoVLo/h+5qyXVg9uKhWsqr4bcDJecsjPfFGqiyqm41JI7IXRXG6LQ8iFvzcL6Vo?= =?utf-8?q?ndIk0o5WTtLH1nSATLmItZPHvmZeZgc2tUuTAXo5sE0nop2ahn6t0bZmak1aK2+xQ?= =?utf-8?q?431jFOb/ntxpiu1AD03XqDccC49TP3kvGKV1Bip5h8AT6UMJzOivY5Zz1BS5Noh71?= =?utf-8?q?Bhvd2VevIfgosihq9ar98laSNckiCOP4VoGLkhvlrLw+Ly+LEOksUsYfcyV+Yy2j/?= =?utf-8?q?cMQnd+Dd5BSPAwNIRViEheZVOHQAhwIEuOp1Nx1B+nDuxI8uExwR3i/UmeaItx4F3?= =?utf-8?q?8c6ln9Tr07LObAVv+vWUAojZy1ZrWgU0OB0u61lmkauN2PLFhoP7+nAvhd7zAxRcE?= =?utf-8?q?I1/g8RN2R9P7iv7FpcExT4tnh84kXpIG8Z4fJJ8MAq/yl2Eztbw5fhQvwZvJA4Oot?= =?utf-8?q?vKPZk2+Zau9kaXQTzRUAQs83RRBZwyk/dAyXaFoi4mVVjiFiMEodq8/8yxcwEZ0/G?= =?utf-8?q?2Jpz+9kJ2W8m3ol/OgibkS9Lw8O5lQfPWg=3D=3D?= X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 26da423c-805d-40e8-25c8-08db3715370a X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5602.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Apr 2023 03:07:26.1237 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /MmNL8hFOCHmtAthRsWewSeR4nYqHtAtluwLUwEcNB8wap1kq9fx41p2DshmH0+aEQVlPbA8sEyjE/jlkzIO2g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7829 X-Proofpoint-ORIG-GUID: orLjEQ2xt-9NrVQfDYClZ4gYRKG5SghD X-Proofpoint-GUID: orLjEQ2xt-9NrVQfDYClZ4gYRKG5SghD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-06_14,2023-04-06_03,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 spamscore=0 clxscore=1015 mlxlogscore=483 lowpriorityscore=0 impostorscore=0 adultscore=0 phishscore=0 mlxscore=0 priorityscore=1501 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304070028 X-MIME-Autoconverted: from 8bit to quoted-printable by mx0a-0064b401.pphosted.com id 3372iTZj004635 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 07 Apr 2023 03:07:38 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14683 According to kernel PSI doc[1], for /proc/pressure/* interface files, the first line is the 'some' line and the second line is the 'full' line. Quoting from the doc: """ The "some" line indicates the share of time in which at least some tasks are stalled on a given resource. The "full" line indicates the share of time in which all non-idle tasks are stalled on a given resource simultaneously. In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. This has severe impact on performance, and it’s useful to distinguish this situation from a state where some tasks are stalled but the CPU is still doing productive work. """ We can see that the 'full' line is a better measurement to check if things are slowed down as a whole. Also, the /proc/pressure/cpu's 'full' line may not be available on some systems, so fall back to the 'some' line. [1] https://www.kernel.org/doc/html/latest/accounting/psi.html Signed-off-by: Chen Qi --- bitbake/lib/bb/runqueue.py | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 02f1474540..e3198dcdeb 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -173,10 +173,13 @@ class RunQueueScheduler(object): with open("/proc/pressure/cpu") as cpu_pressure_fds, \ open("/proc/pressure/io") as io_pressure_fds, \ open("/proc/pressure/memory") as memory_pressure_fds: - - self.prev_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1] - self.prev_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1] - self.prev_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1] + cpu_lines = cpu_pressure_fds.readlines() + if len(cpu_lines) == 1: + self.prev_cpu_pressure = cpu_lines[0].split()[4].split("=")[1] + else: + self.prev_cpu_pressure = cpu_lines[1].split()[4].split("=")[1] + self.prev_io_pressure = io_pressure_fds.readlines()[1].split()[4].split("=")[1] + self.prev_memory_pressure = memory_pressure_fds.readlines()[1].split()[4].split("=")[1] self.prev_pressure_time = time.time() self.check_pressure = True except: @@ -194,10 +197,14 @@ class RunQueueScheduler(object): with open("/proc/pressure/cpu") as cpu_pressure_fds, \ open("/proc/pressure/io") as io_pressure_fds, \ open("/proc/pressure/memory") as memory_pressure_fds: - # extract "total" from /proc/pressure/{cpu|io} - curr_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1] - curr_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1] - curr_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1] + # extract "total" from /proc/pressure/{cpu|io|memory} + cpu_lines = cpu_pressure_fds.readlines() + if len(cpu_lines) == 1: + curr_cpu_pressure = cpu_lines[0].split()[4].split("=")[1] + else: + curr_cpu_pressure = cpu_lines[1].split()[4].split("=")[1] + curr_io_pressure = io_pressure_fds.readlines()[1].split()[4].split("=")[1] + curr_memory_pressure = memory_pressure_fds.readlines()[1].split()[4].split("=")[1] now = time.time() tdiff = now - self.prev_pressure_time if tdiff > 1.0: