From patchwork Thu Sep 21 13:48:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30892 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E597AE71072 for ; Thu, 21 Sep 2023 13:48:40 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.17971.1695304115787547849 for ; Thu, 21 Sep 2023 06:48:36 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8FD1A169E; Thu, 21 Sep 2023 06:49:12 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EB4B73F59C; Thu, 21 Sep 2023 06:48:34 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 1/8] oeqa/runtime/parselogs: remove unused imports Date: Thu, 21 Sep 2023 14:48:26 +0100 Message-Id: <20230921134833.582827-2-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187997 From: Ross Burton Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index e67d3750dad..8fd5a86e141 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -10,8 +10,6 @@ from subprocess import check_output from shutil import rmtree from oeqa.runtime.case import OERuntimeTestCase from oeqa.core.decorator.depends import OETestDepends -from oeqa.core.decorator.data import skipIfDataVar -from oeqa.runtime.decorator.package import OEHasPackage #in the future these lists could be moved outside of module errors = ["error", "cannot", "can\'t", "failed"] From patchwork Thu Sep 21 13:48:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C496E71072 for ; Thu, 21 Sep 2023 13:48:51 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.17947.1695304122504079869 for ; Thu, 21 Sep 2023 06:48:42 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 375FF16A3; Thu, 21 Sep 2023 06:49:13 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9BBDE3F59C; Thu, 21 Sep 2023 06:48:35 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 2/8] oeqa/runtime/parselogs: don't bother to show target hardware information Date: Thu, 21 Sep 2023 14:48:27 +0100 Message-Id: <20230921134833.582827-3-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/188005 From: Ross Burton This information is pretty useless as we know what the target is from the build configuration (be it a qemu machine or real hardware). Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 29 ------------------------ 1 file changed, 29 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index 8fd5a86e141..e62ab2c44ec 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -237,34 +237,6 @@ class ParseLogsTest(OERuntimeTestCase): def getWorkdir(self): return self.td.get('WORKDIR', '') - # Get some information on the CPU of the machine to display at the - # beginning of the output. This info might be useful in some cases. - def getHardwareInfo(self): - hwi = "" - cmd = ('cat /proc/cpuinfo | grep "model name" | head -n1 | ' - " awk 'BEGIN{FS=\":\"}{print $2}'") - _, cpu_name = self.target.run(cmd) - - cmd = ('cat /proc/cpuinfo | grep "cpu cores" | head -n1 | ' - "awk {'print $4'}") - _, cpu_physical_cores = self.target.run(cmd) - - cmd = 'cat /proc/cpuinfo | grep "processor" | wc -l' - _, cpu_logical_cores = self.target.run(cmd) - - _, cpu_arch = self.target.run('uname -m') - - hwi += 'Machine information: \n' - hwi += '*******************************\n' - hwi += 'Machine name: ' + self.getMachine() + '\n' - hwi += 'CPU: ' + str(cpu_name) + '\n' - hwi += 'Arch: ' + str(cpu_arch)+ '\n' - hwi += 'Physical cores: ' + str(cpu_physical_cores) + '\n' - hwi += 'Logical cores: ' + str(cpu_logical_cores) + '\n' - hwi += '*******************************\n' - - return hwi - # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file # just add it to that list. @@ -376,7 +348,6 @@ class ParseLogsTest(OERuntimeTestCase): self.write_dmesg() log_list = self.get_local_log_list(self.log_locations) result = self.parse_logs(self.errors, self.ignore_errors, log_list) - print(self.getHardwareInfo()) errcount = 0 for log in result: self.msg += 'Log: ' + log + '\n' From patchwork Thu Sep 21 13:48:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30894 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CA1DE7107A for ; Thu, 21 Sep 2023 13:48:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.17975.1695304119072718471 for ; Thu, 21 Sep 2023 06:48:39 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DC36716F2; Thu, 21 Sep 2023 06:49:13 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4326E3F59C; Thu, 21 Sep 2023 06:48:36 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 3/8] oeqa/runtime/parselogs: remove obsolete LSB testing support Date: Thu, 21 Sep 2023 14:48:28 +0100 Message-Id: <20230921134833.582827-4-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/188002 From: Ross Burton The LSB compliance tests were removed in 2019[1], so this is obsolete. [1] oe-core fb064356 Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 8 -------- 1 file changed, 8 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index e62ab2c44ec..e670df3cedd 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -69,9 +69,6 @@ common_errors = [ "Failed to read LoaderEntryOneShot variable, ignoring: Operation not supported", ] -video_related = [ -] - x86_common = [ '[drm:psb_do_init] *ERROR* Debug is', 'wrong ELF class', @@ -225,11 +222,6 @@ class ParseLogsTest(OERuntimeTestCase): cls.ignore_errors = ignore_errors cls.log_locations = log_locations cls.msg = '' - is_lsb, _ = cls.tc.target.run("which LSB_Test.sh") - if is_lsb == 0: - for machine in cls.ignore_errors: - cls.ignore_errors[machine] = cls.ignore_errors[machine] \ - + video_related def getMachine(self): return self.td.get('MACHINE', '') From patchwork Thu Sep 21 13:48:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E72B0E71073 for ; Thu, 21 Sep 2023 13:48:40 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.17974.1695304117713351821 for ; Thu, 21 Sep 2023 06:48:37 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8416116F3; Thu, 21 Sep 2023 06:49:14 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E80B93F59C; Thu, 21 Sep 2023 06:48:36 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 4/8] oeqa/runtime/parselogs: inline single-caller functions Date: Thu, 21 Sep 2023 14:48:29 +0100 Message-Id: <20230921134833.582827-5-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/187999 From: Ross Burton There's no need to have one-liner functions to get the MACHINE or WORKDIR when they're only called once. Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index e670df3cedd..6e5dc753060 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -223,12 +223,6 @@ class ParseLogsTest(OERuntimeTestCase): cls.log_locations = log_locations cls.msg = '' - def getMachine(self): - return self.td.get('MACHINE', '') - - def getWorkdir(self): - return self.td.get('WORKDIR', '') - # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file # just add it to that list. @@ -251,7 +245,7 @@ class ParseLogsTest(OERuntimeTestCase): # Copy the log files to be parsed locally def transfer_logs(self, log_list): - workdir = self.getWorkdir() + workdir = self.td.get('WORKDIR') self.target_logs = workdir + '/' + 'target_logs' target_logs = self.target_logs if os.path.exists(target_logs): @@ -278,7 +272,7 @@ class ParseLogsTest(OERuntimeTestCase): grepcmd += '" ' + str(log) + " | grep -Eiv \'" try: - errorlist = ignore_errors[self.getMachine()] + errorlist = ignore_errors[self.td.get('MACHINE')] except KeyError: self.msg += 'No ignore list found for this machine, using default\n' errorlist = ignore_errors['default'] From patchwork Thu Sep 21 13:48:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1D29E71076 for ; Thu, 21 Sep 2023 13:48:40 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.17974.1695304117713351821 for ; Thu, 21 Sep 2023 06:48:38 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2D114169C; Thu, 21 Sep 2023 06:49:15 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 900793F59C; Thu, 21 Sep 2023 06:48:37 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 5/8] oeqa/runtime/parselogs: improve find call Date: Thu, 21 Sep 2023 14:48:30 +0100 Message-Id: <20230921134833.582827-6-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/188000 From: Ross Burton getLogList() uses remote find invocations to find the logs. Instead of relying on shell expansion of wildcards and redundant use of -maxdepth (pointless as the shell expansion means the find is passed the files to return), invoke find idiomatically by telling it what directory to search for and escape the glob so find processes it. Also remove many pointless str() calls. Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index 6e5dc753060..93782b844b9 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -229,18 +229,18 @@ class ParseLogsTest(OERuntimeTestCase): def getLogList(self, log_locations): logs = [] for location in log_locations: - status, _ = self.target.run('test -f ' + str(location)) + status, _ = self.target.run('test -f %s' % location) if status == 0: - logs.append(str(location)) + logs.append(location) else: - status, _ = self.target.run('test -d ' + str(location)) + status, _ = self.target.run('test -d %s' % location) if status == 0: - cmd = 'find ' + str(location) + '/*.log -maxdepth 1 -type f' + cmd = 'find %s -name \\*.log -maxdepth 1 -type f' % location status, output = self.target.run(cmd) if status == 0: output = output.splitlines() for logfile in output: - logs.append(os.path.join(location, str(logfile))) + logs.append(os.path.join(location, logfile)) return logs # Copy the log files to be parsed locally From patchwork Thu Sep 21 13:48:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00949E71074 for ; Thu, 21 Sep 2023 13:48:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.17974.1695304117713351821 for ; Thu, 21 Sep 2023 06:48:38 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C80F916F8; Thu, 21 Sep 2023 06:49:15 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 374233F59C; Thu, 21 Sep 2023 06:48:38 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 6/8] oeqa/runtime/parselogs: don't pass around members Date: Thu, 21 Sep 2023 14:48:31 +0100 Message-Id: <20230921134833.582827-7-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/188001 From: Ross Burton There's no point in passing around member fields, just access them directly. Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index 93782b844b9..0262f574d1a 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -263,19 +263,19 @@ class ParseLogsTest(OERuntimeTestCase): return logs # Build the grep command to be used with filters and exclusions - def build_grepcmd(self, errors, ignore_errors, log): + def build_grepcmd(self, log): grepcmd = 'grep ' grepcmd += '-Ei "' - for error in errors: + for error in self.errors: grepcmd += r'\<' + error + r'\>' + '|' grepcmd = grepcmd[:-1] grepcmd += '" ' + str(log) + " | grep -Eiv \'" try: - errorlist = ignore_errors[self.td.get('MACHINE')] + errorlist = self.ignore_errors[self.td.get('MACHINE')] except KeyError: self.msg += 'No ignore list found for this machine, using default\n' - errorlist = ignore_errors['default'] + errorlist = self.ignore_errors['default'] for ignore_error in errorlist: ignore_error = ignore_error.replace('(', r'\(') @@ -292,17 +292,21 @@ class ParseLogsTest(OERuntimeTestCase): return grepcmd - # Grep only the errors so that their context could be collected. - # Default context is 10 lines before and after the error itself - def parse_logs(self, errors, ignore_errors, logs, - lines_before = 10, lines_after = 10): + def parse_logs(self, logs, lines_before=10, lines_after=10): + """ + Search the log files @logs looking for error lines (marked by + @self.errors), ignoring anything listed in @self.ignore_errors. + + Returns a dictionary of log filenames to a dictionary of error lines to + the error context (controlled by @lines_before and @lines_after). + """ results = {} rez = [] grep_output = '' for log in logs: result = None - thegrep = self.build_grepcmd(errors, ignore_errors, log) + thegrep = self.build_grepcmd(log) try: result = check_output(thegrep, shell=True).decode('utf-8') @@ -333,7 +337,7 @@ class ParseLogsTest(OERuntimeTestCase): def test_parselogs(self): self.write_dmesg() log_list = self.get_local_log_list(self.log_locations) - result = self.parse_logs(self.errors, self.ignore_errors, log_list) + result = self.parse_logs(log_list) errcount = 0 for log in result: self.msg += 'Log: ' + log + '\n' From patchwork Thu Sep 21 13:48:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DC34E7107B for ; Thu, 21 Sep 2023 13:48:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.17974.1695304117713351821 for ; Thu, 21 Sep 2023 06:48:39 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6E3C9169E; Thu, 21 Sep 2023 06:49:16 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D2B173F59C; Thu, 21 Sep 2023 06:48:38 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 7/8] oeqa/runtime/parselogs: move some variables out of global scope Date: Thu, 21 Sep 2023 14:48:32 +0100 Message-Id: <20230921134833.582827-8-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/188003 From: Ross Burton errors and log_locations can be trivially set in the class directly, instead of being defined in the module and then copied into the class. Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index 0262f574d1a..3f205661ea9 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -11,9 +11,6 @@ from shutil import rmtree from oeqa.runtime.case import OERuntimeTestCase from oeqa.core.decorator.depends import OETestDepends -#in the future these lists could be moved outside of module -errors = ["error", "cannot", "can\'t", "failed"] - common_errors = [ "(WW) warning, (EE) error, (NI) not implemented, (??) unknown.", "dma timeout", @@ -201,17 +198,19 @@ ignore_errors = { ] + common_errors, } -log_locations = ["/var/log/","/var/log/dmesg", "/tmp/dmesg_output.log"] - class ParseLogsTest(OERuntimeTestCase): + # Which log files should be collected + log_locations = ["/var/log/", "/var/log/dmesg", "/tmp/dmesg_output.log"] + + # The keywords that identify error messages in the log files + errors = ["error", "cannot", "can't", "failed"] + @classmethod def setUpClass(cls): - cls.errors = errors - # When systemd is enabled we need to notice errors on # circular dependencies in units. - if 'systemd' in cls.td.get('DISTRO_FEATURES', ''): + if 'systemd' in cls.td.get('DISTRO_FEATURES'): cls.errors.extend([ 'Found ordering cycle on', 'Breaking ordering cycle by deleting job', @@ -220,8 +219,6 @@ class ParseLogsTest(OERuntimeTestCase): ]) cls.ignore_errors = ignore_errors - cls.log_locations = log_locations - cls.msg = '' # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file @@ -338,7 +335,9 @@ class ParseLogsTest(OERuntimeTestCase): self.write_dmesg() log_list = self.get_local_log_list(self.log_locations) result = self.parse_logs(log_list) + errcount = 0 + self.msg = "" for log in result: self.msg += 'Log: ' + log + '\n' self.msg += '-----------------------\n' From patchwork Thu Sep 21 13:48:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 30895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D22CE7107D for ; Thu, 21 Sep 2023 13:48:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.17974.1695304117713351821 for ; Thu, 21 Sep 2023 06:48:40 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1591816F2; Thu, 21 Sep 2023 06:49:17 -0700 (PDT) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7A08A3F59C; Thu, 21 Sep 2023 06:48:39 -0700 (PDT) From: ross.burton@arm.com To: openembedded-core@lists.openembedded.org Cc: nd@arm.com Subject: [PATCH 8/8] oeqa/runtime/parselogs: select the correct machine-specific ignores early Date: Thu, 21 Sep 2023 14:48:33 +0100 Message-Id: <20230921134833.582827-9-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230921134833.582827-1-ross.burton@arm.com> References: <20230921134833.582827-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Sep 2023 13:48:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/188004 From: Ross Burton This has no impact to the execution, but makes the following changes neater. Signed-off-by: Ross Burton --- meta/lib/oeqa/runtime/cases/parselogs.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py index 3f205661ea9..c889be8f6ea 100644 --- a/meta/lib/oeqa/runtime/cases/parselogs.py +++ b/meta/lib/oeqa/runtime/cases/parselogs.py @@ -218,7 +218,11 @@ class ParseLogsTest(OERuntimeTestCase): 'Ordering cycle found, skipping', ]) - cls.ignore_errors = ignore_errors + try: + cls.ignore_errors = ignore_errors[cls.td.get('MACHINE')] + except KeyError: + cls.logger.info('No ignore list found for this machine, using default') + cls.ignore_errors = cls.ignore_errors['default'] # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file @@ -268,13 +272,8 @@ class ParseLogsTest(OERuntimeTestCase): grepcmd = grepcmd[:-1] grepcmd += '" ' + str(log) + " | grep -Eiv \'" - try: - errorlist = self.ignore_errors[self.td.get('MACHINE')] - except KeyError: - self.msg += 'No ignore list found for this machine, using default\n' - errorlist = self.ignore_errors['default'] - for ignore_error in errorlist: + for ignore_error in self.ignore_errors: ignore_error = ignore_error.replace('(', r'\(') ignore_error = ignore_error.replace(')', r'\)') ignore_error = ignore_error.replace("'", '.')