From patchwork Tue Dec 19 14:35:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 36663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BD32C41535 for ; Tue, 19 Dec 2023 14:35:40 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.15004.1702996534802274192 for ; Tue, 19 Dec 2023 06:35:35 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=LSpgQ+wh; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-40d12ade25dso32614715e9.2 for ; Tue, 19 Dec 2023 06:35:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1702996533; x=1703601333; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=5eTLlPe0lWzbN9ISuG25z9byrRgwYPd9KUOukWx5hT4=; b=LSpgQ+whm30Ml2JQUe1UPG5CfkjmYMJZzRPjngMw0ZuKQ5XETfC9t6XJFxfQlcp7S6 sdHnWpcgJ9o1tTOFWikQd/XcXvBn3LQrY82odBnB+ftRiWwe4PF2DyCU2FqlBUmN4yz9 VO+mh5gCoalOfTL+Hn32qU0/JCQ4Ppc7YMxL0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702996533; x=1703601333; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5eTLlPe0lWzbN9ISuG25z9byrRgwYPd9KUOukWx5hT4=; b=wiqJP5qnFCOg+IBSGZegDZEDgaIpiGURo8FLSNB0pDLddAn5GlgortfUlcUo73zrQM pG6eVUpNM4PG00IGc9dnfHi3QlTrdkcGTNS8PE8MbYSW/uLsPbH8EGI57fRCNnENCAZT h+PSqm83EsI/nBYfgrGADxs+6p1AwFedRYGNgewt4kq0VbdrjoFDtfZlM+K/zklYn+Pn jfP+HJSi3y8I8N6s5Hi0M/ibGM01X8Qw+dYwPJTZrjEISJ+aNG7f/VKJpFYstYpPfmnp f2tCmj3f7iawhgl8gx/RZ2fxMjJwK9wndWUAcvPlXFPz1UYzJPhE42C3ai9KP09ojOym BTmw== X-Gm-Message-State: AOJu0YzS6B4UokNXU//qxYuj8ZwkP9Gu8oTyFUBvNtw9eD7sTZbygX8W ZrRBu6vEcQUETSzI+Fr/gxjbZAfF+diJq/FxvN0= X-Google-Smtp-Source: AGHT+IGHkQbv9UHOwjDArq1QIzwL1oqluBWwdO0EslM7EKT/pOqT64h4WmoIqAPNMpbwPO5qEWtLYw== X-Received: by 2002:a05:600c:198f:b0:40c:707d:16f0 with SMTP id t15-20020a05600c198f00b0040c707d16f0mr2971049wmq.23.1702996532893; Tue, 19 Dec 2023 06:35:32 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b87d:537f:7c6:e419]) by smtp.gmail.com with ESMTPSA id m5-20020a05600c4f4500b0040c57e4ea28sm3168889wmq.17.2023.12.19.06.35.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 06:35:32 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/5] testimage: Exclude wtmp from target-dumper commands Date: Tue, 19 Dec 2023 14:35:27 +0000 Message-Id: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Dec 2023 14:35:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192740 wtmp is filled with binary data which the run_serial command can't cope with. Catting this results in confusion of the serial interface and potentially large backlogs of data in the buffers which can hang qemu. Exclude the problematic files from the command. Signed-off-by: Richard Purdie --- meta/classes-recipe/testimage.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass index 7c56fe9674c..c6b7a1e142f 100644 --- a/meta/classes-recipe/testimage.bbclass +++ b/meta/classes-recipe/testimage.bbclass @@ -121,7 +121,7 @@ testimage_dump_target () { netstat -an ip address # Next command will dump logs from /var/log/ - find /var/log/ -type f 2>/dev/null -exec echo "====================" \; -exec echo {} \; -exec echo "====================" \; -exec cat {} \; -exec echo "" \; + find /var/log/ -type f -name !wtmp* 2>/dev/null -exec echo "====================" \; -exec echo {} \; -exec echo "====================" \; -exec cat {} \; -exec echo "" \; } testimage_dump_monitor () { From patchwork Tue Dec 19 14:35:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 36667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29587C46CD7 for ; Tue, 19 Dec 2023 14:35:40 +0000 (UTC) Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by mx.groups.io with SMTP id smtpd.web11.15005.1702996535496103182 for ; Tue, 19 Dec 2023 06:35:35 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Hhq3LFSt; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-3364c9ff8e1so2537681f8f.0 for ; Tue, 19 Dec 2023 06:35:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1702996533; x=1703601333; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=mcUB0zdAvoy0z09thntI4WuPRyb3nadRKCMzxIW/3RM=; b=Hhq3LFStH5HiBgxc5kluosjiPPrSKszjka4bxOE8Y7LUav4KG2zAgKxsT2H6oKagtA iz0KjoXmO1wYXLpEVNBYm2AENkS2R2w//blxSjrICASfGRDwPo8AgDxFsCpIU7g1JUFE fKUfdney+HumixmkRjIh7RIO2BzsUFU3G/Qf8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702996533; x=1703601333; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mcUB0zdAvoy0z09thntI4WuPRyb3nadRKCMzxIW/3RM=; b=CXnfsOjB5fZ+kDRDoA2uApJZ6bZzYeGw6vhVKp8eMuoxeQN/NM5WSRsT2GA4rcwfih X4PG+h2+LD962/hk/Kc3EtX9kkMgHbPGT2C45W9MrStK8xeF21RoXJY5ilopdbARE0Z6 T+km+v1buT1fiWbEOAHKDosSxK0/+d4n7xgTW/LOLdLCxqw1l+9CtZ/Uk4An4unv5s3p 4mWuM48Wqf58uV6ERM8MwQv3wD0KR0luEPkTZ1hEMfy/k4KyVvX28+c55Pwwdo+cmj61 /zP/sURu6qCE1kKh2R4XqiXVbg73dCfIhZWTI/lFmx80oXSrHkuiV1Tco5WOhMpxY+mu +YWQ== X-Gm-Message-State: AOJu0YzFxc4d8peOCIzBiKkrLEiWEXmqXt9n+5Dtfip3monbW54iyHY6 YCjuwLzs0y8e+yLqa+0NOUdINjTGQVfzSBLDQWc= X-Google-Smtp-Source: AGHT+IEux/mSYWFX9yuyM8XirTNuP5enYHCvTM+OAWvFdBIePSAIF+f/fZwk6b9Yi67cW29ygQUfJA== X-Received: by 2002:a1c:7505:0:b0:40c:6555:8faf with SMTP id o5-20020a1c7505000000b0040c65558fafmr613284wmc.180.1702996533574; Tue, 19 Dec 2023 06:35:33 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b87d:537f:7c6:e419]) by smtp.gmail.com with ESMTPSA id m5-20020a05600c4f4500b0040c57e4ea28sm3168889wmq.17.2023.12.19.06.35.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 06:35:33 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/5] qemurunner: Improve stdout logging handling Date: Tue, 19 Dec 2023 14:35:28 +0000 Message-Id: <20231219143531.3514266-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> References: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Dec 2023 14:35:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192741 We need to ensure we read from and log the output from qemu stdout as otherwise the buffers can fill and block, leading qemu to hand. Use our existing logging thread to do this. Signed-off-by: Richard Purdie --- meta/lib/oeqa/utils/qemurunner.py | 54 ++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py index ee367078000..cdd0db58776 100644 --- a/meta/lib/oeqa/utils/qemurunner.py +++ b/meta/lib/oeqa/utils/qemurunner.py @@ -29,6 +29,15 @@ control_chars = [chr(x) for x in control_range if chr(x) not in string.printable] re_control_char = re.compile('[%s]' % re.escape("".join(control_chars))) +def getOutput(o): + import fcntl + fl = fcntl.fcntl(o, fcntl.F_GETFL) + fcntl.fcntl(o, fcntl.F_SETFL, fl | os.O_NONBLOCK) + try: + return os.read(o.fileno(), 1000000).decode("utf-8") + except BlockingIOError: + return "" + class QemuRunner: def __init__(self, machine, rootfs, display, tmpdir, deploy_dir_image, logfile, boottime, dump_dir, use_kvm, logger, use_slirp=False, @@ -55,6 +64,7 @@ class QemuRunner: self.boottime = boottime self.logged = False self.thread = None + self.threadsock = None self.use_kvm = use_kvm self.use_ovmf = use_ovmf self.use_slirp = use_slirp @@ -119,21 +129,11 @@ class QemuRunner: f.write(msg) self.msg += self.decode_qemulog(msg) - def getOutput(self, o): - import fcntl - fl = fcntl.fcntl(o, fcntl.F_GETFL) - fcntl.fcntl(o, fcntl.F_SETFL, fl | os.O_NONBLOCK) - try: - return os.read(o.fileno(), 1000000).decode("utf-8") - except BlockingIOError: - return "" - - def handleSIGCHLD(self, signum, frame): if self.runqemu and self.runqemu.poll(): if self.runqemu.returncode: self.logger.error('runqemu exited with code %d' % self.runqemu.returncode) - self.logger.error('Output from runqemu:\n%s' % self.getOutput(self.runqemu.stdout)) + self.logger.error('Output from runqemu:\n%s' % getOutput(self.runqemu.stdout)) self.stop() def start(self, qemuparams = None, get_ip = True, extra_bootparams = None, runqemuparams='', launch_cmd=None, discard_writes=True): @@ -282,7 +282,7 @@ class QemuRunner: if self.runqemu.returncode: # No point waiting any longer self.logger.warning('runqemu exited with code %d' % self.runqemu.returncode) - self.logger.warning("Output from runqemu:\n%s" % self.getOutput(output)) + self.logger.warning("Output from runqemu:\n%s" % getOutput(output)) self.stop() return False time.sleep(0.5) @@ -309,7 +309,7 @@ class QemuRunner: ps = subprocess.Popen(['ps', 'axww', '-o', 'pid,ppid,pri,ni,command '], stdout=subprocess.PIPE).communicate()[0] processes = ps.decode("utf-8") self.logger.debug("Running processes:\n%s" % processes) - op = self.getOutput(output) + op = getOutput(output) self.stop() if op: self.logger.error("Output from runqemu:\n%s" % op) @@ -387,7 +387,7 @@ class QemuRunner: time.time() - connect_time)) # We are alive: qemu is running - out = self.getOutput(output) + out = getOutput(output) netconf = False # network configuration is not required by default self.logger.debug("qemu started in %.2f seconds - qemu procces pid is %s (%s)" % (time.time() - (endtime - self.runqemutime), @@ -430,9 +430,10 @@ class QemuRunner: self.logger.debug("Target IP: %s" % self.ip) self.logger.debug("Server IP: %s" % self.server_ip) + self.thread = LoggingThread(self.log, self.threadsock, self.logger, self.runqemu.stdout) + self.thread.start() + if self.serial_ports >= 2: - self.thread = LoggingThread(self.log, self.threadsock, self.logger) - self.thread.start() if not self.thread.connection_established.wait(self.boottime): self.logger.error("Didn't receive a console connection from qemu. " "Here is the qemu command line used:\n%s\nand " @@ -444,7 +445,7 @@ class QemuRunner: self.logger.debug("Waiting at most %d seconds for login banner (%s)" % (self.boottime, time.strftime("%D %H:%M:%S"))) endtime = time.time() + self.boottime - filelist = [self.server_socket, self.runqemu.stdout] + filelist = [self.server_socket] reachedlogin = False stopread = False qemusock = None @@ -564,7 +565,7 @@ class QemuRunner: self.logger.debug("Sending SIGKILL to runqemu") os.killpg(os.getpgid(self.runqemu.pid), signal.SIGKILL) if not self.runqemu.stdout.closed: - self.logger.info("Output from runqemu:\n%s" % self.getOutput(self.runqemu.stdout)) + self.logger.info("Output from runqemu:\n%s" % getOutput(self.runqemu.stdout)) self.runqemu.stdin.close() self.runqemu.stdout.close() self.runqemu_exited = True @@ -700,9 +701,11 @@ class QemuRunner: # event loop. The mechanism for stopping the thread is a simple pipe which # will wake up the poll and allow for tearing everything down. class LoggingThread(threading.Thread): - def __init__(self, logfunc, sock, logger): + def __init__(self, logfunc, sock, logger, qemuoutput): self.connection_established = threading.Event() + self.serversock = sock + self.qemuoutput = qemuoutput self.logfunc = logfunc self.logger = logger self.readsock = None @@ -732,7 +735,8 @@ class LoggingThread(threading.Thread): def teardown(self): self.logger.debug("Tearing down logging thread") - self.close_socket(self.serversock) + if self.serversock: + self.close_socket(self.serversock) if self.readsock is not None: self.close_socket(self.readsock) @@ -747,7 +751,9 @@ class LoggingThread(threading.Thread): def eventloop(self): poll = select.poll() event_read_mask = self.errorevents | self.readevents - poll.register(self.serversock.fileno()) + if self.serversock: + poll.register(self.serversock.fileno()) + poll.register(self.qemuoutput.fileno()) poll.register(self.readpipe, event_read_mask) breakout = False @@ -767,7 +773,7 @@ class LoggingThread(threading.Thread): break # A connection request was received - elif self.serversock.fileno() == event[0]: + elif self.serversock and self.serversock.fileno() == event[0]: self.logger.debug("Connection request received") self.readsock, _ = self.serversock.accept() self.readsock.setblocking(0) @@ -781,6 +787,10 @@ class LoggingThread(threading.Thread): elif self.readsock.fileno() == event[0]: data = self.recv(1024) self.logfunc(data) + elif self.qemuoutput.fileno() == event[0]: + data = self.qemuoutput.read() + self.logger.debug("Data received on qemu stdout %s" % data) + self.logfunc(data, ".stdout") # Since the socket is non-blocking make sure to honor EAGAIN # and EWOULDBLOCK. From patchwork Tue Dec 19 14:35:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 36666 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22131C46CD4 for ; Tue, 19 Dec 2023 14:35:40 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web10.14867.1702996536199470183 for ; Tue, 19 Dec 2023 06:35:36 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=HoFHO+8D; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-40d2376db79so7941705e9.0 for ; Tue, 19 Dec 2023 06:35:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1702996534; x=1703601334; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=KIU3sidFJlHRS+DYETettLZILKmVSXMjku4Rr5R6wyk=; b=HoFHO+8DdRh1ay1B/pZ7/jMI4P6hQAv0MYdXsIHCYmBTnI2Zx1Ju6o3E7YD+CGJqWp Qp1gRPS3eXX7C3B0myqqrqIFmY/1/hcRnATYL03nLw0O1Lf/sqtiGU+eSiA/keYHwGiX fwKbNFsSXDdwAithBKXxNQJo+ih/QpqAo5vpY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702996534; x=1703601334; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KIU3sidFJlHRS+DYETettLZILKmVSXMjku4Rr5R6wyk=; b=usFuvJnNN/sR3MNgXFgPPU0fdWTYrsVzPiPpU1PkO15zaQYWGcfQsvWC40G2zNigSY A8Nvg8owCi1tH8KbRP28d8xqbCzHQ0k1BF050/qYIbpa/4JEUfR+DA2gp47eIwUqG/kp A/mOIlIQZmAQtuMsLNYvk+Xwvum65JGHAkvkadani/C4wpJJj1zDKB2SRXD9fyVZBZur BJXqq0hVhzwUvpcuyeOZ4Grl6IjhGef8JA7Zks2dgggouFSTJ2L3j60pDMsX/vDMRfs/ hGpmfoIl/1lf3K3LyGxpB7AT7WasMRzpiA5YfnJb1piDMmXiChNqlh2JGHEWYZ+CAAV7 2WJA== X-Gm-Message-State: AOJu0YwHGFmgLWig5NcokPD0Z9S9iALG8iyfEwXccGHwheqsexVUXGE6 EiOk3SUqWvOVDdtlQpjOf97IcC7HrKqoGqq7wY0= X-Google-Smtp-Source: AGHT+IGQ6T9KENsCkcDRVwm/vh2PifUIukwWFQGILQvQ4Lr7TldM3UQvpzm/jo1pfQvNex/OLfPePw== X-Received: by 2002:a05:600c:1da9:b0:40d:247f:3266 with SMTP id p41-20020a05600c1da900b0040d247f3266mr638744wms.42.1702996534430; Tue, 19 Dec 2023 06:35:34 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b87d:537f:7c6:e419]) by smtp.gmail.com with ESMTPSA id m5-20020a05600c4f4500b0040c57e4ea28sm3168889wmq.17.2023.12.19.06.35.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 06:35:33 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/5] qemurunner: Improve handling of serial port output blocking Date: Tue, 19 Dec 2023 14:35:29 +0000 Message-Id: <20231219143531.3514266-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> References: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Dec 2023 14:35:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192742 Similar to stdout in the previous commit, we need to ensure serial output if written is read and put somewhere, else qemu might block on writes to the serial port leading to hangs in the kernel. Use our existing logging thread to log data when run_serial is not in use. Signed-off-by: Richard Purdie --- meta/lib/oeqa/utils/qemurunner.py | 98 ++++++++++++++++++++++--------- 1 file changed, 69 insertions(+), 29 deletions(-) diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py index cdd0db58776..4a2246733f8 100644 --- a/meta/lib/oeqa/utils/qemurunner.py +++ b/meta/lib/oeqa/utils/qemurunner.py @@ -21,7 +21,9 @@ import threading import codecs import tempfile from collections import defaultdict +from contextlib import contextmanager import importlib +import traceback # Get Unicode non printable control chars control_range = list(range(0,32))+list(range(127,160)) @@ -517,8 +519,12 @@ class QemuRunner: except Exception as e: self.logger.warning('Extra log data exception %s' % repr(e)) data = None + self.thread.serial_lock.release() return False + with self.thread.serial_lock: + self.thread.set_serialsock(self.server_socket) + # If we are not able to login the tests can continue try: (status, output) = self.run_serial(self.boot_patterns['send_login_user'], raw=True, timeout=120) @@ -653,31 +659,32 @@ class QemuRunner: data = '' status = 0 - self.server_socket.sendall(command.encode('utf-8')) - start = time.time() - end = start + timeout - while True: - now = time.time() - if now >= end: - data += "<<< run_serial(): command timed out after %d seconds without output >>>\r\n\r\n" % timeout - break - try: - sread, _, _ = select.select([self.server_socket],[],[], end - now) - except InterruptedError: - continue - if sread: - # try to avoid reading single character at a time - time.sleep(0.1) - answer = self.server_socket.recv(1024) - if answer: - data += answer.decode('utf-8') - # Search the prompt to stop - if re.search(self.boot_patterns['search_cmd_finished'], data): - break - else: - if self.canexit: - return (1, "") - raise Exception("No data on serial console socket, connection closed?") + with self.thread.serial_lock: + self.server_socket.sendall(command.encode('utf-8')) + start = time.time() + end = start + timeout + while True: + now = time.time() + if now >= end: + data += "<<< run_serial(): command timed out after %d seconds without output >>>\r\n\r\n" % timeout + break + try: + sread, _, _ = select.select([self.server_socket],[],[], end - now) + except InterruptedError: + continue + if sread: + # try to avoid reading single character at a time + time.sleep(0.1) + answer = self.server_socket.recv(1024) + if answer: + data += answer.decode('utf-8') + # Search the prompt to stop + if re.search(self.boot_patterns['search_cmd_finished'], data): + break + else: + if self.canexit: + return (1, "") + raise Exception("No data on serial console socket, connection closed?") if data: if raw: @@ -696,6 +703,15 @@ class QemuRunner: status = 1 return (status, str(data)) +@contextmanager +def nonblocking_lock(lock): + locked = lock.acquire(False) + try: + yield locked + finally: + if locked: + lock.release() + # This class is for reading data from a socket and passing it to logfunc # to be processed. It's completely event driven and has a straightforward # event loop. The mechanism for stopping the thread is a simple pipe which @@ -703,8 +719,10 @@ class QemuRunner: class LoggingThread(threading.Thread): def __init__(self, logfunc, sock, logger, qemuoutput): self.connection_established = threading.Event() + self.serial_lock = threading.Lock() self.serversock = sock + self.serialsock = None self.qemuoutput = qemuoutput self.logfunc = logfunc self.logger = logger @@ -717,9 +735,14 @@ class LoggingThread(threading.Thread): threading.Thread.__init__(self, target=self.threadtarget) + def set_serialsock(self, serialsock): + self.serialsock = serialsock + def threadtarget(self): try: self.eventloop() + except Exception as e: + self.logger.warning("Exception %s in logging thread" % traceback.format_exception(e)) finally: self.teardown() @@ -753,6 +776,7 @@ class LoggingThread(threading.Thread): event_read_mask = self.errorevents | self.readevents if self.serversock: poll.register(self.serversock.fileno()) + serial_registered = False poll.register(self.qemuoutput.fileno()) poll.register(self.readpipe, event_read_mask) @@ -760,7 +784,7 @@ class LoggingThread(threading.Thread): self.running = True self.logger.debug("Starting thread event loop") while not breakout: - events = poll.poll() + events = poll.poll(2) for event in events: # An error occurred, bail out if event[1] & self.errorevents: @@ -785,18 +809,34 @@ class LoggingThread(threading.Thread): # Actual data to be logged elif self.readsock.fileno() == event[0]: - data = self.recv(1024) + data = self.recv(1024, self.readsock) self.logfunc(data) elif self.qemuoutput.fileno() == event[0]: data = self.qemuoutput.read() self.logger.debug("Data received on qemu stdout %s" % data) self.logfunc(data, ".stdout") + elif self.serialsock and self.serialsock.fileno() == event[0]: + if self.serial_lock.acquire(blocking=False): + data = self.recv(1024, self.serialsock) + self.logger.debug("Data received serial thread %s" % data.decode('utf-8', 'replace')) + self.logfunc(data, ".2") + self.serial_lock.release() + else: + serial_registered = False + poll.unregister(self.serialsock.fileno()) + + if not serial_registered and self.serialsock: + with nonblocking_lock(self.serial_lock) as l: + if l: + serial_registered = True + poll.register(self.serialsock.fileno(), event_read_mask) + # Since the socket is non-blocking make sure to honor EAGAIN # and EWOULDBLOCK. - def recv(self, count): + def recv(self, count, sock): try: - data = self.readsock.recv(count) + data = sock.recv(count) except socket.error as e: if e.errno == errno.EAGAIN or e.errno == errno.EWOULDBLOCK: return b'' From patchwork Tue Dec 19 14:35:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 36664 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CE01C46CCD for ; Tue, 19 Dec 2023 14:35:40 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web11.15007.1702996536963126552 for ; Tue, 19 Dec 2023 06:35:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=cuBF/X9X; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-40c39e936b4so52992125e9.1 for ; Tue, 19 Dec 2023 06:35:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1702996535; x=1703601335; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=XjYSuiliGnom0jflKC4XX/meK062ruYw+E4D8oNNlRc=; b=cuBF/X9XkrBmNO7oauRiygnm1zVQhKyMZ+Wxr3paF92/HPWOhXUKdB/i5i0WeNJuip LFWKxk3QeG7TUbCe8BCFJMFUvbAW6liRAsQzsKszGE5ZTZCHwm7CfNzOSWW/FuqZiPTI 0rs5QGHk/FR3QgkdsCycdPVTCoEjP26LSOiaQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702996535; x=1703601335; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XjYSuiliGnom0jflKC4XX/meK062ruYw+E4D8oNNlRc=; b=Ca4IOkjWqJw58MHcHJDvigZCupt/eUJhAkP7+T19ttS0Vli+wK/XqAoodvamPxby9A 4CiHELMOWv2ySl7A99hokigJplYE5arBpdCe+HbQ4Ib4PCtHgyaubnhsB0vl+M12BZqj qeptX/9MI64rjRjUdehOgUL75yYFAmfCNHCZ5U2QK7t7eHjkTghTl05b/XJj6wWdWiRB XndLbRuV4Jqy+DZUx4pbepJLFuzYiSElCn4yY8m2vWqJaQ5FOZVTFMgy898/tO6IQtHH J8g5bqbvs/fsNF8pP/wmik6w0h1301gHHuNRu6935db05zVkrSCssfTjMiO6CL9euf4O tROQ== X-Gm-Message-State: AOJu0YxbhrD9cE5pyvs90tJuSM/UD+A+jovYm+Q1C5qpDvtE2cBaBy5r wIKLT22n0tQ25FzGfK/SxL2PEMZofczsTdEM3is= X-Google-Smtp-Source: AGHT+IHZB301jpGbP7A+BBiIkINJ/3i7CdzPQCfxZsYXu/2DZeK2C0w1J18XAFK5oyr+5iIET9T3/w== X-Received: by 2002:a05:600c:44c7:b0:40b:5f03:b3cc with SMTP id f7-20020a05600c44c700b0040b5f03b3ccmr4594554wmo.238.1702996535328; Tue, 19 Dec 2023 06:35:35 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b87d:537f:7c6:e419]) by smtp.gmail.com with ESMTPSA id m5-20020a05600c4f4500b0040c57e4ea28sm3168889wmq.17.2023.12.19.06.35.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 06:35:34 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 4/5] sstatesig: Ensure spdx dependencies update when abisafe recipes change Date: Tue, 19 Dec 2023 14:35:30 +0000 Message-Id: <20231219143531.3514266-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> References: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Dec 2023 14:35:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192743 If you build openssh, bump the version of base-passwd, then rerun openssh:do_create_spdx, it will fail due to references to the old version of base-passwd. This is due to base-passwd being listed in SIGGEN_EXCLUDERECIPES_ABISAFE. Ignore these recipes for the purposes of create_spdx and force rebuilds when things change. Signed-off-by: Richard Purdie --- meta/lib/oe/sstatesig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index 0342bcdc87a..1434690f1d1 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -63,7 +63,7 @@ def sstate_rundepfilter(siggen, fn, recipename, task, dep, depname, dataCaches): return False # Exclude well defined machine specific configurations which don't change ABI - if depname in siggen.abisaferecipes and not isImage(mc, fn): + if depname in siggen.abisaferecipes and not isImage(mc, fn) and deptaskname != "do_create_spdx": return False # Kernel modules are well namespaced. We don't want to depend on the kernel's checksum From patchwork Tue Dec 19 14:35:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 36665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A0F4C46CD3 for ; Tue, 19 Dec 2023 14:35:40 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web10.14868.1702996537549288786 for ; Tue, 19 Dec 2023 06:35:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=IFPuSqt6; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-40b5155e154so58299185e9.3 for ; Tue, 19 Dec 2023 06:35:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1702996536; x=1703601336; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=n7NKDTHaF8fv1LwwjSQ1JUQHVZB5W8i34Gd6edgJvjg=; b=IFPuSqt6Th9Jf3GP1r3H5CBSgmx5l2eZ+sCBFO/JKCyBaF+vKtP0rR71pkerfeMQaf lNc1+40fpBc9KsJUlIH/s4XRXe0LTNZwBcFEULqp6invvjpRlR32Dj1692LAJSSxZxDq lvdouqtTJZmlEx6CcXPcafhfb4GlsinaGaeRg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702996536; x=1703601336; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n7NKDTHaF8fv1LwwjSQ1JUQHVZB5W8i34Gd6edgJvjg=; b=tJUXqlGH+gd+2V6K2tkx9zeIV81SXwYiIupAqOS1p4NFWnFGDJG639UTpS+4mbu2Tk JE7LA5AmrNtN4CH2MbhbcuwDupEK2GmW8lUR/6qnS+JHOqaLQ0SiYdM2kORYzn3GTSTJ KVs3n8fi2886u+AsGBBnYKAk7MhJvK0VZmmDNK2zFF/aI8oJNSN2cT5JKNb5TmuPeocv bhbD8v6txXFlmG2OwW5qHkZj+nsUlVEiR1nYQmmVrGl7xKvnpUZG58/4lkIVjT1WlR3B xO1ikmxf/8esEgCe3SndOs7fqAGMDC2hCe3unXPQDXfoUrQf8wDNGfQfPPvIHofAFAwr vtUg== X-Gm-Message-State: AOJu0YwbV02Nort8hsI6LywVKf60GXR37cf3Ta4Her8cL633Ebk/1vSh FJfMmWDycfCYrs+pEL5YT/C1MvwKkneljG9T+o0= X-Google-Smtp-Source: AGHT+IFqzLcqPAk1tjQpPq/pNKQJgEqQAc3JZj+qDiI56IbtO3ySyBYOOqiCIesm4Q46YwrGlVvQvQ== X-Received: by 2002:a05:600c:1715:b0:40c:24b6:ba53 with SMTP id c21-20020a05600c171500b0040c24b6ba53mr8563949wmn.101.1702996535954; Tue, 19 Dec 2023 06:35:35 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b87d:537f:7c6:e419]) by smtp.gmail.com with ESMTPSA id m5-20020a05600c4f4500b0040c57e4ea28sm3168889wmq.17.2023.12.19.06.35.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 06:35:35 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 5/5] oeqa/selftest/overlayfs: Don't overwrite DISTRO_FEATURES Date: Tue, 19 Dec 2023 14:35:31 +0000 Message-Id: <20231219143531.3514266-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> References: <20231219143531.3514266-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 19 Dec 2023 14:35:40 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/192744 Whilst debugging an autobuilder failure, I wondered why it was rebuilding qemu-system-native instead of reusing from sstate. The reason was it was overwriting DISTRO_FEATURES, in this case removing opengl which caused much to rebuild. The test doesn't need that so don't do it. Signed-off-by: Richard Purdie --- meta/lib/oeqa/selftest/cases/overlayfs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py index 4031ded0fe4..a20fa6acc2d 100644 --- a/meta/lib/oeqa/selftest/cases/overlayfs.py +++ b/meta/lib/oeqa/selftest/cases/overlayfs.py @@ -79,7 +79,7 @@ DISTRO_FEATURES:append = " systemd overlayfs usrmerge" config = """ IMAGE_INSTALL:append = " overlayfs-user" -DISTRO_FEATURES += "systemd overlayfs usrmerge" +DISTRO_FEATURES:append "systemd overlayfs usrmerge" OVERLAYFS_QA_SKIP[mnt-overlay] = "mount-configured" """