From patchwork Sun Dec 17 13:30:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 36513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29C1EC46CCD for ; Sun, 17 Dec 2023 13:30:38 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web10.18342.1702819836712831041 for ; Sun, 17 Dec 2023 05:30:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=nbJL4fSC; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id D792C60004; Sun, 17 Dec 2023 13:30:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1702819835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oiGTRRSOBvfAh88IfwjahVNov3deCCDVvdBfJK4Y5lA=; b=nbJL4fSCQ5Zi4NTDkf74Cjm702k9nRzz7gZzeICH3qg42sEJrkXtxcoBtNZQljFPXuA/j+ GCmBEKqSL4QiHtcjUkwZvqPrUN8ArSUMZutKmer8gf4wTyqHY+tVgBKivZE+wzu1739KZy EgcKWxHMXR79z7VcxHHXZgQ/C7fRO1ECaSxEh0+mTPw/zVKT4um2/PfhfNVr0SnXq+6IUl MRYw/G2J7CXL+t37SW4coARHFmmm0k+Ss93lSTDryoEr77Fr+MFoiQvjt44WmB5Ko5p5Gf yZBnRfzHh3mAMlDdn9p2pzt5L6E+XpU3O8TpoyKEVrmyDLiIeMr6pk5dfRC4jg== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [yocto-autobuilder-helper][PATCH 1/4] scripts: send_qa_email: slightly rework previous tag test Date: Sun, 17 Dec 2023 14:30:29 +0100 Message-ID: <20231217133032.27231-2-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231217133032.27231-1-alexis.lothore@bootlin.com> References: <20231217133032.27231-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: alexis.lothore@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 17 Dec 2023 13:30:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61945 From: Alexis Lothoré Use intermediate variables for test input/output, and print expected output in subtest to get more info when a test fails Signed-off-by: Alexis Lothoré --- scripts/test_send_qa_email.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/test_send_qa_email.py b/scripts/test_send_qa_email.py index 5509b3c2510e..8ab91538748a 100755 --- a/scripts/test_send_qa_email.py +++ b/scripts/test_send_qa_email.py @@ -57,10 +57,10 @@ class TestVersion(unittest.TestCase): def test_versions(self): for data in self.test_data_get_version: - test_name = data["input"]["version"] - with self.subTest(f"Test {test_name} previous tag"): - self.assertEqual(send_qa_email.get_previous_tag(os.environ.get( - "POKY_PATH"), data["input"]["version"]), data["expected"]) + input_version = data["input"]["version"] + expected = data["expected"] + with self.subTest(f"Test {input_version} previous tag (expecting {expected})"): + self.assertEqual(send_qa_email.get_previous_tag(os.environ.get("POKY_PATH"), input_version), expected) def test_is_release_version(self): for data in self.test_data_is_release_version: From patchwork Sun Dec 17 13:30:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 36510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BEC7C46CD2 for ; Sun, 17 Dec 2023 13:30:38 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web11.18447.1702819837292832805 for ; Sun, 17 Dec 2023 05:30:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=dlDvvr/k; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 2576B60005; Sun, 17 Dec 2023 13:30:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1702819835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gTadV1/rNLo96VM5eP8TuAkE9LTlUAM+xcdebuX1mYI=; b=dlDvvr/kvRfyz802of25BpWdorJFDoxlc1wmatOq5419oHzwOU+6ViCr7kqPf7mMnFVH9x txkRNYpahk1p5lYjR2TTWVzEsPdqNa0RsNziYpzd82pIvY801II36ygwSb/LlNByYLoIdv kg4XAEGTFNv1s2HxCMRIn8jB04HdGz3yQ9HN/mKVZ+aTCn4vQt0cFv+fU3I5+fWt6OMXRO 3nW2dtttENM0mlZ0maBvUxJfgwFkSqF8rVLxvkQcdN/c3Ln7IGrTwhacDpAYbmHqFIUDB+ 4+kNMB2mqj7XRdrzc9Wt5bJU2lGcc7X7NhtJVch5v/VeFxDjAeBrVbfFdfoPkg== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [yocto-autobuilder-helper][PATCH 2/4] scripts: send_qa_email: allow testing against non fixed tags Date: Sun, 17 Dec 2023 14:30:30 +0100 Message-ID: <20231217133032.27231-3-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231217133032.27231-1-alexis.lothore@bootlin.com> References: <20231217133032.27231-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: alexis.lothore@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 17 Dec 2023 13:30:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61947 From: Alexis Lothoré send_qa_email currently deals with tags in a "reproducible" way: despite new versions being released on different branches, the computation of the "reference" version for a specific input version always remain the same. This behavior does not match perfectly real expectations: if at the some point we get version 4.3.1 as a comparison reference for a regression report, and 4.3.2 is released some time later, we want the next comparision to be done against 4.3.2. Start introducing this new behavior by allowing the tests to check returned version against regex patterns instead of static strings, so we can for example use wildcards on the "micro" version Signed-off-by: Alexis Lothoré --- scripts/test_send_qa_email.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/test_send_qa_email.py b/scripts/test_send_qa_email.py index 8ab91538748a..2dd946f32bf1 100755 --- a/scripts/test_send_qa_email.py +++ b/scripts/test_send_qa_email.py @@ -12,6 +12,7 @@ import sys import unittest import send_qa_email import logging +import re logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") log = logging.getLogger('send-qa-email') @@ -60,7 +61,8 @@ class TestVersion(unittest.TestCase): input_version = data["input"]["version"] expected = data["expected"] with self.subTest(f"Test {input_version} previous tag (expecting {expected})"): - self.assertEqual(send_qa_email.get_previous_tag(os.environ.get("POKY_PATH"), input_version), expected) + result = send_qa_email.get_previous_tag(os.environ.get("POKY_PATH"), input_version) + self.assertIsNotNone(re.match(data["expected"], result), msg=f"get_previous_tag returned {result}") def test_is_release_version(self): for data in self.test_data_is_release_version: From patchwork Sun Dec 17 13:30:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 36512 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DFECC46CD4 for ; Sun, 17 Dec 2023 13:30:38 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web10.18344.1702819837124965038 for ; Sun, 17 Dec 2023 05:30:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=HejUOAzo; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 5B47260007; Sun, 17 Dec 2023 13:30:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1702819835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o4pIJPFIEIFz+lxd/CfM1g4THMKSyRu9/w6yf8NDrws=; b=HejUOAzov1G2g1mfsN2vFLtJxUcmTFEaoMJFBK5I1Bs9TTo5tbOPWMajG+2nxRbDUMoPvy 8D9WqrYRrfu4S0ZwspnXGGdZ2LrBUb9wf8DuGGvdfy5XfOlEcUjTmsQ5s9lsEBEhvsqdEL 65llzwFEV8YmmUJNdACMobAdxJFA+JjHN8kKl9FOB1xs+tr6l3AHwBTsK6UJdldt/glfvQ UfaPRwWN0wG704HjXgWQzoY55me8MGwlpj9BJFE8q9OTkaLBv1W8HNyfEctLTlNGJFtMAh eoRNbM8VsY5NZQ85PIQFWL/TMS367x4ul3XAx4cQyAUyf6hddMOnAVmCecF/rw== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [yocto-autobuilder-helper][PATCH 3/4] scripts: send_qa_email: properly compute previous tag for new major release tag Date: Sun, 17 Dec 2023 14:30:31 +0100 Message-ID: <20231217133032.27231-4-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231217133032.27231-1-alexis.lothore@bootlin.com> References: <20231217133032.27231-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: alexis.lothore@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 17 Dec 2023 13:30:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61946 From: Alexis Lothoré send_qa_email currently fails when dealing with release version starting a new major, for example 5.0. This has been observed for example when trying to generate 5.0_M1.rc1 This specific versioning makes previous tag computation method fall through last branch which currently expects that the current release tag indeed exists (5.0_M1), which is true when checking regression reports a posteriori, but not in an autobuilder run (tag is added only when the release has been "validated") Fix tag computation for this case by getting previous release tag with git ls-remote, instead of relying on git describe with a possibly non-existing tag. While doing so, add a few tests about this specific case. Signed-off-by: Alexis Lothoré --- scripts/send_qa_email.py | 9 +++++++-- scripts/test_send_qa_email.py | 5 ++++- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/scripts/send_qa_email.py b/scripts/send_qa_email.py index 232f360621fe..8422e4937102 100755 --- a/scripts/send_qa_email.py +++ b/scripts/send_qa_email.py @@ -42,10 +42,15 @@ def get_previous_tag(targetrepodir, version): # Process first milestone or release if not first in major release elif compareversionminor > 0: previousversion = compareversion[:-1] + [compareversion[-1] - 1] - # Otherwise : format it as tag (which must exist) and search previous tag + # Otherwise : compare against last release on previous major. This can + # not be computed: we need to retrieve the exact tag from remote else: comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone) - return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip() + previous_major = compareversion[0] - 1 + tags_list = subprocess.check_output(["git", "ls-remote", "--refs", "-t", "origin", f"refs/tags/yocto-{previous_major}*"], cwd=targetrepodir).decode('utf-8').strip() + # Get last tag from list, pick only the tag part, and remove the + # "refs/tags/" part + return tags_list.splitlines()[-1].split()[1].split('/')[-1] return utils.get_tag_from_version(previousversion, previousmilestone) diff --git a/scripts/test_send_qa_email.py b/scripts/test_send_qa_email.py index 2dd946f32bf1..415d8cbbbccd 100755 --- a/scripts/test_send_qa_email.py +++ b/scripts/test_send_qa_email.py @@ -30,7 +30,10 @@ class TestVersion(unittest.TestCase): {"input": {"version": "4.1_M3.rc4"}, "expected": "4.1_M2"}, {"input": {"version": "4.1_M1.rc1"}, "expected": "yocto-4.0"}, {"input": {"version": "4.1_M1.rc4"}, "expected": "yocto-4.0"}, - {"input": {"version": "4.1.rc4"}, "expected": "yocto-4.0"} + {"input": {"version": "4.1.rc4"}, "expected": "yocto-4.0"}, + {"input": {"version": "yocto-5.0_M1.rc1"}, "expected": "yocto-4.3(\.\d)?"}, + {"input": {"version": "yocto-5.0_M1.rc2"}, "expected": "yocto-4.3(\.\d)?"}, + {"input": {"version": "yocto-5.0_M2.rc1"}, "expected": "5.0_M1"}, ] test_data_is_release_version = [ From patchwork Sun Dec 17 13:30:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 36511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3895CC46CD7 for ; Sun, 17 Dec 2023 13:30:38 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web11.18445.1702819837141836725 for ; Sun, 17 Dec 2023 05:30:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=QUp3o0GX; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 930E960008; Sun, 17 Dec 2023 13:30:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1702819835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PhAKbPcuPuJTatYuzclCX9WYCn1k56GVALGhqDcEzHY=; b=QUp3o0GX7aLjaYclnkHnzc4bWDbEy+ilCOqfoSLc3sKfKn1xPR+VqkkQw6BiwfHnrTtoEF OGQJu+AZv6J3eL7Qg3DfrB4JiUcgTED6NHJbBxz31aXdKIp11qBRMd/ZCIwoUeG8YbgoXf UvCLV6YZJTeDE/Xd1LB4pwWqLFLAoR3Wgpzwv6nLd6lnK7ZV9SQ9tv+MmBwEdjQlBWWahC Tq44+GIJ1/sDTXa1nK3m6XTTz0U3E32wK+eEgfE7dFfnnW+BTJJ1LLqCaGNTk/jkxsTMyd AbMm0rwbNyPIwqG+SqWC+NFY35uyEBLg06nG7igfl8szJrHXPwVDtnIiNfr7cw== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [yocto-autobuilder-helper][PATCH 4/4] scripts: send_qa_email: protect get_regression_base_and_target from exceptions Date: Sun, 17 Dec 2023 14:30:32 +0100 Message-ID: <20231217133032.27231-5-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231217133032.27231-1-alexis.lothore@bootlin.com> References: <20231217133032.27231-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: alexis.lothore@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 17 Dec 2023 13:30:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61949 From: Alexis Lothoré generate_regression_reports is currently protect in a try/catch block to prevent it from canceling QA email generation when encountering an issue, but get_regression_base_and_target is not. Make sure that get_regression_base_and_target can not prevent QA email from being generated by adding it to the try/catch block protecting send_qa_email. While doing so, make sure to preserve the exitcode variable to make sure that the step is still marked as fail in autobuilder to make sure the error does not go silent. However the variable is not needed as global anymore since it is now used in a single function. Signed-off-by: Alexis Lothoré --- scripts/send_qa_email.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/scripts/send_qa_email.py b/scripts/send_qa_email.py index 8422e4937102..fe612c7e114a 100755 --- a/scripts/send_qa_email.py +++ b/scripts/send_qa_email.py @@ -16,7 +16,6 @@ import logging import utils TEST_RESULTS_REPOSITORY_URL="git://git.yoctoproject.org/yocto-testresults" -exitcode = 0 def is_release_version(version): p = re.compile('\d{8}-\d+') @@ -89,21 +88,16 @@ def get_regression_base_and_target(targetbranch, basebranch, release, targetrepo def generate_regression_report(querytool, targetrepodir, base, target, resultdir, outputdir, log): log.info(f"Comparing {target} to {base}") - global exitcode - try: - regreport = subprocess.check_output([querytool, "regression-report", base, target, '-t', resultdir]) - with open(outputdir + "/testresult-regressions-report.txt", "wb") as f: - f.write(regreport) - except subprocess.CalledProcessError as e: - error = str(e) - exitcode = 1 - log.error(f"Error while generating report between {target} and {base} : {error}") + regreport = subprocess.check_output([querytool, "regression-report", base, target, '-t', resultdir]) + with open(outputdir + "/testresult-regressions-report.txt", "wb") as f: + f.write(regreport) def send_qa_email(): # Setup logging logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") log = logging.getLogger('send-qa-email') + exitcode = 0 parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.') @@ -195,9 +189,15 @@ def send_qa_email(): # log.warning("Test results not published on release version. Faulty AB configuration ?") utils.printheader("Processing regression report") - regression_base, regression_target = get_regression_base_and_target(targetbranch, basebranch, args.release, targetrepodir, log) - if regression_base and regression_target: + try: + regression_base, regression_target = get_regression_base_and_target(targetbranch, basebranch, args.release, targetrepodir, log) + log.info(f"Generating regression report between {regression_base} and {regression_target}") generate_regression_report(querytool, targetrepodir, regression_base, regression_target, tempdir, args.results_dir, log) + except subprocess.CalledProcessError as e: + error = str(e) + exitcode = 1 + log.error(f"Error while generating regression report: {error}") + finally: log.info(f"[SKIP] delete {tempdir}")