From patchwork Mon Oct 23 15:43:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 32794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 760ECC004C0 for ; Mon, 23 Oct 2023 15:44:02 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web11.124403.1698075839547107792 for ; Mon, 23 Oct 2023 08:43:59 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=HU4nOYel; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-40839652b97so26814355e9.3 for ; Mon, 23 Oct 2023 08:43:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1698075838; x=1698680638; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=7IipQmhl/lANcCOpifbVrQ/DJZOahGjuZ/dx2XgcBLg=; b=HU4nOYel/I+n/nH/PsB3XbVM7rmTzfQ0jSae5RCNr691g23bOYGcImMBt/iYmk8aCe /jXY6bSbAURMAy8fdC2vhkK9BWRAuHHv2WSeEfejpEp9jrdUHmUzWZwY5diP5+9e6RJJ 6dPmpULWEkMggHQtWIdVy1YIeq2dNaKpCFOMc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698075838; x=1698680638; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7IipQmhl/lANcCOpifbVrQ/DJZOahGjuZ/dx2XgcBLg=; b=W2mxCohvVkDB/1SINGfeIC4JlaAaInz+ileyvU1LYH4n3+bUfRTVdUMJkL/J6MSCyq dAWbaQ4kCD2Bz3HZY5tcdt83rqDwcpT5grgi5cZZlpVYwYn0PQyNiYdkNH7CijX84tgI O0VhEu7gHpfiHydJlkWyqmBDbIyummia0G8A8d5w0+IjlfQ1IfZuhbnsKvejdCV9v7me idOLoguxk+ve+RY8R6f563JTnyH7fEuc8X2rrkZWmUF/JWX124QeQvXTf6o4G5and3Ag 7ogHIa79PfLXmk607w2K8d3+DPH6pIMnHazGnmN9YAe76J1rA9uZgetF2JyJVeJNpKEF LUXQ== X-Gm-Message-State: AOJu0YyVrQLyh3Y3TdzYkTja9X5Q40d77ZQjv/jdKIkedCT2H72tEVJ3 F4vZAjn5kXanCXkdtYOh0ahAaJKmClQ+/w3/mwY= X-Google-Smtp-Source: AGHT+IHnx8IOiRkJe69qVW1g5o0Bi+7RHJiA+0JAZrXmaZS3eiM4TACUt8SZ4yarAuAKrHP/+RvM+A== X-Received: by 2002:a05:600c:4ece:b0:408:3ab3:a05e with SMTP id g14-20020a05600c4ece00b004083ab3a05emr7596175wmq.38.1698075837522; Mon, 23 Oct 2023 08:43:57 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3e73:a41c:6787:e5d4]) by smtp.gmail.com with ESMTPSA id ay20-20020a05600c1e1400b00407460234f9sm9745346wmb.21.2023.10.23.08.43.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 08:43:57 -0700 (PDT) From: Richard Purdie To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper] [PATCH 1/5] scripts/run-patchmetrics: Only clone metrics if it isn't present Date: Mon, 23 Oct 2023 16:43:52 +0100 Message-Id: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 Oct 2023 15:44:02 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61454 To prepare for splitting things up, only clone the metrics repo if it isn't present. Signed-off-by: Richard Purdie --- scripts/run-patchmetrics | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics index fc3f214..abe58c7 100755 --- a/scripts/run-patchmetrics +++ b/scripts/run-patchmetrics @@ -13,7 +13,9 @@ TIMESTAMP=`date +"%s"` # # Patch Metrics # -git clone ssh://git@push.yoctoproject.org/yocto-metrics $PARENTDIR/yocto-metrics +if [ ! -e $PARENTDIR/yocto-metrics ]; then + git clone ssh://git@push.yoctoproject.org/yocto-metrics $PARENTDIR/yocto-metrics +fi $OURDIR/patchmetrics-update --repo $PARENTDIR --patchscript $PARENTDIR/scripts/contrib/patchreview.py --metadata $TARGETDIR --json $PARENTDIR/yocto-metrics/patch-status.json git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new patch stats" git -C $PARENTDIR/yocto-metrics push From patchwork Mon Oct 23 15:43:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 32791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BE30C001E0 for ; Mon, 23 Oct 2023 15:44:02 +0000 (UTC) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by mx.groups.io with SMTP id smtpd.web11.124404.1698075840541954026 for ; Mon, 23 Oct 2023 08:44:00 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=IrAefcgN; spf=pass (domain: linuxfoundation.org, ip: 209.85.208.178, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-2c5087d19a6so51636271fa.0 for ; Mon, 23 Oct 2023 08:44:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1698075838; x=1698680638; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=NxDlk3wHQDomCTgubnlM4oCZLUiUpihGFwRS70wBE0Q=; b=IrAefcgNvwYEln7G/hi/TP7SrM4cRIuAMJXjA7dCBrwNK6NIOyHuirhebhem5alXHY 5d2UKNxOv5VT+VtkTounXS0AI46FoKh4VaovL42DSKClCS+7EPbg7GEe+Cjh2tigIVgx 6MKAmAIaH8Rb31UijOe2TVbVZyvAcoLzYLq8s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698075838; x=1698680638; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NxDlk3wHQDomCTgubnlM4oCZLUiUpihGFwRS70wBE0Q=; b=fcGm2ZsUr5aKoL781XrwdbTm64Oy5dUOHErqi2WNxJ418uLmo330F5DsAvBfJKPJbs Vve8JvsntmxQsCM6BV0HTflBA90y5P5MIZVlnUSbVzk3IahbD0wwKKU413ekDZjCvdBM xixU/orleA2t0F2k1Mf7asj4iZsKEL3/hG66BFgs16IvHZLDd53rJPlhIzH8KhnFAXzr TtrxVJyujVHCAtFYZItGixqfCsV9KKxIN0OAorhXPgRJckSb6zukJTwfTSDdnaN5Z9zg ZbKmLqxfrvoxieUwh+fVWeCAb7kKhTLIXLx06e9Iw7A4q8yW8CndkEPvU923bgqSqT27 fTLA== X-Gm-Message-State: AOJu0YxlnBbsqu0jTsoZNW8F8aPrH73tohpwnNK7nMqn2k3+sOkRBjnI HmnC2IhPHbdPJZgDUMJCJo00QF5MEwf3t+kVSOM= X-Google-Smtp-Source: AGHT+IFP3xoEiFjx/vzXtqfLl/+fh4fonB+C+5d//ANw63zBFrkgdBq2PMwnQny4oP8jVOgz3twdiw== X-Received: by 2002:a2e:a589:0:b0:2c5:1a8e:e4c9 with SMTP id m9-20020a2ea589000000b002c51a8ee4c9mr6926161ljp.31.1698075838170; Mon, 23 Oct 2023 08:43:58 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3e73:a41c:6787:e5d4]) by smtp.gmail.com with ESMTPSA id ay20-20020a05600c1e1400b00407460234f9sm9745346wmb.21.2023.10.23.08.43.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 08:43:57 -0700 (PDT) From: Richard Purdie To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper] [PATCH 2/5] scripts/run-patchmetrics: Split out CVE checks Date: Mon, 23 Oct 2023 16:43:53 +0100 Message-Id: <20231023154356.1145934-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> References: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 Oct 2023 15:44:02 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61455 Split the CVE checks from the patch metrics script Signed-off-by: Richard Purdie --- config.json | 6 +++++ scripts/run-cvecheck | 54 ++++++++++++++++++++++++++++++++++++++++ scripts/run-patchmetrics | 36 --------------------------- 3 files changed, 60 insertions(+), 36 deletions(-) create mode 100755 scripts/run-cvecheck diff --git a/config.json b/config.json index bebd999..f225148 100644 --- a/config.json +++ b/config.json @@ -1208,8 +1208,14 @@ "BB_SERVER_TIMEOUT = '0'" ], "step1" : { + "shortname" : "Generating patch metrics", "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-patchmetrics ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."] + }, + "step2" : { + "shortname" : "Running CVE checks", + "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-cvecheck ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."] } + }, "meta-mingw" : { "NEEDREPOS" : ["poky", "meta-mingw"], diff --git a/scripts/run-cvecheck b/scripts/run-cvecheck new file mode 100755 index 0000000..35c796b --- /dev/null +++ b/scripts/run-cvecheck @@ -0,0 +1,54 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +PARENTDIR=`realpath $1` +TARGETDIR=`realpath $2` +RESULTSDIR=`realpath -m $3` +BUILDDIR=`realpath $4` +OURDIR=`dirname $0` + +TIMESTAMP=`date +"%s"` + +# +# CVE Checks +# +if [ ! -e $PARENTDIR/yocto-metrics ]; then + git clone ssh://git@push.yoctoproject.org/yocto-metrics $PARENTDIR/yocto-metrics +fi + +if [ ! -d $RESULTSDIR ]; then + mkdir $RESULTSDIR +fi + +for branch in master mickledore langdale kirkstone dunfell; do + mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch/ + git -C $PARENTDIR reset origin/$branch --hard + rm conf/local.conf + rm conf/bblayers.conf + rm -f conf/templateconf.cfg + rm tmp/ -rf + unset BB_ENV_PASSTHROUGH_ADDITIONS + unset BB_ENV_EXTRAWHITE + cd .. + . oe-init-build-env build + bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc + if [ -e tmp/log/cve/cve-summary.json ]; then + git -C $PARENTDIR/yocto-metrics rm cve-check/$branch/*.json + mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch + cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$branch/$TIMESTAMP.json + git -C $PARENTDIR/yocto-metrics add cve-check/$branch/$TIMESTAMP.json + git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $branch" + git -C $PARENTDIR/yocto-metrics push + $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$branch.txt + fi +done + +mkdir -p $PARENTDIR/yocto-metrics/cve-check/ +$OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/ +git -C $PARENTDIR/yocto-metrics add cve-count-byday.json +git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts" +git -C $PARENTDIR/yocto-metrics push + +cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR +cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics index abe58c7..e45d463 100755 --- a/scripts/run-patchmetrics +++ b/scripts/run-patchmetrics @@ -27,39 +27,3 @@ fi $OURDIR/patchmetrics-generate-chartdata --json $PARENTDIR/yocto-metrics/patch-status.json --outputdir $RESULTSDIR cp $PARENTDIR/yocto-metrics/patch-status.json $RESULTSDIR cp $PARENTDIR/yocto-metrics/patch-status/* $RESULTSDIR - -# -# CVE Checks -# -for branch in master mickledore langdale kirkstone dunfell; do - mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch/ - git -C $PARENTDIR reset origin/$branch --hard - rm conf/local.conf - rm conf/bblayers.conf - rm -f conf/templateconf.cfg - rm tmp/ -rf - unset BB_ENV_PASSTHROUGH_ADDITIONS - unset BB_ENV_EXTRAWHITE - cd .. - . oe-init-build-env build - bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc - if [ -e tmp/log/cve/cve-summary.json ]; then - git -C $PARENTDIR/yocto-metrics rm cve-check/$branch/*.json - mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch - cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$branch/$TIMESTAMP.json - git -C $PARENTDIR/yocto-metrics add cve-check/$branch/$TIMESTAMP.json - git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $branch" - git -C $PARENTDIR/yocto-metrics push - $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$branch.txt - fi -done - -mkdir -p $PARENTDIR/yocto-metrics/cve-check/ -$OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/ -git -C $PARENTDIR/yocto-metrics add cve-count-byday.json -git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts" -git -C $PARENTDIR/yocto-metrics push - -cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR -cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR - From patchwork Mon Oct 23 15:43:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 32790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CC75C00A8F for ; Mon, 23 Oct 2023 15:44:02 +0000 (UTC) Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by mx.groups.io with SMTP id smtpd.web10.124082.1698075840770202248 for ; Mon, 23 Oct 2023 08:44:01 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=RgR8Trva; spf=pass (domain: linuxfoundation.org, ip: 209.85.208.182, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-lj1-f182.google.com with SMTP id 38308e7fff4ca-2c523ac38fbso50536041fa.0 for ; Mon, 23 Oct 2023 08:44:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1698075839; x=1698680639; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=gRjf5QWe/Ur5Bq84ijgiC/Vw1F1jwVBwTYwBn64gbWk=; b=RgR8TrvaGitrRHvWL14faps0SaCJ4SRWNRGMarlgs8DcY+T+kRtS88NbEeW8JeOK5e 4CitPQAK/XOeiY6THCLTqBAEmKZtezOFnIaZ9GDWcQkG083LWqDiPTsnV2ukX/k4G2xv xgZdvTuVnkFxl/NFsWjGXMXKNP5TPrIyR31sw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698075839; x=1698680639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gRjf5QWe/Ur5Bq84ijgiC/Vw1F1jwVBwTYwBn64gbWk=; b=SbQClargbvypz+WaDE7a5dMFrgiJEmd0LS5fhQnlSKJFaUCfMb9/S2TZAtXHl+ZqvD 2IyJv8VcrOPvigQuCeRIBZNQMPxmjSrJaLYFAKFrrX3JSCwl7uQ7P8sgLBHyDfVUGGzm itqEtzKKSOq4AZ+4MIb17WzplATOxgb2TqF0BLxKHDJfmuoOWSrmGavUEHKaooBU9a/c 5SI6yWWW/cdfaTQhZqBQBi2cPHeugLrZzU7syoe4mI2ihy9asUh0hc0rq8lkhhcGjPVL YdK6tCqbk4fG83PQR3X+MX91XRnx9eqol1Kvv/tI5VfC7NSsyUUAZpSbLCq1nuiR0kqV yAvQ== X-Gm-Message-State: AOJu0Yyg+24iLsD5dGLNzt6th2YIMpEsYJ6DniVi0CUzYRJm7KdOSmdH Azs3IOCZSN5gYLlcBxsHh8pXLaHmq5DsF9RtqJs= X-Google-Smtp-Source: AGHT+IGp0kpfZ/9Ip4UwJAEXvkmss1yoCWJ84Q0B1ioMaWUiig20vnTp96bmorIqRRX++opsZ/kv3A== X-Received: by 2002:a2e:7a04:0:b0:2c5:106f:5416 with SMTP id v4-20020a2e7a04000000b002c5106f5416mr6143359ljc.53.1698075838863; Mon, 23 Oct 2023 08:43:58 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3e73:a41c:6787:e5d4]) by smtp.gmail.com with ESMTPSA id ay20-20020a05600c1e1400b00407460234f9sm9745346wmb.21.2023.10.23.08.43.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 08:43:58 -0700 (PDT) From: Richard Purdie To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper] [PATCH 3/5] metrics: Pass branchname to scripts Date: Mon, 23 Oct 2023 16:43:54 +0100 Message-Id: <20231023154356.1145934-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> References: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 Oct 2023 15:44:02 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61456 To prepapre to run the scripts per branch, pass the branchname to the scripts. Signed-off-by: Richard Purdie --- config.json | 4 ++-- scripts/run-cvecheck | 1 + scripts/run-patchmetrics | 1 + 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/config.json b/config.json index f225148..0c35632 100644 --- a/config.json +++ b/config.json @@ -1209,11 +1209,11 @@ ], "step1" : { "shortname" : "Generating patch metrics", - "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-patchmetrics ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."] + "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-patchmetrics ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics . ${HELPERBRANCHNAME}"] }, "step2" : { "shortname" : "Running CVE checks", - "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-cvecheck ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."] + "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-cvecheck ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics . ${HELPERBRANCHNAME}"] } }, diff --git a/scripts/run-cvecheck b/scripts/run-cvecheck index 35c796b..d48fd68 100755 --- a/scripts/run-cvecheck +++ b/scripts/run-cvecheck @@ -6,6 +6,7 @@ PARENTDIR=`realpath $1` TARGETDIR=`realpath $2` RESULTSDIR=`realpath -m $3` BUILDDIR=`realpath $4` +BRANCH=$5 OURDIR=`dirname $0` TIMESTAMP=`date +"%s"` diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics index e45d463..20e6268 100755 --- a/scripts/run-patchmetrics +++ b/scripts/run-patchmetrics @@ -6,6 +6,7 @@ PARENTDIR=`realpath $1` TARGETDIR=`realpath $2` RESULTSDIR=`realpath -m $3` BUILDDIR=`realpath $4` +BRANCH=$5 OURDIR=`dirname $0` TIMESTAMP=`date +"%s"` From patchwork Mon Oct 23 15:43:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 32793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CCF7C25B41 for ; Mon, 23 Oct 2023 15:44:02 +0000 (UTC) Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) by mx.groups.io with SMTP id smtpd.web10.124084.1698075841447650211 for ; Mon, 23 Oct 2023 08:44:01 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=bg3rPx2f; spf=pass (domain: linuxfoundation.org, ip: 209.85.208.175, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-lj1-f175.google.com with SMTP id 38308e7fff4ca-2c50ec238aeso49641601fa.0 for ; Mon, 23 Oct 2023 08:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1698075839; x=1698680639; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=GTezZEBIQK/UFsgRRWaYTh3VuiUf3MbFc0xHWaRqlKg=; b=bg3rPx2f1KhaXiRJRfztNd18JRTr06iiVccykjBSUpHkZvn2FYNOcTHswzoGPtbTIg N9/IpkxmDvRXaVoVN1r4zrMqtseMI8+u6VYDWBK/5PIYTmY5pZa/H+Htf//HtvsBIg1H Q4JoATge3RwOMMWdvTADJ7rJGq7gMvoigHvQE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698075839; x=1698680639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GTezZEBIQK/UFsgRRWaYTh3VuiUf3MbFc0xHWaRqlKg=; b=kR5Vf95OpCLLKQ7HSWM7rncLJimf8c2w8qE/X5rJBdRLV7IVl8D7T+zYVCgKpB85V7 ZaZtCDdhV62tOe6V2g2JcLZJ2RO9qzKnl5wCaOq/WXyqn8kUzb4gHx/PDwJJuXjhXq3Z JlX3lBVm6NRiej0rNpwHwjEsw0aXyTXhPRWFJ22tzjJvnPiZ0jOeiQ1mWybS+3psO3xq svxRmOIvoNwl5of6la3r3BMn94qe7nR2MhYr8RFoVO6J/S5KtufBiKlRL9cSjjrVTXks nnq8pWJtP67KEq+gwjjSP9Ae2qK/rUGLsJy2XqIZmoScfd3eENeAnrGKjxarCgO6UpAB VUoQ== X-Gm-Message-State: AOJu0YxAmIh2F53Y59rL1cQHpg8hMQjpLE7VfYiDxwOh7NhOXdcfLrEy mpM5IjkDVHUihCb5RjETDNu6HVNzMIueXCu8Fyw= X-Google-Smtp-Source: AGHT+IGFqXNKEXIXHZL2qVDpV6b9VK1Bw4WjvFos+JAULO/4eEX1MJvAK2HeMbz4DK/J3DLduaU06A== X-Received: by 2002:a2e:a796:0:b0:2c5:1388:e36a with SMTP id c22-20020a2ea796000000b002c51388e36amr8884703ljf.29.1698075839551; Mon, 23 Oct 2023 08:43:59 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3e73:a41c:6787:e5d4]) by smtp.gmail.com with ESMTPSA id ay20-20020a05600c1e1400b00407460234f9sm9745346wmb.21.2023.10.23.08.43.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 08:43:59 -0700 (PDT) From: Richard Purdie To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper] [PATCH 4/5] scripts/run-patchmetrics: Only monitor master branch Date: Mon, 23 Oct 2023 16:43:55 +0100 Message-Id: <20231023154356.1145934-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> References: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 Oct 2023 15:44:02 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61457 We only monitor the master branch for patch metrics as we can't really make improvements to release branches. Signed-off-by: Richard Purdie --- scripts/run-patchmetrics | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics index 20e6268..391ac45 100755 --- a/scripts/run-patchmetrics +++ b/scripts/run-patchmetrics @@ -11,6 +11,11 @@ OURDIR=`dirname $0` TIMESTAMP=`date +"%s"` +# We only monitor patch metrics on the master branch +if [ "$BRANCH" != "master" ]; then + exit 0 +fi + # # Patch Metrics # From patchwork Mon Oct 23 15:43:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 32792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90DE7C25B45 for ; Mon, 23 Oct 2023 15:44:02 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx.groups.io with SMTP id smtpd.web11.124408.1698075842038023223 for ; Mon, 23 Oct 2023 08:44:02 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=GV5TZt/e; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-4083dbc43cfso25347445e9.3 for ; Mon, 23 Oct 2023 08:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1698075840; x=1698680640; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=YVhpovKu+j/bmDatsNStMwQ5fkAJ+Oq4LlobgizqHiA=; b=GV5TZt/eUdLVrjqR6HAz2BfBk36UjxeeA61XwI4HW++8nWC2i9CvHlBEc3hvG+/G4f 0NQNhqIlC+ysJpt9E98F3QxCMqYc7PfUfMfQ1OhhuffqQRT2eTgLyQ4pmkoaunrS9cQV ai/Ia2BMep4IEfv7MFLnLu8RX6vVee+gmUd2c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698075840; x=1698680640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YVhpovKu+j/bmDatsNStMwQ5fkAJ+Oq4LlobgizqHiA=; b=qAzPSczKnGHLe5iyP6v3gazwd3mQ9Sgzxw4iHcjZz+QK0C3OEt6mt/Ozm/djhikoZd 8VSk7g5kPjatmPt3nn/IbPI0aoqPj7z4VZEW/n2paCe0/p88WsTd969YkTsGK6+CPQrj zDf8CJqHyo7KeoGTZmkvvvOZY7Qf8DwP24a3Uqel2ayC9W351ANNP5B9ZBrkndjH5aHM Uk2k4PxwacEr/swlEN3gfTQCXu/VNkgsJeBa8va1P9UMRzltcqbMBAAm7asQqir53LDp G4CVFnGBhD56FrpZVgR3Wldi3bGkwdHVatjuoO0K6HMjXxjbtbJ57gJOKG9jltuS1qPk 4okQ== X-Gm-Message-State: AOJu0YxXfqixaIJCWYHtUb10qPjyZsjQ4GR/a5y9bPHPLL0MH7el0HDL x7X/x6tbb6sJFDhMF5iSpWwyGj68TlssTmMIG3Y= X-Google-Smtp-Source: AGHT+IEgCGuTKB/zXauaednDgWwqBYe4L8+suDev1VA8+mGlrg44Ex22/WdFdZOLPelPO47ZxrVtwA== X-Received: by 2002:a05:600c:4fc7:b0:407:5b54:bb10 with SMTP id o7-20020a05600c4fc700b004075b54bb10mr7725745wmq.8.1698075840334; Mon, 23 Oct 2023 08:44:00 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3e73:a41c:6787:e5d4]) by smtp.gmail.com with ESMTPSA id ay20-20020a05600c1e1400b00407460234f9sm9745346wmb.21.2023.10.23.08.43.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 08:43:59 -0700 (PDT) From: Richard Purdie To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper] [PATCH 5/5] scripts/run-cvecheck: Remove branch iteration Date: Mon, 23 Oct 2023 16:43:56 +0100 Message-Id: <20231023154356.1145934-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> References: <20231023154356.1145934-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 23 Oct 2023 15:44:02 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/61458 Rather than running multiple checkouts, lets move this to the autobuilder to handle and have it trigger the builds with the right checkouts. Signed-off-by: Richard Purdie --- scripts/run-cvecheck | 52 ++++++++++++++++++++------------------------ 1 file changed, 23 insertions(+), 29 deletions(-) diff --git a/scripts/run-cvecheck b/scripts/run-cvecheck index d48fd68..6294fe6 100755 --- a/scripts/run-cvecheck +++ b/scripts/run-cvecheck @@ -22,34 +22,28 @@ if [ ! -d $RESULTSDIR ]; then mkdir $RESULTSDIR fi -for branch in master mickledore langdale kirkstone dunfell; do - mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch/ - git -C $PARENTDIR reset origin/$branch --hard - rm conf/local.conf - rm conf/bblayers.conf - rm -f conf/templateconf.cfg - rm tmp/ -rf - unset BB_ENV_PASSTHROUGH_ADDITIONS - unset BB_ENV_EXTRAWHITE - cd .. - . oe-init-build-env build - bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc - if [ -e tmp/log/cve/cve-summary.json ]; then - git -C $PARENTDIR/yocto-metrics rm cve-check/$branch/*.json - mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch - cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$branch/$TIMESTAMP.json - git -C $PARENTDIR/yocto-metrics add cve-check/$branch/$TIMESTAMP.json - git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $branch" - git -C $PARENTDIR/yocto-metrics push - $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$branch.txt - fi -done +mkdir -p $PARENTDIR/yocto-metrics/cve-check/$BRANCH/ +cd .. +. oe-init-build-env build +bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc +if [ -e tmp/log/cve/cve-summary.json ]; then + git -C $PARENTDIR/yocto-metrics rm cve-check/$BRANCH/*.json + mkdir -p $PARENTDIR/yocto-metrics/cve-check/$BRANCH + cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$BRANCH/$TIMESTAMP.json + git -C $PARENTDIR/yocto-metrics add cve-check/$BRANCH/$TIMESTAMP.json + git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $BRANCH" + git -C $PARENTDIR/yocto-metrics push + $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$BRANCH.txt +fi + +if [ "$BRANCH" = "master" ]; then + mkdir -p $PARENTDIR/yocto-metrics/cve-check/ + $OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/ + git -C $PARENTDIR/yocto-metrics add cve-count-byday.json + git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts" + git -C $PARENTDIR/yocto-metrics push -mkdir -p $PARENTDIR/yocto-metrics/cve-check/ -$OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/ -git -C $PARENTDIR/yocto-metrics add cve-count-byday.json -git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts" -git -C $PARENTDIR/yocto-metrics push + cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR + cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR +fi -cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR -cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR