From patchwork Thu Apr 28 21:46:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 7330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E69AC433FE for ; Thu, 28 Apr 2022 21:47:21 +0000 (UTC) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by mx.groups.io with SMTP id smtpd.web11.2996.1651182434300789118 for ; Thu, 28 Apr 2022 14:47:14 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=1BjfsS0b; spf=softfail (domain: sakoman.com, ip: 209.85.214.181, mailfrom: steve@sakoman.com) Received: by mail-pl1-f181.google.com with SMTP id n18so5499452plg.5 for ; Thu, 28 Apr 2022 14:47:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=um4bSmZRomlxzWEvQK94NHv2rSO4nhVf67uZxVO1g3s=; b=1BjfsS0bSSoutXo4LmxUzM6UoXDpl5iomKxkOoTvtbyTmu7UFkKmDFRHfW/aIpGD9y 0hyZ5xTWwezu/aN8x8iGKGjxBbGtwCrHqlH58NjO8p5a2rgLezwbgp3OOz4RxlSPm/Nv gpH6lSSuTXpZKuTHShy+fbRUhrJcptIhgVDy+ruaILShlGktHsxWFowVRolKD39BGeKg DcBOCR7d0IC5XI4/Meeb7qcCBH77rKulfQTDFxEXn6CpiEy6dgDOx8zBZrYQRu82p3ui YVFS4T7iDnE9sbd4symClLm8n1p4XoYHBYy5PeK67Vs7InAhh5W9VZMFLFURKCNZOJdk 7p+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=um4bSmZRomlxzWEvQK94NHv2rSO4nhVf67uZxVO1g3s=; b=dcagxjmu0vpgAYy6J1Qa6MJIt3DYN8Sb+0laWVdP8jYcwH6WXcea+UsojMJxIg6aKj hJ0DGyOHx2myrXUdtfi97ZdpPhUx62lWN4hDfj5noBFvuM7I7CScOYJze4owg9sRsu3U FHMNsavcnJ5Q8J8zY/9ZQC2000iawgbGA915SqHZppWb6k+rfURpayZ/bTeqlbWtbhMv /psHQ6w9waS+YPOUPqA1J3PwQU8BkIbhFHqLG7j5tA/F2BNaDXb5PUDHyNMzCHAmfNfg KrK+pWd5rGc+Spw9EqJqW0p8rs9Hx9QKHKxzjhDzKF2AfsBS0MdqvU6e4sDRIABE3Kqm H3PA== X-Gm-Message-State: AOAM5301V6o8N7tz57/uwEhWaoxwhe1kL9ky+jiCSZrfwLGtLq2ZZW7+ LaEwSDw+kH+zRr+zOggqdVVNbkHkl1jDFVhWUNE= X-Google-Smtp-Source: ABdhPJy4M7DKcaNFOXVg8yXBsuV6Cuab7NJ6HvMB7nw4aw60wNBo3upPQeZdP21t8Ud7tqEPyfEcmw== X-Received: by 2002:a17:903:1205:b0:151:8ae9:93ea with SMTP id l5-20020a170903120500b001518ae993eamr35540408plh.37.1651182432906; Thu, 28 Apr 2022 14:47:12 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id c138-20020a624e90000000b005081f92826dsm749742pfb.99.2022.04.28.14.47.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 14:47:12 -0700 (PDT) From: Steve Sakoman To: openembedded-core@lists.openembedded.org Subject: [OE-core][dunfell 1/6] tiff: Fix CVE-2022-0891 Date: Thu, 28 Apr 2022 11:46:31 -1000 Message-Id: <512a8b30c816d2c9d85af7d7a1850b0450f1b6f4.1651182153.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 28 Apr 2022 21:47:21 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164979 From: sana kazi Fix CVE-2022-0891 for tiff Link: https://sources.debian.org/src/tiff/4.1.0+git191117-2%7Edeb10u4/debian/patches/CVE-2022-0891.patch/ Signed-off-by: Sana Kazi Signed-off-by: Sana Kazi Signed-off-by: Steve Sakoman --- .../libtiff/files/CVE-2022-0891.patch | 217 ++++++++++++++++++ meta/recipes-multimedia/libtiff/tiff_4.1.0.bb | 1 + 2 files changed, 218 insertions(+) create mode 100644 meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch diff --git a/meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch b/meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch new file mode 100644 index 0000000000..e2f1bd3056 --- /dev/null +++ b/meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch @@ -0,0 +1,217 @@ +From 232282fd8f9c21eefe8d2d2b96cdbbb172fe7b7c Mon Sep 17 00:00:00 2001 +From: Su Laus +Date: Tue, 8 Mar 2022 17:02:44 +0000 +Subject: [PATCH] tiffcrop: fix issue #380 and #382 heap buffer overflow in + extractImageSection + +CVE: CVE-2022-0891 +Upstream-Status: Backport [https://sources.debian.org/src/tiff/4.1.0+git191117-2%7Edeb10u4/debian/patches/CVE-2022-0891.patch/] +Comment: No change in any hunk +Signed-off-by: Sana Kazi +--- + tools/tiffcrop.c | 92 +++++++++++++++++++----------------------------- + 1 file changed, 36 insertions(+), 56 deletions(-) + +diff --git a/tools/tiffcrop.c b/tools/tiffcrop.c +index f2e5474a..e62bcc71 100644 +--- a/tools/tiffcrop.c ++++ b/tools/tiffcrop.c +@@ -105,8 +105,8 @@ + * of messages to monitor progess without enabling dump logs. + */ + +-static char tiffcrop_version_id[] = "2.4"; +-static char tiffcrop_rev_date[] = "12-13-2010"; ++static char tiffcrop_version_id[] = "2.4.1"; ++static char tiffcrop_rev_date[] = "03-03-2010"; + + #include "tif_config.h" + #include "tiffiop.h" +@@ -6670,10 +6670,10 @@ extractImageSection(struct image_data *image, struct pageseg *section, + #ifdef DEVELMODE + uint32 img_length; + #endif +- uint32 j, shift1, shift2, trailing_bits; ++ uint32 j, shift1, trailing_bits; + uint32 row, first_row, last_row, first_col, last_col; + uint32 src_offset, dst_offset, row_offset, col_offset; +- uint32 offset1, offset2, full_bytes; ++ uint32 offset1, full_bytes; + uint32 sect_width; + #ifdef DEVELMODE + uint32 sect_length; +@@ -6683,7 +6683,6 @@ extractImageSection(struct image_data *image, struct pageseg *section, + #ifdef DEVELMODE + int k; + unsigned char bitset; +- static char *bitarray = NULL; + #endif + + img_width = image->width; +@@ -6701,17 +6700,12 @@ extractImageSection(struct image_data *image, struct pageseg *section, + dst_offset = 0; + + #ifdef DEVELMODE +- if (bitarray == NULL) +- { +- if ((bitarray = (char *)malloc(img_width)) == NULL) +- { +- TIFFError ("", "DEBUG: Unable to allocate debugging bitarray"); +- return (-1); +- } +- } ++ char bitarray[39]; + #endif + +- /* rows, columns, width, length are expressed in pixels */ ++ /* rows, columns, width, length are expressed in pixels ++ * first_row, last_row, .. are index into image array starting at 0 to width-1, ++ * last_col shall be also extracted. */ + first_row = section->y1; + last_row = section->y2; + first_col = section->x1; +@@ -6721,9 +6715,14 @@ extractImageSection(struct image_data *image, struct pageseg *section, + #ifdef DEVELMODE + sect_length = last_row - first_row + 1; + #endif +- img_rowsize = ((img_width * bps + 7) / 8) * spp; +- full_bytes = (sect_width * spp * bps) / 8; /* number of COMPLETE bytes per row in section */ +- trailing_bits = (sect_width * bps) % 8; ++ /* The read function loadImage() used copy separate plane data into a buffer as interleaved ++ * samples rather than separate planes so the same logic works to extract regions ++ * regardless of the way the data are organized in the input file. ++ * Furthermore, bytes and bits are arranged in buffer according to COMPRESSION=1 and FILLORDER=1 ++ */ ++ img_rowsize = (((img_width * spp * bps) + 7) / 8); /* row size in full bytes of source image */ ++ full_bytes = (sect_width * spp * bps) / 8; /* number of COMPLETE bytes per row in section */ ++ trailing_bits = (sect_width * spp * bps) % 8; /* trailing bits within the last byte of destination buffer */ + + #ifdef DEVELMODE + TIFFError ("", "First row: %d, last row: %d, First col: %d, last col: %d\n", +@@ -6736,10 +6735,9 @@ extractImageSection(struct image_data *image, struct pageseg *section, + + if ((bps % 8) == 0) + { +- col_offset = first_col * spp * bps / 8; ++ col_offset = (first_col * spp * bps) / 8; + for (row = first_row; row <= last_row; row++) + { +- /* row_offset = row * img_width * spp * bps / 8; */ + row_offset = row * img_rowsize; + src_offset = row_offset + col_offset; + +@@ -6752,14 +6750,12 @@ extractImageSection(struct image_data *image, struct pageseg *section, + } + else + { /* bps != 8 */ +- shift1 = spp * ((first_col * bps) % 8); +- shift2 = spp * ((last_col * bps) % 8); ++ shift1 = ((first_col * spp * bps) % 8); /* shift1 = bits to skip in the first byte of source buffer*/ + for (row = first_row; row <= last_row; row++) + { + /* pull out the first byte */ + row_offset = row * img_rowsize; +- offset1 = row_offset + (first_col * bps / 8); +- offset2 = row_offset + (last_col * bps / 8); ++ offset1 = row_offset + ((first_col * spp * bps) / 8); /* offset1 = offset into source of byte with first bits to be extracted */ + + #ifdef DEVELMODE + for (j = 0, k = 7; j < 8; j++, k--) +@@ -6771,12 +6767,12 @@ extractImageSection(struct image_data *image, struct pageseg *section, + sprintf(&bitarray[9], " "); + for (j = 10, k = 7; j < 18; j++, k--) + { +- bitset = *(src_buff + offset2) & (((unsigned char)1 << k)) ? 1 : 0; ++ bitset = *(src_buff + offset1 + full_bytes) & (((unsigned char)1 << k)) ? 1 : 0; + sprintf(&bitarray[j], (bitset) ? "1" : "0"); + } + bitarray[18] = '\0'; +- TIFFError ("", "Row: %3d Offset1: %d, Shift1: %d, Offset2: %d, Shift2: %d\n", +- row, offset1, shift1, offset2, shift2); ++ TIFFError ("", "Row: %3d Offset1: %"PRIu32", Shift1: %"PRIu32", Offset2: %"PRIu32", Trailing_bits: %"PRIu32"\n", ++ row, offset1, shift1, offset1+full_bytes, trailing_bits); + #endif + + bytebuff1 = bytebuff2 = 0; +@@ -6800,11 +6796,12 @@ extractImageSection(struct image_data *image, struct pageseg *section, + + if (trailing_bits != 0) + { +- bytebuff2 = src_buff[offset2] & ((unsigned char)255 << (7 - shift2)); ++ /* Only copy higher bits of samples and mask lower bits of not wanted column samples to zero */ ++ bytebuff2 = src_buff[offset1 + full_bytes] & ((unsigned char)255 << (8 - trailing_bits)); + sect_buff[dst_offset] = bytebuff2; + #ifdef DEVELMODE + TIFFError ("", " Trailing bits src offset: %8d, Dst offset: %8d\n", +- offset2, dst_offset); ++ offset1 + full_bytes, dst_offset); + for (j = 30, k = 7; j < 38; j++, k--) + { + bitset = *(sect_buff + dst_offset) & (((unsigned char)1 << k)) ? 1 : 0; +@@ -6823,8 +6820,10 @@ extractImageSection(struct image_data *image, struct pageseg *section, + #endif + for (j = 0; j <= full_bytes; j++) + { +- bytebuff1 = src_buff[offset1 + j] & ((unsigned char)255 >> shift1); +- bytebuff2 = src_buff[offset1 + j + 1] & ((unsigned char)255 << (7 - shift1)); ++ /* Skip the first shift1 bits and shift the source up by shift1 bits before save to destination.*/ ++ /* Attention: src_buff size needs to be some bytes larger than image size, because could read behind image here. */ ++ bytebuff1 = src_buff[offset1 + j] & ((unsigned char)255 >> shift1); ++ bytebuff2 = src_buff[offset1 + j + 1] & ((unsigned char)255 << (8 - shift1)); + sect_buff[dst_offset + j] = (bytebuff1 << shift1) | (bytebuff2 >> (8 - shift1)); + } + #ifdef DEVELMODE +@@ -6840,36 +6839,17 @@ extractImageSection(struct image_data *image, struct pageseg *section, + #endif + dst_offset += full_bytes; + ++ /* Copy the trailing_bits for the last byte in the destination buffer. ++ Could come from one ore two bytes of the source buffer. */ + if (trailing_bits != 0) + { + #ifdef DEVELMODE +- TIFFError ("", " Trailing bits src offset: %8d, Dst offset: %8d\n", offset1 + full_bytes, dst_offset); +-#endif +- if (shift2 > shift1) +- { +- bytebuff1 = src_buff[offset1 + full_bytes] & ((unsigned char)255 << (7 - shift2)); +- bytebuff2 = bytebuff1 & ((unsigned char)255 << shift1); +- sect_buff[dst_offset] = bytebuff2; +-#ifdef DEVELMODE +- TIFFError ("", " Shift2 > Shift1\n"); ++ TIFFError("", " Trailing bits %4"PRIu32" src offset: %8"PRIu32", Dst offset: %8"PRIu32"\n", trailing_bits, offset1 + full_bytes, dst_offset); + #endif ++ /* More than necessary bits are already copied into last destination buffer, ++ * only masking of last byte in destination buffer is necessary.*/ ++ sect_buff[dst_offset] &= ((uint8_t)0xFF << (8 - trailing_bits)); + } +- else +- { +- if (shift2 < shift1) +- { +- bytebuff2 = ((unsigned char)255 << (shift1 - shift2 - 1)); +- sect_buff[dst_offset] &= bytebuff2; +-#ifdef DEVELMODE +- TIFFError ("", " Shift2 < Shift1\n"); +-#endif +- } +-#ifdef DEVELMODE +- else +- TIFFError ("", " Shift2 == Shift1\n"); +-#endif +- } +- } + #ifdef DEVELMODE + sprintf(&bitarray[28], " "); + sprintf(&bitarray[29], " "); +@@ -7022,7 +7002,7 @@ writeImageSections(TIFF *in, TIFF *out, struct image_data *image, + width = sections[i].x2 - sections[i].x1 + 1; + length = sections[i].y2 - sections[i].y1 + 1; + sectsize = (uint32) +- ceil((width * image->bps + 7) / (double)8) * image->spp * length; ++ ceil((width * image->bps * image->spp + 7) / (double)8) * length; + /* allocate a buffer if we don't have one already */ + if (createImageSection(sectsize, sect_buff_ptr)) + { +-- +GitLab diff --git a/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb b/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb index 9db247ecc7..e3ffb12f9e 100644 --- a/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb +++ b/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb @@ -18,6 +18,7 @@ SRC_URI = "http://download.osgeo.org/libtiff/tiff-${PV}.tar.gz \ file://0001-tiffset-fix-global-buffer-overflow-for-ASCII-tags-wh.patch \ file://561599c99f987dc32ae110370cfdd7df7975586b.patch \ file://eecb0712f4c3a5b449f70c57988260a667ddbdef.patch \ + file://CVE-2022-0891.patch \ " SRC_URI[md5sum] = "2165e7aba557463acc0664e71a3ed424" SRC_URI[sha256sum] = "5d29f32517dadb6dbcd1255ea5bbc93a2b54b94fbf83653b4d65c7d6775b8634" From patchwork Thu Apr 28 21:46:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 7331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D4EBC433F5 for ; Thu, 28 Apr 2022 21:47:21 +0000 (UTC) Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by mx.groups.io with SMTP id smtpd.web09.2987.1651182436629476256 for ; Thu, 28 Apr 2022 14:47:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=HRe/UFcw; spf=softfail (domain: sakoman.com, ip: 209.85.216.42, mailfrom: steve@sakoman.com) Received: by mail-pj1-f42.google.com with SMTP id iq10so5465571pjb.0 for ; Thu, 28 Apr 2022 14:47:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Ex44/V1Wugw7hn6efPWrV7AkFBlh3zmoMrodlnxbOxY=; b=HRe/UFcwdetE3RDTAjNiK3QyQRMcXIt9hBhcrrWJ0kX94tEqfHTmY2QUNf5t1bCd1B jA2eyQW9bys+jfXu/OuFOgTxn98B35XudVOVyZLWneD89CW6nK03o3nMcd5981Wc9R35 /LjOWYzJwzXR7T1DIbvtiNZ1qCkBaH9dDg4Q3aZJIGRqE3IK7m7SJS318fqgJFLpjncd eVYTJYUHnPEZ0fII1ATDQ0CWQSHiQ+OOEc1JmdP04rU+qJ8Mw7LFxf9rR2uJP8pca6+O XQd0jSr9nYw0riFWG0p2qcz6JrnFzuU2Rle4cMrV7vvJOaM+kEIlSZdbOJhUdZm124xk XjwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ex44/V1Wugw7hn6efPWrV7AkFBlh3zmoMrodlnxbOxY=; b=EkAmzyyMPKYyJOPhhZsOxeQJl/pLDvX33lvlJkxn4xNwDngUQT1LjqjSyA3mauDwQ4 /z4QVbWrU9m1f/+tyI0HsJvs71gGbBj0M0+dLiWvmsvQkM2Xsus2HxOhjZDuNnXttkA+ zNo5oAiHSr329pPLarO6oE9U4MJHenHKPk+QEmuULJI8EMqxylUn3+689+h8ZYtz5mtA RFuKfAuS3VkFY0tvGhh8GiAAz1uu4+kpO5sYoBjmbM5+R6p/pcE/gWrJabLb57kDNlE+ kq6y4rHuEpBzyinIsRcvOspYNEep6++NUvqfm1VjZVZTwSwgdbZzF2V1Z1ijwFuCOi8a eDFA== X-Gm-Message-State: AOAM533ilYVgDg6zHJkqim8X8aSaEMlg6PSJ+VrDMzk94564wkL6M0rm or3lmpXgO/Ffz5XoDUrMEktP5uGMO3/EHByRq9c= X-Google-Smtp-Source: ABdhPJyZ8fRxJLe05oa2tpxCaV1FKzUyK1AnvXTFwGhXxEW20VyWicdEmU/eBlo0V/J5rFoNONvV9g== X-Received: by 2002:a17:903:208:b0:15c:7dd0:eb64 with SMTP id r8-20020a170903020800b0015c7dd0eb64mr32964731plh.40.1651182435441; Thu, 28 Apr 2022 14:47:15 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id c138-20020a624e90000000b005081f92826dsm749742pfb.99.2022.04.28.14.47.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 14:47:14 -0700 (PDT) From: Steve Sakoman To: openembedded-core@lists.openembedded.org Subject: [OE-core][dunfell 2/6] boost: don't specify gcc version Date: Thu, 28 Apr 2022 11:46:32 -1000 Message-Id: <9ef2a0d98d705dacf8909d846993a6d68c80e4aa.1651182153.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 28 Apr 2022 21:47:21 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164980 From: Ross Burton There's no need to specify an ancient GCC version here as Boost will probe it. Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Tim Orling Signed-off-by: Steve Sakoman --- meta/recipes-support/boost/boost.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-support/boost/boost.inc b/meta/recipes-support/boost/boost.inc index 829e728b6d..1c13fb3599 100644 --- a/meta/recipes-support/boost/boost.inc +++ b/meta/recipes-support/boost/boost.inc @@ -165,7 +165,7 @@ do_configure() { # D2194:Fixing the failure of "error: duplicate initialization of gcc with the following parameters" during compilation. rm -f ${WORKDIR}/user-config.jam - echo 'using gcc : 4.3.1 : ${CXX} : "${CFLAGS}" "${CXXFLAGS}" "${LDFLAGS}" ;' >> ${WORKDIR}/user-config.jam + echo 'using gcc : : ${CXX} : "${CFLAGS}" "${CXXFLAGS}" "${LDFLAGS}" ;' >> ${WORKDIR}/user-config.jam # If we want Python then we need to tell Boost *exactly* where to find it if ${@bb.utils.contains('BOOST_LIBS', 'python', 'true', 'false', d)}; then From patchwork Thu Apr 28 21:46:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 7329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C9E6C433EF for ; Thu, 28 Apr 2022 21:47:21 +0000 (UTC) Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by mx.groups.io with SMTP id smtpd.web11.2998.1651182438961801138 for ; Thu, 28 Apr 2022 14:47:19 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=z/ujMq/j; spf=softfail (domain: sakoman.com, ip: 209.85.210.180, mailfrom: steve@sakoman.com) Received: by mail-pf1-f180.google.com with SMTP id y38so5334176pfa.6 for ; Thu, 28 Apr 2022 14:47:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=xD0Eo/IM4jfvODru564CWFYw6K/YO65XhmXSk+nZ+Wo=; b=z/ujMq/jp3h2ULJR80PI1Npa2OG2/Tpgn8jqjJtFq4Xm1lSaargv1EWtipA3sWyW5W zM4QPLKvgN8mhWvyq+jveCziF0xrtjfVOywfNCrLs0YLzeC8H4/JgNIY7GRdmn0Q8Ptx ofM+Ui/DeBMwCxZGckcworTY0F91ECG/vxtUKDf0EeVU1wwcKKvc3cYfgTJshf0BBY+Z Ep3dplTjcjxZ45RtrZO8xepyTOLZ0BODLznJAiteD2JKpGLVR6U3ToFEBmjy1KaUquDJ R7AASQ+vBLDdAp4+ib5WZ5KjqUyO0M/BlAQkS+PO3Uer5simOK3ItCN9D3FnvKrUSvXs 96dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xD0Eo/IM4jfvODru564CWFYw6K/YO65XhmXSk+nZ+Wo=; b=qsrJZZj8r0ASZfKJoiHBaus59zp6sMf2ILlUtcjPiuRFkr2XBSil/kM5xQCOKM+kHV wcDSh4bX+nTd0FkkALjMyL9yZFHC0zWPwQSG+Fa58ZD4jCAgQRYyKDsG0lGYHCC8uuTS oYIzuwlhBhvZX1ylkg5egfkI2BT+/AYCRksDqE473dJY4ut1WmwuG0LqDjHjD8xvapA+ 0mSAXIWtheQAwMhzex+CjlwnA6QQenh4mKRMUNEosaaBoq47KDXmP3nZ7eJLW96FM2kq Vm3sRlp+PdoJgd8d88Bc/xKAmip/W7E8YlHF38BoNQRuz2ujFtfZzyXtXkjV6jB2Ckgx 979Q== X-Gm-Message-State: AOAM531K+tUQhKuPsgJ5C3Ui0Ui8XNhjukDEPhOvRzxMTvhv8YW/CDZB gmg/dPsNTir0ylMMaehjgA0ck+nd+FizlEBLbvA= X-Google-Smtp-Source: ABdhPJx14BpWrkhDqiaCKRD77RytKBBnO1u4LOSs9Dek7kr57cn+EcuRWQtlYAuYdJJSxtIgs4gMsA== X-Received: by 2002:a63:224f:0:b0:399:4a1a:b01f with SMTP id t15-20020a63224f000000b003994a1ab01fmr30261135pgm.123.1651182437751; Thu, 28 Apr 2022 14:47:17 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id c138-20020a624e90000000b005081f92826dsm749742pfb.99.2022.04.28.14.47.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 14:47:17 -0700 (PDT) From: Steve Sakoman To: openembedded-core@lists.openembedded.org Subject: [OE-core][dunfell 3/6] linux-firmware: correct license for ar3k firmware Date: Thu, 28 Apr 2022 11:46:33 -1000 Message-Id: <8c0aa16d76e5492f774fcfe08c829c877991afbd.1651182153.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 28 Apr 2022 21:47:21 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164981 From: Dmitry Baryshkov According to the WHENCE file, some a3k firmware files are licensed under the special ar3k license, while others are licensed under the more generic Atheros license. Document this by adding extending the LICENSE_${PN}-ar3k and depending on both of them. Signed-off-by: Dmitry Baryshkov Signed-off-by: Luca Ceresoli (cherry picked from commit 8e651814af706285d64b532095fcd6f5f02629ba) Signed-off-by: Steve Sakoman --- meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb b/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb index 04a6cefaa1..ad8dbac17f 100644 --- a/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb +++ b/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb @@ -352,7 +352,7 @@ FILES_${PN}-carl9170 = " \ RDEPENDS_${PN}-carl9170 += "${PN}-gplv2-license" # For QualCommAthos -LICENSE_${PN}-ar3k = "Firmware-qualcommAthos_ar3k" +LICENSE_${PN}-ar3k = "Firmware-qualcommAthos_ar3k & Firmware-atheros_firmware" LICENSE_${PN}-ar3k-license = "Firmware-qualcommAthos_ar3k" LICENSE_${PN}-ath10k = "Firmware-qualcommAthos_ath10k" LICENSE_${PN}-ath10k-license = "Firmware-qualcommAthos_ath10k" @@ -376,7 +376,7 @@ FILES_${PN}-qca = " \ ${nonarch_base_libdir}/firmware/qca \ " -RDEPENDS_${PN}-ar3k += "${PN}-ar3k-license" +RDEPENDS_${PN}-ar3k += "${PN}-ar3k-license ${PN}-atheros-license" RDEPENDS_${PN}-ath10k += "${PN}-ath10k-license" RDEPENDS_${PN}-ath11k += "${PN}-ath10k-license" RDEPENDS_${PN}-qca += "${PN}-ath10k-license" From patchwork Thu Apr 28 21:46:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 7332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E022C4332F for ; Thu, 28 Apr 2022 21:47:21 +0000 (UTC) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mx.groups.io with SMTP id smtpd.web08.2946.1651182441011175862 for ; Thu, 28 Apr 2022 14:47:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=E2ISupu1; spf=softfail (domain: sakoman.com, ip: 209.85.216.43, mailfrom: steve@sakoman.com) Received: by mail-pj1-f43.google.com with SMTP id j8-20020a17090a060800b001cd4fb60dccso5608038pjj.2 for ; Thu, 28 Apr 2022 14:47:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bkMeNNHEJRCOix41sp3liAyAArRWXnPUDUE/v+1u8gw=; b=E2ISupu1ZddNUO8b3QbWo7IiSBDBpfuuruKlN78Q8NiMcbapLFLlGXSCoU82sDBxoQ bawEvniXOhrPtw80s6fqJzzXaNa+3jCG1buMyC1zXPn/InLfreiudhMbKHUF9lEMXnbe iWpvf3xowNc9whcqM0VFPdL6v7Oe2qIhjZQSBDg4BrOnvPwoq2TixZ8fxNPQPH8eQjc0 wO9l7K/NJpVE1IvSSdrzA88mwz4J9gX+mtsm5DwdUstqU0ZTC7UrJTmENcVqTcmjFUaT EVqSqNlmKFmFMBP3db3nPnYep6X/JWYElZk5lNUNJ9dz+607Kmf68IFNnzjXfz257LXa NHNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bkMeNNHEJRCOix41sp3liAyAArRWXnPUDUE/v+1u8gw=; b=3DMdgOnrXrz/0vjf6O3QyACcs43947SJ3yvNx2q+F8zCZL1MRnFc2GmpD5XjilDxqp LCPdp/0zswscuoWLzeTwYORcC5568gqErN7qHeFz+JyaxC7K7wWEToHqdvIsEmnpC2jy oh9sTdk3TyC7r3ZJmsoI1Ec4boBF5Xxdy/MpfY4lhXiDQyGgtXKTEvBKREHOk7mLUPP4 pqksXE8xe8TWB0rcwHciySM+6o/NDG8kyMsxE8nzNIHpP/QvTtBOO00d1DMF3ivfXGde pHKzHLuFixvwPLhnIaUOWLnFlRz7rbqpHigMrwUKA4n0YRRJw/YTCWokGxfa8ElP8b6B 26kA== X-Gm-Message-State: AOAM533yxiHwilXze/36OfT0JGOD3hMiYR69URVX37eEIDgLq8keqpXg RjM7XR/Fh5q9nsCUI4fGiYZZR4jXCyAF2B4tyqI= X-Google-Smtp-Source: ABdhPJwu1UEfZmBdvvcnckUqM0jkIpozAW9mJ+8dkEe3D5zuZvKT+m2XBDxcYCkqhdPQX+5EiG/QUQ== X-Received: by 2002:a17:90b:3805:b0:1db:c60d:59ed with SMTP id mq5-20020a17090b380500b001dbc60d59edmr324362pjb.97.1651182439929; Thu, 28 Apr 2022 14:47:19 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id c138-20020a624e90000000b005081f92826dsm749742pfb.99.2022.04.28.14.47.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 14:47:19 -0700 (PDT) From: Steve Sakoman To: openembedded-core@lists.openembedded.org Subject: [OE-core][dunfell 4/6] cve-check: add json format Date: Thu, 28 Apr 2022 11:46:34 -1000 Message-Id: <92b6011ab25fd36e2f8900a4db6883cdebc3cd3d.1651182153.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 28 Apr 2022 21:47:21 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164982 From: Marta Rybczynska Backport to dunfell from master df567de36ae5964bee433ebb97e8bf702034994a Add an option to output the CVE check in a JSON-based format. This format is easier to parse in software than the original text-based one and allows post-processing by other tools. Output formats are now handed by CVE_CHECK_FORMAT_TEXT and CVE_CHECK_FORMAT_JSON. The text format is enabled by default to maintain compatibility, while the JSON format is disabled by default. The JSON output format gets generated in a similar way to the text format with the exception of the manifest: appending to JSON arrays requires parsing the file. Because of that we first write JSON fragments and then assemble them in one pass at the end. Signed-off-by: Marta Rybczynska Signed-off-by: Steve Sakoman --- meta/classes/cve-check.bbclass | 144 ++++++++++++++++++++++++++++++++- meta/lib/oe/cve_check.py | 16 ++++ 2 files changed, 159 insertions(+), 1 deletion(-) diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 75c5b92b96..a7156cbdfb 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -34,15 +34,27 @@ CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check" CVE_CHECK_SUMMARY_DIR ?= "${LOG_DIR}/cve" CVE_CHECK_SUMMARY_FILE_NAME ?= "cve-summary" CVE_CHECK_SUMMARY_FILE ?= "${CVE_CHECK_SUMMARY_DIR}/${CVE_CHECK_SUMMARY_FILE_NAME}" +CVE_CHECK_SUMMARY_FILE_NAME_JSON = "cve-summary.json" +CVE_CHECK_SUMMARY_INDEX_PATH = "${CVE_CHECK_SUMMARY_DIR}/cve-summary-index.txt" + +CVE_CHECK_LOG_JSON ?= "${T}/cve.json" CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve" CVE_CHECK_RECIPE_FILE ?= "${CVE_CHECK_DIR}/${PN}" +CVE_CHECK_RECIPE_FILE_JSON ?= "${CVE_CHECK_DIR}/${PN}_cve.json" CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cve" +CVE_CHECK_MANIFEST_JSON ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.json" CVE_CHECK_COPY_FILES ??= "1" CVE_CHECK_CREATE_MANIFEST ??= "1" CVE_CHECK_REPORT_PATCHED ??= "1" +# Provide text output +CVE_CHECK_FORMAT_TEXT ??= "1" + +# Provide JSON output - disabled by default for backward compatibility +CVE_CHECK_FORMAT_JSON ??= "0" + # Whitelist for packages (PN) CVE_CHECK_PN_WHITELIST ?= "" @@ -118,6 +130,7 @@ python cve_check_cleanup () { Delete the file used to gather all the CVE information. """ bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE")) + bb.utils.remove(e.data.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")) } addhandler cve_check_cleanup @@ -129,11 +142,15 @@ python cve_check_write_rootfs_manifest () { """ import shutil + from oe.cve_check import cve_check_merge_jsons if d.getVar("CVE_CHECK_COPY_FILES") == "1": deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE") if os.path.exists(deploy_file): bb.utils.remove(deploy_file) + deploy_file_json = d.getVar("CVE_CHECK_RECIPE_FILE_JSON") + if os.path.exists(deploy_file_json): + bb.utils.remove(deploy_file_json) if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE")): bb.note("Writing rootfs CVE manifest") @@ -152,6 +169,26 @@ python cve_check_write_rootfs_manifest () { os.remove(manifest_link) os.symlink(os.path.basename(manifest_name), manifest_link) bb.plain("Image CVE report stored in: %s" % manifest_name) + + if os.path.exists(d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")): + import json + bb.note("Generating JSON CVE manifest") + deploy_dir = d.getVar("DEPLOY_DIR_IMAGE") + link_name = d.getVar("IMAGE_LINK_NAME") + manifest_name = d.getVar("CVE_CHECK_MANIFEST_JSON") + index_file = d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH") + manifest = {"version":"1", "package": []} + with open(index_file) as f: + filename = f.readline() + while filename: + with open(filename.rstrip()) as j: + data = json.load(j) + cve_check_merge_jsons(manifest, data) + filename = f.readline() + + with open(manifest_name, "w") as f: + json.dump(manifest, f, indent=2) + bb.plain("Image CVE report stored in: %s" % manifest_name) } ROOTFS_POSTPROCESS_COMMAND_prepend = "${@'cve_check_write_rootfs_manifest; ' if d.getVar('CVE_CHECK_CREATE_MANIFEST') == '1' else ''}" @@ -337,7 +374,7 @@ def get_cve_info(d, cves): conn.close() return cve_data -def cve_write_data(d, patched, unpatched, whitelisted, cve_data): +def cve_write_data_text(d, patched, unpatched, whitelisted, cve_data): """ Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and CVE manifest if enabled. @@ -403,3 +440,108 @@ def cve_write_data(d, patched, unpatched, whitelisted, cve_data): with open(d.getVar("CVE_CHECK_TMP_FILE"), "a") as f: f.write("%s" % write_string) + +def cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file): + """ + Write CVE information in the JSON format: to WORKDIR; and to + CVE_CHECK_DIR, if CVE manifest if enabled, write fragment + files that will be assembled at the end in cve_check_write_rootfs_manifest. + """ + + import json + + write_string = json.dumps(output, indent=2) + with open(direct_file, "w") as f: + bb.note("Writing file %s with CVE information" % direct_file) + f.write(write_string) + + if d.getVar("CVE_CHECK_COPY_FILES") == "1": + bb.utils.mkdirhier(os.path.dirname(deploy_file)) + with open(deploy_file, "w") as f: + f.write(write_string) + + if d.getVar("CVE_CHECK_CREATE_MANIFEST") == "1": + cvelogpath = d.getVar("CVE_CHECK_SUMMARY_DIR") + index_path = d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH") + bb.utils.mkdirhier(cvelogpath) + fragment_file = os.path.basename(deploy_file) + fragment_path = os.path.join(cvelogpath, fragment_file) + with open(fragment_path, "w") as f: + f.write(write_string) + with open(index_path, "a+") as f: + f.write("%s\n" % fragment_path) + +def cve_write_data_json(d, patched, unpatched, ignored, cve_data): + """ + Prepare CVE data for the JSON format, then write it. + """ + + output = {"version":"1", "package": []} + nvd_link = "https://nvd.nist.gov/vuln/detail/" + + fdir_name = d.getVar("FILE_DIRNAME") + layer = fdir_name.split("/")[-3] + + include_layers = d.getVar("CVE_CHECK_LAYER_INCLUDELIST").split() + exclude_layers = d.getVar("CVE_CHECK_LAYER_EXCLUDELIST").split() + + if exclude_layers and layer in exclude_layers: + return + + if include_layers and layer not in include_layers: + return + + unpatched_cves = [] + + package_version = "%s%s" % (d.getVar("EXTENDPE"), d.getVar("PV")) + package_data = { + "name" : d.getVar("PN"), + "layer" : layer, + "version" : package_version + } + cve_list = [] + + for cve in sorted(cve_data): + is_patched = cve in patched + status = "Unpatched" + if is_patched and (d.getVar("CVE_CHECK_REPORT_PATCHED") != "1"): + continue + if cve in ignored: + status = "Ignored" + elif is_patched: + status = "Patched" + else: + # default value of status is Unpatched + unpatched_cves.append(cve) + + issue_link = "%s%s" % (nvd_link, cve) + + cve_item = { + "id" : cve, + "summary" : cve_data[cve]["summary"], + "scorev2" : cve_data[cve]["scorev2"], + "scorev3" : cve_data[cve]["scorev3"], + "vector" : cve_data[cve]["vector"], + "status" : status, + "link": issue_link + } + cve_list.append(cve_item) + + package_data["issue"] = cve_list + output["package"].append(package_data) + + direct_file = d.getVar("CVE_CHECK_LOG_JSON") + deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE_JSON") + manifest_file = d.getVar("CVE_CHECK_SUMMARY_FILE_NAME_JSON") + + cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file) + +def cve_write_data(d, patched, unpatched, ignored, cve_data): + """ + Write CVE data in each enabled format. + """ + + if d.getVar("CVE_CHECK_FORMAT_TEXT") == "1": + cve_write_data_text(d, patched, unpatched, ignored, cve_data) + if d.getVar("CVE_CHECK_FORMAT_JSON") == "1": + cve_write_data_json(d, patched, unpatched, ignored, cve_data) diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py index a1d7c292af..1d3c775bbe 100644 --- a/meta/lib/oe/cve_check.py +++ b/meta/lib/oe/cve_check.py @@ -63,3 +63,19 @@ def _cmpkey(release, patch_l, pre_l, pre_v): else: _pre = float(pre_v) if pre_v else float('-inf') return _release, _patch, _pre + +def cve_check_merge_jsons(output, data): + """ + Merge the data in the "package" property to the main data file + output + """ + if output["version"] != data["version"]: + bb.error("Version mismatch when merging JSON outputs") + return + + for product in output["package"]: + if product["name"] == data["package"][0]["name"]: + bb.error("Error adding the same package twice") + return + + output["package"].append(data["package"][0]) From patchwork Thu Apr 28 21:46:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 7333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 709DEC433EF for ; Thu, 28 Apr 2022 21:47:31 +0000 (UTC) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mx.groups.io with SMTP id smtpd.web10.2988.1651182443529051846 for ; Thu, 28 Apr 2022 14:47:23 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=DH9Z2L4s; spf=softfail (domain: sakoman.com, ip: 209.85.210.173, mailfrom: steve@sakoman.com) Received: by mail-pf1-f173.google.com with SMTP id p8so5326817pfh.8 for ; Thu, 28 Apr 2022 14:47:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Aq2U1YTdeve7Ovc+I8J+wOUaDfdo0lh+64TNGIhnRFc=; b=DH9Z2L4saDB3oqLzmBaqhUd6F3ysxR9TXfLg8zGyGY517n7XDRo6sLQzH/MnDEIQQz 7xYYcH7QdhPARwRH6+cRR6j2X9czDhsKA1vkuI1tlFZF5ekmno4iQ4MMgar9pL+xBBZs WvTYtMASDpqQgnbSkvzMP0YlXJixhsC3iUVh5r516lWm7Dw7F7KOfz9oF4vnggGmCyZq zTl/14EAHKQDWUKz2jzdUXdlWo6fCNXXUeNNveAwiRa4nrKiS/4YMRFTiEvte61rJVaF tOvYLWRrmpTS97M/8M2DfYbno8fGsOvUz1IhhCsLZh8vq1j+3sv/w0XH9jzkLBG4zKY0 YSMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Aq2U1YTdeve7Ovc+I8J+wOUaDfdo0lh+64TNGIhnRFc=; b=drJdMesUGZjteIzJkdoc36JqLsA01t0GjzvgGlwsT2KMiIkJvy16f4zs0W6mCKEbWI 9zqN+eVIPKZQHMs6vFTnJmH4pUSXjwrKLa7wYPcaul+3VcrLDdz2a1MpicpO5AGblVXH SEGFfg4YZKPn3ev4z/BjbzGsAI/zhh++LFoDNIcGfPE69+y9PWZvJZNbmwvy/06NpCv6 fkQKxjI3snQEXUNBHrVLl/M7p6Y2wpt9Ml1NAjD6r7MDtTBc9ZR0/yzCaail8tPZox3/ m2vbBB1TMn+Wx8jtFSl1NfotO3uccY4dtDLTPR2mjW5LN8JksKFV18DvH8U52+Oi1xDC Zltw== X-Gm-Message-State: AOAM532kDTg4WbYSsZ7lY25t7nXyR62/nU0N+5hHWXlaUb+89m2nvPoX 0DAQeLp2wS5StfU7ybJ2LyJq3rqJk5xdC/0rfRY= X-Google-Smtp-Source: ABdhPJx73lV29MQpkuzXwlCR8YMOxO0EIMJ7BscSXRwTlubzMdPDX/bpRm8uABgGbVxbboLhX1U6/A== X-Received: by 2002:a63:564c:0:b0:3c1:976d:9f03 with SMTP id g12-20020a63564c000000b003c1976d9f03mr2217026pgm.354.1651182442310; Thu, 28 Apr 2022 14:47:22 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id c138-20020a624e90000000b005081f92826dsm749742pfb.99.2022.04.28.14.47.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 14:47:21 -0700 (PDT) From: Steve Sakoman To: openembedded-core@lists.openembedded.org Subject: [OE-core][dunfell 5/6] perf-build-test/report: Drop phantomjs and html email reports support Date: Thu, 28 Apr 2022 11:46:35 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 28 Apr 2022 21:47:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164983 From: Richard Purdie phantomjs isn't reliable and we've moved to sharing the reports via a webserver. Update the scripts to more match those being used in the autobuilder helper where the html email support was removed. Signed-off-by: Richard Purdie (cherry picked from commit b5c131006e3fad0a15e6cdf81f71dc1e96647028) Signed-off-by: Steve Sakoman --- scripts/contrib/build-perf-test-wrapper.sh | 15 +- scripts/contrib/oe-build-perf-report-email.py | 154 +----------------- 2 files changed, 8 insertions(+), 161 deletions(-) diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh index fa71d4a2e9..0a85e6e708 100755 --- a/scripts/contrib/build-perf-test-wrapper.sh +++ b/scripts/contrib/build-perf-test-wrapper.sh @@ -87,21 +87,10 @@ if [ $# -ne 0 ]; then exit 1 fi -if [ -n "$email_to" ]; then - if ! [ -x "$(command -v phantomjs)" ]; then - echo "ERROR: Sending email needs phantomjs." - exit 1 - fi - if ! [ -x "$(command -v optipng)" ]; then - echo "ERROR: Sending email needs optipng." - exit 1 - fi -fi - # Open a file descriptor for flock and acquire lock LOCK_FILE="/tmp/oe-build-perf-test-wrapper.lock" if ! exec 3> "$LOCK_FILE"; then - echo "ERROR: Unable to open lock file" + echo "ERROR: Unable to open loemack file" exit 1 fi if ! flock -n 3; then @@ -226,7 +215,7 @@ if [ -n "$results_repo" ]; then if [ -n "$email_to" ]; then echo "Emailing test report" os_name=`get_os_release_var PRETTY_NAME` - "$script_dir"/oe-build-perf-report-email.py --to "$email_to" --subject "Build Perf Test Report for $os_name" --text $report_txt --html $report_html "${OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS[@]}" + "$script_dir"/oe-build-perf-report-email.py --to "$email_to" --subject "Build Perf Test Report for $os_name" --text $report_txt "${OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS[@]}" fi # Upload report files, unless we're on detached head diff --git a/scripts/contrib/oe-build-perf-report-email.py b/scripts/contrib/oe-build-perf-report-email.py index de3862c897..c900720f6e 100755 --- a/scripts/contrib/oe-build-perf-report-email.py +++ b/scripts/contrib/oe-build-perf-report-email.py @@ -19,8 +19,6 @@ import socket import subprocess import sys import tempfile -from email.mime.image import MIMEImage -from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText @@ -29,19 +27,6 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") log = logging.getLogger('oe-build-perf-report') -# Find js scaper script -SCRAPE_JS = os.path.join(os.path.dirname(__file__), '..', 'lib', 'build_perf', - 'scrape-html-report.js') -if not os.path.isfile(SCRAPE_JS): - log.error("Unableto find oe-build-perf-report-scrape.js") - sys.exit(1) - - -class ReportError(Exception): - """Local errors""" - pass - - def check_utils(): """Check that all needed utils are installed in the system""" missing = [] @@ -77,137 +62,19 @@ def parse_args(argv): "the email parts") parser.add_argument('--text', help="Plain text message") - parser.add_argument('--html', - help="HTML peport generated by oe-build-perf-report") - parser.add_argument('--phantomjs-args', action='append', - help="Extra command line arguments passed to PhantomJS") args = parser.parse_args(argv) - if not args.html and not args.text: - parser.error("Please specify --html and/or --text") + if not args.text: + parser.error("Please specify --text") return args -def decode_png(infile, outfile): - """Parse/decode/optimize png data from a html element""" - with open(infile) as f: - raw_data = f.read() - - # Grab raw base64 data - b64_data = re.sub('^.*href="data:image/png;base64,', '', raw_data, 1) - b64_data = re.sub('">.+$', '', b64_data, 1) - - # Replace file with proper decoded png - with open(outfile, 'wb') as f: - f.write(base64.b64decode(b64_data)) - - subprocess.check_output(['optipng', outfile], stderr=subprocess.STDOUT) - - -def mangle_html_report(infile, outfile, pngs): - """Mangle html file into a email compatible format""" - paste = True - png_dir = os.path.dirname(outfile) - with open(infile) as f_in: - with open(outfile, 'w') as f_out: - for line in f_in.readlines(): - stripped = line.strip() - # Strip out scripts - if stripped == '': - paste = False - elif stripped == '': - paste = True - elif paste: - if re.match('^.+href="data:image/png;base64', stripped): - # Strip out encoded pngs (as they're huge in size) - continue - elif 'www.gstatic.com' in stripped: - # HACK: drop references to external static pages - continue - - # Replace charts with elements - match = re.match('
X-Patchwork-Id: 7334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70A12C4332F for ; Thu, 28 Apr 2022 21:47:31 +0000 (UTC) Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by mx.groups.io with SMTP id smtpd.web10.2990.1651182446167603629 for ; Thu, 28 Apr 2022 14:47:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=z8oxvWB0; spf=softfail (domain: sakoman.com, ip: 209.85.214.174, mailfrom: steve@sakoman.com) Received: by mail-pl1-f174.google.com with SMTP id s14so5490140plk.8 for ; Thu, 28 Apr 2022 14:47:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Td5p42OazpTYQgEhP72s6RFxDvo1/7PA2UUMD9lGQlk=; b=z8oxvWB0K0kScN4aNXfTO0lqWwGugLgAztFeByYmW45j2Y7+FiFAK4+xOWVfjRKicE z9C6G0utOyb+daCKI0w4jDwOUul/7rFakqIjeYT8zsPRs4D+0qzv8TlaF2jVLqvZFLCe 5P1LCimZjQBnK/fi4aeXKPFmKZ8R8Kp/qLwQlV5oASZXN9maQm0vUIrUP9Ry/Q9VAVnh TPrhmrFyw513o3OJbldnwXWY5Zl/VgsrlYQRj7XnRrRSRg1oDARlmOg1GaDIVyCISBX2 ow0pBWmUqqK8ZjTNPvRbbe+7IMQHAgTsfoLIIo9svWjmWuaE4zRm85zNVCFEPYFOl/tF u77Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Td5p42OazpTYQgEhP72s6RFxDvo1/7PA2UUMD9lGQlk=; b=YDJOdUK98TagWIO5ZmZVH3w4e9ZYbm+FJ6A0sgGiTORQLiAV86YeF6o2ujB2qvABVW gxlPtUZ2cwBfsa7vizEVi0aLxzKG3r/ISJHPVAHy+UHF1HBruAjsq/fOSH3CZe7c19s+ 0Ttq6mbVNCfyQKBIRkgqyVR+flJYUd0T3i3sbCoFwEw6f0GSIsqtpi4h79erXgG1Aj9s BvBTHvMK2elY/dRThHToeCDeAXi4ZK12wPJGK62Pa8wGoyhHHT9w/AxpcvdeiLK4YRMH P/EoiPE+a4lTg0SsXv9Tn/07nRmCmUx9NHTjJ3XILYZAiZc8Pe5rInD5sruqId01eB+e tFbw== X-Gm-Message-State: AOAM531XmZXQ5OyNRAHc+DE2lStzV+2638MC4hXlk7ohollU47DEGeNq AAoQZ+hJeArQBcea1m/jh2UBznv65cqLphNHlFY= X-Google-Smtp-Source: ABdhPJxOQDjJTEE8NuLsv5ohDFCz/3ojgfWUHzXXyLN9gUG8BBSBDaCC/ModryT7P5q3D+HTyWXzlg== X-Received: by 2002:a17:902:e5d2:b0:15d:9e66:b3f7 with SMTP id u18-20020a170902e5d200b0015d9e66b3f7mr7591887plf.141.1651182445119; Thu, 28 Apr 2022 14:47:25 -0700 (PDT) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id c138-20020a624e90000000b005081f92826dsm749742pfb.99.2022.04.28.14.47.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Apr 2022 14:47:24 -0700 (PDT) From: Steve Sakoman To: openembedded-core@lists.openembedded.org Subject: [OE-core][dunfell 6/6] scripts/contrib/oe-build-perf-report-email.py: remove obsolete check for phantomjs and optipng Date: Thu, 28 Apr 2022 11:46:36 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 28 Apr 2022 21:47:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/164984 Use of those tools was removed in b5c131006e3fad0a15e6cdf81f71dc1e96647028 perf-build-test/report: Drop phantomjs and html email reports support Signed-off-by: Steve Sakoman --- scripts/contrib/oe-build-perf-report-email.py | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/scripts/contrib/oe-build-perf-report-email.py b/scripts/contrib/oe-build-perf-report-email.py index c900720f6e..7192113c28 100755 --- a/scripts/contrib/oe-build-perf-report-email.py +++ b/scripts/contrib/oe-build-perf-report-email.py @@ -27,17 +27,6 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") log = logging.getLogger('oe-build-perf-report') -def check_utils(): - """Check that all needed utils are installed in the system""" - missing = [] - for cmd in ('phantomjs', 'optipng'): - if not shutil.which(cmd): - missing.append(cmd) - if missing: - log.error("The following tools are missing: %s", ' '.join(missing)) - sys.exit(1) - - def parse_args(argv): """Parse command line arguments""" description = """Email build perf test report""" @@ -101,8 +90,6 @@ def main(argv=None): if args.debug: log.setLevel(logging.DEBUG) - check_utils() - if args.outdir: outdir = args.outdir if not os.path.exists(outdir):