From patchwork Wed Nov 1 23:10:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charles-Antoine Couret X-Patchwork-Id: 33426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 749B1C4167B for ; Wed, 1 Nov 2023 23:11:23 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web11.2553.1698880279021954983 for ; Wed, 01 Nov 2023 16:11:19 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@mind.be header.s=google header.b=UtY1tKCV; spf=pass (domain: essensium.com, ip: 209.85.128.44, mailfrom: charles-antoine.couret@essensium.com) Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-4083dbc43cfso2038385e9.3 for ; Wed, 01 Nov 2023 16:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mind.be; s=google; t=1698880277; x=1699485077; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mN7ScpIHoHFZpMmQrLEskAiIeB5gMQfYDf5bd+vgIP0=; b=UtY1tKCVT+eN9gyqlCLR4mnt5Yd8UgIlz3jNkzswmaeixzh5a+V+nV0YJnzlN4PnTa m1tT2zXUk53AESlLwm5MTWsXOmhXlepeXKvZcGCqmCL7U8TsoJ2g20QHRYXq0AvC/U88 cIQzVgcm2tIZuEOnytsBsl+Nbe05ZppBPGDkNEQ4wnb32+dZ2qCQG+FxK1TkuIoBSn2v 24p8I5FvnUC4QA9Zf7jblgKX6uOKgWFWLN99Dwv5RMlv/TPjF4YJ2Dtwh6H0k8F2/39b sgcVCaO6G42A3faNgrdz/4k+kJGNA1P7Lpk9N8Uo92JLSDbNpXTKTJNcHDVXGPGZ9wCp N1ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698880277; x=1699485077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mN7ScpIHoHFZpMmQrLEskAiIeB5gMQfYDf5bd+vgIP0=; b=cWVDTgwhvhXW7BlLNOPtvFikipbURozJfzaMozmgJURnFMHkwKcsw/ZG3fkNc6P9mE 0MCB7ZCdcy4Wdp82qOKaNgI+wYhtpF2EV76pHAyWk+eMTVAbK41B2Rao3loBzRVSGoWp gU4VEmbM677kdgzhlPA98as+QSSz9NXmSlJuPpIoLcO61rM7AIZ8vBtdEILS1bV7slbQ 6CvJJStBoAV8jLoQhdpz1NbW++QnqVIY1nscKfrSIB6NoeBWnfapaZCWSloqfTbrOeSk PwZ+4g97n8CINSX5iG0wk7TCgrM8WVLGCxFFWvcrcD9rKJxOh2bHJoMaJ9OylPtuZkCD ssvw== X-Gm-Message-State: AOJu0YxjZPd81cbvMsPEhjk6PoJjqzcdJlqKDmkdbidGMBbGmeOaNe3T /+0xZLfsXwUpyAsi+4Uqvr+mxBWyE9bWSerNv+s= X-Google-Smtp-Source: AGHT+IHnTqFKKLrq7qmSgCCakFIDfLLwhm4TOmQKoeilyN+BDV3A2bsXvZ15JN/k9p2ADxAIcvMHBg== X-Received: by 2002:a5d:64ce:0:b0:32d:b031:9719 with SMTP id f14-20020a5d64ce000000b0032db0319719mr13962464wri.42.1698880277150; Wed, 01 Nov 2023 16:11:17 -0700 (PDT) Received: from Bishop.fritz.box ([2a02:578:85c6:1101:e7a2:3f2c:a83f:5e92]) by smtp.gmail.com with ESMTPSA id m11-20020adff38b000000b0032db4e660d9sm854601wro.56.2023.11.01.16.11.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 16:11:16 -0700 (PDT) From: Charles-Antoine Couret To: openembedded-core@lists.openembedded.org Cc: Charles-Antoine Couret Subject: [PATCH 1/5 v3] image_types: add python function to get the IMAGE_FILE_MAXSIZE:fstype value Date: Thu, 2 Nov 2023 00:10:50 +0100 Message-ID: <20231101231058.86928-2-charles-antoine.couret@mind.be> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231101231058.86928-1-charles-antoine.couret@mind.be> References: <20231101231058.86928-1-charles-antoine.couret@mind.be> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Nov 2023 23:11:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190049 It returns 0 if the variable is not set for this filesystem. In case of fixed partitionning where the rootfs partition can't exceed an amount of bytes, there is currently no automatic and no generic way to have this requirement met in any case. Until now, ROOTFS_SIZE value got from directory_size() does not takes into account the size of required metadata for the filesystem itself (and does not work well for other block size than 4k BTW). Obviously it's a difficult task which depends on rootfs size and filesystem type. The workaround was to set IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE to add the required extra margins. But when the final rootfs is closed to the maximum size, it's difficult to adjust them correctly. And if you remove or add new recipes in your image, you've to recompute these margins to have enough space for these metadata when the rootfs is small, and to not have too big final image when the rootfs is big. It's cumbersome and error prone to just have a build failure when the final output can't be flashed into the partition. The solution is to follow how it's implemented in buildroot by having a specific variable, here IMAGE_FILE_MAXSIZE, to create the final sparse file and trying to fill it with the content of rootfs. If there is enough space, margins are well compressed and does not consume space in the filesystem. If there is no enough space, an error is triggered to warm the developer before trying to use it in the device. If IMAGE_FILE_MAXSIZE is not set, the idea is to keep the previous behaviour for compatibility reason and to met other requirements. Signed-off-by: Charles-Antoine Couret --- meta/classes-recipe/image_types.bbclass | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass index 4aed64e27f..be8197f1f6 100644 --- a/meta/classes-recipe/image_types.bbclass +++ b/meta/classes-recipe/image_types.bbclass @@ -54,6 +54,13 @@ def imagetypes_getdepends(d): # Sort the set so that ordering is consistant return " ".join(sorted(deps)) +def get_max_image_size(d, fs): + max_size = d.getVar("IMAGE_FILE_MAXSIZE:%s" % fs) + if max_size is not None: + return max_size + + return 0 + XZ_COMPRESSION_LEVEL ?= "-9" XZ_INTEGRITY_CHECK ?= "crc32" From patchwork Wed Nov 1 23:10:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charles-Antoine Couret X-Patchwork-Id: 33427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C4C9C001B5 for ; Wed, 1 Nov 2023 23:11:23 +0000 (UTC) Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by mx.groups.io with SMTP id smtpd.web10.2506.1698880279983380483 for ; Wed, 01 Nov 2023 16:11:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@mind.be header.s=google header.b=BiBf9Zog; spf=pass (domain: essensium.com, ip: 209.85.221.51, mailfrom: charles-antoine.couret@essensium.com) Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-32ded3eb835so182565f8f.0 for ; Wed, 01 Nov 2023 16:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mind.be; s=google; t=1698880278; x=1699485078; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xLymyTckY/0P8pSBxfSX5eIB5/EtSaUUZE5sJp67OzE=; b=BiBf9ZogoAP+TQJJqLjBnBqIHV9rx0HA6JvqfKyLCQXDsYV+egDFUquMBvyrrdo0os cprRyRvUna6FiQPQmjWpfz1hkzgfrF/CdaLFSgcPKdjs4RHDw+REYtjv9VgaghaZW8Q0 E5MwYtQ9ir+McRf8GgbGWosCR3ZTu6I3kGcKxYmRgvaSWSX2RzZDYN7dJEmSsfV2AgT3 LGn5OneVwZKsGcJznqE3pFFZ6zEvc+u3LJco8/I1ZgI+BTDvC7Uga3p8RIHS0eiOJDJ2 fzh7LeeUxrOcM2ugPRfZqHM8S2FNoqRXanIRdif+F+hqNRCRAk3rtVwX0Py8VbG951qT Tdfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698880278; x=1699485078; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xLymyTckY/0P8pSBxfSX5eIB5/EtSaUUZE5sJp67OzE=; b=p4ZzZGRoUbYmVmMwokcHMBx7HP8DS6ZbTAL4hexqG8UuEZzjb4KfdrrzN8JSaWOyQf R32S3p+qIoQblfJagNnvjfMmZ4Uh+pdx6Q6Rvy8SMF15cp11SYCQlbnrswpRraBJoQUI EwpZCZQNNcuyE+x/u04a26pCOhBDw4y+jJP0Douexjt8mKHOdPOY8NtBXlYtjH3ksfMw 9RHOnyOE773GzXIKDtDmmoBGQz8zZVD3m98KhINLgQpfEfO2wDKo09RfQW/o7304JzI1 2JHgtYGCIDqdg5jiL5x6lQXfqTfmRQaxcGArGE2M9UqeI4dzC3i+Fu2jFSxaa1tQDIhF cBMg== X-Gm-Message-State: AOJu0YyXb4YjJrzhUOTdIPcCewrwmylsBrbWWUgh4l54msS+C7Z8F7q6 GLnj7kIVkeqQte4//sE3UOOogJes3redFb0ApiU= X-Google-Smtp-Source: AGHT+IFb14SVsoez9EEmy4eutAV+bZiaQ4QE/17Dd0/7OOHrAj8FOVt9jX6LAOYta8gR7jIryPcrvQ== X-Received: by 2002:a05:6000:1052:b0:32d:aa20:3d05 with SMTP id c18-20020a056000105200b0032daa203d05mr13506887wrx.27.1698880278417; Wed, 01 Nov 2023 16:11:18 -0700 (PDT) Received: from Bishop.fritz.box ([2a02:578:85c6:1101:e7a2:3f2c:a83f:5e92]) by smtp.gmail.com with ESMTPSA id m11-20020adff38b000000b0032db4e660d9sm854601wro.56.2023.11.01.16.11.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 16:11:18 -0700 (PDT) From: Charles-Antoine Couret To: openembedded-core@lists.openembedded.org Cc: Charles-Antoine Couret Subject: [PATCH 2/5 v3] image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types Date: Thu, 2 Nov 2023 00:10:51 +0100 Message-ID: <20231101231058.86928-3-charles-antoine.couret@mind.be> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231101231058.86928-1-charles-antoine.couret@mind.be> References: <20231101231058.86928-1-charles-antoine.couret@mind.be> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Nov 2023 23:11:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190050 If defined, this variable value overrides the size of ext* partition file created by mkfs. Otherwise previous logic based on ROOTFS_SIZE variable is used. It should be set when the final file size would not be above a specific value due to fixed partitionning for example. Signed-off-by: Charles-Antoine Couret --- meta/classes-recipe/image_types.bbclass | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass index be8197f1f6..fb1e33cf3e 100644 --- a/meta/classes-recipe/image_types.bbclass +++ b/meta/classes-recipe/image_types.bbclass @@ -79,24 +79,32 @@ IMAGE_CMD:cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}.cr oe_mkext234fs () { fstype=$1 + image_file_maxsize=$2 extra_imagecmd="" + rootfs_file_size=$ROOTFS_SIZE - if [ $# -gt 1 ]; then - shift + if [ $# -gt 2 ]; then + shift 2 extra_imagecmd=$@ fi + + if [[ "${image_file_maxsize}" -ne 0 ]]; then + rootfs_file_size=${image_file_maxsize} + fi + # If generating an empty image the size of the sparse block should be large # enough to allocate an ext4 filesystem using 4096 bytes per inode, this is # about 60K, so dd needs a minimum count of 60, with bs=1024 (bytes per IO) eval local COUNT=\"0\" eval local MIN_COUNT=\"60\" - if [ $ROOTFS_SIZE -lt $MIN_COUNT ]; then + if [ $rootfs_file_size -lt $MIN_COUNT ]; then eval COUNT=\"$MIN_COUNT\" fi + # Create a sparse image block - bbdebug 1 Executing "dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024" - dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024 + bbdebug 1 Executing "dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype seek=$rootfs_file_size count=$COUNT bs=1024" + dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype seek=$rootfs_file_size count=$COUNT bs=1024 bbdebug 1 "Actual Rootfs size: `du -s ${IMAGE_ROOTFS}`" bbdebug 1 "Actual Partition size: `stat -c '%s' ${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype`" bbdebug 1 Executing "mkfs.$fstype -F $extra_imagecmd ${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype -d ${IMAGE_ROOTFS}" @@ -105,9 +113,9 @@ oe_mkext234fs () { fsck.$fstype -pvfD ${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype || [ $? -le 3 ] } -IMAGE_CMD:ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" -IMAGE_CMD:ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" -IMAGE_CMD:ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" +IMAGE_CMD:ext2 = "oe_mkext234fs ext2 ${@get_max_image_size(d, 'ext2')} ${EXTRA_IMAGECMD}" +IMAGE_CMD:ext3 = "oe_mkext234fs ext3 ${@get_max_image_size(d, 'ext3')} ${EXTRA_IMAGECMD}" +IMAGE_CMD:ext4 = "oe_mkext234fs ext4 ${@get_max_image_size(d, 'ext4')} ${EXTRA_IMAGECMD}" MIN_BTRFS_SIZE ?= "16384" IMAGE_CMD:btrfs () { From patchwork Wed Nov 1 23:10:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charles-Antoine Couret X-Patchwork-Id: 33428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A2F6C41535 for ; Wed, 1 Nov 2023 23:11:23 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web11.2555.1698880281317888188 for ; Wed, 01 Nov 2023 16:11:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@mind.be header.s=google header.b=ahVMjroc; spf=pass (domain: essensium.com, ip: 209.85.128.44, mailfrom: charles-antoine.couret@essensium.com) Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-4094301d505so2372415e9.2 for ; Wed, 01 Nov 2023 16:11:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mind.be; s=google; t=1698880279; x=1699485079; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yjb6oFVhxVokTB+M07EYVOSfJtbkPXR5TBbiVtAEZf0=; b=ahVMjrocuRqctwxedVp61rxA/forJKjoSLJVwKtGtLtSc2pf3chV9n8TKSmYG/43kI nRvLt04ln6AEFj1GM8y7KQB4oEbXYqdc2K5TPaVcMwG/Fwr9t6IQWikTD/7Nm/qJkSXM MJY3spbCHQHmnOBIlHjPQwiKqw7PfKgEVdGK9VRwjavQAMRY3PS5e7uA3g7CuTvoVXHV YFMuMUvcM+clkmPtOY3lbn2oHdge0GvdG8dzb21NFWNL8DnUtVHSs8SLy3ekfTcOW2zh lexEhrJivq1Q/VsIzpSTZQ3DzGaWug2PodhEck44DCTGk7c+axWjukVET0gCc3wrP5Yu FPow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698880279; x=1699485079; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yjb6oFVhxVokTB+M07EYVOSfJtbkPXR5TBbiVtAEZf0=; b=fxYFXwx1Re4Ub7b5GWRvxx1PT70mdfpzM6M35YX31jsRyB5x4kwjmWPL8MTvfqgQpd HRIJRFIJb+qMWHIzSce/mSzm8VznL0DFRHQF/bSLIT2W/GNl5DEFk+FC6SOM59C+2YN8 4jiIVf3eaFXDaleQtpzEHh0zcmicHXF8BZAjO2Y31HkyqtlI55CcVnZNKI4OW0KrH0Ui BfVotPTAR/2CXEhYrmBUDLKXM2Jno2NvixCq2SkPMlDbiZ7A81O24JXPRCsaUAhHgiye NelumxUi0+wwDUT0SQeUe+P7z10uAFLeFVMLYglePTEdctbaRodJtXnTkWnNCh5kzrP7 TURA== X-Gm-Message-State: AOJu0YyDva/AAi0/ieBo2RkthZuQJB1otFDmR3EaSmjV25d4xH+AYFA/ WP1b5AdzgiCGyBxtt59dNOkDYZkquIvxLfVds2I= X-Google-Smtp-Source: AGHT+IHQalmjxuxjHhei2Yz06B/2+T/kwlLXCbgh3lESsd25aa6rJBoTe+YyO4thy2IJIb9Jm12FNg== X-Received: by 2002:a5d:64ae:0:b0:32f:a3fb:835b with SMTP id m14-20020a5d64ae000000b0032fa3fb835bmr2155152wrp.44.1698880279527; Wed, 01 Nov 2023 16:11:19 -0700 (PDT) Received: from Bishop.fritz.box ([2a02:578:85c6:1101:e7a2:3f2c:a83f:5e92]) by smtp.gmail.com with ESMTPSA id m11-20020adff38b000000b0032db4e660d9sm854601wro.56.2023.11.01.16.11.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 16:11:19 -0700 (PDT) From: Charles-Antoine Couret To: openembedded-core@lists.openembedded.org Cc: Charles-Antoine Couret Subject: [PATCH 3/5 v3] image_types: use IMAGE_FILE_MAXSIZE variable for btrfs image types Date: Thu, 2 Nov 2023 00:10:52 +0100 Message-ID: <20231101231058.86928-4-charles-antoine.couret@mind.be> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231101231058.86928-1-charles-antoine.couret@mind.be> References: <20231101231058.86928-1-charles-antoine.couret@mind.be> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Nov 2023 23:11:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190051 If defined, this variable value overrides the size of btrfs partition file created by mkfs. Otherwise previous logic based on ROOTFS_SIZE variable is used. It should be set when the final file size would not be above a specific value due to fixed partitionning for example. Signed-off-by: Charles-Antoine Couret --- meta/classes-recipe/image_types.bbclass | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass index fb1e33cf3e..1356552445 100644 --- a/meta/classes-recipe/image_types.bbclass +++ b/meta/classes-recipe/image_types.bbclass @@ -120,6 +120,12 @@ IMAGE_CMD:ext4 = "oe_mkext234fs ext4 ${@get_max_image_size(d, 'ext4')} ${EXTRA_I MIN_BTRFS_SIZE ?= "16384" IMAGE_CMD:btrfs () { size=${ROOTFS_SIZE} + image_file_maxsize=${@get_max_image_size(d, "btrfs")} + + if [[ ${image_file_maxsize} -ne 0 ]]; then + size=${image_file_maxsize} + fi + if [ ${size} -lt ${MIN_BTRFS_SIZE} ] ; then size=${MIN_BTRFS_SIZE} bbwarn "Rootfs size is too small for BTRFS. Filesystem will be extended to ${size}K" From patchwork Wed Nov 1 23:10:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charles-Antoine Couret X-Patchwork-Id: 33424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66F6DC4167D for ; Wed, 1 Nov 2023 23:11:23 +0000 (UTC) Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) by mx.groups.io with SMTP id smtpd.web10.2507.1698880282554186402 for ; Wed, 01 Nov 2023 16:11:22 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@mind.be header.s=google header.b=NpX30nO1; spf=pass (domain: essensium.com, ip: 209.85.167.52, mailfrom: charles-antoine.couret@essensium.com) Received: by mail-lf1-f52.google.com with SMTP id 2adb3069b0e04-507a29c7eefso313261e87.1 for ; Wed, 01 Nov 2023 16:11:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mind.be; s=google; t=1698880281; x=1699485081; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tPkSr7ECjACBiOXbSwkVaMT9agDEBfFY9xLGpIpwJrs=; b=NpX30nO1+iD7WIctiXoe5sDTx6vuCU6UrvQLaw+6A8oGKzJ7by8D8g93PeU0YPv+NC Cp1VVHqmIk07eLd2ZwC8DkOby9i6LucqFp9aT2iERdco0NiknxKYWlqO0l7p9pFx1aQY AbtRiQRiyf/FyApC6Iged0WG0p9t2Via52Uv2hCK9+22ZZuTvUrPwh1IN1qOF3xUkjVF Q2rUKUsTuWVfElFzX+CcKsS+KEREh4/ZFDvdJReSsKMH0qzTXZq550S2WkEQJnDApHZe g6fUPoOn63IgA7qlWNlvRU+zV1IZcxfVLFxvM5eEMjLivgebErervYtU3NDDLF9A4c3q woew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698880281; x=1699485081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tPkSr7ECjACBiOXbSwkVaMT9agDEBfFY9xLGpIpwJrs=; b=XTRd++Q+qnWLFew9QpbFzPoyQb2oXlvmOWCsgOu95jORjyPXqfmdcsNh3tld6lftzr 8WhPOakCHiTacLwBFaM276nNS+PFi02REggTnDqPjc0R5NaYp6rKq1/pP0H4HfnAoX8r pe1hY7jyXI2+ri/XstU5c0RgrQum3H2AVFy3YTx4Jx8HY2cyqcbOzVEq9Ocdra7DBVDr xS60uQ+JYdi3il4+LZUoN1csSES+ElFNoiUT8aartLO5kqzizz4aomma60iRhSjPesqO 7sercfaeFD/SiiFZOMDecqrAn+gC202JTYyV9LupHvq9weBT+FG63CwwFZIq3eeCTXdF 5AoQ== X-Gm-Message-State: AOJu0YzJ0mzzGd4CYB3OFO9Dr4E+iIlx1ys30wueQL/1Y1RiF0dSj3A4 iXJvZts8U9je6cW2W+pzSmK09JsV7+CiOmOW+Ig= X-Google-Smtp-Source: AGHT+IFKgl1wM1YDjLV65hHj96rbC1rWiGtKNeDk6tT2mUtVvWE8NXNfLBAIPslhCDcMqRpQnDvgUg== X-Received: by 2002:ac2:46c3:0:b0:507:a70e:c619 with SMTP id p3-20020ac246c3000000b00507a70ec619mr10652189lfo.67.1698880280580; Wed, 01 Nov 2023 16:11:20 -0700 (PDT) Received: from Bishop.fritz.box ([2a02:578:85c6:1101:e7a2:3f2c:a83f:5e92]) by smtp.gmail.com with ESMTPSA id m11-20020adff38b000000b0032db4e660d9sm854601wro.56.2023.11.01.16.11.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 16:11:20 -0700 (PDT) From: Charles-Antoine Couret To: openembedded-core@lists.openembedded.org Cc: Charles-Antoine Couret Subject: [PATCH 4/5 v3] image_types: use IMAGE_FILE_MAXSIZE variable for f2fs image types Date: Thu, 2 Nov 2023 00:10:53 +0100 Message-ID: <20231101231058.86928-5-charles-antoine.couret@mind.be> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231101231058.86928-1-charles-antoine.couret@mind.be> References: <20231101231058.86928-1-charles-antoine.couret@mind.be> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Nov 2023 23:11:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190052 If defined, this variable value overrides the size of f2fs partition file created by mkfs. Otherwise previous logic based on ROOTFS_SIZE variable is used. It should be set when the final file size would not be above a specific value due to fixed partitionning for example. Signed-off-by: Charles-Antoine Couret --- meta/classes-recipe/image_types.bbclass | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass index 1356552445..6e3190fa3f 100644 --- a/meta/classes-recipe/image_types.bbclass +++ b/meta/classes-recipe/image_types.bbclass @@ -272,6 +272,12 @@ IMAGE_CMD:f2fs () { # 500M the standard IMAGE_OVERHEAD_FACTOR does not work, so add additional # space here when under 500M size=${ROOTFS_SIZE} + image_file_maxsize=${@get_max_image_size(d, "f2fs")} + + if [[ "${image_file_maxsize}" -ne 0 ]]; then + size=${image_file_maxsize} + fi + if [ ${size} -lt ${MIN_F2FS_SIZE} ] ; then size=${MIN_F2FS_SIZE} bbwarn "Rootfs size is too small for F2FS. Filesystem will be extended to ${size}K" From patchwork Wed Nov 1 23:10:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charles-Antoine Couret X-Patchwork-Id: 33425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A263C001DB for ; Wed, 1 Nov 2023 23:11:23 +0000 (UTC) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by mx.groups.io with SMTP id smtpd.web10.2509.1698880282942016326 for ; Wed, 01 Nov 2023 16:11:23 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@mind.be header.s=google header.b=gPMVVuoQ; spf=pass (domain: essensium.com, ip: 209.85.128.53, mailfrom: charles-antoine.couret@essensium.com) Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-40853c639abso2442425e9.0 for ; Wed, 01 Nov 2023 16:11:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mind.be; s=google; t=1698880281; x=1699485081; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ljk/g3f+mwq6sFg3lD4Dg1L+bLMIEcW1+oVhI5Zy0Kg=; b=gPMVVuoQEwEqyNoW4lX/gv9c9x9sifBOmRgDKTHeKd1dxy1b42LIz9MnqtgIiML2eO zwiGQSZhGd5vvCRLYmXs/AnZgtr753thBQynAHWvpRF+ZsernEGsS8gStU3XUvpB//SV Kpp7tg70swvyiBGOriC2GGW+6kfmkoGmtWLs7+NryG5CxvheAEOMyleiL/XzMYlddE6u 6gR5pDP6CJxWO3F6SyXAaFjuJdTNfTdgM2DEZxGrDJRakIFFrO4ruo6rl005yRkYvnAi FRsKRu9bmkc0xcr5Q1syOXM271nsYinFsfR2qeAGDZ7a1X6uOjqlFQ/qsvltC2Az5KHX E3Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698880281; x=1699485081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ljk/g3f+mwq6sFg3lD4Dg1L+bLMIEcW1+oVhI5Zy0Kg=; b=Od54OOoFAtgZetCCabBOBorclUbr7an9gzaSsh1pk9AEh5YgUrfmc835bV7zCDGjtC q6UQrzd4fsYkIDvlu4QRJK/RflV99uEX7S31L61kXckUOaK4RZ4LC/ZZk+jZHcEC2IIm SVPwYYORczp9KJ4nPX60++K2/OlKVfvQJBR31CMI3xa4p0+cvv0Das1a+ped4KXX/P7R bA8IHo3ZECTQN5LEjvGjq4ic4Akul/Sn/9T+yt67mGGbn2WUdDnfpZspuhbrOwr5U5sr 6ccN1ugKqAm5gUq4jLH4vFWmc+DtTxNkyEUdLc4D75RUq4lPcdAyIyiKthT4ZgEsMWxk kibg== X-Gm-Message-State: AOJu0YyuUhjuWusjA9szBI+CNHZWn6reYtKJ9NoZhMOj9cqaNinuqeVf dvQk5MZ61b/aOmBXFyttXfa78Z6x8k5ChgfhofI= X-Google-Smtp-Source: AGHT+IGle/MAf2Uh4JeYzPqUv34EoYhTZ1Xqh6/c0mizcLoS7OCUjyaZY+HIBYLx143mxPrOFDJskQ== X-Received: by 2002:adf:eb0a:0:b0:32d:884b:7403 with SMTP id s10-20020adfeb0a000000b0032d884b7403mr12651625wrn.66.1698880281479; Wed, 01 Nov 2023 16:11:21 -0700 (PDT) Received: from Bishop.fritz.box ([2a02:578:85c6:1101:e7a2:3f2c:a83f:5e92]) by smtp.gmail.com with ESMTPSA id m11-20020adff38b000000b0032db4e660d9sm854601wro.56.2023.11.01.16.11.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 16:11:21 -0700 (PDT) From: Charles-Antoine Couret To: openembedded-core@lists.openembedded.org Cc: Charles-Antoine Couret Subject: [PATCH 5/5 v3] image: add check_image_max_size as post function to check file size against IMAGE_FILE_MAXSIZE Date: Thu, 2 Nov 2023 00:10:54 +0100 Message-ID: <20231101231058.86928-6-charles-antoine.couret@mind.be> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231101231058.86928-1-charles-antoine.couret@mind.be> References: <20231101231058.86928-1-charles-antoine.couret@mind.be> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Nov 2023 23:11:23 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190053 Trigger an error if the final size is above IMAGE_FILE_MAXSIZE value. Which is relevant if the partition size is fixed and the user wants to be sure that the image can be entirely installed into its partition. If the variable is not set, no error is trigger. It works for all filesystems. Signed-off-by: Charles-Antoine Couret --- meta/classes-recipe/image.bbclass | 30 +++++++++++ meta/lib/oeqa/selftest/cases/imagefeatures.py | 52 +++++++++++++++++++ 2 files changed, 82 insertions(+) diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass index 7231fad940..d2fcf078c9 100644 --- a/meta/classes-recipe/image.bbclass +++ b/meta/classes-recipe/image.bbclass @@ -509,6 +509,7 @@ python () { d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') d.prependVarFlag(task, 'postfuncs', 'create_symlinks ') + d.prependVarFlag(task, 'postfuncs', 'check_image_max_size ') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', ' DATETIME DATE ' + ' '.join(vardepsexclude)) @@ -609,6 +610,35 @@ python create_symlinks() { bb.note("Skipping symlink, source does not exist: %s -> %s" % (dst, src)) } +# +# Check if image size is lighter than maximum size +# +python check_image_max_size() { + def get_max_image_size(d, fs): + max_size = d.getVar("IMAGE_FILE_MAXSIZE:%s" % fs) + if max_size is not None: + return int(max_size) + + return None + + deploy_dir = d.getVar('IMGDEPLOYDIR') + img_name = d.getVar('IMAGE_NAME') + taskname = d.getVar("BB_CURRENTTASK") + subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or "").split() + imgsuffix = d.getVarFlag("do_" + taskname, 'imgsuffix') or d.expand("${IMAGE_NAME_SUFFIX}.") + + for type in subimages: + file_name = os.path.join(deploy_dir, img_name + imgsuffix + type) + + if os.path.exists(file_name): + file_size = os.stat(file_name).st_size / 1024 + max_size = get_max_image_size(d, type) + if max_size is not None: + if file_size > max_size: + bb.error("The image %s size %d(K) exceeds IMAGE_FILE_MAXSIZE: %d(K)" % \ + (file_name, file_size, max_size)) +} + MULTILIBRE_ALLOW_REP =. "${base_bindir}|${base_sbindir}|${bindir}|${sbindir}|${libexecdir}|${sysconfdir}|${nonarch_base_libdir}/udev|/lib/modules/[^/]*/modules.*|" MULTILIB_CHECK_FILE = "${WORKDIR}/multilib_check.py" MULTILIB_TEMP_ROOTFS = "${WORKDIR}/multilib" diff --git a/meta/lib/oeqa/selftest/cases/imagefeatures.py b/meta/lib/oeqa/selftest/cases/imagefeatures.py index dc88c222bd..afdc7a72fa 100644 --- a/meta/lib/oeqa/selftest/cases/imagefeatures.py +++ b/meta/lib/oeqa/selftest/cases/imagefeatures.py @@ -234,6 +234,58 @@ UBINIZE_ARGS_mtd_4_256 ?= "-m 4096 -p 256KiB" self.assertTrue(os.path.exists(image_path), "%s image %s doesn't exist" % (itype, image_path)) + def test_image_maxsize_success(self): + """ + Summary: Check if the image built is below maximum size if set + Expected: image is built, size below the limit and no error are reported + Product: oe-core + Author: Charles-Antoine Couret + """ + image = 'core-image-minimal' + + config = """ +IMAGE_FSTYPES += "ext4" +IMAGE_FILE_MAXSIZE:ext4 = "300000" +""" + self.write_config(config) + + res = bitbake(image) + bb_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'], image) + + image_path = os.path.join(bb_vars['DEPLOY_DIR_IMAGE'], "%s.%s" % (bb_vars['IMAGE_LINK_NAME'], "ext4")) + # check if result image is in deploy directory + self.assertTrue(os.path.exists(image_path), + "%s image %s doesn't exist" % ("ext4", image_path)) + # check if result image size is below than maximum value + self.assertTrue(os.stat(image_path).st_size / 1024 <= 300000) + # No error during execution + self.assertEqual(res.status, 0) + + def test_image_maxsize_failure(self): + """ + Summary: Check if the image built with size above limit is triggering error + Expected: the image size is above limit and triggers error + Product: oe-core + Author: Charles-Antoine Couret + """ + image = 'core-image-minimal' + + config = """ +IMAGE_FSTYPES += "ext4" +IMAGE_FILE_MAXSIZE:ext4 = "1000" +""" + self.write_config(config) + + res = bitbake(image, ignore_status=True) + bb_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'], image) + + image_path = os.path.join(bb_vars['DEPLOY_DIR_IMAGE'], "%s.%s" % (bb_vars['IMAGE_LINK_NAME'], "ext4")) + # check if result image is not generated + self.assertFalse(os.path.exists(image_path), + "%s image %s doesn't exist" % ("ext4", image_path)) + # Error triggered during execution + self.assertNotEqual(res.status, 0) + def test_useradd_static(self): config = """ USERADDEXTENSION = "useradd-staticids"