[1/5] RFC image_types.bbclass: add image size limit for tar image type

Submitted by Mikko Rapeli on Nov. 29, 2018, 12:21 p.m. | Patch ID: 156747

Details

Message ID 1543494097-19534-2-git-send-email-mikko.rapeli@bmw.de
State New
Headers show

Commit Message

Mikko Rapeli Nov. 29, 2018, 12:21 p.m.
If IMAGE_ROOTFS_SIZE_LIMIT is defined in image configuration, then
build will fail if for tar image type the uncompressed tar ball size
exceeds the value, as reported by 'du -ks'.

This check could be extended to other image types as well.

There already exists a check with IMAGE_ROOTFS_SIZE variable
but I could not figure out how to actually use it. It does
some quite complex overhead calculations which did not seem
to work for me on sumo.

When the image size is exceeded, build fails like this:

ERROR: image-1.0-r0 do_image_tar: Image size 170712 of /home/builder/src/base/build/tmp/work/linux/image/1.0-r0/deploy-image-complete/image.rootfs.tar reported by 'du -ks' is larger than limit IMAGE_ROOTFS_SIZE_LIMIT 170000

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
---
 meta/classes/image.bbclass       |  2 +-
 meta/classes/image_types.bbclass | 11 ++++++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)

Patch hide | download patch | download mbox

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 276d0d3..34a567f 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -127,7 +127,7 @@  def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS', 'IMAGE_ROOTFS_SIZE_LIMIT']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 5c40648..0481703 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -125,7 +125,16 @@  IMAGE_CMD_squashfs-lz4 = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM
 # required when extracting, but it seems prudent to use it in both cases.
 IMAGE_CMD_TAR ?= "tar"
 # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
-IMAGE_CMD_tar = "${IMAGE_CMD_TAR} --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+IMAGE_CMD_tar() {
+	set -ex
+	"${IMAGE_CMD_TAR}" --numeric-owner -cf "${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar" -C "${IMAGE_ROOTFS}" . || [ $? -eq 1 ]
+	# fail build if IMAGE_ROOTFS_SIZE_LIMIT is exceeded
+	if [ -n "${IMAGE_ROOTFS_SIZE_LIMIT}" ]; then
+		imagesize=$( du -ks "${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar" | awk '{ print $1 }' )
+		[ "${imagesize}" -gt "${IMAGE_ROOTFS_SIZE_LIMIT}" ] && \
+		bberror "Image size ${imagesize} of ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar reported by 'du -ks' is larger than limit IMAGE_ROOTFS_SIZE_LIMIT ${IMAGE_ROOTFS_SIZE_LIMIT}"
+	fi
+}
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {

Comments

Richard Purdie Nov. 29, 2018, 2:04 p.m.
On Thu, 2018-11-29 at 14:21 +0200, Mikko Rapeli wrote:
> If IMAGE_ROOTFS_SIZE_LIMIT is defined in image configuration, then
> build will fail if for tar image type the uncompressed tar ball size
> exceeds the value, as reported by 'du -ks'.
> 
> This check could be extended to other image types as well.
> 
> There already exists a check with IMAGE_ROOTFS_SIZE variable
> but I could not figure out how to actually use it. It does
> some quite complex overhead calculations which did not seem
> to work for me on sumo.
> 
> When the image size is exceeded, build fails like this:
> 
> ERROR: image-1.0-r0 do_image_tar: Image size 170712 of
> /home/builder/src/base/build/tmp/work/linux/image/1.0-r0/deploy-
> image-complete/image.rootfs.tar reported by 'du -ks' is larger than
> limit IMAGE_ROOTFS_SIZE_LIMIT 170000

What about IMAGE_ROOTFS_MAXSIZE?

I think IMAGE_ROOTFS_SIZE is about adding extra space to the image, not
limiting its size...

Cheers,

Richard
Mikko Rapeli Nov. 29, 2018, 2:17 p.m.
On Thu, Nov 29, 2018 at 02:04:14PM +0000, Richard Purdie wrote:
> On Thu, 2018-11-29 at 14:21 +0200, Mikko Rapeli wrote:
> > If IMAGE_ROOTFS_SIZE_LIMIT is defined in image configuration, then
> > build will fail if for tar image type the uncompressed tar ball size
> > exceeds the value, as reported by 'du -ks'.
> > 
> > This check could be extended to other image types as well.
> > 
> > There already exists a check with IMAGE_ROOTFS_SIZE variable
> > but I could not figure out how to actually use it. It does
> > some quite complex overhead calculations which did not seem
> > to work for me on sumo.
> > 
> > When the image size is exceeded, build fails like this:
> > 
> > ERROR: image-1.0-r0 do_image_tar: Image size 170712 of
> > /home/builder/src/base/build/tmp/work/linux/image/1.0-r0/deploy-
> > image-complete/image.rootfs.tar reported by 'du -ks' is larger than
> > limit IMAGE_ROOTFS_SIZE_LIMIT 170000
> 
> What about IMAGE_ROOTFS_MAXSIZE?
>
> I think IMAGE_ROOTFS_SIZE is about adding extra space to the image, not
> limiting its size...

Yea, forgot to mention that I tried that too but got inconsisten results.
I know it's bad but it was easier for me to add this new test than to figure
out what's wrong with current yocto image size checks. Hence RFC.

-Mikko

> Cheers,
> 
> Richard
Christopher Larson Nov. 29, 2018, 3:21 p.m.
This seems like it’d make a good general image qa check instead, rather
than being part of do_image_tar.

On Thu, Nov 29, 2018 at 7:29 AM <Mikko.Rapeli@bmw.de> wrote:

> On Thu, Nov 29, 2018 at 02:04:14PM +0000, Richard Purdie wrote:
> > On Thu, 2018-11-29 at 14:21 +0200, Mikko Rapeli wrote:
> > > If IMAGE_ROOTFS_SIZE_LIMIT is defined in image configuration, then
> > > build will fail if for tar image type the uncompressed tar ball size
> > > exceeds the value, as reported by 'du -ks'.
> > >
> > > This check could be extended to other image types as well.
> > >
> > > There already exists a check with IMAGE_ROOTFS_SIZE variable
> > > but I could not figure out how to actually use it. It does
> > > some quite complex overhead calculations which did not seem
> > > to work for me on sumo.
> > >
> > > When the image size is exceeded, build fails like this:
> > >
> > > ERROR: image-1.0-r0 do_image_tar: Image size 170712 of
> > > /home/builder/src/base/build/tmp/work/linux/image/1.0-r0/deploy-
> > > image-complete/image.rootfs.tar reported by 'du -ks' is larger than
> > > limit IMAGE_ROOTFS_SIZE_LIMIT 170000
> >
> > What about IMAGE_ROOTFS_MAXSIZE?
> >
> > I think IMAGE_ROOTFS_SIZE is about adding extra space to the image, not
> > limiting its size...
>
> Yea, forgot to mention that I tried that too but got inconsisten results.
> I know it's bad but it was easier for me to add this new test than to
> figure
> out what's wrong with current yocto image size checks. Hence RFC.
>
> -Mikko
>
> > Cheers,
> >
> > Richard
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
Richard Purdie Nov. 29, 2018, 4:34 p.m.
On Thu, 2018-11-29 at 14:17 +0000, Mikko.Rapeli@bmw.de wrote:
> On Thu, Nov 29, 2018 at 02:04:14PM +0000, Richard Purdie wrote:
> > On Thu, 2018-11-29 at 14:21 +0200, Mikko Rapeli wrote:
> > > If IMAGE_ROOTFS_SIZE_LIMIT is defined in image configuration,
> > > then
> > > build will fail if for tar image type the uncompressed tar ball
> > > size
> > > exceeds the value, as reported by 'du -ks'.
> > > 
> > > This check could be extended to other image types as well.
> > > 
> > > There already exists a check with IMAGE_ROOTFS_SIZE variable
> > > but I could not figure out how to actually use it. It does
> > > some quite complex overhead calculations which did not seem
> > > to work for me on sumo.
> > > 
> > > When the image size is exceeded, build fails like this:
> > > 
> > > ERROR: image-1.0-r0 do_image_tar: Image size 170712 of
> > > /home/builder/src/base/build/tmp/work/linux/image/1.0-r0/deploy-
> > > image-complete/image.rootfs.tar reported by 'du -ks' is larger
> > > than
> > > limit IMAGE_ROOTFS_SIZE_LIMIT 170000
> > 
> > What about IMAGE_ROOTFS_MAXSIZE?
> > 
> > I think IMAGE_ROOTFS_SIZE is about adding extra space to the image,
> > not
> > limiting its size...
> 
> Yea, forgot to mention that I tried that too but got inconsisten
> results.
> I know it's bad but it was easier for me to add this new test than to
> figure out what's wrong with current yocto image size checks. Hence
> RFC.

How would we document this new variable? Recommend users to set both
and hope for the best? :)

Seriously, we need one good way of doing this which works. That means
either you debug the IMAGE_ROOTFS_MAXSIZE option or at least file a bug
with as much information as you can about the problems you're seeing...

Cheers,

Richard