[0/3] image_types: use IMAGE_FILE_MAXSIZE variable to create fixed partition size

In case of fixed partitionning where the rootfs partition can't exceed an
amount of bytes, there is currently no automatic and no generic way to have
this requirement met in any case.

Until now, ROOTFS_SIZE value got from directory_size() does not takes into account
the size of required metadata for the filesystem itself (and does not work well
for other block size than 4k BTW).
Obviously it's a difficult task which depends on rootfs size and filesystem type.

The workaround was to set IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
to add the required extra margins. But when the final rootfs is closed to the
maximum size, it's difficult to adjust them correctly  And if you remove
or add new recipes in your image, you've to recompute these margins to have enough
space for these metadata when the rootfs is small, and to not have too big final
file when the rootfs is big.

It's cumbersome and error prone to just have a build failure when the final output
can't be flashed into the partition.

The solution is to follow how it's implemented in buildroot by having a
specific variable, here IMAGE_FILE_MAXSIZE, to create the final sparse file
and trying to fill it with the content of rootfs. If there is enough space,
margins are well compressed and does not consume space in the filesystem.
If there is no enough space, an error is triggered to warm the developer before
trying to use it in the device.

If IMAGE_FILE_MAXSIZE is not set, the idea is to keep the previous behaviour
for compatibility reason and to met other requirements.

Charles-Antoine Couret (3):
  image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types
  image_types: use IMAGE_FILE_MAXSIZE variable for btrfs image types
  image_types: use IMAGE_FILE_MAXSIZE variable for f2fs image types

 meta/classes-recipe/image_types.bbclass | 34 +++++++++++++++++++------
 1 file changed, 26 insertions(+), 8 deletions(-)

Hello,

On 04/06/2023 14:37:52+0200, Charles-Antoine Couret via lists.openembedded.org wrote:
> In case of fixed partitionning where the rootfs partition can't exceed an
> amount of bytes, there is currently no automatic and no generic way to have
> this requirement met in any case.
> 
> Until now, ROOTFS_SIZE value got from directory_size() does not takes into account
> the size of required metadata for the filesystem itself (and does not work well
> for other block size than 4k BTW).
> Obviously it's a difficult task which depends on rootfs size and filesystem type.
> 
> The workaround was to set IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
> to add the required extra margins. But when the final rootfs is closed to the
> maximum size, it's difficult to adjust them correctly  And if you remove
> or add new recipes in your image, you've to recompute these margins to have enough
> space for these metadata when the rootfs is small, and to not have too big final
> file when the rootfs is big.
> 
> It's cumbersome and error prone to just have a build failure when the final output
> can't be flashed into the partition.
> 
> The solution is to follow how it's implemented in buildroot by having a
> specific variable, here IMAGE_FILE_MAXSIZE, to create the final sparse file
> and trying to fill it with the content of rootfs. If there is enough space,
> margins are well compressed and does not consume space in the filesystem.
> If there is no enough space, an error is triggered to warm the developer before
> trying to use it in the device.
> 
> If IMAGE_FILE_MAXSIZE is not set, the idea is to keep the previous behaviour
> for compatibility reason and to met other requirements.

It would be great if you could add test cases that ensure that the
generated images are indeed fitting the size.

> 
> Charles-Antoine Couret (3):
>   image_types: use IMAGE_FILE_MAXSIZE variable for ext2/3/4 image types
>   image_types: use IMAGE_FILE_MAXSIZE variable for btrfs image types
>   image_types: use IMAGE_FILE_MAXSIZE variable for f2fs image types
> 
>  meta/classes-recipe/image_types.bbclass | 34 +++++++++++++++++++------
>  1 file changed, 26 insertions(+), 8 deletions(-)
> 
> -- 
> 2.40.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#182359): https://lists.openembedded.org/g/openembedded-core/message/182359
> Mute This Topic: https://lists.openembedded.org/mt/99320002/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On 4 Jun 2023, at 13:37, Charles-Antoine Couret via lists.openembedded.org <charles-antoine.couret=mind.be@lists.openembedded.org> wrote:
> The solution is to follow how it's implemented in buildroot by having a
> specific variable, here IMAGE_FILE_MAXSIZE, to create the final sparse file
> and trying to fill it with the content of rootfs. If there is enough space,
> margins are well compressed and does not consume space in the filesystem.
> If there is no enough space, an error is triggered to warm the developer before
> trying to use it in the device.

This seems like a useful feature but the autobuilder failed and now the patches don’t apply to master.  Will you be fixing the issues the AB exposed and rebasing?

Ross