[master,v2] image_types: Add 7-Zip support in conversion types and commands

s=20181004; d=freemail.hu;

	h=From:To:Cc:Subject:Date:MIME-Version:Content-Type:Content-Transfer-Encoding;

	l=3376; bh=04eYxfqiZL3vrO/7ppKX0DrhEm26mc8zxfsQxtyIEvA=;

	b=J1FpU65DNnm3ThXCeKWpTAuyYDI/CActnIhVRoihZbPwBUIwlOgcE/WbHbZ4vEmU

	be/a+/S+xV/JA4UWZgTi9/XzDy9KkMwfUnygR6ynUxvVubvpLBeCJMj4Nzor/pD+8RY

	pKPOomavveo7X7+y6beSB3OvCifBKNq9U9A940dJS8MRNQ2Fz5+qtvePxOce9QPuWF3

	NlgS/JI0MvGriwwksVZe4O2d8kP5kq1puXhIgXKbVOwXq8G3M3abuqwKP6eeggHYuGl

	tLmEt5OmpFFn6uiOT+tEZ349Sdyt8560IAcGBBOh9VN2gd5ZQMwl4dAc1ebw8wFeGsc

	UC+cosaPOA==
Content-Transfer-Encoding: quoted-printable

From: Benjamin Sz=C5=91ke <egyszeregy@freemail.hu>

Add 7-Zip support in conversion types and commands. Default arguments of =
7-Zip are compression level: 9, method: BZip2, extension: zip. All these =
are overridable from local.con for any other use case in user side. First=
 goal was to make an alternative zip conversion which can operate faster =
in multi-thread systems (on many CPU core), normal zip is just a single t=
hread compression. 7-Zip can do a zip compression in multi-thread process=
ing, so it is fine to use and speed up it.
---
 meta/classes-recipe/image_types.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

 qcow2 base64 gzsync zsync ${COMPRESSIONTYPES}"
 CONVERSION_CMD:lzma =3D "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}=
.${type}"
 CONVERSION_CMD:gz =3D "gzip -f -9 -n -c --rsyncable ${IMAGE_NAME}${IMAGE=
_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 CONVERSION_CMD:bz2 =3D "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$=
{type}"
@@ -312,6 +316,7 @@ CONVERSION_CMD:xz =3D "xz -f -k -c ${XZ_COMPRESSION_L=
EVEL} ${XZ_DEFAULTS} --check=3D
 CONVERSION_CMD:lz4 =3D "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$=
{type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
 CONVERSION_CMD:lzo =3D "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type=
}"
 CONVERSION_CMD:zip =3D "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAG=
E_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD:7zip =3D "7za a -mx=3D${7ZIP_COMPRESSION_LEVEL} -mm=3D${7=
ZIP_COMPRESSION_METHOD} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.${7ZIP_=
EXTENSION} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD:zst =3D "zstd -f -k -T0 -c ${ZSTD_COMPRESSION_LEVEL} ${IM=
AGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}=
.${type}.zst"
 CONVERSION_CMD:sum =3D "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${t=
ype} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_A=
RGS}"
 CONVERSION_CMD:md5sum =3D "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${ty=
pe} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
@@ -337,6 +342,7 @@ CONVERSION_DEPENDS_xz =3D "xz-native"
 CONVERSION_DEPENDS_lz4 =3D "lz4-native"
 CONVERSION_DEPENDS_lzo =3D "lzop-native"
 CONVERSION_DEPENDS_zip =3D "zip-native"
+CONVERSION_DEPENDS_7zip =3D "p7zip-native"
 CONVERSION_DEPENDS_zst =3D "zstd-native"
 CONVERSION_DEPENDS_sum =3D "mtd-utils-native"
 CONVERSION_DEPENDS_bmap =3D "bmap-tools-native"
--=20
2.38.1.windows.1

Hi Benjamin,

On 11/6/22 14:31, Livius wrote:
> From: Benjamin Szőke <egyszeregy@freemail.hu>
> 
> Add 7-Zip support in conversion types and commands. Default arguments of 7-Zip are compression level: 9, method: BZip2, extension: zip. All these are overridable from local.con for any other use case in user side. First goal was to make an alternative zip conversion which can operate faster in multi-thread systems (on many CPU core), normal zip is just a single thread compression. 7-Zip can do a zip compression in multi-thread processing, so it is fine to use and speed up it.

Your signed-off-by is missing here, it is usually added when you run
git commit --signoff
git commit -s

You can add your signature to your patch locally with
git commit --signoff --amend --no-edit

> ---
>   meta/classes-recipe/image_types.bbclass | 8 +++++++-
>   1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
> index 764e6a5574..f70c71b9d4 100644
> --- a/meta/classes-recipe/image_types.bbclass
> +++ b/meta/classes-recipe/image_types.bbclass
> @@ -59,6 +59,10 @@ XZ_INTEGRITY_CHECK ?= "crc32"
>   
>   ZIP_COMPRESSION_LEVEL ?= "-9"
>   
> +7ZIP_COMPRESSION_LEVEL ?= "9"
> +7ZIP_COMPRESSION_METHOD ?= "BZip2"
> +7ZIP_EXTENSION ?= "zip"

Wouldn't this conflict with the output of the zip compression? 
Basically, if I have 7zip and zip for an archive, they will overwrite 
each other un-deterministically. Maybe the default .7z extension is better?

> +
>   ZSTD_COMPRESSION_LEVEL ?= "-3"
>   
>   JFFS2_SUM_EXTRA_ARGS ?= ""
> @@ -304,7 +308,7 @@ IMAGE_TYPES:append:x86-64 = " hddimg iso"
>   # CONVERSION_CMD/DEPENDS.
>   COMPRESSIONTYPES ?= ""
>   
> -CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vhd vhdx vdi qcow2 base64 gzsync zsync ${COMPRESSIONTYPES}"
> +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip 7zip zst sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vhd vhdx vdi qcow2 base64 gzsync zsync ${COMPRESSIONTYPES}"
>   CONVERSION_CMD:lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>   CONVERSION_CMD:gz = "gzip -f -9 -n -c --rsyncable ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>   CONVERSION_CMD:bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> @@ -312,6 +316,7 @@ CONVERSION_CMD:xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_DEFAULTS} --check=
>   CONVERSION_CMD:lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>   CONVERSION_CMD:lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>   CONVERSION_CMD:zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> +CONVERSION_CMD:7zip = "7za a -mx=${7ZIP_COMPRESSION_LEVEL} -mm=${7ZIP_COMPRESSION_METHOD} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.${7ZIP_EXTENSION} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>   CONVERSION_CMD:zst = "zstd -f -k -T0 -c ${ZSTD_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
>   CONVERSION_CMD:sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
>   CONVERSION_CMD:md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
> @@ -337,6 +342,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
>   CONVERSION_DEPENDS_lz4 = "lz4-native"
>   CONVERSION_DEPENDS_lzo = "lzop-native"
>   CONVERSION_DEPENDS_zip = "zip-native"
> +CONVERSION_DEPENDS_7zip = "p7zip-native"

This tool is in meta-openembedded/meta-oe and not openembedded-core, 
should we actually move the recipe into oe-core?

Cheers,
Quentin

On Mon, 7 Nov 2022 at 10:46, Quentin Schulz via lists.openembedded.org
<quentin.schulz=theobroma-systems.com@lists.openembedded.org> wrote:
> > +CONVERSION_DEPENDS_7zip = "p7zip-native"
>
> This tool is in meta-openembedded/meta-oe and not openembedded-core,
> should we actually move the recipe into oe-core?

I don't think so. There's plenty of references in oe-core recipes to
recipes that are not in core, as long as they're not used by default.

Alex

Good point for this zip extension conflict, you are right. If somebody is a overzealous developer can use IMAGE_TYPES = "wic.zip wic.7zip" then sure there will be an extension conflict after the compression in deploy time. This escaped my notice this because this use case is not so realistic (normaly enough only one zip pack for wic image) but can be a real issue. 100% sure 7ZIP_EXTENSION ?= 7z can be better for default value, then it can be overridable to .zip extension in users's local.conf if needed (for example for me it .zip will be better).

Do i need to resend again the patch in a new thread/topic if i like to solve the missing signed-off issue?

Hi Benjamin,

On 11/7/22 12:33, Livius wrote:
> Good point for this zip extension conflict, you are right. If somebody is a overzealous developer can use IMAGE_TYPES = "wic.zip wic.7zip" then sure there will be an extension conflict after the compression in deploy time. This escaped my notice this because this use case is not so realistic (normaly enough only one zip pack for wic image) but can be a real issue. 100% sure 7ZIP_EXTENSION ?= 7z can be better for default value, then it can be overridable to .zip extension in users's local.conf if needed (for example for me it .zip will be better).
> 
> Do i need to resend again the patch in a new thread/topic if i like to solve the missing signed-off issue?
> 

Yes, please make it a v2 also (git format-patch -v 2 when creating the 
patch before sending it with git-send-email).

Cheers,
Quentin