[bitbake-devel,2/5] opkg-utils: Fix silent empty/broken opkg package creation

Submitted by Richard Purdie on Nov. 21, 2019, 8:54 p.m. | Patch ID: 167265

Details

Message ID 20191121205415.25946-2-richard.purdie@linuxfoundation.org
State Master Next
Commit 163f3400a84cb6d0b8c7202288ffb5e0e298095b
Headers show

Commit Message

Richard Purdie Nov. 21, 2019, 8:54 p.m.
opkg-build was failing on hosts where tar < 1.28 and reproducibile builds
were enabled but it was doing this silently and generating corrupted
(empty) ipk files. Add a fix for this (submitted upstream).

The fix requires bash but if you're building ipk files this shoudn't be
a problem.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../opkg-utils/opkg-utils/pipefail.patch      | 31 +++++++++++++++++++
 .../opkg-utils/opkg-utils_0.4.1.bb            |  3 ++
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch

Patch hide | download patch | download mbox

diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
new file mode 100644
index 00000000000..55ddcc1fd20
--- /dev/null
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
@@ -0,0 +1,31 @@ 
+We need opkg-build to fail if for example the tar command is passed invalid 
+options. Without this, we see silently created empty packaged where data.tar
+is zero bytes in size. This creates hard to debug problems.
+
+An example is when reproducible builds are enabled and run on old hosts like
+centos7 which has tar < 1.28:
+
+Subprocess output:tar: unrecognized option '--clamp-mtime'
+Try `tar --help' or `tar --usage' for more information.
+
+Upstream-Status: Pending
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+
+Index: opkg-utils-0.4.1/opkg-build
+===================================================================
+--- opkg-utils-0.4.1.orig/opkg-build
++++ opkg-utils-0.4.1/opkg-build
+@@ -1,4 +1,4 @@
+-#!/bin/sh
++#!/bin/bash
+ 
+ : <<=cut
+ =head1 NAME
+@@ -12,6 +12,7 @@ opkg-build - construct an .opk from a di
+ #   Updated to work on Familiar Pre0.7rc1, with busybox tar.
+ #   Note it Requires: binutils-ar (since the busybox ar can't create)
+ set -e
++set -o pipefail
+ 
+ version=1.0
+ 
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
index cf1e4670c65..eb6c7a3a6ad 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
@@ -10,6 +10,7 @@  PROVIDES += "${@bb.utils.contains('PACKAGECONFIG', 'update-alternatives', 'virtu
 SRC_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/${BPN}/snapshot/${BPN}-${PV}.tar.gz \
            file://0001-Switch-all-scripts-to-use-Python-3.x.patch \
            file://0001-opkg-build-clamp-mtimes-to-SOURCE_DATE_EPOCH.patch \
+           file://pipefail.patch \
 "
 UPSTREAM_CHECK_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/refs/"
 
@@ -19,6 +20,8 @@  SRC_URI[sha256sum] = "9ea9efdd9fe13661ad251e3a2860c1c93045adcfaa6659c3e86d9748ec
 
 TARGET_CC_ARCH += "${LDFLAGS}"
 
+RDEPENDS_${PN} += "bash"
+
 # For native builds we use the host Python
 PYTHONRDEPS = "python3 python3-shell python3-io python3-math python3-crypt python3-logging python3-fcntl python3-pickle python3-compression python3-stringold"
 PYTHONRDEPS_class-native = ""

Comments

Khem Raj Nov. 22, 2019, 12:38 a.m.
On Thu, Nov 21, 2019 at 12:54 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> opkg-build was failing on hosts where tar < 1.28 and reproducibile builds
> were enabled but it was doing this silently and generating corrupted
> (empty) ipk files. Add a fix for this (submitted upstream).
>
> The fix requires bash but if you're building ipk files this shoudn't be
> a problem.

is the problem similar to this one

http://cfajohnson.com/shell/cus-faq-2.html#Q11

does it impact build times when default shell is not bash ?

>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  .../opkg-utils/opkg-utils/pipefail.patch      | 31 +++++++++++++++++++
>  .../opkg-utils/opkg-utils_0.4.1.bb            |  3 ++
>  2 files changed, 34 insertions(+)
>  create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
>
> diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
> new file mode 100644
> index 00000000000..55ddcc1fd20
> --- /dev/null
> +++ b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
> @@ -0,0 +1,31 @@
> +We need opkg-build to fail if for example the tar command is passed invalid
> +options. Without this, we see silently created empty packaged where data.tar
> +is zero bytes in size. This creates hard to debug problems.
> +
> +An example is when reproducible builds are enabled and run on old hosts like
> +centos7 which has tar < 1.28:
> +
> +Subprocess output:tar: unrecognized option '--clamp-mtime'
> +Try `tar --help' or `tar --usage' for more information.
> +
> +Upstream-Status: Pending
> +Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> +
> +Index: opkg-utils-0.4.1/opkg-build
> +===================================================================
> +--- opkg-utils-0.4.1.orig/opkg-build
> ++++ opkg-utils-0.4.1/opkg-build
> +@@ -1,4 +1,4 @@
> +-#!/bin/sh
> ++#!/bin/bash
> +
> + : <<=cut
> + =head1 NAME
> +@@ -12,6 +12,7 @@ opkg-build - construct an .opk from a di
> + #   Updated to work on Familiar Pre0.7rc1, with busybox tar.
> + #   Note it Requires: binutils-ar (since the busybox ar can't create)
> + set -e
> ++set -o pipefail
> +
> + version=1.0
> +
> diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
> index cf1e4670c65..eb6c7a3a6ad 100644
> --- a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
> +++ b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
> @@ -10,6 +10,7 @@ PROVIDES += "${@bb.utils.contains('PACKAGECONFIG', 'update-alternatives', 'virtu
>  SRC_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/${BPN}/snapshot/${BPN}-${PV}.tar.gz \
>             file://0001-Switch-all-scripts-to-use-Python-3.x.patch \
>             file://0001-opkg-build-clamp-mtimes-to-SOURCE_DATE_EPOCH.patch \
> +           file://pipefail.patch \
>  "
>  UPSTREAM_CHECK_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/refs/"
>
> @@ -19,6 +20,8 @@ SRC_URI[sha256sum] = "9ea9efdd9fe13661ad251e3a2860c1c93045adcfaa6659c3e86d9748ec
>
>  TARGET_CC_ARCH += "${LDFLAGS}"
>
> +RDEPENDS_${PN} += "bash"
> +
>  # For native builds we use the host Python
>  PYTHONRDEPS = "python3 python3-shell python3-io python3-math python3-crypt python3-logging python3-fcntl python3-pickle python3-compression python3-stringold"
>  PYTHONRDEPS_class-native = ""
> --
> 2.20.1
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Richard Purdie Nov. 22, 2019, 9:03 a.m.
On Thu, 2019-11-21 at 16:38 -0800, Khem Raj wrote:
> On Thu, Nov 21, 2019 at 12:54 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > opkg-build was failing on hosts where tar < 1.28 and reproducibile
> > builds
> > were enabled but it was doing this silently and generating
> > corrupted
> > (empty) ipk files. Add a fix for this (submitted upstream).
> > 
> > The fix requires bash but if you're building ipk files this
> > shoudn't be
> > a problem.
> 
> is the problem similar to this one
> 
> http://cfajohnson.com/shell/cus-faq-2.html#Q11

Yes.

> does it impact build times when default shell is not bash ?

I doubt very much whether it makes much difference at all. Its a single
execution per package of a script which takes a comparatively long
time. 

There are multiple places pipelines are used in the script and it would
be fairly invasive to try and use tricks like the ones in the link
above.

If it were something which looped tightly or has a short execution
time, it would make a difference but that isn't the case.

Cheers,

Richard