openssl: fix occasionally install fail for parallel build

Submitted by changqing.li@windriver.com on Oct. 24, 2018, 3:30 a.m. | Patch ID: 155872

Details

Message ID 1540351835-16861-1-git-send-email-changqing.li@windriver.com
State New
Headers show

Commit Message

changqing.li@windriver.com Oct. 24, 2018, 3:30 a.m.
From: Changqing Li <changqing.li@windriver.com>

when do parallel build, like run "make install -j 48", openssl will do_install
failed with below error:
mv: cannot stat 'openssl/1.1.1-r0/image/usr/lib/libcrypto.so.1.1.new': No such file or directory
Makefile:318: recipe for target 'install_dev' failed

problem caused by target install_dev and target install_runtime, both installed
libcrypto.so.1.1, result in log:
*** Installing development files
...
install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
link /var/tmp/foo/lib/libcrypto.so -> /var/tmp/foo/lib/libcrypto.so.1.1
...
*** Installing runtime files
install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1

command in install_dev and install_runtime will have parallel competition.

have create an issue to upstream:
https://github.com/openssl/openssl/issues/7466#issuecomment-432148137

but upstream won't support this condition, so use this fix as a workaround
to fix the problem.

Signed-off-by: Changqing Li <changqing.li@windriver.com>
---
 ...-occasionally-install-fail-for-parallel-b.patch | 55 ++++++++++++++++++++++
 meta/recipes-connectivity/openssl/openssl_1.1.1.bb |  1 +
 2 files changed, 56 insertions(+)
 create mode 100644 meta/recipes-connectivity/openssl/openssl/0001-openssl-fix-occasionally-install-fail-for-parallel-b.patch

Patch hide | download patch | download mbox

diff --git a/meta/recipes-connectivity/openssl/openssl/0001-openssl-fix-occasionally-install-fail-for-parallel-b.patch b/meta/recipes-connectivity/openssl/openssl/0001-openssl-fix-occasionally-install-fail-for-parallel-b.patch
new file mode 100644
index 0000000..afb974b
--- /dev/null
+++ b/meta/recipes-connectivity/openssl/openssl/0001-openssl-fix-occasionally-install-fail-for-parallel-b.patch
@@ -0,0 +1,55 @@ 
+From e5baebbf3273ed4f9618eac5b180219e42ffe8c7 Mon Sep 17 00:00:00 2001
+From: Changqing Li <changqing.li@windriver.com>
+Date: Wed, 24 Oct 2018 10:59:02 +0800
+Subject: [PATCH] openssl: fix occasionally install fail for parallel build
+
+when do parallel build, like run "make install -j 48", openssl will do_install
+failed with below error:
+mv: cannot stat 'openssl/1.1.1-r0/image/usr/lib/libcrypto.so.1.1.new': No such file or directory
+Makefile:318: recipe for target 'install_dev' failed
+
+problem caused by target install_dev and target install_runtime, both installed
+libcrypto.so.1.1, result in log:
+*** Installing development files
+...
+install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
+link /var/tmp/foo/lib/libcrypto.so -> /var/tmp/foo/lib/libcrypto.so.1.1
+...
+*** Installing runtime files
+install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
+
+command in install_dev and install_runtime will have parallel competition.
+
+have create an issue to upstream:
+https://github.com/openssl/openssl/issues/7466#issuecomment-432148137
+
+but upstream won't support this condition, so use this fix as a workaround
+to fix the problem.
+
+Upstream-Status: Pending
+
+Signed-off-by: Changqing Li <changqing.li@windriver.com>
+---
+ Configurations/unix-Makefile.tmpl | 6 ++++--
+ 1 file changed, 4 insertions(+), 2 deletions(-)
+
+diff --git a/Configurations/unix-Makefile.tmpl b/Configurations/unix-Makefile.tmpl
+index 16af4d2..d4c7ae6 100644
+--- a/Configurations/unix-Makefile.tmpl
++++ b/Configurations/unix-Makefile.tmpl
+@@ -436,8 +436,10 @@ depend:
+ 	@: {- output_on() if $disabled{makedepend}; "" -}
+ 
+ # Install helper targets #############################################
+-
+-install_sw: all install_dev install_engines install_runtime
++intermediate: all install_dev install_engines
++install_sw:
++	@$(MAKE) intermediate
++	@$(MAKE) install_runtime
+ 
+ uninstall_sw: uninstall_runtime uninstall_engines uninstall_dev
+ 
+-- 
+2.7.4
+
diff --git a/meta/recipes-connectivity/openssl/openssl_1.1.1.bb b/meta/recipes-connectivity/openssl/openssl_1.1.1.bb
index af9038a..dcfefb5 100644
--- a/meta/recipes-connectivity/openssl/openssl_1.1.1.bb
+++ b/meta/recipes-connectivity/openssl/openssl_1.1.1.bb
@@ -15,6 +15,7 @@  SRC_URI = "http://www.openssl.org/source/openssl-${PV}.tar.gz \
            file://run-ptest \
            file://openssl-c_rehash.sh \
            file://0001-skip-test_symbol_presence.patch \
+           file://0001-openssl-fix-occasionally-install-fail-for-parallel-b.patch \
            "
 
 SRC_URI_append_class-nativesdk = " \

Comments

Richard Purdie Oct. 24, 2018, 6:04 a.m.
On Wed, 2018-10-24 at 11:30 +0800, changqing.li@windriver.com wrote:
> From: Changqing Li <changqing.li@windriver.com>
> 
> when do parallel build, like run "make install -j 48", openssl will
> do_install
> failed with below error:
> mv: cannot stat 'openssl/1.1.1-
> r0/image/usr/lib/libcrypto.so.1.1.new': No such file or directory
> Makefile:318: recipe for target 'install_dev' failed
> 
> problem caused by target install_dev and target install_runtime, both
> installed
> libcrypto.so.1.1, result in log:
> *** Installing development files
> ...
> install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> link /var/tmp/foo/lib/libcrypto.so ->
> /var/tmp/foo/lib/libcrypto.so.1.1
> ...
> *** Installing runtime files
> install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> 
> command in install_dev and install_runtime will have parallel
> competition.
> 
> have create an issue to upstream:
> https://github.com/openssl/openssl/issues/7466#issuecomment-432148137
> 
> but upstream won't support this condition, so use this fix as a
> workaround to fix the problem.

This means that:

a) Upstream don't support parallel make install
b) Reject patches fixing any bugs in parallel make install

This means the Upstream-Status should be rejected with the link above,
not pending. It also means we may need to think about not using
parallel install although since its working, its tempting to carry
patches and keep using it if it works.

It may be worth pointing out to upstream that it is mostly working for
us? I am a little worried they know something we don't about problems
this could run into.

Cheers,

Richard
changqing.li@windriver.com Oct. 24, 2018, 6:52 a.m.
On 10/24/2018 02:04 PM, Richard Purdie wrote:
> On Wed, 2018-10-24 at 11:30 +0800, changqing.li@windriver.com wrote:
>> From: Changqing Li <changqing.li@windriver.com>
>>
>> when do parallel build, like run "make install -j 48", openssl will
>> do_install
>> failed with below error:
>> mv: cannot stat 'openssl/1.1.1-
>> r0/image/usr/lib/libcrypto.so.1.1.new': No such file or directory
>> Makefile:318: recipe for target 'install_dev' failed
>>
>> problem caused by target install_dev and target install_runtime, both
>> installed
>> libcrypto.so.1.1, result in log:
>> *** Installing development files
>> ...
>> install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
>> link /var/tmp/foo/lib/libcrypto.so ->
>> /var/tmp/foo/lib/libcrypto.so.1.1
>> ...
>> *** Installing runtime files
>> install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
>>
>> command in install_dev and install_runtime will have parallel
>> competition.
>>
>> have create an issue to upstream:
>> https://github.com/openssl/openssl/issues/7466#issuecomment-432148137
>>
>> but upstream won't support this condition, so use this fix as a
>> workaround to fix the problem.
> This means that:
>
> a) Upstream don't support parallel make install
> b) Reject patches fixing any bugs in parallel make install
>
> This means the Upstream-Status should be rejected with the link above,
> not pending. It also means we may need to think about not using
> parallel install although since its working, its tempting to carry
> patches and keep using it if it works.
>
> It may be worth pointing out to upstream that it is mostly working for
> us? I am a little worried they know something we don't about problems
> this could run into.
>
> Cheers,
>
> Richard
>
>
I will point out to upstream that it is mostly working for us,  and
ask upstream suggestion about potential problem of parallel build.

if upstream think no obvious potential problem can seen,  I will change
upstream-status to reject and resend the patch.

if upstream recommend not to do parallel build,  I will add something 
like below
to stop parallel build for make install, and resend the patch, but in 
this case,  build
openssl  will very slow.

ifneq ($(filter install,$(MAKECMDGOALS)),)
.NOTPARALLEL:
endif
Khem Raj Oct. 29, 2018, 4:18 p.m.
On Tue, Oct 23, 2018 at 11:53 PM Changqing Li
<changqing.li@windriver.com> wrote:
>
>
>
> On 10/24/2018 02:04 PM, Richard Purdie wrote:
> > On Wed, 2018-10-24 at 11:30 +0800, changqing.li@windriver.com wrote:
> >> From: Changqing Li <changqing.li@windriver.com>
> >>
> >> when do parallel build, like run "make install -j 48", openssl will
> >> do_install
> >> failed with below error:
> >> mv: cannot stat 'openssl/1.1.1-
> >> r0/image/usr/lib/libcrypto.so.1.1.new': No such file or directory
> >> Makefile:318: recipe for target 'install_dev' failed
> >>
> >> problem caused by target install_dev and target install_runtime, both
> >> installed
> >> libcrypto.so.1.1, result in log:
> >> *** Installing development files
> >> ...
> >> install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> >> link /var/tmp/foo/lib/libcrypto.so ->
> >> /var/tmp/foo/lib/libcrypto.so.1.1
> >> ...
> >> *** Installing runtime files
> >> install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> >>
> >> command in install_dev and install_runtime will have parallel
> >> competition.
> >>
> >> have create an issue to upstream:
> >> https://github.com/openssl/openssl/issues/7466#issuecomment-432148137
> >>
> >> but upstream won't support this condition, so use this fix as a
> >> workaround to fix the problem.
> > This means that:
> >
> > a) Upstream don't support parallel make install
> > b) Reject patches fixing any bugs in parallel make install
> >
> > This means the Upstream-Status should be rejected with the link above,
> > not pending. It also means we may need to think about not using
> > parallel install although since its working, its tempting to carry
> > patches and keep using it if it works.
> >
> > It may be worth pointing out to upstream that it is mostly working for
> > us? I am a little worried they know something we don't about problems
> > this could run into.
> >
> > Cheers,
> >
> > Richard
> >
> >
> I will point out to upstream that it is mostly working for us,  and
> ask upstream suggestion about potential problem of parallel build.
>
> if upstream think no obvious potential problem can seen,  I will change
> upstream-status to reject and resend the patch.
>
> if upstream recommend not to do parallel build,  I will add something
> like below
> to stop parallel build for make install, and resend the patch, but in
> this case,  build
> openssl  will very slow.
>
> ifneq ($(filter install,$(MAKECMDGOALS)),)
> .NOTPARALLEL:
> endif

I think its best to convince or follow upstream here. Making install
target as serial might be best approach if there are other issues with
parallel install that upstream is uncomfortable with such a patch we
should try to find more.
Richard Purdie Oct. 29, 2018, 4:30 p.m.
On Mon, 2018-10-29 at 09:18 -0700, Khem Raj wrote:
> On Tue, Oct 23, 2018 at 11:53 PM Changqing Li
> <changqing.li@windriver.com> wrote:
> > 
> > 
> > 
> > On 10/24/2018 02:04 PM, Richard Purdie wrote:
> > > On Wed, 2018-10-24 at 11:30 +0800, changqing.li@windriver.com
> > > wrote:
> > > > From: Changqing Li <changqing.li@windriver.com>
> > > > 
> > > > when do parallel build, like run "make install -j 48", openssl
> > > > will
> > > > do_install
> > > > failed with below error:
> > > > mv: cannot stat 'openssl/1.1.1-
> > > > r0/image/usr/lib/libcrypto.so.1.1.new': No such file or
> > > > directory
> > > > Makefile:318: recipe for target 'install_dev' failed
> > > > 
> > > > problem caused by target install_dev and target
> > > > install_runtime, both
> > > > installed
> > > > libcrypto.so.1.1, result in log:
> > > > *** Installing development files
> > > > ...
> > > > install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> > > > link /var/tmp/foo/lib/libcrypto.so ->
> > > > /var/tmp/foo/lib/libcrypto.so.1.1
> > > > ...
> > > > *** Installing runtime files
> > > > install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> > > > 
> > > > command in install_dev and install_runtime will have parallel
> > > > competition.
> > > > 
> > > > have create an issue to upstream:
> > > > 
https://github.com/openssl/openssl/issues/7466#issuecomment-432148137
> > > > 
> > > > but upstream won't support this condition, so use this fix as a
> > > > workaround to fix the problem.
> > > 
> > > This means that:
> > > 
> > > a) Upstream don't support parallel make install
> > > b) Reject patches fixing any bugs in parallel make install
> > > 
> > > This means the Upstream-Status should be rejected with the link
> > > above,
> > > not pending. It also means we may need to think about not using
> > > parallel install although since its working, its tempting to
> > > carry
> > > patches and keep using it if it works.
> > > 
> > > It may be worth pointing out to upstream that it is mostly
> > > working for
> > > us? I am a little worried they know something we don't about
> > > problems
> > > this could run into.
> > > 
> > > Cheers,
> > > 
> > > Richard
> > > 
> > > 
> > 
> > I will point out to upstream that it is mostly working for us,  and
> > ask upstream suggestion about potential problem of parallel build.
> > 
> > if upstream think no obvious potential problem can seen,  I will
> > change
> > upstream-status to reject and resend the patch.
> > 
> > if upstream recommend not to do parallel build,  I will add
> > something
> > like below
> > to stop parallel build for make install, and resend the patch, but
> > in
> > this case,  build
> > openssl  will very slow.
> > 
> > ifneq ($(filter install,$(MAKECMDGOALS)),)
> > .NOTPARALLEL:
> > endif
> 
> I think its best to convince or follow upstream here. Making install
> target as serial might be best approach if there are other issues
> with
> parallel install that upstream is uncomfortable with such a patch we
> should try to find more.

If you follow the upstream discussion they originally said no, then
they are possibly reconsidering after I explained more about the
situation. I've not seen a fix land yet but they did say they are
working on one.

If anyone wants to +1 them fixing it, feel free:

https://github.com/openssl/openssl/issues/7466#issuecomment-432938981

Cheers,

Richard
Khem Raj Oct. 29, 2018, 4:40 p.m.
Cool, thanks for update I will chime in
On Mon, Oct 29, 2018 at 9:30 AM <richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2018-10-29 at 09:18 -0700, Khem Raj wrote:
> > On Tue, Oct 23, 2018 at 11:53 PM Changqing Li
> > <changqing.li@windriver.com> wrote:
> > >
> > >
> > >
> > > On 10/24/2018 02:04 PM, Richard Purdie wrote:
> > > > On Wed, 2018-10-24 at 11:30 +0800, changqing.li@windriver.com
> > > > wrote:
> > > > > From: Changqing Li <changqing.li@windriver.com>
> > > > >
> > > > > when do parallel build, like run "make install -j 48", openssl
> > > > > will
> > > > > do_install
> > > > > failed with below error:
> > > > > mv: cannot stat 'openssl/1.1.1-
> > > > > r0/image/usr/lib/libcrypto.so.1.1.new': No such file or
> > > > > directory
> > > > > Makefile:318: recipe for target 'install_dev' failed
> > > > >
> > > > > problem caused by target install_dev and target
> > > > > install_runtime, both
> > > > > installed
> > > > > libcrypto.so.1.1, result in log:
> > > > > *** Installing development files
> > > > > ...
> > > > > install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> > > > > link /var/tmp/foo/lib/libcrypto.so ->
> > > > > /var/tmp/foo/lib/libcrypto.so.1.1
> > > > > ...
> > > > > *** Installing runtime files
> > > > > install libcrypto.so.1.1 -> /var/tmp/foo/lib/libcrypto.so.1.1
> > > > >
> > > > > command in install_dev and install_runtime will have parallel
> > > > > competition.
> > > > >
> > > > > have create an issue to upstream:
> > > > >
> https://github.com/openssl/openssl/issues/7466#issuecomment-432148137
> > > > >
> > > > > but upstream won't support this condition, so use this fix as a
> > > > > workaround to fix the problem.
> > > >
> > > > This means that:
> > > >
> > > > a) Upstream don't support parallel make install
> > > > b) Reject patches fixing any bugs in parallel make install
> > > >
> > > > This means the Upstream-Status should be rejected with the link
> > > > above,
> > > > not pending. It also means we may need to think about not using
> > > > parallel install although since its working, its tempting to
> > > > carry
> > > > patches and keep using it if it works.
> > > >
> > > > It may be worth pointing out to upstream that it is mostly
> > > > working for
> > > > us? I am a little worried they know something we don't about
> > > > problems
> > > > this could run into.
> > > >
> > > > Cheers,
> > > >
> > > > Richard
> > > >
> > > >
> > >
> > > I will point out to upstream that it is mostly working for us,  and
> > > ask upstream suggestion about potential problem of parallel build.
> > >
> > > if upstream think no obvious potential problem can seen,  I will
> > > change
> > > upstream-status to reject and resend the patch.
> > >
> > > if upstream recommend not to do parallel build,  I will add
> > > something
> > > like below
> > > to stop parallel build for make install, and resend the patch, but
> > > in
> > > this case,  build
> > > openssl  will very slow.
> > >
> > > ifneq ($(filter install,$(MAKECMDGOALS)),)
> > > .NOTPARALLEL:
> > > endif
> >
> > I think its best to convince or follow upstream here. Making install
> > target as serial might be best approach if there are other issues
> > with
> > parallel install that upstream is uncomfortable with such a patch we
> > should try to find more.
>
> If you follow the upstream discussion they originally said no, then
> they are possibly reconsidering after I explained more about the
> situation. I've not seen a fix land yet but they did say they are
> working on one.
>
> If anyone wants to +1 them fixing it, feel free:
>
> https://github.com/openssl/openssl/issues/7466#issuecomment-432938981
>
> Cheers,
>
> Richard
>
>
>