Patchwork [1/1] webkit-gtk: limit ld memory requirement

login
register
mail settings
Submitter Joe Slater
Date Sept. 13, 2013, 11:28 p.m.
Message ID <1379114920-23851-1-git-send-email-jslater@windriver.com>
Download mbox | patch
Permalink /patch/58011/
State Accepted
Commit 1146eeb5b79e7437718495ee820bde059a75db3c
Headers show

Comments

Joe Slater - Sept. 13, 2013, 11:28 p.m.
Add --no-keep-memory to LDFLAGS.

Signed-off-by: Joe Slater <jslater@windriver.com>
---
 meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
Khem Raj - Sept. 14, 2013, 4:35 a.m.
On Friday, September 13, 2013, Joe Slater wrote:

> Add --no-keep-memory to LDFLAGS.


It does not come with out a price. Have you measured how much performance
degradation it brings in ?
We need that to assess the trade off and also describe the machine
configuration where this option is helping


>
> Signed-off-by: Joe Slater <jslater@windriver.com <javascript:;>>
> ---
>  meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
>
> diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bbb/meta/recipes-sato/webkit/
> webkit-gtk_1.8.3.bb
> index 5691d3f..9304243 100644
> --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> @@ -58,6 +58,10 @@ CPPFLAGS_append_powerpc = "
> -I${STAGING_INCDIR}/pango-1.0 \
>                              -I${STAGING_LIBDIR}/glib-2.0/include \
>                              -I${STAGING_INCDIR}/glib-2.0"
>
> +# ld can run out of memory linking libwebkitgtk!
> +#
> +LDFLAGS += "-Wl,--no-keep-memory"
> +
>  EXTRA_AUTORECONF = " -I Source/autotools "
>
>
> --
> 1.7.3.4
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org <javascript:;>
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
Martin Jansa - Sept. 14, 2013, 8:08 a.m.
On Fri, Sep 13, 2013 at 09:35:36PM -0700, Khem Raj wrote:
> On Friday, September 13, 2013, Joe Slater wrote:
> 
> > Add --no-keep-memory to LDFLAGS.
> 

I think it was discussed on this ML before and conclusion was that it's
distro/builder policy and shouldn't be default in the recipe.
 
> It does not come with out a price. Have you measured how much performance
> degradation it brings in ?
> We need that to assess the trade off and also describe the machine
> configuration where this option is helping
> 
> 
> >
> > Signed-off-by: Joe Slater <jslater@windriver.com <javascript:;>>
> > ---
> >  meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb |    4 ++++
> >  1 files changed, 4 insertions(+), 0 deletions(-)
> >
> > diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bbb/meta/recipes-sato/webkit/
> > webkit-gtk_1.8.3.bb
> > index 5691d3f..9304243 100644
> > --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> > +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> > @@ -58,6 +58,10 @@ CPPFLAGS_append_powerpc = "
> > -I${STAGING_INCDIR}/pango-1.0 \
> >                              -I${STAGING_LIBDIR}/glib-2.0/include \
> >                              -I${STAGING_INCDIR}/glib-2.0"
> >
> > +# ld can run out of memory linking libwebkitgtk!
> > +#
> > +LDFLAGS += "-Wl,--no-keep-memory"
> > +
> >  EXTRA_AUTORECONF = " -I Source/autotools "
> >
> >
> > --
> > 1.7.3.4
> >
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org <javascript:;>
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >

> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
Saul Wold - Sept. 15, 2013, 12:39 a.m.
On 09/14/2013 12:10 PM, Burton, Ross wrote:
> On 14 September 2013 05:35, Khem Raj <raj.khem@gmail.com> wrote:
>> It does not come with out a price. Have you measured how much performance
>> degradation it brings in ?
>> We need that to assess the trade off and also describe the machine
>> configuration where this option is helping
>
> Agreed - there's a bug which is tracking this from Kai.  As far as I'm
> aware the failure cases are 32-bit hosts (3GB limit per process) or
> 64-bit hosts with limited memory.  I suggested that we benchmark with
> and without this - if the performance hit is measured in seconds then
> we can probably enable it for this recipe out of the box.
>
Joe / Kai:

Please test as Khem and Ross suggest and report back with the results, 
also please include the [YOCTO #5160].

Ross, I am going to assign this to Joe.

Sau!

> Ross
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
Kang Kai - Sept. 17, 2013, 5:11 a.m.
On 2013?09?15? 08:39, Saul Wold wrote:
> On 09/14/2013 12:10 PM, Burton, Ross wrote:
>> On 14 September 2013 05:35, Khem Raj <raj.khem@gmail.com> wrote:
>>> It does not come with out a price. Have you measured how much 
>>> performance
>>> degradation it brings in ?
>>> We need that to assess the trade off and also describe the machine
>>> configuration where this option is helping
>>
>> Agreed - there's a bug which is tracking this from Kai. As far as I'm
>> aware the failure cases are 32-bit hosts (3GB limit per process) or
>> 64-bit hosts with limited memory. I suggested that we benchmark with
>> and without this - if the performance hit is measured in seconds then
>> we can probably enable it for this recipe out of the box.
>>
> Joe / Kai:
>
> Please test as Khem and Ross suggest and report back with the results, 
> also please include the [YOCTO #5160].

run command 'time bitbake webkit-gtk -c compile' on the host which failed:

without patch till compile fails:
real 49m2.496s
user 102m47.789s
sys 31m18.698s

with the patch:
real 44m9.000s
user 106m45.182s
sys 30m53.223s


On the host build doesn't fail:

Without patch:
real 76m1.646s
user 86m36.704s
sys 38m39.103s

With patch:
real 63m36.193s
user 86m9.274s
sys 37m48.431s

It seems the link option doesn't increase the compile time.

Regards,
Kai

>
> Ross, I am going to assign this to Joe.
>
> Sau!
>
>> Ross
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>>
>
>
Khem Raj - Sept. 17, 2013, 5:31 a.m.
On Mon, Sep 16, 2013 at 10:11 PM, Kang Kai <Kai.Kang@windriver.com> wrote:
> Without patch:
> real 76m1.646s
> user 86m36.704s
> sys 38m39.103s
>
> With patch:
> real 63m36.193s
> user 86m9.274s
> sys 37m48.431s
>
> It seems the link option doesn't increase the compile time.

on the contrary its reducing the build time which is kind of
interesting and seems a win-win. are you able to extract the failing
link command and just run that with/without the linker option to
discard memory ? and measure just the linking time on the host where
it already builds without this option?
Martin Jansa - Sept. 17, 2013, 10:30 a.m.
On Mon, Sep 16, 2013 at 10:31:18PM -0700, Khem Raj wrote:
> On Mon, Sep 16, 2013 at 10:11 PM, Kang Kai <Kai.Kang@windriver.com> wrote:
> > Without patch:
> > real 76m1.646s
> > user 86m36.704s
> > sys 38m39.103s
> >
> > With patch:
> > real 63m36.193s
> > user 86m9.274s
> > sys 37m48.431s
> >
> > It seems the link option doesn't increase the compile time.
> 
> on the contrary its reducing the build time which is kind of
> interesting and seems a win-win. are you able to extract the failing
> link command and just run that with/without the linker option to
> discard memory ? and measure just the linking time on the host where
> it already builds without this option?

I think someone should measure it on machine with a lot of ram. If the
test was executed on machine where it was linking mostly from swap,
then it's not so surprising it was faster with the option enabled.
Ross Burton - Sept. 17, 2013, 10:53 a.m.
On 17 September 2013 11:30, Martin Jansa <martin.jansa@gmail.com> wrote:
> I think someone should measure it on machine with a lot of ram. If the
> test was executed on machine where it was linking mostly from swap,
> then it's not so surprising it was faster with the option enabled.

My very rough test says no significant difference between the option
being enabled or disabled (~23m +- 30s each way).  i7 with 16GB RAM.

Anyone else want to add a data point?

Ross
Slater, Joseph - Sept. 23, 2013, 9:51 p.m.
> -----Original Message-----
> From: Burton, Ross [mailto:ross.burton@intel.com]
> Sent: Tuesday, September 17, 2013 3:53 AM
> To: Martin Jansa
> Cc: Khem Raj; Slater, Joseph; Patches and discussions about the oe-core layer
> Subject: Re: [OE-core] [PATCH 1/1] webkit-gtk: limit ld memory requirement
> 
> On 17 September 2013 11:30, Martin Jansa <martin.jansa@gmail.com> wrote:
> > I think someone should measure it on machine with a lot of ram. If the
> > test was executed on machine where it was linking mostly from swap,
> > then it's not so surprising it was faster with the option enabled.
> 
> My very rough test says no significant difference between the option
> being enabled or disabled (~23m +- 30s each way).  i7 with 16GB RAM.
> 
> Anyone else want to add a data point?

I am the only user on the machine below.  I compiled twice, each way --

model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz

MemTotal:       12326492 kB


build - present recipe
----------------------
real	48m26.620s
user	96m15.939s
sys	17m43.639s

compile - present recipe
------------------------
real	25m39.845s
user	85m31.102s
sys	10m17.016s

real	25m31.344s
user	85m35.210s
sys	9m46.309s

compile - new recipe
------------------------
real	25m28.338s
user	85m27.088s
sys	10m19.696s

real	25m22.355s
user	85m24.133s
sys	10m13.151s





> 
> Ross
Ross Burton - Sept. 24, 2013, 4:34 p.m.
On 23 September 2013 22:51, Slater, Joseph <joe.slater@windriver.com> wrote:
>> Anyone else want to add a data point?
>
> I am the only user on the machine below.  I compiled twice, each way --

[snip]

Unless I'm reading that wrong, it looks like again its actually a
small win to use -no-keep-memory.  Sounds like we should just add it.

Ross
Richard Purdie - Sept. 24, 2013, 4:37 p.m.
On Tue, 2013-09-24 at 17:34 +0100, Burton, Ross wrote:
> On 23 September 2013 22:51, Slater, Joseph <joe.slater@windriver.com> wrote:
> >> Anyone else want to add a data point?
> >
> > I am the only user on the machine below.  I compiled twice, each way --
> 
> [snip]
> 
> Unless I'm reading that wrong, it looks like again its actually a
> small win to use -no-keep-memory.  Sounds like we should just add it.

Its in.

Cheers,

Richard
Khem Raj - Sept. 24, 2013, 4:39 p.m.
On Sep 24, 2013, at 9:34 AM, "Burton, Ross" <ross.burton@intel.com> wrote:

> On 23 September 2013 22:51, Slater, Joseph <joe.slater@windriver.com> wrote:
>>> Anyone else want to add a data point?
>> 
>> I am the only user on the machine below.  I compiled twice, each way --
> 
> [snip]
> 
> Unless I'm reading that wrong, it looks like again its actually a
> small win to use -no-keep-memory.  Sounds like we should just add it.
> 

yes indeed. Thanks all for chiming in.

> Ross

Patch

diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
index 5691d3f..9304243 100644
--- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
+++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
@@ -58,6 +58,10 @@  CPPFLAGS_append_powerpc = " -I${STAGING_INCDIR}/pango-1.0 \
                             -I${STAGING_LIBDIR}/glib-2.0/include \
                             -I${STAGING_INCDIR}/glib-2.0"
 
+# ld can run out of memory linking libwebkitgtk!
+#
+LDFLAGS += "-Wl,--no-keep-memory"
+
 EXTRA_AUTORECONF = " -I Source/autotools "