Patchwork [1/2] kernel: stop using -exec rm for deleting files

login
register
mail settings
Submitter Bruce Ashfield
Date Feb. 4, 2014, 5:34 p.m.
Message ID <8a4a98cc53998f6bbf08763359e1b560b2da3e54.1391534980.git.bruce.ashfield@windriver.com>
Download mbox | patch
Permalink /patch/66319/
State New
Headers show

Comments

Bruce Ashfield - Feb. 4, 2014, 5:34 p.m.
Removing files from the source tree via find, exec and rm is not the
most efficient operation, due to (among other things) the many forked
processes.

If we use -delete, it saves a significant amount of time. But -delete
does not work with -prune (since it forces -depth). To maintain the
lib, tools and scripts source files, we can hide them temporarily,
skip their hidden directories and then finally restore them.

Time for install before this change:

 real    2m48.563s
 user    0m35.220s
 sys     0m33.036s

Time for install after this change:

 real    1m21.301s
 user    0m33.160s
 sys     0m28.388s

We could further speed this up by using inline perl to delete the files,
but that complexity is avoided for now.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/classes/kernel.bbclass | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
Richard Purdie - Feb. 4, 2014, 11:26 p.m.
On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
> Removing files from the source tree via find, exec and rm is not the
> most efficient operation, due to (among other things) the many forked
> processes.
> 
> If we use -delete, it saves a significant amount of time. But -delete
> does not work with -prune (since it forces -depth). To maintain the
> lib, tools and scripts source files, we can hide them temporarily,
> skip their hidden directories and then finally restore them.
> 
> Time for install before this change:
> 
>  real    2m48.563s
>  user    0m35.220s
>  sys     0m33.036s
> 
> Time for install after this change:
> 
>  real    1m21.301s
>  user    0m33.160s
>  sys     0m28.388s
> 
> We could further speed this up by using inline perl to delete the files,
> but that complexity is avoided for now.
> 
> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
> ---
>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index 51626b03f824..b76a65699755 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -260,9 +260,21 @@ kernel_do_install() {
>  	# we clean the scripts dir while leaving the generated config
>  	# and include files.
>  	#
> -	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
> -	make -C $kerneldir _mrproper_scripts
> -	find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;

If we wanted to keep the single expression like this, we could do
something like:

find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -print0 | xargs -0 rm -f

?

I don't have strong feelings about it, I'm just curious if you tried it
really. The xargs should batch things up enough that the fork overhead
is much less.

Cheers,

Richard


> +	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts 
> +
> +	# hide directories that shouldn't have their .c, s and S files deleted
> +	for d in tools scripts lib; do
> +		mv $kerneldir/$d $kerneldir/.$d
> +	done
> +
> +	# delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is 
> +	# much faster than find -prune and -exec
> +	find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
> +
> +	# put the hidden dirs back
> +	for d in tools scripts lib; do
> +		mv $kerneldir/.$d $kerneldir/$d
> +	done
>  
>  	# As of Linux kernel version 3.0.1, the clean target removes
>  	# arch/powerpc/lib/crtsavres.o which is present in
Bruce Ashfield - Feb. 5, 2014, 12:35 a.m.
On Tue, Feb 4, 2014 at 6:26 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>> Removing files from the source tree via find, exec and rm is not the
>> most efficient operation, due to (among other things) the many forked
>> processes.
>>
>> If we use -delete, it saves a significant amount of time. But -delete
>> does not work with -prune (since it forces -depth). To maintain the
>> lib, tools and scripts source files, we can hide them temporarily,
>> skip their hidden directories and then finally restore them.
>>
>> Time for install before this change:
>>
>>  real    2m48.563s
>>  user    0m35.220s
>>  sys     0m33.036s
>>
>> Time for install after this change:
>>
>>  real    1m21.301s
>>  user    0m33.160s
>>  sys     0m28.388s
>>
>> We could further speed this up by using inline perl to delete the files,
>> but that complexity is avoided for now.
>>
>> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>> ---
>>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>> index 51626b03f824..b76a65699755 100644
>> --- a/meta/classes/kernel.bbclass
>> +++ b/meta/classes/kernel.bbclass
>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>       # we clean the scripts dir while leaving the generated config
>>       # and include files.
>>       #
>> -     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>> -     make -C $kerneldir _mrproper_scripts
>> -     find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>
> If we wanted to keep the single expression like this, we could do
> something like:
>
> find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -print0 | xargs -0 rm -f
>
> ?
>
> I don't have strong feelings about it, I'm just curious if you tried it
> really. The xargs should batch things up enough that the fork overhead
> is much less.

I did try xargs, and a few other things in between. I had a 7 hour plane flight
to look into this (and packaging overhead .. that's next) and I tried
a few other
techniques. As I hinted in my commit message, the fastest is to actually
locate the files with perl and calling unlink() directly .. but that's
even more of
a departure from what we had before :)

find, prune and friends get so unreadable, that when I found the faster way, I
tossed them in the bin and didn't look back!

Cheers,

Bruce

>
> Cheers,
>
> Richard
>
>
>> +     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
>> +
>> +     # hide directories that shouldn't have their .c, s and S files deleted
>> +     for d in tools scripts lib; do
>> +             mv $kerneldir/$d $kerneldir/.$d
>> +     done
>> +
>> +     # delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is
>> +     # much faster than find -prune and -exec
>> +     find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
>> +
>> +     # put the hidden dirs back
>> +     for d in tools scripts lib; do
>> +             mv $kerneldir/.$d $kerneldir/$d
>> +     done
>>
>>       # As of Linux kernel version 3.0.1, the clean target removes
>>       # arch/powerpc/lib/crtsavres.o which is present in
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
Paul Barker - Feb. 5, 2014, 1:36 a.m.
On 5 February 2014 00:35, Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
> On Tue, Feb 4, 2014 at 6:26 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>>> Removing files from the source tree via find, exec and rm is not the
>>> most efficient operation, due to (among other things) the many forked
>>> processes.
>>>
>>> If we use -delete, it saves a significant amount of time. But -delete
>>> does not work with -prune (since it forces -depth). To maintain the
>>> lib, tools and scripts source files, we can hide them temporarily,
>>> skip their hidden directories and then finally restore them.
>>>
>>> Time for install before this change:
>>>
>>>  real    2m48.563s
>>>  user    0m35.220s
>>>  sys     0m33.036s
>>>
>>> Time for install after this change:
>>>
>>>  real    1m21.301s
>>>  user    0m33.160s
>>>  sys     0m28.388s
>>>
>>> We could further speed this up by using inline perl to delete the files,
>>> but that complexity is avoided for now.
>>>
>>> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>>> ---
>>>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>>> index 51626b03f824..b76a65699755 100644
>>> --- a/meta/classes/kernel.bbclass
>>> +++ b/meta/classes/kernel.bbclass
>>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>>       # we clean the scripts dir while leaving the generated config
>>>       # and include files.
>>>       #
>>> -     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>>> -     make -C $kerneldir _mrproper_scripts
>>> -     find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>>
>> If we wanted to keep the single expression like this, we could do
>> something like:
>>
>> find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -print0 | xargs -0 rm -f
>>
>> ?
>>
>> I don't have strong feelings about it, I'm just curious if you tried it
>> really. The xargs should batch things up enough that the fork overhead
>> is much less.
>
> I did try xargs, and a few other things in between. I had a 7 hour plane flight
> to look into this (and packaging overhead .. that's next) and I tried
> a few other
> techniques. As I hinted in my commit message, the fastest is to actually
> locate the files with perl and calling unlink() directly .. but that's
> even more of
> a departure from what we had before :)
>
> find, prune and friends get so unreadable, that when I found the faster way, I
> tossed them in the bin and didn't look back!
>

How does """find $kerneldir -not -regex
$kerneldir'/\(lib\|tools\|scripts)/.*' -type f -name "*.[csS]"
-delete""" stack up?
Bruce Ashfield - Feb. 5, 2014, 3:10 a.m.
On Tue, Feb 4, 2014 at 8:36 PM, Paul Barker <paul@paulbarker.me.uk> wrote:
> On 5 February 2014 00:35, Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
>> On Tue, Feb 4, 2014 at 6:26 PM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>>>> Removing files from the source tree via find, exec and rm is not the
>>>> most efficient operation, due to (among other things) the many forked
>>>> processes.
>>>>
>>>> If we use -delete, it saves a significant amount of time. But -delete
>>>> does not work with -prune (since it forces -depth). To maintain the
>>>> lib, tools and scripts source files, we can hide them temporarily,
>>>> skip their hidden directories and then finally restore them.
>>>>
>>>> Time for install before this change:
>>>>
>>>>  real    2m48.563s
>>>>  user    0m35.220s
>>>>  sys     0m33.036s
>>>>
>>>> Time for install after this change:
>>>>
>>>>  real    1m21.301s
>>>>  user    0m33.160s
>>>>  sys     0m28.388s
>>>>
>>>> We could further speed this up by using inline perl to delete the files,
>>>> but that complexity is avoided for now.
>>>>
>>>> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>>>> ---
>>>>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>>>> index 51626b03f824..b76a65699755 100644
>>>> --- a/meta/classes/kernel.bbclass
>>>> +++ b/meta/classes/kernel.bbclass
>>>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>>>       # we clean the scripts dir while leaving the generated config
>>>>       # and include files.
>>>>       #
>>>> -     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>>>> -     make -C $kerneldir _mrproper_scripts
>>>> -     find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>>>
>>> If we wanted to keep the single expression like this, we could do
>>> something like:
>>>
>>> find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -print0 | xargs -0 rm -f
>>>
>>> ?
>>>
>>> I don't have strong feelings about it, I'm just curious if you tried it
>>> really. The xargs should batch things up enough that the fork overhead
>>> is much less.
>>
>> I did try xargs, and a few other things in between. I had a 7 hour plane flight
>> to look into this (and packaging overhead .. that's next) and I tried
>> a few other
>> techniques. As I hinted in my commit message, the fastest is to actually
>> locate the files with perl and calling unlink() directly .. but that's
>> even more of
>> a departure from what we had before :)
>>
>> find, prune and friends get so unreadable, that when I found the faster way, I
>> tossed them in the bin and didn't look back!
>>
>
> How does """find $kerneldir -not -regex
> $kerneldir'/\(lib\|tools\|scripts)/.*' -type f -name "*.[csS]"
> -delete""" stack up?

timewise it is similar, but I could have cooked that up myself .. I just don't
like incantations of find and its arcane options.

Bruce

>
> --
> Paul Barker
>
> Email: paul@paulbarker.me.uk
> http://www.paulbarker.me.uk
Richard Purdie - Feb. 9, 2014, 12:06 p.m.
On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
> Removing files from the source tree via find, exec and rm is not the
> most efficient operation, due to (among other things) the many forked
> processes.
> 
> If we use -delete, it saves a significant amount of time. But -delete
> does not work with -prune (since it forces -depth). To maintain the
> lib, tools and scripts source files, we can hide them temporarily,
> skip their hidden directories and then finally restore them.
> 
> Time for install before this change:
> 
>  real    2m48.563s
>  user    0m35.220s
>  sys     0m33.036s
> 
> Time for install after this change:
> 
>  real    1m21.301s
>  user    0m33.160s
>  sys     0m28.388s
> 
> We could further speed this up by using inline perl to delete the files,
> but that complexity is avoided for now.
> 
> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
> ---
>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index 51626b03f824..b76a65699755 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -260,9 +260,21 @@ kernel_do_install() {
>  	# we clean the scripts dir while leaving the generated config
>  	# and include files.
>  	#
> -	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
> -	make -C $kerneldir _mrproper_scripts
> -	find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
> +	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts 
> +
> +	# hide directories that shouldn't have their .c, s and S files deleted
> +	for d in tools scripts lib; do
> +		mv $kerneldir/$d $kerneldir/.$d
> +	done
> +
> +	# delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is 
> +	# much faster than find -prune and -exec
> +	find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
> +
> +	# put the hidden dirs back
> +	for d in tools scripts lib; do
> +		mv $kerneldir/.$d $kerneldir/$d
> +	done
>  
>  	# As of Linux kernel version 3.0.1, the clean target removes
>  	# arch/powerpc/lib/crtsavres.o which is present in


I think this patch is resulting in:

http://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc-lsb/builds/22/steps/BuildImages/logs/stdio

Cheers,

Richard
Bruce Ashfield - Feb. 9, 2014, 2:36 p.m.
On Sun, Feb 9, 2014 at 7:06 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>> Removing files from the source tree via find, exec and rm is not the
>> most efficient operation, due to (among other things) the many forked
>> processes.
>>
>> If we use -delete, it saves a significant amount of time. But -delete
>> does not work with -prune (since it forces -depth). To maintain the
>> lib, tools and scripts source files, we can hide them temporarily,
>> skip their hidden directories and then finally restore them.
>>
>> Time for install before this change:
>>
>>  real    2m48.563s
>>  user    0m35.220s
>>  sys     0m33.036s
>>
>> Time for install after this change:
>>
>>  real    1m21.301s
>>  user    0m33.160s
>>  sys     0m28.388s
>>
>> We could further speed this up by using inline perl to delete the files,
>> but that complexity is avoided for now.
>>
>> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>> ---
>>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>> index 51626b03f824..b76a65699755 100644
>> --- a/meta/classes/kernel.bbclass
>> +++ b/meta/classes/kernel.bbclass
>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>       # we clean the scripts dir while leaving the generated config
>>       # and include files.
>>       #
>> -     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>> -     make -C $kerneldir _mrproper_scripts
>> -     find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>> +     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
>> +
>> +     # hide directories that shouldn't have their .c, s and S files deleted
>> +     for d in tools scripts lib; do
>> +             mv $kerneldir/$d $kerneldir/.$d
>> +     done
>> +
>> +     # delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is
>> +     # much faster than find -prune and -exec
>> +     find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
>> +
>> +     # put the hidden dirs back
>> +     for d in tools scripts lib; do
>> +             mv $kerneldir/.$d $kerneldir/$d
>> +     done
>>
>>       # As of Linux kernel version 3.0.1, the clean target removes
>>       # arch/powerpc/lib/crtsavres.o which is present in
>
>
> I think this patch is resulting in:
>
> http://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc-lsb/builds/22/steps/BuildImages/logs/stdio
>

Could be. I'll launch a build and see if it happens here as well.

Bruce

> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
Bruce Ashfield - Feb. 10, 2014, 1:32 p.m.
On 14-02-09 07:06 AM, Richard Purdie wrote:
> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>> Removing files from the source tree via find, exec and rm is not the
>> most efficient operation, due to (among other things) the many forked
>> processes.
>>
>> If we use -delete, it saves a significant amount of time. But -delete
>> does not work with -prune (since it forces -depth). To maintain the
>> lib, tools and scripts source files, we can hide them temporarily,
>> skip their hidden directories and then finally restore them.
>>
>> Time for install before this change:
>>
>>   real    2m48.563s
>>   user    0m35.220s
>>   sys     0m33.036s
>>
>> Time for install after this change:
>>
>>   real    1m21.301s
>>   user    0m33.160s
>>   sys     0m28.388s
>>
>> We could further speed this up by using inline perl to delete the files,
>> but that complexity is avoided for now.
>>
>> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>> ---
>>   meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>   1 file changed, 15 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>> index 51626b03f824..b76a65699755 100644
>> --- a/meta/classes/kernel.bbclass
>> +++ b/meta/classes/kernel.bbclass
>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>   	# we clean the scripts dir while leaving the generated config
>>   	# and include files.
>>   	#
>> -	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>> -	make -C $kerneldir _mrproper_scripts
>> -	find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>> +	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
>> +
>> +	# hide directories that shouldn't have their .c, s and S files deleted
>> +	for d in tools scripts lib; do
>> +		mv $kerneldir/$d $kerneldir/.$d
>> +	done
>> +
>> +	# delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is
>> +	# much faster than find -prune and -exec
>> +	find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
>> +
>> +	# put the hidden dirs back
>> +	for d in tools scripts lib; do
>> +		mv $kerneldir/.$d $kerneldir/$d
>> +	done
>>
>>   	# As of Linux kernel version 3.0.1, the clean target removes
>>   	# arch/powerpc/lib/crtsavres.o which is present in
>
>
> I think this patch is resulting in:
>
> http://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc-lsb/builds/22/steps/BuildImages/logs/stdio

I've reproduced the issue locally, patch should be incoming shortly.

Bruce

>
> Cheers,
>
> Richard
>
>
>
Bruce Ashfield - Feb. 10, 2014, 5:17 p.m.
On 14-02-09 07:06 AM, Richard Purdie wrote:
> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>> Removing files from the source tree via find, exec and rm is not the
>> most efficient operation, due to (among other things) the many forked
>> processes.
>>
>> If we use -delete, it saves a significant amount of time. But -delete
>> does not work with -prune (since it forces -depth). To maintain the
>> lib, tools and scripts source files, we can hide them temporarily,
>> skip their hidden directories and then finally restore them.
>>
>> Time for install before this change:
>>
>>   real    2m48.563s
>>   user    0m35.220s
>>   sys     0m33.036s
>>
>> Time for install after this change:
>>
>>   real    1m21.301s
>>   user    0m33.160s
>>   sys     0m28.388s
>>
>> We could further speed this up by using inline perl to delete the files,
>> but that complexity is avoided for now.
>>
>> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
>> ---
>>   meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>   1 file changed, 15 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>> index 51626b03f824..b76a65699755 100644
>> --- a/meta/classes/kernel.bbclass
>> +++ b/meta/classes/kernel.bbclass
>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>   	# we clean the scripts dir while leaving the generated config
>>   	# and include files.
>>   	#
>> -	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>> -	make -C $kerneldir _mrproper_scripts
>> -	find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>> +	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
>> +
>> +	# hide directories that shouldn't have their .c, s and S files deleted
>> +	for d in tools scripts lib; do
>> +		mv $kerneldir/$d $kerneldir/.$d
>> +	done
>> +
>> +	# delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is
>> +	# much faster than find -prune and -exec
>> +	find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
>> +
>> +	# put the hidden dirs back
>> +	for d in tools scripts lib; do
>> +		mv $kerneldir/.$d $kerneldir/$d
>> +	done
>>
>>   	# As of Linux kernel version 3.0.1, the clean target removes
>>   	# arch/powerpc/lib/crtsavres.o which is present in
>
>
> I think this patch is resulting in:
>
> http://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc-lsb/builds/22/steps/BuildImages/logs/stdio

I've fixed this now.

For staged patches, do you prefer a resend or an incremental update patch ?

Bruce

>
> Cheers,
>
> Richard
>
>
>
Richard Purdie - Feb. 10, 2014, 5:23 p.m.
On Mon, 2014-02-10 at 12:17 -0500, Bruce Ashfield wrote:
> On 14-02-09 07:06 AM, Richard Purdie wrote:
> > On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
> >> Removing files from the source tree via find, exec and rm is not the
> >> most efficient operation, due to (among other things) the many forked
> >> processes.
> >>
> >> If we use -delete, it saves a significant amount of time. But -delete
> >> does not work with -prune (since it forces -depth). To maintain the
> >> lib, tools and scripts source files, we can hide them temporarily,
> >> skip their hidden directories and then finally restore them.
> >>
> >> Time for install before this change:
> >>
> >>   real    2m48.563s
> >>   user    0m35.220s
> >>   sys     0m33.036s
> >>
> >> Time for install after this change:
> >>
> >>   real    1m21.301s
> >>   user    0m33.160s
> >>   sys     0m28.388s
> >>
> >> We could further speed this up by using inline perl to delete the files,
> >> but that complexity is avoided for now.
> >>
> >> Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
> >> ---
> >>   meta/classes/kernel.bbclass | 18 +++++++++++++++---
> >>   1 file changed, 15 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> >> index 51626b03f824..b76a65699755 100644
> >> --- a/meta/classes/kernel.bbclass
> >> +++ b/meta/classes/kernel.bbclass
> >> @@ -260,9 +260,21 @@ kernel_do_install() {
> >>   	# we clean the scripts dir while leaving the generated config
> >>   	# and include files.
> >>   	#
> >> -	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
> >> -	make -C $kerneldir _mrproper_scripts
> >> -	find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
> >> +	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
> >> +
> >> +	# hide directories that shouldn't have their .c, s and S files deleted
> >> +	for d in tools scripts lib; do
> >> +		mv $kerneldir/$d $kerneldir/.$d
> >> +	done
> >> +
> >> +	# delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is
> >> +	# much faster than find -prune and -exec
> >> +	find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
> >> +
> >> +	# put the hidden dirs back
> >> +	for d in tools scripts lib; do
> >> +		mv $kerneldir/.$d $kerneldir/$d
> >> +	done
> >>
> >>   	# As of Linux kernel version 3.0.1, the clean target removes
> >>   	# arch/powerpc/lib/crtsavres.o which is present in
> >
> >
> > I think this patch is resulting in:
> >
> > http://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc-lsb/builds/22/steps/BuildImages/logs/stdio
> 
> I've fixed this now.
> 
> For staged patches, do you prefer a resend or an incremental update patch ?

A resend is fine thanks.

Cheers,

Richard

Patch

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 51626b03f824..b76a65699755 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -260,9 +260,21 @@  kernel_do_install() {
 	# we clean the scripts dir while leaving the generated config
 	# and include files.
 	#
-	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
-	make -C $kerneldir _mrproper_scripts
-	find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
+	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts 
+
+	# hide directories that shouldn't have their .c, s and S files deleted
+	for d in tools scripts lib; do
+		mv $kerneldir/$d $kerneldir/.$d
+	done
+
+	# delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is 
+	# much faster than find -prune and -exec
+	find . -not -path '*/\.*' -type f -name "*.[csS]" -delete
+
+	# put the hidden dirs back
+	for d in tools scripts lib; do
+		mv $kerneldir/.$d $kerneldir/$d
+	done
 
 	# As of Linux kernel version 3.0.1, the clean target removes
 	# arch/powerpc/lib/crtsavres.o which is present in