Patchwork [0/3] Fix xz compression command and optimize compression time

login
register
mail settings
Submitter Andrei Gherzan
Date July 12, 2012, 6:13 p.m.
Message ID <cover.1342116650.git.andrei@gherzan.ro>
Download mbox
Permalink /patch/31815/
State New
Headers show

Pull-request

git://git.yoctoproject.org/poky-contrib ag/xz

Comments

Andrei Gherzan - July 12, 2012, 6:13 p.m.
Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. 

-T threads, --threads=threads
              Specify the number of worker threads to use.  Setting threads to
              a special value 0 makes xz use as many threads as there are  CPU
              cores  on  the system.  The actual number of threads can be less
              than threads if the input file is not big enough  for  threading
              with  the  given  settings or if using more threads would exceed
              the memory usage limit.

-------------------

Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4

File to compress: 1.9G

xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb
real    6m37.170s
user    6m35.389s
sys     0m0.884s
Compressed file size: 3.4Mb

xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory

xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G
real    3m24.462s
user    12m8.502s
sys     0m2.180s
Compressed file size: 3.4Mb

xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb
real    1m9.265s
user    4m8.972s
sys     0m0.944s
Compressed file size: 3.4Mb

So my conclusion would be to use the default -e -6 with -T 0.

The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:

  upstream_tracking: update boost (2012-07-09 17:21:38 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ag/xz
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz

Andrei Gherzan (3):
  image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
    data to file
  image_types.bbclass: Add XZ variable to set number of threads to be
    used while compressing
  image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6

 meta/classes/image_types.bbclass |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Koen Kooi - July 12, 2012, 6:58 p.m.
Any volunteers to test this on a system with >4 real cores? 

Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:

> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. 
> 
> -T threads, --threads=threads
>              Specify the number of worker threads to use.  Setting threads to
>              a special value 0 makes xz use as many threads as there are  CPU
>              cores  on  the system.  The actual number of threads can be less
>              than threads if the input file is not big enough  for  threading
>              with  the  given  settings or if using more threads would exceed
>              the memory usage limit.
> 
> -------------------
> 
> Memory: 7.8 GiB
> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
> 
> File to compress: 1.9G
> 
> xz -f -k -e -9 --check=crc32 (current configuration)
> Memory: 673Mb
> real    6m37.170s
> user    6m35.389s
> sys     0m0.884s
> Compressed file size: 3.4Mb
> 
> xz -f -k -e -9 -T 4 --check=crc32
> Memory: Cannot allocate memory
> 
> xz -f -k -e -8 -T 4 --check=crc32
> Memory: 1.8G
> real    3m24.462s
> user    12m8.502s
> sys     0m2.180s
> Compressed file size: 3.4Mb
> 
> xz -f -k -T 4 --check=crc32 (-e defaults to -6)
> Memory: 471Mb
> real    1m9.265s
> user    4m8.972s
> sys     0m0.944s
> Compressed file size: 3.4Mb
> 
> So my conclusion would be to use the default -e -6 with -T 0.
> 
> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
> 
>  upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
> 
> are available in the git repository at:
> 
>  git://git.yoctoproject.org/poky-contrib ag/xz
>  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
> 
> Andrei Gherzan (3):
>  image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
>    data to file
>  image_types.bbclass: Add XZ variable to set number of threads to be
>    used while compressing
>  image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
> 
> meta/classes/image_types.bbclass |    5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
> 
> -- 
> 1.7.9.5
>
McClintock Matthew-B29882 - July 12, 2012, 7:20 p.m.
On Thu, Jul 12, 2012 at 1:58 PM, Koen Kooi <koen@dominion.thruhere.net> wrote:
> Any volunteers to test this on a system with >4 real cores?

How do you get the memory usage bit?

-M

>
> Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:
>
>> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.
>>
>> -T threads, --threads=threads
>>              Specify the number of worker threads to use.  Setting threads to
>>              a special value 0 makes xz use as many threads as there are  CPU
>>              cores  on  the system.  The actual number of threads can be less
>>              than threads if the input file is not big enough  for  threading
>>              with  the  given  settings or if using more threads would exceed
>>              the memory usage limit.
>>
>> -------------------
>>
>> Memory: 7.8 GiB
>> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
>>
>> File to compress: 1.9G
>>
>> xz -f -k -e -9 --check=crc32 (current configuration)
>> Memory: 673Mb
>> real    6m37.170s
>> user    6m35.389s
>> sys     0m0.884s
>> Compressed file size: 3.4Mb
>>
>> xz -f -k -e -9 -T 4 --check=crc32
>> Memory: Cannot allocate memory
>>
>> xz -f -k -e -8 -T 4 --check=crc32
>> Memory: 1.8G
>> real    3m24.462s
>> user    12m8.502s
>> sys     0m2.180s
>> Compressed file size: 3.4Mb
>>
>> xz -f -k -T 4 --check=crc32 (-e defaults to -6)
>> Memory: 471Mb
>> real    1m9.265s
>> user    4m8.972s
>> sys     0m0.944s
>> Compressed file size: 3.4Mb
>>
>> So my conclusion would be to use the default -e -6 with -T 0.
>>
>> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
>>
>>  upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
>>
>> are available in the git repository at:
>>
>>  git://git.yoctoproject.org/poky-contrib ag/xz
>>  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
>>
>> Andrei Gherzan (3):
>>  image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
>>    data to file
>>  image_types.bbclass: Add XZ variable to set number of threads to be
>>    used while compressing
>>  image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
>>
>> meta/classes/image_types.bbclass |    5 +++--
>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> --
>> 1.7.9.5
>>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Saul Wold - July 12, 2012, 7:22 p.m.
On 07/12/2012 11:58 AM, Koen Kooi wrote:
> Any volunteers to test this on a system with >4 real cores?
>

Koen,

Does OE-Core or Poky have an image setup for using .xz by default?

I will run it on a machine I have, just want to make sure I am doing the 
same tests that Andrei is running.

I was about to ask what msm asked about the Memory info.


Sau!

> Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:
>
>> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.
>>
>> -T threads, --threads=threads
>>               Specify the number of worker threads to use.  Setting threads to
>>               a special value 0 makes xz use as many threads as there are  CPU
>>               cores  on  the system.  The actual number of threads can be less
>>               than threads if the input file is not big enough  for  threading
>>               with  the  given  settings or if using more threads would exceed
>>               the memory usage limit.
>>
>> -------------------
>>
>> Memory: 7.8 GiB
>> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
>>
>> File to compress: 1.9G
>>
>> xz -f -k -e -9 --check=crc32 (current configuration)
>> Memory: 673Mb
>> real    6m37.170s
>> user    6m35.389s
>> sys     0m0.884s
>> Compressed file size: 3.4Mb
>>
>> xz -f -k -e -9 -T 4 --check=crc32
>> Memory: Cannot allocate memory
>>
>> xz -f -k -e -8 -T 4 --check=crc32
>> Memory: 1.8G
>> real    3m24.462s
>> user    12m8.502s
>> sys     0m2.180s
>> Compressed file size: 3.4Mb
>>
>> xz -f -k -T 4 --check=crc32 (-e defaults to -6)
>> Memory: 471Mb
>> real    1m9.265s
>> user    4m8.972s
>> sys     0m0.944s
>> Compressed file size: 3.4Mb
>>
>> So my conclusion would be to use the default -e -6 with -T 0.
>>
>> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
>>
>>   upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
>>
>> are available in the git repository at:
>>
>>   git://git.yoctoproject.org/poky-contrib ag/xz
>>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
>>
>> Andrei Gherzan (3):
>>   image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
>>     data to file
>>   image_types.bbclass: Add XZ variable to set number of threads to be
>>     used while compressing
>>   image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
>>
>> meta/classes/image_types.bbclass |    5 +++--
>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> --
>> 1.7.9.5
>>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
Andrei Gherzan - July 12, 2012, 7:59 p.m.
On Thu, Jul 12, 2012 at 10:22 PM, Saul Wold <sgw@linux.intel.com> wrote:

> On 07/12/2012 11:58 AM, Koen Kooi wrote:
>
>> Any volunteers to test this on a system with >4 real cores?
>>
>>
> Koen,
>
> Does OE-Core or Poky have an image setup for using .xz by default?
>
> I will run it on a machine I have, just want to make sure I am doing the
> same tests that Andrei is running.
>
> I was about to ask what msm asked about the Memory info.
>
>
That field in results is not from "time". It's from gnome-system-monitor.
Sorry for being a little confusing.

@g
Khem Raj - July 12, 2012, 8:14 p.m.
On Thu, Jul 12, 2012 at 12:59 PM, Andrei Gherzan <andrei@gherzan.ro> wrote:
> On Thu, Jul 12, 2012 at 10:22 PM, Saul Wold <sgw@linux.intel.com> wrote:
>>
>> On 07/12/2012 11:58 AM, Koen Kooi wrote:
>>>
>>> Any volunteers to test this on a system with >4 real cores?
>>>
>>
>> Koen,
>>
>> Does OE-Core or Poky have an image setup for using .xz by default?
>>
>> I will run it on a machine I have, just want to make sure I am doing the
>> same tests that Andrei is running.
>>
>> I was about to ask what msm asked about the Memory info.


use htop

> That field in results is not from "time". It's from gnome-system-monitor.
> Sorry for being a little confusing.
>
> @g
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
Koen Kooi - July 13, 2012, 6:45 a.m.
Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:

> On 07/12/2012 11:58 AM, Koen Kooi wrote:
>> Any volunteers to test this on a system with >4 real cores?
>> 
> 
> Koen,
> 
> Does OE-Core or Poky have an image setup for using .xz by default?

No, and as you can see from Andrei's patches, it would have never worked :)

regards,

Koen

> 
> I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running.
> 
> I was about to ask what msm asked about the Memory info.
> 
> 
> Sau!
> 
>> Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:
>> 
>>> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.
>>> 
>>> -T threads, --threads=threads
>>>              Specify the number of worker threads to use.  Setting threads to
>>>              a special value 0 makes xz use as many threads as there are  CPU
>>>              cores  on  the system.  The actual number of threads can be less
>>>              than threads if the input file is not big enough  for  threading
>>>              with  the  given  settings or if using more threads would exceed
>>>              the memory usage limit.
>>> 
>>> -------------------
>>> 
>>> Memory: 7.8 GiB
>>> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
>>> 
>>> File to compress: 1.9G
>>> 
>>> xz -f -k -e -9 --check=crc32 (current configuration)
>>> Memory: 673Mb
>>> real    6m37.170s
>>> user    6m35.389s
>>> sys     0m0.884s
>>> Compressed file size: 3.4Mb
>>> 
>>> xz -f -k -e -9 -T 4 --check=crc32
>>> Memory: Cannot allocate memory
>>> 
>>> xz -f -k -e -8 -T 4 --check=crc32
>>> Memory: 1.8G
>>> real    3m24.462s
>>> user    12m8.502s
>>> sys     0m2.180s
>>> Compressed file size: 3.4Mb
>>> 
>>> xz -f -k -T 4 --check=crc32 (-e defaults to -6)
>>> Memory: 471Mb
>>> real    1m9.265s
>>> user    4m8.972s
>>> sys     0m0.944s
>>> Compressed file size: 3.4Mb
>>> 
>>> So my conclusion would be to use the default -e -6 with -T 0.
>>> 
>>> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
>>> 
>>>  upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
>>> 
>>> are available in the git repository at:
>>> 
>>>  git://git.yoctoproject.org/poky-contrib ag/xz
>>>  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
>>> 
>>> Andrei Gherzan (3):
>>>  image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
>>>    data to file
>>>  image_types.bbclass: Add XZ variable to set number of threads to be
>>>    used while compressing
>>>  image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
>>> 
>>> meta/classes/image_types.bbclass |    5 +++--
>>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>> 
>>> --
>>> 1.7.9.5
>>> 
>> 
>> 
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>> 
>> 
>
Andrea Adami - July 13, 2012, 8:19 a.m.
On Fri, Jul 13, 2012 at 8:45 AM, Koen Kooi <koen@dominion.thruhere.net> wrote:
>
> Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:
>
>> On 07/12/2012 11:58 AM, Koen Kooi wrote:
>>> Any volunteers to test this on a system with >4 real cores?
>>>
>>
>> Koen,
>>
>> Does OE-Core or Poky have an image setup for using .xz by default?
>
> No, and as you can see from Andrei's patches, it would have never worked :)
>
> regards,
>
> Koen
>

This leads to a similar remark for lzma:

The xz/lzma have been added with commit  38334ac and at that time both
where using XZ_COMPRESSION_LEVEL.

The code has been refactored afterwards and what happened is that lzma
now defaults to compression 7 (the default) while xz is way too high
(default is 6).

There is a well hidden pitfall with that, being that any image,
including the initramfs, will use that compression factor.

As you can notice in that (old) man xz [2]  the memory requirements
are very high and what did happen to me was that the cpio.lzma could
not be decompressed thus kernel could not boot (we override now the
value setting "-e2" in our BSP).

Adding more threads will multiply that figures.

My 2 cents

Andrea

[1] http://cgit.openembedded.org/openembedded-core/log/meta/classes/image_types.bbclass
[2] http://www.linuxcertif.com/man/1/xz

>>
>> I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running.
>>
>> I was about to ask what msm asked about the Memory info.
>>
>>
>> Sau!
>>
>>> Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:
>>>
>>>> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.
>>>>
>>>> -T threads, --threads=threads
>>>>              Specify the number of worker threads to use.  Setting threads to
>>>>              a special value 0 makes xz use as many threads as there are  CPU
>>>>              cores  on  the system.  The actual number of threads can be less
>>>>              than threads if the input file is not big enough  for  threading
>>>>              with  the  given  settings or if using more threads would exceed
>>>>              the memory usage limit.
>>>>
>>>> -------------------
>>>>
>>>> Memory: 7.8 GiB
>>>> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
>>>>
>>>> File to compress: 1.9G
>>>>
>>>> xz -f -k -e -9 --check=crc32 (current configuration)
>>>> Memory: 673Mb
>>>> real    6m37.170s
>>>> user    6m35.389s
>>>> sys     0m0.884s
>>>> Compressed file size: 3.4Mb
>>>>
>>>> xz -f -k -e -9 -T 4 --check=crc32
>>>> Memory: Cannot allocate memory
>>>>
>>>> xz -f -k -e -8 -T 4 --check=crc32
>>>> Memory: 1.8G
>>>> real    3m24.462s
>>>> user    12m8.502s
>>>> sys     0m2.180s
>>>> Compressed file size: 3.4Mb
>>>>
>>>> xz -f -k -T 4 --check=crc32 (-e defaults to -6)
>>>> Memory: 471Mb
>>>> real    1m9.265s
>>>> user    4m8.972s
>>>> sys     0m0.944s
>>>> Compressed file size: 3.4Mb
>>>>
>>>> So my conclusion would be to use the default -e -6 with -T 0.
>>>>
>>>> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
>>>>
>>>>  upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
>>>>
>>>> are available in the git repository at:
>>>>
>>>>  git://git.yoctoproject.org/poky-contrib ag/xz
>>>>  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
>>>>
>>>> Andrei Gherzan (3):
>>>>  image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
>>>>    data to file
>>>>  image_types.bbclass: Add XZ variable to set number of threads to be
>>>>    used while compressing
>>>>  image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
>>>>
>>>> meta/classes/image_types.bbclass |    5 +++--
>>>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>>>
>>>> --
>>>> 1.7.9.5
>>>>
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>
>>>
>>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Andrei Gherzan - July 13, 2012, 9:44 a.m.
On Fri, Jul 13, 2012 at 11:19 AM, Andrea Adami <andrea.adami@gmail.com>wrote:

> On Fri, Jul 13, 2012 at 8:45 AM, Koen Kooi <koen@dominion.thruhere.net>
> wrote:
> >
> > Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:
> >
> >> On 07/12/2012 11:58 AM, Koen Kooi wrote:
> >>> Any volunteers to test this on a system with >4 real cores?
> >>>
> >>
> >> Koen,
> >>
> >> Does OE-Core or Poky have an image setup for using .xz by default?
> >
> > No, and as you can see from Andrei's patches, it would have never worked
> :)
> >
> > regards,
> >
> > Koen
> >
>
> This leads to a similar remark for lzma:
>
> The xz/lzma have been added with commit  38334ac and at that time both
> where using XZ_COMPRESSION_LEVEL.
>
> The code has been refactored afterwards and what happened is that lzma
> now defaults to compression 7 (the default) while xz is way too high
> (default is 6).
>
>
Nope. It's default value is -9.


> There is a well hidden pitfall with that, being that any image,
> including the initramfs, will use that compression factor.
>
> As you can notice in that (old) man xz [2]  the memory requirements
>

What old version?


> are very high and what did happen to me was that the cpio.lzma could
> not be decompressed thus kernel could not boot (we override now the
> value setting "-e2" in our BSP).
>
>
Without threads, memory demand was around 700mb. So it should be ok for
most of machines.

Adding more threads will multiply that figures.
>
>
Indeed. With 8GB memory at my disposal i couldn't compress an image with -e
-9 and -T 0 (-T 4)/

@g
Andrea Adami - July 13, 2012, 9:54 a.m.
On Fri, Jul 13, 2012 at 11:44 AM, Andrei Gherzan <andrei@gherzan.ro> wrote:
> On Fri, Jul 13, 2012 at 11:19 AM, Andrea Adami <andrea.adami@gmail.com>
> wrote:
>>
>> On Fri, Jul 13, 2012 at 8:45 AM, Koen Kooi <koen@dominion.thruhere.net>
>> wrote:
>> >
>> > Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:
>> >
>> >> On 07/12/2012 11:58 AM, Koen Kooi wrote:
>> >>> Any volunteers to test this on a system with >4 real cores?
>> >>>
>> >>
>> >> Koen,
>> >>
>> >> Does OE-Core or Poky have an image setup for using .xz by default?
>> >
>> > No, and as you can see from Andrei's patches, it would have never worked
>> > :)
>> >
>> > regards,
>> >
>> > Koen
>> >
>>
>> This leads to a similar remark for lzma:
>>
>> The xz/lzma have been added with commit  38334ac and at that time both
>> where using XZ_COMPRESSION_LEVEL.
>>
>> The code has been refactored afterwards and what happened is that lzma
>> now defaults to compression 7 (the default) while xz is way too high
>> (default is 6).
>>
>
> Nope. It's default value is -9.
>

Well, in oe-core maybe.
Upstream they are more conservative, see that man.

"The default preset level in LZMA Utils is -7 while in XZ Utils it is
-6, so both use 8 MiB dictionary by default."

Cheers

Andrea

>>
>> There is a well hidden pitfall with that, being that any image,
>> including the initramfs, will use that compression factor.
>>
>> As you can notice in that (old) man xz [2]  the memory requirements
>
>
> What old version?
>
>>
>> are very high and what did happen to me was that the cpio.lzma could
>> not be decompressed thus kernel could not boot (we override now the
>> value setting "-e2" in our BSP).
>>
>
> Without threads, memory demand was around 700mb. So it should be ok for most
> of machines.
>
>> Adding more threads will multiply that figures.
>>
>
> Indeed. With 8GB memory at my disposal i couldn't compress an image with -e
> -9 and -T 0 (-T 4)/
>
> @g
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
Andrei Gherzan - July 13, 2012, 9:59 a.m.
>
>
> Well, in oe-core maybe.
> Upstream they are more conservative, see that man.
>
> "The default preset level in LZMA Utils is -7 while in XZ Utils it is
> -6, so both use 8 MiB dictionary by default."
>
>
So you were talking about upstream. Ok got that.

@g
Saul Wold - July 17, 2012, 4:27 p.m.
On 07/12/2012 11:13 AM, Andrei Gherzan wrote:
> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.
>
> -T threads, --threads=threads
>                Specify the number of worker threads to use.  Setting threads to
>                a special value 0 makes xz use as many threads as there are  CPU
>                cores  on  the system.  The actual number of threads can be less
>                than threads if the input file is not big enough  for  threading
>                with  the  given  settings or if using more threads would exceed
>                the memory usage limit.
>
> -------------------
>
> Memory: 7.8 GiB
> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
>
> File to compress: 1.9G
>
> xz -f -k -e -9 --check=crc32 (current configuration)
> Memory: 673Mb
> real    6m37.170s
> user    6m35.389s
> sys     0m0.884s
> Compressed file size: 3.4Mb
>
> xz -f -k -e -9 -T 4 --check=crc32
> Memory: Cannot allocate memory
>
> xz -f -k -e -8 -T 4 --check=crc32
> Memory: 1.8G
> real    3m24.462s
> user    12m8.502s
> sys     0m2.180s
> Compressed file size: 3.4Mb
>
> xz -f -k -T 4 --check=crc32 (-e defaults to -6)
> Memory: 471Mb
> real    1m9.265s
> user    4m8.972s
> sys     0m0.944s
> Compressed file size: 3.4Mb
>
> So my conclusion would be to use the default -e -6 with -T 0.
>
> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
>
>    upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib ag/xz
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
>
> Andrei Gherzan (3):
>    image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
>      data to file
>    image_types.bbclass: Add XZ variable to set number of threads to be
>      used while compressing
>    image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
>
>   meta/classes/image_types.bbclass |    5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
>

Merged into OE-Core

Thanks
	Sau!