Patchwork [0/1] genext2fs: support large files and filesystems without using large amounts of memor

login
register
mail settings
Submitter Dexuan Cui
Date March 28, 2012, 4:51 p.m.
Message ID <cover.1332952962.git.dexuan.cui@intel.com>
Download mbox
Permalink /patch/24763/
State New
Headers show

Pull-request

git://git.pokylinux.org/poky-contrib dcui/master

Comments

Dexuan Cui - March 28, 2012, 4:51 p.m.
Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.

Let's figure out if this big patch is accepatable or not...

With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
The speed is slow -- I spent about 1.5 hours.

The following changes since commit 265903bdffb10c95ceaf7a892151a50b67939c71:

  procps: don't print error message with kernel 3.0+ (2012-03-28 10:16:30 +0100)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib dcui/master
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master

Dexuan Cui (1):
  genext2fs: support large files and  filesystems without using large
    amounts of memory

 ...01-Fix-warnings-remove-some-unused-macros.patch |   70 ++
 .../0002-Add-put_blk-and-put_nod-routines.patch    | 1121 ++++++++++++++++++++
 .../0003-Add-get_blkmap-and-put_blkmap.patch       |  220 ++++
 ...lker-for-walking-through-directory-entrie.patch |  355 +++++++
 ...05-Make-filesystem-struct-not-an-overloay.patch |  372 +++++++
 ...0006-Improve-the-efficiency-of-extend_blk.patch |  270 +++++
 ...ove-hdlinks-into-the-filesystem-structure.patch |  173 +++
 ...t-the-creation-of-the-filesystem-structur.patch |   93 ++
 ...e-byte-swapping-into-the-get-put-routines.patch |  419 ++++++++
 ...rt-over-to-keeping-the-filesystem-on-disk.patch |  837 +++++++++++++++
 ...les-into-the-filesystem-a-piece-at-a-time.patch |  101 ++
 ...upport-large-file-support-and-rework-hole.patch |  209 ++++
 .../0013-Add-volume-id-support.patch               |   84 ++
 ...014-Remove-unneeded-setting-of-s_reserved.patch |   26 +
 ...-Rework-creating-the-lost-found-directory.patch |   55 +
 ...ix-the-documentation-for-the-new-L-option.patch |   27 +
 .../0017-Fix-file-same-comparison.patch            |   28 +
 ...andle-files-changing-while-we-are-working.patch |   87 ++
 ...ke-sure-superblock-is-clear-on-allocation.patch |   40 +
 .../genext2fs-1.4.1/fix-nbblocks-cast.patch        |   18 +-
 .../genext2fs/genext2fs-1.4.1/update_to_1.95.patch |  119 +++
 meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb |   24 +-
 22 files changed, 4738 insertions(+), 10 deletions(-)
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0001-Fix-warnings-remove-some-unused-macros.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0002-Add-put_blk-and-put_nod-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0003-Add-get_blkmap-and-put_blkmap.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0004-Add-a-dirwalker-for-walking-through-directory-entrie.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0005-Make-filesystem-struct-not-an-overloay.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0006-Improve-the-efficiency-of-extend_blk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0007-Move-hdlinks-into-the-filesystem-structure.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0008-Separate-out-the-creation-of-the-filesystem-structur.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0009-Move-byte-swapping-into-the-get-put-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0010-Convert-over-to-keeping-the-filesystem-on-disk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0011-Copy-files-into-the-filesystem-a-piece-at-a-time.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0012-Add-rev-1-support-large-file-support-and-rework-hole.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0013-Add-volume-id-support.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0014-Remove-unneeded-setting-of-s_reserved.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0015-Rework-creating-the-lost-found-directory.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0016-Fix-the-documentation-for-the-new-L-option.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0017-Fix-file-same-comparison.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0018-Handle-files-changing-while-we-are-working.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0019-Make-sure-superblock-is-clear-on-allocation.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/update_to_1.95.patch
Dexuan Cui - March 29, 2012, 8:57 a.m.
Dexuan Cui wrote on 2012-03-29:
> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
> 
> Let's figure out if this big patch is accepatable or not...
> 
> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
> The speed is slow -- I spent about 1.5 hours.
Based on the latest poky master, I built from scratch with DL_DIR pointing
to an empty directory. After the building finished, I got more data:

the size of DL_DIR is about 3.1 GB.
genext2fs took about 65 minutes to create a 5.4GB .ext3 file.
The final .vmdk file's size is about 3.5GB.

> 
> The following changes since commit
> 265903bdffb10c95ceaf7a892151a50b67939c71:
> 
>   procps: don't print error message with kernel 3.0+ (2012-03-28
> 10:16:30
> +0100)
> 

Thanks,
-- Dexuan
Richard Purdie - March 29, 2012, 10:07 p.m.
On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote:
> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
> 
> Let's figure out if this big patch is accepatable or not...
> 
> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
> The speed is slow -- I spent about 1.5 hours.

If these patches solved all the problems and made things work
wonderfully I'd probably say we'd take them. Unfortunately I don't
consider taking 1.5 hours to build am 8GB filesystem "wonderful", its
rather worrying and I don't think its performing any where need fast
enough for our needs :(.

Patching genext2fs at this point in the cycle is a rather risky
undertaking too and I'm getting very concerned about this. 

We might have to look at alternative ways to solve this problem. I think
Darren said he might help look at this and has some other ideas. Darren?

Cheers,

Richard
McClintock Matthew-B29882 - March 29, 2012, 10:27 p.m.
On Thu, Mar 29, 2012 at 5:07 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote:
>> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
>>
>> Let's figure out if this big patch is accepatable or not...
>>
>> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
>> The speed is slow -- I spent about 1.5 hours.
>
> If these patches solved all the problems and made things work
> wonderfully I'd probably say we'd take them. Unfortunately I don't
> consider taking 1.5 hours to build am 8GB filesystem "wonderful", its
> rather worrying and I don't think its performing any where need fast
> enough for our needs :(.
>
> Patching genext2fs at this point in the cycle is a rather risky
> undertaking too and I'm getting very concerned about this.
>
> We might have to look at alternative ways to solve this problem. I think
> Darren said he might help look at this and has some other ideas. Darren?

Can fuse offer some option here?

-M
Paul Eggleton - March 29, 2012, 10:31 p.m.
On Thursday 29 March 2012 22:27:54 McClintock Matthew-B29882 wrote:
> Can fuse offer some option here?

Only if we could rely on fuse being available on everyone's system, which I'm 
not sure is the case.

Cheers,
Paul
Darren Hart - March 29, 2012, 10:55 p.m.
On 03/29/2012 03:07 PM, Richard Purdie wrote:
> On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote:
>> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
>>
>> Let's figure out if this big patch is accepatable or not...
>>
>> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
>> The speed is slow -- I spent about 1.5 hours.
> 
> If these patches solved all the problems and made things work
> wonderfully I'd probably say we'd take them. Unfortunately I don't
> consider taking 1.5 hours to build am 8GB filesystem "wonderful", its
> rather worrying and I don't think its performing any where need fast
> enough for our needs :(.
> 
> Patching genext2fs at this point in the cycle is a rather risky
> undertaking too and I'm getting very concerned about this. 
> 
> We might have to look at alternative ways to solve this problem. I think
> Darren said he might help look at this and has some other ideas. Darren?

Without having looked into this very far, I seem to recall this was
generating sparse files. This means every time we copy something into
the filesystem it has to allocate the space on the drive. This is great
for building big filesystems that will be mostly empty - but for big
filesystems that we plan to mostly populate, I believe we would be
better off doing pre-allocation (I'm taking this from my experience
creating VMs in virtmanager). However, since using dd to create the file
would require mounting it in order to copy files across (and we want to
avoid requiring root permissions during build), this may not be an option.

Have we tried without the -z option? (-z allows holes in files - I
presume this means sparse files).

Depending on how much of the 8.5 GB image we are filling, we may get a
significant speed increase by first generating a smaller image and then
using resize2fs to grow it to the desired size (referencing
http://landley.livejournal.com/47024.html).


A quick test without passing an initial rootfs showed no difference
between -z and no -z. It may still be worth trying with the actual
population to see if that makes a difference.
Dexuan Cui - March 30, 2012, 3:48 p.m.
Darren Hart wrote on 2012-03-30:
> On 03/29/2012 03:07 PM, Richard Purdie wrote:
>> On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote:
>>> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
>>> 
>>> Let's figure out if this big patch is acceptable or not...
>>> 
>>> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
>>> The speed is slow -- I spent about 1.5 hours.
>> 
>> If these patches solved all the problems and made things work
>> wonderfully I'd probably say we'd take them. Unfortunately I don't
>> consider taking 1.5 hours to build am 8GB filesystem "wonderful",
>> its rather worrying and I don't think its performing any where need
>> fast enough for our needs :(.
>> 
>> Patching genext2fs at this point in the cycle is a rather risky
>> undertaking too and I'm getting very concerned about this.
I agree. 
I understand the risk.

But, BTW, I personally tend to think the quality of the patches (which
Come from the genext2fs mailing list) are good:
please see my test results
http://sourceforge.net/mailarchive/message.php?msg_id=29062014

>> We might have to look at alternative ways to solve this problem. I
>> think Darren said he might help look at this and has some other ideas.
> Darren?
> 
> Without having looked into this very far, I seem to recall this was
> generating sparse files. This means every time we copy something into
> the filesystem it has to allocate the space on the drive. This is
> great for building big filesystems that will be mostly empty - but for
> big filesystems that we plan to mostly populate, I believe we would be
> better off doing pre-allocation (I'm taking this from my experience
> creating VMs in virtmanager). However, since using dd to create the
> file would require mounting it in order to copy files across (and we
> want to avoid requiring root permissions during build), this may not be an option.
> 
> Have we tried without the -z option? (-z allows holes in files - I
> presume this means sparse files).
I tried the -z option of genext2fs, but the speed to create the filesystem
is the same.

> 
> Depending on how much of the 8.5 GB image we are filling, we may get a
> significant speed increase by first generating a smaller image and then
> using resize2fs to grow it to the desired size (referencing
> http://landley.livejournal.com/47024.html).
> 
> 
> A quick test without passing an initial rootfs showed no difference
> between -z and no -z. It may still be worth trying with the actual
> population to see if that makes a difference.
>

Thanks,
-- Dexuan
Koen Kooi - March 30, 2012, 8:43 p.m.
Op 29 mrt. 2012, om 15:07 heeft Richard Purdie het volgende geschreven:

> On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote:
>> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
>> 
>> Let's figure out if this big patch is accepatable or not...
>> 
>> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
>> The speed is slow -- I spent about 1.5 hours.
> 
> If these patches solved all the problems and made things work
> wonderfully I'd probably say we'd take them. Unfortunately I don't
> consider taking 1.5 hours to build am 8GB filesystem "wonderful", its
> rather worrying and I don't think its performing any where need fast
> enough for our needs :(.
> 
> Patching genext2fs at this point in the cycle is a rather risky
> undertaking too and I'm getting very concerned about this. 

The biggest issue I have with genext2fs is that it currently allocates RAM for the complete image, which in my case will cause it to swap and for other users cause it to crash, since image > (ram + swap). Fixing the memory issue before the release would be greatly appreciated.

What about putting it on the shortlist for backports after the release? That should give us all more time to test it and eventually get into the hands of stable branch users. Of course I'd still prefer it to go in before the release and backport fixes :)

regards,

Koen
Richard Purdie - March 30, 2012, 9:49 p.m.
On Fri, 2012-03-30 at 13:43 -0700, Koen Kooi wrote:
> Op 29 mrt. 2012, om 15:07 heeft Richard Purdie het volgende geschreven:
> 
> > On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote:
> >> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
> >> 
> >> Let's figure out if this big patch is accepatable or not...
> >> 
> >> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
> >> The speed is slow -- I spent about 1.5 hours.
> > 
> > If these patches solved all the problems and made things work
> > wonderfully I'd probably say we'd take them. Unfortunately I don't
> > consider taking 1.5 hours to build am 8GB filesystem "wonderful", its
> > rather worrying and I don't think its performing any where need fast
> > enough for our needs :(.
> > 
> > Patching genext2fs at this point in the cycle is a rather risky
> > undertaking too and I'm getting very concerned about this. 
> 
> The biggest issue I have with genext2fs is that it currently allocates
> RAM for the complete image, which in my case will cause it to swap and
> for other users cause it to crash, since image > (ram + swap). Fixing
> the memory issue before the release would be greatly appreciated.
> 
> What about putting it on the shortlist for backports after the
> release? That should give us all more time to test it and eventually
> get into the hands of stable branch users. Of course I'd still prefer
> it to go in before the release and backport fixes :)

I should point out one of the patches in this mega patch set is one for
the memory issue.

I'd appreciate some others giving this patch set a go, see if they can
find any regressions for their usual use cases.

Based on the feedback I've had so far I'm leaning towards merging it but
I'm not making a final decision yet, I'd like some test reports...

Cheers,

Richard
Koen Kooi - March 30, 2012, 9:50 p.m.
Op 30 mrt. 2012, om 14:49 heeft Richard Purdie het volgende geschreven:

> On Fri, 2012-03-30 at 13:43 -0700, Koen Kooi wrote:
>> Op 29 mrt. 2012, om 15:07 heeft Richard Purdie het volgende geschreven:
>> 
>>> On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote:
>>>> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.
>>>> 
>>>> Let's figure out if this big patch is accepatable or not...
>>>> 
>>>> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs.
>>>> The speed is slow -- I spent about 1.5 hours.
>>> 
>>> If these patches solved all the problems and made things work
>>> wonderfully I'd probably say we'd take them. Unfortunately I don't
>>> consider taking 1.5 hours to build am 8GB filesystem "wonderful", its
>>> rather worrying and I don't think its performing any where need fast
>>> enough for our needs :(.
>>> 
>>> Patching genext2fs at this point in the cycle is a rather risky
>>> undertaking too and I'm getting very concerned about this. 
>> 
>> The biggest issue I have with genext2fs is that it currently allocates
>> RAM for the complete image, which in my case will cause it to swap and
>> for other users cause it to crash, since image > (ram + swap). Fixing
>> the memory issue before the release would be greatly appreciated.
>> 
>> What about putting it on the shortlist for backports after the
>> release? That should give us all more time to test it and eventually
>> get into the hands of stable branch users. Of course I'd still prefer
>> it to go in before the release and backport fixes :)
> 
> I should point out one of the patches in this mega patch set is one for
> the memory issue.

Hence my mail :)