| Submitter | Dexuan Cui |
|---|---|
| Date | March 28, 2012, 4:51 p.m. |
| Message ID | <cover.1332952962.git.dexuan.cui@intel.com> |
| Download | mbox |
| Permalink | /patch/24763/ |
| State | New |
| Headers | show |
Pull-request
git://git.pokylinux.org/poky-contrib dcui/masterComments
Dexuan Cui wrote on 2012-03-29: > Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. > > Let's figure out if this big patch is accepatable or not... > > With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. > The speed is slow -- I spent about 1.5 hours. Based on the latest poky master, I built from scratch with DL_DIR pointing to an empty directory. After the building finished, I got more data: the size of DL_DIR is about 3.1 GB. genext2fs took about 65 minutes to create a 5.4GB .ext3 file. The final .vmdk file's size is about 3.5GB. > > The following changes since commit > 265903bdffb10c95ceaf7a892151a50b67939c71: > > procps: don't print error message with kernel 3.0+ (2012-03-28 > 10:16:30 > +0100) > Thanks, -- Dexuan
On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote: > Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. > > Let's figure out if this big patch is accepatable or not... > > With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. > The speed is slow -- I spent about 1.5 hours. If these patches solved all the problems and made things work wonderfully I'd probably say we'd take them. Unfortunately I don't consider taking 1.5 hours to build am 8GB filesystem "wonderful", its rather worrying and I don't think its performing any where need fast enough for our needs :(. Patching genext2fs at this point in the cycle is a rather risky undertaking too and I'm getting very concerned about this. We might have to look at alternative ways to solve this problem. I think Darren said he might help look at this and has some other ideas. Darren? Cheers, Richard
On Thu, Mar 29, 2012 at 5:07 PM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote: >> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. >> >> Let's figure out if this big patch is accepatable or not... >> >> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. >> The speed is slow -- I spent about 1.5 hours. > > If these patches solved all the problems and made things work > wonderfully I'd probably say we'd take them. Unfortunately I don't > consider taking 1.5 hours to build am 8GB filesystem "wonderful", its > rather worrying and I don't think its performing any where need fast > enough for our needs :(. > > Patching genext2fs at this point in the cycle is a rather risky > undertaking too and I'm getting very concerned about this. > > We might have to look at alternative ways to solve this problem. I think > Darren said he might help look at this and has some other ideas. Darren? Can fuse offer some option here? -M
On Thursday 29 March 2012 22:27:54 McClintock Matthew-B29882 wrote:
> Can fuse offer some option here?
Only if we could rely on fuse being available on everyone's system, which I'm
not sure is the case.
Cheers,
Paul
On 03/29/2012 03:07 PM, Richard Purdie wrote: > On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote: >> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. >> >> Let's figure out if this big patch is accepatable or not... >> >> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. >> The speed is slow -- I spent about 1.5 hours. > > If these patches solved all the problems and made things work > wonderfully I'd probably say we'd take them. Unfortunately I don't > consider taking 1.5 hours to build am 8GB filesystem "wonderful", its > rather worrying and I don't think its performing any where need fast > enough for our needs :(. > > Patching genext2fs at this point in the cycle is a rather risky > undertaking too and I'm getting very concerned about this. > > We might have to look at alternative ways to solve this problem. I think > Darren said he might help look at this and has some other ideas. Darren? Without having looked into this very far, I seem to recall this was generating sparse files. This means every time we copy something into the filesystem it has to allocate the space on the drive. This is great for building big filesystems that will be mostly empty - but for big filesystems that we plan to mostly populate, I believe we would be better off doing pre-allocation (I'm taking this from my experience creating VMs in virtmanager). However, since using dd to create the file would require mounting it in order to copy files across (and we want to avoid requiring root permissions during build), this may not be an option. Have we tried without the -z option? (-z allows holes in files - I presume this means sparse files). Depending on how much of the 8.5 GB image we are filling, we may get a significant speed increase by first generating a smaller image and then using resize2fs to grow it to the desired size (referencing http://landley.livejournal.com/47024.html). A quick test without passing an initial rootfs showed no difference between -z and no -z. It may still be worth trying with the actual population to see if that makes a difference.
Darren Hart wrote on 2012-03-30: > On 03/29/2012 03:07 PM, Richard Purdie wrote: >> On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote: >>> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. >>> >>> Let's figure out if this big patch is acceptable or not... >>> >>> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. >>> The speed is slow -- I spent about 1.5 hours. >> >> If these patches solved all the problems and made things work >> wonderfully I'd probably say we'd take them. Unfortunately I don't >> consider taking 1.5 hours to build am 8GB filesystem "wonderful", >> its rather worrying and I don't think its performing any where need >> fast enough for our needs :(. >> >> Patching genext2fs at this point in the cycle is a rather risky >> undertaking too and I'm getting very concerned about this. I agree. I understand the risk. But, BTW, I personally tend to think the quality of the patches (which Come from the genext2fs mailing list) are good: please see my test results http://sourceforge.net/mailarchive/message.php?msg_id=29062014 >> We might have to look at alternative ways to solve this problem. I >> think Darren said he might help look at this and has some other ideas. > Darren? > > Without having looked into this very far, I seem to recall this was > generating sparse files. This means every time we copy something into > the filesystem it has to allocate the space on the drive. This is > great for building big filesystems that will be mostly empty - but for > big filesystems that we plan to mostly populate, I believe we would be > better off doing pre-allocation (I'm taking this from my experience > creating VMs in virtmanager). However, since using dd to create the > file would require mounting it in order to copy files across (and we > want to avoid requiring root permissions during build), this may not be an option. > > Have we tried without the -z option? (-z allows holes in files - I > presume this means sparse files). I tried the -z option of genext2fs, but the speed to create the filesystem is the same. > > Depending on how much of the 8.5 GB image we are filling, we may get a > significant speed increase by first generating a smaller image and then > using resize2fs to grow it to the desired size (referencing > http://landley.livejournal.com/47024.html). > > > A quick test without passing an initial rootfs showed no difference > between -z and no -z. It may still be worth trying with the actual > population to see if that makes a difference. > Thanks, -- Dexuan
Op 29 mrt. 2012, om 15:07 heeft Richard Purdie het volgende geschreven: > On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote: >> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. >> >> Let's figure out if this big patch is accepatable or not... >> >> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. >> The speed is slow -- I spent about 1.5 hours. > > If these patches solved all the problems and made things work > wonderfully I'd probably say we'd take them. Unfortunately I don't > consider taking 1.5 hours to build am 8GB filesystem "wonderful", its > rather worrying and I don't think its performing any where need fast > enough for our needs :(. > > Patching genext2fs at this point in the cycle is a rather risky > undertaking too and I'm getting very concerned about this. The biggest issue I have with genext2fs is that it currently allocates RAM for the complete image, which in my case will cause it to swap and for other users cause it to crash, since image > (ram + swap). Fixing the memory issue before the release would be greatly appreciated. What about putting it on the shortlist for backports after the release? That should give us all more time to test it and eventually get into the hands of stable branch users. Of course I'd still prefer it to go in before the release and backport fixes :) regards, Koen
On Fri, 2012-03-30 at 13:43 -0700, Koen Kooi wrote: > Op 29 mrt. 2012, om 15:07 heeft Richard Purdie het volgende geschreven: > > > On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote: > >> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. > >> > >> Let's figure out if this big patch is accepatable or not... > >> > >> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. > >> The speed is slow -- I spent about 1.5 hours. > > > > If these patches solved all the problems and made things work > > wonderfully I'd probably say we'd take them. Unfortunately I don't > > consider taking 1.5 hours to build am 8GB filesystem "wonderful", its > > rather worrying and I don't think its performing any where need fast > > enough for our needs :(. > > > > Patching genext2fs at this point in the cycle is a rather risky > > undertaking too and I'm getting very concerned about this. > > The biggest issue I have with genext2fs is that it currently allocates > RAM for the complete image, which in my case will cause it to swap and > for other users cause it to crash, since image > (ram + swap). Fixing > the memory issue before the release would be greatly appreciated. > > What about putting it on the shortlist for backports after the > release? That should give us all more time to test it and eventually > get into the hands of stable branch users. Of course I'd still prefer > it to go in before the release and backport fixes :) I should point out one of the patches in this mega patch set is one for the memory issue. I'd appreciate some others giving this patch set a go, see if they can find any regressions for their usual use cases. Based on the feedback I've had so far I'm leaning towards merging it but I'm not making a final decision yet, I'd like some test reports... Cheers, Richard
Op 30 mrt. 2012, om 14:49 heeft Richard Purdie het volgende geschreven: > On Fri, 2012-03-30 at 13:43 -0700, Koen Kooi wrote: >> Op 29 mrt. 2012, om 15:07 heeft Richard Purdie het volgende geschreven: >> >>> On Thu, 2012-03-29 at 00:51 +0800, Dexuan Cui wrote: >>>> Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. >>>> >>>> Let's figure out if this big patch is accepatable or not... >>>> >>>> With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. >>>> The speed is slow -- I spent about 1.5 hours. >>> >>> If these patches solved all the problems and made things work >>> wonderfully I'd probably say we'd take them. Unfortunately I don't >>> consider taking 1.5 hours to build am 8GB filesystem "wonderful", its >>> rather worrying and I don't think its performing any where need fast >>> enough for our needs :(. >>> >>> Patching genext2fs at this point in the cycle is a rather risky >>> undertaking too and I'm getting very concerned about this. >> >> The biggest issue I have with genext2fs is that it currently allocates >> RAM for the complete image, which in my case will cause it to swap and >> for other users cause it to crash, since image > (ram + swap). Fixing >> the memory issue before the release would be greatly appreciated. >> >> What about putting it on the shortlist for backports after the >> release? That should give us all more time to test it and eventually >> get into the hands of stable branch users. Of course I'd still prefer >> it to go in before the release and backport fixes :) > > I should point out one of the patches in this mega patch set is one for > the memory issue. Hence my mail :)
Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment. Let's figure out if this big patch is accepatable or not... With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. The speed is slow -- I spent about 1.5 hours. The following changes since commit 265903bdffb10c95ceaf7a892151a50b67939c71: procps: don't print error message with kernel 3.0+ (2012-03-28 10:16:30 +0100) are available in the git repository at: git://git.pokylinux.org/poky-contrib dcui/master http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master Dexuan Cui (1): genext2fs: support large files and filesystems without using large amounts of memory ...01-Fix-warnings-remove-some-unused-macros.patch | 70 ++ .../0002-Add-put_blk-and-put_nod-routines.patch | 1121 ++++++++++++++++++++ .../0003-Add-get_blkmap-and-put_blkmap.patch | 220 ++++ ...lker-for-walking-through-directory-entrie.patch | 355 +++++++ ...05-Make-filesystem-struct-not-an-overloay.patch | 372 +++++++ ...0006-Improve-the-efficiency-of-extend_blk.patch | 270 +++++ ...ove-hdlinks-into-the-filesystem-structure.patch | 173 +++ ...t-the-creation-of-the-filesystem-structur.patch | 93 ++ ...e-byte-swapping-into-the-get-put-routines.patch | 419 ++++++++ ...rt-over-to-keeping-the-filesystem-on-disk.patch | 837 +++++++++++++++ ...les-into-the-filesystem-a-piece-at-a-time.patch | 101 ++ ...upport-large-file-support-and-rework-hole.patch | 209 ++++ .../0013-Add-volume-id-support.patch | 84 ++ ...014-Remove-unneeded-setting-of-s_reserved.patch | 26 + ...-Rework-creating-the-lost-found-directory.patch | 55 + ...ix-the-documentation-for-the-new-L-option.patch | 27 + .../0017-Fix-file-same-comparison.patch | 28 + ...andle-files-changing-while-we-are-working.patch | 87 ++ ...ke-sure-superblock-is-clear-on-allocation.patch | 40 + .../genext2fs-1.4.1/fix-nbblocks-cast.patch | 18 +- .../genext2fs/genext2fs-1.4.1/update_to_1.95.patch | 119 +++ meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb | 24 +- 22 files changed, 4738 insertions(+), 10 deletions(-) create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0001-Fix-warnings-remove-some-unused-macros.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0002-Add-put_blk-and-put_nod-routines.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0003-Add-get_blkmap-and-put_blkmap.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0004-Add-a-dirwalker-for-walking-through-directory-entrie.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0005-Make-filesystem-struct-not-an-overloay.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0006-Improve-the-efficiency-of-extend_blk.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0007-Move-hdlinks-into-the-filesystem-structure.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0008-Separate-out-the-creation-of-the-filesystem-structur.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0009-Move-byte-swapping-into-the-get-put-routines.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0010-Convert-over-to-keeping-the-filesystem-on-disk.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0011-Copy-files-into-the-filesystem-a-piece-at-a-time.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0012-Add-rev-1-support-large-file-support-and-rework-hole.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0013-Add-volume-id-support.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0014-Remove-unneeded-setting-of-s_reserved.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0015-Rework-creating-the-lost-found-directory.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0016-Fix-the-documentation-for-the-new-L-option.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0017-Fix-file-same-comparison.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0018-Handle-files-changing-while-we-are-working.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0019-Make-sure-superblock-is-clear-on-allocation.patch create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/update_to_1.95.patch