Patchwork [0/6] Setup for VMDK to use Direct Disk

login
register
mail settings
Submitter Saul Wold
Date March 27, 2012, 5:42 a.m.
Message ID <cover.1332826876.git.sgw@linux.intel.com>
Download mbox
Permalink /patch/24613/
State New
Headers show

Pull-request

git://git.openembedded.org/openembedded-core-contrib sgw/self

Comments

Saul Wold - March 27, 2012, 5:42 a.m.
Updated comments per Darren's request, added cleanup to
image-types to only use one -i (inode-count) parameter.

Sau!
The following changes since commit 644b7503c37fd73730dd3d7841463b158b8934ed:

  guile: Deal with hardcoded path issues (2012-03-27 00:28:41 +0100)

are available in the git repository at:
  git://git.openembedded.org/openembedded-core-contrib sgw/self
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=sgw/self

Dexuan Cui (1):
  self-hosted-image: pre-populate the builder user with poky source

Saul Wold (5):
  image-vmdk: Add symbolic link for a short named vmdk image
  boot-directdisk: Fix Block Calcuation
  vmdk: Update for direstdisk
  builder: Enable auto starting of Hob
  image_types: remove duplicate setting of -i for inode count

 meta/classes/boot-directdisk.bbclass               |   19 ++++++---
 meta/classes/image-vmdk.bbclass                    |   17 +++++++-
 meta/classes/image_types.bbclass                   |    5 +-
 meta/recipes-core/images/self-hosted-image.bb      |   41 +++++++++++++++++++-
 meta/recipes-graphics/builder/builder_0.1.bb       |    2 +-
 .../builder/files/builder_hob_start.sh             |   10 +++-
 6 files changed, 77 insertions(+), 17 deletions(-)
Paul Eggleton - March 27, 2012, 2:37 p.m.
On Monday 26 March 2012 22:42:54 Saul Wold wrote:
> Updated comments per Darren's request, added cleanup to
> image-types to only use one -i (inode-count) parameter.

I tried this series on my build machine, but do_rootfs failed with "genext2fs: 
not enough memory for filesystem".

The machine has 6GB RAM and 8GB swap. The rootfs directory is 15GB. 

I guess this might have something to do with my DL_DIR not being clean 
beforehand (since it seems to be copying it into the rootfs in its entirety).

Cheers,
Paul
Saul Wold - March 27, 2012, 6:16 p.m.
On 03/27/2012 07:37 AM, Paul Eggleton wrote:
> On Monday 26 March 2012 22:42:54 Saul Wold wrote:
>> Updated comments per Darren's request, added cleanup to
>> image-types to only use one -i (inode-count) parameter.
>
> I tried this series on my build machine, but do_rootfs failed with "genext2fs:
> not enough memory for filesystem".

Yes, there is an issue with genext2fs needing memory to build the 
filesystem first.  I noted in another email thread that the Debian build 
system had so resort to dd, mkfs.ext3, mounting and copying. The issue 
with that is that it requires root privileges which is forbidden with 
bitbake.

genext2fs combines the making and copying of the fs into one process, if 
there is a way to populate the filesystem image with the bits without 
root privs that's what we need to look towards.

This is turning into a non-trivial problem.

I am beginning to think that we might have to have 2 vmdk images for the 
build appliance to work correctly. 1 for the rootfs and the second for 
the build space (10G-20G) with DL_DIR populated.

>
> The machine has 6GB RAM and 8GB swap. The rootfs directory is 15GB.
>
> I guess this might have something to do with my DL_DIR not being clean
> beforehand (since it seems to be copying it into the rootfs in its entirety).
>
I have created a clean DL_DIR to work with by doing a world fetch and 
then using that DL_DIR to build into the image.

> Cheers,
> Paul
>
Paul Eggleton - March 27, 2012, 8:21 p.m.
On Tuesday 27 March 2012 11:16:10 Saul Wold wrote:
> Yes, there is an issue with genext2fs needing memory to build the
> filesystem first.  I noted in another email thread that the Debian build
> system had so resort to dd, mkfs.ext3, mounting and copying. The issue
> with that is that it requires root privileges which is forbidden with
> bitbake.
> 
> genext2fs combines the making and copying of the fs into one process, if
> there is a way to populate the filesystem image with the bits without
> root privs that's what we need to look towards.

Yes, the mount alternative won't work for us. virt-make-fs from the libguestfs 
tools might be a possibility but its large set of dependencies could be an 
issue unless we can pare them down somehow.

> This is turning into a non-trivial problem.

Indeed :/

> I am beginning to think that we might have to have 2 vmdk images for the
> build appliance to work correctly. 1 for the rootfs and the second for
> the build space (10G-20G) with DL_DIR populated.

That might mitigate the problem, yes. If genext2fs requires the same amount of 
RAM as the target filesystem then maybe we need to do an available  RAM check 
first and bail out if there isn't enough.

> > I guess this might have something to do with my DL_DIR not being clean
> > beforehand (since it seems to be copying it into the rootfs in its
> > entirety).
>
> I have created a clean DL_DIR to work with by doing a world fetch and
> then using that DL_DIR to build into the image.

I guess we can just put that in the docs for now. For future improvement 
though I wonder if we can trick the fetcher into just pulling across the 
current files into the rootfs. I'll put that on my todo list for later.

Cheers,
Paul
João Henrique Freitas - March 27, 2012, 8:45 p.m.
Hi,

I guess we can just put that in the docs for now. For future improvement
> though I wonder if we can trick the fetcher into just pulling across the
> current files into the rootfs. I'll put that on my todo list for later.
>


The following thread suggest a patch to fix it:

http://sourceforge.net/mailarchive/forum.php?thread_name=1307458172.10974.64.camel%40skunk&forum_name=genext2fs-devel

Thanks.
Dexuan Cui - March 28, 2012, 3:21 p.m.
Hi joaohf,

Thank for the hint!

I have got the patches from genext2fs-devel mailing list and did some tests:

I can successfully create a .ext2 file of 8.5GB with genext2fs now!



I’ll send an update to the genext2fs recipe later.



Thanks,

-- Dexuan

From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Jo?o Henrique Freitas

Sent: Wednesday, March 28, 2012 4:46 AM
To: Patches and discussions about the oe-core layer
Subject: Re: [OE-core] [PATCH 0/6] Setup for VMDK to use Direct Disk


Hi,
I guess we can just put that in the docs for now. For future improvement
though I wonder if we can trick the fetcher into just pulling across the
current files into the rootfs. I'll put that on my todo list for later.


The following thread suggest a patch to fix it:

http://sourceforge.net/mailarchive/forum.php?thread_name=1307458172.10974.64.camel%40skunk&forum_name=genext2fs-devel

Thanks.

--
-----------------------------------------------------------
João Henrique Freitas - joaohf_at_gmail.com<http://joaohf_at_gmail.com>
Campinas-SP-Brasil
BSD051283
LPI 1
http://www.joaohfreitas.eti.br
Paul Eggleton - March 30, 2012, 2:17 p.m.
On Monday 26 March 2012 22:42:54 Saul Wold wrote:
> Updated comments per Darren's request, added cleanup to
> image-types to only use one -i (inode-count) parameter.
> 
> Sau!
> The following changes since commit 644b7503c37fd73730dd3d7841463b158b8934ed:
> 
>   guile: Deal with hardcoded path issues (2012-03-27 00:28:41 +0100)
> 
> are available in the git repository at:
>   git://git.openembedded.org/openembedded-core-contrib sgw/self
>  
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=sgw/
> self

So these patches have been merged now; I updated to latest master and re-ran 
bitbake self-hosted-image; unfortunately the output doesn't appear to be 
usable. I don't know what has gone wrong but during boot there are complaints 
that the filesystem is corrupt:

SYSLINUX 4.03 2010-10-22 EDD Copyright (C) 1994-2010 H. Peter Anvin et al
EXT3-fs error (device hda2): ext3_lookup: deleted inode referenced: 581713
EXT3-fs error (device hda2): ext3_lookup: deleted inode referenced: 610383
Kernel panic - not syncing: no init found.  Try passing init= option to 
kernel.

Running e2fsck -fn on the rootfs.ext3 file shows quite a number of errors. 
There were no unusual errors in the log.do_rootfs.

Cheers,
Paul
Dexuan Cui - March 30, 2012, 9:16 p.m.
Paul Eggleton wrote on 2012-03-30:
> On Monday 26 March 2012 22:42:54 Saul Wold wrote:
>> Updated comments per Darren's request, added cleanup to image-types
>> to only use one -i (inode-count) parameter.
>> 
>> Sau! The following changes since commit
>> 644b7503c37fd73730dd3d7841463b158b8934ed:
>> 
>>   guile: Deal with hardcoded path issues (2012-03-27 00:28:41 +0100)
>> are available in the git repository at:
>>   git://git.openembedded.org/openembedded-core-contrib sgw/self
>> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/
>> ?h
>> =sgw/
>> self
> 
> So these patches have been merged now; I updated to latest master and
> re-ran bitbake self-hosted-image; unfortunately the output doesn't
> appear to be usable. I don't know what has gone wrong but during boot
> there are complaints that the filesystem is corrupt:
> 
> SYSLINUX 4.03 2010-10-22 EDD Copyright (C) 1994-2010 H. Peter Anvin et
> al EXT3-fs error (device hda2): ext3_lookup: deleted inode referenced:
> 581713 EXT3-fs error (device hda2): ext3_lookup: deleted inode
> referenced: 610383 Kernel panic - not syncing: no init found.  Try passing init= option to kernel.
> 
> Running e2fsck -fn on the rootfs.ext3 file shows quite a number of errors.
> There were no unusual errors in the log.do_rootfs.
> 
Hi Paul,
I can reproduce the same I issue, too...
I'll try to look into this, but at the first glance, I don't know what has gone wrong, either...

Thanks,
-- Dexuan
Dexuan Cui - March 31, 2012, 5:11 a.m.
Cui, Dexuan wrote on 2012-03-31:
> Paul Eggleton wrote on 2012-03-30:
>> On Monday 26 March 2012 22:42:54 Saul Wold wrote:
>>> Updated comments per Darren's request, added cleanup to image-types
>>> to only use one -i (inode-count) parameter.
>>> 
>>> Sau! The following changes since commit
>>> 644b7503c37fd73730dd3d7841463b158b8934ed:
>>> 
>>>   guile: Deal with hardcoded path issues (2012-03-27 00:28:41 +0100)
>>>   are available in the git repository at:
>>>   git://git.openembedded.org/openembedded-core-contrib sgw/self
>>> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log
>>> /
>>> ?h
>>> =sgw/
>>> self
>> 
>> So these patches have been merged now; I updated to latest master
>> and re-ran bitbake self-hosted-image; unfortunately the output
>> doesn't appear to be usable. I don't know what has gone wrong but
>> during boot there are complaints that the filesystem is corrupt:
>> 
>> SYSLINUX 4.03 2010-10-22 EDD Copyright (C) 1994-2010 H. Peter Anvin et
>> al EXT3-fs error (device hda2): ext3_lookup: deleted inode referenced:
>> 581713 EXT3-fs error (device hda2): ext3_lookup: deleted inode
>> referenced: 610383 Kernel panic - not syncing: no init found.  Try
>> passing init= option to kernel.
>> 
>> Running e2fsck -fn on the rootfs.ext3 file shows quite a number of errors.
>> There were no unusual errors in the log.do_rootfs.
>> 
> Hi Paul,
> I can reproduce the same I issue, too...
> I'll try to look into this, but at the first glance, I don't know what
> has gone wrong, either...
Can we make a conclusion the current genext2fs is buggy here
when creating a big image(e.g., >4GB)?

Thanks,
-- Dexuan
Paul Eggleton - April 1, 2012, 7:43 p.m.
On Sunday 01 April 2012 10:28:35 Cui, Dexuan wrote:
> Hi Paul, I believe this is true:
> genext2fs-1.4.1 can create a corrupt file system when
> the size of the file system exceeds some limit.
> 
> e.g., when I create an image of 2.8GB, the image can be mounted
> properly, but when I create an image of about 4.5GB(yes, I found
> genext2fs-1.4.1 is actually able to create an image of 4.5GB; I also
> found it's unable to create an image of about 5.5GB, always
> reporting "couldn't allocate a block (no free space)"),
> the generated image can show something like this when it's
> booted:
> 
> EXT3-fs error (device hda2): ext3_lookup: deleted inode referenced...
> 
> However, with Corey's patches (from the genext2fs's mailing list)
> applied, I can't see the issues.

I can confirm this.

With these patches applied I also compared (a) using genext2fs to create a 
30GB rootfs, and (b) letting genext2fs create a filesystem with just enough 
space and then resizing it to 30GB with resize2fs. There was no benefit in the 
latter - it took 5 minutes longer.

Cheers,
Paul