Patchwork [0/1,Yocto,Bug,1700] Fix for buildstats diskio on non physical disks

login
register
mail settings
Submitter Elizabeth Flanagan
Date Nov. 2, 2011, 6:41 a.m.
Message ID <cover.1320190748.git.elizabeth.flanagan@intel.com>
Download mbox
Permalink /patch/14191/
State New, archived
Headers show

Pull-request

git://git.pokylinux.org/poky-contrib eflanagan/diskio_bug1700

Comments

Elizabeth Flanagan - Nov. 2, 2011, 6:41 a.m.
From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>

tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
buildstats to not collect diskio statistics when we encounter a case where 
the os.major/os.minor is not represented with an entry in /proc/diskstats.

Longer term solution to this is to:
- Identify all fs types that don't have representation in /proc/diskstats
- Gather meaningful iostats of these fs types if possible.

The following changes since commit 4a951e0433a99cd985515843f0a48fadc7050aca:

  Fix HOMEPAGE values in libzypp and sat-solver .bb files (2011-11-01 18:28:19 +0000)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib eflanagan/diskio_bug1700
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/diskio_bug1700

Elizabeth Flanagan (1):
  [Yocto Bug 1700] Fix for buildstats on tmpfs

 meta/classes/buildstats.bbclass |   37 ++++++++++++++++++++++++++-----------
 1 files changed, 26 insertions(+), 11 deletions(-)
Wolfram Stering - Nov. 8, 2011, 10:15 a.m.
On 11/02/2011 07:41 AM, Beth Flanagan wrote:
> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>
> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
> buildstats to not collect diskio statistics when we encounter a case where 
> the os.major/os.minor is not represented with an entry in /proc/diskstats.

A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes.
The problem there is, that btrfs's stat() reports fake device ids that
cannot be found in /proc/diskstats.

> Longer term solution to this is to:
> - Identify all fs types that don't have representation in /proc/diskstats
> - Gather meaningful iostats of these fs types if possible.
>
> The following changes since commit 4a951e0433a99cd985515843f0a48fadc7050aca:
>
>   Fix HOMEPAGE values in libzypp and sat-solver .bb files (2011-11-01 18:28:19 +0000)
>
> are available in the git repository at:
>   git://git.pokylinux.org/poky-contrib eflanagan/diskio_bug1700
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/diskio_bug1700
>
> Elizabeth Flanagan (1):
>   [Yocto Bug 1700] Fix for buildstats on tmpfs
>
>  meta/classes/buildstats.bbclass |   37 ++++++++++++++++++++++++++-----------
>  1 files changed, 26 insertions(+), 11 deletions(-)
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Richard Purdie - Nov. 8, 2011, 2:12 p.m.
On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
> > From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
> >
> > tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
> > buildstats to not collect diskio statistics when we encounter a case where 
> > the os.major/os.minor is not represented with an entry in /proc/diskstats.
> 
> A similar issue exists for building on a btrfs partition.
> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
> The problem there is, that btrfs's stat() reports fake device ids that
> cannot be found in /proc/diskstats.

Did this patch help address that problem for you too?

Cheers,

Richard
Wolfram Stering - Nov. 8, 2011, 3:53 p.m.
On 11/08/2011 03:12 PM, Richard Purdie wrote:
> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>
>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
>>> buildstats to not collect diskio statistics when we encounter a case where 
>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>> A similar issue exists for building on a btrfs partition.
>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>> The problem there is, that btrfs's stat() reports fake device ids that
>> cannot be found in /proc/diskstats.
> Did this patch help address that problem for you too?

I'll be able to check that tomorrow and report back.

regards,

- wolfi

> Cheers,
>
> Richard
Elizabeth Flanagan - Nov. 8, 2011, 3:57 p.m.
On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>
>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>> A similar issue exists for building on a btrfs partition.
>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>> The problem there is, that btrfs's stat() reports fake device ids that
>>> cannot be found in /proc/diskstats.
>> Did this patch help address that problem for you too?
>
> I'll be able to check that tomorrow and report back.

It should as the patch will just disable diskio collection if it
cannot find a valid device id in /proc/diskstats. Let me know if it
doesn't and I'll rework it and resubmit.

-b

>
> regards,
>
> - wolfi
>
>> Cheers,
>>
>> Richard
>
>
>
> --
> Wolfram Stering
>  (Entwicklung)
> HALE electronic GmbH
> Eugen-Müller-Straße 18, 5020 Salzburg, Austria
>  Tel: +43 (662) 439011 550
>  Fax: +43 (662) 439011 9
> http://www.hale.at/
> Firmenbuchnummer: FN 66801m HG Salzburg
>
>
>
> --
> Scanned by MailScanner.
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
Wolfram Stering - Nov. 9, 2011, noon
On 11/08/2011 04:57 PM, Flanagan, Elizabeth wrote:
> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>>
>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>> A similar issue exists for building on a btrfs partition.
>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>> cannot be found in /proc/diskstats.
>>> Did this patch help address that problem for you too?
>> I'll be able to check that tomorrow and report back.
> It should as the patch will just disable diskio collection if it
> cannot find a valid device id in /proc/diskstats. Let me know if it
> doesn't and I'll rework it and resubmit.
>
> -b

Your patch fixes the buildstats issue for building on a btrfs volume as
well.
bitbake no longer hits the exception and diskstats are omitted.

Theoretically, this information would be available for btrfs, but it is
not discoverable
in the way buildstats currently collects the disk statistics. However,
this is a seaprate
issue, I think.

Thanks a lot for fixing this,

-wolfi
Koen Kooi - Nov. 9, 2011, 12:13 p.m.
Op 8 nov. 2011, om 16:57 heeft Flanagan, Elizabeth het volgende geschreven:

> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>> 
>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>> A similar issue exists for building on a btrfs partition.
>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>> cannot be found in /proc/diskstats.
>>> Did this patch help address that problem for you too?
>> 
>> I'll be able to check that tomorrow and report back.
> 
> It should as the patch will just disable diskio collection if it
> cannot find a valid device id in /proc/diskstats. Let me know if it
> doesn't and I'll rework it and resubmit.

Related question: what do you use to visualize the buildstats? I heard rumours about getting bootchart to support our format natively, is that the case? I will be needing pretty graphics for a report on the oe-core buildflow in a few weeks :)

regards,

Koen
Elizabeth Flanagan - Nov. 9, 2011, 6:11 p.m.
On Wed, Nov 9, 2011 at 4:00 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
> On 11/08/2011 04:57 PM, Flanagan, Elizabeth wrote:
>> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>>>
>>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>>> A similar issue exists for building on a btrfs partition.
>>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>>> cannot be found in /proc/diskstats.
>>>> Did this patch help address that problem for you too?
>>> I'll be able to check that tomorrow and report back.
>> It should as the patch will just disable diskio collection if it
>> cannot find a valid device id in /proc/diskstats. Let me know if it
>> doesn't and I'll rework it and resubmit.
>>
>> -b
>
> Your patch fixes the buildstats issue for building on a btrfs volume as
> well.
> bitbake no longer hits the exception and diskstats are omitted.
>
> Theoretically, this information would be available for btrfs, but it is
> not discoverable
> in the way buildstats currently collects the disk statistics. However,
> this is a seaprate
> issue, I think.

Yes, and I'm open for suggestions on the best way to implement finding
disk io data for the myriad of filesystem types out there. :/ Or at
the very list  a way to pull diskstats from a trimmed down list that
includes:

ext2/3/4
btrfs
tmpfs
encryptfs....

I'm not sure performance data is even collected for for non-block
devices. dstat and iostat as far as I can tell can't collect data for
tmpfs.

Ideas?

-b
Elizabeth Flanagan - Nov. 9, 2011, 6:21 p.m.
On Wed, Nov 9, 2011 at 4:13 AM, Koen Kooi <koen@dominion.thruhere.net> wrote:
>
> Op 8 nov. 2011, om 16:57 heeft Flanagan, Elizabeth het volgende geschreven:
>
>> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>>>
>>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>>> A similar issue exists for building on a btrfs partition.
>>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>>> cannot be found in /proc/diskstats.
>>>> Did this patch help address that problem for you too?
>>>
>>> I'll be able to check that tomorrow and report back.
>>
>> It should as the patch will just disable diskio collection if it
>> cannot find a valid device id in /proc/diskstats. Let me know if it
>> doesn't and I'll rework it and resubmit.
>
> Related question: what do you use to visualize the buildstats? I heard rumours about getting bootchart to support our format natively, is that the case? I will be needing pretty graphics for a report on the oe-core buildflow in a few weeks :)
>
> regards,

Koen,

There is a patch to pybootchartgui that allows visualization. RP has
the patchset IIRC.  I'm going to be working on buildstats
visualization soon and I'm leaning towards using jquery/flot to create
a way to allow end users to generate via browser some comparative
graphing of buildstats against different builds/statistics. This way I
can set it up for the AB to compare all the builds we're doing there
and end users can just use it through a browser either locally or via
a server.

-b

>
> Koen
> -----BEGIN PGP SIGNATURE-----
>
> iEYEARECAAYFAk66bmcACgkQMkyGM64RGpHMCQCguHdgtS3SrIVV5+T1dOktTpAT
> FkQAnRZzZqwPJlGa3QlGGmxRrZnNtv/8
> =fEA3
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>