Patchwork [0/8] Missing deps discovered in per-recipe builds

login
register
mail settings
Submitter Martin Jansa
Date July 5, 2013, 1:28 a.m.
Message ID <cover.1372987648.git.Martin.Jansa@gmail.com>
Download mbox
Permalink /patch/53075/
State Accepted, archived
Headers show

Pull-request

git://git.openembedded.org/openembedded-core-contrib jansa/deps

Comments

Martin Jansa - July 5, 2013, 1:28 a.m.
I was using simple script to build each component alone:

#!/bin/bash

rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/*
rm -rf log.*

ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u > component.list
ncomponents=`cat component.list | wc -l`
i=1
for component in `cat component.list`; do
  echo "Building component: ${component} ($i/$ncomponents)"
  bitbake -c cleansstate ${component} | tee log.${component};
  bitbake ${component} | tee -a log.${component};
  rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/*
  i=`expr $i + 1`
done
grep "NOTE: Tasks Summary: Attempted.*failed.$" log.*

That should cover missing mandatory dependencies.
Comparing depends.dot files in buildhistory between incremental build and this per-component build should detect at least some autodetected dependencies. In per-component builds it can find only libraries already included in dependency tree, so dependencies lost in buildhistory aren't deterministic.
Even better would be to run per-component build once after building world without removing tmp-eglibc, that will allow components to autodetect all possible libraries and then run another per-component build with removing tmp-eglibc which allows to detect libraries from dependency-tree.


The following changes since commit 3e1dbabbf33a2e461abc92ff10cd970fe604ee38:

  sysstat: backport a patch to fix a parallel building error (2013-07-02 07:58:10 -0700)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib jansa/deps
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/deps

Martin Jansa (8):
  (lib)telepathy*: add missing dependency on libxslt-native
  gst-plugins-bad: inherit gsettings
  gst-plugins-base: add missing dependency on glib-2.0-native
  taglib: add missing dependency on zlib
  quota: inherit pkgconfig
  libpam: inherit pkgconfig
  systemtap: inherit pkgconfig
  eglibc-locale: add missing dependency on
    virtual/${TARGET_PREFIX}binutils

 meta/recipes-connectivity/telepathy/libtelepathy_0.3.3.bb            | 2 +-
 meta/recipes-connectivity/telepathy/telepathy-glib_0.20.1.bb         | 3 +--
 meta/recipes-connectivity/telepathy/telepathy-idle_0.1.14.bb         | 3 +--
 .../telepathy/telepathy-mission-control_5.14.0.bb                    | 4 +---
 meta/recipes-connectivity/telepathy/telepathy-python_0.15.19.bb      | 1 +
 meta/recipes-core/eglibc/eglibc-locale.inc                           | 5 +++++
 meta/recipes-extended/pam/libpam_1.1.6.bb                            | 2 +-
 meta/recipes-extended/quota/quota_4.01.bb                            | 2 +-
 meta/recipes-kernel/systemtap/systemtap_git.bb                       | 2 +-
 meta/recipes-multimedia/gstreamer/gst-plugins-bad_0.10.23.bb         | 2 +-
 meta/recipes-multimedia/gstreamer/gst-plugins-base_0.10.36.bb        | 2 +-
 meta/recipes-support/taglib/taglib_1.8.bb                            | 2 ++
 12 files changed, 17 insertions(+), 13 deletions(-)
Ross Burton - July 5, 2013, 11:17 a.m.
On 5 July 2013 02:28, Martin Jansa <martin.jansa@gmail.com> wrote:
> I was using simple script to build each component alone:

I was literally going to write this script on Monday, great stuff.
Surprisingly few patches for oe-core! :)

Ross
Paul Eggleton - July 5, 2013, 3:28 p.m.
On Friday 05 July 2013 03:28:21 Martin Jansa wrote:
> I was using simple script to build each component alone:
> 
> #!/bin/bash
> 
> rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* rm -rf log.*
> 
> ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u >
> component.list ncomponents=`cat component.list | wc -l`
> i=1
> for component in `cat component.list`; do
>   echo "Building component: ${component} ($i/$ncomponents)"
>   bitbake -c cleansstate ${component} | tee log.${component};
>   bitbake ${component} | tee -a log.${component};
>   rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* i=`expr $i +
> 1`
> done

A cleaned up form of this would be a great addition to the scripts/contrib 
directory if you have time...

Cheers,
Paul
Ross Burton - July 5, 2013, 3:37 p.m.
On 5 July 2013 16:28, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
>>   rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
>> tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
>> tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
>> tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/*

Do you really need all those rms?  wipe-sysroot will blast away the
sysroot which will then be re-populated from sstate, and the
per-package work directories will be cleaned out by the -ccleansstate
(that does a clean + remove from sstate, right?)

Ross
Martin Jansa - July 6, 2013, 12:29 p.m.
On Fri, Jul 05, 2013 at 04:28:24PM +0100, Paul Eggleton wrote:
> On Friday 05 July 2013 03:28:21 Martin Jansa wrote:
> > I was using simple script to build each component alone:
> > 
> > #!/bin/bash
> > 
> > rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> > tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> > tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> > tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* rm -rf log.*
> > 
> > ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u >
> > component.list ncomponents=`cat component.list | wc -l`
> > i=1
> > for component in `cat component.list`; do
> >   echo "Building component: ${component} ($i/$ncomponents)"
> >   bitbake -c cleansstate ${component} | tee log.${component};
> >   bitbake ${component} | tee -a log.${component};
> >   rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> > tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> > tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> > tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* i=`expr $i +
> > 1`
> > done
> 
> A cleaned up form of this would be a great addition to the scripts/contrib 
> directory if you have time...

I'm working on improved version of this script which will detect
automagically enabled dependencies. Once I get it working I'll clean it
and send patch with script.

Unfortunately running it takes very long time (even on fast server),
normal "bitbake world" is done in +- 24 hours, but with repopulating
tmp-eglibc so many times (even from sstate-cache) it's almost 48 hours.