Patchwork [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

login
register
mail settings
Submitter Martin Jansa
Date July 10, 2013, 12:38 p.m.
Message ID <1373459893-23611-1-git-send-email-Martin.Jansa@gmail.com>
Download mbox | patch
Permalink /patch/53459/
State Accepted, archived
Headers show

Comments

Martin Jansa - July 10, 2013, 12:38 p.m.
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/test-dependencies.sh | 253 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 253 insertions(+)
 create mode 100755 scripts/test-dependencies.sh
Martin Jansa - July 10, 2013, 12:41 p.m.
On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  scripts/test-dependencies.sh | 253 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 253 insertions(+)
>  create mode 100755 scripts/test-dependencies.sh

Build with last version of this script is still running (after 37 hours)
Building recipe: gnome-desktop (485/1848)
but I've tested it in smaller builds and I'm happy with current
functionality, I'll send found issues in tomorrow or day after that when
the build is complete.

> diff --git a/scripts/test-dependencies.sh b/scripts/test-dependencies.sh
> new file mode 100755
> index 0000000..405c14e
> --- /dev/null
> +++ b/scripts/test-dependencies.sh
> @@ -0,0 +1,253 @@
> +#!/bin/sh
> +
> +# Author: Martin Jansa <martin.jansa@gmail.com>
> +#
> +# Copyright (c) 2013 Martin Jansa <Martin.Jansa@gmail.com>
> +
> +# Used to detect missing dependencies or automagically
> +# enabled dependencies which aren't explicitly enabled
> +# or disabled.
> +
> +# It does 3 builds of <target>
> +# 1st to populate sstate-cache directory and sysroot
> +# 2nd to rebuild each recipe with every possible
> +#     dependency found in sysroot (which stays populated
> +#     from 1st build
> +# 3rd to rebuild each recipe only with dependencies defined
> +#     in DEPENDS
> +# 4th (optional) repeat build like 3rd to make sure that
> +#     minimal versions of dependencies defined in DEPENDS
> +#     is also enough
> +
> +# Global vars
> +tmpdir=
> +targets=
> +recipes=
> +buildhistory=
> +buildtype=
> +default_targets="world"
> +default_buildhistory="buildhistory"
> +default_buildtype="1 2 3 c"
> +
> +usage () {
> +  cat << EOF
> +Welcome to utility to detect missing or autoenabled dependencies.
> +WARNING: this utility will completely remove your tmpdir (make sure
> +         you don't have important buildhistory or persistent dir there).
> +$0 <OPTION>
> +
> +Options:
> +  -h, --help
> +        Display this help and exit.
> +
> +  --tmpdir=<tmpdir>
> +        Specify tmpdir, will use the environment variable TMPDIR if it is not specified.
> +        Something like /OE/oe-core/tmp-eglibc (no / at the end).
> +
> +  --targets=<targets>
> +        List of targets separated by space, will use the environment variable TARGETS if it is not specified.
> +        It will run "bitbake <targets>" to populate sysroots.
> +        Default value is "world".
> +
> +  --recipes=<recipes>
> +        File with list of recipes we want to rebuild with minimal and maximal sysroot.
> +        Will use the environment variable RECIPES if it is not specified.
> +        Default value will use all packages ever recorded in buildhistory directory.
> +
> +  --buildhistory=<buildhistory>
> +        Path to buildhistory directory, it needs to be enabled in your config,
> +        because it's used to detect different dependencies and to create list
> +        of recipes to rebuild when it's not specified.
> +        Will use the environment variable BUILDHISTORY if it is not specified.
> +        Default value is "buildhistory"
> +
> +  --buildtype=<buildtype>
> +        There are 4 types of build:
> +          1: build to populate sstate-cache directory and sysroot
> +          2: build to rebuild each recipe with every possible dep
> +          3: build to rebuild each recipe with minimal dependencies
> +          4: build to rebuild each recipe again with minimal dependencies
> +          c: compare buildhistory directories from build 2 and 3
> +        Will use the environment variable BUILDTYPE if it is not specified.
> +        Default value is "1 2 3 c", order is important, type 4 is optional.
> +EOF
> +}
> +
> +# Print error information and exit.
> +echo_error () {
> +  echo "ERROR: $1" >&2
> +  exit 1
> +}
> +
> +while [ -n "$1" ]; do
> +  case $1 in
> +    --tmpdir=*)
> +      tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e`
> +      [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir"
> +      shift
> +        ;;
> +    --targets=*)
> +      targets=`echo $1 | sed -e 's#^--targets="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --recipes=*)
> +      recipes=`echo $1 | sed -e 's#^--recipes="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --buildhistory=*)
> +      buildhistory=`echo $1 | sed -e 's#^--buildhistory="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --buildtype=*)
> +      buildtype=`echo $1 | sed -e 's#^--buildtype="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --help|-h)
> +      usage
> +      exit 0
> +        ;;
> +    *)
> +      echo "Invalid arguments $*"
> +      echo_error "Try '$0 -h' for more information."
> +        ;;
> +  esac
> +done
> +
> +# tmpdir directory, use environment variable TMPDIR
> +# if it was not specified, otherwise, error.
> +[ -n "$tmpdir" ] || tmpdir=$TMPDIR
> +[ -n "$tmpdir" ] || echo_error "No tmpdir found!"
> +[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\""
> +[ -n "$targets" ] || targets=$TARGETS
> +[ -n "$targets" ] || targets=$default_targets
> +[ -n "$recipes" ] || recipes=$RECIPES
> +[ -n "$recipes" -a ! -f "$recipes" ] && echo_error "Invalid file with list of recipes to rebuild"
> +[ -n "$recipes" ] || echo "All packages ever recorded in buildhistory directory will be rebuilt"
> +[ -n "$buildhistory" ] || buildhistory=$BUILDHISTORY
> +[ -n "$buildhistory" ] || buildhistory=$default_buildhistory
> +[ -d "$buildhistory" ] || echo_error "Invalid buildhistory directory \"$buildhistory\""
> +[ -n "$buildtype" ] || buildtype=$BUILDTYPE
> +[ -n "$buildtype" ] || buildtype=$default_buildtype
> +echo "$buildtype" | grep -v '^[1234c ]*$' && echo_error "Invalid buildtype \"$buildtype\", only some combination of 1, 2, 3, 4, c separated by space is allowed"
> +
> +OUTPUT_BASE=test-dependencies/`date "+%s"`
> +
> +build_all() {
> +  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
> +  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
> +  mkdir -p ${OUTPUT1}
> +  echo "Logs will be stored in ${OUTPUT1} directory"
> +  bitbake -k $targets | tee -a ${OUTPUT1}/complete.log
> +}
> +
> +build_every_recipe() {
> +  if [ "${TYPE}" = "2" ] ; then
> +    echo "===== 2nd build to rebuild each recipe with every possible dep ====="
> +    OUTPUT_MAX=${OUTPUT_BASE}/${TYPE}_max
> +    OUTPUTB=${OUTPUT_MAX}
> +  else
> +    echo "===== 3rd or 4th build to rebuild each recipe with minimal dependencies ====="
> +    OUTPUT_MIN=${OUTPUT_BASE}/${TYPE}_min
> +    OUTPUTB=${OUTPUT_MIN}
> +  fi
> +  
> +  mkdir -p ${OUTPUTB} ${OUTPUTB}/failed ${OUTPUTB}/ok
> +  echo "Logs will be stored in ${OUTPUTB} directory"
> +  if [ -z "$recipes" ]; then
> +    ls -d $buildhistory/packages/*/* | xargs -n 1 basename | sort -u > ${OUTPUTB}/recipe.list
> +    recipes=${OUTPUTB}/recipe.list
> +  fi
> +  if [ "${TYPE}" != "2" ] ; then
> +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
> +  fi
> +  i=1
> +  count=`cat $recipes | wc -l`
> +  for recipe in `cat $recipes`; do
> +    echo "Building recipe: ${recipe} ($i/$count)"
> +    bitbake -c cleansstate ${recipe} > ${OUTPUTB}/log.${recipe} 2>&1;
> +    bitbake ${recipe} >> ${OUTPUTB}/log.${recipe} 2>&1;
> +    grep "ERROR: Task.*failed" ${OUTPUTB}/log.${recipe} && mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/failed/${recipe} || mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/ok/${recipe}
> +    if [ "${TYPE}" != "2" ] ; then
> +      rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
> +    fi
> +    i=`expr $i + 1`
> +  done
> +  echo "Copying buildhistory/packages to ${OUTPUTB}"
> +  cp -ra $buildhistory/packages ${OUTPUTB}
> +  # This will be usefull to see which library is pulling new dependency
> +  echo "Copying do_package logs to ${OUTPUTB}/do_package/"
> +  mkdir ${OUTPUTB}/do_package
> +  find $tmpdir/work/ -name log.do_package | while read f; do
> +    # pn is 3 levels back, but we don't know if there is just one log per pn (only one arch and version)
> +    # dest=`echo $f | sed 's#^.*/\([^/]*\)/\([^/]*\)/\([^/]*\)/log.do_package#\1#g'`
> +    dest=`echo $f | sed "s#$tmpdir/work/##g; s#/#_#g"`
> +    cp $f ${OUTPUTB}/do_package/$dest
> +  done
> +  grep "ERROR: Task.*failed" ${OUTPUTB}/failed/*
> +}
> +
> +compare_deps() {
> +  # you can run just compare task with command like this
> +  # OUTPUT_MAX=test-dependencies/1373140172/2_max \
> +  # OUTPUT_MIN=test-dependencies/1373140172/3_min \
> +  # OUTPUTC=test-dependencies/1373140172 \
> +  # openembedded-core/scripts/test-dependencies.sh --tmpdir=tmp-eglibc --targets=glib-2.0 --recipes=recipe_list --buildtype=c
> +  echo "===== Compare dependencies recorded in \"${OUTPUT_MAX}\" and \"${OUTPUT_MIN}\" ====="
> +  [ -n "${OUTPUTC}" ] || OUTPUTC=${OUTPUT_BASE}
> +  mkdir -p ${OUTPUTC}
> +  OUTPUT_FILE=${OUTPUTC}/dependency-changes
> +  echo "Differences will be stored in ${OUTPUT_FILE}, dot is shown for every 100 of checked packages"
> +  echo > ${OUTPUT_FILE}
> +
> +  [ -d ${OUTPUT_MAX} ] || echo_error "Directory with output from build 2 \"${OUTPUT_MAX}\" does not exist"
> +  [ -d ${OUTPUT_MIN} ] || echo_error "Directory with output from build 3 \"${OUTPUT_MIN}\" does not exist"
> +  [ -d ${OUTPUT_MAX}/packages/ ] || echo_error "Directory with packages from build 2 \"${OUTPUT_MAX}/packages/\" does not exist"
> +  [ -d ${OUTPUT_MIN}/packages/ ] || echo_error "Directory with packages from build 3 \"${OUTPUT_MIN}/packages/\" does not exist"
> +  i=0
> +  find ${OUTPUT_MAX}/packages/ -name latest | sed "s#${OUTPUT_MAX}/##g" | while read pkg; do
> +    max_pkg=${OUTPUT_MAX}/${pkg}
> +    min_pkg=${OUTPUT_MIN}/${pkg}
> +    if [ ! -f "${min_pkg}" ] ; then
> +      echo "ERROR: ${min_pkg} doesn't exist" | tee -a ${OUTPUT_FILE}
> +      continue
> +    fi
> +    # strip version information in parenthesis
> +    max_deps=`grep "^RDEPENDS = " ${max_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
> +    min_deps=`grep "^RDEPENDS = " ${min_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
> +    if [ "$i" = 100 ] ; then
> +      echo -n "." # cheap progressbar
> +      i=0
> +    fi
> +    if [ "${max_deps}" = "${min_deps}" ] ; then
> +      # it's annoying long, but at least it's showing some progress, warnings are grepped at the end
> +      echo "NOTE: ${pkg} dependencies weren't changed" >> ${OUTPUT_FILE}
> +    else
> +      missing_deps=
> +      for dep in ${max_deps}; do
> +        echo "${min_deps}" | grep -q " ${dep} " || missing_deps="${missing_deps} ${dep}"
> +      done
> +      if [ -n "${missing_deps}" ] ; then
> +        echo # to get rid of dots on last line
> +        echo "WARN: ${pkg} lost dependency on ${missing_deps}" | tee -a ${OUTPUT_FILE}
> +      fi
> +    fi
> +    i=`expr $i + 1`
> +  done
> +  echo # to get rid of dots on last line
> +  echo "Found differences: "
> +  grep "^WARN: " ${OUTPUT_FILE}
> +  echo "Found errors: "
> +  grep "^ERROR: " ${OUTPUT_FILE}
> +}
> +
> +for TYPE in $buildtype; do
> +  case ${TYPE} in
> +    1) build_all;;
> +    2) build_every_recipe;;
> +    3) build_every_recipe;;
> +    4) build_every_recipe;;
> +    c) compare_deps;;
> +    *) echo_error "Invalid buildtype \"$TYPE\""
> +  esac
> +done
> -- 
> 1.8.2.1
>
mike.looijmans@topic.nl - July 10, 2013, 1:05 p.m.
I added a buildserver that also exports its "sstate-cache" directory, so 
that other build machines can grab their stuff from it. This works fine, 
but I have one problem. Some packages are meant to be dependent on the 
system that built it. I want to enforce that each build machine creates 
its own package, and does not grab it from the sstate-cache of the 
central server.

For example,

$ cat recipes-core/meta/distro-feed-configs.bbappend
PRINC="1"
DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
DISTRO_FEED_NAME ?= "feed"
DISTRO_FEED_PREFIX = "topic"
DISTRO_FEED_URI = 
"http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"


The purpose being that the host name of the machien that built the image 
ends up in the opkg config files. This works just fine.
Now that we have a central server, all images on all build hosts grab 
the package from the sstate-cache server, resulting in the feed pointing 
to the central server instead of the private one, which is not what I 
wanted to happen.

After reading the documentation, I added the following line to the recipe:

PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"

This did not have the desired effect. The package is still retrieved 
from the cache, and not rebuilt locally. Even if I clean and force a 
rebuild of the distro-feed-configs package, the package that ends up in 
the image is still the one from the central server.

What am I missing here?
Ross Burton - July 10, 2013, 2:34 p.m.
On 10 July 2013 13:41, Martin Jansa <martin.jansa@gmail.com> wrote:
> Build with last version of this script is still running (after 37 hours)
> Building recipe: gnome-desktop (485/1848)
> but I've tested it in smaller builds and I'm happy with current
> functionality, I'll send found issues in tomorrow or day after that when
> the build is complete.

As this has been on my todo list for a while now, come to ELC-E and
I'll get you a beer. :)

Anyone volunteering to run this on oe-core master this weekend?

Ross
Martin Jansa - July 10, 2013, 2:57 p.m.
On Wed, Jul 10, 2013 at 03:34:58PM +0100, Burton, Ross wrote:
> On 10 July 2013 13:41, Martin Jansa <martin.jansa@gmail.com> wrote:
> > Build with last version of this script is still running (after 37 hours)
> > Building recipe: gnome-desktop (485/1848)
> > but I've tested it in smaller builds and I'm happy with current
> > functionality, I'll send found issues in tomorrow or day after that when
> > the build is complete.
> 
> As this has been on my todo list for a while now, come to ELC-E and
> I'll get you a beer. :)

OK :)

> Anyone volunteering to run this on oe-core master this weekend?

I have it running with more layers included so it will take longer, but
will report more "autodetected" deps.

Well for some reason the server is very slow now, it moved only couple
recipes since writting that previous e-mail :/

Building recipe: gnumeric (505/1848)
Paul Eggleton - July 10, 2013, 5:39 p.m.
Hi Mike,

On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:
> I added a buildserver that also exports its "sstate-cache" directory, so
> that other build machines can grab their stuff from it. This works fine,
> but I have one problem. Some packages are meant to be dependent on the
> system that built it. I want to enforce that each build machine creates
> its own package, and does not grab it from the sstate-cache of the
> central server.
> 
> For example,
> 
> $ cat recipes-core/meta/distro-feed-configs.bbappend
> PRINC="1"
> DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
> DISTRO_FEED_NAME ?= "feed"
> DISTRO_FEED_PREFIX = "topic"
> DISTRO_FEED_URI =
> "http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"
> 
> 
> The purpose being that the host name of the machien that built the image
> ends up in the opkg config files. This works just fine.
> Now that we have a central server, all images on all build hosts grab
> the package from the sstate-cache server, resulting in the feed pointing
> to the central server instead of the private one, which is not what I
> wanted to happen.
> 
> After reading the documentation, I added the following line to the recipe:
> 
> PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"
> 
> This did not have the desired effect. The package is still retrieved
> from the cache, and not rebuilt locally. Even if I clean and force a
> rebuild of the distro-feed-configs package, the package that ends up in
> the image is still the one from the central server.
> 
> What am I missing here?

I think distro-feed-configs (from meta-oe, not OE-Core) is for package 
installation on the target, and has nothing to do with shared state.

I'm not sure if it's entirely appropriate, but you could take a similar 
approach to the one we use for preventing native sstate packages from mixing 
across different distributions (mostly to sidestep host glibc version 
dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you 
don't want to share and then the packages will go into a subdirectory named 
with that value; you could then delete these automatically from the server.

Cheers,
Paul

PS if you could please avoid replying to an existing thread for a new 
discussion, that would be great.
Martin Jansa - July 10, 2013, 5:45 p.m.
On Wed, Jul 10, 2013 at 06:39:20PM +0100, Paul Eggleton wrote:
> Hi Mike,
> 
> On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:
> > I added a buildserver that also exports its "sstate-cache" directory, so
> > that other build machines can grab their stuff from it. This works fine,
> > but I have one problem. Some packages are meant to be dependent on the
> > system that built it. I want to enforce that each build machine creates
> > its own package, and does not grab it from the sstate-cache of the
> > central server.
> > 
> > For example,
> > 
> > $ cat recipes-core/meta/distro-feed-configs.bbappend
> > PRINC="1"
> > DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
> > DISTRO_FEED_NAME ?= "feed"
> > DISTRO_FEED_PREFIX = "topic"
> > DISTRO_FEED_URI =
> > "http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"
> > 
> > 
> > The purpose being that the host name of the machien that built the image
> > ends up in the opkg config files. This works just fine.
> > Now that we have a central server, all images on all build hosts grab
> > the package from the sstate-cache server, resulting in the feed pointing
> > to the central server instead of the private one, which is not what I
> > wanted to happen.
> > 
> > After reading the documentation, I added the following line to the recipe:
> > 
> > PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"
> > 
> > This did not have the desired effect. The package is still retrieved
> > from the cache, and not rebuilt locally. Even if I clean and force a
> > rebuild of the distro-feed-configs package, the package that ends up in
> > the image is still the one from the central server.
> > 
> > What am I missing here?
> 
> I think distro-feed-configs (from meta-oe, not OE-Core) is for package 
> installation on the target, and has nothing to do with shared state.
> 
> I'm not sure if it's entirely appropriate, but you could take a similar 
> approach to the one we use for preventing native sstate packages from mixing 
> across different distributions (mostly to sidestep host glibc version 
> dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you 
> don't want to share and then the packages will go into a subdirectory named 
> with that value; you could then delete these automatically from the server.

I would try to add
DISTRO_HOST_NAME[vardepvalue] = "${DISTRO_HOST_NAME}"

that way each builder should create own sstate archive.

use bitbake-diffsigs path-to-distro-feeds.sigdata-file to check that
value of os.uname is included in signature.

Regards,
Martin Jansa - July 11, 2013, 12:38 p.m.
On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  scripts/test-dependencies.sh | 253 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 253 insertions(+)
>  create mode 100755 scripts/test-dependencies.sh
> 
> +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null

I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
takes too long already without reparsing all recipes every time.

Originally I was removing it as work around for
http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
but that's not needed for master and people probably won't be changing
PNBLACKLIST while this script is running.
Chris Larson - July 11, 2013, 1:28 p.m.
On Thu, Jul 11, 2013 at 5:38 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  scripts/test-dependencies.sh | 253
> +++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 253 insertions(+)
> >  create mode 100755 scripts/test-dependencies.sh
> >
> > +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> > +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata
> $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work
> $tmpdir/work-shared 2>/dev/null
>
> I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
> takes too long already without reparsing all recipes every time.
>
> Originally I was removing it as work around for
> http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
> but that's not needed for master and people probably won't be changing
> PNBLACKLIST while this script is running.
>

This is quite an interesting script. It looks quite similar to my
greedy-deps-tests[1] script, though you've taken it a step further in
adding builds #3 and #4.

1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts
Martin Jansa - July 11, 2013, 1:49 p.m.
On Thu, Jul 11, 2013 at 06:28:04AM -0700, Chris Larson wrote:
> On Thu, Jul 11, 2013 at 5:38 AM, Martin Jansa <martin.jansa@gmail.com>wrote:
> 
> > On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > > ---
> > >  scripts/test-dependencies.sh | 253
> > +++++++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 253 insertions(+)
> > >  create mode 100755 scripts/test-dependencies.sh
> > >
> > > +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> > > +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata
> > $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work
> > $tmpdir/work-shared 2>/dev/null
> >
> > I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
> > takes too long already without reparsing all recipes every time.
> >
> > Originally I was removing it as work around for
> > http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
> > but that's not needed for master and people probably won't be changing
> > PNBLACKLIST while this script is running.
> >
> 
> This is quite an interesting script. It looks quite similar to my
> greedy-deps-tests[1] script, though you've taken it a step further in
> adding builds #3 and #4.
> 
> 1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts

pity I didn't know about your script before writing mine, it looks more
elegant in python

I've started with #3 the idea to detect greedy came later

Do you plan to integrate your script into oe-core? I'm fine with keeping
my shell script locally until you do. Your python script would better fit
in bitbake/oe environment and reloc-tests also looks useful.
Chris Larson - July 11, 2013, 1:56 p.m.
On Thu, Jul 11, 2013 at 6:49 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> On Thu, Jul 11, 2013 at 06:28:04AM -0700, Chris Larson wrote:
> > On Thu, Jul 11, 2013 at 5:38 AM, Martin Jansa <martin.jansa@gmail.com
> >wrote:
> >
> > > On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> > > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > > > ---
> > > >  scripts/test-dependencies.sh | 253
> > > +++++++++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 253 insertions(+)
> > > >  create mode 100755 scripts/test-dependencies.sh
> > > >
> > > > +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> > > > +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata
> > > $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work
> > > $tmpdir/work-shared 2>/dev/null
> > >
> > > I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
> > > takes too long already without reparsing all recipes every time.
> > >
> > > Originally I was removing it as work around for
> > >
> http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
> > > but that's not needed for master and people probably won't be changing
> > > PNBLACKLIST while this script is running.
> > >
> >
> > This is quite an interesting script. It looks quite similar to my
> > greedy-deps-tests[1] script, though you've taken it a step further in
> > adding builds #3 and #4.
> >
> > 1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts
>
> pity I didn't know about your script before writing mine, it looks more
> elegant in python
>

Heh, it was shell at first, but I wanted to experiment. It ended up being a
good exercise in clean handling of SIGINT and SIGTERM.

I've started with #3 the idea to detect greedy came later
>
> Do you plan to integrate your script into oe-core? I'm fine with keeping
> my shell script locally until you do. Your python script would better fit
> in bitbake/oe environment and reloc-tests also looks useful.


I hadn't thought about it, but it's a good idea. Perhaps we can get the
best-of-both-worlds and integrate the features of both our scripts.
mike.looijmans@topic.nl - July 29, 2013, 1:17 p.m.
?On 07/10/2013 07:45 PM, Martin Jansa wrote:
> On Wed, Jul 10, 2013 at 06:39:20PM +0100, Paul Eggleton wrote:
>> Hi Mike,
>>
>> On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:
>>> I added a buildserver that also exports its "sstate-cache" directory, so
>>> that other build machines can grab their stuff from it. This works fine,
>>> but I have one problem. Some packages are meant to be dependent on the
>>> system that built it. I want to enforce that each build machine creates
>>> its own package, and does not grab it from the sstate-cache of the
>>> central server.
>>>
>>> For example,
>>>
>>> $ cat recipes-core/meta/distro-feed-configs.bbappend
>>> PRINC="1"
>>> DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
>>> DISTRO_FEED_NAME ?= "feed"
>>> DISTRO_FEED_PREFIX = "topic"
>>> DISTRO_FEED_URI =
>>> "http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"
>>>
>>>
>>> The purpose being that the host name of the machien that built the image
>>> ends up in the opkg config files. This works just fine.
>>> Now that we have a central server, all images on all build hosts grab
>>> the package from the sstate-cache server, resulting in the feed pointing
>>> to the central server instead of the private one, which is not what I
>>> wanted to happen.
>>>
>>> After reading the documentation, I added the following line to the recipe:
>>>
>>> PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"
>>>
>>> This did not have the desired effect. The package is still retrieved
>>> from the cache, and not rebuilt locally. Even if I clean and force a
>>> rebuild of the distro-feed-configs package, the package that ends up in
>>> the image is still the one from the central server.
>>>
>>> What am I missing here?
>>
>> I think distro-feed-configs (from meta-oe, not OE-Core) is for package
>> installation on the target, and has nothing to do with shared state.
>>
>> I'm not sure if it's entirely appropriate, but you could take a similar
>> approach to the one we use for preventing native sstate packages from mixing
>> across different distributions (mostly to sidestep host glibc version
>> dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you
>> don't want to share and then the packages will go into a subdirectory named
>> with that value; you could then delete these automatically from the server.
>
> I would try to add
> DISTRO_HOST_NAME[vardepvalue] = "${DISTRO_HOST_NAME}"
>
> that way each builder should create own sstate archive.

I forgot to give my final feedback on this, it would be impolite to ask 
for advise and don't report back on whether it was useful, so here goes:

I've been using that setting for a while now and it does exactly what I 
wanted it to do. Every build machine creates its own feed configuration, 
while the big buildserver creates about everything else. The shared 
state cache is a big time saver for us.

Thanks!

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

Patch

diff --git a/scripts/test-dependencies.sh b/scripts/test-dependencies.sh
new file mode 100755
index 0000000..405c14e
--- /dev/null
+++ b/scripts/test-dependencies.sh
@@ -0,0 +1,253 @@ 
+#!/bin/sh
+
+# Author: Martin Jansa <martin.jansa@gmail.com>
+#
+# Copyright (c) 2013 Martin Jansa <Martin.Jansa@gmail.com>
+
+# Used to detect missing dependencies or automagically
+# enabled dependencies which aren't explicitly enabled
+# or disabled.
+
+# It does 3 builds of <target>
+# 1st to populate sstate-cache directory and sysroot
+# 2nd to rebuild each recipe with every possible
+#     dependency found in sysroot (which stays populated
+#     from 1st build
+# 3rd to rebuild each recipe only with dependencies defined
+#     in DEPENDS
+# 4th (optional) repeat build like 3rd to make sure that
+#     minimal versions of dependencies defined in DEPENDS
+#     is also enough
+
+# Global vars
+tmpdir=
+targets=
+recipes=
+buildhistory=
+buildtype=
+default_targets="world"
+default_buildhistory="buildhistory"
+default_buildtype="1 2 3 c"
+
+usage () {
+  cat << EOF
+Welcome to utility to detect missing or autoenabled dependencies.
+WARNING: this utility will completely remove your tmpdir (make sure
+         you don't have important buildhistory or persistent dir there).
+$0 <OPTION>
+
+Options:
+  -h, --help
+        Display this help and exit.
+
+  --tmpdir=<tmpdir>
+        Specify tmpdir, will use the environment variable TMPDIR if it is not specified.
+        Something like /OE/oe-core/tmp-eglibc (no / at the end).
+
+  --targets=<targets>
+        List of targets separated by space, will use the environment variable TARGETS if it is not specified.
+        It will run "bitbake <targets>" to populate sysroots.
+        Default value is "world".
+
+  --recipes=<recipes>
+        File with list of recipes we want to rebuild with minimal and maximal sysroot.
+        Will use the environment variable RECIPES if it is not specified.
+        Default value will use all packages ever recorded in buildhistory directory.
+
+  --buildhistory=<buildhistory>
+        Path to buildhistory directory, it needs to be enabled in your config,
+        because it's used to detect different dependencies and to create list
+        of recipes to rebuild when it's not specified.
+        Will use the environment variable BUILDHISTORY if it is not specified.
+        Default value is "buildhistory"
+
+  --buildtype=<buildtype>
+        There are 4 types of build:
+          1: build to populate sstate-cache directory and sysroot
+          2: build to rebuild each recipe with every possible dep
+          3: build to rebuild each recipe with minimal dependencies
+          4: build to rebuild each recipe again with minimal dependencies
+          c: compare buildhistory directories from build 2 and 3
+        Will use the environment variable BUILDTYPE if it is not specified.
+        Default value is "1 2 3 c", order is important, type 4 is optional.
+EOF
+}
+
+# Print error information and exit.
+echo_error () {
+  echo "ERROR: $1" >&2
+  exit 1
+}
+
+while [ -n "$1" ]; do
+  case $1 in
+    --tmpdir=*)
+      tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e`
+      [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir"
+      shift
+        ;;
+    --targets=*)
+      targets=`echo $1 | sed -e 's#^--targets="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --recipes=*)
+      recipes=`echo $1 | sed -e 's#^--recipes="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildhistory=*)
+      buildhistory=`echo $1 | sed -e 's#^--buildhistory="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildtype=*)
+      buildtype=`echo $1 | sed -e 's#^--buildtype="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --help|-h)
+      usage
+      exit 0
+        ;;
+    *)
+      echo "Invalid arguments $*"
+      echo_error "Try '$0 -h' for more information."
+        ;;
+  esac
+done
+
+# tmpdir directory, use environment variable TMPDIR
+# if it was not specified, otherwise, error.
+[ -n "$tmpdir" ] || tmpdir=$TMPDIR
+[ -n "$tmpdir" ] || echo_error "No tmpdir found!"
+[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\""
+[ -n "$targets" ] || targets=$TARGETS
+[ -n "$targets" ] || targets=$default_targets
+[ -n "$recipes" ] || recipes=$RECIPES
+[ -n "$recipes" -a ! -f "$recipes" ] && echo_error "Invalid file with list of recipes to rebuild"
+[ -n "$recipes" ] || echo "All packages ever recorded in buildhistory directory will be rebuilt"
+[ -n "$buildhistory" ] || buildhistory=$BUILDHISTORY
+[ -n "$buildhistory" ] || buildhistory=$default_buildhistory
+[ -d "$buildhistory" ] || echo_error "Invalid buildhistory directory \"$buildhistory\""
+[ -n "$buildtype" ] || buildtype=$BUILDTYPE
+[ -n "$buildtype" ] || buildtype=$default_buildtype
+echo "$buildtype" | grep -v '^[1234c ]*$' && echo_error "Invalid buildtype \"$buildtype\", only some combination of 1, 2, 3, 4, c separated by space is allowed"
+
+OUTPUT_BASE=test-dependencies/`date "+%s"`
+
+build_all() {
+  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
+  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
+  mkdir -p ${OUTPUT1}
+  echo "Logs will be stored in ${OUTPUT1} directory"
+  bitbake -k $targets | tee -a ${OUTPUT1}/complete.log
+}
+
+build_every_recipe() {
+  if [ "${TYPE}" = "2" ] ; then
+    echo "===== 2nd build to rebuild each recipe with every possible dep ====="
+    OUTPUT_MAX=${OUTPUT_BASE}/${TYPE}_max
+    OUTPUTB=${OUTPUT_MAX}
+  else
+    echo "===== 3rd or 4th build to rebuild each recipe with minimal dependencies ====="
+    OUTPUT_MIN=${OUTPUT_BASE}/${TYPE}_min
+    OUTPUTB=${OUTPUT_MIN}
+  fi
+  
+  mkdir -p ${OUTPUTB} ${OUTPUTB}/failed ${OUTPUTB}/ok
+  echo "Logs will be stored in ${OUTPUTB} directory"
+  if [ -z "$recipes" ]; then
+    ls -d $buildhistory/packages/*/* | xargs -n 1 basename | sort -u > ${OUTPUTB}/recipe.list
+    recipes=${OUTPUTB}/recipe.list
+  fi
+  if [ "${TYPE}" != "2" ] ; then
+    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
+    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
+  fi
+  i=1
+  count=`cat $recipes | wc -l`
+  for recipe in `cat $recipes`; do
+    echo "Building recipe: ${recipe} ($i/$count)"
+    bitbake -c cleansstate ${recipe} > ${OUTPUTB}/log.${recipe} 2>&1;
+    bitbake ${recipe} >> ${OUTPUTB}/log.${recipe} 2>&1;
+    grep "ERROR: Task.*failed" ${OUTPUTB}/log.${recipe} && mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/failed/${recipe} || mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/ok/${recipe}
+    if [ "${TYPE}" != "2" ] ; then
+      rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
+    fi
+    i=`expr $i + 1`
+  done
+  echo "Copying buildhistory/packages to ${OUTPUTB}"
+  cp -ra $buildhistory/packages ${OUTPUTB}
+  # This will be usefull to see which library is pulling new dependency
+  echo "Copying do_package logs to ${OUTPUTB}/do_package/"
+  mkdir ${OUTPUTB}/do_package
+  find $tmpdir/work/ -name log.do_package | while read f; do
+    # pn is 3 levels back, but we don't know if there is just one log per pn (only one arch and version)
+    # dest=`echo $f | sed 's#^.*/\([^/]*\)/\([^/]*\)/\([^/]*\)/log.do_package#\1#g'`
+    dest=`echo $f | sed "s#$tmpdir/work/##g; s#/#_#g"`
+    cp $f ${OUTPUTB}/do_package/$dest
+  done
+  grep "ERROR: Task.*failed" ${OUTPUTB}/failed/*
+}
+
+compare_deps() {
+  # you can run just compare task with command like this
+  # OUTPUT_MAX=test-dependencies/1373140172/2_max \
+  # OUTPUT_MIN=test-dependencies/1373140172/3_min \
+  # OUTPUTC=test-dependencies/1373140172 \
+  # openembedded-core/scripts/test-dependencies.sh --tmpdir=tmp-eglibc --targets=glib-2.0 --recipes=recipe_list --buildtype=c
+  echo "===== Compare dependencies recorded in \"${OUTPUT_MAX}\" and \"${OUTPUT_MIN}\" ====="
+  [ -n "${OUTPUTC}" ] || OUTPUTC=${OUTPUT_BASE}
+  mkdir -p ${OUTPUTC}
+  OUTPUT_FILE=${OUTPUTC}/dependency-changes
+  echo "Differences will be stored in ${OUTPUT_FILE}, dot is shown for every 100 of checked packages"
+  echo > ${OUTPUT_FILE}
+
+  [ -d ${OUTPUT_MAX} ] || echo_error "Directory with output from build 2 \"${OUTPUT_MAX}\" does not exist"
+  [ -d ${OUTPUT_MIN} ] || echo_error "Directory with output from build 3 \"${OUTPUT_MIN}\" does not exist"
+  [ -d ${OUTPUT_MAX}/packages/ ] || echo_error "Directory with packages from build 2 \"${OUTPUT_MAX}/packages/\" does not exist"
+  [ -d ${OUTPUT_MIN}/packages/ ] || echo_error "Directory with packages from build 3 \"${OUTPUT_MIN}/packages/\" does not exist"
+  i=0
+  find ${OUTPUT_MAX}/packages/ -name latest | sed "s#${OUTPUT_MAX}/##g" | while read pkg; do
+    max_pkg=${OUTPUT_MAX}/${pkg}
+    min_pkg=${OUTPUT_MIN}/${pkg}
+    if [ ! -f "${min_pkg}" ] ; then
+      echo "ERROR: ${min_pkg} doesn't exist" | tee -a ${OUTPUT_FILE}
+      continue
+    fi
+    # strip version information in parenthesis
+    max_deps=`grep "^RDEPENDS = " ${max_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    min_deps=`grep "^RDEPENDS = " ${min_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    if [ "$i" = 100 ] ; then
+      echo -n "." # cheap progressbar
+      i=0
+    fi
+    if [ "${max_deps}" = "${min_deps}" ] ; then
+      # it's annoying long, but at least it's showing some progress, warnings are grepped at the end
+      echo "NOTE: ${pkg} dependencies weren't changed" >> ${OUTPUT_FILE}
+    else
+      missing_deps=
+      for dep in ${max_deps}; do
+        echo "${min_deps}" | grep -q " ${dep} " || missing_deps="${missing_deps} ${dep}"
+      done
+      if [ -n "${missing_deps}" ] ; then
+        echo # to get rid of dots on last line
+        echo "WARN: ${pkg} lost dependency on ${missing_deps}" | tee -a ${OUTPUT_FILE}
+      fi
+    fi
+    i=`expr $i + 1`
+  done
+  echo # to get rid of dots on last line
+  echo "Found differences: "
+  grep "^WARN: " ${OUTPUT_FILE}
+  echo "Found errors: "
+  grep "^ERROR: " ${OUTPUT_FILE}
+}
+
+for TYPE in $buildtype; do
+  case ${TYPE} in
+    1) build_all;;
+    2) build_every_recipe;;
+    3) build_every_recipe;;
+    4) build_every_recipe;;
+    c) compare_deps;;
+    *) echo_error "Invalid buildtype \"$TYPE\""
+  esac
+done