Patchwork Does support for external toolchains working in current OE?

login
register
mail settings
Submitter Marcin Juszkiewicz
Date April 5, 2013, 12:21 p.m.
Message ID <515EC1B2.4060103@linaro.org>
Download mbox | patch
Permalink /patch/47493/
State New
Headers show

Comments

Marcin Juszkiewicz - April 5, 2013, 12:21 p.m.
W dniu 05.04.2013 11:20, Marcin Juszkiewicz pisze:

> ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
>  This usually means one provides something the other doesn't and should.

Let's enable debug:

DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']
NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc

And now - let's dig deep into BitBake code. We want two files:

- lib/bb/providers.py
- lib/bb/taskdata.py

1. Bitbake jumps into taskdata/add_provider_internal(). 
2. Then asks bb.providers.filterProviders() 
   "is there any provider for 'virtual/libc'"?
3. In providers/_filterProviders this output is printed:
   "DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']"
4. providers/filterProviders() (note lack of "_") checks for
   PREFERRED_PROVIDER_virtual/libc and outputs:
   NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
   sets foundUnique to True, outputs:
   DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
   and returns all providers + foundUnique
5. We are back at taskdata/add_provider_internal() and we
   have more then one entry in "eligible" array plus
   foundUnique set to True.
6. Here we are happy of results and skips two "if" checks.
7. Then we add each entry as good one with this output:
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc

The question is - why providers/filterProviders() returns more 
then one entry when it founds that one of them is preferred?

So I got with this change:

And when I added "eglibc" into PROVIDES in 
external-linaro-toolchain.bb I got bitbake pass!!!
NO MULTIPLE RECIPES!!! :D
Marcin Juszkiewicz - April 5, 2013, 12:26 p.m.
W dniu 05.04.2013 14:21, Marcin Juszkiewicz pisze:

> So I got with this change:
> 
> diff --git a/lib/bb/providers.py b/lib/bb/providers.py
> index fcee6dc..0b37c44 100644
> --- a/lib/bb/providers.py
> +++ b/lib/bb/providers.py
> @@ -296,8 +296,7 @@ def filterProviders(providers, item, cfgData, dataCache):
>              pn = dataCache.pkg_fn[p]
>              if dataCache.preferred[item] == pn:
>                  logger.verbose("selecting %s to satisfy %s due to PREFERRED_PROVIDERS", pn, item)
> -                eligible.remove(p)
> -                eligible = [p] + eligible
> +                eligible = [p]
>                  foundUnique = True
>                  break

Ignore patch.

> And when I added "eglibc" into PROVIDES in external-linaro-toolchain.bb

This was a source of issues.
Richard Purdie - April 5, 2013, 12:31 p.m.
On Fri, 2013-04-05 at 14:21 +0200, Marcin Juszkiewicz wrote:
> W dniu 05.04.2013 11:20, Marcin Juszkiewicz pisze:
> 
> > ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
> >  This usually means one provides something the other doesn't and should.
> 
> Let's enable debug:
> 
> DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']
> NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
> DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc
> 
> And now - let's dig deep into BitBake code. We want two files:
> 
> - lib/bb/providers.py
> - lib/bb/taskdata.py
> 
> 1. Bitbake jumps into taskdata/add_provider_internal(). 
> 2. Then asks bb.providers.filterProviders() 
>    "is there any provider for 'virtual/libc'"?
> 3. In providers/_filterProviders this output is printed:
>    "DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']"
> 4. providers/filterProviders() (note lack of "_") checks for
>    PREFERRED_PROVIDER_virtual/libc and outputs:
>    NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
>    sets foundUnique to True, outputs:
>    DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
>    and returns all providers + foundUnique
> 5. We are back at taskdata/add_provider_internal() and we
>    have more then one entry in "eligible" array plus
>    foundUnique set to True.
> 6. Here we are happy of results and skips two "if" checks.
> 7. Then we add each entry as good one with this output:
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc
> 
> The question is - why providers/filterProviders() returns more 
> then one entry when it founds that one of them is preferred?

This is primarily due to the way bitbake once worked which was to
execute A, then stop and figure out what it should execute next.

When we enabled multiple tasks in parallel, we had to start computing
the dependency tree in advance. The trouble is I was told at the time we
had to continue to support the "if X is unavailable, try building with
Y" in --continue mode. The only way to do this is to be able to
recompute dependency trees upon a build failure.

Over time we've decided deterministic builds are actually a good think
and we shouldn't change behaviour upon task failures so we now default
to not recomputing dependencies upon failure.

The code therefore has lists for each provider sorted by priority.

So that is the history lesson and the code does it for a reason.

That said, I'm unsure how it manages to build with the code change you
mention without warnings, that is rather puzzling as I'd not have
expected that.

Cheers,

Richard

Patch

diff --git a/lib/bb/providers.py b/lib/bb/providers.py
index fcee6dc..0b37c44 100644
--- a/lib/bb/providers.py
+++ b/lib/bb/providers.py
@@ -296,8 +296,7 @@  def filterProviders(providers, item, cfgData, dataCache):
             pn = dataCache.pkg_fn[p]
             if dataCache.preferred[item] == pn:
                 logger.verbose("selecting %s to satisfy %s due to PREFERRED_PROVIDERS", pn, item)
-                eligible.remove(p)
-                eligible = [p] + eligible
+                eligible = [p]
                 foundUnique = True
                 break