Patchwork net-snmp: disable libnl use

login
register
mail settings
Submitter Steffen Sledz
Date March 11, 2011, 11:04 a.m.
Message ID <1299841477-23487-1-git-send-email-sledz@dresearch.de>
Download mbox | patch
Permalink /patch/1297/
State Accepted
Headers show

Comments

Steffen Sledz - March 11, 2011, 11:04 a.m.
Fixes compile error

| ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_destroy'
| ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_alloc'

which occurred sometimes depending on build order (not in clean package
only builds).

Signed-off-by: Steffen Sledz <sledz@dresearch.de>
---
 recipes/net-snmp/net-snmp_svn.bb |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
Koen Kooi - March 11, 2011, 1:05 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11-03-11 12:04, Steffen Sledz wrote:
> Fixes compile error
> 
> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_destroy'
> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_alloc'
> 
> which occurred sometimes depending on build order (not in clean package
> only builds).

What features do we loose when disabling netlink support? I'd rather get
the libnl recipe fixed (e.g. static lib only) instead of hacking the
dependants.

regards,

Koen

> 
> Signed-off-by: Steffen Sledz <sledz@dresearch.de>
> ---
>  recipes/net-snmp/net-snmp_svn.bb |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/recipes/net-snmp/net-snmp_svn.bb b/recipes/net-snmp/net-snmp_svn.bb
> index 2a75722..c53c91f 100644
> --- a/recipes/net-snmp/net-snmp_svn.bb
> +++ b/recipes/net-snmp/net-snmp_svn.bb
> @@ -1,7 +1,7 @@
>  require net-snmp.inc
>  DEPENDS += "libtool libtool-native libtool-cross"
>  SRCREV = "20069"
> -PR = "${INC_PR}.0"
> +PR = "${INC_PR}.1"
>  
>  S = "${WORKDIR}/net-snmp"
>  
> @@ -10,7 +10,7 @@ SRC_URI = "svn://net-snmp.svn.sourceforge.net/svnroot/net-snmp/trunk;module=net-
>          file://snmpd.conf \
>          file://snmptrapd.conf"
>  
> -EXTRA_OECONF += "--disable-embedded-perl --with-perl-modules=no GREP=/bin/grep SED=/bin/sed"
> +EXTRA_OECONF += "--disable-embedded-perl --with-perl-modules=no --without-nl GREP=/bin/grep SED=/bin/sed"
>  EXTRA_OEMAKE = "INSTALL_PREFIX=${D}"
>  
>  CCACHE = ""

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFNeh4JMkyGM64RGpERAoHZAJ9qMCGgy94d0SRjWn8BYytVOgRTxwCgj+GR
tioY8TZybHS74EH8FMVubQ0=
=euaE
-----END PGP SIGNATURE-----
Khem Raj - March 12, 2011, 12:05 a.m.
On (11/03/11 16:53), Steffen Sledz wrote:
> On 03/11/2011 02:05 PM, Koen Kooi wrote:
> > On 11-03-11 12:04, Steffen Sledz wrote:
> >> Fixes compile error
> > 
> >> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_destroy'
> >> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_alloc'
> > 
> >> which occurred sometimes depending on build order (not in clean package
> >> only builds).
> > 
> > What features do we loose when disabling netlink support? I'd rather get
> > the libnl recipe fixed (e.g. static lib only) instead of hacking the
> > dependants.
> 
> I'm not sure.
> 
> But there is no dependency to libnl. If *libnl was not built before net-snmp* the autotools/configure does not find it. So in that case libnl wasn't used anyway.
> 
> Just in case *it was built before* net-snmp tries to use it and hits the mentioned error.

So assuming that successful builds before did not link in libnl this
seems ok to me.
> 
> Regards,
> Steffen
> 
> -- 
> DResearch Fahrzeugelektronik GmbH
> Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
> Tel: +49 30 515932-237 mailto:sledz@DResearch.de
> Fax: +49 30 515932-299
> Geschäftsführer: Dr. Michael Weber, Werner Mögle;
> Amtsgericht Berlin Charlottenburg; HRB 130120 B;
> Ust.-IDNr. DE273952058
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Steffen Sledz - March 14, 2011, 7:06 a.m.
Am 12.03.2011 01:05, schrieb Khem Raj:
> On (11/03/11 16:53), Steffen Sledz wrote:
>> On 03/11/2011 02:05 PM, Koen Kooi wrote:
>>> On 11-03-11 12:04, Steffen Sledz wrote:
>>>> Fixes compile error
>>>
>>>> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_destroy'
>>>> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_alloc'
>>>
>>>> which occurred sometimes depending on build order (not in clean package
>>>> only builds).
>>>
>>> What features do we loose when disabling netlink support? I'd rather get
>>> the libnl recipe fixed (e.g. static lib only) instead of hacking the
>>> dependants.
>>
>> I'm not sure.
>>
>> But there is no dependency to libnl. If *libnl was not built before net-snmp* the autotools/configure does not find it. So in that case libnl wasn't used anyway.
>>
>> Just in case *it was built before* net-snmp tries to use it and hits the mentioned error.
> 
> So assuming that successful builds before did not link in libnl this
> seems ok to me.

Can i take that as an ACK?

Steffen
Khem Raj - March 14, 2011, 4:39 p.m.
On (14/03/11 08:06), Steffen Sledz wrote:
> Am 12.03.2011 01:05, schrieb Khem Raj:
> > On (11/03/11 16:53), Steffen Sledz wrote:
> >> On 03/11/2011 02:05 PM, Koen Kooi wrote:
> >>> On 11-03-11 12:04, Steffen Sledz wrote:
> >>>> Fixes compile error
> >>>
> >>>> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_destroy'
> >>>> | ./.libs/libnetsnmpmibs.so: undefined reference to `nl_handle_alloc'
> >>>
> >>>> which occurred sometimes depending on build order (not in clean package
> >>>> only builds).
> >>>
> >>> What features do we loose when disabling netlink support? I'd rather get
> >>> the libnl recipe fixed (e.g. static lib only) instead of hacking the
> >>> dependants.
> >>
> >> I'm not sure.
> >>
> >> But there is no dependency to libnl. If *libnl was not built before net-snmp* the autotools/configure does not find it. So in that case libnl wasn't used anyway.
> >>
> >> Just in case *it was built before* net-snmp tries to use it and hits the mentioned error.
> > 
> > So assuming that successful builds before did not link in libnl this
> > seems ok to me.
> 
> Can i take that as an ACK?

Acked-by: Khem Raj <raj.khem@gmail.com>
> 
> Steffen
> 
> -- 
> DResearch Fahrzeugelektronik GmbH
> Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
> Tel: +49 30 515932-237 mailto:sledz@DResearch.de
> Fax: +49 30 515932-299
> Geschäftsführer: Dr. Michael Weber, Werner Mögle;
> Amtsgericht Berlin Charlottenburg; HRB 130120 B;
> Ust.-IDNr. DE273952058
>
Esben Haabendal - March 15, 2011, 9:08 a.m.
On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote:
> which occurred sometimes depending on build order (not in clean
> package only builds).

I would like to raise awareness of the underlying problem here.

The current dependency/staging model of OE basically has this feature
that a build can be influenced not only by it's own dependencies, but
also what has been build before it (or not).

I strongly believe that this has to be fixed on the architectural level,
and not just on a case-by-case level as is currently needed.

I haven't received much feedback on the preivous posting about the
per-recipe staging principle implemented in OE-lite, but I decided to
take this opportunity to re-iterate the fact that the OE-lite
implementation of staging and build dependencies eliminates this
problem.

I am still very much interested in discussing how to move this
technology from OE-lite to OE, but as it impacts all recipe metadata
(build dependencies has to be redefined), OE community at a large really
needs to value the benefits of solving this problem.

Best regards,
Esben
Denys Dmytriyenko - March 15, 2011, 10:03 p.m.
On Tue, Mar 15, 2011 at 10:08:37AM +0100, Esben Haabendal wrote:
> On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote:
> > which occurred sometimes depending on build order (not in clean
> > package only builds).
> 
> I would like to raise awareness of the underlying problem here.
> 
> The current dependency/staging model of OE basically has this feature
> that a build can be influenced not only by it's own dependencies, but
> also what has been build before it (or not).
> 
> I strongly believe that this has to be fixed on the architectural level,
> and not just on a case-by-case level as is currently needed.
> 
> I haven't received much feedback on the preivous posting about the
> per-recipe staging principle implemented in OE-lite, but I decided to
> take this opportunity to re-iterate the fact that the OE-lite
> implementation of staging and build dependencies eliminates this
> problem.
> 
> I am still very much interested in discussing how to move this
> technology from OE-lite to OE, but as it impacts all recipe metadata
> (build dependencies has to be redefined), OE community at a large really
> needs to value the benefits of solving this problem.

Esben,

Thanks for raising this issue and making people aware of it!

This has previously been discussed several times, including last time during 
the Yocto Summit in December, where I brought up the question of per-package 
staging/dependency to Richard's attention, when he introduced the new Shared 
State (sstate). While per-package dependency wasn't considered a critical 
must-have feature right away, it was acknowledged as something worthwhile 
looking at and, according to Richard, should be easy to accomplish with the 
new sstate functionality...
Graham Gower - March 15, 2011, 11:15 p.m.
On 03/15/2011 07:38 PM, Esben Haabendal wrote:
> On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote:
>> which occurred sometimes depending on build order (not in clean
>> package only builds).
> 
> I would like to raise awareness of the underlying problem here.
> 
> The current dependency/staging model of OE basically has this feature
> that a build can be influenced not only by it's own dependencies, but
> also what has been build before it (or not).
> 
> I strongly believe that this has to be fixed on the architectural level,
> and not just on a case-by-case level as is currently needed.
> 
> I haven't received much feedback on the preivous posting about the
> per-recipe staging principle implemented in OE-lite, but I decided to
> take this opportunity to re-iterate the fact that the OE-lite
> implementation of staging and build dependencies eliminates this
> problem.
> 
> I am still very much interested in discussing how to move this
> technology from OE-lite to OE, but as it impacts all recipe metadata
> (build dependencies has to be redefined), OE community at a large really
> needs to value the benefits of solving this problem.
> 
> Best regards,
> Esben
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


I did some work last year to fix some of the low hanging fruit (on a
case by case basis).
My methodology was to do two runs of bitbake -k world, then compare
the output. In case anyone is interested, I used the script below to
identify the problematic recipes. Last time I ran it was in November,
by which time the output had reduced to a list of 5-10 items most of
the time.


#!/bin/sh
#
# Quick and dirty script to determine which tasks fail in one world.log,
# where they succeeded in another. world.log should be generated with, e.g.
# $ bitbake -k world | tee world.log
#
# If there are too many failures, this script will take too long as grep -f
# is slooow.

if [ $# != 2 ]; then
	echo "usage: $0 world.log.1 world.log.2"
	exit 1
fi

tasks="do_setscene do_fetch do_distribute_sources do_unpack do_prepsources
	do_patch do_configure do_qa_configure do_compile do_install
	do_populate_sysroot do_package do_package_write_ipk
	do_package_write do_package_stage do_package_stage_all do_build
	do_qa_staging"

tmp1=`mktemp /tmp/world_regress.XXXXXX`
tmp2=`mktemp /tmp/world_regress.XXXXXX`

for task in $tasks; do
	grep "task $task: Failed" $2 \
		| sed -e 's/^.*NOTE: package //' \
			-e 's/task \(.*\): Failed/\1/' \
		> $tmp1

	grep "task $task: Succeeded" $1 \
		| sed -e 's/^.*NOTE: package //' \
			-e 's/task \(.*\): Succeeded/\1/' \
		> $tmp2

	grep -f $tmp1 $tmp2 \
		| sort -n
done

rm -f $tmp1 $tmp2
Esben Haabendal - March 16, 2011, 5:47 a.m.
On Tue, 2011-03-15 at 18:03 -0400, Denys Dmytriyenko wrote:
> On Tue, Mar 15, 2011 at 10:08:37AM +0100, Esben Haabendal wrote:
> > On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote:
> > > which occurred sometimes depending on build order (not in clean
> > > package only builds).
> > 
> > I would like to raise awareness of the underlying problem here.
> > 
> > The current dependency/staging model of OE basically has this feature
> > that a build can be influenced not only by it's own dependencies, but
> > also what has been build before it (or not).
> > 
> > I strongly believe that this has to be fixed on the architectural level,
> > and not just on a case-by-case level as is currently needed.
> > 
> > I haven't received much feedback on the preivous posting about the
> > per-recipe staging principle implemented in OE-lite, but I decided to
> > take this opportunity to re-iterate the fact that the OE-lite
> > implementation of staging and build dependencies eliminates this
> > problem.
> > 
> > I am still very much interested in discussing how to move this
> > technology from OE-lite to OE, but as it impacts all recipe metadata
> > (build dependencies has to be redefined), OE community at a large really
> > needs to value the benefits of solving this problem.
> 
> Esben,
> 
> Thanks for raising this issue and making people aware of it!
> 
> This has previously been discussed several times, including last time during 
> the Yocto Summit in December, where I brought up the question of per-package 
> staging/dependency to Richard's attention, when he introduced the new Shared 
> State (sstate). While per-package dependency wasn't considered a critical 
> must-have feature right away, it was acknowledged as something worthwhile 
> looking at and, according to Richard, should be easy to accomplish with the 
> new sstate functionality...

There is two different objectives in this.

Per-recipe staging and per-package build-time dependencies.  While I
assume per-recipe staging will be doable to build on top of sstate,
per-package build-time dependencies is not really related to sstate, but
is mostly a meta-data issue (man-hours wise), while also requiring
changes to tbe BitBake cooker.

The final thing that I have found while solving the above issues, is how
simple (KISS!) all this can be addressed when going from per-recipe
build-time dependencies to per-package.  And I really think that we need
more KISS in OE ;-)

/Esben
Noor Ahsan - March 16, 2011, 6:22 a.m.
Hello,

I was looking at the libxml2 package. While going through the recipes I
came to know that its native package depends on python-native package as
well. This dependency is defined in libxml2.inc file. 

DEPENDS_virtclass-native = "python-native"

This package build fine after removing this dependency. I am not clear
why this dependency is being added in libxml2 package. Can somebody
clear that to me. While looking at the git log I came to know that this
change is coming from libxml2-native.inc file. But I am not clear why
this dependency was added as this package build fine without it.

Please help me in understanding this dependency. Thanks.

Regards,
Noor
Frans Meulenbroeks - March 16, 2011, 6:35 a.m.
2011/3/16 Esben Haabendal <eha@doredevelopment.dk>:
> On Tue, 2011-03-15 at 18:03 -0400, Denys Dmytriyenko wrote:
>> On Tue, Mar 15, 2011 at 10:08:37AM +0100, Esben Haabendal wrote:
>> > On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote:
>> > > which occurred sometimes depending on build order (not in clean
>> > > package only builds).
>> >
>> > I would like to raise awareness of the underlying problem here.
>> >
>> > The current dependency/staging model of OE basically has this feature
>> > that a build can be influenced not only by it's own dependencies, but
>> > also what has been build before it (or not).
>> >
>> > I strongly believe that this has to be fixed on the architectural level,
>> > and not just on a case-by-case level as is currently needed.
>> >
>> > I haven't received much feedback on the preivous posting about the
>> > per-recipe staging principle implemented in OE-lite, but I decided to
>> > take this opportunity to re-iterate the fact that the OE-lite
>> > implementation of staging and build dependencies eliminates this
>> > problem.
>> >
>> > I am still very much interested in discussing how to move this
>> > technology from OE-lite to OE, but as it impacts all recipe metadata
>> > (build dependencies has to be redefined), OE community at a large really
>> > needs to value the benefits of solving this problem.
>>
>> Esben,
>>
>> Thanks for raising this issue and making people aware of it!
>>
>> This has previously been discussed several times, including last time during
>> the Yocto Summit in December, where I brought up the question of per-package
>> staging/dependency to Richard's attention, when he introduced the new Shared
>> State (sstate). While per-package dependency wasn't considered a critical
>> must-have feature right away, it was acknowledged as something worthwhile
>> looking at and, according to Richard, should be easy to accomplish with the
>> new sstate functionality...
>
> There is two different objectives in this.
>
> Per-recipe staging and per-package build-time dependencies.  While I
> assume per-recipe staging will be doable to build on top of sstate,
> per-package build-time dependencies is not really related to sstate, but
> is mostly a meta-data issue (man-hours wise), while also requiring
> changes to tbe BitBake cooker.
>
> The final thing that I have found while solving the above issues, is how
> simple (KISS!) all this can be addressed when going from per-recipe
> build-time dependencies to per-package.  And I really think that we need
> more KISS in OE ;-)
>
> /Esben

From a consistency point of view per package build-time dependencies
are highly desirable to get a consistent and reproducible system as
you are always getting the same output irrespective of build order.
Currently this is not the case because configure might pick up
whatever package is build first.
So for (deeply) embedded product development this is a very desirable
feature, QA wise, if not a must have.

Frans
Khem Raj - March 16, 2011, 7:08 a.m.
On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
> Hello,
>
> I was looking at the libxml2 package. While going through the recipes I
> came to know that its native package depends on python-native package as
> well. This dependency is defined in libxml2.inc file.
>
> DEPENDS_virtclass-native = "python-native"
>
> This package build fine after removing this dependency. I am not clear
> why this dependency is being added in libxml2 package. Can somebody
> clear that to me. While looking at the git log I came to know that this
> change is coming from libxml2-native.inc file. But I am not clear why
> this dependency was added as this package build fine without it.
>
> Please help me in understanding this dependency. Thanks.

Try to build this recipe from scratch and see if it still works then we 
can remove the dep
>
> Regards,
> Noor
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Frans Meulenbroeks - March 16, 2011, 7:28 a.m.
2011/3/16 Khem Raj <raj.khem@gmail.com>:
> On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
>>
>> Hello,
>>
>> I was looking at the libxml2 package. While going through the recipes I
>> came to know that its native package depends on python-native package as
>> well. This dependency is defined in libxml2.inc file.
>>
>> DEPENDS_virtclass-native = "python-native"
>>
>> This package build fine after removing this dependency. I am not clear
>> why this dependency is being added in libxml2 package. Can somebody
>> clear that to me. While looking at the git log I came to know that this
>> change is coming from libxml2-native.inc file. But I am not clear why
>> this dependency was added as this package build fine without it.
>>
>> Please help me in understanding this dependency. Thanks.
>
> Try to build this recipe from scratch and see if it still works then we can
> remove the dep

I'm not sure if that is the right way.
It might well be that typically python-native is already there in a
scratch build because some earlier package needed it (or even because
of the scheduling of build steps.

Probably a better way is to build with packaged staging (with staging
dir outside TMPDIR) then rm TMPDIR and bake the package.
Alternately peek into the package to see if it needs python. If it is
needed then the DEPENDS should stay.

Looking at the issue at hand maybe the best person to ask is Martin
Jansa as he last touched that line according to git blame:
3cd9ddbf recipes/libxml/libxml2.inc  (Martin Jansa
2010-02-26 12:32:53 +0100  5) DEPENDS_virtclass-native =
"python-native"

The other thing is that further on in the .inc file I see:
XPY_virtclass-native = " --with-python=${PYTHON_DIR}"

This strongly suggests a hard dependency on python, so removing the
DEPENDS "because it builds without it" does not seem to be a good plan
to me.

My strong feeling is that recipes should be self-contained (so should
list all packages they need to build, minus the ones that are
inherited from classes (and maybe a few implicit ones like maybe make)

Frans
Khem Raj - March 16, 2011, 7:43 a.m.
On 3/16/2011 12:28 AM, Frans Meulenbroeks wrote:
> 2011/3/16 Khem Raj<raj.khem@gmail.com>:
>> On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
>>>
>>> Hello,
>>>
>>> I was looking at the libxml2 package. While going through the recipes I
>>> came to know that its native package depends on python-native package as
>>> well. This dependency is defined in libxml2.inc file.
>>>
>>> DEPENDS_virtclass-native = "python-native"
>>>
>>> This package build fine after removing this dependency. I am not clear
>>> why this dependency is being added in libxml2 package. Can somebody
>>> clear that to me. While looking at the git log I came to know that this
>>> change is coming from libxml2-native.inc file. But I am not clear why
>>> this dependency was added as this package build fine without it.
>>>
>>> Please help me in understanding this dependency. Thanks.
>>
>> Try to build this recipe from scratch and see if it still works then we can
>> remove the dep
>
> I'm not sure if that is the right way.
> It might well be that typically python-native is already there in a
> scratch build because some earlier package needed it (or even because
> of the scheduling of build steps.
>
> Probably a better way is to build with packaged staging (with staging
> dir outside TMPDIR) then rm TMPDIR and bake the package.
> Alternately peek into the package to see if it needs python. If it is
> needed then the DEPENDS should stay.


Did you read carefully what I wrote ?
Frans Meulenbroeks - March 16, 2011, 8 a.m.
2011/3/16 Khem Raj <raj.khem@gmail.com>:
> On 3/16/2011 12:28 AM, Frans Meulenbroeks wrote:
>>
>> 2011/3/16 Khem Raj<raj.khem@gmail.com>:
>>>
>>> On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
>>>>
>>>> Hello,
>>>>
>>>> I was looking at the libxml2 package. While going through the recipes I
>>>> came to know that its native package depends on python-native package as
>>>> well. This dependency is defined in libxml2.inc file.
>>>>
>>>> DEPENDS_virtclass-native = "python-native"
>>>>
>>>> This package build fine after removing this dependency. I am not clear
>>>> why this dependency is being added in libxml2 package. Can somebody
>>>> clear that to me. While looking at the git log I came to know that this
>>>> change is coming from libxml2-native.inc file. But I am not clear why
>>>> this dependency was added as this package build fine without it.
>>>>
>>>> Please help me in understanding this dependency. Thanks.
>>>
>>> Try to build this recipe from scratch and see if it still works then we
>>> can
>>> remove the dep
>>
>> I'm not sure if that is the right way.
>> It might well be that typically python-native is already there in a
>> scratch build because some earlier package needed it (or even because
>> of the scheduling of build steps.
>>
>> Probably a better way is to build with packaged staging (with staging
>> dir outside TMPDIR) then rm TMPDIR and bake the package.
>> Alternately peek into the package to see if it needs python. If it is
>> needed then the DEPENDS should stay.
>
>
> Did you read carefully what I wrote ?

Yes. From scratch for me (and maybe also for others) means rm TMPDIR
(including pstaging).
In that case you might pick up stray packages.
You probably meant keeping pstaging, but at least for some of the
readers that is not obvious.

Frans
Martin Jansa - March 16, 2011, 8:05 a.m.
On Wed, Mar 16, 2011 at 08:28:39AM +0100, Frans Meulenbroeks wrote:
> 2011/3/16 Khem Raj <raj.khem@gmail.com>:
> > On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
> >>
> >> Hello,
> >>
> >> I was looking at the libxml2 package. While going through the recipes I
> >> came to know that its native package depends on python-native package as
> >> well. This dependency is defined in libxml2.inc file.
> >>
> >> DEPENDS_virtclass-native = "python-native"
> >>
> >> This package build fine after removing this dependency. I am not clear
> >> why this dependency is being added in libxml2 package. Can somebody
> >> clear that to me. While looking at the git log I came to know that this
> >> change is coming from libxml2-native.inc file. But I am not clear why
> >> this dependency was added as this package build fine without it.
> >>
> >> Please help me in understanding this dependency. Thanks.
> >
> > Try to build this recipe from scratch and see if it still works then we can
> > remove the dep

That's not enough, libxml2-native builds fine.. but then xml2po
will fail if there isn't libxml2-native built with python support.

see a25f07ca6159e1cd5b8a215ca5da8405fb1b9053

So yes.. for such tests you should build recipe from scratch _and_ check
if resulting package/sysroot is still the same.

Regards,

> I'm not sure if that is the right way.
> It might well be that typically python-native is already there in a
> scratch build because some earlier package needed it (or even because
> of the scheduling of build steps.
> 
> Probably a better way is to build with packaged staging (with staging
> dir outside TMPDIR) then rm TMPDIR and bake the package.
> Alternately peek into the package to see if it needs python. If it is
> needed then the DEPENDS should stay.
> 
> Looking at the issue at hand maybe the best person to ask is Martin
> Jansa as he last touched that line according to git blame:
> 3cd9ddbf recipes/libxml/libxml2.inc  (Martin Jansa
> 2010-02-26 12:32:53 +0100  5) DEPENDS_virtclass-native =
> "python-native"
> 
> The other thing is that further on in the .inc file I see:
> XPY_virtclass-native = " --with-python=${PYTHON_DIR}"
> 
> This strongly suggests a hard dependency on python, so removing the
> DEPENDS "because it builds without it" does not seem to be a good plan
> to me.
> 
> My strong feeling is that recipes should be self-contained (so should
> list all packages they need to build, minus the ones that are
> inherited from classes (and maybe a few implicit ones like maybe make)
> 
> Frans
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Noor Ahsan - March 16, 2011, 8:38 a.m.
Hello,

I made 2 changes, first one was to remove python-native from DEPENDS and
second was commented out XPY_virtclass-native = "
--with-python=${PYTHON_DIR}". Moreover I removed all the stuff from
build dir, so there were not python-native for me. After kicking this
build libxml2-native build fine.

I was looking for xml2po recipe, I couldn't find it. Can you tell me how
to build xml2po package so that I can reproduce the problem.

Regards,
Noor

-----Original Message-----
From: openembedded-devel-bounces@lists.openembedded.org
[mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
Martin Jansa
Sent: Wednesday, March 16, 2011 1:05 PM
To: openembedded-devel@lists.openembedded.org
Subject: Re: [oe] Python-native dependency in libxml2

On Wed, Mar 16, 2011 at 08:28:39AM +0100, Frans Meulenbroeks wrote:
> 2011/3/16 Khem Raj <raj.khem@gmail.com>:
> > On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
> >>
> >> Hello,
> >>
> >> I was looking at the libxml2 package. While going through the
recipes I
> >> came to know that its native package depends on python-native
package as
> >> well. This dependency is defined in libxml2.inc file.
> >>
> >> DEPENDS_virtclass-native = "python-native"
> >>
> >> This package build fine after removing this dependency. I am not
clear
> >> why this dependency is being added in libxml2 package. Can somebody
> >> clear that to me. While looking at the git log I came to know that
this
> >> change is coming from libxml2-native.inc file. But I am not clear
why
> >> this dependency was added as this package build fine without it.
> >>
> >> Please help me in understanding this dependency. Thanks.
> >
> > Try to build this recipe from scratch and see if it still works then
we can
> > remove the dep

That's not enough, libxml2-native builds fine.. but then xml2po
will fail if there isn't libxml2-native built with python support.

see a25f07ca6159e1cd5b8a215ca5da8405fb1b9053

So yes.. for such tests you should build recipe from scratch _and_ check
if resulting package/sysroot is still the same.

Regards,

> I'm not sure if that is the right way.
> It might well be that typically python-native is already there in a
> scratch build because some earlier package needed it (or even because
> of the scheduling of build steps.
> 
> Probably a better way is to build with packaged staging (with staging
> dir outside TMPDIR) then rm TMPDIR and bake the package.
> Alternately peek into the package to see if it needs python. If it is
> needed then the DEPENDS should stay.
> 
> Looking at the issue at hand maybe the best person to ask is Martin
> Jansa as he last touched that line according to git blame:
> 3cd9ddbf recipes/libxml/libxml2.inc  (Martin Jansa
> 2010-02-26 12:32:53 +0100  5) DEPENDS_virtclass-native =
> "python-native"
> 
> The other thing is that further on in the .inc file I see:
> XPY_virtclass-native = " --with-python=${PYTHON_DIR}"
> 
> This strongly suggests a hard dependency on python, so removing the
> DEPENDS "because it builds without it" does not seem to be a good plan
> to me.
> 
> My strong feeling is that recipes should be self-contained (so should
> list all packages they need to build, minus the ones that are
> inherited from classes (and maybe a few implicit ones like maybe make)
> 
> Frans
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Noor Ahsan - March 17, 2011, 10:40 a.m.
Hello,

Today I got the fresh stuff and tried to build gnome-doc-utils-native
(to build xml2po package). I am building on Ubuntu host 9.10. First it
gave me an error that it could not find intltoolize command. You can see
the attached build log "build_intltool_error,log". So I added intltool
package in gnome-doc-utils-native DEPENDS so it built successfully. You
can see the build log "build_intltool_fix.log". Then I removed all the
build stuff from build DIR and commented DEPENDS_virtclass-native =
"python-native" and XPY_virtclass-native = "
--with-python=${PYTHON_DIR}" in libxml2.inc file. Rebuild
gnome-doc-utils-native. It built successfully again. The build log is
also attached "build_no_python.log". I think there is no need of python
dependency in libxml2 package. Kindly correct me if I am doing anything
wrong.

We need to fix gnome-doc-utils-native recipe, so that it does not show
intltoolize error.

Regards,
Noor

-----Original Message-----
From: openembedded-devel-bounces@lists.openembedded.org
[mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
Ahsan, Noor
Sent: Wednesday, March 16, 2011 1:39 PM
To: openembedded-devel@lists.openembedded.org
Subject: Re: [oe] Python-native dependency in libxml2

Hello,

I made 2 changes, first one was to remove python-native from DEPENDS and
second was commented out XPY_virtclass-native = "
--with-python=${PYTHON_DIR}". Moreover I removed all the stuff from
build dir, so there were not python-native for me. After kicking this
build libxml2-native build fine.

I was looking for xml2po recipe, I couldn't find it. Can you tell me how
to build xml2po package so that I can reproduce the problem.

Regards,
Noor

-----Original Message-----
From: openembedded-devel-bounces@lists.openembedded.org
[mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
Martin Jansa
Sent: Wednesday, March 16, 2011 1:05 PM
To: openembedded-devel@lists.openembedded.org
Subject: Re: [oe] Python-native dependency in libxml2

On Wed, Mar 16, 2011 at 08:28:39AM +0100, Frans Meulenbroeks wrote:
> 2011/3/16 Khem Raj <raj.khem@gmail.com>:
> > On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
> >>
> >> Hello,
> >>
> >> I was looking at the libxml2 package. While going through the
recipes I
> >> came to know that its native package depends on python-native
package as
> >> well. This dependency is defined in libxml2.inc file.
> >>
> >> DEPENDS_virtclass-native = "python-native"
> >>
> >> This package build fine after removing this dependency. I am not
clear
> >> why this dependency is being added in libxml2 package. Can somebody
> >> clear that to me. While looking at the git log I came to know that
this
> >> change is coming from libxml2-native.inc file. But I am not clear
why
> >> this dependency was added as this package build fine without it.
> >>
> >> Please help me in understanding this dependency. Thanks.
> >
> > Try to build this recipe from scratch and see if it still works then
we can
> > remove the dep

That's not enough, libxml2-native builds fine.. but then xml2po
will fail if there isn't libxml2-native built with python support.

see a25f07ca6159e1cd5b8a215ca5da8405fb1b9053

So yes.. for such tests you should build recipe from scratch _and_ check
if resulting package/sysroot is still the same.

Regards,

> I'm not sure if that is the right way.
> It might well be that typically python-native is already there in a
> scratch build because some earlier package needed it (or even because
> of the scheduling of build steps.
> 
> Probably a better way is to build with packaged staging (with staging
> dir outside TMPDIR) then rm TMPDIR and bake the package.
> Alternately peek into the package to see if it needs python. If it is
> needed then the DEPENDS should stay.
> 
> Looking at the issue at hand maybe the best person to ask is Martin
> Jansa as he last touched that line according to git blame:
> 3cd9ddbf recipes/libxml/libxml2.inc  (Martin Jansa
> 2010-02-26 12:32:53 +0100  5) DEPENDS_virtclass-native =
> "python-native"
> 
> The other thing is that further on in the .inc file I see:
> XPY_virtclass-native = " --with-python=${PYTHON_DIR}"
> 
> This strongly suggests a hard dependency on python, so removing the
> DEPENDS "because it builds without it" does not seem to be a good plan
> to me.
> 
> My strong feeling is that recipes should be self-contained (so should
> list all packages they need to build, minus the ones that are
> inherited from classes (and maybe a few implicit ones like maybe make)
> 
> Frans
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Phil Blundell - March 17, 2011, 11:18 a.m.
On Tue, 2011-03-15 at 10:08 +0100, Esben Haabendal wrote:
> The current dependency/staging model of OE basically has this feature
> that a build can be influenced not only by it's own dependencies, but
> also what has been build before it (or not).
> 
> I strongly believe that this has to be fixed on the architectural level,
> and not just on a case-by-case level as is currently needed.

Yes, agreed.  This clearly sucks, and indeed I think it is probably the
largest single area of architectural suckage in OE today.

> I am still very much interested in discussing how to move this
> technology from OE-lite to OE, but as it impacts all recipe metadata
> (build dependencies has to be redefined), OE community at a large really
> needs to value the benefits of solving this problem.

The benefits of solving the problem are clearly very great, but I don't
think OE itself is really in a position to embrace a "big bang" kind of
change which would require redefining all the build dependencies.  I
think we need to find a technological solution which will work with our
current DEPENDS scheme.

I did do some work in that direction about a year ago, but I got
sidetracked into dealing with some toolchain suckage (which was making
it difficult to construct the per-recipe sysroots in the way that I
wanted) and then ran out of time.  I would like to get back to working
on this stuff but right now it is difficult for me to set aside the time
for it.

p.
Esben Haabendal - March 17, 2011, 2:43 p.m.
On Thu, 2011-03-17 at 11:18 +0000, Phil Blundell wrote:
> 
> > I am still very much interested in discussing how to move this
> > technology from OE-lite to OE, but as it impacts all recipe metadata
> > (build dependencies has to be redefined), OE community at a large
> really
> > needs to value the benefits of solving this problem.
> 
> The benefits of solving the problem are clearly very great, but I
> don't think OE itself is really in a position to embrace a "big bang"
> kind of change which would require redefining all the build
> dependencies.  I think we need to find a technological solution which
> will work with our current DEPENDS scheme. 

I know that this seems to be the general consensus in OE, but I do not
agree with it.  Sometimes architectural work requires working out the
details on a branch, and then when it is mostly done, do the "big bang"
which will then not really be that "big" anymore, as most of the issues
have been resolved.

Is OE really in a position to permantly settle for something suboptimal
in such a central area?

/Esben
Graeme Gregory - March 17, 2011, 2:52 p.m.
On 17/03/2011 14:43, Esben Haabendal wrote:
> On Thu, 2011-03-17 at 11:18 +0000, Phil Blundell wrote:
>>> I am still very much interested in discussing how to move this
>>> technology from OE-lite to OE, but as it impacts all recipe metadata
>>> (build dependencies has to be redefined), OE community at a large
>> really
>>> needs to value the benefits of solving this problem.
>> The benefits of solving the problem are clearly very great, but I
>> don't think OE itself is really in a position to embrace a "big bang"
>> kind of change which would require redefining all the build
>> dependencies.  I think we need to find a technological solution which
>> will work with our current DEPENDS scheme. 
> I know that this seems to be the general consensus in OE, but I do not
> agree with it.  Sometimes architectural work requires working out the
> details on a branch, and then when it is mostly done, do the "big bang"
> which will then not really be that "big" anymore, as most of the issues
> have been resolved.
>
> Is OE really in a position to permantly settle for something suboptimal
> in such a central area?
>
I would suggest a branch of oe-core is the ideal place to do this work.
It can be proved on oe-core then rolled out to meta-oe at a later date.

Graeme
Phil Blundell - March 17, 2011, 3:07 p.m.
On Thu, 2011-03-17 at 15:43 +0100, Esben Haabendal wrote:
> Is OE really in a position to permantly settle for something suboptimal
> in such a central area?

No, but rejecting the big bang doesn't mean that we can't make the
change; it just means that we need to find a way to make the old and new
arrangements coexist for a transition period.  This is the way every
other architectural change in OE has been done and I don't see any
reason that this one needs to be different.

If we really did want to go ahead and have a metadata flag day then no
doubt there are plenty of other things we would like to change at the
same time.  But, thus far, this has never seemed to justify the
disruption of breaking every recipe in both the main OE repo and third
party overlays.

I must admit that I'm also slightly unclear about why the change in
build dependency specifications is a prerequisite for doing per-recipe
sysroots.  Is that just an artifact of the way you implemented it in
OE-lite or is there some fundamental constraint that means it needs to
be this way?

p.
Koen Kooi - March 17, 2011, 3:24 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 17-03-11 15:52, Graeme Gregory wrote:
> On 17/03/2011 14:43, Esben Haabendal wrote:
>> On Thu, 2011-03-17 at 11:18 +0000, Phil Blundell wrote:
>>>> I am still very much interested in discussing how to move this
>>>> technology from OE-lite to OE, but as it impacts all recipe metadata
>>>> (build dependencies has to be redefined), OE community at a large
>>> really
>>>> needs to value the benefits of solving this problem.
>>> The benefits of solving the problem are clearly very great, but I
>>> don't think OE itself is really in a position to embrace a "big bang"
>>> kind of change which would require redefining all the build
>>> dependencies.  I think we need to find a technological solution which
>>> will work with our current DEPENDS scheme. 
>> I know that this seems to be the general consensus in OE, but I do not
>> agree with it.  Sometimes architectural work requires working out the
>> details on a branch, and then when it is mostly done, do the "big bang"
>> which will then not really be that "big" anymore, as most of the issues
>> have been resolved.
>>
>> Is OE really in a position to permantly settle for something suboptimal
>> in such a central area?
>>
> I would suggest a branch of oe-core is the ideal place to do this work.
> It can be proved on oe-core then rolled out to meta-oe at a later date.

OE-core already features per-machine sysroots, so the main problem we
had, relocatability, has already been solved.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFNgielMkyGM64RGpERAqcIAKCyT44a9MzDF+esO6DgRbtBXi4L8ACdGzpN
4vN5d5EtUwbo3cNHfMl9XQ4=
=eFCq
-----END PGP SIGNATURE-----
Esben Haabendal - March 17, 2011, 5:52 p.m.
On Thu, 2011-03-17 at 15:07 +0000, Phil Blundell wrote:
> On Thu, 2011-03-17 at 15:43 +0100, Esben Haabendal wrote:
> > Is OE really in a position to permantly settle for something suboptimal
> > in such a central area?
> 
> No, but rejecting the big bang doesn't mean that we can't make the
> change; it just means that we need to find a way to make the old and new
> arrangements coexist for a transition period.  This is the way every
> other architectural change in OE has been done and I don't see any
> reason that this one needs to be different.

Exactly

> 
> If we really did want to go ahead and have a metadata flag day then no
> doubt there are plenty of other things we would like to change at the
> same time.  But, thus far, this has never seemed to justify the
> disruption of breaking every recipe in both the main OE repo and third
> party overlays.

Well, it might be possible to minimize the disruption for a transitional
period by carefully specifying some catch-all build-time package
dependencies pulling in all packages for recipes not ported yet.

I think it would be much more worthwhile to put effort into this than to
push for per-recipe sysroot with the current sstate solution.

Having package-based build-time dependencies, using the same packages as
when building run-time images, is much simpler than what is currently
done in OE, while at the same time giving increased flexibility in
specifying the build-time dependencies needed for a recipe and it's
packages.

But you might have to try it out before really understanding why you
cannot live without it ;-)

> I must admit that I'm also slightly unclear about why the change in
> build dependency specifications is a prerequisite for doing per-recipe
> sysroots.

It might not be, but the result will of-course not be the same.

>Is that just an artifact of the way you implemented it in
> OE-lite or is there some fundamental constraint that means it needs to
> be this way?

I don't think there are any fundamental constraint.  Sometimes doing
things right are simply so much easier.  As a result, I already do have
a solution with both per-recipe staging/sysroot and buld-time
dependencies using target packages.  I feel pretty confident that this
was the fast path, which is sort of backed by the fact that it is done,
whereas the path (detour?) currently being worked on in OE is still just
on it's way.

/Esben
Phil Blundell - March 17, 2011, 6:05 p.m.
On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote:
> Well, it might be possible to minimize the disruption for a transitional
> period by carefully specifying some catch-all build-time package
> dependencies pulling in all packages for recipes not ported yet.

Yes, that's the sort of thing I was thinking of.  It isn't even totally
obvious to me that specifying individual packages is any better than
"all packages from recipe X", since with the former you then have a
potential maintenance headache if files get moved from one package to
another.

p.
Esben Haabendal - March 17, 2011, 7:58 p.m.
On Thu, 2011-03-17 at 18:05 +0000, Phil Blundell wrote:
> On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote:
> > Well, it might be possible to minimize the disruption for a transitional
> > period by carefully specifying some catch-all build-time package
> > dependencies pulling in all packages for recipes not ported yet.
> 
> Yes, that's the sort of thing I was thinking of.  It isn't even totally
> obvious to me that specifying individual packages is any better than
> "all packages from recipe X", since with the former you then have a
> potential maintenance headache if files get moved from one package to
> another.

There is a number of ways that I believe package based build
dependencies are better than recipe based.

a) It is possible to depend on parts of a recipe, which fx. is useful
when a recipe provides more than 1 library, where you might not want all
of them.

b) To build a recipe, you depend on some stuff which you don't need to,
or perhaps even really don't want to pass on to recipes depending on
this recipe.

c) KISS. Using the actual target packages for satisfying build-time
dependencies are a very simple approach, which I strongly believe will
make OE a better tool in the long run, by reducing complexity, and thus
lowering the bar for contributing to OE archictural work.

The maintenance headache you talk about is already there.  In OE-lite
build-time dependency 95% of the time just follow the run-time
dependencies, perhaps making it easier to maintain than OE, as we don't
have to think about another type of "item names" to depend on.

/Esben
Phil Blundell - March 17, 2011, 9 p.m.
On Thu, 2011-03-17 at 20:58 +0100, Esben Haabendal wrote:
> On Thu, 2011-03-17 at 18:05 +0000, Phil Blundell wrote:
> > On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote:
> > > Well, it might be possible to minimize the disruption for a transitional
> > > period by carefully specifying some catch-all build-time package
> > > dependencies pulling in all packages for recipes not ported yet.
> > 
> > Yes, that's the sort of thing I was thinking of.  It isn't even totally
> > obvious to me that specifying individual packages is any better than
> > "all packages from recipe X", since with the former you then have a
> > potential maintenance headache if files get moved from one package to
> > another.
> 
> There is a number of ways that I believe package based build
> dependencies are better than recipe based.
> 
> a) It is possible to depend on parts of a recipe, which fx. is useful
> when a recipe provides more than 1 library, where you might not want all
> of them.

You'd still need to have built all of them (at least as far as the end
of do_compile) though, right?  In that case it doesn't seem as though
installing the unnecessary ones in the sysroot is all that big a
hardship.

> b) To build a recipe, you depend on some stuff which you don't need to,
> or perhaps even really don't want to pass on to recipes depending on
> this recipe.

Sorry, I don't understand what you're saying here.

> c) KISS. Using the actual target packages for satisfying build-time
> dependencies are a very simple approach, which I strongly believe will
> make OE a better tool in the long run, by reducing complexity, and thus
> lowering the bar for contributing to OE archictural work.

I agree with this, but it seems unrelated to the question of whether
build dependencies should be specified in terms of recipes or output
packages.

> The maintenance headache you talk about is already there.  In OE-lite
> build-time dependency 95% of the time just follow the run-time
> dependencies, perhaps making it easier to maintain than OE, as we don't
> have to think about another type of "item names" to depend on.

(How) do you deal with library package renaming in OE-lite?

p.
Esben Haabendal - March 18, 2011, 5:29 a.m.
On Thu, 2011-03-17 at 21:00 +0000, Phil Blundell wrote:
> On Thu, 2011-03-17 at 20:58 +0100, Esben Haabendal wrote:
> > There is a number of ways that I believe package based build
> > dependencies are better than recipe based.
> > 
> > a) It is possible to depend on parts of a recipe, which fx. is useful
> > when a recipe provides more than 1 library, where you might not want all
> > of them.
> 
> You'd still need to have built all of them (at least as far as the end
> of do_compile) though, right?

Of-course.  All recipes providing something for  a do_stage must be
build up till the do_package stage, so I can unpack the actual target
packages needed.

> In that case it doesn't seem as though
> installing the unnecessary ones in the sysroot is all that big a
> hardship.

It is not to save time, it is about controlling what goes into a build.
As many software packages scans for presence of various components
through the availability of headers and libraries, I believe it is
important to decide not just what goes into the stage directory, but
also what does not.  So in OE-lite, if I say I depend on libfoo, I do
not get foobar, just because it was built by the same recipe.

> > b) To build a recipe, you depend on some stuff which you don't need to,
> > or perhaps even really don't want to pass on to recipes depending on
> > this recipe.
> 
> Sorry, I don't understand what you're saying here.

Example:

I build recipe A.
Recipe A depends on package B-b from recipe B.
Recipe B depends on package C-c from recipe C, and D-d-native from
recipe D.
Recipe A only get's package B-b staged, unless the B-b package has
specified some build-time dependencies.

This is relevant both for target library and native tools dependencies.

> > c) KISS. Using the actual target packages for satisfying build-time
> > dependencies are a very simple approach, which I strongly believe will
> > make OE a better tool in the long run, by reducing complexity, and thus
> > lowering the bar for contributing to OE archictural work.
> 
> I agree with this, but it seems unrelated to the question of whether
> build dependencies should be specified in terms of recipes or output
> packages.

Yes, I get your point.  You could of-course still specify build
dependencies using recipe names/provides, and then just unpack all
target packages build by that recipe.

This would give you the major part of the KISS win I see, except for in
the dependency handling code of BitBake (and in the learning curve of OE
developers).  Keep in mind that by using packages for build-time
dependencies, the mechanism is exactly identical for run-time and
build-time dependencies.  Exactly the same code in BitBake handles both,
and the same mechanism needs to be understood.  This is also part of the
KISS win of doing it this way.

> > The maintenance headache you talk about is already there.  In OE-lite
> > build-time dependency 95% of the time just follow the run-time
> > dependencies, perhaps making it easier to maintain than OE, as we don't
> > have to think about another type of "item names" to depend on.
> 
> (How) do you deal with library package renaming in OE-lite?

What exactly do you mean?  (We are doing several things with library
package naming...)

/Esben
Noor Ahsan - March 18, 2011, 7:41 a.m.
Hello Martin/Frans and Khem,

Any recommendation after following analysis that how should I proceed.
Should I send the patches to community?

Regards,
Noor

-----Original Message-----
From: openembedded-devel-bounces@lists.openembedded.org
[mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
Ahsan, Noor
Sent: Thursday, March 17, 2011 3:41 PM
To: openembedded-devel@lists.openembedded.org
Subject: Re: [oe] Python-native dependency in libxml2

Hello,

Today I got the fresh stuff and tried to build gnome-doc-utils-native
(to build xml2po package). I am building on Ubuntu host 9.10. First it
gave me an error that it could not find intltoolize command. You can see
the attached build log "build_intltool_error,log". So I added intltool
package in gnome-doc-utils-native DEPENDS so it built successfully. You
can see the build log "build_intltool_fix.log". Then I removed all the
build stuff from build DIR and commented DEPENDS_virtclass-native =
"python-native" and XPY_virtclass-native = "
--with-python=${PYTHON_DIR}" in libxml2.inc file. Rebuild
gnome-doc-utils-native. It built successfully again. The build log is
also attached "build_no_python.log". I think there is no need of python
dependency in libxml2 package. Kindly correct me if I am doing anything
wrong.

We need to fix gnome-doc-utils-native recipe, so that it does not show
intltoolize error.

Regards,
Noor

-----Original Message-----
From: openembedded-devel-bounces@lists.openembedded.org
[mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
Ahsan, Noor
Sent: Wednesday, March 16, 2011 1:39 PM
To: openembedded-devel@lists.openembedded.org
Subject: Re: [oe] Python-native dependency in libxml2

Hello,

I made 2 changes, first one was to remove python-native from DEPENDS and
second was commented out XPY_virtclass-native = "
--with-python=${PYTHON_DIR}". Moreover I removed all the stuff from
build dir, so there were not python-native for me. After kicking this
build libxml2-native build fine.

I was looking for xml2po recipe, I couldn't find it. Can you tell me how
to build xml2po package so that I can reproduce the problem.

Regards,
Noor

-----Original Message-----
From: openembedded-devel-bounces@lists.openembedded.org
[mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
Martin Jansa
Sent: Wednesday, March 16, 2011 1:05 PM
To: openembedded-devel@lists.openembedded.org
Subject: Re: [oe] Python-native dependency in libxml2

On Wed, Mar 16, 2011 at 08:28:39AM +0100, Frans Meulenbroeks wrote:
> 2011/3/16 Khem Raj <raj.khem@gmail.com>:
> > On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
> >>
> >> Hello,
> >>
> >> I was looking at the libxml2 package. While going through the
recipes I
> >> came to know that its native package depends on python-native
package as
> >> well. This dependency is defined in libxml2.inc file.
> >>
> >> DEPENDS_virtclass-native = "python-native"
> >>
> >> This package build fine after removing this dependency. I am not
clear
> >> why this dependency is being added in libxml2 package. Can somebody
> >> clear that to me. While looking at the git log I came to know that
this
> >> change is coming from libxml2-native.inc file. But I am not clear
why
> >> this dependency was added as this package build fine without it.
> >>
> >> Please help me in understanding this dependency. Thanks.
> >
> > Try to build this recipe from scratch and see if it still works then
we can
> > remove the dep

That's not enough, libxml2-native builds fine.. but then xml2po
will fail if there isn't libxml2-native built with python support.

see a25f07ca6159e1cd5b8a215ca5da8405fb1b9053

So yes.. for such tests you should build recipe from scratch _and_ check
if resulting package/sysroot is still the same.

Regards,

> I'm not sure if that is the right way.
> It might well be that typically python-native is already there in a
> scratch build because some earlier package needed it (or even because
> of the scheduling of build steps.
> 
> Probably a better way is to build with packaged staging (with staging
> dir outside TMPDIR) then rm TMPDIR and bake the package.
> Alternately peek into the package to see if it needs python. If it is
> needed then the DEPENDS should stay.
> 
> Looking at the issue at hand maybe the best person to ask is Martin
> Jansa as he last touched that line according to git blame:
> 3cd9ddbf recipes/libxml/libxml2.inc  (Martin Jansa
> 2010-02-26 12:32:53 +0100  5) DEPENDS_virtclass-native =
> "python-native"
> 
> The other thing is that further on in the .inc file I see:
> XPY_virtclass-native = " --with-python=${PYTHON_DIR}"
> 
> This strongly suggests a hard dependency on python, so removing the
> DEPENDS "because it builds without it" does not seem to be a good plan
> to me.
> 
> My strong feeling is that recipes should be self-contained (so should
> list all packages they need to build, minus the ones that are
> inherited from classes (and maybe a few implicit ones like maybe make)
> 
> Frans
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Martin Jansa - March 18, 2011, 8:38 a.m.
On Fri, Mar 18, 2011 at 08:41:40AM +0100, Ahsan, Noor wrote:
> Hello Martin/Frans and Khem,
> Any recommendation after following analysis that how should I proceed.
> Should I send the patches to community?

Maybe koen will reply with more info about xml2po issue he found, imho
it wasn't in xml2po build but while using xml2to to generate docs in some
other recipe.
 
IIRC I had similar issue in some other recipe, but unfortunatelly don't 
remember which one.

Regards,

> 
> Regards,
> Noor
> 
> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> Ahsan, Noor
> Sent: Thursday, March 17, 2011 3:41 PM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Python-native dependency in libxml2
> 
> Hello,
> 
> Today I got the fresh stuff and tried to build gnome-doc-utils-native
> (to build xml2po package). I am building on Ubuntu host 9.10. First it
> gave me an error that it could not find intltoolize command. You can see
> the attached build log "build_intltool_error,log". So I added intltool
> package in gnome-doc-utils-native DEPENDS so it built successfully. You
> can see the build log "build_intltool_fix.log". Then I removed all the
> build stuff from build DIR and commented DEPENDS_virtclass-native =
> "python-native" and XPY_virtclass-native = "
> --with-python=${PYTHON_DIR}" in libxml2.inc file. Rebuild
> gnome-doc-utils-native. It built successfully again. The build log is
> also attached "build_no_python.log". I think there is no need of python
> dependency in libxml2 package. Kindly correct me if I am doing anything
> wrong.
> 
> We need to fix gnome-doc-utils-native recipe, so that it does not show
> intltoolize error.
> 
> Regards,
> Noor
> 
> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> Ahsan, Noor
> Sent: Wednesday, March 16, 2011 1:39 PM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Python-native dependency in libxml2
> 
> Hello,
> 
> I made 2 changes, first one was to remove python-native from DEPENDS and
> second was commented out XPY_virtclass-native = "
> --with-python=${PYTHON_DIR}". Moreover I removed all the stuff from
> build dir, so there were not python-native for me. After kicking this
> build libxml2-native build fine.
> 
> I was looking for xml2po recipe, I couldn't find it. Can you tell me how
> to build xml2po package so that I can reproduce the problem.
> 
> Regards,
> Noor
> 
> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> Martin Jansa
> Sent: Wednesday, March 16, 2011 1:05 PM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Python-native dependency in libxml2
> 
> On Wed, Mar 16, 2011 at 08:28:39AM +0100, Frans Meulenbroeks wrote:
> > 2011/3/16 Khem Raj <raj.khem@gmail.com>:
> > > On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
> > >>
> > >> Hello,
> > >>
> > >> I was looking at the libxml2 package. While going through the
> recipes I
> > >> came to know that its native package depends on python-native
> package as
> > >> well. This dependency is defined in libxml2.inc file.
> > >>
> > >> DEPENDS_virtclass-native = "python-native"
> > >>
> > >> This package build fine after removing this dependency. I am not
> clear
> > >> why this dependency is being added in libxml2 package. Can somebody
> > >> clear that to me. While looking at the git log I came to know that
> this
> > >> change is coming from libxml2-native.inc file. But I am not clear
> why
> > >> this dependency was added as this package build fine without it.
> > >>
> > >> Please help me in understanding this dependency. Thanks.
> > >
> > > Try to build this recipe from scratch and see if it still works then
> we can
> > > remove the dep
> 
> That's not enough, libxml2-native builds fine.. but then xml2po
> will fail if there isn't libxml2-native built with python support.
> 
> see a25f07ca6159e1cd5b8a215ca5da8405fb1b9053
> 
> So yes.. for such tests you should build recipe from scratch _and_ check
> if resulting package/sysroot is still the same.
> 
> Regards,
> 
> > I'm not sure if that is the right way.
> > It might well be that typically python-native is already there in a
> > scratch build because some earlier package needed it (or even because
> > of the scheduling of build steps.
> > 
> > Probably a better way is to build with packaged staging (with staging
> > dir outside TMPDIR) then rm TMPDIR and bake the package.
> > Alternately peek into the package to see if it needs python. If it is
> > needed then the DEPENDS should stay.
> > 
> > Looking at the issue at hand maybe the best person to ask is Martin
> > Jansa as he last touched that line according to git blame:
> > 3cd9ddbf recipes/libxml/libxml2.inc  (Martin Jansa
> > 2010-02-26 12:32:53 +0100  5) DEPENDS_virtclass-native =
> > "python-native"
> > 
> > The other thing is that further on in the .inc file I see:
> > XPY_virtclass-native = " --with-python=${PYTHON_DIR}"
> > 
> > This strongly suggests a hard dependency on python, so removing the
> > DEPENDS "because it builds without it" does not seem to be a good plan
> > to me.
> > 
> > My strong feeling is that recipes should be self-contained (so should
> > list all packages they need to build, minus the ones that are
> > inherited from classes (and maybe a few implicit ones like maybe make)
> > 
> > Frans
> > 
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Frans Meulenbroeks - March 18, 2011, 9:01 a.m.
2011/3/18 Ahsan, Noor <Noor_Ahsan@mentor.com>:
> Hello Martin/Frans and Khem,
>
> Any recommendation after following analysis that how should I proceed.
> Should I send the patches to community?
>
> Regards,
> Noor

I think Martin is in a better position to judge this than I am.
Did you also verify that the generated output (the packages and the
contents of the packages) is identical in the python and no-python
case?

Best regards, Frans
Koen Kooi - March 18, 2011, 9:47 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18-03-11 09:38, Martin Jansa wrote:
> On Fri, Mar 18, 2011 at 08:41:40AM +0100, Ahsan, Noor wrote:
>> Hello Martin/Frans and Khem,
>> Any recommendation after following analysis that how should I proceed.
>> Should I send the patches to community?
> 
> Maybe koen will reply with more info about xml2po issue he found, imho
> it wasn't in xml2po build but while using xml2to to generate docs in some
> other recipe.

The issue I had that I needed xml2{t,p}o to work on the target as well
for native compilation. If that still works, Noors changes is fine by me.

regards,

Koen

>  
> IIRC I had similar issue in some other recipe, but unfortunatelly don't 
> remember which one.
> 
> Regards,
> 
>>
>> Regards,
>> Noor
>>
>> -----Original Message-----
>> From: openembedded-devel-bounces@lists.openembedded.org
>> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
>> Ahsan, Noor
>> Sent: Thursday, March 17, 2011 3:41 PM
>> To: openembedded-devel@lists.openembedded.org
>> Subject: Re: [oe] Python-native dependency in libxml2
>>
>> Hello,
>>
>> Today I got the fresh stuff and tried to build gnome-doc-utils-native
>> (to build xml2po package). I am building on Ubuntu host 9.10. First it
>> gave me an error that it could not find intltoolize command. You can see
>> the attached build log "build_intltool_error,log". So I added intltool
>> package in gnome-doc-utils-native DEPENDS so it built successfully. You
>> can see the build log "build_intltool_fix.log". Then I removed all the
>> build stuff from build DIR and commented DEPENDS_virtclass-native =
>> "python-native" and XPY_virtclass-native = "
>> --with-python=${PYTHON_DIR}" in libxml2.inc file. Rebuild
>> gnome-doc-utils-native. It built successfully again. The build log is
>> also attached "build_no_python.log". I think there is no need of python
>> dependency in libxml2 package. Kindly correct me if I am doing anything
>> wrong.
>>
>> We need to fix gnome-doc-utils-native recipe, so that it does not show
>> intltoolize error.
>>
>> Regards,
>> Noor
>>
>> -----Original Message-----
>> From: openembedded-devel-bounces@lists.openembedded.org
>> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
>> Ahsan, Noor
>> Sent: Wednesday, March 16, 2011 1:39 PM
>> To: openembedded-devel@lists.openembedded.org
>> Subject: Re: [oe] Python-native dependency in libxml2
>>
>> Hello,
>>
>> I made 2 changes, first one was to remove python-native from DEPENDS and
>> second was commented out XPY_virtclass-native = "
>> --with-python=${PYTHON_DIR}". Moreover I removed all the stuff from
>> build dir, so there were not python-native for me. After kicking this
>> build libxml2-native build fine.
>>
>> I was looking for xml2po recipe, I couldn't find it. Can you tell me how
>> to build xml2po package so that I can reproduce the problem.
>>
>> Regards,
>> Noor
>>
>> -----Original Message-----
>> From: openembedded-devel-bounces@lists.openembedded.org
>> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
>> Martin Jansa
>> Sent: Wednesday, March 16, 2011 1:05 PM
>> To: openembedded-devel@lists.openembedded.org
>> Subject: Re: [oe] Python-native dependency in libxml2
>>
>> On Wed, Mar 16, 2011 at 08:28:39AM +0100, Frans Meulenbroeks wrote:
>>> 2011/3/16 Khem Raj <raj.khem@gmail.com>:
>>>> On 3/15/2011 11:22 PM, Ahsan, Noor wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I was looking at the libxml2 package. While going through the
>> recipes I
>>>>> came to know that its native package depends on python-native
>> package as
>>>>> well. This dependency is defined in libxml2.inc file.
>>>>>
>>>>> DEPENDS_virtclass-native = "python-native"
>>>>>
>>>>> This package build fine after removing this dependency. I am not
>> clear
>>>>> why this dependency is being added in libxml2 package. Can somebody
>>>>> clear that to me. While looking at the git log I came to know that
>> this
>>>>> change is coming from libxml2-native.inc file. But I am not clear
>> why
>>>>> this dependency was added as this package build fine without it.
>>>>>
>>>>> Please help me in understanding this dependency. Thanks.
>>>>
>>>> Try to build this recipe from scratch and see if it still works then
>> we can
>>>> remove the dep
>>
>> That's not enough, libxml2-native builds fine.. but then xml2po
>> will fail if there isn't libxml2-native built with python support.
>>
>> see a25f07ca6159e1cd5b8a215ca5da8405fb1b9053
>>
>> So yes.. for such tests you should build recipe from scratch _and_ check
>> if resulting package/sysroot is still the same.
>>
>> Regards,
>>
>>> I'm not sure if that is the right way.
>>> It might well be that typically python-native is already there in a
>>> scratch build because some earlier package needed it (or even because
>>> of the scheduling of build steps.
>>>
>>> Probably a better way is to build with packaged staging (with staging
>>> dir outside TMPDIR) then rm TMPDIR and bake the package.
>>> Alternately peek into the package to see if it needs python. If it is
>>> needed then the DEPENDS should stay.
>>>
>>> Looking at the issue at hand maybe the best person to ask is Martin
>>> Jansa as he last touched that line according to git blame:
>>> 3cd9ddbf recipes/libxml/libxml2.inc  (Martin Jansa
>>> 2010-02-26 12:32:53 +0100  5) DEPENDS_virtclass-native =
>>> "python-native"
>>>
>>> The other thing is that further on in the .inc file I see:
>>> XPY_virtclass-native = " --with-python=${PYTHON_DIR}"
>>>
>>> This strongly suggests a hard dependency on python, so removing the
>>> DEPENDS "because it builds without it" does not seem to be a good plan
>>> to me.
>>>
>>> My strong feeling is that recipes should be self-contained (so should
>>> list all packages they need to build, minus the ones that are
>>> inherited from classes (and maybe a few implicit ones like maybe make)
>>>
>>> Frans
>>>
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>
>> -- 
>> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFNgypFMkyGM64RGpERAka3AJ91Dpu6+SCx3MH0HtK6LUyEnaH7tACeM4EP
b0nPj51+QTs4wdONs4BCP7Y=
=Zxcy
-----END PGP SIGNATURE-----
Phil Blundell - March 18, 2011, 10:26 a.m.
On Fri, 2011-03-18 at 06:29 +0100, Esben Haabendal wrote:
> Yes, I get your point.  You could of-course still specify build
> dependencies using recipe names/provides, and then just unpack all
> target packages build by that recipe.
> 
> This would give you the major part of the KISS win I see, except for in
> the dependency handling code of BitBake (and in the learning curve of OE
> developers).  Keep in mind that by using packages for build-time
> dependencies, the mechanism is exactly identical for run-time and
> build-time dependencies.  Exactly the same code in BitBake handles both,
> and the same mechanism needs to be understood.  This is also part of the
> KISS win of doing it this way.

Right.  So, it seems to me that the appropriate way to attack the
problem in OE is to start by doing exactly that: leave the syntax of
DEPENDS alone for now, treat each entry in DEPENDS as meaning "all
packages built by the named recipe", and then implement the per-recipe
sysroot construction that I think we are agreed is highly desirable.

Then, as a later refinement, we could introduce a mechanism for
specifying dependencies more precisely, at the package level rather than
the recipe level.  We could do that either by extending the syntax of
DEPENDS, something like:

DEPENDS = "boost:boost-iostreams"

(to say that you wanted just the boost-iostreams package from the boost
recipe)

or, alternately, by introducing a new variable which would either
supplement or replace DEPENDS.  Either of those would allow us to
migrate to fine-grained dependencies without breaking existing recipes.

> > (How) do you deal with library package renaming in OE-lite?
> 
> What exactly do you mean?  (We are doing several things with library
> package naming...)

I was thinking of situations like the Debian package autonamer, ie the
thing that causes glibc-dev to come out named "libc6-dev.ipk" or similar
depending on the soname of the library that was built.  It seems like
this would be a bit awkward to deal with if your dependencies were
specified purely in terms of output packages.

p.
Esben Haabendal - March 18, 2011, 12:14 p.m.
Phil Blundell <philb@gnu.org> writes:

> On Fri, 2011-03-18 at 06:29 +0100, Esben Haabendal wrote:
>> Yes, I get your point.  You could of-course still specify build
>> dependencies using recipe names/provides, and then just unpack all
>> target packages build by that recipe.
>> 
>> This would give you the major part of the KISS win I see, except for in
>> the dependency handling code of BitBake (and in the learning curve of OE
>> developers).  Keep in mind that by using packages for build-time
>> dependencies, the mechanism is exactly identical for run-time and
>> build-time dependencies.  Exactly the same code in BitBake handles both,
>> and the same mechanism needs to be understood.  This is also part of the
>> KISS win of doing it this way.
>
> Right.  So, it seems to me that the appropriate way to attack the
> problem in OE is to start by doing exactly that: leave the syntax of
> DEPENDS alone for now, treat each entry in DEPENDS as meaning "all
> packages built by the named recipe", and then implement the per-recipe
> sysroot construction that I think we are agreed is highly desirable.

By doing it this way, I don't think we are getting much closer to the
second step (package-based build-time dependencies).  The OE-lite
implementation of per-recipe staging builds on top of package-based
build-time dependencies, so to do it the other way around, would imply
taking the sstate concept and extend it to do per-recipe sysroots.
After that, I believe you have exactly the same amount of development
left to get where OE-lite is in respect to package-based build-time
depencies and per-recipe staging.

As I see it, package-based build-time dependencies practically obsoletes
sstate.  So if we for a moment imagine that we have moved OE to have
sstate based per-recipe staging, we still have the same amount of work
to do in the following areas:

* Modifying BitBake and OE metadata to handle package-based build-time
  dependencies
* Modifying existing recipes to include package-based build-time
  dependency information

You can see the per-recipe staging as a postive side-effect of the
package-based build-time dependency implementation.

So why take the detour instead of just doing it?

> Then, as a later refinement, we could introduce a mechanism for
> specifying dependencies more precisely, at the package level rather than
> the recipe level.  We could do that either by extending the syntax of
> DEPENDS, something like:
>
> DEPENDS = "boost:boost-iostreams"
>
> (to say that you wanted just the boost-iostreams package from the boost
> recipe)

That is IMO just adding more cruft to OE.  Not really what we need.

> or, alternately, by introducing a new variable which would either
> supplement or replace DEPENDS.  Either of those would allow us to
> migrate to fine-grained dependencies without breaking existing recipes.

Sounds better.

Also, just as for run-time dependencies, the build-time package
dependencies is not necesarely the actual package names, but some
provides names, so having a "recipe:package" syntax really does not make
much sense.

>> > (How) do you deal with library package renaming in OE-lite?
>> 
>> What exactly do you mean?  (We are doing several things with library
>> package naming...)
>
> I was thinking of situations like the Debian package autonamer, ie the
> thing that causes glibc-dev to come out named "libc6-dev.ipk" or similar
> depending on the soname of the library that was built.  It seems like
> this would be a bit awkward to deal with if your dependencies were
> specified purely in terms of output packages.

As OE-lite is not OE, we are currently not supporting such type of
package renaming.  Let's leave that to Debian guys to fight with ;-)

But as we build-time dependencies are typically in the form of some
package "provides", it doesn't really matter if the package is renamed.

/Esben
Richard Purdie - March 19, 2011, 12:18 a.m.
On Thu, 2011-03-17 at 20:58 +0100, Esben Haabendal wrote:
> On Thu, 2011-03-17 at 18:05 +0000, Phil Blundell wrote:
> > On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote:
> > > Well, it might be possible to minimize the disruption for a transitional
> > > period by carefully specifying some catch-all build-time package
> > > dependencies pulling in all packages for recipes not ported yet.
> > 
> > Yes, that's the sort of thing I was thinking of.  It isn't even totally
> > obvious to me that specifying individual packages is any better than
> > "all packages from recipe X", since with the former you then have a
> > potential maintenance headache if files get moved from one package to
> > another.
> 
> There is a number of ways that I believe package based build
> dependencies are better than recipe based.
> 
> a) It is possible to depend on parts of a recipe, which fx. is useful
> when a recipe provides more than 1 library, where you might not want all
> of them.
> 
> b) To build a recipe, you depend on some stuff which you don't need to,
> or perhaps even really don't want to pass on to recipes depending on
> this recipe.
> 
> c) KISS. Using the actual target packages for satisfying build-time
> dependencies are a very simple approach, which I strongly believe will
> make OE a better tool in the long run, by reducing complexity, and thus
> lowering the bar for contributing to OE archictural work.

I'd like to step in here and point out that your approach is not as
simple as you claim. It certainly is a really nice solution for one very
specific use case.

There are however many other problems which appear when you step away
from your isolated usecase.

Take the case of different package managers, how is that handled? Can
you only have one enabled at a given time? What happens if you switch
package manager half way through a build. Currently package managers and
package format are abstracted, I suspect we lose that with your
implementation. What if the package managers have slightly different
behaviours?

Secondly, how about task output that isn't packages? The output of
do_deploy for example(kernel/firmware/bootloader)? The pkgdata generated
by do_package? A generic extra task added by the user which outputs some
data?

What you're proposing is to totally actually totally collapse our
current generic task structure into something entirely governed by
target packages and their specific requirements. Of course you could say
the existing mechanisms are still available but then where is the
simplification?

sstate is very simple in concept and very simple in operation compared
to anything we've previously had. It is generic being task based and can
apply to any task as required. It doesn't require radical changes to the
existing model, there is a clear migration path and it also embraces
things like the checksum/signature generation which I believe are going
to play a significant role in the future.

This isn't to say what you're doing is wrong or that we shouldn't be
looking at it for ideas but I believe it needs to be something
incremental on top of what we already have and functionality we have now
cannot simply be lost as people use it.

Cheers,

Richard
Richard Purdie - March 19, 2011, 12:32 a.m.
On Fri, 2011-03-18 at 13:14 +0100, Esben Haabendal wrote:
> By doing it this way, I don't think we are getting much closer to the
> second step (package-based build-time dependencies).  The OE-lite
> implementation of per-recipe staging builds on top of package-based
> build-time dependencies, so to do it the other way around, would imply
> taking the sstate concept and extend it to do per-recipe sysroots.
> After that, I believe you have exactly the same amount of development
> left to get where OE-lite is in respect to package-based build-time
> depencies and per-recipe staging.
> 
> As I see it, package-based build-time dependencies practically obsoletes
> sstate. 

This is where I think you misunderstand why sstate has been written and
what its role is.

Its a generic abstraction of taking the output from a task, saving it
away somewhere and then being able to reuse a prebuilt version of it.
There is no dependency on a package manager, package format or special
dependency formats. sstate simply looks for a prebuilt version (matching
a checksum), uses it if present, otherwise builds from "scratch".

> > I was thinking of situations like the Debian package autonamer, ie the
> > thing that causes glibc-dev to come out named "libc6-dev.ipk" or similar
> > depending on the soname of the library that was built.  It seems like
> > this would be a bit awkward to deal with if your dependencies were
> > specified purely in terms of output packages.
> 
> As OE-lite is not OE, we are currently not supporting such type of
> package renaming.  Let's leave that to Debian guys to fight with ;-)

What other features are you not supporting? Are you prepared to think
about them or is that someone else's problem?

Cheers,

Richard
Esben Haabendal - March 22, 2011, 9 a.m.
Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> On Thu, 2011-03-17 at 20:58 +0100, Esben Haabendal wrote:
>> On Thu, 2011-03-17 at 18:05 +0000, Phil Blundell wrote:
>> > On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote:
>> > > Well, it might be possible to minimize the disruption for a transitional
>> > > period by carefully specifying some catch-all build-time package
>> > > dependencies pulling in all packages for recipes not ported yet.
>> > 
>> > Yes, that's the sort of thing I was thinking of.  It isn't even totally
>> > obvious to me that specifying individual packages is any better than
>> > "all packages from recipe X", since with the former you then have a
>> > potential maintenance headache if files get moved from one package to
>> > another.
>> 
>> There is a number of ways that I believe package based build
>> dependencies are better than recipe based.
>> 
>> a) It is possible to depend on parts of a recipe, which fx. is useful
>> when a recipe provides more than 1 library, where you might not want all
>> of them.
>> 
>> b) To build a recipe, you depend on some stuff which you don't need to,
>> or perhaps even really don't want to pass on to recipes depending on
>> this recipe.
>> 
>> c) KISS. Using the actual target packages for satisfying build-time
>> dependencies are a very simple approach, which I strongly believe will
>> make OE a better tool in the long run, by reducing complexity, and thus
>> lowering the bar for contributing to OE archictural work.
>
> I'd like to step in here and point out that your approach is not as
> simple as you claim. It certainly is a really nice solution for one very
> specific use case.
>
> There are however many other problems which appear when you step away
> from your isolated usecase.
>
> Take the case of different package managers, how is that handled?
> Can you only have one enabled at a given time? What happens if you
> switch package manager half way through a build. Currently package
> managers and package format are abstracted, I suspect we lose that
> with your implementation. What if the package managers have slightly
> different behaviours?

In OE-lite, which is designed for a much more specific use-case than OE,
it is handled very simple.  OE-lite (currently) does not support package
managers at all.  The package format used is simple flat tarballs.  The
way I see package managers handled would be to add them on top of that.
With that I mean that the meta-data contains the necessary packaging
information, this is stored in some standardized meta-data container
stored together with the tarballs (but propbably in separate files), and
packages (RPM/dpkg/ipkg) generation would be implemented on top of that.

I don't believe it is worth the added complexity to support package
managers internally in OE-lite.

But if OE were to implement something similar, then I don't see any real
showstoppers to extending the do_stage/do_rstage functions to handle
multiple package managers.  But I believe it would be best to let
BitBake handle all the dependencie stuff as OE-lite of-course does.

> Secondly, how about task output that isn't packages?

As of now, there is no other type of output.  But I don't see any real
blockers for having non-package output.  That said, in OE-lite the goal
is to only have packages as task output.

> The output of do_deploy for example(kernel/firmware/bootloader)?

The output from do_deploy is never used by other tasks.  Instead, the
recipes that uses do_deploy can just arrange for a package to generated
with the stuff that needs by other tasks (like the things you mentioned
above).

> The pkgdata generated by do_package?

As OE-lite does not generate packages for real package managers (just
tarballs), this is currently not handled.  But I would just store it in
some way together with the tarball (fx. *.tar and *.pkgdata).

> A generic extra task added by the user which outputs some data?

Like what?

> What you're proposing is to totally actually totally collapse our
> current generic task structure into something entirely governed by
> target packages and their specific requirements.

I partly agree.  The proposal is to let target packages have a much more
central role than in current OE.  But the task structure in OE-lite is
actually very much similar to in OE, so I don't think really agree with
your wording (totally collapse).

> Of course you could say the existing mechanisms are still available

And they are.

> but then where is the simplification?

The simplification I have been trying to explain here is the fact that I
don't have to handle two different ways if dependency-handling for
build- and run-time dependencies.  They are completely identical.  Also,
I don't have to care about crafting any kind of special sysroot/sstate
packages, just target packages (tarballs).

> sstate is very simple in concept and very simple in operation compared
> to anything we've previously had.

Could be.  But my main point is that it still builds on what I believe
is a fundamentally flawed concept, ie. the idea of sharing anything
between recipes other than clearly defined entities (ie. packages).

> It is generic being task based and can apply to any task as
> required. It doesn't require radical changes to the existing model,
> there is a clear migration path

I agree on that.

> and it also embraces things like the checksum/signature generation
> which I believe are going to play a significant role in the future.

Yes, that will definitely be important in the future.  But I don't see
how checksum/signature generation is made much easier by the design
concepts being discussed here.

> This isn't to say what you're doing is wrong or that we shouldn't be
> looking at it for ideas but I believe it needs to be something
> incremental on top of what we already have and functionality we have now
> cannot simply be lost as people use it.

And that is probably the core of this discussion.

I am happy to see that there seems to be some kind of concensus that
what I am doing/showing in OE-lite has some clear merits.

So the important choice is if the absolute driving force for development
has to be that everything has to be "incremental on top of what we
already have", or it is sometimes better to replace something.

I basically does not see as much value in "incremental on top of ..." as
you do, but understand and appreciate the need for a transitional phase
for meta-data.  But these are really two different issues.  If something
(like this proposal for "real" package-based staging/sysroot) is
chosen for the OE future, a plan for transition needs to be made and
executed.  But that is not the same as to say that we cannot at the same
time replace some core parts/concepts.

/Esben
Esben Haabendal - March 22, 2011, 9:01 a.m.
Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> On Fri, 2011-03-18 at 13:14 +0100, Esben Haabendal wrote:
>> By doing it this way, I don't think we are getting much closer to the
>> second step (package-based build-time dependencies).  The OE-lite
>> implementation of per-recipe staging builds on top of package-based
>> build-time dependencies, so to do it the other way around, would imply
>> taking the sstate concept and extend it to do per-recipe sysroots.
>> After that, I believe you have exactly the same amount of development
>> left to get where OE-lite is in respect to package-based build-time
>> depencies and per-recipe staging.
>> 
>> As I see it, package-based build-time dependencies practically obsoletes
>> sstate. 
>
> This is where I think you misunderstand why sstate has been written and
> what its role is.

Please note my use of "practically", which should have been quoted or
something...

> Its a generic abstraction of taking the output from a task, saving it
> away somewhere and then being able to reuse a prebuilt version of it.

I know.  And this is clearly different to what package-based build-time
dependencies give.  Without sstate, you will not be able to reuse
prebuilt versions of any random task.  You can only reuse prebuilt
versions of packages (ie. do_package tasks).  In practice, I believe
that is sufficient for most needs.  But I don't see any showstoppers for
using something sstate together with package-based build-time
dependencies.  I am just saying that what I believes is the most
important value of sstate, more robus builds, is already provided the
package-based build-time dependencies.

> There is no dependency on a package manager, package format or special
> dependency formats. sstate simply looks for a prebuilt version (matching
> a checksum), uses it if present, otherwise builds from "scratch".
>
>> > I was thinking of situations like the Debian package autonamer, ie the
>> > thing that causes glibc-dev to come out named "libc6-dev.ipk" or similar
>> > depending on the soname of the library that was built.  It seems like
>> > this would be a bit awkward to deal with if your dependencies were
>> > specified purely in terms of output packages.
>> 
>> As OE-lite is not OE, we are currently not supporting such type of
>> package renaming.  Let's leave that to Debian guys to fight with ;-)
>
> What other features are you not supporting? Are you prepared to think
> about them or is that someone else's problem?

I am not sure what direction you are trying to push the discussion now.

I am not prepared to single-handedly (without a large $$$ sponsor) take
over complete responsibility for all of OE.

I am though prepared to think about them, and also prepared to work with
the OE community to find good solution to everybody else's problems.

I don't think it is totally fair to imply that I am being completely
ignorant to other people's problems.  I know we are ignoring a lot of OE
features in OE-lite, but that is the reason it is done as OE-lite and
not in OE.  I did not have time and manpower to implement what I needed
in the timeframes I had without ignoring those features.

I am though now approach you (OE community) trying to find out if we can
find a way to get some of the work done in OE-lite back to OE, just as
you have done several times with features developed in Poky going back
to OE.

Talking about that, with Poky being much more directly coupled with OE,
where will new features be developed in the future?  This discussion
seems to indicate that it is not going to be in OE as such.

/Esben
Frans Meulenbroeks - March 23, 2011, 8:31 p.m.
Dear all,

pb and I had a discussion about this on irc earlier today.

Actually we feel there are three different issues in the proposal:
1. only stage those recipes that are in DEPENDS.
2. stage based on packages instead of recipes (so e.g. you can stage
only a lib package)
3. use tar (or a different tool) to stage things; or in other words,
decouple the storage format and the package format.

I think each of these has its merits. I also understand that
incremental introduction is desired above a big bang approach.

Let us look at the issues in detail:

1. only stage those recipes that are in DEPENDS.

This seems pretty useful as it gives consistent builds not depending
on whatever you have build before (and so making things deterministic)
This is especially useful when doing incremental builds. It can also
solve races.
Introducing it would mean changes in staging; incremental introduction
is possible if e.g. we add a flag PACKAGE_BASED_STAGING or so to the
recipes we want this to have (or add a special class or so).
Once all recipes are moved, the flag can be removed again.

Question is how efficient this can be. It mostly depends on how we
want to achieve the staging.
I'm definitely no expert, but we could e.g.
- create the staging area from earlier package images each time we
build it. Does not seem overly efficient
- stage each package in its own dir and create symlinks in the
required staging area.
- stage each pacakge in its own dir and add to -I and -L to only stage
the packages that are needed.

The staging area for a package to be build should probably be within
the work dir of the recipe otherwise we'll loose the capability to
build things in parallel

2. stage based on packages instead of recipes (so e.g. you can stage
only a lib package)

This gives more fine grained control. Not sure how much it brings and
what it will cost.
It might be possible to do it incrementally by e.g. having a keyword
PACKAGE_DEPENDS.
Question is if it is worth the effort

3. use tar (or a different tool) to stage things; or in other words,
decouple the storage format and the package format.

To me that is a completely different question only loosely related to
the above.
If staging is from packaged images, it would help though to have a
format that is fast to unpack.

Another, perhaps more interesting reason, would be to use a package
manager agnostic format and use that to build packages from (for the
package manager you want to have it).
I don't oversee the impact of this one though.


Your opinion on this proposal, improvement suggestions, discussion on
the best way forward etc is appreciated.

Frans

Patch

diff --git a/recipes/net-snmp/net-snmp_svn.bb b/recipes/net-snmp/net-snmp_svn.bb
index 2a75722..c53c91f 100644
--- a/recipes/net-snmp/net-snmp_svn.bb
+++ b/recipes/net-snmp/net-snmp_svn.bb
@@ -1,7 +1,7 @@ 
 require net-snmp.inc
 DEPENDS += "libtool libtool-native libtool-cross"
 SRCREV = "20069"
-PR = "${INC_PR}.0"
+PR = "${INC_PR}.1"
 
 S = "${WORKDIR}/net-snmp"
 
@@ -10,7 +10,7 @@  SRC_URI = "svn://net-snmp.svn.sourceforge.net/svnroot/net-snmp/trunk;module=net-
         file://snmpd.conf \
         file://snmptrapd.conf"
 
-EXTRA_OECONF += "--disable-embedded-perl --with-perl-modules=no GREP=/bin/grep SED=/bin/sed"
+EXTRA_OECONF += "--disable-embedded-perl --with-perl-modules=no --without-nl GREP=/bin/grep SED=/bin/sed"
 EXTRA_OEMAKE = "INSTALL_PREFIX=${D}"
 
 CCACHE = ""