Patchwork Using external source trees with OE-Core

login
register
mail settings
Submitter Richard Purdie
Date Feb. 24, 2012, 12:55 p.m.
Message ID <1330088135.32006.14.camel@ted>
Download mbox | patch
Permalink /patch/21919/
State New
Headers show

Comments

Richard Purdie - Feb. 24, 2012, 12:55 p.m.
Someone recently asked me about using external source trees with
OE-Core. I was aware of srctree.bbclass in OE-Classic and did start
looking at it but it has various elements I wasn't so keen on. I
therefore wondered if I could improve upon it. I did start from that
code base but came up with:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
[patch inline below]

which I've called externalsrc.bbclass since it behaves a bit
Andreas Oberritter - Feb. 24, 2012, 1:49 p.m.
On 24.02.2012 13:55, Richard Purdie wrote:
> Someone recently asked me about using external source trees with
> OE-Core. I was aware of srctree.bbclass in OE-Classic and did start
> looking at it but it has various elements I wasn't so keen on. I
> therefore wondered if I could improve upon it. I did start from that
> code base but came up with:
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
> [patch inline below]
> 
> which I've called externalsrc.bbclass since it behaves a bit
> differently. In particular, all the usual task targets are available.
> 
> As an example usage with the above patch applied:
> 
> cd ~
> tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
> (creates libfm-0.1.17 in my homedir)
> 
> edit libfm_0.1.17.bb and add:
> 
> inherit externalsrc
> S = "/home/richard/libfm-0.1.17"

Do I need to edit libfm_0.1.17.bb in-place, or can it be stored outside
my layers to be built with bitbake -b, for example?

Does S need to contain an absolute path, or could it be set to a path
relative to the location of the recipe?

It would be nice if it was possible to include bitbake recipes in the
source trees of my projects, which "just work".

I think this was possible with srctree.bbclass, though I never had it
working reliably. I think bitbake -cclean always raised errors.

Regards,
Andreas
Bruce Ashfield - Feb. 24, 2012, 2:01 p.m.
On Fri, Feb 24, 2012 at 7:55 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Someone recently asked me about using external source trees with
> OE-Core. I was aware of srctree.bbclass in OE-Classic and did start
> looking at it but it has various elements I wasn't so keen on. I
> therefore wondered if I could improve upon it. I did start from that
> code base but came up with:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
> [patch inline below]
>
> which I've called externalsrc.bbclass since it behaves a bit
> differently. In particular, all the usual task targets are available.
>
> As an example usage with the above patch applied:
>
> cd ~
> tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
> (creates libfm-0.1.17 in my homedir)
>
> edit libfm_0.1.17.bb and add:
>
> inherit externalsrc
> S = "/home/richard/libfm-0.1.17"
>
> bitbake libfm
>
> and it does what you'd expect. You can also -c clean and it will wipe
> out WORKDIR but it won't touch ${S}. I picked libfm effectively at
> random. It won't remove the autoreconf changes from configure but I'm
> not sure I care much about that.
>
> Admittedly, I did have to fix one Makefile.am which did '-I../' instead
> of "-I$(srcdir)/../" but that is a bug in libfm.
>
> Obviously this won't work quite as well with some recipes like
> linux-yocto which manipulate ${S} a lot more but it should work in most
> cases. I also found it very hard to remove tasks from the anonymous
> python, we probably need to improve the API for this.

I was just looking at this myself, and I agree that when using this class,
you'd want to drop a few phases that linux-yocto does and just let things
run from compile onwards. You aren't using much infrastructure that is
provided in this scenario .. but that is the point :)

I like the option that this provides for a development workflow, with some
safety and with associated risks. But overall, if used properly .. I see the
value.

Cheers,

Bruce

>
> Opinions on including this class?
>
> Cheers,
>
> Richard
>
>
>
> From ac8a54c18c33aaab6758c34a3d6093d3d3384cee Mon Sep 17 00:00:00 2001
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date: Fri, 24 Feb 2012 12:29:36 +0000
> Subject: externalsrc.bbclass: Add class for handling external source trees
>
> This is loosly based upon srctree.bbclass from OE-Classic but with some
> changes appropriate to OE-Core.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
> new file mode 100644
> index 0000000..7e00ef8
> --- a/dev/null
> +++ b/meta/classes/externalsrc.bbclass
> @@ -0,0 +1,53 @@
> +# Copyright (C) 2012 Linux Foundation
> +# Author: Richard Purdie
> +# Some code and influence taken from srctree.bbclass:
> +# Copyright (C) 2009 Chris Larson <clarson@kergoth.com>
> +# Released under the MIT license (see COPYING.MIT for the terms)
> +#
> +# externalsrc.bbclass enables use of an existing source tree, usually external to
> +# the build system to build a piece of software rather than the usual fetch/unpack/patch
> +# process.
> +#
> +# To use, set S to point at the directory you want to use containing the sources
> +# e.g. S = "/path/to/my/source/tree"
> +#
> +# If the class is to work for both target and native versions (or with multilibs/
> +# cross or other BBCLASSEXTEND variants), its expected that setting B to point to
> +# where to place the compiled binaries will work (split source and build directories).
> +# This is the default but B can be set to S if circumstaces dictate.
> +#
> +
> +SRC_URI = ""
> +SRCTREECOVEREDTASKS ?= "do_patch do_unpack do_fetch"
> +B = "${WORKDIR}/${BPN}-${PV}/"
> +
> +def remove_tasks(tasks, deltasks, d):
> +    for task in tasks:
> +        deps = d.getVarFlag(task, "deps")
> +        for preptask in deltasks:
> +            if preptask in deps:
> +                deps.remove(preptask)
> +        d.setVarFlag(task, "deps", deps)
> +    # Poking around bitbake internal variables is evil but there appears to be no better way :(
> +    tasklist = d.getVar('__BBTASKS') or []
> +    for task in deltasks:
> +        d.delVarFlag(task, "task")
> +        if task in tasklist:
> +            tasklist.remove(task)
> +    d.setVar('__BBTASKS', tasklist)
> +
> +python () {
> +    tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys())
> +    covered = d.getVar("SRCTREECOVEREDTASKS", True).split()
> +
> +    for task in tasks:
> +        if task.endswith("_setscene"):
> +            # sstate is never going to work for external source trees, disable it
> +            covered.append(task)
> +        else:
> +            # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time
> +            d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock")
> +
> +    remove_tasks(tasks, covered, d)
> +}
> +
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Mark Hatle - Feb. 24, 2012, 2:41 p.m.
On 2/24/12 7:49 AM, Andreas Oberritter wrote:
> On 24.02.2012 13:55, Richard Purdie wrote:
>> Someone recently asked me about using external source trees with
>> OE-Core. I was aware of srctree.bbclass in OE-Classic and did start
>> looking at it but it has various elements I wasn't so keen on. I
>> therefore wondered if I could improve upon it. I did start from that
>> code base but came up with:
>>
>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
>> [patch inline below]
>>
>> which I've called externalsrc.bbclass since it behaves a bit
>> differently. In particular, all the usual task targets are available.
>>
>> As an example usage with the above patch applied:
>>
>> cd ~
>> tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
>> (creates libfm-0.1.17 in my homedir)
>>
>> edit libfm_0.1.17.bb and add:
>>
>> inherit externalsrc
>> S = "/home/richard/libfm-0.1.17"
>
> Do I need to edit libfm_0.1.17.bb in-place, or can it be stored outside
> my layers to be built with bitbake -b, for example?

You should be able to add a bbappend for this item, in a custom development 
layer and not modify the original .bb.  It will still require the two line 
configuration change (and a custom layer configuration... but I recommend that 
anyway.)

> Does S need to contain an absolute path, or could it be set to a path
> relative to the location of the recipe?

I don't know for this development.... but relative can get very messy, 
especially when all of the project, work dir, etc can be overridden to create a 
custom environment for someone.  Enforcing an absolute path isn't a bad idea here.

> It would be nice if it was possible to include bitbake recipes in the
> source trees of my projects, which "just work".

Might be possible in something like base.bbclass to add the inherit if a value 
is set.. then in local.conf you could do something like:

EXTERNALSRC_libfm = "/home/richard/libfm-0.1.17"

And if it's set, the class could do the reset?  That would certainly be easier 
then a layer.

--Mark

> I think this was possible with srctree.bbclass, though I never had it
> working reliably. I think bitbake -cclean always raised errors.
>
> Regards,
> Andreas
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
McClintock Matthew-B29882 - Feb. 24, 2012, 4:55 p.m.
Richard,

We do something very similar to this currently. I've pushed what we
are using here:

https://github.com/mattsm/meta-fsl-local-sdk

It would be nice to consolidate all the required bits upstream... =)

-M

On Fri, Feb 24, 2012 at 6:55 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Someone recently asked me about using external source trees with
> OE-Core. I was aware of srctree.bbclass in OE-Classic and did start
> looking at it but it has various elements I wasn't so keen on. I
> therefore wondered if I could improve upon it. I did start from that
> code base but came up with:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
> [patch inline below]
>
> which I've called externalsrc.bbclass since it behaves a bit
> differently. In particular, all the usual task targets are available.
>
> As an example usage with the above patch applied:
>
> cd ~
> tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
> (creates libfm-0.1.17 in my homedir)
>
> edit libfm_0.1.17.bb and add:
>
> inherit externalsrc
> S = "/home/richard/libfm-0.1.17"
>
> bitbake libfm
>
> and it does what you'd expect. You can also -c clean and it will wipe
> out WORKDIR but it won't touch ${S}. I picked libfm effectively at
> random. It won't remove the autoreconf changes from configure but I'm
> not sure I care much about that.
>
> Admittedly, I did have to fix one Makefile.am which did '-I../' instead
> of "-I$(srcdir)/../" but that is a bug in libfm.
>
> Obviously this won't work quite as well with some recipes like
> linux-yocto which manipulate ${S} a lot more but it should work in most
> cases. I also found it very hard to remove tasks from the anonymous
> python, we probably need to improve the API for this.
>
> Opinions on including this class?
>
> Cheers,
>
> Richard
>
>
>
> From ac8a54c18c33aaab6758c34a3d6093d3d3384cee Mon Sep 17 00:00:00 2001
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date: Fri, 24 Feb 2012 12:29:36 +0000
> Subject: externalsrc.bbclass: Add class for handling external source trees
>
> This is loosly based upon srctree.bbclass from OE-Classic but with some
> changes appropriate to OE-Core.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
> new file mode 100644
> index 0000000..7e00ef8
> --- a/dev/null
> +++ b/meta/classes/externalsrc.bbclass
> @@ -0,0 +1,53 @@
> +# Copyright (C) 2012 Linux Foundation
> +# Author: Richard Purdie
> +# Some code and influence taken from srctree.bbclass:
> +# Copyright (C) 2009 Chris Larson <clarson@kergoth.com>
> +# Released under the MIT license (see COPYING.MIT for the terms)
> +#
> +# externalsrc.bbclass enables use of an existing source tree, usually external to
> +# the build system to build a piece of software rather than the usual fetch/unpack/patch
> +# process.
> +#
> +# To use, set S to point at the directory you want to use containing the sources
> +# e.g. S = "/path/to/my/source/tree"
> +#
> +# If the class is to work for both target and native versions (or with multilibs/
> +# cross or other BBCLASSEXTEND variants), its expected that setting B to point to
> +# where to place the compiled binaries will work (split source and build directories).
> +# This is the default but B can be set to S if circumstaces dictate.
> +#
> +
> +SRC_URI = ""
> +SRCTREECOVEREDTASKS ?= "do_patch do_unpack do_fetch"
> +B = "${WORKDIR}/${BPN}-${PV}/"
> +
> +def remove_tasks(tasks, deltasks, d):
> +    for task in tasks:
> +        deps = d.getVarFlag(task, "deps")
> +        for preptask in deltasks:
> +            if preptask in deps:
> +                deps.remove(preptask)
> +        d.setVarFlag(task, "deps", deps)
> +    # Poking around bitbake internal variables is evil but there appears to be no better way :(
> +    tasklist = d.getVar('__BBTASKS') or []
> +    for task in deltasks:
> +        d.delVarFlag(task, "task")
> +        if task in tasklist:
> +            tasklist.remove(task)
> +    d.setVar('__BBTASKS', tasklist)
> +
> +python () {
> +    tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys())
> +    covered = d.getVar("SRCTREECOVEREDTASKS", True).split()
> +
> +    for task in tasks:
> +        if task.endswith("_setscene"):
> +            # sstate is never going to work for external source trees, disable it
> +            covered.append(task)
> +        else:
> +            # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time
> +            d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock")
> +
> +    remove_tasks(tasks, covered, d)
> +}
> +
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Chris Larson - Feb. 24, 2012, 4:59 p.m.
On Fri, Feb 24, 2012 at 5:55 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> You can also -c clean and it will wipe
> out WORKDIR but it won't touch ${S}. I picked libfm effectively at
> random. It won't remove the autoreconf changes from configure but I'm
> not sure I care much about that.

This is one of the things that srctree handled a bit differently. I
think we should think about pushing the clean improvement bits into a
core class, to make it easy to select slightly different clean
behavior. Either that, or add a hook for it. As it is today, it'd be
slightly painful to make the recipe both wipe workdir and make clean
your source tree.
Richard Purdie - Feb. 24, 2012, 5:36 p.m.
On Fri, 2012-02-24 at 16:55 +0000, McClintock Matthew-B29882 wrote:
> We do something very similar to this currently. I've pushed what we
> are using here:
> 
> https://github.com/mattsm/meta-fsl-local-sdk
> 
> It would be nice to consolidate all the required bits upstream... =)

I would warn you that there is code in there that is clearly
inappropriate for OE-Core.

It would also explain that bug report you sent me which referenced
"do_populate_sysroot_post" and I asked what local modifications you had
and you said "none" ;-).

Does the patch in this thread take care of your needs regarding the
srctree and clean classes?

For cleanobjs, I can see advantages in merging something like that into
the core...

Cheers,

Richard
McClintock Matthew-B29882 - Feb. 24, 2012, 5:44 p.m.
On Fri, Feb 24, 2012 at 11:36 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2012-02-24 at 16:55 +0000, McClintock Matthew-B29882 wrote:
>> We do something very similar to this currently. I've pushed what we
>> are using here:
>>
>> https://github.com/mattsm/meta-fsl-local-sdk
>>
>> It would be nice to consolidate all the required bits upstream... =)
>
> I would warn you that there is code in there that is clearly
> inappropriate for OE-Core.

No surprise at all...

> It would also explain that bug report you sent me which referenced
> "do_populate_sysroot_post" and I asked what local modifications you had
> and you said "none" ;-).

I don't use this... we use this for internal developers... so the
error was not using this...

> Does the patch in this thread take care of your needs regarding the
> srctree and clean classes?

Not sure, hoping Zhenhua can take a look.

> For cleanobjs, I can see advantages in merging something like that into
> the core...

....

-M
Rich Pixley - Feb. 24, 2012, 6:25 p.m.
On 2/24/12 04:55 , Richard Purdie wrote:
> Someone recently asked me about using external source trees with
> OE-Core. [...]
> Opinions on including this class?
I see value.

Our workflow, (based on an ancient branch of oe), uses something 
similar.  When working on a component in context, (as distinct from 
standalone), the developer sets S to point to his own directory and then 
just avoids the tasks with awkward side effects like clean and mrproper.

We had problems initially with people committing bb files with S still 
pointed to the local values.  There needs to be a safety catch for this 
somewhere as this mistake is too easy to make and affects everyone when 
it happens.

Against my better judgment, someone locally implemented a ~/oe.conf 
arrangement that developers now use to override S for the components in 
which they work.  That seems to have largely addressed the problem of 
accidental commits of bb files with bad S values, but has opened a 
completely different set of problems, of course.  I don't think this is 
the right solution to the accidental commit problem, but I think some 
sanity check or "best practice" for overriding S is required before 
including this class.

--rich

Patch

differently. In particular, all the usual task targets are available.

As an example usage with the above patch applied:

cd ~
tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
(creates libfm-0.1.17 in my homedir)

edit libfm_0.1.17.bb and add:

inherit externalsrc
S = "/home/richard/libfm-0.1.17"

bitbake libfm

and it does what you'd expect. You can also -c clean and it will wipe
out WORKDIR but it won't touch ${S}. I picked libfm effectively at
random. It won't remove the autoreconf changes from configure but I'm
not sure I care much about that.

Admittedly, I did have to fix one Makefile.am which did '-I../' instead
of "-I$(srcdir)/../" but that is a bug in libfm.

Obviously this won't work quite as well with some recipes like
linux-yocto which manipulate ${S} a lot more but it should work in most
cases. I also found it very hard to remove tasks from the anonymous
python, we probably need to improve the API for this.

Opinions on including this class?

Cheers,

Richard



From ac8a54c18c33aaab6758c34a3d6093d3d3384cee Mon Sep 17 00:00:00 2001
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Fri, 24 Feb 2012 12:29:36 +0000
Subject: externalsrc.bbclass: Add class for handling external source trees

This is loosly based upon srctree.bbclass from OE-Classic but with some
changes appropriate to OE-Core.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
new file mode 100644
index 0000000..7e00ef8
--- a/dev/null
+++ b/meta/classes/externalsrc.bbclass
@@ -0,0 +1,53 @@ 
+# Copyright (C) 2012 Linux Foundation
+# Author: Richard Purdie
+# Some code and influence taken from srctree.bbclass:
+# Copyright (C) 2009 Chris Larson <clarson@kergoth.com>
+# Released under the MIT license (see COPYING.MIT for the terms)
+#
+# externalsrc.bbclass enables use of an existing source tree, usually external to 
+# the build system to build a piece of software rather than the usual fetch/unpack/patch
+# process.
+#
+# To use, set S to point at the directory you want to use containing the sources
+# e.g. S = "/path/to/my/source/tree"
+#
+# If the class is to work for both target and native versions (or with multilibs/
+# cross or other BBCLASSEXTEND variants), its expected that setting B to point to 
+# where to place the compiled binaries will work (split source and build directories).
+# This is the default but B can be set to S if circumstaces dictate.
+#
+
+SRC_URI = ""
+SRCTREECOVEREDTASKS ?= "do_patch do_unpack do_fetch"
+B = "${WORKDIR}/${BPN}-${PV}/"
+
+def remove_tasks(tasks, deltasks, d):
+    for task in tasks:
+        deps = d.getVarFlag(task, "deps")
+        for preptask in deltasks:
+            if preptask in deps:
+                deps.remove(preptask)
+        d.setVarFlag(task, "deps", deps)
+    # Poking around bitbake internal variables is evil but there appears to be no better way :(
+    tasklist = d.getVar('__BBTASKS') or []
+    for task in deltasks:
+        d.delVarFlag(task, "task")
+        if task in tasklist:
+            tasklist.remove(task)
+    d.setVar('__BBTASKS', tasklist)
+
+python () {
+    tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys())
+    covered = d.getVar("SRCTREECOVEREDTASKS", True).split()
+
+    for task in tasks:
+        if task.endswith("_setscene"):
+            # sstate is never going to work for external source trees, disable it
+            covered.append(task)
+        else:
+            # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time
+            d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock")
+
+    remove_tasks(tasks, covered, d)
+}
+