Patchwork [bitbake-devel] world-image: add extra target

login
register
mail settings
Submitter Martin Jansa
Date Jan. 25, 2013, 7:38 p.m.
Message ID <1359142708-24667-1-git-send-email-Martin.Jansa@gmail.com>
Download mbox | patch
Permalink /patch/43405/
State New
Headers show

Comments

Martin Jansa - Jan. 25, 2013, 7:38 p.m.
* similar to world target, but collects all PACKAGES
* recipes marked with EXCLUDE_FROM_WORLD are excluded from world as well
  as world-image
* recipes marked with EXCLUDE_FROM_WORLD_IMAGE are excluded completely
  from world-image, sometimes you can build 2 recipes in same sysroot,
  but cannot install them in the same image
* PACKAGES listed in EXCLUDE_PACKAGES_FROM_WORLD_IMAGE are also
  excluded. You can remove e.g. ${PN}-dbg, but keep all other PACKAGES.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 lib/bb/cache.py  | 17 ++++++++++++++++-
 lib/bb/cooker.py | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 1 deletion(-)
Chris Larson - Jan. 25, 2013, 9:45 p.m.
On Fri, Jan 25, 2013 at 12:38 PM, Martin Jansa <martin.jansa@gmail.com>wrote:

> * similar to world target, but collects all PACKAGES
> * recipes marked with EXCLUDE_FROM_WORLD are excluded from world as well
>   as world-image
> * recipes marked with EXCLUDE_FROM_WORLD_IMAGE are excluded completely
>   from world-image, sometimes you can build 2 recipes in same sysroot,
>   but cannot install them in the same image
> * PACKAGES listed in EXCLUDE_PACKAGES_FROM_WORLD_IMAGE are also
>   excluded. You can remove e.g. ${PN}-dbg, but keep all other PACKAGES.
>
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>

bitbake should have zero knowledge of what an image even is, much less
hardcoding awareness of the IMAGE_INSTALL variable. Instead, we should
provide a means to hook into bitbake's knowledge of the recipes included in
the world image, and let a world-image recipe utilize that.
Martin Jansa - Jan. 25, 2013, 10:13 p.m.
Agreed, I've tried with handler added in world-image.bbclass (cumulating
all PACKAGES on RecipeParsed event) and failed, that's why this ended in
bitbake code, but I'm open to ideas how to hook it properly.


On Fri, Jan 25, 2013 at 10:45 PM, Chris Larson <clarson@kergoth.com> wrote:

>
> On Fri, Jan 25, 2013 at 12:38 PM, Martin Jansa <martin.jansa@gmail.com>wrote:
>
>> * similar to world target, but collects all PACKAGES
>> * recipes marked with EXCLUDE_FROM_WORLD are excluded from world as well
>>   as world-image
>> * recipes marked with EXCLUDE_FROM_WORLD_IMAGE are excluded completely
>>   from world-image, sometimes you can build 2 recipes in same sysroot,
>>   but cannot install them in the same image
>> * PACKAGES listed in EXCLUDE_PACKAGES_FROM_WORLD_IMAGE are also
>>   excluded. You can remove e.g. ${PN}-dbg, but keep all other PACKAGES.
>>
>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>>
>
> bitbake should have zero knowledge of what an image even is, much less
> hardcoding awareness of the IMAGE_INSTALL variable. Instead, we should
> provide a means to hook into bitbake's knowledge of the recipes included in
> the world image, and let a world-image recipe utilize that.
> --
> Christopher Larson
>
Chris Larson - Jan. 25, 2013, 10:17 p.m.
On Fri, Jan 25, 2013 at 3:13 PM, Martin Jansa <martin.jansa@gmail.com>wrote:

> Agreed, I've tried with handler added in world-image.bbclass (cumulating
> all PACKAGES on RecipeParsed event) and failed, that's why this ended in
> bitbake code, but I'm open to ideas how to hook it properly.


RecipeParsed isn't ideal, better to let bitbake collate the world recipe
list, pass that along via a config level event (not ConfigParsed, but after
the cooker and taskdata preparation), and map that to packages with pkgdata
from the RecipeParsed.

Patch

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index fd5fbb3..47d3f2d 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -43,7 +43,7 @@  except ImportError:
     logger.info("Importing cPickle failed. "
                 "Falling back to a very slow implementation.")
 
-__cache_version__ = "145"
+__cache_version__ = "146"
 
 def getCacheFile(path, filename, data_hash):
     return os.path.join(path, filename + "." + data_hash)
@@ -128,6 +128,8 @@  class CoreRecipeInfo(RecipeInfoCommon):
         self.defaultpref = self.intvar('DEFAULT_PREFERENCE', metadata)
         self.broken = self.getvar('BROKEN', metadata)
         self.not_world = self.getvar('EXCLUDE_FROM_WORLD', metadata)
+        self.not_world_image = self.getvar('EXCLUDE_FROM_WORLD_IMAGE', metadata)
+        self.not_world_image_packages = self.getvar('EXCLUDE_PACKAGES_FROM_WORLD_IMAGE', metadata).split(' ')
         self.stamp = self.getvar('STAMP', metadata)
         self.stampclean = self.getvar('STAMPCLEAN', metadata)        
         self.stamp_base = self.flaglist('stamp-base', self.tasks, metadata)
@@ -176,6 +178,8 @@  class CoreRecipeInfo(RecipeInfoCommon):
         cachedata.rundeps = defaultdict(lambda: defaultdict(list))
         cachedata.runrecs = defaultdict(lambda: defaultdict(list))
         cachedata.possible_world = []
+        cachedata.possible_world_image = []
+        cachedata.possible_world_image_packages = []
         cachedata.universe_target = []
         cachedata.hashfn = {}
 
@@ -235,6 +239,16 @@  class CoreRecipeInfo(RecipeInfoCommon):
         # calculations
         if not self.broken and not self.not_world:
             cachedata.possible_world.append(fn)
+            # Collect files we may need for possible world-image
+            # calculations anc packages which are not explicitly excluded
+            if not self.not_world_image:
+                cachedata.possible_world_image.append(fn)
+                if self.not_world_image_packages:
+                    for package in self.packages:
+                        if not package in self.not_world_image_packages:
+                            cachedata.possible_world_image_packages.append(package)
+                else:
+                    cachedata.possible_world_image_packages.extend(self.packages)
 
         # create a collection of all targets for sanity checking
         # tasks, such as upstream versions, license, and tools for
@@ -710,6 +724,7 @@  class CacheData(object):
         # Indirect Cache variables (set elsewhere)
         self.ignored_dependencies = []
         self.world_target = set()
+        self.world_image_target = set()
         self.bbfile_priority = {}
 
     def add_from_recipeinfo(self, fn, info_array):
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index caf1123..b733066 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -227,6 +227,8 @@  class BBCooker:
         if self.configuration.show_environment:
             if 'world' in self.configuration.pkgs_to_build:
                 self.commandlineAction['msg'] = "'world' is not a valid target for --environment."
+            if 'world-image' in self.configuration.pkgs_to_build:
+                self.commandlineAction['msg'] = "'world-image' is not a valid target for --environment."
             elif 'universe' in self.configuration.pkgs_to_build:
                 self.commandlineAction['msg'] = "'universe' is not a valid target for --environment."
             elif len(self.configuration.pkgs_to_build) > 1:
@@ -817,6 +819,32 @@  class BBCooker:
             if terminal:
                 self.status.world_target.add(pn)
 
+    def buildWorldImageTargetList(self):
+        """
+         Build package list for "bitbake world-image"
+        """
+        all_depends = self.status.all_depends
+        pn_provides = self.status.pn_provides
+        parselog.debug(1, "collating packages for \"world-image\"")
+        for f in self.status.possible_world_image:
+            terminal = True
+            pn = self.status.pkg_fn[f]
+
+            for p in pn_provides[pn]:
+                if p.startswith('virtual/'):
+                    parselog.debug(2, "World image build skipping %s due to %s provider starting with virtual/", f, p)
+                    terminal = False
+                    break
+                for pf in self.status.providers[p]:
+                    if self.status.pkg_fn[pf] != pn:
+                        parselog.debug(2, "World image build skipping %s due to both us and %s providing %s", f, pf, p)
+                        terminal = False
+                        break
+            if terminal:
+                self.status.world_image_target.add(pn)
+        parselog.debug(2, "collated packages for \"world-image\": '%s'" % self.status.possible_world_image_packages)
+        return self.status.possible_world_image_packages
+
     def interactiveMode( self ):
         """Drop off into a shell"""
         try:
@@ -1253,6 +1281,12 @@  class BBCooker:
             for t in self.status.world_target:
                 pkgs_to_build.append(t)
 
+        if 'world-image' in pkgs_to_build:
+            packages = ' '.join(self.buildWorldImageTargetList())
+            self.configuration.data.setVar('IMAGE_INSTALL_pn-world-image', packages)
+            for t in self.status.world_image_target:
+                pkgs_to_build.append(t)
+
         if 'universe' in pkgs_to_build:
             parselog.warn("The \"universe\" target is only intended for testing and may produce errors.")
             parselog.debug(1, "collating packages for \"universe\"")