[bitbake-devel,1/1] toaster: fix SDK artifact capture

Submitted by Reyna, David on April 10, 2017, 5:07 a.m. | Patch ID: 138970

Details

Message ID 07c4805b646e9c2ff9d9f75febdabcf1c4c3a485.1491800468.git.David.Reyna@windriver.com
State New
Headers show

Commit Message

Reyna, David April 10, 2017, 5:07 a.m.
From: David Reyna <David.Reyna@windriver.com>

Use the TaskArtifacts event to scan the SDK and ESDK manifests
to cleanly collect the respective artifact files.

The previous method was broken when the SDK file deployment moved
from the do_populate_sdk[_ext] tasks to their sstate tasks. That
method is disabled (but not yet removed) in preparation for the
rest of refactor work for the parent #10283 work.

[YOCTO #10850]

Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 30 ++++++++++++++++++++++++++++++
 bitbake/lib/bb/ui/toasterui.py       |  4 +---
 meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----
 3 files changed, 47 insertions(+), 8 deletions(-)

Patch hide | download patch | download mbox

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 92d1a1c..46be5a5 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -1663,6 +1663,36 @@  class BuildInfoHelper(object):
                 break
         return endswith
 
+    def scan_task_artifacts(self, event):
+        """
+        The 'TaskArtifacts' event passes the manifest file content for the
+        tasks 'do_deploy', 'do_image_complete', 'do_populate_sdk', and
+        'do_populate_sdk_ext'. The first two will be implemented later.
+        """
+        task_vars = BuildInfoHelper._get_data_from_event(event)
+        task_name = task_vars['task'][task_vars['task'].find(':')+1:]
+        task_artifacts = task_vars['artifacts']
+
+        if task_name in ['do_populate_sdk', 'do_populate_sdk_ext']:
+            targets = [target for target in self.internal_state['targets'] \
+                if target.task == task_name[3:]]
+            if not targets:
+                logger.warning("scan_task_artifacts: SDK targets not found: %s\n", task_name)
+                return
+            for artifact_path in task_artifacts:
+                if not os.path.isfile(artifact_path):
+                    logger.warning("scan_task_artifacts: artifact file not found: %s\n", artifact_path)
+                    continue
+                for target in targets:
+                    # don't record the file if it's already been added
+                    # to this target
+                    matching_files = TargetSDKFile.objects.filter(
+                        target=target, file_name=artifact_path)
+                    if matching_files.count() == 0:
+                        artifact_size = os.stat(artifact_path).st_size
+                        self.orm_wrapper.save_target_sdk_file(
+                            target, artifact_path, artifact_size)
+
     def _get_image_files(self, deploy_dir_image, image_name, image_file_extensions):
         """
         Find files in deploy_dir_image whose basename starts with the
diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 1729902..71f04fa 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -438,9 +438,7 @@  def main(server, eventHandler, params):
                 elif event.type == "SetBRBE":
                     buildinfohelper.brbe = buildinfohelper._get_data_from_event(event)
                 elif event.type == "TaskArtifacts":
-                    # not implemented yet
-                    # see https://bugzilla.yoctoproject.org/show_bug.cgi?id=10283 for details
-                    pass
+                    buildinfohelper.scan_task_artifacts(event)
                 elif event.type == "OSErrorException":
                     logger.error(event)
                 else:
diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
index 4de380b..296e476 100644
--- a/meta/classes/toaster.bbclass
+++ b/meta/classes/toaster.bbclass
@@ -328,8 +328,18 @@  python toaster_artifacts() {
     if e.taskname in ["do_deploy", "do_image_complete", "do_populate_sdk", "do_populate_sdk_ext"]:
         d2 = d.createCopy()
         d2.setVar('FILE', e.taskfile)
-        d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
+        # Use 'stamp-extra-info' if present, else use workaround
+        # to determine 'SSTATE_MANMACH'
+        extrainf = d2.getVarFlag(e.taskname, 'stamp-extra-info')
+        if extrainf:
+            d2.setVar('SSTATE_MANMACH', extrainf)
+        else:
+            if "do_populate_sdk" == e.taskname:
+                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}${SDKMACHINE}"))
+            else:
+                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
         manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:], d2)[0]
+
         if os.access(manifest, os.R_OK):
             with open(manifest) as fmanifest:
                 artifacts = [fname.strip() for fname in fmanifest]
@@ -356,8 +366,9 @@  do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
 
-do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
-do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+#do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
+#do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+
+#do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
+#do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
 
-do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
-do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "

Comments

Richard Purdie April 11, 2017, 11:31 a.m.
On Sun, 2017-04-09 at 22:07 -0700, David Reyna wrote:
> From: David Reyna <David.Reyna@windriver.com>
> 
> Use the TaskArtifacts event to scan the SDK and ESDK manifests
> to cleanly collect the respective artifact files.
> 
> The previous method was broken when the SDK file deployment moved
> from the do_populate_sdk[_ext] tasks to their sstate tasks. That
> method is disabled (but not yet removed) in preparation for the
> rest of refactor work for the parent #10283 work.
> 
> [YOCTO #10850]
> 
> Signed-off-by: David Reyna <David.Reyna@windriver.com>
> ---
>  bitbake/lib/bb/ui/buildinfohelper.py | 30
> ++++++++++++++++++++++++++++++
>  bitbake/lib/bb/ui/toasterui.py       |  4 +---
>  meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----
>  3 files changed, 47 insertions(+), 8 deletions(-)

This isn't really in the right format, the patches against
toaster.bbclass needs to be sent separately to OE-Core since its in a
separate repository.

I've sorted this out and merged it as we're close to release builds and
I appreciate this needed fixing, please just keep it in mind next time,
thanks!

Richard