toaster: fix SDK artifact capture

Submitted by Reyna, David on April 8, 2017, 3:34 a.m. | Patch ID: 138962

Details

Message ID 1491622440-24352-1-git-send-email-david.reyna@windriver.com
State Superseded
Headers show

Commit Message

Reyna, David April 8, 2017, 3:34 a.m.
From: David Reyna <David.Reyna@windriver.com>

Use the TaskArtifacts event to scan the SDK and ESDK manifests
to cleanly collect the artifact files.
The previous method was broken when the SDK file deployment moved
from the do_populate_sdk[_ext] tasks to their sstate tasks.

[YOCTO #10850]

Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 30 ++++++++++++++++++++++++++++++
 bitbake/lib/bb/ui/toasterui.py       |  4 +---
 meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----
 3 files changed, 47 insertions(+), 8 deletions(-)

Patch hide | download patch | download mbox

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 92d1a1c..077c668 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -1663,6 +1663,36 @@  class BuildInfoHelper(object):
                 break
         return endswith
 
+    def scan_task_artifacts(self, event):
+        """
+        The 'TaskArtifacts' event passes the manifest file content for the
+        tasks 'do_deploy', 'do_image_complete', 'do_populate_sdk', and
+        'do_populate_sdk_ext'. The first two will be implemented later.
+        """
+        task_vars = BuildInfoHelper._get_data_from_event(event)
+        task_name = task_vars['task'][task_vars['task'].find(':')+1:]
+        task_artifacts = task_vars['artifacts']
+
+        if task_name in ['do_populate_sdk', 'do_populate_sdk_ext']:
+            targets = [target for target in self.internal_state['targets'] \
+                if target.task == task_name[3:]]
+            if not targets:
+                logger.warning("scan_task_artifacts: SDK targets not found: \n%s", task_name)
+                return
+            for artifact_path in task_artifacts:
+                if not os.path.isfile(artifact_path):
+                    logger.warning("scan_task_artifacts: artifact file not found: \n%s", artifact_path)
+                    continue
+                for target in targets:
+                    # don't record the file if it's already been added
+                    # to this target
+                    matching_files = TargetSDKFile.objects.filter(
+                        target=target, file_name=artifact_path)
+                    if matching_files.count() == 0:
+                        artifact_size = os.stat(artifact_path).st_size
+                        self.orm_wrapper.save_target_sdk_file(
+                            target, artifact_path, artifact_size)
+
     def _get_image_files(self, deploy_dir_image, image_name, image_file_extensions):
         """
         Find files in deploy_dir_image whose basename starts with the
diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 1729902..71f04fa 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -438,9 +438,7 @@  def main(server, eventHandler, params):
                 elif event.type == "SetBRBE":
                     buildinfohelper.brbe = buildinfohelper._get_data_from_event(event)
                 elif event.type == "TaskArtifacts":
-                    # not implemented yet
-                    # see https://bugzilla.yoctoproject.org/show_bug.cgi?id=10283 for details
-                    pass
+                    buildinfohelper.scan_task_artifacts(event)
                 elif event.type == "OSErrorException":
                     logger.error(event)
                 else:
diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
index 4de380b..296e476 100644
--- a/meta/classes/toaster.bbclass
+++ b/meta/classes/toaster.bbclass
@@ -328,8 +328,18 @@  python toaster_artifacts() {
     if e.taskname in ["do_deploy", "do_image_complete", "do_populate_sdk", "do_populate_sdk_ext"]:
         d2 = d.createCopy()
         d2.setVar('FILE', e.taskfile)
-        d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
+        # Use 'stamp-extra-info' if present, else use workaround
+        # to determine 'SSTATE_MANMACH'
+        extrainf = d2.getVarFlag(e.taskname, 'stamp-extra-info')
+        if extrainf:
+            d2.setVar('SSTATE_MANMACH', extrainf)
+        else:
+            if "do_populate_sdk" == e.taskname:
+                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}${SDKMACHINE}"))
+            else:
+                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
         manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:], d2)[0]
+
         if os.access(manifest, os.R_OK):
             with open(manifest) as fmanifest:
                 artifacts = [fname.strip() for fname in fmanifest]
@@ -356,8 +366,9 @@  do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
 
-do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
-do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+#do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
+#do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+
+#do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
+#do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
 
-do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
-do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "

Comments

brian avery April 9, 2017, 5:49 p.m.
one quick comment:
the poky-contrib branch you pushed up was :

username/toaster/sdk_artifacts_10850

but should have been

dreyna/toaster/sdk_artifacts_10850

:)

-b

an intel employee

On Fri, Apr 7, 2017 at 8:34 PM, David Reyna <david.reyna@windriver.com>
wrote:

> From: David Reyna <David.Reyna@windriver.com>
>
> Use the TaskArtifacts event to scan the SDK and ESDK manifests
> to cleanly collect the artifact files.
> The previous method was broken when the SDK file deployment moved
> from the do_populate_sdk[_ext] tasks to their sstate tasks.
>
> [YOCTO #10850]
>
> Signed-off-by: David Reyna <David.Reyna@windriver.com>
> ---
>  bitbake/lib/bb/ui/buildinfohelper.py | 30 ++++++++++++++++++++++++++++++
>  bitbake/lib/bb/ui/toasterui.py       |  4 +---
>  meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----
>  3 files changed, 47 insertions(+), 8 deletions(-)
>
> diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/
> buildinfohelper.py
> index 92d1a1c..077c668 100644
> --- a/bitbake/lib/bb/ui/buildinfohelper.py
> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
> @@ -1663,6 +1663,36 @@ class BuildInfoHelper(object):
>                  break
>          return endswith
>
> +    def scan_task_artifacts(self, event):
> +        """
> +        The 'TaskArtifacts' event passes the manifest file content for the
> +        tasks 'do_deploy', 'do_image_complete', 'do_populate_sdk', and
> +        'do_populate_sdk_ext'. The first two will be implemented later.
> +        """
> +        task_vars = BuildInfoHelper._get_data_from_event(event)
> +        task_name = task_vars['task'][task_vars['task'].find(':')+1:]
> +        task_artifacts = task_vars['artifacts']
> +
> +        if task_name in ['do_populate_sdk', 'do_populate_sdk_ext']:
> +            targets = [target for target in
> self.internal_state['targets'] \
> +                if target.task == task_name[3:]]
> +            if not targets:
> +                logger.warning("scan_task_artifacts: SDK targets not
> found: \n%s", task_name)
> +                return
> +            for artifact_path in task_artifacts:
> +                if not os.path.isfile(artifact_path):
> +                    logger.warning("scan_task_artifacts: artifact file
> not found: \n%s", artifact_path)
> +                    continue
> +                for target in targets:
> +                    # don't record the file if it's already been added
> +                    # to this target
> +                    matching_files = TargetSDKFile.objects.filter(
> +                        target=target, file_name=artifact_path)
> +                    if matching_files.count() == 0:
> +                        artifact_size = os.stat(artifact_path).st_size
> +                        self.orm_wrapper.save_target_sdk_file(
> +                            target, artifact_path, artifact_size)
> +
>      def _get_image_files(self, deploy_dir_image, image_name,
> image_file_extensions):
>          """
>          Find files in deploy_dir_image whose basename starts with the
> diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.
> py
> index 1729902..71f04fa 100644
> --- a/bitbake/lib/bb/ui/toasterui.py
> +++ b/bitbake/lib/bb/ui/toasterui.py
> @@ -438,9 +438,7 @@ def main(server, eventHandler, params):
>                  elif event.type == "SetBRBE":
>                      buildinfohelper.brbe = buildinfohelper._get_data_
> from_event(event)
>                  elif event.type == "TaskArtifacts":
> -                    # not implemented yet
> -                    # see https://bugzilla.yoctoproject.
> org/show_bug.cgi?id=10283 for details
> -                    pass
> +                    buildinfohelper.scan_task_artifacts(event)
>                  elif event.type == "OSErrorException":
>                      logger.error(event)
>                  else:
> diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
> index 4de380b..296e476 100644
> --- a/meta/classes/toaster.bbclass
> +++ b/meta/classes/toaster.bbclass
> @@ -328,8 +328,18 @@ python toaster_artifacts() {
>      if e.taskname in ["do_deploy", "do_image_complete",
> "do_populate_sdk", "do_populate_sdk_ext"]:
>          d2 = d.createCopy()
>          d2.setVar('FILE', e.taskfile)
> -        d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
> +        # Use 'stamp-extra-info' if present, else use workaround
> +        # to determine 'SSTATE_MANMACH'
> +        extrainf = d2.getVarFlag(e.taskname, 'stamp-extra-info')
> +        if extrainf:
> +            d2.setVar('SSTATE_MANMACH', extrainf)
> +        else:
> +            if "do_populate_sdk" == e.taskname:
> +                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}${
> SDKMACHINE}"))
> +            else:
> +                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
>          manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:],
> d2)[0]
> +
>          if os.access(manifest, os.R_OK):
>              with open(manifest) as fmanifest:
>                  artifacts = [fname.strip() for fname in fmanifest]
> @@ -356,8 +366,9 @@ do_packagedata_setscene[vardepsexclude] +=
> "toaster_package_dumpdata "
>  do_package[postfuncs] += "toaster_package_dumpdata "
>  do_package[vardepsexclude] += "toaster_package_dumpdata "
>
> -do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
> -do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
> +#do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
> +#do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
> +
> +#do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
> +#do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
>
> -do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
> -do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
> --
> 1.9.1
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>
Reyna, David April 9, 2017, 11:07 p.m.
Hi Brian,

> username/toaster/sdk_artifacts_10850

Hmm, you are right. I was using an incomplete template. I have refactored the patch and it should appear shortly (I see in poky-contrib now).

Would you possibly be able to submit upstream this patch and #11213 patch ([Toaster] [PATCH] toaster: fix add layer button for Machines) today in time for the April 10 deadline? I understand from Mark Hatle that the deadline is per Richard’s timezone, which means tonight. I tested #11213 against both Firefox and Chrome, and I tested #10850 against qemux86, qemuarm, qemuarm64, qemumips64, and genericx86-64 for a mix MACHINES and 64/32.

If you were not available I was going to self-submit today (my Sunday) it on our behalf, with an apology to Richard.

BTW, you probably observed that my “sdk_artifacts_10850” patch was in one patch file. Is that ok for this small patch? I could break it up if that is required (I assume by making a separate commit per patch directory before sending).

Thanks,
David

From: Brian Avery [mailto:avery.brian@gmail.com]

Sent: Sunday, April 09, 2017 10:49 AM
To: Reyna, David
Cc: toaster@yoctoproject.org
Subject: Re: [Toaster] [PATCH] toaster: fix SDK artifact capture

one quick comment:
the poky-contrib branch you pushed up was :

username/toaster/sdk_artifacts_10850

but should have been

dreyna/toaster/sdk_artifacts_10850

:)

-b

an intel employee

On Fri, Apr 7, 2017 at 8:34 PM, David Reyna <david.reyna@windriver.com<mailto:david.reyna@windriver.com>> wrote:
From: David Reyna <David.Reyna@windriver.com<mailto:David.Reyna@windriver.com>>


Use the TaskArtifacts event to scan the SDK and ESDK manifests
to cleanly collect the artifact files.
The previous method was broken when the SDK file deployment moved
from the do_populate_sdk[_ext] tasks to their sstate tasks.

[YOCTO #10850]

Signed-off-by: David Reyna <David.Reyna@windriver.com<mailto:David.Reyna@windriver.com>>

---
 bitbake/lib/bb/ui/buildinfohelper.py | 30 ++++++++++++++++++++++++++++++
 bitbake/lib/bb/ui/toasterui.py       |  4 +---
 meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----
 3 files changed, 47 insertions(+), 8 deletions(-)

--
1.9.1

--
_______________________________________________
toaster mailing list
toaster@yoctoproject.org<mailto:toaster@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/toaster
Reyna, David April 10, 2017, 8:55 a.m.
This is a system generated Comment: Patch 138962 was automatically marked as superseded by patch 138967.