[1/1] sstate: list missing files for toaster

Submitted by Alexandru DAMIAN on Feb. 17, 2014, 5:33 p.m.

Details

Message ID 1392658389-19826-1-git-send-email-alexandru.damian@intel.com
State Accepted
Commit 109ae6c5c981610ab0d63d2c83dcd50b2e93276b
Headers show

Commit Message

Alexandru DAMIAN Feb. 17, 2014, 5:33 p.m.
From: Alexandru DAMIAN <alexandru.damian@intel.com>

Toaster needs to record the attempts to restore
setscene tasks that don't have a sstate file.

We build a list of tasks for which we can't find an
sstate file, and if we're running under Toaster data
collection, we send it off with a MetadataEvent.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
 meta/classes/sstate.bbclass | 11 +++++++++++
 1 file changed, 11 insertions(+)

Patch hide | download patch | download mbox

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index f7bd117..f70c62f 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -621,6 +621,7 @@  BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     ret = []
+    missed = []
 
     def getpathcomponents(task, d):
         # Magic data from BB_HASHFILENAME
@@ -642,11 +643,13 @@  def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
         spec, extrapath, tname = getpathcomponents(task, d)
 
         sstatefile = d.expand("${SSTATE_DIR}/" + extrapath + generate_sstatefn(spec, sq_hash[task], d) + "_" + tname + ".tgz.siginfo")
+
         if os.path.exists(sstatefile):
             bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
             ret.append(task)
             continue
         else:
+            missed.append(task)
             bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile)
 
     mirrors = d.getVar("SSTATE_MIRRORS", True)
@@ -684,9 +687,17 @@  def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
                 bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
                 ret.append(task)
             except:
+                missed.append(task)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri)
                 pass     
 
+    inheritlist = d.getVar("INHERIT", True)
+    if "toaster" in inheritlist:
+        evdata = []
+        for task in missed:
+            evdata.append( (sq_fn[task], sq_task[task], sq_hash[task], generate_sstatefn(spec, sq_hash[task],d) ) )
+        bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"

Comments

Richard Purdie Feb. 20, 2014, 2:30 p.m.
On Mon, 2014-02-17 at 17:33 +0000, Alex DAMIAN wrote:
> From: Alexandru DAMIAN <alexandru.damian@intel.com>
> 
> Toaster needs to record the attempts to restore
> setscene tasks that don't have a sstate file.
> 
> We build a list of tasks for which we can't find an
> sstate file, and if we're running under Toaster data
> collection, we send it off with a MetadataEvent.
> 
> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
> ---
>  meta/classes/sstate.bbclass | 11 +++++++++++
>  1 file changed, 11 insertions(+)

I'm wondering if we should perhaps do this in bitbake and add a setscene
"missed" event type there based upon the return value of this function?

Ideally I'd like to be able to report the stats in knotty too.

Cheers,

Richard
Alexandru DAMIAN Feb. 20, 2014, 3:41 p.m.
I would think against it, because

- it's very inefficient to send a Skipped event for each missed task than
to collect all the data and send it in one shot
- it's faster to send the event from the sstate.bbclass than to return the
task ids and then call again a python function to generate the sstate file
names - basically replicating the sstate.bbclass code in bitbake
- the queueTaskSkipped event has a different semantic than what we need
here to return a whole batch of data; creating a new event type from
scratch is a bit pointless since the inheritance would need to be broken
- knotty can receive MetadataEvents just as well as sceneTaskSkipped event,
or any variation of it

I would suggest actually just sending this event at all times, not just
when toaster is enabled.

Please advise,
Alex


On Thu, Feb 20, 2014 at 2:30 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2014-02-17 at 17:33 +0000, Alex DAMIAN wrote:
> > From: Alexandru DAMIAN <alexandru.damian@intel.com>
> >
> > Toaster needs to record the attempts to restore
> > setscene tasks that don't have a sstate file.
> >
> > We build a list of tasks for which we can't find an
> > sstate file, and if we're running under Toaster data
> > collection, we send it off with a MetadataEvent.
> >
> > Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
> > ---
> >  meta/classes/sstate.bbclass | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
>
> I'm wondering if we should perhaps do this in bitbake and add a setscene
> "missed" event type there based upon the return value of this function?
>
> Ideally I'd like to be able to report the stats in knotty too.
>
> Cheers,
>
> Richard
>
>
Richard Purdie Feb. 20, 2014, 4:06 p.m.
On Thu, 2014-02-20 at 15:41 +0000, Damian, Alexandru wrote:
> I would think against it, because

> - it's very inefficient to send a Skipped event for each missed task
> than to collect all the data and send it in one shot
> - it's faster to send the event from the sstate.bbclass than to return
> the task ids and then call again a python function to generate the
> sstate file names - basically replicating the sstate.bbclass code in
> bitbake
> - the queueTaskSkipped event has a different semantic than what we
> need here to return a whole batch of data; creating a new event type
> from scratch is a bit pointless since the inheritance would need to be
> broken
> - knotty can receive MetadataEvents just as well as sceneTaskSkipped
> event, or any variation of it

I guess I'd like to understand which information from this even you're
particularly interested in?

Do I understand correctly that you want the actual filenames attempted?

Cheers,

Richard
Alexandru DAMIAN Feb. 20, 2014, 8:29 p.m.
Yep, tasks that were looked up, and actual filenames attempted.

Alex


On Thu, Feb 20, 2014 at 4:06 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2014-02-20 at 15:41 +0000, Damian, Alexandru wrote:
> > I would think against it, because
>
> > - it's very inefficient to send a Skipped event for each missed task
> > than to collect all the data and send it in one shot
> > - it's faster to send the event from the sstate.bbclass than to return
> > the task ids and then call again a python function to generate the
> > sstate file names - basically replicating the sstate.bbclass code in
> > bitbake
> > - the queueTaskSkipped event has a different semantic than what we
> > need here to return a whole batch of data; creating a new event type
> > from scratch is a bit pointless since the inheritance would need to be
> > broken
> > - knotty can receive MetadataEvents just as well as sceneTaskSkipped
> > event, or any variation of it
>
> I guess I'd like to understand which information from this even you're
> particularly interested in?
>
> Do I understand correctly that you want the actual filenames attempted?
>
> Cheers,
>
> Richard
>
>
>
Alex Damian March 13, 2014, 12:28 p.m.
Hello,

I am following up with this patch. If there are no other objects, can we
please merge it ?

Thanks,
Alex


On Thu, Feb 20, 2014 at 8:29 PM, Damian, Alexandru <
alexandru.damian@intel.com> wrote:

> Yep, tasks that were looked up, and actual filenames attempted.
>
> Alex
>
>
> On Thu, Feb 20, 2014 at 4:06 PM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> On Thu, 2014-02-20 at 15:41 +0000, Damian, Alexandru wrote:
>> > I would think against it, because
>>
>> > - it's very inefficient to send a Skipped event for each missed task
>> > than to collect all the data and send it in one shot
>> > - it's faster to send the event from the sstate.bbclass than to return
>> > the task ids and then call again a python function to generate the
>> > sstate file names - basically replicating the sstate.bbclass code in
>> > bitbake
>> > - the queueTaskSkipped event has a different semantic than what we
>> > need here to return a whole batch of data; creating a new event type
>> > from scratch is a bit pointless since the inheritance would need to be
>> > broken
>> > - knotty can receive MetadataEvents just as well as sceneTaskSkipped
>> > event, or any variation of it
>>
>> I guess I'd like to understand which information from this even you're
>> particularly interested in?
>>
>> Do I understand correctly that you want the actual filenames attempted?
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>
>
> --
> Alex Damian
> Yocto Project
> SSG / OTC
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>