[v4,2/5] bitbake: cooker: add a new function to retrieve task signatures

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the
signatures using tinfoil

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)

On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> adding a new command in cooker to compute and get task signatures
>
> this commit also add the associated command and event needed to get the
> signatures using tinfoil
>
> Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> ---
>  bitbake/lib/bb/command.py |  6 ++++++
>  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
>  bitbake/lib/bb/event.py   |  8 ++++++++
>  3 files changed, 30 insertions(+)
>
> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> index a355f56c60c..12202779ac0 100644
> --- a/bitbake/lib/bb/command.py
> +++ b/bitbake/lib/bb/command.py
> @@ -776,3 +776,9 @@ class CommandsAsync:
>          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
>          command.finishAsyncCommand()
>      findSigInfo.needcache = False
> +
> +    def getTaskSignatures(self, command, params):
> +        res = command.cooker.getTaskSignatures(params[0], params[1])
> +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> +        command.finishAsyncCommand()
> +    getTaskSignatures.needcache = True
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 11c9fa2c40d..687cdde5e6d 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1542,6 +1542,22 @@ class BBCooker:
>
>          self.idleCallBackRegister(buildFileIdle, rq)
>
> +    def getTaskSignatures(self, target, task):
> +        sig = []
> +
> +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> +        rq.rqdata.prepare()
> +
> +        for key in rq.rqdata.runtaskentries:
> +            pn = bb.parse.siggen.tidtopn[key]
> +            taskname = bb.runqueue.taskname_from_tid(key)
> +            if pn in target:
> +                if (task and taskname in task) or (not task):
> +                    rq.rqdata.prepare_task_hash(key)
> +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> +        return sig
> +

Hi all,

I would like some feedback on this function. The goal of this function
was to be able to get a task signature (for bblock to use it, see bblock
series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks

Using buildTaskData with "do_build" task gives me almost all tasks
signatures, at least, the "normal recipe build tasks" according to the doc [1].
But we don't get the signatures for the "Manually called Tasks"

At first I thought it would be good enough for bblock use case, but now
I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,
  but running `bitbake -c listtasks` gives *all* the tasks
* this function as is is not generic enough: we cannot get the task
  signature for all the "manually called tasks" (but is that really a
  problem?)
* bblock cannot lock those tasks

So what do you think? Should I make this function more generic and
called it several times to get *all* the tasks signature?

Cheers
Julien
[1]: https://docs.yoctoproject.org/ref-manual/tasks.html

>      def buildTargets(self, targets, task):
>          """
>          Attempt to build the targets specified
> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
> index 0d0e0a68aac..f8acacd80d1 100644
> --- a/bitbake/lib/bb/event.py
> +++ b/bitbake/lib/bb/event.py
> @@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
>          Event.__init__(self)
>          self.result = result
>
> +class GetTaskSignatureResult(Event):
> +    """
> +    Event to return results from GetTaskSignatures command
> +    """
> +    def __init__(self, sig):
> +        Event.__init__(self)
> +        self.sig = sig
> +
>  class ParseError(Event):
>      """
>      Event to indicate parse failed
> --
> 2.41.0
>

On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > adding a new command in cooker to compute and get task signatures
> > 
> > this commit also add the associated command and event needed to get the
> > signatures using tinfoil
> > 
> > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > ---
> >  bitbake/lib/bb/command.py |  6 ++++++
> >  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
> >  bitbake/lib/bb/event.py   |  8 ++++++++
> >  3 files changed, 30 insertions(+)
> > 
> > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > index a355f56c60c..12202779ac0 100644
> > --- a/bitbake/lib/bb/command.py
> > +++ b/bitbake/lib/bb/command.py
> > @@ -776,3 +776,9 @@ class CommandsAsync:
> >          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> >          command.finishAsyncCommand()
> >      findSigInfo.needcache = False
> > +
> > +    def getTaskSignatures(self, command, params):
> > +        res = command.cooker.getTaskSignatures(params[0], params[1])
> > +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > +        command.finishAsyncCommand()
> > +    getTaskSignatures.needcache = True
> > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > index 11c9fa2c40d..687cdde5e6d 100644
> > --- a/bitbake/lib/bb/cooker.py
> > +++ b/bitbake/lib/bb/cooker.py
> > @@ -1542,6 +1542,22 @@ class BBCooker:
> > 
> >          self.idleCallBackRegister(buildFileIdle, rq)
> > 
> > +    def getTaskSignatures(self, target, task):
> > +        sig = []
> > +
> > +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > +        rq.rqdata.prepare()
> > +
> > +        for key in rq.rqdata.runtaskentries:
> > +            pn = bb.parse.siggen.tidtopn[key]
> > +            taskname = bb.runqueue.taskname_from_tid(key)
> > +            if pn in target:
> > +                if (task and taskname in task) or (not task):
> > +                    rq.rqdata.prepare_task_hash(key)
> > +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > +        return sig
> > +

Firstly, thanks for working on this! These changes aren't simple and it
means the review takes longer as we need to get this right.

> I would like some feedback on this function. The goal of this function
> was to be able to get a task signature (for bblock to use it, see bblock
> series). The idea was:
> - get the task signatures of "task"
> - if "task" is empty, get the signature of all tasks
> 
> Using buildTaskData with "do_build" task gives me almost all tasks
> signatures, at least, the "normal recipe build tasks" according to the doc [1].
> But we don't get the signatures for the "Manually called Tasks"
> 
> At first I thought it would be good enough for bblock use case, but now
> I am not so sure anymore:
> * for oeqa self test, I need to have access to the same list of tasks,

I think you need to query the tasks available for a recipe, remove the
"nostamp" ones and then return the signatures for the rest.

>   but running `bitbake -c listtasks` gives *all* the tasks
> * this function as is is not generic enough: we cannot get the task
>   signature for all the "manually called tasks" (but is that really a
>   problem?)
> * bblock cannot lock those tasks
> 
> So what do you think? Should I make this function more generic and
> called it several times to get *all* the tasks signature?

I think bitbake should internally be able to work out the nostamp ones
and return a list of all of them with the nostamp ones excluded.

Let me know if you need some pointers on how to do that in bitbake, it
isn't entirely straightforward and I'd have to look it up. I *think*
(without looking to know for sure) that the dataCache object does have
the task data in it which can be looked up by target and that lists
which ones are nostamp.

Cheers,

Richard

Le ven. 11 août 2023 à 17:16, Richard Purdie
<richard.purdie@linuxfoundation.org> a écrit :
>
> On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> > On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > > adding a new command in cooker to compute and get task signatures
> > >
> > > this commit also add the associated command and event needed to get the
> > > signatures using tinfoil
> > >
> > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > ---
> > >  bitbake/lib/bb/command.py |  6 ++++++
> > >  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
> > >  bitbake/lib/bb/event.py   |  8 ++++++++
> > >  3 files changed, 30 insertions(+)
> > >
> > > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > > index a355f56c60c..12202779ac0 100644
> > > --- a/bitbake/lib/bb/command.py
> > > +++ b/bitbake/lib/bb/command.py
> > > @@ -776,3 +776,9 @@ class CommandsAsync:
> > >          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> > >          command.finishAsyncCommand()
> > >      findSigInfo.needcache = False
> > > +
> > > +    def getTaskSignatures(self, command, params):
> > > +        res = command.cooker.getTaskSignatures(params[0], params[1])
> > > +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > > +        command.finishAsyncCommand()
> > > +    getTaskSignatures.needcache = True
> > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > > index 11c9fa2c40d..687cdde5e6d 100644
> > > --- a/bitbake/lib/bb/cooker.py
> > > +++ b/bitbake/lib/bb/cooker.py
> > > @@ -1542,6 +1542,22 @@ class BBCooker:
> > >
> > >          self.idleCallBackRegister(buildFileIdle, rq)
> > >
> > > +    def getTaskSignatures(self, target, task):
> > > +        sig = []
> > > +
> > > +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > > +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > > +        rq.rqdata.prepare()
> > > +
> > > +        for key in rq.rqdata.runtaskentries:
> > > +            pn = bb.parse.siggen.tidtopn[key]
> > > +            taskname = bb.runqueue.taskname_from_tid(key)
> > > +            if pn in target:
> > > +                if (task and taskname in task) or (not task):
> > > +                    rq.rqdata.prepare_task_hash(key)
> > > +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > > +        return sig
> > > +
>
> Firstly, thanks for working on this! These changes aren't simple and it
> means the review takes longer as we need to get this right.
>
> > I would like some feedback on this function. The goal of this function
> > was to be able to get a task signature (for bblock to use it, see bblock
> > series). The idea was:
> > - get the task signatures of "task"
> > - if "task" is empty, get the signature of all tasks
> >
> > Using buildTaskData with "do_build" task gives me almost all tasks
> > signatures, at least, the "normal recipe build tasks" according to the doc [1].
> > But we don't get the signatures for the "Manually called Tasks"
> >
> > At first I thought it would be good enough for bblock use case, but now
> > I am not so sure anymore:
> > * for oeqa self test, I need to have access to the same list of tasks,
>
> I think you need to query the tasks available for a recipe, remove the
> "nostamp" ones and then return the signatures for the rest.
>
> >   but running `bitbake -c listtasks` gives *all* the tasks
> > * this function as is is not generic enough: we cannot get the task
> >   signature for all the "manually called tasks" (but is that really a
> >   problem?)
> > * bblock cannot lock those tasks
> >
> > So what do you think? Should I make this function more generic and
> > called it several times to get *all* the tasks signature?
>
> I think bitbake should internally be able to work out the nostamp ones
> and return a list of all of them with the nostamp ones excluded.
>
> Let me know if you need some pointers on how to do that in bitbake, it
> isn't entirely straightforward and I'd have to look it up. I *think*
> (without looking to know for sure) that the dataCache object does have
> the task data in it which can be looked up by target and that lists
> which ones are nostamp.

Hi Richard,

Sorry for the late reply I was off last week.
Using rq.rqdata.dataCaches I am able to get the full list of tasks and
filter out the no stamps.
But it is still unclear for me what to do :( Here is what I
understood, please correct me if I am wrong:
* in getTaskSignatures (cooker.py): I should get the tasks list and
filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ),
then compute the signature for all
* in oeqa self test, I get the list of tasks using 'bitbake -c
listtasks', but how to filter out the 'nostamps' ones?

Cheers
Julien
>
> Cheers,
>
> Richard
>
>

Hi,

Sorry, I meant to reply to this and somehow it got lost with everything
else going on.

On Tue, 2023-08-22 at 11:27 +0200, Julien Stephan wrote:
> Le ven. 11 août 2023 à 17:16, Richard Purdie
> <richard.purdie@linuxfoundation.org> a écrit :
> > 
> > On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> > > On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > > > adding a new command in cooker to compute and get task signatures
> > > > 
> > > > this commit also add the associated command and event needed to get the
> > > > signatures using tinfoil
> > > > 
> > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > > ---
> > > >  bitbake/lib/bb/command.py |  6 ++++++
> > > >  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
> > > >  bitbake/lib/bb/event.py   |  8 ++++++++
> > > >  3 files changed, 30 insertions(+)
> > > > 
> > > > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > > > index a355f56c60c..12202779ac0 100644
> > > > --- a/bitbake/lib/bb/command.py
> > > > +++ b/bitbake/lib/bb/command.py
> > > > @@ -776,3 +776,9 @@ class CommandsAsync:
> > > >          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> > > >          command.finishAsyncCommand()
> > > >      findSigInfo.needcache = False
> > > > +
> > > > +    def getTaskSignatures(self, command, params):
> > > > +        res = command.cooker.getTaskSignatures(params[0], params[1])
> > > > +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > > > +        command.finishAsyncCommand()
> > > > +    getTaskSignatures.needcache = True
> > > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > > > index 11c9fa2c40d..687cdde5e6d 100644
> > > > --- a/bitbake/lib/bb/cooker.py
> > > > +++ b/bitbake/lib/bb/cooker.py
> > > > @@ -1542,6 +1542,22 @@ class BBCooker:
> > > > 
> > > >          self.idleCallBackRegister(buildFileIdle, rq)
> > > > 
> > > > +    def getTaskSignatures(self, target, task):
> > > > +        sig = []
> > > > +
> > > > +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > > > +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > > > +        rq.rqdata.prepare()
> > > > +
> > > > +        for key in rq.rqdata.runtaskentries:
> > > > +            pn = bb.parse.siggen.tidtopn[key]
> > > > +            taskname = bb.runqueue.taskname_from_tid(key)
> > > > +            if pn in target:
> > > > +                if (task and taskname in task) or (not task):
> > > > +                    rq.rqdata.prepare_task_hash(key)
> > > > +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > > > +        return sig
> > > > +
> > 
> > Firstly, thanks for working on this! These changes aren't simple and it
> > means the review takes longer as we need to get this right.
> > 
> > > I would like some feedback on this function. The goal of this function
> > > was to be able to get a task signature (for bblock to use it, see bblock
> > > series). The idea was:
> > > - get the task signatures of "task"
> > > - if "task" is empty, get the signature of all tasks
> > > 
> > > Using buildTaskData with "do_build" task gives me almost all tasks
> > > signatures, at least, the "normal recipe build tasks" according to the doc [1].
> > > But we don't get the signatures for the "Manually called Tasks"
> > > 
> > > At first I thought it would be good enough for bblock use case, but now
> > > I am not so sure anymore:
> > > * for oeqa self test, I need to have access to the same list of tasks,
> > 
> > I think you need to query the tasks available for a recipe, remove the
> > "nostamp" ones and then return the signatures for the rest.
> > 
> > >   but running `bitbake -c listtasks` gives *all* the tasks
> > > * this function as is is not generic enough: we cannot get the task
> > >   signature for all the "manually called tasks" (but is that really a
> > >   problem?)
> > > * bblock cannot lock those tasks
> > > 
> > > So what do you think? Should I make this function more generic and
> > > called it several times to get *all* the tasks signature?
> > 
> > I think bitbake should internally be able to work out the nostamp ones
> > and return a list of all of them with the nostamp ones excluded.
> > 
> > Let me know if you need some pointers on how to do that in bitbake, it
> > isn't entirely straightforward and I'd have to look it up. I *think*
> > (without looking to know for sure) that the dataCache object does have
> > the task data in it which can be looked up by target and that lists
> > which ones are nostamp.
> 
> Sorry for the late reply I was off last week.
> Using rq.rqdata.dataCaches I am able to get the full list of tasks and
> filter out the no stamps.
> But it is still unclear for me what to do :( Here is what I
> understood, please correct me if I am wrong:
> * in getTaskSignatures (cooker.py): I should get the tasks list and
> filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ),

You can skip the setscene ones as they're not interesting or needed for
what we need here, at least as I understand it as we never lock those.

> then compute the signature for all
> * in oeqa self test, I get the list of tasks using 'bitbake -c
> listtasks', but how to filter out the 'nostamps' ones?

Just thinking out loud, for a given recipe you could run
d.getVarFlag("do_clean", "nostamp") and see that it was set?

"bitbake-getvar -r bash do_clean -f nostamp --value" also shows that.

Cheers,

Richard