Patchwork [bitbake-devel,v4,14/18] runqueue.py, build.py: Invalidate setscene tasks based on do_unpack stamps

If you have a fully populated sstate cache and have used it to
execute a build, it is not possible to invalidate repackage
an intermediate build after you have forced a compiled

Example when you have build from a complete sstate cache build:
  bitbake core-image-sato
  bitbake -c patch acl
     *** Make some changes to the C files for experimentation.
  bitbake -f -c compile acl
  bitbake acl

The bitbake will refuse to build the acl package at this
point and instead keep populating it from the sstate.  Using
the cleanstate is no longer a good option because it will
also erase the scratch area.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/build.py    |    8 ++++++++
 lib/bb/runqueue.py |   14 ++++++++++++++
 2 files changed, 22 insertions(+)

On Fri, 2012-06-08 at 08:41 -0500, Jason Wessel wrote:
> If you have a fully populated sstate cache and have used it to
> execute a build, it is not possible to invalidate repackage
> an intermediate build after you have forced a compiled
> 
> Example when you have build from a complete sstate cache build:
>   bitbake core-image-sato
>   bitbake -c patch acl
>      *** Make some changes to the C files for experimentation.
>   bitbake -f -c compile acl
>   bitbake acl
> 
> The bitbake will refuse to build the acl package at this
> point and instead keep populating it from the sstate.  Using
> the cleanstate is no longer a good option because it will
> also erase the scratch area.
> 
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>  lib/bb/build.py    |    8 ++++++++
>  lib/bb/runqueue.py |   14 ++++++++++++++
>  2 files changed, 22 insertions(+)
> 
> diff --git a/lib/bb/build.py b/lib/bb/build.py
> index fb61b00..18c28aa 100644
> --- a/lib/bb/build.py
> +++ b/lib/bb/build.py
> @@ -463,6 +463,14 @@ def del_stamp(task, d, file_name = None):
>      stamp = stamp_internal(task, d, file_name)
>      bb.utils.remove(stamp)
>  
> +def exists_stamp(task, d, file_name = None):
> +    """
> +    Removes a stamp for a given task
> +    (d can be a data dict or dataCache)
> +    """
> +    stamp = stamp_internal(task, d, file_name)
> +    return os.path.exists(stamp)
> +
>  def stampfile(taskname, d, file_name = None):
>      """
>      Return the stamp for a given task
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index da3fdf9..6c802af 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -718,6 +718,20 @@ class RunQueueData:
>                      for dep in self.runq_depends[task]:
>                          procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
>                      self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
> +        try:
> +            new_setscene = []
> +            for task in self.runq_setscene:
> +                try:
> +                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
> +                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
> +                        logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
> +                    else:
> +                        new_setscene.append(task)
> +                except:
> +                    logger.debug(2, 'Failed do_unpack check for %s', task)
> +            self.runq_setscene = new_setscene
> +        except:
> +            logger.debug(2, 'Failed to update runq_setscene')

We've gone to quite a bit of trouble to keep knowledge of the specific
tasks out of bitbake. I've noticed a few of your patches are "blurring"
the separation between bitbake and the metadata.

Here, bitbake shouldn't have knowledge of "unpack" or why its special.
I'd also argue its not in fact special and there are probably other
"force" scenarios which would break with a similar problem even with
this patch. I agree there is a problem here but we need to fix it in a
more generic way. There is already a bug open on this kind of problem
(#2256).

Cheers,

Richard

On 06/11/2012 08:52 AM, Richard Purdie wrote:
> On Fri, 2012-06-08 at 08:41 -0500, Jason Wessel wrote:
>> If you have a fully populated sstate cache and have used it to
>> execute a build, it is not possible to invalidate repackage
>> an intermediate build after you have forced a compiled
>>
>> Example when you have build from a complete sstate cache build:
>>   bitbake core-image-sato
>>   bitbake -c patch acl
>>      *** Make some changes to the C files for experimentation.
>>   bitbake -f -c compile acl
>>   bitbake acl
>>
>> The bitbake will refuse to build the acl package at this
>> point and instead keep populating it from the sstate.  Using
>> the cleanstate is no longer a good option because it will
>> also erase the scratch area.
>>
>> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
>> ---
>>  lib/bb/build.py    |    8 ++++++++
>>  lib/bb/runqueue.py |   14 ++++++++++++++
>>  2 files changed, 22 insertions(+)
>>
>> diff --git a/lib/bb/build.py b/lib/bb/build.py
>> index fb61b00..18c28aa 100644
>> --- a/lib/bb/build.py
>> +++ b/lib/bb/build.py
>> @@ -463,6 +463,14 @@ def del_stamp(task, d, file_name = None):
>>      stamp = stamp_internal(task, d, file_name)
>>      bb.utils.remove(stamp)
>>  
>> +def exists_stamp(task, d, file_name = None):
>> +    """
>> +    Removes a stamp for a given task
>> +    (d can be a data dict or dataCache)
>> +    """
>> +    stamp = stamp_internal(task, d, file_name)
>> +    return os.path.exists(stamp)
>> +
>>  def stampfile(taskname, d, file_name = None):
>>      """
>>      Return the stamp for a given task
>> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
>> index da3fdf9..6c802af 100644
>> --- a/lib/bb/runqueue.py
>> +++ b/lib/bb/runqueue.py
>> @@ -718,6 +718,20 @@ class RunQueueData:
>>                      for dep in self.runq_depends[task]:
>>                          procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
>>                      self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
>> +        try:
>> +            new_setscene = []
>> +            for task in self.runq_setscene:
>> +                try:
>> +                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
>> +                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
>> +                        logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
>> +                    else:
>> +                        new_setscene.append(task)
>> +                except:
>> +                    logger.debug(2, 'Failed do_unpack check for %s', task)
>> +            self.runq_setscene = new_setscene
>> +        except:
>> +            logger.debug(2, 'Failed to update runq_setscene')
> We've gone to quite a bit of trouble to keep knowledge of the specific
> tasks out of bitbake. I've noticed a few of your patches are "blurring"
> the separation between bitbake and the metadata.
>
> Here, bitbake shouldn't have knowledge of "unpack" or why its special.
> I'd also argue its not in fact special and there are probably other
> "force" scenarios which would break with a similar problem even with
> this patch. I agree there is a problem here but we need to fix it in a
> more generic way. There is already a bug open on this kind of problem
> (#2256).

As I mentioned in the 0/18 summary this is certainly the case that there needs to be an API and this should be treated as more of an RFC.

I will break whole series down into a few smaller chunks since there are some important fixes vs the features.


Jason.