Patchwork [0/3,RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?)

login
register
mail settings
Submitter Hongxu Jia
Date Aug. 6, 2014, 7:15 a.m.
Message ID <cover.1407306581.git.hongxu.jia@windriver.com>
Download mbox | patch
Permalink /patch/77355/
State New
Headers show

Pull-request

git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache

Comments

Hongxu Jia - Aug. 6, 2014, 7:15 a.m.
Issue description:

The issue is that the developer who demand only the "new" software
they write is allowed to be compiled from source, they only want to
reuse binaries from an existed sstate-cache, if the developer makes
a change that triggers a rebuild, it should be an instant error. 

The purpose of this is for the sstate-cache to check if the item
exists or not. If it doesn't the item needs to be in a whitelist
or we need to fail.

I dig into three approaches to implement, and send them to oe-core
list to see if the community has a preference as to the approach,
and any additional comments.

In these approaches, I'm not sure we should protect the 'clean' or
not. Since it's ok to clean the sstate-cache, as long as a mirror
fetch will pull it back down. Any suggestion is welcomed.

If you have any better ideas, please don't hesitate to share with us.

Test steps:

1) For approach 1
   INHERIT += 'sstate_readonly'

   For approach 2
   INHERIT += 'sstate_readonly_2'

2) Create a sstate cache for testing
$ bitbake db

3) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache,
vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip'
...

4) Remove tmp dir and build db from sstate-cache succeed
$ mv tmp tmp-back && bitbake db

5) tweak db's do_configure task by adding comments
$ bitbake db

7) clean db and there is a clean failure
bitbake db -ccleansstate

8) Append the missing recipe's PN to SSTATECACHE_WHITELIST
vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
...

9) Build db and clean db succeed
$ bitbake db && bitbake db -ccleansstate

//Hongxu

The following changes since commit 870bb8d35547b8313b3a487d7e8b914ab9470e64:

  local.conf.sample.extended: fix example for EXTRA_USERS_PARAMS (2014-08-04 17:38:24 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache

Hongxu Jia (3):
  bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS
  bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted
    time
  bbclass/sstate approach 3: add checking in the return path of
    sstate_checkhashes

 meta/classes/sstate.bbclass            | 40 ++++++++++++++++++++++++++++++++++
 meta/classes/sstate_readonly.bbclass   | 38 ++++++++++++++++++++++++++++++++
 meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++
 3 files changed, 103 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass
 create mode 100644 meta/classes/sstate_readonly_2.bbclass
Hongxu Jia - Aug. 20, 2014, 10:32 a.m.
Ping

//Hongxu

On 08/06/2014 03:15 PM, Hongxu Jia wrote:
> Issue description:
>
> The issue is that the developer who demand only the "new" software
> they write is allowed to be compiled from source, they only want to
> reuse binaries from an existed sstate-cache, if the developer makes
> a change that triggers a rebuild, it should be an instant error.
>
> The purpose of this is for the sstate-cache to check if the item
> exists or not. If it doesn't the item needs to be in a whitelist
> or we need to fail.
>
> I dig into three approaches to implement, and send them to oe-core
> list to see if the community has a preference as to the approach,
> and any additional comments.
>
> In these approaches, I'm not sure we should protect the 'clean' or
> not. Since it's ok to clean the sstate-cache, as long as a mirror
> fetch will pull it back down. Any suggestion is welcomed.
>
> If you have any better ideas, please don't hesitate to share with us.
>
> Test steps:
>
> 1) For approach 1
>     INHERIT += 'sstate_readonly'
>
>     For approach 2
>     INHERIT += 'sstate_readonly_2'
>
> 2) Create a sstate cache for testing
> $ bitbake db
>
> 3) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache,
> vim local.conf
> ...
> SSTATECACHE_WHITELIST = 'gzip'
> ...
>
> 4) Remove tmp dir and build db from sstate-cache succeed
> $ mv tmp tmp-back && bitbake db
>
> 5) tweak db's do_configure task by adding comments
> --- a/meta/recipes-support/db/db_6.0.30.bb
> +++ b/meta/recipes-support/db/db_6.0.30.bb
> @@ -27,6 +27,8 @@ LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"
> @@ -82,6 +84,7 @@ do_configure() {
>          gnu-configize --force ${S}
>          export STRIP="true"
>          oe_runconf
> +       echo "hello"
>   }
>
> 6) build db and there is a build failure
> $ bitbake db
>
> 7) clean db and there is a clean failure
> bitbake db -ccleansstate
>
> 8) Append the missing recipe's PN to SSTATECACHE_WHITELIST
> vim local.conf
> ...
> SSTATECACHE_WHITELIST = 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
> ...
>
> 9) Build db and clean db succeed
> $ bitbake db && bitbake db -ccleansstate
>
> //Hongxu
>
> The following changes since commit 870bb8d35547b8313b3a487d7e8b914ab9470e64:
>
>    local.conf.sample.extended: fix example for EXTRA_USERS_PARAMS (2014-08-04 17:38:24 +0100)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache
>
> Hongxu Jia (3):
>    bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS
>    bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted
>      time
>    bbclass/sstate approach 3: add checking in the return path of
>      sstate_checkhashes
>
>   meta/classes/sstate.bbclass            | 40 ++++++++++++++++++++++++++++++++++
>   meta/classes/sstate_readonly.bbclass   | 38 ++++++++++++++++++++++++++++++++
>   meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++
>   3 files changed, 103 insertions(+)
>   create mode 100644 meta/classes/sstate_readonly.bbclass
>   create mode 100644 meta/classes/sstate_readonly_2.bbclass
>

Patch

--- a/meta/recipes-support/db/db_6.0.30.bb
+++ b/meta/recipes-support/db/db_6.0.30.bb
@@ -27,6 +27,8 @@  LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"
@@ -82,6 +84,7 @@  do_configure() {
        gnu-configize --force ${S}
        export STRIP="true"
        oe_runconf
+       echo "hello"
 }

6) build db and there is a build failure