| Submitter | Richard Purdie |
|---|---|
| Date | May 29, 2012, 12:29 p.m. |
| Message ID | <1338294584.20169.103.camel@ted> |
| Download | mbox | patch |
| Permalink | /patch/28879/ |
| State | Accepted |
| Commit | 2acf8da4f13c175ea818b9514677b7059de1e3e2 |
| Headers | show |
Comments
On 05/29/2012 07:29 AM, Richard Purdie wrote: > Currently, ccache is used if it is present. When building from scratch it gives > no performance improvement and creates a ton of empty directories even when its > not in use. > > This change moves ccache support to a bbclass file which the user can choose to > enable. This should make builds more determinstic and make it easier/clearer > to the end user when its being used and when it is not. Are the sstate sums the same with and without the use of the new bbclass ccache? It is not clear to me that this is or is not the case based on this single commit header. Jason.
On Tue, 2012-05-29 at 07:36 -0500, Jason Wessel wrote: > On 05/29/2012 07:29 AM, Richard Purdie wrote: > > Currently, ccache is used if it is present. When building from scratch it gives > > no performance improvement and creates a ton of empty directories even when its > > not in use. > > > > This change moves ccache support to a bbclass file which the user can choose to > > enable. This should make builds more determinstic and make it easier/clearer > > to the end user when its being used and when it is not. > > Are the sstate sums the same with and without the use of the new bbclass ccache? > > It is not clear to me that this is or is not the case based on this single commit header. There is a problem with the existing sstate checksums since CCACHE itself is not being excluded, only CCACHE_DIR so you'd end up with sstate built using ccache and sstate built without using cccache with different checksums. This change unifies those to have matching sstate checksums. So with and without use of the class, the checksums should now be the same. Cheers, Richard
Richard Purdie <richard.purdie-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org> writes: > Currently, ccache is used if it is present. When building from scratch > it gives no performance improvement Here, I have a 10% cache hit ratio in a scratch build: cache hit (direct) 2841 cache hit (preprocessed) 1370 cache miss 39957 called for link 3369 called for preprocessing 3207 multiple source files 8 compile failed 2531 preprocessor error 432 couldn't find the compiler 111 bad compiler arguments 468 unsupported source language 532 autoconf compile/link 6613 unsupported compiler option 132 no input file 2919 files in cache 94205 cache size 2.0 Gbytes max cache size 3.0 Gbytes Enrico
On Tue, 2012-05-29 at 15:40 +0200, Enrico Scholz wrote: > Richard Purdie > <richard.purdie-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org> > writes: > > > Currently, ccache is used if it is present. When building from scratch > > it gives no performance improvement > > Here, I have a 10% cache hit ratio in a scratch build: > > cache hit (direct) 2841 > cache hit (preprocessed) 1370 > cache miss 39957 > called for link 3369 > called for preprocessing 3207 > multiple source files 8 > compile failed 2531 > preprocessor error 432 > couldn't find the compiler 111 > bad compiler arguments 468 > unsupported source language 532 > autoconf compile/link 6613 > unsupported compiler option 132 > no input file 2919 > files in cache 94205 > cache size 2.0 Gbytes > max cache size 3.0 Gbytes > That's good data, thanks for sharing it. In any measurement I've made, it doesn't seem to help the wall time for the build though :( We're not removing the ability to use ccache, just changing the way its enabled which all things considered I still think is a good move... Cheers, Richard
Patch
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass index b166d62..17aa5e1 100644 --- a/meta/classes/base.bbclass +++ b/meta/classes/base.bbclass @@ -271,7 +271,7 @@ python base_eventhandler() { } addtask configure after do_patch -do_configure[dirs] = "${CCACHE_DIR} ${S} ${B}" +do_configure[dirs] = "${S} ${B}" do_configure[deptask] = "do_populate_sysroot" base_do_configure() { : diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass new file mode 100644 index 0000000..10f9b9f --- a/dev/null +++ b/meta/classes/ccache.bbclass @@ -0,0 +1,5 @@ +CCACHE = "${@bb.which(d.getVar('PATH', True), 'ccache') and 'ccache '}" +export CCACHE_DIR = "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}" + +do_configure[dirs] =+ "${CCACHE_DIR}" +do_kernel_configme[dirs] =+ "${CCACHE_DIR}" diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass index e931630..2a3373c 100644 --- a/meta/classes/kernel-yocto.bbclass +++ b/meta/classes/kernel-yocto.bbclass @@ -161,7 +161,7 @@ do_kernel_checkout[dirs] = "${S}" addtask kernel_checkout before do_patch after do_unpack -do_kernel_configme[dirs] = "${CCACHE_DIR} ${S} ${B}" +do_kernel_configme[dirs] = "${S} ${B}" do_kernel_configme() { echo "[INFO] doing kernel configme" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 5fda36b..48287a7 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -409,10 +409,9 @@ export PATH # Build utility info. ################################################################## -CCACHE = "${@bb.which(d.getVar('PATH', True), 'ccache') and 'ccache '}" +CCACHE ??= "" TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}" -export CCACHE_DIR = "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}" export CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}" export CXX = "${CCACHE}${HOST_PREFIX}g++ ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}" export F77 = "${CCACHE}${HOST_PREFIX}g77 ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}" @@ -746,7 +745,7 @@ BB_CONSOLELOG ?= "${TMPDIR}/cooker.log.${DATETIME}" # Setup our default hash policy BB_SIGNATURE_HANDLER ?= "OEBasic" -BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR EXTERNAL_TOOLCHAIN" +BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE" BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SESSION_MANAGER DBUS_SESSION_BUS_ADDRESS SSH_AGENT_PID XDG_SESSION_COOKIE SSH_AUTH_SOCK XAUTHORITY" MLPREFIX ??= ""
Currently, ccache is used if it is present. When building from scratch it gives no performance improvement and creates a ton of empty directories even when its not in use. This change moves ccache support to a bbclass file which the user can choose to enable. This should make builds more determinstic and make it easier/clearer to the end user when its being used and when it is not. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> ---