[1/1,RFC/RFT] sstate: Shorten file names in sstate-cache

Submitted by Paul Barker on Jan. 11, 2019, 4:44 p.m. | Patch ID: 157800

Details

Message ID 20190111164438.19749-2-paul@betafive.co.uk
State New
Headers show

Commit Message

Paul Barker Jan. 11, 2019, 4:44 p.m.
With the recent move to using sha256 checksums in sstate, the lengths of
some file names can now exceed the 255 char limit seen in most Linux
filesystems. When this occurs, bitbake crashes as it is unable to create
the sstate archive or sigfile.

To avoid this issue we need to shorten the length of the file names used
in the sstate cache. We can remove the 'sstate:' prefix as these files
already reside in a directory which only contains sstate files. We can
also remove the target triplet (set by the expansion of
${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}) as we also have the
package arch later via the expansion of ${SSTATE_PKGARCH}. And for
simplicity we can replace the expansion of ${PN}, ${PV} and ${PR} with
${PF}.

With the above changes we may still get bitten by packages with
exceptionally long names and/or version strings. To avoid this we limit
the length of the expansions of ${PF} to 100 chars and ${SSTATE_PKGARCH}
to 32 chars. Allowing a further 64 chars for the sha256 hash this leaves
about 50 chars for the task name and extensions (in the worst case)
which should be sufficient.

Signed-off-by: Paul Barker <paul@betafive.co.uk>
---
 meta/classes/sstate.bbclass | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

Patch hide | download patch | download mbox

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index da0807d6e9..987218afaf 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -8,9 +8,12 @@  def generate_sstatefn(spec, hash, d):
         hash = "INVALID"
     return hash[:2] + "/" + spec + hash
 
+def limit_sstate_frag(frag, limit, d):
+    return frag[:limit]
+
 SSTATE_PKGARCH    = "${PACKAGE_ARCH}"
-SSTATE_PKGSPEC    = "sstate:${PN}:${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}:${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:"
-SSTATE_SWSPEC     = "sstate:${PN}::${PV}:${PR}::${SSTATE_VERSION}:"
+SSTATE_PKGSPEC    = "${@limit_sstate_frag(d.getVar('PF'), 100, d)}:${@limit_sstate_frag(d.getVar('SSTATE_PKGARCH'), 32, d)}:${SSTATE_VERSION}:"
+SSTATE_SWSPEC     = "${@limit_sstate_frag(d.getVar('PF'), 100, d)}::${SSTATE_VERSION}:"
 SSTATE_PKGNAME    = "${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC'), d.getVar('BB_UNIHASH'), d)}"
 SSTATE_PKG        = "${SSTATE_DIR}/${SSTATE_PKGNAME}"
 SSTATE_EXTRAPATH   = ""

Comments

Richard Purdie Jan. 12, 2019, 4:20 p.m.
Hi Paul,

On Fri, 2019-01-11 at 16:44 +0000, Paul Barker wrote:
> With the recent move to using sha256 checksums in sstate, the lengths
> of some file names can now exceed the 255 char limit seen in most
> Linux filesystems. When this occurs, bitbake crashes as it is unable
> to create the sstate archive or sigfile. 

> To avoid this issue we need to shorten the length of the file names
> used in the sstate cache. We can remove the 'sstate:' prefix as these
> files already reside in a directory which only contains sstate files.
> We can also remove the target triplet (set by the expansion of
> ${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}) as we also have the
> package arch later via the expansion of ${SSTATE_PKGARCH}. And for
> simplicity we can replace the expansion of ${PN}, ${PV} and ${PR}
> with ${PF}.

Thanks for looking at this. Firstly, could you split this patch up?
You're making multiple changes in one patch and I'd really like to make
it clear what we're changing here.

You've removed the sstate prefix. I'm torn on that since it does tell
you what the file is when looked at in isolation so I've mixed
feelings. I guess it was insurance in case we ever place other files in
the sstate directory.

In SSTATE_SWSPEC the expression is: "${PN}::${PV}:${PR}" and we need
to be very clear this is not equal to ${PF} (which is ${PN}-
${EXTENDPE}${PV}-${PR}). Its a different set of fields and a different
set of delimiters.

The delimiter was specifically chosen to be ":" so that we could stand
a chance of separating the data back out by machine. PN/PV/PR can
contain "-" which makes it a bad choice of delimiter and makes it hard
to delete " all the sstate for perl-native" or similar.

I agree that the duplication of PACKAGE_ARCH in there is probably
unneeded and can likely remove it, the ${TARGET_VENDOR}-${TARGET_OS}
piece may be useful to people with multiple distro configs. These may
be better captured in SSTATE_PKGARCH, not sure but it is a loss of
information.

I think we need to make a conscious decision on each of these changes
as well as length limitation changes.

Cheers,

Richard