[bitbake-devel,v2] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories

Submitted by Aníbal Limón on Nov. 15, 2019, 2:47 p.m. | Patch ID: 167006

Details

Message ID 20191115144723.7452-1-anibal.limon@linaro.org
State New
Headers show

Commit Message

Aníbal Limón Nov. 15, 2019, 2:47 p.m.
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.

Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"

Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
---
 lib/bb/checksum.py        | 5 +++--
 lib/bb/fetch2/__init__.py | 4 ++--
 lib/bb/siggen.py          | 5 +++--
 3 files changed, 8 insertions(+), 6 deletions(-)

Patch hide | download patch | download mbox

diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
index 5bc8a8fc..677020f4 100644
--- a/lib/bb/checksum.py
+++ b/lib/bb/checksum.py
@@ -74,7 +74,7 @@  class FileChecksumCache(MultiProcessCache):
             else:
                 dest[0][h] = source[0][h]
 
-    def get_checksums(self, filelist, pn):
+    def get_checksums(self, filelist, pn, localdirsexclude):
         """Get checksums for a list of files"""
 
         def checksum_file(f):
@@ -90,7 +90,8 @@  class FileChecksumCache(MultiProcessCache):
             if pth == "/":
                 bb.fatal("Refusing to checksum /")
             dirchecksums = []
-            for root, dirs, files in os.walk(pth):
+            for root, dirs, files in os.walk(pth, topdown=True):
+                [dirs.remove(d) for d in list(dirs) if d in localdirsexclude]
                 for name in files:
                     fullpth = os.path.join(root, name)
                     checksum = checksum_file(fullpth)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 07de6c26..731c1608 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1197,14 +1197,14 @@  def get_checksum_file_list(d):
 
     return " ".join(filelist)
 
-def get_file_checksums(filelist, pn):
+def get_file_checksums(filelist, pn, localdirsexclude):
     """Get a list of the checksums for a list of local files
 
     Returns the checksums for a list of local files, caching the results as
     it proceeds
 
     """
-    return _checksum_cache.get_checksums(filelist, pn)
+    return _checksum_cache.get_checksums(filelist, pn, localdirsexclude)
 
 
 class FetchData(object):
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index a4bb1ff7..2445a340 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -123,6 +123,7 @@  class SignatureGeneratorBasic(SignatureGenerator):
 
         self.unihash_cache = bb.cache.SimpleCache("1")
         self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {})
+        self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc .p4 .repo .svn").split()
 
     def init_rundepcheck(self, data):
         self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None
@@ -221,9 +222,9 @@  class SignatureGeneratorBasic(SignatureGenerator):
 
         if task in dataCache.file_checksums[fn]:
             if self.checksum_cache:
-                checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename)
+                checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
             else:
-                checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
+                checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
             for (f,cs) in checksums:
                 self.file_checksum_values[tid].append((f,cs))
                 if cs:

Comments

Peter Kjellerstedt Nov. 15, 2019, 4:29 p.m.
> -----Original Message-----

> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-

> bounces@lists.openembedded.org> On Behalf Of Aníbal Limón

> Sent: den 15 november 2019 15:47

> To: bitbake-devel@lists.openembedded.org

> Subject: [bitbake-devel] [PATCHv2] lib/bb: Add

> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories

> 

> The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list

> of directories to exclude when making taskhash, our specific case

> is using SRC_URI that points local VCS directory.

> 

> Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo

> .svn"

> 

> Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>

> ---

>  lib/bb/checksum.py        | 5 +++--

>  lib/bb/fetch2/__init__.py | 4 ++--

>  lib/bb/siggen.py          | 5 +++--

>  3 files changed, 8 insertions(+), 6 deletions(-)


[cut]

> --- a/lib/bb/siggen.py

> +++ b/lib/bb/siggen.py

> @@ -123,6 +123,7 @@ class SignatureGeneratorBasic(SignatureGenerator):

> 

>          self.unihash_cache = bb.cache.SimpleCache("1")

>          self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {})

> +        self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc .p4 .repo .svn").split()


I don't like this construct, because if I do:

BB_SIGNATURE_LOCAL_DIRS_EXCLUDE_append = " some_dir"

it means that all of a sudden the SCM directories listed above will no 
longer be excluded since BB_SIGNATURE_LOCAL_DIRS_EXCLUDE will now be 
defined, but only contain " some_dir". Isn't it better to add:

BB_SIGNATURE_LOCAL_DIRS_EXCLUDE ??= "CVS .bzr .git .hg .osc .p4 .repo .svn"

somewhere in OE-Core (and it can then also have a suitable 
BB_SIGNATURE_LOCAL_DIRS_EXCLUDE[doc], though it seems BB_-variables 
are only documented in meta/conf/documentation.conf)?

Alternatively, to keep the list above within bitbake, make it two 
variables, one for the list of directories above that is not really 
intended to be modified, and one for additional directories (which 
should still be documented in meta/conf/documentation.conf).

//Peter
Mark Hatle Nov. 15, 2019, 5:12 p.m.
On 11/15/19 10:29 AM, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>> bounces@lists.openembedded.org> On Behalf Of Aníbal Limón
>> Sent: den 15 november 2019 15:47
>> To: bitbake-devel@lists.openembedded.org
>> Subject: [bitbake-devel] [PATCHv2] lib/bb: Add
>> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
>>
>> The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
>> of directories to exclude when making taskhash, our specific case
>> is using SRC_URI that points local VCS directory.
>>
>> Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo
>> .svn"
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
>> ---
>>  lib/bb/checksum.py        | 5 +++--
>>  lib/bb/fetch2/__init__.py | 4 ++--
>>  lib/bb/siggen.py          | 5 +++--
>>  3 files changed, 8 insertions(+), 6 deletions(-)
> 
> [cut]
> 
>> --- a/lib/bb/siggen.py
>> +++ b/lib/bb/siggen.py
>> @@ -123,6 +123,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>>
>>          self.unihash_cache = bb.cache.SimpleCache("1")
>>          self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {})
>> +        self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc .p4 .repo .svn").split()
> 
> I don't like this construct, because if I do:
> 
> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE_append = " some_dir"
> 
> it means that all of a sudden the SCM directories listed above will no 
> longer be excluded since BB_SIGNATURE_LOCAL_DIRS_EXCLUDE will now be 
> defined, but only contain " some_dir". Isn't it better to add:
> 
> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE ??= "CVS .bzr .git .hg .osc .p4 .repo .svn"

I agree.. this really should be put into the bitbake.conf file in OE-core.  (I'd
also suggest adding it to the bitbake.conf file in bitbake itself, as an example.)

--Mark

> somewhere in OE-Core (and it can then also have a suitable 
> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE[doc], though it seems BB_-variables 
> are only documented in meta/conf/documentation.conf)?
> 
> Alternatively, to keep the list above within bitbake, make it two 
> variables, one for the list of directories above that is not really 
> intended to be modified, and one for additional directories (which 
> should still be documented in meta/conf/documentation.conf).
> 
> //Peter
>
Aníbal Limón Nov. 18, 2019, 3:22 p.m.
On Fri, 15 Nov 2019 at 11:12, Mark Hatle <mark.hatle@kernel.crashing.org>
wrote:

>
>
> On 11/15/19 10:29 AM, Peter Kjellerstedt wrote:
> >> -----Original Message-----
> >> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
> >> bounces@lists.openembedded.org> On Behalf Of Aníbal Limón
> >> Sent: den 15 november 2019 15:47
> >> To: bitbake-devel@lists.openembedded.org
> >> Subject: [bitbake-devel] [PATCHv2] lib/bb: Add
> >> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
> >>
> >> The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
> >> of directories to exclude when making taskhash, our specific case
> >> is using SRC_URI that points local VCS directory.
> >>
> >> Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo
> >> .svn"
> >>
> >> Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
> >> ---
> >>  lib/bb/checksum.py        | 5 +++--
> >>  lib/bb/fetch2/__init__.py | 4 ++--
> >>  lib/bb/siggen.py          | 5 +++--
> >>  3 files changed, 8 insertions(+), 6 deletions(-)
> >
> > [cut]
> >
> >> --- a/lib/bb/siggen.py
> >> +++ b/lib/bb/siggen.py
> >> @@ -123,6 +123,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> >>
> >>          self.unihash_cache = bb.cache.SimpleCache("1")
> >>          self.unitaskhashes = self.unihash_cache.init_cache(data,
> "bb_unihashes.dat", {})
> >> +        self.localdirsexclude =
> (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc
> .p4 .repo .svn").split()
> >
> > I don't like this construct, because if I do:
> >
> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE_append = " some_dir"
> >
> > it means that all of a sudden the SCM directories listed above will no
> > longer be excluded since BB_SIGNATURE_LOCAL_DIRS_EXCLUDE will now be
> > defined, but only contain " some_dir". Isn't it better to add:
> >
> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE ??= "CVS .bzr .git .hg .osc .p4 .repo
> .svn"
>
> I agree.. this really should be put into the bitbake.conf file in
> OE-core.  (I'd
> also suggest adding it to the bitbake.conf file in bitbake itself, as an
> example.)
>
> --Mark
>
> > somewhere in OE-Core (and it can then also have a suitable
> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE[doc], though it seems BB_-variables
> > are only documented in meta/conf/documentation.conf)?
> >
> > Alternatively, to keep the list above within bitbake, make it two
> > variables, one for the list of directories above that is not really
> > intended to be modified, and one for additional directories (which
> > should still be documented in meta/conf/documentation.conf).
>

For me it is fine if we set this variable in OE-Core with a default value
to VCS directories this is covered by v1 PATCH, after merge I will send
a patch to OE-Core adding the default value.

Regards,
Anibal


> >
> > //Peter
> >
>
Aníbal Limón Dec. 16, 2019, 6:55 p.m.
Hi, any update?

Regards,
Anibal

El lun., 18 de nov. de 2019 09:22, Anibal Limon <anibal.limon@linaro.org>
escribió:

>
>
> On Fri, 15 Nov 2019 at 11:12, Mark Hatle <mark.hatle@kernel.crashing.org>
> wrote:
>
>>
>>
>> On 11/15/19 10:29 AM, Peter Kjellerstedt wrote:
>> >> -----Original Message-----
>> >> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>> >> bounces@lists.openembedded.org> On Behalf Of Aníbal Limón
>> >> Sent: den 15 november 2019 15:47
>> >> To: bitbake-devel@lists.openembedded.org
>> >> Subject: [bitbake-devel] [PATCHv2] lib/bb: Add
>> >> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
>> >>
>> >> The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
>> >> of directories to exclude when making taskhash, our specific case
>> >> is using SRC_URI that points local VCS directory.
>> >>
>> >> Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4
>> .repo
>> >> .svn"
>> >>
>> >> Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
>> >> ---
>> >>  lib/bb/checksum.py        | 5 +++--
>> >>  lib/bb/fetch2/__init__.py | 4 ++--
>> >>  lib/bb/siggen.py          | 5 +++--
>> >>  3 files changed, 8 insertions(+), 6 deletions(-)
>> >
>> > [cut]
>> >
>> >> --- a/lib/bb/siggen.py
>> >> +++ b/lib/bb/siggen.py
>> >> @@ -123,6 +123,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> >>
>> >>          self.unihash_cache = bb.cache.SimpleCache("1")
>> >>          self.unitaskhashes = self.unihash_cache.init_cache(data,
>> "bb_unihashes.dat", {})
>> >> +        self.localdirsexclude =
>> (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc
>> .p4 .repo .svn").split()
>> >
>> > I don't like this construct, because if I do:
>> >
>> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE_append = " some_dir"
>> >
>> > it means that all of a sudden the SCM directories listed above will no
>> > longer be excluded since BB_SIGNATURE_LOCAL_DIRS_EXCLUDE will now be
>> > defined, but only contain " some_dir". Isn't it better to add:
>> >
>> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE ??= "CVS .bzr .git .hg .osc .p4 .repo
>> .svn"
>>
>> I agree.. this really should be put into the bitbake.conf file in
>> OE-core.  (I'd
>> also suggest adding it to the bitbake.conf file in bitbake itself, as an
>> example.)
>>
>> --Mark
>>
>> > somewhere in OE-Core (and it can then also have a suitable
>> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE[doc], though it seems BB_-variables
>> > are only documented in meta/conf/documentation.conf)?
>> >
>> > Alternatively, to keep the list above within bitbake, make it two
>> > variables, one for the list of directories above that is not really
>> > intended to be modified, and one for additional directories (which
>> > should still be documented in meta/conf/documentation.conf).
>>
>
> For me it is fine if we set this variable in OE-Core with a default value
> to VCS directories this is covered by v1 PATCH, after merge I will send
> a patch to OE-Core adding the default value.
>
> Regards,
> Anibal
>
>
>> >
>> > //Peter
>> >
>>
>