Patchwork gitpkgv.bbclass: cache GITPKGV result

login
register
mail settings
Submitter Enrico Scholz
Date Nov. 16, 2012, 4:19 p.m.
Message ID <1353082752-8187-1-git-send-email-enrico.scholz@sigma-chemnitz.de>
Download mbox | patch
Permalink /patch/39159/
State Superseded
Headers show

Comments

Enrico Scholz - Nov. 16, 2012, 4:19 p.m.
gitpkgv runs the 'git rev-list | wc -l' several times when processing a
package using GITPKGV.  This takes ages for packages like the linux
kernel which has a) a large repository and b) lots of subpackages.

This patch caches the result of 'git rev-list' into the sources cache
directory and uses it on the next run.  Because collisions of the sha1
hash are very unlikely, the git revision is used directly as the key.

[this is a slightly fixed version of classic OE's
 8c61580bc303911cf77a16eefb661414fee637f4]

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta-oe/classes/gitpkgv.bbclass | 34 ++++++++++++++++++++++++++++------
 1 file changed, 28 insertions(+), 6 deletions(-)
Otavio Salvador - Nov. 16, 2012, 5:28 p.m.
On Fri, Nov 16, 2012 at 2:19 PM, Enrico Scholz <
enrico.scholz@sigma-chemnitz.de> wrote:

> gitpkgv runs the 'git rev-list | wc -l' several times when processing a
> package using GITPKGV.  This takes ages for packages like the linux
> kernel which has a) a large repository and b) lots of subpackages.
>
> This patch caches the result of 'git rev-list' into the sources cache
> directory and uses it on the next run.  Because collisions of the sha1
> hash are very unlikely, the git revision is used directly as the key.
>
> [this is a slightly fixed version of classic OE's
>  8c61580bc303911cf77a16eefb661414fee637f4]
>
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
>

I'd prefer to have the hashes cache in same DL_DIR so we don't have another
directory there. It is a file so I see no need for another dir. What do you
think?
Enrico Scholz - Nov. 16, 2012, 5:43 p.m.
Otavio Salvador <otavio@ossystems.com.br> writes:

>> This patch caches the result of 'git rev-list' into the sources cache
>> directory and uses it on the next run.  Because collisions of the sha1
>> hash are very unlikely, the git revision is used directly as the key.
> ...
> I'd prefer to have the hashes cache in same DL_DIR so we don't have another
> directory there. It is a file so I see no need for another dir. What do you
> think?

good idea; I will submit an updated patch soon.


Enrico

Patch

diff --git a/meta-oe/classes/gitpkgv.bbclass b/meta-oe/classes/gitpkgv.bbclass
index 165ba1e..3819eff 100644
--- a/meta-oe/classes/gitpkgv.bbclass
+++ b/meta-oe/classes/gitpkgv.bbclass
@@ -50,11 +50,15 @@  def gitpkgv_drop_tag_prefix(version):
 def get_git_pkgv(d, use_tags):
     import os
     import bb
+    from pipes import quote
 
     src_uri = d.getVar('SRC_URI', 1).split()
+    cachedir = d.expand("${DL_DIR}/gitpkgv", True)
     fetcher = bb.fetch2.Fetch(src_uri, d)
     ud = fetcher.ud
 
+    bb.utils.mkdirhier(cachedir)
+
     #
     # If SRCREV_FORMAT is set respect it for tags
     #
@@ -71,22 +75,40 @@  def get_git_pkgv(d, use_tags):
 
                 found = True
 
-                cwd = os.getcwd()
-                os.chdir(url.localpath)
+                vars = { 'repodir' : quote(url.localpath),
+                         'rev' : quote(rev),
+                       }
+
+                rev = bb.fetch2.get_srcrev(d).split('+')[1]
+                rev_file = os.path.join(cachedir, rev)
 
-                commits = bb.fetch2.runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % rev, d, quiet=True).strip()
+                if not os.path.exists(rev_file) or os.path.getsize(rev_file)==0:
+                    commits = bb.fetch2.runfetchcmd(
+                        "cd %(repodir)s && "
+                        "git rev-list %(rev)s -- 2> /dev/null "
+                        "| wc -l" % vars,
+                        d, quiet=True).strip().lstrip('0')
+
+                    if commits != "":
+                        oe.path.remove(rev_file, recurse=False)
+                        open(rev_file, "w").write("%d\n" % int(commits))
+                    else:
+                        commits = "0"
+                else:
+                    commits = open(rev_file, "r").readline(128).strip()
 
                 if use_tags:
                     try:
-                        output = bb.fetch2.runfetchcmd("git describe %s 2>/dev/null" % rev, d, quiet=True).strip()
+                        output = bb.fetch2.runfetchcmd(
+                            "cd %(repodir)s && "
+                            "git describe %(rev)s 2>/dev/null" % vars,
+                            d, quiet=True).strip()
                         ver = gitpkgv_drop_tag_prefix(output)
                     except Exception:
                         ver = "0.0-%s-g%s" % (commits, rev[:7])
                 else:
                     ver = "%s+%s" % (commits, rev[:7])
 
-                os.chdir(cwd)
-
                 format = format.replace(name, ver)
 
     if found: