Patchwork [bitbake-devel,4/4] fetch2/git.py: Optimize clone fall back when it is local

login
mail settings
Submitter Jason Wessel
Date June 14, 2012, 1:33 p.m.
Message ID <4FD9E81D.6030808@windriver.com>
Download mbox | patch
Permalink /patch/29837/
State New
Headers show

Comments

Jason Wessel - June 14, 2012, 1:33 p.m.
On 06/14/2012 08:03 AM, Richard Purdie wrote:
> On Wed, 2012-06-13 at 22:09 -0500, Jason Wessel wrote:
>> A file:// url should use "clone -s" to greatly speed
>> up the clone in the case of a kernel when it is local.
>>
>> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
>> ---
>>  lib/bb/fetch2/git.py |    8 +++++++-
>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
>> index 1ad9213..f5a3983 100644
>> --- a/lib/bb/fetch2/git.py
>> +++ b/lib/bb/fetch2/git.py
>> @@ -192,7 +192,13 @@ class Git(FetchMethod):
>>  
>>          # If the repo still doesn't exist, fallback to cloning it
>>          if not os.path.exists(ud.clonedir):
>> -            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
>> +            if repourl.startswith("file://"):
>> +                use_s = "-s"
>> +                repo = repourl[7:]
>> +            else:
>> +                use_s = ""
>> +                repo = repourl
>> +            clone_cmd = "%s clone %s --bare --mirror %s %s" % (ud.basecmd, use_s, repo, ud.clonedir)
>>              if ud.proto.lower() != 'file':
>>                  bb.fetch2.check_network_access(d, clone_cmd)
>>              runfetchcmd(clone_cmd, d)
> 
> How about always using the -l option instead?
> 

I just had to put it through some testing first.  Yes I agree it is fine to use -l as well.

Attached is the new version of the patch.

Jason.
Richard Purdie - June 14, 2012, 2:26 p.m.
On Thu, 2012-06-14 at 08:33 -0500, Jason Wessel wrote:
> On 06/14/2012 08:03 AM, Richard Purdie wrote:
> > On Wed, 2012-06-13 at 22:09 -0500, Jason Wessel wrote:
> >> A file:// url should use "clone -s" to greatly speed
> >> up the clone in the case of a kernel when it is local.
> >>
> >> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> >> ---
> >>  lib/bb/fetch2/git.py |    8 +++++++-
> >>  1 file changed, 7 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> >> index 1ad9213..f5a3983 100644
> >> --- a/lib/bb/fetch2/git.py
> >> +++ b/lib/bb/fetch2/git.py
> >> @@ -192,7 +192,13 @@ class Git(FetchMethod):
> >>  
> >>          # If the repo still doesn't exist, fallback to cloning it
> >>          if not os.path.exists(ud.clonedir):
> >> -            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
> >> +            if repourl.startswith("file://"):
> >> +                use_s = "-s"
> >> +                repo = repourl[7:]
> >> +            else:
> >> +                use_s = ""
> >> +                repo = repourl
> >> +            clone_cmd = "%s clone %s --bare --mirror %s %s" % (ud.basecmd, use_s, repo, ud.clonedir)
> >>              if ud.proto.lower() != 'file':
> >>                  bb.fetch2.check_network_access(d, clone_cmd)
> >>              runfetchcmd(clone_cmd, d)
> > 

I think you can then simplify this to:

@@ -192,7 +192,9 @@ class Git(FetchMethod):
  
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
-            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
+            if repourl.startswith("file://"):
+                repo = repourl[7:]
+            clone_cmd = "%s clone -l --bare --mirror %s %s" % (ud.basecmd, repo, ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd)
             runfetchcmd(clone_cmd, d)
 
Cheers,

Richard
Jason Wessel - June 14, 2012, 2:31 p.m.
On 06/14/2012 09:26 AM, Richard Purdie wrote:
> On Thu, 2012-06-14 at 08:33 -0500, Jason Wessel wrote:
>> On 06/14/2012 08:03 AM, Richard Purdie wrote:
>>> On Wed, 2012-06-13 at 22:09 -0500, Jason Wessel wrote:
>>>> A file:// url should use "clone -s" to greatly speed
>>>> up the clone in the case of a kernel when it is local.
>>>>
>>>> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
>>>> ---
>>>>  lib/bb/fetch2/git.py |    8 +++++++-
>>>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
>>>> index 1ad9213..f5a3983 100644
>>>> --- a/lib/bb/fetch2/git.py
>>>> +++ b/lib/bb/fetch2/git.py
>>>> @@ -192,7 +192,13 @@ class Git(FetchMethod):
>>>>  
>>>>          # If the repo still doesn't exist, fallback to cloning it
>>>>          if not os.path.exists(ud.clonedir):
>>>> -            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
>>>> +            if repourl.startswith("file://"):
>>>> +                use_s = "-s"
>>>> +                repo = repourl[7:]
>>>> +            else:
>>>> +                use_s = ""
>>>> +                repo = repourl
>>>> +            clone_cmd = "%s clone %s --bare --mirror %s %s" % (ud.basecmd, use_s, repo, ud.clonedir)
>>>>              if ud.proto.lower() != 'file':
>>>>                  bb.fetch2.check_network_access(d, clone_cmd)
>>>>              runfetchcmd(clone_cmd, d)
> I think you can then simplify this to:
>
> @@ -192,7 +192,9 @@ class Git(FetchMethod):
>   
>          # If the repo still doesn't exist, fallback to cloning it
>          if not os.path.exists(ud.clonedir):
> -            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
> +            if repourl.startswith("file://"):
> +                repo = repourl[7:]
> +            clone_cmd = "%s clone -l --bare --mirror %s %s" % (ud.basecmd, repo, ud.clonedir)
>              if ud.proto.lower() != 'file':
>                  bb.fetch2.check_network_access(d, clone_cmd)
>              runfetchcmd(clone_cmd, d)
>  
>

That is fine with me, given that it does the same thing.

Jason.
Jason Wessel - June 14, 2012, 9:11 p.m.
On 06/14/2012 09:26 AM, Richard Purdie wrote:
> On Thu, 2012-06-14 at 08:33 -0500, Jason Wessel wrote:
>> On 06/14/2012 08:03 AM, Richard Purdie wrote:
>>> On Wed, 2012-06-13 at 22:09 -0500, Jason Wessel wrote:
>>>> A file:// url should use "clone -s" to greatly speed
>>>> up the clone in the case of a kernel when it is local.
>>>>
>>>> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
>>>> ---
>>>>  lib/bb/fetch2/git.py |    8 +++++++-
>>>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
>>>> index 1ad9213..f5a3983 100644
>>>> --- a/lib/bb/fetch2/git.py
>>>> +++ b/lib/bb/fetch2/git.py
>>>> @@ -192,7 +192,13 @@ class Git(FetchMethod):
>>>>  
>>>>          # If the repo still doesn't exist, fallback to cloning it
>>>>          if not os.path.exists(ud.clonedir):
>>>> -            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
>>>> +            if repourl.startswith("file://"):
>>>> +                use_s = "-s"
>>>> +                repo = repourl[7:]
>>>> +            else:
>>>> +                use_s = ""
>>>> +                repo = repourl
>>>> +            clone_cmd = "%s clone %s --bare --mirror %s %s" % (ud.basecmd, use_s, repo, ud.clonedir)
>>>>              if ud.proto.lower() != 'file':
>>>>                  bb.fetch2.check_network_access(d, clone_cmd)
>>>>              runfetchcmd(clone_cmd, d)
> I think you can then simplify this to:
>
> @@ -192,7 +192,9 @@ class Git(FetchMethod):
>   
>          # If the repo still doesn't exist, fallback to cloning it
>          if not os.path.exists(ud.clonedir):
> -            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
> +            if repourl.startswith("file://"):
> +                repo = repourl[7:]
> +            clone_cmd = "%s clone -l --bare --mirror %s %s" % (ud.basecmd, repo, ud.clonedir)
>              if ud.proto.lower() != 'file':
>                  bb.fetch2.check_network_access(d, clone_cmd)
>              runfetchcmd(clone_cmd, d)

This had one defect in it which I didn't notice until I tested it  "repo" should have been "repourl", so I'll send new version along with a respin of the local git:/// -> file:/// PREMIRROR patch.

Many thanks for your comments.

Cheers,
Jason.

Patch

From daa5cc78b294406fd2376697d44c4de1a42d6f34 Mon Sep 17 00:00:00 2001
From: Jason Wessel <jason.wessel@windriver.com>
Date: Tue, 12 Jun 2012 13:23:34 -0500
Subject: [PATCH] fetch2/git.py: Optimize clone fall back when it is local

A file:// url should use "clone -l" to greatly speed
up the clone in the case of a kernel when it is local.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/fetch2/git.py |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index ecc5e0d..d6a08ac 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -188,7 +188,13 @@  class Git(FetchMethod):
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
-            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
+            if repourl.startswith("file://"):
+                use_s = "-l"
+                repo = repourl[7:]
+            else:
+                use_s = ""
+                repo = repourl
+            clone_cmd = "%s clone %s --bare --mirror %s %s" % (ud.basecmd, use_s, repo, ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd)
             runfetchcmd(clone_cmd, d)
-- 
1.7.10