[1/3] gitsm.py: use download cache for git submodules too

Submitted by Mikko Rapeli on Sept. 6, 2017, 11:08 a.m. | Patch ID: 143711

Details

Message ID 1504696096-13095-1-git-send-email-mikko.rapeli@bmw.de
State New
Headers show

Commit Message

Mikko Rapeli Sept. 6, 2017, 11:08 a.m.
Currently download caches for recipes using gitsm are filled also with
the submodule repositories but when the git submodule tree is reused from
cache, only the base repository is cloned. After a source URL rewrite,
'git submodule update --init --recursive' is called in the tree
which downloads the needed submodules from upstream URL's.
This wastes bandwidth since download cache already had the needed
commits for the submodule repositories.

With this change, also the submodule repositories are copied to the
work space from download cache and the download from upstream can
be avoided. Using plain 'cp -a' since git cloning doesn't work for the
submodule directories.

Note that using BB_NO_NETWORK does not stop the submodule downloads
from 'git submodule update --init --recursive'. Thus tested by
breaking access to remote repository via /etc/hosts line like:

127.0.0.1       localhost github.com

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
---
 bitbake/lib/bb/fetch2/gitsm.py | 8 ++++++++
 1 file changed, 8 insertions(+)

Patch hide | download patch | download mbox

diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index 0aff100..20a9c40 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -131,5 +131,13 @@  class GitSM(Git):
         Git.unpack(self, ud, destdir, d)
 
         if self.uses_submodules(ud, d, ud.destdir):
+            # Copy also submodule trees from download cache instead of
+            # downloading again from the upstream repository.
+            # For some reason git does not clone them.
+            clone_modules = os.path.join(ud.clonedir, "modules")
+            if os.path.exists(clone_modules):
+                dest_modules = os.path.join(ud.destdir, ".git")
+                runfetchcmd("cp -a " + clone_modules + " " + dest_modules, d)
+
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
             runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)

Comments

Mikko Rapeli Sept. 6, 2017, 12:14 p.m.
On Wed, Sep 06, 2017 at 11:34:54AM +0000, Patchwork wrote:
> == Series Details ==
> 
> Series: "gitsm.py: use download cache f..." and 2 more
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/8734/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
>   Suggested fix    Rebase your series on top of targeted branch
>   Targeted branch  master (currently at cc319b6dcc)

Uh, the patch is based on poky master at:

commit 8b4f16a9cbbaf521461f699b7264fac2ac872581 (refs/remotes/upstream/master-ne
Author: Jussi Kukkonen <jussi.kukkonen@intel.com>
Date:   Mon Sep 4 11:39:24 2017 +0300

    mesa-gl: Fix build after recent mesa PACKAGECONFIG changes
    
    48d39cf43b added "opengl" PACKAGECONFIG option to mesa: before that
    the configuration was always enabled. "opengl" should have been added
    to mesa-gl default PACKAGECONFIG but wasn't: do it now.
    
    (From OE-Core rev: cc319b6dcc5b4a5019fb91c9771b12ce17f3c953)
    
    Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

and there are no newer changes available. Should I port the patch to oe-core
directly and submit again?

Maybe this worked in the past because someone else did the applying and
merging manually?

-Mikko
Mikko Rapeli Sept. 6, 2017, 12:22 p.m.
On Wed, Sep 06, 2017 at 12:14:48PM +0000, Mikko.Rapeli@bmw.de wrote:
> On Wed, Sep 06, 2017 at 11:34:54AM +0000, Patchwork wrote:
> > == Series Details ==
> > 
> > Series: "gitsm.py: use download cache f..." and 2 more
> > Revision: 1
> > URL   : https://patchwork.openembedded.org/series/8734/
> > State : failure
> > 
> > == Summary ==
> > 
> > 
> > Thank you for submitting this patch series to OpenEmbedded Core. This is
> > an automated response. Several tests have been executed on the proposed
> > series by patchtest resulting in the following failures:
> > 
> > 
> > 
> > * Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
> >   Suggested fix    Rebase your series on top of targeted branch
> >   Targeted branch  master (currently at cc319b6dcc)
> 
> Uh, the patch is based on poky master at:
> 
> commit 8b4f16a9cbbaf521461f699b7264fac2ac872581 (refs/remotes/upstream/master-ne
> Author: Jussi Kukkonen <jussi.kukkonen@intel.com>
> Date:   Mon Sep 4 11:39:24 2017 +0300
> 
>     mesa-gl: Fix build after recent mesa PACKAGECONFIG changes
>     
>     48d39cf43b added "opengl" PACKAGECONFIG option to mesa: before that
>     the configuration was always enabled. "opengl" should have been added
>     to mesa-gl default PACKAGECONFIG but wasn't: do it now.
>     
>     (From OE-Core rev: cc319b6dcc5b4a5019fb91c9771b12ce17f3c953)
>     
>     Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
>     Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> and there are no newer changes available. Should I port the patch to oe-core
> directly and submit again?
> 
> Maybe this worked in the past because someone else did the applying and
> merging manually?

Ah, sorry. The patch is for bitbake, not oe-core. Will send to correct list.

-Mikko