Patchwork [bitbake-devel,PATCHv3] git: remove symling before updating mirror tarball

login
register
mail settings
Submitter Martin Jansa
Date May 7, 2013, 4:39 p.m.
Message ID <1367944760-16756-1-git-send-email-Martin.Jansa@gmail.com>
Download mbox | patch
Permalink /patch/49487/
State Accepted
Commit 3627b02f77c78beedadadd77c619b9e5edaae076
Headers show

Comments

Martin Jansa - May 7, 2013, 4:39 p.m.
* with read-only PREMIRROR (e.g. mounted over NFS or CIFS
  and referenced as file:///mnt/premirror) we cannot use
  BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
  files later became just symlinks to read-only location in PREMIRROR
  (it works fine on first build and for new components, because
  at that time there isn't tarball on PREMIRROR yet).

  ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
  tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
  tar (child): Error is not recoverable: exiting now

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 lib/bb/fetch2/git.py | 4 ++++
 1 file changed, 4 insertions(+)
Martin Jansa - May 7, 2013, 9:51 p.m.
On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote:
> * with read-only PREMIRROR (e.g. mounted over NFS or CIFS
>   and referenced as file:///mnt/premirror) we cannot use
>   BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
>   files later became just symlinks to read-only location in PREMIRROR
>   (it works fine on first build and for new components, because
>   at that time there isn't tarball on PREMIRROR yet).
> 
>   ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
>   tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
>   tar (child): Error is not recoverable: exiting now

This one is tested and I would like to get it also in 1.16 and 1.18

Steps to reproduce the issue (without redownloading anything big)

# mv downloads downloads-ro
# mkdir downloads
# mkdir tmp-eglibc/downloads
# sudo mount -o ro,bind downloads-ro tmp-eglibc/downloads
# sudo mount -o remount,ro tmp-eglibc/downloads

conf/local.conf:
SOURCE_MIRROR_URL = "file:///OE/oe-core/tmp-eglibc/downloads"
INHERIT += "own-mirrors"

# bitbake -c fetch qtbase
# ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz 
  should be a symlink

bump SRCREV in qtbase_git.bb to some revision not included in
git2_qt.gitorious.org.qt.qtbase.git.tar.gz

# bitbake -c fetch qtbase
WARNING: Failed to fetch URL git://qt.gitorious.org/qt/qtbase.git;branch=stable, attempting MIRRORS if available
DEBUG: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /OE/oe-core/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz: Cannot open: Read-only file system
tar (child): Error is not recoverable: exiting now

apply this patch
# bitbake -c fetch qtbase
works fine
# ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
-rw-r--r-- 1 bitbake bitbake 176M May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
is now proper file instead of symlink

> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  lib/bb/fetch2/git.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 052802e..6175e4c 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -217,6 +217,10 @@ class Git(FetchMethod):
>      def build_mirror_data(self, url, ud, d):
>          # Generate a mirror tarball if needed
>          if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
> +            # it's possible that this symlink points to read-only filesystem with PREMIRROR
> +            if os.path.islink(ud.fullmirror):
> +                os.unlink(ud.fullmirror)
> +
>              os.chdir(ud.clonedir)
>              logger.info("Creating tarball of git repository")
>              runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
> -- 
> 1.8.2.1
>
Martin Jansa - May 7, 2013, 10:10 p.m.
On Tue, May 07, 2013 at 11:51:08PM +0200, Martin Jansa wrote:
> On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote:
> > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS
> >   and referenced as file:///mnt/premirror) we cannot use
> >   BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
> >   files later became just symlinks to read-only location in PREMIRROR
> >   (it works fine on first build and for new components, because
> >   at that time there isn't tarball on PREMIRROR yet).
> > 
> >   ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
> >   tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
> >   tar (child): Error is not recoverable: exiting now
> 
> This one is tested and I would like to get it also in 1.16 and 1.18
> 
> Steps to reproduce the issue (without redownloading anything big)
> 
> # mv downloads downloads-ro
> # mkdir downloads
> # mkdir tmp-eglibc/downloads
> # sudo mount -o ro,bind downloads-ro tmp-eglibc/downloads
> # sudo mount -o remount,ro tmp-eglibc/downloads
> 
> conf/local.conf:
> SOURCE_MIRROR_URL = "file:///OE/oe-core/tmp-eglibc/downloads"
> INHERIT += "own-mirrors"
> 
> # bitbake -c fetch qtbase
> # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz 
>   should be a symlink
> 
> bump SRCREV in qtbase_git.bb to some revision not included in
> git2_qt.gitorious.org.qt.qtbase.git.tar.gz
> 
> # bitbake -c fetch qtbase
> WARNING: Failed to fetch URL git://qt.gitorious.org/qt/qtbase.git;branch=stable, attempting MIRRORS if available
> DEBUG: Fetcher failure: Fetch command failed with exit code 141, output:
> tar (child): /OE/oe-core/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz: Cannot open: Read-only file system
> tar (child): Error is not recoverable: exiting now
> 
> apply this patch
> # bitbake -c fetch qtbase
> works fine
> # ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
> -rw-r--r-- 1 bitbake bitbake 176M May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
> is now proper file instead of symlink

There is one more interesting behavior with such setup where file://
fetcher is used in PREMIRROR

AFAIK it's completely harmless and also not influenced by this change,
but .done stamps for files downloaded from premirror are stored in
${DL_DIR}<path-to-local-premirror>/file.done, e.g.
downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done

qtbase was recreated locally thanks to this patch, that's why it has 2 .done stamps, one in
downloads, 2nd in downloads/OE/oe-core/tmp-eglibc/downloads/

qtdeclarative has only symlink and .done stamp in downloads/OE/oe-core/tmp-eglibc/downloads/

OE @ ~/oe-core $ ll downloads/*qt{base,declarative}.git*
-rw-r--r-- 1 bitbake bitbake 184015350 May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
-rw-r--r-- 1 bitbake bitbake         0 May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done
lrwxrwxrwx 1 bitbake bitbake        82 May  8 00:00 downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz -> /OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz

OE @ ~/oe-core $ ll downloads/OE/oe-core/tmp-eglibc/downloads/*qt{base,declarative}.git*
-rw-r--r-- 1 bitbake bitbake 0 May  7 23:41 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done
-rw-r--r-- 1 bitbake bitbake 0 May  8 00:00 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz.done

> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  lib/bb/fetch2/git.py | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> > index 052802e..6175e4c 100644
> > --- a/lib/bb/fetch2/git.py
> > +++ b/lib/bb/fetch2/git.py
> > @@ -217,6 +217,10 @@ class Git(FetchMethod):
> >      def build_mirror_data(self, url, ud, d):
> >          # Generate a mirror tarball if needed
> >          if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
> > +            # it's possible that this symlink points to read-only filesystem with PREMIRROR
> > +            if os.path.islink(ud.fullmirror):
> > +                os.unlink(ud.fullmirror)
> > +
> >              os.chdir(ud.clonedir)
> >              logger.info("Creating tarball of git repository")
> >              runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
> > -- 
> > 1.8.2.1
> > 
> 
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com
Martin Jansa - May 10, 2013, 9:35 a.m.
On Tue, May 07, 2013 at 11:51:08PM +0200, Martin Jansa wrote:
> On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote:
> > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS
> >   and referenced as file:///mnt/premirror) we cannot use
> >   BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
> >   files later became just symlinks to read-only location in PREMIRROR
> >   (it works fine on first build and for new components, because
> >   at that time there isn't tarball on PREMIRROR yet).
> > 
> >   ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
> >   tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
> >   tar (child): Error is not recoverable: exiting now
> 
> This one is tested and I would like to get it also in 1.16 and 1.18

cherry-picking 3627b02f77c78beedadadd77c619b9e5edaae076 works fine for
both, do you prefer to send patches with [1.16]/[1.18] in subject or
should I just wait a bit more with patch in master and then ping you
later?
Richard Purdie - May 10, 2013, 12:32 p.m.
On Fri, 2013-05-10 at 11:35 +0200, Martin Jansa wrote:
> On Tue, May 07, 2013 at 11:51:08PM +0200, Martin Jansa wrote:
> > On Tue, May 07, 2013 at 06:39:20PM +0200, Martin Jansa wrote:
> > > * with read-only PREMIRROR (e.g. mounted over NFS or CIFS
> > >   and referenced as file:///mnt/premirror) we cannot use
> > >   BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
> > >   files later became just symlinks to read-only location in PREMIRROR
> > >   (it works fine on first build and for new components, because
> > >   at that time there isn't tarball on PREMIRROR yet).
> > > 
> > >   ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
> > >   tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
> > >   tar (child): Error is not recoverable: exiting now
> > 
> > This one is tested and I would like to get it also in 1.16 and 1.18
> 
> cherry-picking 3627b02f77c78beedadadd77c619b9e5edaae076 works fine for
> both, do you prefer to send patches with [1.16]/[1.18] in subject or
> should I just wait a bit more with patch in master and then ping you
> later?

Just give this a short while, then we can backport. I don't like
instantly backporting to other releases...

Cheers,

Richard

Patch

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 052802e..6175e4c 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -217,6 +217,10 @@  class Git(FetchMethod):
     def build_mirror_data(self, url, ud, d):
         # Generate a mirror tarball if needed
         if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(ud.fullmirror)
+
             os.chdir(ud.clonedir)
             logger.info("Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)