[bitbake-devel,dunfell,1.46,1/2] bitbake: fetch/git: add support for SRC_URI containing spaces in url

Submitted by Steve Sakoman on Oct. 12, 2020, 3:07 p.m. | Patch ID: 177162

Details

Message ID 0a8a13131b2d520a405877cb9d39f3eae7d6882a.1602515077.git.steve@sakoman.com
State New
Headers show

Commit Message

Steve Sakoman Oct. 12, 2020, 3:07 p.m. 
From: Charlie Davies <charles.davies@whitetree.xyz>

Microsoft's TFS VCS system allows for spaces in a git repository url.
An example of a valid url is:

ssh://tfs-my-company.org:22/tfs/My Projects/FooBar

This commit adds support for such urls by implementing two changes.
Firstly, when bitbake makes a git command line call the url is
surrounded by quotes so that the url, regardless of spaces, is
treated as one argument. Secondly, additional parsing of various
filepath variables, which are based off of the url, are now
completed with any spaces in the url replaced with underscores.

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eb38b6f0935763f7ba19e5618f376fcae1dac41a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/fetch2/git.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

Patch hide | download patch | download mbox 

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 2ce9395f..dcecff5d 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -236,7 +236,7 @@  class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.'))
+        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_'))
         if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]
 
@@ -342,7 +342,7 @@  class Git(FetchMethod):
             # We do this since git will use a "-l" option automatically for local urls where possible
             if repourl.startswith("file://"):
                 repourl = repourl[7:]
-            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
+            clone_cmd = "LANG=C %s clone --bare --mirror \"%s\" %s --progress" % (ud.basecmd, repourl, ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
@@ -354,8 +354,8 @@  class Git(FetchMethod):
             if "origin" in output:
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
-            runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, repourl)
+            runfetchcmd("%s remote add --mirror=fetch origin \"%s\"" % (ud.basecmd, repourl), d, workdir=ud.clonedir)
+            fetch_cmd = "LANG=C %s fetch -f --progress \"%s\" refs/*:refs/*" % (ud.basecmd, repourl)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
@@ -501,7 +501,7 @@  class Git(FetchMethod):
             raise bb.fetch2.UnpackError("No up to date source found: " + "; ".join(source_error), ud.url)
 
         repourl = self._get_repo_url(ud)
-        runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir)
+        runfetchcmd("%s remote set-url origin \"%s\"" % (ud.basecmd, repourl), d, workdir=destdir)
 
         if self._contains_lfs(ud, d, destdir):
             if need_lfs and not self._find_git_lfs(d):
@@ -613,7 +613,7 @@  class Git(FetchMethod):
         d.setVar('_BB_GIT_IN_LSREMOTE', '1')
         try:
             repourl = self._get_repo_url(ud)
-            cmd = "%s ls-remote %s %s" % \
+            cmd = "%s ls-remote \"%s\" %s" % \
                 (ud.basecmd, repourl, search)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, cmd, repourl)

Comments

Enrico Scholz via lists.openembedded.org Oct. 13, 2020, 10:51 a.m. 
"Steve Sakoman" <steve@sakoman.com> writes:

> This commit adds support for such urls by implementing two changes.
> Firstly, when bitbake makes a git command line call the url is
> surrounded by quotes so that the url, regardless of spaces, is
> treated as one argument. Secondly, additional parsing of various
> filepath variables, which are based off of the url, are now
> completed with any spaces in the url replaced with underscores.
>
> -            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
> +            clone_cmd = "LANG=C %s clone --bare --mirror \"%s\" %s --progress" % (ud.basecmd, repourl, ud.clonedir)

Why do you quote it manually (and with double instead of single quote))
instead of using shlex.quote() or making the clone_cmd an array?


Enrico
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#11751): https://lists.openembedded.org/g/bitbake-devel/message/11751
Mute This Topic: https://lists.openembedded.org/mt/77462759/3617530
Group Owner: bitbake-devel+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Steve Sakoman Oct. 13, 2020, 6:20 p.m. 
On Tue, Oct 13, 2020 at 12:51 AM Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
>
> "Steve Sakoman" <steve@sakoman.com> writes:
>
> > This commit adds support for such urls by implementing two changes.
> > Firstly, when bitbake makes a git command line call the url is
> > surrounded by quotes so that the url, regardless of spaces, is
> > treated as one argument. Secondly, additional parsing of various
> > filepath variables, which are based off of the url, are now
> > completed with any spaces in the url replaced with underscores.
> >
> > -            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
> > +            clone_cmd = "LANG=C %s clone --bare --mirror \"%s\" %s --progress" % (ud.basecmd, repourl, ud.clonedir)
>
> Why do you quote it manually (and with double instead of single quote))
> instead of using shlex.quote() or making the clone_cmd an array?

This question is best addressed to the patch author (copied)

Note that this patch has already been accepted into the master branch,
and my request for comment was just for cherry-picking the commit into
the dunfell branch.

Steve
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#11752): https://lists.openembedded.org/g/bitbake-devel/message/11752
Mute This Topic: https://lists.openembedded.org/mt/77462759/3617530
Group Owner: bitbake-devel+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Charlie Davies Oct. 13, 2020, 9:44 p.m. 
Thank you for the feedback Enrico.

I agree that using shlex.quote() is a better method to implement this functionality.

I have written a patch using shlex.quote() instead of escaped double quotes. I will run this patch through my local setup tonight before submitting in the morning.

Steve, I will submit against master, dunfell and 1.46.

Charlie
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#11753): https://lists.openembedded.org/g/bitbake-devel/message/11753
Mute This Topic: https://lists.openembedded.org/mt/77462759/3617530
Group Owner: bitbake-devel+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-