Patchwork [bitbake-devel] fetch2: Add new mirror syntax to simplify mirror specifications

login
register
mail settings
Submitter Richard Purdie
Date June 21, 2012, 2:34 p.m.
Message ID <1340289297.1640.122.camel@ted>
Download mbox | patch
Permalink /patch/30405/
State New
Headers show

Comments

Richard Purdie - June 21, 2012, 2:34 p.m.
When writing mirror specifications, the current regexp syntax can be awkward
and hard to get it to do what you want. For example, extracting the 'basename'
of a repository:

PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"

can now become:

PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"

which is much clearer. A MIRRORNAME substitution is also added which contains
an encoded form of both host and path. One of the problems with the existing
regexp syntax is you couldn't access HOST information from PATH and vice-versa
which is an issue this patch also addresses.

Tests for the new syntax are also added.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
Martin Jansa - June 26, 2012, 5:31 p.m.
On Thu, Jun 21, 2012 at 03:34:57PM +0100, Richard Purdie wrote:
> When writing mirror specifications, the current regexp syntax can be awkward
> and hard to get it to do what you want. For example, extracting the 'basename'
> of a repository:
> 
> PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
> 
> can now become:
> 
> PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
> 
> which is much clearer. A MIRRORNAME substitution is also added which contains
> an encoded form of both host and path. One of the problems with the existing
> regexp syntax is you couldn't access HOST information from PATH and vice-versa
> which is an issue this patch also addresses.

One of this fetch2 patches break https?:// in MIRRORS

It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
https?$://.*/.* http://sources.openembedded.org/

I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch
log shorter) on libcap (which doesn't exist on upstream URL).

PREMIRRORS_append () {
cvs://.*/.* http://build.shr-project.org/sources/
svn://.*/.* http://build.shr-project.org/sources/
git://.*/.* http://build.shr-project.org/sources/
hg://.*/.* http://build.shr-project.org/sources/
bzr://.*/.* http://build.shr-project.org/sources/
https?$://.*/.* http://build.shr-project.org/sources/
}

bitbake d316f28ed725ff40daa8771c1aa224ac46d5b224:
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished

bitbake e6ff1d4bab43fdcd8af1230f1d54615f53c1978e
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2
...and after trying Upstream and all MIRRORS (none will match) it fails to fetch it..

Removing '$' doesn't work too:
PREMIRRORS_append () {
https?://.*/.* http://build.shr-project.org/sources/
}
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2

Using http$ doesn't work too
EBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream

And without any regexp with different PREMIRRORS for http and https it also does weird things (looks like trying to resolve returned PREMIRROR again)
PREMIRRORS_append () {
http://.*/.* http://build.shr-project.org/sources/
https://.*/.* http://build.shr-project.org/https-sources/
}

changed libcap SRC_URI to https:
-SRC_URI = "${KERNELORG_MIRROR}/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2"
+SRC_URI = "https://foo.org/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2"


DEBUG: Trying PREMIRRORS
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
NOTE: fetch http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Mirror fetch failure for url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 (original url: https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2)
DEBUG: Fetcher failure: Fetch command export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2' failed with exit code 8, output:
STDOUT:
STDERR: http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2:
2012-06-26 19:23:37 ERROR 404: Not Found.

NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished


Cheers,
> 
> Tests for the new syntax are also added.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 1b99a0f..e6f55f8 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -182,7 +182,7 @@ def encodeurl(decoded):
>  
>      return url
>  
> -def uri_replace(ud, uri_find, uri_replace, d):
> +def uri_replace(ud, uri_find, uri_replace, replacements, d):
>      if not ud.url or not uri_find or not uri_replace:
>          logger.error("uri_replace: passed an undefined value, not replacing")
>          return None
> @@ -213,6 +213,8 @@ def uri_replace(ud, uri_find, uri_replace, d):
>              if not uri_replace_decoded[loc]:
>                  result_decoded[loc] = ""    
>              else:
> +                for k in replacements:
> +                    uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k])
>                  #bb.note("%s %s %s" % (i, uri_replace_decoded[loc], uri_decoded[loc]))
>                  result_decoded[loc] = re.sub(i, uri_replace_decoded[loc], uri_decoded[loc])
>              if loc == 2:
> @@ -485,13 +487,20 @@ def build_mirroruris(origud, mirrors, ld):
>      uris = []
>      uds = []
>  
> +    replacements = {}
> +    replacements["TYPE"] = origud.type
> +    replacements["HOST"] = origud.host
> +    replacements["PATH"] = origud.path
> +    replacements["BASENAME"] = origud.path.split("/")[-1]
> +    replacements["MIRRORNAME"] = origud.host.replace(':','.') + origud.path.replace('/', '.')
> +
>      def adduri(uri, ud, uris, uds):
>          for line in mirrors:
>              try:
>                  (find, replace) = line
>              except ValueError:
>                  continue
> -            newuri = uri_replace(ud, find, replace, ld)
> +            newuri = uri_replace(ud, find, replace, replacements, ld)
>              if not newuri or newuri in uris or newuri == origud.url:
>                  continue
>              uris.append(newuri)
> diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
> index 9961343..1477fab 100644
> --- a/bitbake/lib/bb/tests/fetch.py
> +++ b/bitbake/lib/bb/tests/fetch.py
> @@ -48,7 +48,14 @@ class FetcherTest(unittest.TestCase):
>          ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://www.apache.org/dist", "http://archive.apache.org/dist")
>              : "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2",
>          ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/")
> -            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2"
> +            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2",
> +        ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
> +            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
> +        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
> +            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
> +        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http") 
> +            : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
> +
>          #Renaming files doesn't work
>          #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
>          #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
> @@ -143,8 +150,9 @@ class FetcherTest(unittest.TestCase):
>          for k, v in self.replaceuris.items():
>              ud = bb.fetch.FetchData(k[0], self.d)
>              ud.setup_localpath(self.d)
> -            newuris = bb.fetch2.uri_replace(ud, k[1], k[2], self.d)
> -            self.assertEqual(newuris, v)
> +            mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
> +            newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
> +            self.assertEqual(newuris, [v])
>  
>      def test_urilist1(self):
>          fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
> 
> 
> 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Richard Purdie - June 27, 2012, 10:34 a.m.
On Tue, 2012-06-26 at 19:31 +0200, Martin Jansa wrote:
> On Thu, Jun 21, 2012 at 03:34:57PM +0100, Richard Purdie wrote:
> > When writing mirror specifications, the current regexp syntax can be awkward
> > and hard to get it to do what you want. For example, extracting the 'basename'
> > of a repository:
> > 
> > PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
> > 
> > can now become:
> > 
> > PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
> > 
> > which is much clearer. A MIRRORNAME substitution is also added which contains
> > an encoded form of both host and path. One of the problems with the existing
> > regexp syntax is you couldn't access HOST information from PATH and vice-versa
> > which is an issue this patch also addresses.
> 
> One of this fetch2 patches break https?:// in MIRRORS
> 
> It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
> https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
> https?$://.*/.* http://sources.openembedded.org/
> 
> I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch
> log shorter) on libcap (which doesn't exist on upstream URL).
> 
> PREMIRRORS_append () {
> cvs://.*/.* http://build.shr-project.org/sources/
> svn://.*/.* http://build.shr-project.org/sources/
> git://.*/.* http://build.shr-project.org/sources/
> hg://.*/.* http://build.shr-project.org/sources/
> bzr://.*/.* http://build.shr-project.org/sources/
> https?$://.*/.* http://build.shr-project.org/sources/
> }

This is almost certainly a result of:

http://git.openembedded.org/bitbake/commit/?id=604df1b25cf114e083f52917df2df64e01279c25

since I didn't believe anyone was using the regexp syntax in the base
url type. The bug I wanted to fix was where if you did:

https://.*/.* file:///some/path/

which seems reasonable enough, it would access:

files:///some/path/

which is clearly not what anyone would want.

I'm not sure what to do about fixing this...

Cheers,

Richard
Martin Jansa - June 27, 2012, 11:06 a.m.
On Wed, Jun 27, 2012 at 11:34:13AM +0100, Richard Purdie wrote:
> On Tue, 2012-06-26 at 19:31 +0200, Martin Jansa wrote:
> > On Thu, Jun 21, 2012 at 03:34:57PM +0100, Richard Purdie wrote:
> > > When writing mirror specifications, the current regexp syntax can be awkward
> > > and hard to get it to do what you want. For example, extracting the 'basename'
> > > of a repository:
> > > 
> > > PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
> > > 
> > > can now become:
> > > 
> > > PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
> > > 
> > > which is much clearer. A MIRRORNAME substitution is also added which contains
> > > an encoded form of both host and path. One of the problems with the existing
> > > regexp syntax is you couldn't access HOST information from PATH and vice-versa
> > > which is an issue this patch also addresses.
> > 
> > One of this fetch2 patches break https?:// in MIRRORS
> > 
> > It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
> > https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
> > https?$://.*/.* http://sources.openembedded.org/
> > 
> > I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch
> > log shorter) on libcap (which doesn't exist on upstream URL).
> > 
> > PREMIRRORS_append () {
> > cvs://.*/.* http://build.shr-project.org/sources/
> > svn://.*/.* http://build.shr-project.org/sources/
> > git://.*/.* http://build.shr-project.org/sources/
> > hg://.*/.* http://build.shr-project.org/sources/
> > bzr://.*/.* http://build.shr-project.org/sources/
> > https?$://.*/.* http://build.shr-project.org/sources/
> > }
> 
> This is almost certainly a result of:
> 
> http://git.openembedded.org/bitbake/commit/?id=604df1b25cf114e083f52917df2df64e01279c25
> 
> since I didn't believe anyone was using the regexp syntax in the base
> url type. The bug I wanted to fix was where if you did:
> 
> https://.*/.* file:///some/path/
> 
> which seems reasonable enough, it would access:
> 
> files:///some/path/
> 
> which is clearly not what anyone would want.
> 
> I'm not sure what to do about fixing this...

oe-core mirrors.bbclass is using that..
http://git.openembedded.org/openembedded-core/tree/meta/classes/mirrors.bbclass
so maybe someone should fix mirrors.bbclass if this is expected behavior
with new bitbake..

Cheers,

> 
> Cheers,
> 
> Richard
> 
> 
>

Patch

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 1b99a0f..e6f55f8 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -182,7 +182,7 @@  def encodeurl(decoded):
 
     return url
 
-def uri_replace(ud, uri_find, uri_replace, d):
+def uri_replace(ud, uri_find, uri_replace, replacements, d):
     if not ud.url or not uri_find or not uri_replace:
         logger.error("uri_replace: passed an undefined value, not replacing")
         return None
@@ -213,6 +213,8 @@  def uri_replace(ud, uri_find, uri_replace, d):
             if not uri_replace_decoded[loc]:
                 result_decoded[loc] = ""    
             else:
+                for k in replacements:
+                    uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k])
                 #bb.note("%s %s %s" % (i, uri_replace_decoded[loc], uri_decoded[loc]))
                 result_decoded[loc] = re.sub(i, uri_replace_decoded[loc], uri_decoded[loc])
             if loc == 2:
@@ -485,13 +487,20 @@  def build_mirroruris(origud, mirrors, ld):
     uris = []
     uds = []
 
+    replacements = {}
+    replacements["TYPE"] = origud.type
+    replacements["HOST"] = origud.host
+    replacements["PATH"] = origud.path
+    replacements["BASENAME"] = origud.path.split("/")[-1]
+    replacements["MIRRORNAME"] = origud.host.replace(':','.') + origud.path.replace('/', '.')
+
     def adduri(uri, ud, uris, uds):
         for line in mirrors:
             try:
                 (find, replace) = line
             except ValueError:
                 continue
-            newuri = uri_replace(ud, find, replace, ld)
+            newuri = uri_replace(ud, find, replace, replacements, ld)
             if not newuri or newuri in uris or newuri == origud.url:
                 continue
             uris.append(newuri)
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index 9961343..1477fab 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -48,7 +48,14 @@  class FetcherTest(unittest.TestCase):
         ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://www.apache.org/dist", "http://archive.apache.org/dist")
             : "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2",
         ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/")
-            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2"
+            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2",
+        ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
+            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
+            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http") 
+            : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+
         #Renaming files doesn't work
         #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
         #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
@@ -143,8 +150,9 @@  class FetcherTest(unittest.TestCase):
         for k, v in self.replaceuris.items():
             ud = bb.fetch.FetchData(k[0], self.d)
             ud.setup_localpath(self.d)
-            newuris = bb.fetch2.uri_replace(ud, k[1], k[2], self.d)
-            self.assertEqual(newuris, v)
+            mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
+            newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
+            self.assertEqual(newuris, [v])
 
     def test_urilist1(self):
         fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)