Patchwork [2/2] sanity: implement network connectivity test

login
register
mail settings
Submitter Joshua Lock
Date June 20, 2011, 6:09 p.m.
Message ID <909a6c756247a7a0f68f60a64477d4e8419ca74c.1308593086.git.josh@linux.intel.com>
Download mbox | patch
Permalink /patch/6147/
State New, archived
Headers show

Comments

Joshua Lock - June 20, 2011, 6:09 p.m.
Sanity test to verify files can be fetched from the network using git, http
and https fetchers point users at a page to help get set up in the case of a
failure.

Addresses [YOCTO #933]

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |   34 ++++++++++++++++++++++++++++++++++
 1 files changed, 34 insertions(+), 0 deletions(-)
Saul Wold - June 26, 2011, 12:53 a.m.
On 06/20/2011 11:09 AM, Joshua Lock wrote:
> Sanity test to verify files can be fetched from the network using git, http
> and https fetchers point users at a page to help get set up in the case of a
> failure.
>
> Addresses [YOCTO #933]
>
> Signed-off-by: Joshua Lock<josh@linux.intel.com>
> ---
>   meta/classes/sanity.bbclass |   34 ++++++++++++++++++++++++++++++++++
>   1 files changed, 34 insertions(+), 0 deletions(-)
>
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index bffa4f5..650df5f 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
>
>       # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
>       testmsg = check_create_long_filename(tmpdir, "TMPDIR")
> +    # Check that we can fetch from various network transports
> +    testmsg = testmsg + check_connectivity(data)
>       return testmsg
>
>   def check_sanity_version_change(data):
> @@ -75,6 +77,38 @@ def check_create_long_filename(filepath, pathname):
>               return "Failed to create a file in %s: %s" % (pathname, strerror)
>       return ""
>
> +def check_connectivity(d):
> +    # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable using
> +    # the same syntax as SRC_URI.
> +    test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or "").split()
> +    # If no URI's set, fallback to some default ones we know of
> +    if len(test_uris) == 0:
> +        test_uris = ["http://yoctoproject.org/about",
> +                     "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
> +                     "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
> +    retval = ""
> +
> +    # Only check connectivity if network access and this check enabled.
> +    # Because it's a fairly heavy test allow disabling of just this sanity test
> +    # by setting DISABLE_NETWORK_SANITY.
> +    network_disabled = not bb.data.getVar('BB_NO_NETWORK', d, True)
> +    check_disabled = bb.data.getVar('DISABLE_NETWORK_SANITY', d, True)
> +    if check_disabled or network_disabled:
> +        data = bb.data.createCopy(d)
> +        dldir = bb.data.expand('${TMPDIR}/sanity', data)
> +        bb.data.setVar('DL_DIR', dldir, data)
> +
> +        try:
> +            fetcher = bb.fetch2.Fetch(test_uris, data)
> +            fetcher.download()
> +	    fetcher.clean(test_uris)
> +        except Exception:
> +            retval = "Error connecting to the network to fetch, http/https and git checked.\nPlease check the wiki (https://wiki.yoctoproject.org/wiki/Connectivity\%20Troubleshooting) for more suggestions.\n"
> +        finally:
> +            # Make sure we tidy up the cruft
> +            oe.path.remove(dldir)
> +    return retval
> +
>   def check_sanity(e):
>       from bb import note, error, data, __version__
>

Josh,

When I ran this, I got the following messages about missing SRC_URI 
Checksums and then it failed to parse recipes.  When I removed this 
commit things worked normally.



Pseudo is not present but is required, building this first before the 
main build
Cloning into bare repository 
/intel/poky2/builds/pending/tmp/sanity/git2/git.yoctoproject.org.yocto-firewall-test...
  NOTE: fetch http://yoctoproject.org/about
WARNING: Missing SRC_URI checksum for 
/intel/poky2/builds/pending/tmp/sanity/about, consider adding to the recipe:
SRC_URI[md5sum] = "1ca6c9ee2ce5dc5d251e649cbd4e339f"
SRC_URI[sha256sum] = 
"24cda768cacaf6cc41ab75a46df4936f58075c4788e094c262c35d5ba69544e2"
NOTE: fetch 
https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0
WARNING: Missing SRC_URI checksum for 
/intel/poky2/builds/pending/tmp/sanity/crownbay-bernard-5.0.0, consider 
adding to the recipe:
SRC_URI[md5sum] = "c534ba257c640e47300572607762b22a"
SRC_URI[sha256sum] = 
"485812cf1c43381d70611e85eaf8a74982b452121bdc92accf7a62f0a2707137"
NOTE: fetch 
http://autobuilder.yoctoproject.org/sources/git2_git.yoctoproject.org.yocto-firewall-test.tar.gz
Parsing recipes...ERROR: Command execution failed: Exited with 1
NOTE: fetch 
http://autobuilder.yoctoproject.org/sources/git2_git.yoctoproject.org.yocto-firewall-test.tar.gz
WARNING: Missing SRC_URI checksum for 
/intel/poky2/builds/pending/tmp/sanity/crownbay-bernard-5.0.0, consider 
adding to the recipe:
SRC_URI[md5sum] = "c534ba257c640e47300572607762b22a"
SRC_URI[sha256sum] = 
"485812cf1c43381d70611e85eaf8a74982b452121bdc92accf7a62f0a2707137"
NOTE: fetch 
https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0
WARNING: Missing SRC_URI checksum for 
/intel/poky2/builds/pending/tmp/sanity/about, consider adding to the recipe:
SRC_URI[md5sum] = "1ca6c9ee2ce5dc5d251e649cbd4e339f"
SRC_URI[sha256sum] = 
"24cda768cacaf6cc41ab75a46df4936f58075c4788e094c262c35d5ba69544e2"
NOTE: fetch http://yoctoproject.org/about

Thanks
	Sau!
Joshua Lock - June 26, 2011, 1:56 a.m.
On Sat, 2011-06-25 at 17:53 -0700, Saul Wold wrote:
> On 06/20/2011 11:09 AM, Joshua Lock wrote:
> > Sanity test to verify files can be fetched from the network using git, http
> > and https fetchers point users at a page to help get set up in the case of a
> > failure.
> >
> > Addresses [YOCTO #933]
> >
> > Signed-off-by: Joshua Lock<josh@linux.intel.com>
> > ---
> >   meta/classes/sanity.bbclass |   34 ++++++++++++++++++++++++++++++++++
> >   1 files changed, 34 insertions(+), 0 deletions(-)
> >
> > diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> > index bffa4f5..650df5f 100644
> > --- a/meta/classes/sanity.bbclass
> > +++ b/meta/classes/sanity.bbclass
> > @@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
> >
> >       # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
> >       testmsg = check_create_long_filename(tmpdir, "TMPDIR")
> > +    # Check that we can fetch from various network transports
> > +    testmsg = testmsg + check_connectivity(data)
> >       return testmsg
> >
> >   def check_sanity_version_change(data):
> > @@ -75,6 +77,38 @@ def check_create_long_filename(filepath, pathname):
> >               return "Failed to create a file in %s: %s" % (pathname, strerror)
> >       return ""
> >
> > +def check_connectivity(d):
> > +    # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable using
> > +    # the same syntax as SRC_URI.
> > +    test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or "").split()
> > +    # If no URI's set, fallback to some default ones we know of
> > +    if len(test_uris) == 0:
> > +        test_uris = ["http://yoctoproject.org/about",
> > +                     "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
> > +                     "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
> > +    retval = ""
> > +
> > +    # Only check connectivity if network access and this check enabled.
> > +    # Because it's a fairly heavy test allow disabling of just this sanity test
> > +    # by setting DISABLE_NETWORK_SANITY.
> > +    network_disabled = not bb.data.getVar('BB_NO_NETWORK', d, True)
> > +    check_disabled = bb.data.getVar('DISABLE_NETWORK_SANITY', d, True)
> > +    if check_disabled or network_disabled:
> > +        data = bb.data.createCopy(d)
> > +        dldir = bb.data.expand('${TMPDIR}/sanity', data)
> > +        bb.data.setVar('DL_DIR', dldir, data)
> > +
> > +        try:
> > +            fetcher = bb.fetch2.Fetch(test_uris, data)
> > +            fetcher.download()
> > +	    fetcher.clean(test_uris)
> > +        except Exception:
> > +            retval = "Error connecting to the network to fetch, http/https and git checked.\nPlease check the wiki (https://wiki.yoctoproject.org/wiki/Connectivity\%20Troubleshooting) for more suggestions.\n"
> > +        finally:
> > +            # Make sure we tidy up the cruft
> > +            oe.path.remove(dldir)
> > +    return retval
> > +
> >   def check_sanity(e):
> >       from bb import note, error, data, __version__
> >
> 
> Josh,
> 
> When I ran this, I got the following messages about missing SRC_URI 
> Checksums and then it failed to parse recipes.  When I removed this 
> commit things worked normally.
> 

Ick. I didn't see this when I first wrote the patches but am able to
reproduce now... It's the "ERROR: Command execution failed: Exited with
1" which indicates the real problem. The WARNING's are just extra noise.

Looks like the git fetcher is cd'ing before fetching and then breaking
the parsing after the sanity check is complete by not being in the right
cwd as expected by the parser. Messy.

At least that's my inference from reading this output:

joshual@scimitar:~/Projects/Yocto/oe-core/boom [josh/connection-test]
$ bitbake core-image-minimal
Pseudo is not present but is required, building this first before the
main build
Cloning into bare
repository /home/joshual/Projects/Yocto/oe-core/boom/tmp/sanity/git2/git.yoctoproject.org.yocto-firewall-test...
 Parsing recipes:   0% |                                         | ETA:
--:--:-ERROR: Unable to
parse /home/joshual/Projects/Yocto/oe-core/meta/recipes-graphics/xorg-driver/xf86-video-intel_2.15.0.bb
Traceback (most recent call last):
  File "/home/joshual/Projects/Yocto/bitbake/lib/bb/cooker.py", line
1164, in
parse_file(task=('/home/joshual/Projects/Yocto/oe-core/meta/recipes-graphics/xorg-driver/xf86-video-intel_2.15.0.bb', [], [<class 'bb.cache.CoreRecipeInfo'>])):
         try:
    >        return True, bb.cache.Cache.parse(filename, appends,
parse_file.cfg, caches_array)
         except Exception as exc:
  File "/home/joshual/Projects/Yocto/bitbake/lib/bb/cache.py", line 402,
in parse(cls=<class 'bb.cache.Cache'>,
filename='/home/joshual/Projects/Yocto/oe-core/meta/recipes-graphics/xorg-driver/xf86-video-intel_2.15.0.bb', appends=[], configdata=<bb.data_smart.DataSmart object at 0x961f04c>, caches_array=[<class 'bb.cache.CoreRecipeInfo'>]):
             infos = []
    >        datastores = cls.load_bbfile(filename, appends, configdata)
             depends = set()
  File "/home/joshual/Projects/Yocto/bitbake/lib/bb/cache.py", line 641,
in
load_bbfile(bbfile='/home/joshual/Projects/Yocto/oe-core/meta/recipes-graphics/xorg-driver/xf86-video-intel_2.15.0.bb', appends=[], config=<bb.data_smart.DataSmart object at 0x961f04c>):
             bbfile_loc = os.path.abspath(os.path.dirname(bbfile))
    >        oldpath = os.path.abspath(os.getcwd())
             parse.cached_mtime_noerror(bbfile_loc)
OSError: [Errno 2] No such file or directory

ERROR: Command execution failed: Exited with 1

The following hunk is a workaround but I need to figure out if this is
the right way to fix it and Saturday night isn't when I do my best
thinking ;-)

@@ -90,6 +90,7 @@ def check_connectivity(d):
     network_disabled = not bb.data.getVar('BB_NO_NETWORK', data, True)
     check_disabled = bb.data.getVar('DISABLE_NETWORK_SANITY', data,
True)
     if check_disabled or network_disabled:
+        bookmark = os.getcwd()
         dldir = bb.data.expand('${TMPDIR}/sanity', data)
         bb.data.setVar('DL_DIR', dldir, data)
 
@@ -102,6 +103,7 @@ def check_connectivity(d):
         finally:
             # Make sure we tidy up the cruft
             oe.path.remove(dldir)
+            os.chdir(bookmark)
     return retval

I'll work on a rev two patch and also try and figure out a clean way to
at least hide the SRC_URI checksum warnings.

Thanks,
Joshua
Khem Raj - June 26, 2011, 2:33 a.m.
On 6/25/2011 5:53 PM, Saul Wold wrote:
> On 06/20/2011 11:09 AM, Joshua Lock wrote:
>> Sanity test to verify files can be fetched from the network using git,
>> http
>> and https fetchers point users at a page to help get set up in the
>> case of a
>> failure.
>>
>> Addresses [YOCTO #933]
>>
>> Signed-off-by: Joshua Lock<josh@linux.intel.com>
>> ---
>> meta/classes/sanity.bbclass | 34 ++++++++++++++++++++++++++++++++++
>> 1 files changed, 34 insertions(+), 0 deletions(-)
>>
>> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
>> index bffa4f5..650df5f 100644
>> --- a/meta/classes/sanity.bbclass
>> +++ b/meta/classes/sanity.bbclass
>> @@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
>>
>> # Check that TMPDIR isn't on a filesystem with limited filename length
>> (eg. eCryptFS)
>> testmsg = check_create_long_filename(tmpdir, "TMPDIR")
>> + # Check that we can fetch from various network transports
>> + testmsg = testmsg + check_connectivity(data)
>> return testmsg
>>
>> def check_sanity_version_change(data):
>> @@ -75,6 +77,38 @@ def check_create_long_filename(filepath, pathname):
>> return "Failed to create a file in %s: %s" % (pathname, strerror)
>> return ""
>>
>> +def check_connectivity(d):
>> + # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable
>> using
>> + # the same syntax as SRC_URI.
>> + test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or
>> "").split()
>> + # If no URI's set, fallback to some default ones we know of
>> + if len(test_uris) == 0:
>> + test_uris = ["http://yoctoproject.org/about",
>> +
>> "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
>>
>> + "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
>> + retval = ""

imo this change is yocto specific doesnt belong to core
>> +
>> + # Only check connectivity if network access and this check enabled.
>> + # Because it's a fairly heavy test allow disabling of just this
>> sanity test
>> + # by setting DISABLE_NETWORK_SANITY.
>> + network_disabled = not bb.data.getVar('BB_NO_NETWORK', d, True)
>> + check_disabled = bb.data.getVar('DISABLE_NETWORK_SANITY', d, True)
>> + if check_disabled or network_disabled:
>> + data = bb.data.createCopy(d)
>> + dldir = bb.data.expand('${TMPDIR}/sanity', data)
>> + bb.data.setVar('DL_DIR', dldir, data)
>> +
>> + try:
>> + fetcher = bb.fetch2.Fetch(test_uris, data)
>> + fetcher.download()
>> + fetcher.clean(test_uris)
>> + except Exception:
>> + retval = "Error connecting to the network to fetch, http/https and
>> git checked.\nPlease check the wiki
>> (https://wiki.yoctoproject.org/wiki/Connectivity\%20Troubleshooting)
>> for more suggestions.\n"
>> + finally:
>> + # Make sure we tidy up the cruft
>> + oe.path.remove(dldir)
>> + return retval
>> +
>> def check_sanity(e):
>> from bb import note, error, data, __version__
>>
>
> Josh,
>
> When I ran this, I got the following messages about missing SRC_URI
> Checksums and then it failed to parse recipes. When I removed this
> commit things worked normally.
>
>
>
> Pseudo is not present but is required, building this first before the
> main build
> Cloning into bare repository
> /intel/poky2/builds/pending/tmp/sanity/git2/git.yoctoproject.org.yocto-firewall-test...
>
> NOTE: fetch http://yoctoproject.org/about
> WARNING: Missing SRC_URI checksum for
> /intel/poky2/builds/pending/tmp/sanity/about, consider adding to the
> recipe:
> SRC_URI[md5sum] = "1ca6c9ee2ce5dc5d251e649cbd4e339f"
> SRC_URI[sha256sum] =
> "24cda768cacaf6cc41ab75a46df4936f58075c4788e094c262c35d5ba69544e2"
> NOTE: fetch
> https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0
> WARNING: Missing SRC_URI checksum for
> /intel/poky2/builds/pending/tmp/sanity/crownbay-bernard-5.0.0, consider
> adding to the recipe:
> SRC_URI[md5sum] = "c534ba257c640e47300572607762b22a"
> SRC_URI[sha256sum] =
> "485812cf1c43381d70611e85eaf8a74982b452121bdc92accf7a62f0a2707137"
> NOTE: fetch
> http://autobuilder.yoctoproject.org/sources/git2_git.yoctoproject.org.yocto-firewall-test.tar.gz
>
> Parsing recipes...ERROR: Command execution failed: Exited with 1
> NOTE: fetch
> http://autobuilder.yoctoproject.org/sources/git2_git.yoctoproject.org.yocto-firewall-test.tar.gz
>
> WARNING: Missing SRC_URI checksum for
> /intel/poky2/builds/pending/tmp/sanity/crownbay-bernard-5.0.0, consider
> adding to the recipe:
> SRC_URI[md5sum] = "c534ba257c640e47300572607762b22a"
> SRC_URI[sha256sum] =
> "485812cf1c43381d70611e85eaf8a74982b452121bdc92accf7a62f0a2707137"
> NOTE: fetch
> https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0
> WARNING: Missing SRC_URI checksum for
> /intel/poky2/builds/pending/tmp/sanity/about, consider adding to the
> recipe:
> SRC_URI[md5sum] = "1ca6c9ee2ce5dc5d251e649cbd4e339f"
> SRC_URI[sha256sum] =
> "24cda768cacaf6cc41ab75a46df4936f58075c4788e094c262c35d5ba69544e2"
> NOTE: fetch http://yoctoproject.org/about
>
> Thanks
> Sau!
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Joshua Lock - June 26, 2011, 5:37 p.m.
On Sat, 2011-06-25 at 19:33 -0700, Khem Raj wrote:
> On 6/25/2011 5:53 PM, Saul Wold wrote:
> > On 06/20/2011 11:09 AM, Joshua Lock wrote:
> >> Sanity test to verify files can be fetched from the network using git,
> >> http
> >> and https fetchers point users at a page to help get set up in the
> >> case of a
> >> failure.
> >>
> >> Addresses [YOCTO #933]
> >>
> >> Signed-off-by: Joshua Lock<josh@linux.intel.com>
> >> ---
> >> meta/classes/sanity.bbclass | 34 ++++++++++++++++++++++++++++++++++
> >> 1 files changed, 34 insertions(+), 0 deletions(-)
> >>
> >> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> >> index bffa4f5..650df5f 100644
> >> --- a/meta/classes/sanity.bbclass
> >> +++ b/meta/classes/sanity.bbclass
> >> @@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
> >>
> >> # Check that TMPDIR isn't on a filesystem with limited filename length
> >> (eg. eCryptFS)
> >> testmsg = check_create_long_filename(tmpdir, "TMPDIR")
> >> + # Check that we can fetch from various network transports
> >> + testmsg = testmsg + check_connectivity(data)
> >> return testmsg
> >>
> >> def check_sanity_version_change(data):
> >> @@ -75,6 +77,38 @@ def check_create_long_filename(filepath, pathname):
> >> return "Failed to create a file in %s: %s" % (pathname, strerror)
> >> return ""
> >>
> >> +def check_connectivity(d):
> >> + # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable
> >> using
> >> + # the same syntax as SRC_URI.
> >> + test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or
> >> "").split()
> >> + # If no URI's set, fallback to some default ones we know of
> >> + if len(test_uris) == 0:
> >> + test_uris = ["http://yoctoproject.org/about",
> >> +
> >> "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
> >>
> >> + "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
> >> + retval = ""
> 
> imo this change is yocto specific doesnt belong to core

Are you objecting to the feature (testing whether the fetchers can work
on a newly configured tmpdir) or the implementation (using
yoctoproject.org URI's)?

I can change this patch to only run the check when the
CONNECTIVITY_CHECK_URI's and remove the default uri's. Would that make
the change less objectionable?

Joshua
Khem Raj - June 26, 2011, 6:08 p.m.
On 06/26/2011 10:37 AM, Joshua Lock wrote:
> On Sat, 2011-06-25 at 19:33 -0700, Khem Raj wrote:
>> On 6/25/2011 5:53 PM, Saul Wold wrote:
>>> On 06/20/2011 11:09 AM, Joshua Lock wrote:
>>>> Sanity test to verify files can be fetched from the network using git,
>>>> http
>>>> and https fetchers point users at a page to help get set up in the
>>>> case of a
>>>> failure.
>>>>
>>>> Addresses [YOCTO #933]
>>>>
>>>> Signed-off-by: Joshua Lock<josh@linux.intel.com>
>>>> ---
>>>> meta/classes/sanity.bbclass | 34 ++++++++++++++++++++++++++++++++++
>>>> 1 files changed, 34 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
>>>> index bffa4f5..650df5f 100644
>>>> --- a/meta/classes/sanity.bbclass
>>>> +++ b/meta/classes/sanity.bbclass
>>>> @@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
>>>>
>>>> # Check that TMPDIR isn't on a filesystem with limited filename length
>>>> (eg. eCryptFS)
>>>> testmsg = check_create_long_filename(tmpdir, "TMPDIR")
>>>> + # Check that we can fetch from various network transports
>>>> + testmsg = testmsg + check_connectivity(data)
>>>> return testmsg
>>>>
>>>> def check_sanity_version_change(data):
>>>> @@ -75,6 +77,38 @@ def check_create_long_filename(filepath, pathname):
>>>> return "Failed to create a file in %s: %s" % (pathname, strerror)
>>>> return ""
>>>>
>>>> +def check_connectivity(d):
>>>> + # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable
>>>> using
>>>> + # the same syntax as SRC_URI.
>>>> + test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or
>>>> "").split()
>>>> + # If no URI's set, fallback to some default ones we know of
>>>> + if len(test_uris) == 0:
>>>> + test_uris = ["http://yoctoproject.org/about",
>>>> +
>>>> "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
>>>>
>>>> + "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
>>>> + retval = ""
>>
>> imo this change is yocto specific doesnt belong to core
>
> Are you objecting to the feature (testing whether the fetchers can work
> on a newly configured tmpdir) or the implementation (using
> yoctoproject.org URI's)?

the latter
>
> I can change this patch to only run the check when the
> CONNECTIVITY_CHECK_URI's and remove the default uri's. Would that make
> the change less objectionable?

yes that would be better.
>
> Joshua
Joshua Lock - June 27, 2011, 3:03 p.m.
On Sun, 2011-06-26 at 11:08 -0700, Khem Raj wrote:
> On 06/26/2011 10:37 AM, Joshua Lock wrote:
> > On Sat, 2011-06-25 at 19:33 -0700, Khem Raj wrote:
> >> On 6/25/2011 5:53 PM, Saul Wold wrote:
> >>> On 06/20/2011 11:09 AM, Joshua Lock wrote:
> >>>> Sanity test to verify files can be fetched from the network using git,
> >>>> http
> >>>> and https fetchers point users at a page to help get set up in the
> >>>> case of a
> >>>> failure.
> >>>>
> >>>> Addresses [YOCTO #933]
> >>>>
> >>>> Signed-off-by: Joshua Lock<josh@linux.intel.com>
> >>>> ---
> >>>> meta/classes/sanity.bbclass | 34 ++++++++++++++++++++++++++++++++++
> >>>> 1 files changed, 34 insertions(+), 0 deletions(-)
> >>>>
> >>>> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> >>>> index bffa4f5..650df5f 100644
> >>>> --- a/meta/classes/sanity.bbclass
> >>>> +++ b/meta/classes/sanity.bbclass
> >>>> @@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
> >>>>
> >>>> # Check that TMPDIR isn't on a filesystem with limited filename length
> >>>> (eg. eCryptFS)
> >>>> testmsg = check_create_long_filename(tmpdir, "TMPDIR")
> >>>> + # Check that we can fetch from various network transports
> >>>> + testmsg = testmsg + check_connectivity(data)
> >>>> return testmsg
> >>>>
> >>>> def check_sanity_version_change(data):
> >>>> @@ -75,6 +77,38 @@ def check_create_long_filename(filepath, pathname):
> >>>> return "Failed to create a file in %s: %s" % (pathname, strerror)
> >>>> return ""
> >>>>
> >>>> +def check_connectivity(d):
> >>>> + # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable
> >>>> using
> >>>> + # the same syntax as SRC_URI.
> >>>> + test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or
> >>>> "").split()
> >>>> + # If no URI's set, fallback to some default ones we know of
> >>>> + if len(test_uris) == 0:
> >>>> + test_uris = ["http://yoctoproject.org/about",
> >>>> +
> >>>> "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
> >>>>
> >>>> + "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
> >>>> + retval = ""
> >>
> >> imo this change is yocto specific doesnt belong to core
> >
> > Are you objecting to the feature (testing whether the fetchers can work
> > on a newly configured tmpdir) or the implementation (using
> > yoctoproject.org URI's)?
> 
> the latter
> >
> > I can change this patch to only run the check when the
> > CONNECTIVITY_CHECK_URI's and remove the default uri's. Would that make
> > the change less objectionable?
> 
> yes that would be better.
> >

Okey dokes. Expect a v3 soon.

Thanks,
Joshua
Richard Purdie - June 28, 2011, 2:13 p.m.
On Sun, 2011-06-26 at 11:08 -0700, Khem Raj wrote:
> On 06/26/2011 10:37 AM, Joshua Lock wrote:
> > On Sat, 2011-06-25 at 19:33 -0700, Khem Raj wrote:
> >> On 6/25/2011 5:53 PM, Saul Wold wrote:
> >>>> + # If no URI's set, fallback to some default ones we know of
> >>>> + if len(test_uris) == 0:
> >>>> + test_uris = ["http://yoctoproject.org/about",
> >>>> +
> >>>> "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
> >>>>
> >>>> + "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
> >>>> + retval = ""
> >>
> >> imo this change is yocto specific doesnt belong to core
> >
> > Are you objecting to the feature (testing whether the fetchers can work
> > on a newly configured tmpdir) or the implementation (using
> > yoctoproject.org URI's)?
> 
> the latter

I'd just point out that this is a test URL and turning off a load of
checks designed to improve usability just based on the test url seems a
little counter-intuitive to improving new user usability of OE.

Cheers,

Richard

Patch

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index bffa4f5..650df5f 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -35,6 +35,8 @@  def check_sanity_tmpdir_change(tmpdir, data):
 
     # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
     testmsg = check_create_long_filename(tmpdir, "TMPDIR")
+    # Check that we can fetch from various network transports
+    testmsg = testmsg + check_connectivity(data)
     return testmsg
         
 def check_sanity_version_change(data):
@@ -75,6 +77,38 @@  def check_create_long_filename(filepath, pathname):
             return "Failed to create a file in %s: %s" % (pathname, strerror)
     return ""
 
+def check_connectivity(d):
+    # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable using
+    # the same syntax as SRC_URI.
+    test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or "").split()
+    # If no URI's set, fallback to some default ones we know of
+    if len(test_uris) == 0:
+        test_uris = ["http://yoctoproject.org/about",
+                     "https://eula-downloads.yoctoproject.org/crownbay/crownbay-bernard-5.0.0",
+                     "git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=HEAD"]
+    retval = ""
+
+    # Only check connectivity if network access and this check enabled.
+    # Because it's a fairly heavy test allow disabling of just this sanity test
+    # by setting DISABLE_NETWORK_SANITY.
+    network_disabled = not bb.data.getVar('BB_NO_NETWORK', d, True)
+    check_disabled = bb.data.getVar('DISABLE_NETWORK_SANITY', d, True)
+    if check_disabled or network_disabled:
+        data = bb.data.createCopy(d)
+        dldir = bb.data.expand('${TMPDIR}/sanity', data)
+        bb.data.setVar('DL_DIR', dldir, data)
+
+        try:
+            fetcher = bb.fetch2.Fetch(test_uris, data)
+            fetcher.download()
+	    fetcher.clean(test_uris)
+        except Exception:
+            retval = "Error connecting to the network to fetch, http/https and git checked.\nPlease check the wiki (https://wiki.yoctoproject.org/wiki/Connectivity\%20Troubleshooting) for more suggestions.\n"
+        finally:
+            # Make sure we tidy up the cruft
+            oe.path.remove(dldir)
+    return retval
+
 def check_sanity(e):
     from bb import note, error, data, __version__