diff mbox series

[master,mickledore,kirkstone,dunfell,1/2] cve-update-nvd2-native: retry all errors and sleep between retries

Message ID 20230711063629.7627-1-peter.marko@siemens.com
State Accepted, archived
Commit 88dad8f198baa80af5ab576498f4df6ed639d551
Headers show
Series [master,mickledore,kirkstone,dunfell,1/2] cve-update-nvd2-native: retry all errors and sleep between retries | expand

Commit Message

Peter Marko July 11, 2023, 6:36 a.m. UTC 
From: Peter Marko <peter.marko@siemens.com>

Last couple days it is not possible to update NVD DB as servers
are returning lot of errors.
Mostly "HTTP Error 503: Service Unavailable" is observed but
sporadially also some others.

Retrying helps in most cases, so extend retries to all errors.

Additionally add sleep which is recommended by NVD between requests.
These retries are already implemented between successful requests,
but giving servers time between failed ones is important, too.

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

Comments

Marta Rybczynska July 11, 2023, 12:49 p.m. UTC | #1 
Thank you Peter for debugging this. Could you dump us a log of one of your
typical runs to see what the errors are?
We might consider mirroring at some point.

Kind regards,
Marta

On Tue, Jul 11, 2023 at 8:37 AM Peter Marko via lists.openembedded.org
<peter.marko=siemens.com@lists.openembedded.org> wrote:

> From: Peter Marko <peter.marko@siemens.com>
>
> Last couple days it is not possible to update NVD DB as servers
> are returning lot of errors.
> Mostly "HTTP Error 503: Service Unavailable" is observed but
> sporadially also some others.
>
> Retrying helps in most cases, so extend retries to all errors.
>
> Additionally add sleep which is recommended by NVD between requests.
> These retries are already implemented between successful requests,
> but giving servers time between failed ones is important, too.
>
> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> ---
>  meta/recipes-core/meta/cve-update-nvd2-native.bb | 11 ++++-------
>  1 file changed, 4 insertions(+), 7 deletions(-)
>
> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> index 4585126f73..a7392405e0 100644
> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> @@ -119,6 +119,7 @@ def nvd_request_next(url, api_key, args):
>      import urllib.parse
>      import gzip
>      import http
> +    import time
>
>      headers = {}
>      if api_key:
> @@ -140,13 +141,9 @@ def nvd_request_next(url, api_key, args):
>
>              r.close()
>
> -        except UnicodeDecodeError:
> -            # Received garbage, retry
> -            bb.debug(2, "CVE database: received malformed data, retrying
> (request: %s)" %(full_request))
> -            pass
> -        except http.client.IncompleteRead:
> -            # Read incomplete, let's try again
> -            bb.debug(2, "CVE database: received incomplete data, retrying
> (request: %s)" %(full_request))
> +        except Exception as e:
> +            bb.debug(2, "CVE database: received error (%s), retrying
> (request: %s)" %(e, full_request))
> +            time.sleep(6)
>              pass
>          else:
>              return raw_data
> --
> 2.30.2
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184115):
> https://lists.openembedded.org/g/openembedded-core/message/184115
> Mute This Topic: https://lists.openembedded.org/mt/100074006/5827677
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> rybczynska@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
Peter Marko July 11, 2023, 1:11 p.m. UTC | #2 
Hello,

My testing was done with log increased to warning, and this is a typical outcome:
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=32000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=32000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=58000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=58000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=130000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=130000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=214000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=214000)

Out of 11 runs I had:
- 1 different failure but since at that time I was only catching 503s and thus it crashed I don’t have a cooker log, it was something that remote hang up without providing any data.
- 1 problem that 3 tries were not enough
- 9 times success
But maybe it also depends on region where your machine is doing the requests from or your time when you’re doing the run.
I think the first patch (catching all exceptions) is something really needed and the second one (5 retries) is something not very nice but probably needed for time being.
Possibly we could increase the sleep time instead of increasing retries; I’m open to suggestions as I’d like to see the changes merged soon so we can continue with vulnerability handling.

Regards,
  Peter

From: Marta Rybczynska <rybczynska@gmail.com>
Sent: Tuesday, July 11, 2023 14:50
To: Marko, Peter (ADV D EU SK BFS1) <Peter.Marko@siemens.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core][master][mickledore][kirkstone][dunfell][PATCH 1/2] cve-update-nvd2-native: retry all errors and sleep between retries

Thank you Peter for debugging this. Could you dump us a log of one of your typical runs to see what the errors are?
We might consider mirroring at some point.

Kind regards,
Marta

On Tue, Jul 11, 2023 at 8:37 AM Peter Marko via lists.openembedded.org<http://lists.openembedded.org> <peter.marko=siemens.com@lists.openembedded.org<mailto:siemens.com@lists.openembedded.org>> wrote:
From: Peter Marko <peter.marko@siemens.com<mailto:peter.marko@siemens.com>>

Last couple days it is not possible to update NVD DB as servers
are returning lot of errors.
Mostly "HTTP Error 503: Service Unavailable" is observed but
sporadially also some others.

Retrying helps in most cases, so extend retries to all errors.

Additionally add sleep which is recommended by NVD between requests.
These retries are already implemented between successful requests,
but giving servers time between failed ones is important, too.

Signed-off-by: Peter Marko <peter.marko@siemens.com<mailto:peter.marko@siemens.com>>
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb> | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb> b/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb>
index 4585126f73..a7392405e0 100644
--- a/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb>
+++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb>
@@ -119,6 +119,7 @@ def nvd_request_next(url, api_key, args):
     import urllib.parse
     import gzip
     import http
+    import time

     headers = {}
     if api_key:
@@ -140,13 +141,9 @@ def nvd_request_next(url, api_key, args):

             r.close()

-        except UnicodeDecodeError:
-            # Received garbage, retry
-            bb.debug(2, "CVE database: received malformed data, retrying (request: %s)" %(full_request))
-            pass
-        except http.client.IncompleteRead:
-            # Read incomplete, let's try again
-            bb.debug(2, "CVE database: received incomplete data, retrying (request: %s)" %(full_request))
+        except Exception as e:
+            bb.debug(2, "CVE database: received error (%s), retrying (request: %s)" %(e, full_request))
+            time.sleep(6)
             pass
         else:
             return raw_data
--
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184115): https://lists.openembedded.org/g/openembedded-core/message/184115
Mute This Topic: https://lists.openembedded.org/mt/100074006/5827677
Group Owner: openembedded-core+owner@lists.openembedded.org<mailto:openembedded-core%2Bowner@lists.openembedded.org>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [rybczynska@gmail.com<mailto:rybczynska@gmail.com>]
-=-=-=-=-=-=-=-=-=-=-=-
Peter Marko July 11, 2023, 1:35 p.m. UTC | #3 
Maybe to complete my answer:
With current patch all 11 attempts would pass, but it was iterative approach adding more and more protections.
From the tries my conclusions would be:
- without any patch, there is 0% to get DB fetched (with current NVD infrastructure problems)
- with retrying all errors, 90% chance to get it
- with additionally increasing retry count I’d guess 99%

Peter

From: Marko, Peter (ADV D EU SK BFS1)
Sent: Tuesday, July 11, 2023 15:12
To: Marta Rybczynska <rybczynska@gmail.com>
Cc: openembedded-core@lists.openembedded.org
Subject: RE: [OE-core][master][mickledore][kirkstone][dunfell][PATCH 1/2] cve-update-nvd2-native: retry all errors and sleep between retries

Hello,

My testing was done with log increased to warning, and this is a typical outcome:
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=32000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=32000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=58000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=58000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=130000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=130000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=214000)
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database: received error (HTTP Error 503: Service Unavailable), retrying (request: https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=214000)

Out of 11 runs I had:
- 1 different failure but since at that time I was only catching 503s and thus it crashed I don’t have a cooker log, it was something that remote hang up without providing any data.
- 1 problem that 3 tries were not enough
- 9 times success
But maybe it also depends on region where your machine is doing the requests from or your time when you’re doing the run.
I think the first patch (catching all exceptions) is something really needed and the second one (5 retries) is something not very nice but probably needed for time being.
Possibly we could increase the sleep time instead of increasing retries; I’m open to suggestions as I’d like to see the changes merged soon so we can continue with vulnerability handling.

Regards,
  Peter

From: Marta Rybczynska <rybczynska@gmail.com<mailto:rybczynska@gmail.com>>
Sent: Tuesday, July 11, 2023 14:50
To: Marko, Peter (ADV D EU SK BFS1) <Peter.Marko@siemens.com<mailto:Peter.Marko@siemens.com>>
Cc: openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core][master][mickledore][kirkstone][dunfell][PATCH 1/2] cve-update-nvd2-native: retry all errors and sleep between retries

Thank you Peter for debugging this. Could you dump us a log of one of your typical runs to see what the errors are?
We might consider mirroring at some point.

Kind regards,
Marta

On Tue, Jul 11, 2023 at 8:37 AM Peter Marko via lists.openembedded.org<http://lists.openembedded.org> <peter.marko=siemens.com@lists.openembedded.org<mailto:siemens.com@lists.openembedded.org>> wrote:
From: Peter Marko <peter.marko@siemens.com<mailto:peter.marko@siemens.com>>

Last couple days it is not possible to update NVD DB as servers
are returning lot of errors.
Mostly "HTTP Error 503: Service Unavailable" is observed but
sporadially also some others.

Retrying helps in most cases, so extend retries to all errors.

Additionally add sleep which is recommended by NVD between requests.
These retries are already implemented between successful requests,
but giving servers time between failed ones is important, too.

Signed-off-by: Peter Marko <peter.marko@siemens.com<mailto:peter.marko@siemens.com>>
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb> | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb> b/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb>
index 4585126f73..a7392405e0 100644
--- a/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb>
+++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb<http://cve-update-nvd2-native.bb>
@@ -119,6 +119,7 @@ def nvd_request_next(url, api_key, args):
     import urllib.parse
     import gzip
     import http
+    import time

     headers = {}
     if api_key:
@@ -140,13 +141,9 @@ def nvd_request_next(url, api_key, args):

             r.close()

-        except UnicodeDecodeError:
-            # Received garbage, retry
-            bb.debug(2, "CVE database: received malformed data, retrying (request: %s)" %(full_request))
-            pass
-        except http.client.IncompleteRead:
-            # Read incomplete, let's try again
-            bb.debug(2, "CVE database: received incomplete data, retrying (request: %s)" %(full_request))
+        except Exception as e:
+            bb.debug(2, "CVE database: received error (%s), retrying (request: %s)" %(e, full_request))
+            time.sleep(6)
             pass
         else:
             return raw_data
--
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184115): https://lists.openembedded.org/g/openembedded-core/message/184115
Mute This Topic: https://lists.openembedded.org/mt/100074006/5827677
Group Owner: openembedded-core+owner@lists.openembedded.org<mailto:openembedded-core%2Bowner@lists.openembedded.org>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [rybczynska@gmail.com<mailto:rybczynska@gmail.com>]
-=-=-=-=-=-=-=-=-=-=-=-
diff mbox series

Patch

diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
index 4585126f73..a7392405e0 100644
--- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
+++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
@@ -119,6 +119,7 @@  def nvd_request_next(url, api_key, args):
     import urllib.parse
     import gzip
     import http
+    import time
 
     headers = {}
     if api_key:
@@ -140,13 +141,9 @@  def nvd_request_next(url, api_key, args):
 
             r.close()
 
-        except UnicodeDecodeError:
-            # Received garbage, retry
-            bb.debug(2, "CVE database: received malformed data, retrying (request: %s)" %(full_request))
-            pass
-        except http.client.IncompleteRead:
-            # Read incomplete, let's try again
-            bb.debug(2, "CVE database: received incomplete data, retrying (request: %s)" %(full_request))
+        except Exception as e:
+            bb.debug(2, "CVE database: received error (%s), retrying (request: %s)" %(e, full_request))
+            time.sleep(6)
             pass
         else:
             return raw_data