From patchwork Tue Jul 11 06:36:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Marko, Peter" X-Patchwork-Id: 27169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FCEBEB64DC for ; Tue, 11 Jul 2023 06:37:37 +0000 (UTC) Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net [185.136.64.226]) by mx.groups.io with SMTP id smtpd.web11.1755.1689057454345880625 for ; Mon, 10 Jul 2023 23:37:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=peter.marko@siemens.com header.s=fm1 header.b=AYO29AZ3; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.226, mailfrom: fm-256628-20230711063730edffb6182a24485b00-ivzkrt@rts-flowmailer.siemens.com) Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 20230711063730edffb6182a24485b00 for ; Tue, 11 Jul 2023 08:37:31 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=peter.marko@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; bh=KKwzmrxzN+TcxPFpzgbn8wA62VGyMW0FS9rCjh8+PMw=; b=AYO29AZ3sSXb6EdOoZBJs+NKkCA8AzylcozqRyzPhRKFNA0q7/rOxb4OAwcBgKit6nx6PI oLEkKXuvAczzIwyvb/XeyTXIyoCL71Wqaue03NcQ6Dh9f21uGf82RBsWLg2HM61qdsQ3i2c5 jtaJFAzdwdUMb5k849rOheAbJa3a8=; From: Peter Marko To: openembedded-core@lists.openembedded.org Cc: Peter Marko Subject: [OE-core][master][mickledore][kirkstone][dunfell][PATCH 1/2] cve-update-nvd2-native: retry all errors and sleep between retries Date: Tue, 11 Jul 2023 08:36:28 +0200 Message-Id: <20230711063629.7627-1-peter.marko@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-256628:519-21489:flowmailer List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 11 Jul 2023 06:37:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/184115 From: Peter Marko Last couple days it is not possible to update NVD DB as servers are returning lot of errors. Mostly "HTTP Error 503: Service Unavailable" is observed but sporadially also some others. Retrying helps in most cases, so extend retries to all errors. Additionally add sleep which is recommended by NVD between requests. These retries are already implemented between successful requests, but giving servers time between failed ones is important, too. Signed-off-by: Peter Marko Signed-off-by: Peter Marko > Signed-off-by: Peter Marko > --- meta/recipes-core/meta/cve-update-nvd2-native.bb | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb index 4585126f73..a7392405e0 100644 --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb @@ -119,6 +119,7 @@ def nvd_request_next(url, api_key, args): import urllib.parse import gzip import http + import time headers = {} if api_key: @@ -140,13 +141,9 @@ def nvd_request_next(url, api_key, args): r.close() - except UnicodeDecodeError: - # Received garbage, retry - bb.debug(2, "CVE database: received malformed data, retrying (request: %s)" %(full_request)) - pass - except http.client.IncompleteRead: - # Read incomplete, let's try again - bb.debug(2, "CVE database: received incomplete data, retrying (request: %s)" %(full_request)) + except Exception as e: + bb.debug(2, "CVE database: received error (%s), retrying (request: %s)" %(e, full_request)) + time.sleep(6) pass else: return raw_data