diff mbox series

[1/2] cve-check: get_cve_info should open the database read-only

Message ID 20220223125432.2733396-1-ross.burton@arm.com
State Accepted, archived
Commit 8de517238f1f418d9af1ce312d99de04ce2e26fc
Headers show
Series [1/2] cve-check: get_cve_info should open the database read-only | expand

Commit Message

Ross Burton Feb. 23, 2022, 12:54 p.m. UTC 
All of the function in cve-check should open the database read-only, as
the only writer is the fetch task in cve-update-db.  However,
get_cve_info() was failing to do this, which might be causing locking
issues with sqlite.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/cve-check.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Konrad Weihmann March 3, 2022, 8:10 a.m. UTC | #1 
Sadly I encountered this here last night

NOTE: recipe cve-update-db-native-1.0-r0: task do_cve_check: Started
ERROR: cve-update-db-native-1.0-r0 do_cve_check: Error executing a 
python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
      0001:
  *** 0002:do_cve_check(d)
      0003:
File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno: 
104, function: do_cve_check
      0100:        try:
      0101:            patched_cves = get_patched_cves(d)
      0102:        except FileNotFoundError:
      0103:            bb.fatal("Failure in searching patches")
  *** 0104:        whitelisted, patched, unpatched = check_cves(d, 
patched_cves)
      0105:        if patched or unpatched:
      0106:            cve_data = get_cve_info(d, patched + unpatched)
      0107:            cve_write_data(d, patched, unpatched, 
whitelisted, cve_data)
      0108:    else:
File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno: 
198, function: check_cves
      0194:        else:
      0195:            vendor = "%"
      0196:
      0197:        # Find all relevant CVE IDs.
  *** 0198:        for cverow in conn.execute("SELECT DISTINCT ID FROM 
PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)):
      0199:            cve = cverow[0]
      0200:
      0201:            if cve in cve_whitelist:
      0202:                bb.note("%s-%s has been whitelisted for %s" % 
(product, pv, cve))
Exception: sqlite3.OperationalError: attempt to write a readonly database

So it seems your patch doesn't solve the issue



On 23.02.22 13:54, Ross Burton wrote:
> All of the function in cve-check should open the database read-only, as
> the only writer is the fetch task in cve-update-db.  However,
> get_cve_info() was failing to do this, which might be causing locking
> issues with sqlite.
> 
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>   meta/classes/cve-check.bbclass | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
> index 2d69aeba4b..d715fbf4d8 100644
> --- a/meta/classes/cve-check.bbclass
> +++ b/meta/classes/cve-check.bbclass
> @@ -265,7 +265,8 @@ def get_cve_info(d, cves):
>       import sqlite3
>   
>       cve_data = {}
> -    conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
> +    db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
> +    conn = sqlite3.connect(db_file, uri=True)
>   
>       for cve in cves:
>           for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)):
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#162258): https://lists.openembedded.org/g/openembedded-core/message/162258
> Mute This Topic: https://lists.openembedded.org/mt/89339702/3647476
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Ross Burton March 3, 2022, 11:07 a.m. UTC | #2 
What's the scenario where this is happening?  Multiple build
directories using the same DL_DIR? Or was this a single build?

Ross

On Thu, 3 Mar 2022 at 08:10, Konrad Weihmann <kweihmann@outlook.com> wrote:
>
> Sadly I encountered this here last night
>
> NOTE: recipe cve-update-db-native-1.0-r0: task do_cve_check: Started
> ERROR: cve-update-db-native-1.0-r0 do_cve_check: Error executing a
> python function in exec_func_python() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>       0001:
>   *** 0002:do_cve_check(d)
>       0003:
> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
> 104, function: do_cve_check
>       0100:        try:
>       0101:            patched_cves = get_patched_cves(d)
>       0102:        except FileNotFoundError:
>       0103:            bb.fatal("Failure in searching patches")
>   *** 0104:        whitelisted, patched, unpatched = check_cves(d,
> patched_cves)
>       0105:        if patched or unpatched:
>       0106:            cve_data = get_cve_info(d, patched + unpatched)
>       0107:            cve_write_data(d, patched, unpatched,
> whitelisted, cve_data)
>       0108:    else:
> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
> 198, function: check_cves
>       0194:        else:
>       0195:            vendor = "%"
>       0196:
>       0197:        # Find all relevant CVE IDs.
>   *** 0198:        for cverow in conn.execute("SELECT DISTINCT ID FROM
> PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)):
>       0199:            cve = cverow[0]
>       0200:
>       0201:            if cve in cve_whitelist:
>       0202:                bb.note("%s-%s has been whitelisted for %s" %
> (product, pv, cve))
> Exception: sqlite3.OperationalError: attempt to write a readonly database
>
> So it seems your patch doesn't solve the issue
>
>
>
> On 23.02.22 13:54, Ross Burton wrote:
> > All of the function in cve-check should open the database read-only, as
> > the only writer is the fetch task in cve-update-db.  However,
> > get_cve_info() was failing to do this, which might be causing locking
> > issues with sqlite.
> >
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> >   meta/classes/cve-check.bbclass | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
> > index 2d69aeba4b..d715fbf4d8 100644
> > --- a/meta/classes/cve-check.bbclass
> > +++ b/meta/classes/cve-check.bbclass
> > @@ -265,7 +265,8 @@ def get_cve_info(d, cves):
> >       import sqlite3
> >
> >       cve_data = {}
> > -    conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
> > +    db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
> > +    conn = sqlite3.connect(db_file, uri=True)
> >
> >       for cve in cves:
> >           for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)):
> >
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#162258): https://lists.openembedded.org/g/openembedded-core/message/162258
> > Mute This Topic: https://lists.openembedded.org/mt/89339702/3647476
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
Konrad Weihmann March 3, 2022, 11:10 a.m. UTC | #3 
Single from scratch build of core-image-minimal

On 03.03.22 12:07, Ross Burton wrote:
> What's the scenario where this is happening?  Multiple build
> directories using the same DL_DIR? Or was this a single build?
> 
> Ross
> 
> On Thu, 3 Mar 2022 at 08:10, Konrad Weihmann <kweihmann@outlook.com> wrote:
>>
>> Sadly I encountered this here last night
>>
>> NOTE: recipe cve-update-db-native-1.0-r0: task do_cve_check: Started
>> ERROR: cve-update-db-native-1.0-r0 do_cve_check: Error executing a
>> python function in exec_func_python() autogenerated:
>>
>> The stack trace of python calls that resulted in this exception/failure was:
>> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>>        0001:
>>    *** 0002:do_cve_check(d)
>>        0003:
>> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
>> 104, function: do_cve_check
>>        0100:        try:
>>        0101:            patched_cves = get_patched_cves(d)
>>        0102:        except FileNotFoundError:
>>        0103:            bb.fatal("Failure in searching patches")
>>    *** 0104:        whitelisted, patched, unpatched = check_cves(d,
>> patched_cves)
>>        0105:        if patched or unpatched:
>>        0106:            cve_data = get_cve_info(d, patched + unpatched)
>>        0107:            cve_write_data(d, patched, unpatched,
>> whitelisted, cve_data)
>>        0108:    else:
>> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
>> 198, function: check_cves
>>        0194:        else:
>>        0195:            vendor = "%"
>>        0196:
>>        0197:        # Find all relevant CVE IDs.
>>    *** 0198:        for cverow in conn.execute("SELECT DISTINCT ID FROM
>> PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)):
>>        0199:            cve = cverow[0]
>>        0200:
>>        0201:            if cve in cve_whitelist:
>>        0202:                bb.note("%s-%s has been whitelisted for %s" %
>> (product, pv, cve))
>> Exception: sqlite3.OperationalError: attempt to write a readonly database
>>
>> So it seems your patch doesn't solve the issue
>>
>>
>>
>> On 23.02.22 13:54, Ross Burton wrote:
>>> All of the function in cve-check should open the database read-only, as
>>> the only writer is the fetch task in cve-update-db.  However,
>>> get_cve_info() was failing to do this, which might be causing locking
>>> issues with sqlite.
>>>
>>> Signed-off-by: Ross Burton <ross.burton@arm.com>
>>> ---
>>>    meta/classes/cve-check.bbclass | 3 ++-
>>>    1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
>>> index 2d69aeba4b..d715fbf4d8 100644
>>> --- a/meta/classes/cve-check.bbclass
>>> +++ b/meta/classes/cve-check.bbclass
>>> @@ -265,7 +265,8 @@ def get_cve_info(d, cves):
>>>        import sqlite3
>>>
>>>        cve_data = {}
>>> -    conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
>>> +    db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
>>> +    conn = sqlite3.connect(db_file, uri=True)
>>>
>>>        for cve in cves:
>>>            for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)):
>>>
>>>
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-
>>> Links: You receive all messages sent to this group.
>>> View/Reply Online (#162258): https://lists.openembedded.org/g/openembedded-core/message/162258
>>> Mute This Topic: https://lists.openembedded.org/mt/89339702/3647476
>>> Group Owner: openembedded-core+owner@lists.openembedded.org
>>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com]
>>> -=-=-=-=-=-=-=-=-=-=-=-
>>>
Ross Burton March 3, 2022, 11:27 a.m. UTC | #4 
:tableflip:

Thanks, I guess!

Can you share the full bitbake log from that run? I'd like to see what
jobs are running in parallel.

Ross

On Thu, 3 Mar 2022 at 11:10, Konrad Weihmann <kweihmann@outlook.com> wrote:
>
> Single from scratch build of core-image-minimal
>
> On 03.03.22 12:07, Ross Burton wrote:
> > What's the scenario where this is happening?  Multiple build
> > directories using the same DL_DIR? Or was this a single build?
> >
> > Ross
> >
> > On Thu, 3 Mar 2022 at 08:10, Konrad Weihmann <kweihmann@outlook.com> wrote:
> >>
> >> Sadly I encountered this here last night
> >>
> >> NOTE: recipe cve-update-db-native-1.0-r0: task do_cve_check: Started
> >> ERROR: cve-update-db-native-1.0-r0 do_cve_check: Error executing a
> >> python function in exec_func_python() autogenerated:
> >>
> >> The stack trace of python calls that resulted in this exception/failure was:
> >> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
> >>        0001:
> >>    *** 0002:do_cve_check(d)
> >>        0003:
> >> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
> >> 104, function: do_cve_check
> >>        0100:        try:
> >>        0101:            patched_cves = get_patched_cves(d)
> >>        0102:        except FileNotFoundError:
> >>        0103:            bb.fatal("Failure in searching patches")
> >>    *** 0104:        whitelisted, patched, unpatched = check_cves(d,
> >> patched_cves)
> >>        0105:        if patched or unpatched:
> >>        0106:            cve_data = get_cve_info(d, patched + unpatched)
> >>        0107:            cve_write_data(d, patched, unpatched,
> >> whitelisted, cve_data)
> >>        0108:    else:
> >> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
> >> 198, function: check_cves
> >>        0194:        else:
> >>        0195:            vendor = "%"
> >>        0196:
> >>        0197:        # Find all relevant CVE IDs.
> >>    *** 0198:        for cverow in conn.execute("SELECT DISTINCT ID FROM
> >> PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)):
> >>        0199:            cve = cverow[0]
> >>        0200:
> >>        0201:            if cve in cve_whitelist:
> >>        0202:                bb.note("%s-%s has been whitelisted for %s" %
> >> (product, pv, cve))
> >> Exception: sqlite3.OperationalError: attempt to write a readonly database
> >>
> >> So it seems your patch doesn't solve the issue
> >>
> >>
> >>
> >> On 23.02.22 13:54, Ross Burton wrote:
> >>> All of the function in cve-check should open the database read-only, as
> >>> the only writer is the fetch task in cve-update-db.  However,
> >>> get_cve_info() was failing to do this, which might be causing locking
> >>> issues with sqlite.
> >>>
> >>> Signed-off-by: Ross Burton <ross.burton@arm.com>
> >>> ---
> >>>    meta/classes/cve-check.bbclass | 3 ++-
> >>>    1 file changed, 2 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
> >>> index 2d69aeba4b..d715fbf4d8 100644
> >>> --- a/meta/classes/cve-check.bbclass
> >>> +++ b/meta/classes/cve-check.bbclass
> >>> @@ -265,7 +265,8 @@ def get_cve_info(d, cves):
> >>>        import sqlite3
> >>>
> >>>        cve_data = {}
> >>> -    conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
> >>> +    db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
> >>> +    conn = sqlite3.connect(db_file, uri=True)
> >>>
> >>>        for cve in cves:
> >>>            for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)):
> >>>
> >>>
> >>>
> >>> -=-=-=-=-=-=-=-=-=-=-=-
> >>> Links: You receive all messages sent to this group.
> >>> View/Reply Online (#162258): https://lists.openembedded.org/g/openembedded-core/message/162258
> >>> Mute This Topic: https://lists.openembedded.org/mt/89339702/3647476
> >>> Group Owner: openembedded-core+owner@lists.openembedded.org
> >>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com]
> >>> -=-=-=-=-=-=-=-=-=-=-=-
> >>>
Konrad Weihmann March 3, 2022, 11:43 a.m. UTC | #5 
This one is odd...

After scrolling through the log that I can get from a github pipeline 
(sorry no runner log available) - I noticed

2022-03-03T05:39:23.2334189Z WARNING: cve-update-db-native-1.0-r0 
do_fetch: Failed to fetch CVE data ([Errno 99] Cannot assign requested 
address)
2022-03-03T05:39:23.2569941Z NOTE: recipe cve-update-db-native-1.0-r0: 
task do_fetch: Succeeded

and then do_cve_check run on that recipe

2022-03-03T05:39:56.6239005Z NOTE: Running task 4428 of 5120 
(/opt/build/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb:do_cve_check)

before failing with 2022-03-03T05:39:56.8063851Z Exception: 
sqlite3.OperationalError: attempt to write a readonly database

this doesn't look right to me - maybe there's something special about my 
settings - I will try to recreate that locally

On 03.03.22 12:27, Ross Burton wrote:
> :tableflip:
> 
> Thanks, I guess!
> 
> Can you share the full bitbake log from that run? I'd like to see what
> jobs are running in parallel.
> 
> Ross
> 
> On Thu, 3 Mar 2022 at 11:10, Konrad Weihmann <kweihmann@outlook.com> wrote:
>>
>> Single from scratch build of core-image-minimal
>>
>> On 03.03.22 12:07, Ross Burton wrote:
>>> What's the scenario where this is happening?  Multiple build
>>> directories using the same DL_DIR? Or was this a single build?
>>>
>>> Ross
>>>
>>> On Thu, 3 Mar 2022 at 08:10, Konrad Weihmann <kweihmann@outlook.com> wrote:
>>>>
>>>> Sadly I encountered this here last night
>>>>
>>>> NOTE: recipe cve-update-db-native-1.0-r0: task do_cve_check: Started
>>>> ERROR: cve-update-db-native-1.0-r0 do_cve_check: Error executing a
>>>> python function in exec_func_python() autogenerated:
>>>>
>>>> The stack trace of python calls that resulted in this exception/failure was:
>>>> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>>>>         0001:
>>>>     *** 0002:do_cve_check(d)
>>>>         0003:
>>>> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
>>>> 104, function: do_cve_check
>>>>         0100:        try:
>>>>         0101:            patched_cves = get_patched_cves(d)
>>>>         0102:        except FileNotFoundError:
>>>>         0103:            bb.fatal("Failure in searching patches")
>>>>     *** 0104:        whitelisted, patched, unpatched = check_cves(d,
>>>> patched_cves)
>>>>         0105:        if patched or unpatched:
>>>>         0106:            cve_data = get_cve_info(d, patched + unpatched)
>>>>         0107:            cve_write_data(d, patched, unpatched,
>>>> whitelisted, cve_data)
>>>>         0108:    else:
>>>> File: '/opt/build/sources/poky/meta/classes/cve-check.bbclass', lineno:
>>>> 198, function: check_cves
>>>>         0194:        else:
>>>>         0195:            vendor = "%"
>>>>         0196:
>>>>         0197:        # Find all relevant CVE IDs.
>>>>     *** 0198:        for cverow in conn.execute("SELECT DISTINCT ID FROM
>>>> PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)):
>>>>         0199:            cve = cverow[0]
>>>>         0200:
>>>>         0201:            if cve in cve_whitelist:
>>>>         0202:                bb.note("%s-%s has been whitelisted for %s" %
>>>> (product, pv, cve))
>>>> Exception: sqlite3.OperationalError: attempt to write a readonly database
>>>>
>>>> So it seems your patch doesn't solve the issue
>>>>
>>>>
>>>>
>>>> On 23.02.22 13:54, Ross Burton wrote:
>>>>> All of the function in cve-check should open the database read-only, as
>>>>> the only writer is the fetch task in cve-update-db.  However,
>>>>> get_cve_info() was failing to do this, which might be causing locking
>>>>> issues with sqlite.
>>>>>
>>>>> Signed-off-by: Ross Burton <ross.burton@arm.com>
>>>>> ---
>>>>>     meta/classes/cve-check.bbclass | 3 ++-
>>>>>     1 file changed, 2 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
>>>>> index 2d69aeba4b..d715fbf4d8 100644
>>>>> --- a/meta/classes/cve-check.bbclass
>>>>> +++ b/meta/classes/cve-check.bbclass
>>>>> @@ -265,7 +265,8 @@ def get_cve_info(d, cves):
>>>>>         import sqlite3
>>>>>
>>>>>         cve_data = {}
>>>>> -    conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
>>>>> +    db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
>>>>> +    conn = sqlite3.connect(db_file, uri=True)
>>>>>
>>>>>         for cve in cves:
>>>>>             for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)):
>>>>>
>>>>>
>>>>>
>>>>> -=-=-=-=-=-=-=-=-=-=-=-
>>>>> Links: You receive all messages sent to this group.
>>>>> View/Reply Online (#162258): https://lists.openembedded.org/g/openembedded-core/message/162258
>>>>> Mute This Topic: https://lists.openembedded.org/mt/89339702/3647476
>>>>> Group Owner: openembedded-core+owner@lists.openembedded.org
>>>>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com]
>>>>> -=-=-=-=-=-=-=-=-=-=-=-
>>>>>
Ross Burton March 21, 2022, 1:43 p.m. UTC | #6 
On Thu, 3 Mar 2022 at 11:43, Konrad Weihmann <kweihmann@outlook.com> wrote:
> After scrolling through the log that I can get from a github pipeline
> (sorry no runner log available) - I noticed
>
> 2022-03-03T05:39:23.2334189Z WARNING: cve-update-db-native-1.0-r0
> do_fetch: Failed to fetch CVE data ([Errno 99] Cannot assign requested
> address)
> 2022-03-03T05:39:23.2569941Z NOTE: recipe cve-update-db-native-1.0-r0:
> task do_fetch: Succeeded

That's... special.

> and then do_cve_check run on that recipe
>
> 2022-03-03T05:39:56.6239005Z NOTE: Running task 4428 of 5120
> (/opt/build/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb:do_cve_check)
>
> before failing with 2022-03-03T05:39:56.8063851Z Exception:
> sqlite3.OperationalError: attempt to write a readonly database
>
> this doesn't look right to me - maybe there's something special about my
> settings - I will try to recreate that locally

Have you managed to reproduce this failure since?

Ross
Konrad Weihmann March 21, 2022, 4:55 p.m. UTC | #7 
On 21.03.22 14:43, Ross Burton wrote:
> On Thu, 3 Mar 2022 at 11:43, Konrad Weihmann <kweihmann@outlook.com> wrote:
>> After scrolling through the log that I can get from a github pipeline
>> (sorry no runner log available) - I noticed
>>
>> 2022-03-03T05:39:23.2334189Z WARNING: cve-update-db-native-1.0-r0
>> do_fetch: Failed to fetch CVE data ([Errno 99] Cannot assign requested
>> address)
>> 2022-03-03T05:39:23.2569941Z NOTE: recipe cve-update-db-native-1.0-r0:
>> task do_fetch: Succeeded
> 
> That's... special.
> 
>> and then do_cve_check run on that recipe
>>
>> 2022-03-03T05:39:56.6239005Z NOTE: Running task 4428 of 5120
>> (/opt/build/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb:do_cve_check)
>>
>> before failing with 2022-03-03T05:39:56.8063851Z Exception:
>> sqlite3.OperationalError: attempt to write a readonly database
>>
>> this doesn't look right to me - maybe there's something special about my
>> settings - I will try to recreate that locally
> 
> Have you managed to reproduce this failure since?

I tried, but without success - but I did a bit of reading on the 
reported Error 99, which seems to be related to using containers (in the 
pipeline itself a container based on ubuntu 18.04 is used) to run the build.

looking at the code block from cve-update-db-native:do_fetch

             try:
                 response = urllib.request.urlopen(meta_url)
             except urllib.error.URLError as e:
                 cve_f.write('Warning: CVE db update error, Unable to 
fetch CVE data.\n\n')
                 bb.warn("Failed to fetch CVE data (%s)" % e.reason)
                 return

I'm not a 100% sure, if the sudden return doesn't leave the 
pre-populated DB file in a limbo state.
I at least would expect a flush + close of the DB file here, or even 
better a recreation of an empty DB + flush and close - but I'm not sure 
what to do in the case the network stack of the OS fails us

But as I mentioned a couple of times, this issue is hard to reproduce 
and just hits me in on those mentioned GH pipelines every now and then.

BTW just one idea for improvement would be to globally exclude 
cve-update-db-native from running do_cve_check... as this doesn't make 
much sense to me - guess setting CVE_CHECK_SKIP_RECIPE = 
"cve-update-db-native" as a sane default in cve-check class should do it 
- any thoughts?

> 
> Ross
diff mbox series

Patch

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 2d69aeba4b..d715fbf4d8 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -265,7 +265,8 @@  def get_cve_info(d, cves):
     import sqlite3
 
     cve_data = {}
-    conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE"))
+    db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
+    conn = sqlite3.connect(db_file, uri=True)
 
     for cve in cves:
         for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)):