Patchwork [bitbake-devel,1/1] bitbake/persist_data: Reconnect when DB is locked

login
register
mail settings
Submitter Lianhao Lu
Date Jan. 16, 2012, 4:07 a.m.
Message ID <554ae4c9ee0ac70254c6ca6a5ba44db844cf86b3.1326686612.git.lianhao.lu@intel.com>
Download mbox | patch
Permalink /patch/19413/
State New
Headers show

Comments

Lianhao Lu - Jan. 16, 2012, 4:07 a.m.
[YOCTO #1761]
Reconnect to the backend Sqlite DB in 'database is locked' exception so
the timeout can be leveraged in each time retry.

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
 bitbake/lib/bb/persist_data.py |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)
Richard Purdie - Jan. 19, 2012, 12:16 a.m.
On Mon, 2012-01-16 at 12:07 +0800, Lianhao Lu wrote:
> [YOCTO #1761]
> Reconnect to the backend Sqlite DB in 'database is locked' exception so
> the timeout can be leveraged in each time retry.
> 
> Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
> ---
>  bitbake/lib/bb/persist_data.py |   12 +++++++-----
>  1 files changed, 7 insertions(+), 5 deletions(-)

Merged to master, thanks. Good to finally have a handle on why this kept
coming up! :)

Cheers,

Richard
McClintock Matthew-B29882 - May 31, 2012, 12:33 a.m.
This needs to be in edison-next. It cherry-pick's cleanly as an added
plus (doing more testing now). Seems like it's already in denzil-next

-M

On Sun, Jan 15, 2012 at 10:07 PM, Lianhao Lu <lianhao.lu@intel.com> wrote:
> [YOCTO #1761]
> Reconnect to the backend Sqlite DB in 'database is locked' exception so
> the timeout can be leveraged in each time retry.
>
> Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
> ---
>  bitbake/lib/bb/persist_data.py |   12 +++++++-----
>  1 files changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
> index 17620ef..c4ea23b 100644
> --- a/bitbake/lib/bb/persist_data.py
> +++ b/bitbake/lib/bb/persist_data.py
> @@ -47,9 +47,10 @@ if hasattr(sqlite3, 'enable_shared_cache'):
>  @total_ordering
>  class SQLTable(collections.MutableMapping):
>     """Object representing a table/domain in the database"""
> -    def __init__(self, cursor, table):
> -        self.cursor = cursor
> +    def __init__(self, cachefile, table):
> +        self.cachefile = cachefile
>         self.table = table
> +        self.cursor = connect(self.cachefile)
>
>         self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);"
>                       % table)
> @@ -63,6 +64,8 @@ class SQLTable(collections.MutableMapping):
>             except sqlite3.OperationalError as exc:
>                 if 'database is locked' in str(exc) and count < 500:
>                     count = count + 1
> +                    self.cursor.close()
> +                    self.cursor = connect(self.cachefile)
>                     continue
>                 raise
>
> @@ -188,7 +191,7 @@ class PersistData(object):
>         del self.data[domain][key]
>
>  def connect(database):
> -    return sqlite3.connect(database, timeout=30, isolation_level=None)
> +    return sqlite3.connect(database, timeout=5, isolation_level=None)
>
>  def persist(domain, d):
>     """Convenience factory for SQLTable objects based upon metadata"""
> @@ -201,5 +204,4 @@ def persist(domain, d):
>
>     bb.utils.mkdirhier(cachedir)
>     cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3")
> -    connection = connect(cachefile)
> -    return SQLTable(connection, domain)
> +    return SQLTable(cachefile, domain)
> --
> 1.7.0.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Joshua Lock - May 31, 2012, 5:54 p.m.
On 30/05/12 17:33, McClintock Matthew-B29882 wrote:
> This needs to be in edison-next. It cherry-pick's cleanly as an added
> plus (doing more testing now). Seems like it's already in denzil-next

cherry-picked to my josh/edison branch on poky-contrib - I will try and 
ensure it makes it into the edison branch of the main repository promptly.

Cheers,
Joshua
McClintock Matthew-B29882 - May 31, 2012, 7:45 p.m.
On Thu, May 31, 2012 at 12:54 PM, Joshua Lock <josh@linux.intel.com> wrote:
> On 30/05/12 17:33, McClintock Matthew-B29882 wrote:
>>
>> This needs to be in edison-next. It cherry-pick's cleanly as an added
>> plus (doing more testing now). Seems like it's already in denzil-next
>
>
> cherry-picked to my josh/edison branch on poky-contrib - I will try and
> ensure it makes it into the edison branch of the main repository promptly.

This has resolved my issue and I've seen no new issues after quite a
bit of building.

-M

Patch

diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index 17620ef..c4ea23b 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -47,9 +47,10 @@  if hasattr(sqlite3, 'enable_shared_cache'):
 @total_ordering
 class SQLTable(collections.MutableMapping):
     """Object representing a table/domain in the database"""
-    def __init__(self, cursor, table):
-        self.cursor = cursor
+    def __init__(self, cachefile, table):
+        self.cachefile = cachefile
         self.table = table
+        self.cursor = connect(self.cachefile)
 
         self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);"
                       % table)
@@ -63,6 +64,8 @@  class SQLTable(collections.MutableMapping):
             except sqlite3.OperationalError as exc:
                 if 'database is locked' in str(exc) and count < 500:
                     count = count + 1
+                    self.cursor.close()
+                    self.cursor = connect(self.cachefile)
                     continue
                 raise
 
@@ -188,7 +191,7 @@  class PersistData(object):
         del self.data[domain][key]
 
 def connect(database):
-    return sqlite3.connect(database, timeout=30, isolation_level=None)
+    return sqlite3.connect(database, timeout=5, isolation_level=None)
 
 def persist(domain, d):
     """Convenience factory for SQLTable objects based upon metadata"""
@@ -201,5 +204,4 @@  def persist(domain, d):
 
     bb.utils.mkdirhier(cachedir)
     cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3")
-    connection = connect(cachefile)
-    return SQLTable(connection, domain)
+    return SQLTable(cachefile, domain)