From patchwork Tue Apr 30 17:15:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42980 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2A46C19F4F for ; Tue, 30 Apr 2024 17:15:31 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web10.21659.1714497330150562573 for ; Tue, 30 Apr 2024 10:15:30 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=mfgEHmBp; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 3850AC0003; Tue, 30 Apr 2024 17:15:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497328; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8BiNHWZedTQUDworzQJJA7RM4WypooYTjRHh0Fw20Nw=; b=mfgEHmBpnEvVgpHtTiweGKznAOHskxW+WKqUMXLTxTY1jZ+7lqPJPgJTW+HHrE6hLJ6gEl VD9jdJaTgatHOaZ3+k/yeugjog9u/DeJi4dmD73b7uPG1Vn8x5al29NlOrSgwPtlZR3A63 1yQlmQYTtsHXY9STiOZJ9SUEuRRSDaa/DojyXd9wpU0eJoxY4+uMOEdDzmJFWACWoOyA8w vdHFEOav91OqwdFr8X7fR5V6P0/wdhjHCyu5zdmbtPeJFXEJ8o4D4WYEXKDjcjE01y3q4+ KOyf8hssGQz/TMK6zrfut6oTUEPUoy1XIwLYLP7MEPDzcF0rBd0KyAEDEG5WGg== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 1/8] prserv: declare "max_package_pr" client hook Date: Tue, 30 Apr 2024 19:15:05 +0200 Message-Id: <20240430171512.936371-2-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16166 From: Michael Opdenacker Add missing declaration for the max_package_pr client hook Signed-off-by: Michael Opdenacker Cc: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni --- lib/prserv/client.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/prserv/client.py b/lib/prserv/client.py index 8471ee3046..99fc4e0f7f 100644 --- a/lib/prserv/client.py +++ b/lib/prserv/client.py @@ -65,7 +65,7 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): class PRClient(bb.asyncrpc.Client): def __init__(self): super().__init__() - self._add_methods("getPR", "test_pr", "test_package", "importone", "export", "is_readonly") + self._add_methods("getPR", "test_pr", "test_package", "max_package_pr", "importone", "export", "is_readonly") def _get_async_client(self): return PRAsyncClient() From patchwork Tue Apr 30 17:15:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9A31C4345F for ; Tue, 30 Apr 2024 17:15:41 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web11.21514.1714497331527341917 for ; Tue, 30 Apr 2024 10:15:31 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=X3G+yQp+; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 7018060002; Tue, 30 Apr 2024 17:15:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497329; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A4H02KuFgkkWy+56xbW5//7dIiLLkpHSXOkgl650G/w=; b=X3G+yQp+cigxHQhHlAdj/+MyEOPVezHTI4UtWpXEpmP59UPH/uKaGX0dvBzG1Kor1VCPM+ 3zFVtHU/s1zeP/vbTJn+TX3Uu7Imy778YRSAXOISR3PA/bWyXApBLjPo/gO/AbdFHJqXSz IyMO6fG+tVOYjujOZUWMKQtz483Fm/LYtdiaH8EZy13NSsxMCFM/AEYon++tM6L59wn4Ut iZUvJBzfJmPLpTVJUdxnUya1g6jNOb0Do1Z6s3w3j5B28V0VP1QuVBXqv7qXnVshzWHlUx NBfzJSUU4BUaXKua85Y/X77iZvcws1459kHHSqdM3Cn2lHJBDalvDNTcQWlg5g== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 2/8] prserv: move code from __init__ to bitbake-prserv Date: Tue, 30 Apr 2024 19:15:06 +0200 Message-Id: <20240430171512.936371-3-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16167 From: Michael Opdenacker This script was the only user of this code. Signed-off-by: Michael Opdenacker Cc: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni --- bin/bitbake-prserv | 9 ++++++++- lib/prserv/__init__.py | 13 ------------- 2 files changed, 8 insertions(+), 14 deletions(-) diff --git a/bin/bitbake-prserv b/bin/bitbake-prserv index ad0a069401..920663a1d8 100755 --- a/bin/bitbake-prserv +++ b/bin/bitbake-prserv @@ -21,6 +21,13 @@ VERSION = "1.1.0" PRHOST_DEFAULT="0.0.0.0" PRPORT_DEFAULT=8585 +def init_logger(logfile, loglevel): + numeric_level = getattr(logging, loglevel.upper(), None) + if not isinstance(numeric_level, int): + raise ValueError("Invalid log level: %s" % loglevel) + FORMAT = "%(asctime)-15s %(message)s" + logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT) + def main(): parser = argparse.ArgumentParser( description="BitBake PR Server. Version=%s" % VERSION, @@ -72,7 +79,7 @@ def main(): ) args = parser.parse_args() - prserv.init_logger(os.path.abspath(args.log), args.loglevel) + init_logger(os.path.abspath(args.log), args.loglevel) if args.start: ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) diff --git a/lib/prserv/__init__.py b/lib/prserv/__init__.py index 0e0aa34d0e..94658b815d 100644 --- a/lib/prserv/__init__.py +++ b/lib/prserv/__init__.py @@ -5,16 +5,3 @@ # __version__ = "1.0.0" - -import os, time -import sys, logging - -def init_logger(logfile, loglevel): - numeric_level = getattr(logging, loglevel.upper(), None) - if not isinstance(numeric_level, int): - raise ValueError("Invalid log level: %s" % loglevel) - FORMAT = "%(asctime)-15s %(message)s" - logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT) - -class NotFoundError(Exception): - pass From patchwork Tue Apr 30 17:15:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4299C41513 for ; Tue, 30 Apr 2024 17:15:41 +0000 (UTC) Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by mx.groups.io with SMTP id smtpd.web10.21661.1714497333144768844 for ; Tue, 30 Apr 2024 10:15:33 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=G5cHvMb8; spf=pass (domain: bootlin.com, ip: 217.70.183.199, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id C3E99FF807; Tue, 30 Apr 2024 17:15:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497331; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1JnAyTYs4YAapjKqoVSrW1H27wfA076ofoRN/O5Qoes=; b=G5cHvMb8u6Hs4LHiZz8nCcd9r2+QgmEk4SyF9ZIOOhtyvvdygnfALRrR3MGgbpz/sBCe2j WVxhnD4qPNmqymlT21YEuoTFylrUpdFDShOyqw00mRiU2KwjTLrqNvKlpkOCXOujyCQniu MyB4FdEwh1wpnqr79cS1xYZFI2BDXAgeTwbvmzRh7dySBx1IZKCUWlthMpAUrCJdfSudMC 25xIjQFkN3rcurxMTNdmgdsFGLtyqTJp6PhDXY5q72G6MrFdxfwMcTaCwdQoAcBdBicWj8 EHrpknKsndJRZI/wJ2X2OeZefgUXi6MJVMSiu0wzdAIIJE2ZsDhKUCfvokExNA== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 3/8] prserv: add "upstream" server support Date: Tue, 30 Apr 2024 19:15:07 +0200 Message-Id: <20240430171512.936371-4-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16168 From: Michael Opdenacker Introduce a PRSERVER_UPSTREAM variable that makes the local PR server connect to an "upstream" one. This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y"). Update the calculation of the new revisions to support the case when prior revisions are not integers, but have an "x.y..." format." Set the comments in the handle_get_pr() function in serv.py for details about the calculation of the local revision. This is done by going on supporting the "history" mode that wasn't used so far (revisions can return to a previous historical value), in addition to the default "no history" mode (revisions can never decrease). Rather than storing the history mode in the database table itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode is now passed through the client requests. As a consequence, the table name is now "PRMAIN", which is incompatible with what was generated before, but avoids confusion if we kept the "PRMAIN_nohist" name for both "history" and "no history" modes. Update the server version to "2.0.0". Signed-off-by: Michael Opdenacker Cc: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni --- bin/bitbake-prserv | 17 +++- lib/prserv/__init__.py | 90 +++++++++++++++++- lib/prserv/client.py | 17 ++-- lib/prserv/db.py | 205 ++++++++++++++++++++++------------------- lib/prserv/serv.py | 134 +++++++++++++++++++++++---- 5 files changed, 338 insertions(+), 125 deletions(-) diff --git a/bin/bitbake-prserv b/bin/bitbake-prserv index 920663a1d8..580e021fda 100755 --- a/bin/bitbake-prserv +++ b/bin/bitbake-prserv @@ -16,7 +16,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib import prserv import prserv.serv -VERSION = "1.1.0" +VERSION = "2.0.0" PRHOST_DEFAULT="0.0.0.0" PRPORT_DEFAULT=8585 @@ -77,12 +77,25 @@ def main(): action="store_true", help="open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=os.environ.get("PRSERVER_UPSTREAM", None), + help="Upstream PR service (host:port)", + ) args = parser.parse_args() init_logger(os.path.abspath(args.log), args.loglevel) if args.start: - ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + ret=prserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=prserv.serv.stop_daemon(args.host, args.port) else: diff --git a/lib/prserv/__init__.py b/lib/prserv/__init__.py index 94658b815d..a817b03c1e 100644 --- a/lib/prserv/__init__.py +++ b/lib/prserv/__init__.py @@ -4,4 +4,92 @@ # SPDX-License-Identifier: GPL-2.0-only # -__version__ = "1.0.0" + +__version__ = "2.0.0" + +import logging +logger = logging.getLogger("BitBake.PRserv") + +from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS + +def create_server(addr, dbpath, upstream=None, read_only=False): + from . import serv + + s = serv.PRServer(dbpath, upstream=upstream, read_only=read_only) + host, port = addr.split(":") + s.start_tcp_server(host, int(port)) + + return s + +def increase_revision(ver): + """Take a revision string such as "1" or "1.2.3" or even a number and increase its last number + This fails if the last number is not an integer""" + + fields=str(ver).split('.') + last = fields[-1] + + try: + val = int(last) + except Exception as e: + logger.critical("Unable to increase revision value %s: %s" % (ver, e)) + raise e + + return ".".join(fields[0:-1] + list(str(val + 1))) + +def _revision_greater_or_equal(rev1, rev2): + """Compares x.y.z revision numbers, using integer comparison + Returns True if rev1 is greater or equal to rev2""" + + fields1 = rev1.split(".") + fields2 = rev2.split(".") + l1 = len(fields1) + l2 = len(fields2) + + for i in range(l1): + val1 = int(fields1[i]) + if i < l2: + val2 = int(fields2[i]) + if val2 < val1: + return True + elif val2 > val1: + return False + else: + return True + return True + +def revision_smaller(rev1, rev2): + """Compares x.y.z revision numbers, using integer comparison + Returns True if rev1 is strictly smaller than rev2""" + return not(_revision_greater_or_equal(rev1, rev2)) + +def revision_greater(rev1, rev2): + """Compares x.y.z revision numbers, using integer comparison + Returns True if rev1 is strictly greater than rev2""" + return _revision_greater_or_equal(rev1, rev2) and (rev1 != rev2) + +def create_client(addr): + from . import client + + c = client.PRClient() + + try: + (typ, a) = parse_address(addr) + c.connect_tcp(*a) + return c + except Exception as e: + c.close() + raise e + +async def create_async_client(addr): + from . import client + + c = client.PRAsyncClient() + + try: + (typ, a) = parse_address(addr) + await c.connect_tcp(*a) + return c + + except Exception as e: + await c.close() + raise e diff --git a/lib/prserv/client.py b/lib/prserv/client.py index 99fc4e0f7f..565c6f3872 100644 --- a/lib/prserv/client.py +++ b/lib/prserv/client.py @@ -6,6 +6,7 @@ import logging import bb.asyncrpc +from . import create_async_client logger = logging.getLogger("BitBake.PRserv") @@ -13,16 +14,16 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): def __init__(self): super().__init__("PRSERVICE", "1.0", logger) - async def getPR(self, version, pkgarch, checksum): + async def getPR(self, version, pkgarch, checksum, history=False): response = await self.invoke( - {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}} ) if response: return response["value"] - async def test_pr(self, version, pkgarch, checksum): + async def test_pr(self, version, pkgarch, checksum, history=False): response = await self.invoke( - {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}} ) if response: return response["value"] @@ -41,16 +42,16 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): if response: return response["value"] - async def importone(self, version, pkgarch, checksum, value): + async def importone(self, version, pkgarch, checksum, value, history=False): response = await self.invoke( - {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value}} + {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value, "history": history}} ) if response: return response["value"] - async def export(self, version, pkgarch, checksum, colinfo): + async def export(self, version, pkgarch, checksum, colinfo, history=False): response = await self.invoke( - {"export": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "colinfo": colinfo}} + {"export": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "colinfo": colinfo, "history": history}} ) if response: return (response["metainfo"], response["datainfo"]) diff --git a/lib/prserv/db.py b/lib/prserv/db.py index eb41508198..b2520f3158 100644 --- a/lib/prserv/db.py +++ b/lib/prserv/db.py @@ -10,6 +10,8 @@ import errno import prserv import time +from . import increase_revision, revision_greater, revision_smaller + try: import sqlite3 except ImportError: @@ -32,15 +34,11 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): # class PRTable(object): - def __init__(self, conn, table, nohist, read_only): + def __init__(self, conn, table, read_only): self.conn = conn - self.nohist = nohist self.read_only = read_only self.dirty = False - if nohist: - self.table = "%s_nohist" % table - else: - self.table = "%s_hist" % table + self.table = table if self.read_only: table_exists = self._execute( @@ -53,8 +51,8 @@ class PRTable(object): (version TEXT NOT NULL, \ pkgarch TEXT NOT NULL, \ checksum TEXT NOT NULL, \ - value INTEGER, \ - PRIMARY KEY (version, pkgarch, checksum));" % self.table) + value TEXT, \ + PRIMARY KEY (version, pkgarch, checksum, value));" % self.table) def _execute(self, *query): """Execute a query, waiting to acquire a lock if necessary""" @@ -68,6 +66,28 @@ class PRTable(object): continue raise exc + def _extremum_value(self, rows, is_max): + value = None + + for row in rows: + current_value = row[0] + if value is None: + value = current_value + else: + if is_max: + is_new_extremum = revision_greater(current_value, value) + else: + is_new_extremum = revision_smaller(current_value, value) + if is_new_extremum: + value = current_value + return value + + def _max_value(self, rows): + return self._extremum_value(rows, True) + + def _min_value(self, rows): + return self._extremum_value(rows, False) + def sync(self): if not self.read_only: self.conn.commit() @@ -102,101 +122,93 @@ class PRTable(object): else: return False - def find_value(self, version, pkgarch, checksum): + + def find_package_max_value(self, version, pkgarch): + """Returns the greatest value for (version, pkgarch), or None if not found. Doesn't create a new value""" + + data = self._execute("SELECT value FROM %s where version=? AND pkgarch=?;" % (self.table), + (version, pkgarch)) + rows = data.fetchall() + value = self._max_value(rows) + return value + + def find_value(self, version, pkgarch, checksum, history=False): """Returns the value for the specified checksum if found or None otherwise.""" - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] + if history: + return self.find_min_value(version, pkgarch, checksum) else: - return None + return self.find_max_value(version, pkgarch, checksum) - def find_max_value(self, version, pkgarch): - """Returns the greatest value for (version, pkgarch), or None if not found. Doesn't create a new value""" - data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=?;" % (self.table), + def _find_extremum_value(self, version, pkgarch, checksum, is_max): + """Returns the maximum (if is_max is True) or minimum (if is_max is False) value + for (version, pkgarch, checksum), or None if not found. Doesn't create a new value""" + + data = self._execute("SELECT value FROM %s where version=? AND pkgarch=? AND checksum=?;" % (self.table), + (version, pkgarch, checksum)) + rows = data.fetchall() + return self._extremum_value(rows, is_max) + + def find_max_value(self, version, pkgarch, checksum): + return self._find_extremum_value(version, pkgarch, checksum, True) + + def find_min_value(self, version, pkgarch, checksum): + return self._find_extremum_value(version, pkgarch, checksum, False) + + def find_new_subvalue(self, version, pkgarch, base): + """Take and increase the greatest ".y" value for (version, pkgarch), or return ".0" if not found. + This doesn't store a new value.""" + + data = self._execute("SELECT value FROM %s where version=? AND pkgarch=? AND value LIKE '%s.%%';" % (self.table, base), (version, pkgarch)) - row = data.fetchone() - if row is not None: - return row[0] - else: - return None + rows = data.fetchall() + value = self._max_value(rows) - def _get_value_hist(self, version, pkgarch, checksum): - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] + if value is not None: + return increase_revision(value) else: - #no value found, try to insert - if self.read_only: - data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table), - (version, pkgarch)) - row = data.fetchone() - if row is not None: - return row[0] - else: - return 0 + return base + ".0" - try: - self._execute("INSERT INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));" - % (self.table, self.table), - (version, pkgarch, checksum, version, pkgarch)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) + def store_value(self, version, pkgarch, checksum, value): + """Store new value in the database""" - self.dirty = True + try: + self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), + (version, pkgarch, checksum, value)) + except sqlite3.IntegrityError as exc: + logger.error(str(exc)) - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] - else: - raise prserv.NotFoundError + self.dirty = True - def _get_value_no_hist(self, version, pkgarch, checksum): - data=self._execute("SELECT value FROM %s \ - WHERE version=? AND pkgarch=? AND checksum=? AND \ - value >= (select max(value) from %s where version=? AND pkgarch=?);" - % (self.table, self.table), - (version, pkgarch, checksum, version, pkgarch)) - row=data.fetchone() - if row is not None: - return row[0] - else: - #no value found, try to insert - if self.read_only: - data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table), - (version, pkgarch)) - return data.fetchone()[0] + def _get_value(self, version, pkgarch, checksum, history): - try: - self._execute("INSERT OR REPLACE INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));" - % (self.table, self.table), - (version, pkgarch, checksum, version, pkgarch)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) - self.conn.rollback() + max_value = self.find_package_max_value(version, pkgarch) - self.dirty = True + if max_value is None: + # version, pkgarch completely unknown. Return initial value. + return "0" - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] - else: - raise prserv.NotFoundError + value = self.find_value(version, pkgarch, checksum, history) - def get_value(self, version, pkgarch, checksum): - if self.nohist: - return self._get_value_no_hist(version, pkgarch, checksum) + if value is None: + # version, pkgarch found but not checksum. Create a new value from the maximum one + return increase_revision(max_value) + + if history: + return value + + # "no history" mode - If the value is not the maximum value for the package, need to increase it. + if max_value > value: + return increase_revision(max_value) else: - return self._get_value_hist(version, pkgarch, checksum) + return value + + def get_value(self, version, pkgarch, checksum, history): + value = self._get_value(version, pkgarch, checksum, history) + if not self.read_only: + self.store_value(version, pkgarch, checksum, value) + return value def _import_hist(self, version, pkgarch, checksum, value): if self.read_only: @@ -252,13 +264,13 @@ class PRTable(object): else: return None - def importone(self, version, pkgarch, checksum, value): - if self.nohist: - return self._import_no_hist(version, pkgarch, checksum, value) - else: + def importone(self, version, pkgarch, checksum, value, history=False): + if history: return self._import_hist(version, pkgarch, checksum, value) + else: + return self._import_no_hist(version, pkgarch, checksum, value) - def export(self, version, pkgarch, checksum, colinfo): + def export(self, version, pkgarch, checksum, colinfo, history=False): metainfo = {} #column info if colinfo: @@ -278,12 +290,12 @@ class PRTable(object): #data info datainfo = [] - if self.nohist: + if history: + sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table + else: sqlstmt = "SELECT T1.version, T1.pkgarch, T1.checksum, T1.value FROM %s as T1, \ (SELECT version, pkgarch, max(value) as maxvalue FROM %s GROUP BY version, pkgarch) as T2 \ WHERE T1.version=T2.version AND T1.pkgarch=T2.pkgarch AND T1.value=T2.maxvalue " % (self.table, self.table) - else: - sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table sqlarg = [] where = "" if version: @@ -322,9 +334,8 @@ class PRTable(object): class PRData(object): """Object representing the PR database""" - def __init__(self, filename, nohist=True, read_only=False): + def __init__(self, filename, read_only=False): self.filename=os.path.abspath(filename) - self.nohist=nohist self.read_only = read_only #build directory hierarchy try: @@ -351,7 +362,7 @@ class PRData(object): if tblname in self._tables: return self._tables[tblname] else: - tableobj = self._tables[tblname] = PRTable(self.connection, tblname, self.nohist, self.read_only) + tableobj = self._tables[tblname] = PRTable(self.connection, tblname, self.read_only) return tableobj def __delitem__(self, tblname): diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py index dc4be5b620..05573d06cc 100644 --- a/lib/prserv/serv.py +++ b/lib/prserv/serv.py @@ -12,6 +12,7 @@ import sqlite3 import prserv import prserv.db import errno +from . import create_async_client, revision_smaller, increase_revision import bb.asyncrpc logger = logging.getLogger("BitBake.PRserv") @@ -51,8 +52,9 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): version = request["version"] pkgarch = request["pkgarch"] checksum = request["checksum"] + history = request["history"] - value = self.server.table.find_value(version, pkgarch, checksum) + value = self.server.table.find_value(version, pkgarch, checksum, history) return {"value": value} async def handle_test_package(self, request): @@ -68,22 +70,110 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): version = request["version"] pkgarch = request["pkgarch"] - value = self.server.table.find_max_value(version, pkgarch) + value = self.server.table.find_package_max_value(version, pkgarch) return {"value": value} async def handle_get_pr(self, request): version = request["version"] pkgarch = request["pkgarch"] checksum = request["checksum"] + history = request["history"] - response = None - try: - value = self.server.table.get_value(version, pkgarch, checksum) - response = {"value": value} - except prserv.NotFoundError: - self.logger.error("failure storing value in database for (%s, %s)",version, checksum) + if self.upstream_client is None: + value = self.server.table.get_value(version, pkgarch, checksum, history) + return {"value": value} - return response + # We have an upstream server. + # Check whether the local server already knows the requested configuration. + # If the configuration is a new one, the generated value we will add will + # depend on what's on the upstream server. That's why we're calling find_value() + # instead of get_value() directly. + + value = self.server.table.find_value(version, pkgarch, checksum, history) + upstream_max = await self.upstream_client.max_package_pr(version, pkgarch) + + if value is not None: + + # The configuration is already known locally. + + if history: + value = self.server.table.get_value(version, pkgarch, checksum, history) + else: + existing_value = value + # In "no history", we need to make sure the value doesn't decrease + # and is at least greater than the maximum upstream value + # and the maximum local value + + local_max = self.server.table.find_package_max_value(version, pkgarch) + if revision_smaller(value, local_max): + value = increase_revision(local_max) + + if revision_smaller(value, upstream_max): + # Ask upstream whether it knows the checksum + upstream_value = await self.upstream_client.test_pr(version, pkgarch, checksum) + if upstream_value is None: + # Upstream doesn't have our checksum, let create a new one + value = upstream_max + ".0" + else: + # Fine to take the same value as upstream + value = upstream_max + + if not value == existing_value and not self.server.read_only: + self.server.table.store_value(version, pkgarch, checksum, value) + + return {"value": value} + + # The configuration is a new one for the local server + # Let's ask the upstream server whether it knows it + + known_upstream = await self.upstream_client.test_package(version, pkgarch) + + if not known_upstream: + + # The package is not known upstream, must be a local-only package + # Let's compute the PR number using the local-only method + + value = self.server.table.get_value(version, pkgarch, checksum, history) + return {"value": value} + + # The package is known upstream, let's ask the upstream server + # whether it knows our new output hash + + value = await self.upstream_client.test_pr(version, pkgarch, checksum) + + if value is not None: + + # Upstream knows this output hash, let's store it and use it too. + + if not self.server.read_only: + self.server.table.store_value(version, pkgarch, checksum, value) + # If the local server is read only, won't be able to store the new + # value in the database and will have to keep asking the upstream server + return {"value": value} + + # The output hash doesn't exist upstream, get the most recent number from upstream (x) + # Then, we want to have a new PR value for the local server: x.y + + upstream_max = await self.upstream_client.max_package_pr(version, pkgarch) + # Here we know that the package is known upstream, so upstream_max can't be None + subvalue = self.server.table.find_new_subvalue(version, pkgarch, upstream_max) + + if not self.server.read_only: + self.server.table.store_value(version, pkgarch, checksum, subvalue) + + return {"value": subvalue} + + async def process_requests(self): + if self.server.upstream is not None: + self.upstream_client = await create_async_client(self.server.upstream) + else: + self.upstream_client = None + + try: + await super().process_requests() + finally: + if self.upstream_client is not None: + await self.upstream_client.close() async def handle_import_one(self, request): response = None @@ -92,8 +182,9 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): pkgarch = request["pkgarch"] checksum = request["checksum"] value = request["value"] + history = request["history"] - value = self.server.table.importone(version, pkgarch, checksum, value) + value = self.server.table.importone(version, pkgarch, checksum, value, history) if value is not None: response = {"value": value} @@ -104,9 +195,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): pkgarch = request["pkgarch"] checksum = request["checksum"] colinfo = request["colinfo"] + history = request["history"] try: - (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo) + (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo, history) except sqlite3.Error as exc: self.logger.error(str(exc)) metainfo = datainfo = None @@ -117,11 +209,12 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): return {"readonly": self.server.read_only} class PRServer(bb.asyncrpc.AsyncServer): - def __init__(self, dbfile, read_only=False): + def __init__(self, dbfile, read_only=False, upstream=None): super().__init__(logger) self.dbfile = dbfile self.table = None self.read_only = read_only + self.upstream = upstream def accept_client(self, socket): return PRServerClient(socket, self) @@ -134,6 +227,9 @@ class PRServer(bb.asyncrpc.AsyncServer): self.logger.info("Started PRServer with DBfile: %s, Address: %s, PID: %s" % (self.dbfile, self.address, str(os.getpid()))) + if self.upstream is not None: + self.logger.info("And upstream PRServer: %s " % (self.upstream)) + return tasks async def stop(self): @@ -147,14 +243,15 @@ class PRServer(bb.asyncrpc.AsyncServer): self.table.sync() class PRServSingleton(object): - def __init__(self, dbfile, logfile, host, port): + def __init__(self, dbfile, logfile, host, port, upstream): self.dbfile = dbfile self.logfile = logfile self.host = host self.port = port + self.upstream = upstream def start(self): - self.prserv = PRServer(self.dbfile) + self.prserv = PRServer(self.dbfile, upstream=self.upstream) self.prserv.start_tcp_server(socket.gethostbyname(self.host), self.port) self.process = self.prserv.serve_as_process(log_level=logging.WARNING) @@ -233,7 +330,7 @@ def run_as_daemon(func, pidfile, logfile): os.remove(pidfile) os._exit(0) -def start_daemon(dbfile, host, port, logfile, read_only=False): +def start_daemon(dbfile, host, port, logfile, read_only=False, upstream=None): ip = socket.gethostbyname(host) pidfile = PIDPREFIX % (ip, port) try: @@ -249,7 +346,7 @@ def start_daemon(dbfile, host, port, logfile, read_only=False): dbfile = os.path.abspath(dbfile) def daemon_main(): - server = PRServer(dbfile, read_only=read_only) + server = PRServer(dbfile, read_only=read_only, upstream=upstream) server.start_tcp_server(ip, port) server.serve_forever() @@ -336,6 +433,9 @@ def auto_start(d): host = host_params[0].strip().lower() port = int(host_params[1]) + + upstream = d.getVar("PRSERV_UPSTREAM") or None + if is_local_special(host, port): import bb.utils cachedir = (d.getVar("PERSISTENT_DIR") or d.getVar("CACHE")) @@ -350,7 +450,7 @@ def auto_start(d): auto_shutdown() if not singleton: bb.utils.mkdirhier(cachedir) - singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port) + singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port, upstream) singleton.start() if singleton: host = singleton.host From patchwork Tue Apr 30 17:15:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1BCAC25B5C for ; Tue, 30 Apr 2024 17:15:41 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web11.21515.1714497333523095844 for ; Tue, 30 Apr 2024 10:15:33 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=pxZlVCea; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id E614E60003; Tue, 30 Apr 2024 17:15:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497332; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g6f8PWqWrB6iGR9IWEP3S098DrdRWUcxsXRWbof4kbM=; b=pxZlVCeawXJFl9TttxKI1e1J6WkQ2Ehr6kf5Nq+ZIKThaAxKynVFLfDXmuUSbSAOaCBOkh g7RNtAPuMcKIJry0jfgJf/FOJUKhJTCvpm+qzbRnYvs9MRnkwzsNrYEvi0M4jfORJdBEak Ce0/77Akj8fBA/kI6I9KxmSkUdpJSLtPJnNDG06rSVJxNGVIWD6QcnEtLNt+S6Fvm66cxW LDojGTVwVRyY9LCuRqV2Yhvgm8Ry6GaiuINupqPZKSgHat0hgqLGZjv2vzGBYOxDplgmjv 21VisIDV436OfIyiSbDIlCyTQjtqRKjJ45mgP0UplsAFgwPbAw4QXJsq0FD2Fw== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 4/8] prserv: enable database sharing Date: Tue, 30 Apr 2024 19:15:08 +0200 Message-Id: <20240430171512.936371-5-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16169 From: Michael Opdenacker sqlite3 can allow multiple processes to access the database simultaneously, but it must be opened correctly. The key change is that the database is no longer opened in "exclusive" mode (defaulting to shared mode). In addition, the journal is set to "WAL" mode, as this is the most efficient for dealing with simultaneous access between different processes. In order to keep the database performance, synchronous mode is set to "off". The WAL journal will protect against incomplete transactions in any given client, however the database will not be protected against unexpected power loss from the OS (which is a fine trade off for performance, and also the same as the previous implementation). The use of a database cursor enabled to remove the _execute() wrapper. The cursor automatically makes sure that the query happens in an atomic transaction and commits when finished. This also removes the need for a "dirty" flag for the database and for explicit database syncing, which simplifies the code. Signed-off-by: Michael Opdenacker Signed-off-by: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni Reviewed-by: Jan-Siumon Möller --- lib/prserv/db.py | 322 +++++++++++++++++++++------------------------ lib/prserv/serv.py | 8 -- 2 files changed, 151 insertions(+), 179 deletions(-) diff --git a/lib/prserv/db.py b/lib/prserv/db.py index b2520f3158..f430586d73 100644 --- a/lib/prserv/db.py +++ b/lib/prserv/db.py @@ -8,21 +8,13 @@ import logging import os.path import errno import prserv -import time +import sqlite3 +from contextlib import closing from . import increase_revision, revision_greater, revision_smaller -try: - import sqlite3 -except ImportError: - from pysqlite2 import dbapi2 as sqlite3 - logger = logging.getLogger("BitBake.PRserv") -sqlversion = sqlite3.sqlite_version_info -if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): - raise Exception("sqlite3 version 3.3.0 or later is required.") - # # "No History" mode - for a given query tuple (version, pkgarch, checksum), # the returned value will be the largest among all the values of the same @@ -31,40 +23,28 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): # "History" mode - Return a new higher value for previously unseen query # tuple (version, pkgarch, checksum), otherwise return historical value. # Value can decrement if returning to a previous build. -# class PRTable(object): def __init__(self, conn, table, read_only): self.conn = conn self.read_only = read_only - self.dirty = False self.table = table - if self.read_only: - table_exists = self._execute( - "SELECT count(*) FROM sqlite_master \ - WHERE type='table' AND name='%s'" % (self.table)) - if not table_exists: - raise prserv.NotFoundError - else: - self._execute("CREATE TABLE IF NOT EXISTS %s \ - (version TEXT NOT NULL, \ - pkgarch TEXT NOT NULL, \ - checksum TEXT NOT NULL, \ - value TEXT, \ - PRIMARY KEY (version, pkgarch, checksum, value));" % self.table) - - def _execute(self, *query): - """Execute a query, waiting to acquire a lock if necessary""" - start = time.time() - end = start + 20 - while True: - try: - return self.conn.execute(*query) - except sqlite3.OperationalError as exc: - if "is locked" in str(exc) and end > time.time(): - continue - raise exc + with closing(self.conn.cursor()) as cursor: + if self.read_only: + table_exists = cursor.execute( + "SELECT count(*) FROM sqlite_master \ + WHERE type='table' AND name='%s'" % (self.table)) + if not table_exists: + raise prserv.NotFoundError + else: + cursor.execute("CREATE TABLE IF NOT EXISTS %s \ + (version TEXT NOT NULL, \ + pkgarch TEXT NOT NULL, \ + checksum TEXT NOT NULL, \ + value TEXT, \ + PRIMARY KEY (version, pkgarch, checksum, value));" % self.table) + self.conn.commit() def _extremum_value(self, rows, is_max): value = None @@ -88,49 +68,42 @@ class PRTable(object): def _min_value(self, rows): return self._extremum_value(rows, False) - def sync(self): - if not self.read_only: - self.conn.commit() - self._execute("BEGIN EXCLUSIVE TRANSACTION") - - def sync_if_dirty(self): - if self.dirty: - self.sync() - self.dirty = False - def test_package(self, version, pkgarch): """Returns whether the specified package version is found in the database for the specified architecture""" # Just returns the value if found or None otherwise - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=?;" % self.table, - (version, pkgarch)) - row=data.fetchone() - if row is not None: - return True - else: - return False + with closing(self.conn.cursor()) as cursor: + data=cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=?;" % self.table, + (version, pkgarch)) + row=data.fetchone() + if row is not None: + return True + else: + return False def test_value(self, version, pkgarch, value): """Returns whether the specified value is found in the database for the specified package and architecture""" # Just returns the value if found or None otherwise - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? and value=?;" % self.table, - (version, pkgarch, value)) - row=data.fetchone() - if row is not None: - return True - else: - return False + with closing(self.conn.cursor()) as cursor: + data=cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? and value=?;" % self.table, + (version, pkgarch, value)) + row=data.fetchone() + if row is not None: + return True + else: + return False def find_package_max_value(self, version, pkgarch): """Returns the greatest value for (version, pkgarch), or None if not found. Doesn't create a new value""" - data = self._execute("SELECT value FROM %s where version=? AND pkgarch=?;" % (self.table), - (version, pkgarch)) - rows = data.fetchall() - value = self._max_value(rows) - return value + with closing(self.conn.cursor()) as cursor: + data = cursor.execute("SELECT value FROM %s where version=? AND pkgarch=?;" % (self.table), + (version, pkgarch)) + rows = data.fetchall() + value = self._max_value(rows) + return value def find_value(self, version, pkgarch, checksum, history=False): """Returns the value for the specified checksum if found or None otherwise.""" @@ -145,10 +118,11 @@ class PRTable(object): """Returns the maximum (if is_max is True) or minimum (if is_max is False) value for (version, pkgarch, checksum), or None if not found. Doesn't create a new value""" - data = self._execute("SELECT value FROM %s where version=? AND pkgarch=? AND checksum=?;" % (self.table), - (version, pkgarch, checksum)) - rows = data.fetchall() - return self._extremum_value(rows, is_max) + with closing(self.conn.cursor()) as cursor: + data = cursor.execute("SELECT value FROM %s where version=? AND pkgarch=? AND checksum=?;" % (self.table), + (version, pkgarch, checksum)) + rows = data.fetchall() + return self._extremum_value(rows, is_max) def find_max_value(self, version, pkgarch, checksum): return self._find_extremum_value(version, pkgarch, checksum, True) @@ -160,26 +134,27 @@ class PRTable(object): """Take and increase the greatest ".y" value for (version, pkgarch), or return ".0" if not found. This doesn't store a new value.""" - data = self._execute("SELECT value FROM %s where version=? AND pkgarch=? AND value LIKE '%s.%%';" % (self.table, base), - (version, pkgarch)) - rows = data.fetchall() - value = self._max_value(rows) + with closing(self.conn.cursor()) as cursor: + data = cursor.execute("SELECT value FROM %s where version=? AND pkgarch=? AND value LIKE '%s.%%';" % (self.table, base), + (version, pkgarch)) + rows = data.fetchall() + value = self._max_value(rows) - if value is not None: - return increase_revision(value) - else: - return base + ".0" + if value is not None: + return increase_revision(value) + else: + return base + ".0" def store_value(self, version, pkgarch, checksum, value): """Store new value in the database""" - try: - self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), - (version, pkgarch, checksum, value)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) - - self.dirty = True + with closing(self.conn.cursor()) as cursor: + try: + cursor.execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), + (version, pkgarch, checksum, value)) + except sqlite3.IntegrityError as exc: + logger.error(str(exc)) + self.conn.commit() def _get_value(self, version, pkgarch, checksum, history): @@ -215,54 +190,56 @@ class PRTable(object): return None val = None - data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, + with closing(self.conn.cursor()) as cursor: + data = cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, (version, pkgarch, checksum)) - row = data.fetchone() - if row is not None: - val=row[0] - else: - #no value found, try to insert - try: - self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), - (version, pkgarch, checksum, value)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) + row = data.fetchone() + if row is not None: + val=row[0] + else: + #no value found, try to insert + try: + cursor.execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), + (version, pkgarch, checksum, value)) + except sqlite3.IntegrityError as exc: + logger.error(str(exc)) - self.dirty = True + self.conn.commit() - data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, + data = cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, (version, pkgarch, checksum)) - row = data.fetchone() - if row is not None: - val = row[0] + row = data.fetchone() + if row is not None: + val = row[0] return val def _import_no_hist(self, version, pkgarch, checksum, value): if self.read_only: return None - try: - #try to insert - self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), - (version, pkgarch, checksum, value)) - except sqlite3.IntegrityError as exc: - #already have the record, try to update + with closing(self.conn.cursor()) as cursor: try: - self._execute("UPDATE %s SET value=? WHERE version=? AND pkgarch=? AND checksum=? AND value=?;" % self.table, + data = cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=? AND value>=?;" % self.table, (version, pkgarch, checksum, value)) - row=data.fetchone() - if row is not None: - return row[0] - else: - return None + row=data.fetchone() + if row is not None: + return row[0] + else: + return None def importone(self, version, pkgarch, checksum, value, history=False): if history: @@ -272,56 +249,57 @@ class PRTable(object): def export(self, version, pkgarch, checksum, colinfo, history=False): metainfo = {} - #column info - if colinfo: - metainfo["tbl_name"] = self.table - metainfo["core_ver"] = prserv.__version__ - metainfo["col_info"] = [] - data = self._execute("PRAGMA table_info(%s);" % self.table) + with closing(self.conn.cursor()) as cursor: + #column info + if colinfo: + metainfo["tbl_name"] = self.table + metainfo["core_ver"] = prserv.__version__ + metainfo["col_info"] = [] + data = cursor.execute("PRAGMA table_info(%s);" % self.table) + for row in data: + col = {} + col["name"] = row["name"] + col["type"] = row["type"] + col["notnull"] = row["notnull"] + col["dflt_value"] = row["dflt_value"] + col["pk"] = row["pk"] + metainfo["col_info"].append(col) + + #data info + datainfo = [] + + if history: + sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table + else: + sqlstmt = "SELECT T1.version, T1.pkgarch, T1.checksum, T1.value FROM %s as T1, \ + (SELECT version, pkgarch, max(value) as maxvalue FROM %s GROUP BY version, pkgarch) as T2 \ + WHERE T1.version=T2.version AND T1.pkgarch=T2.pkgarch AND T1.value=T2.maxvalue " % (self.table, self.table) + sqlarg = [] + where = "" + if version: + where += "AND T1.version=? " + sqlarg.append(str(version)) + if pkgarch: + where += "AND T1.pkgarch=? " + sqlarg.append(str(pkgarch)) + if checksum: + where += "AND T1.checksum=? " + sqlarg.append(str(checksum)) + + sqlstmt += where + ";" + + if len(sqlarg): + data = cursor.execute(sqlstmt, tuple(sqlarg)) + else: + data = cursor.execute(sqlstmt) for row in data: - col = {} - col["name"] = row["name"] - col["type"] = row["type"] - col["notnull"] = row["notnull"] - col["dflt_value"] = row["dflt_value"] - col["pk"] = row["pk"] - metainfo["col_info"].append(col) - - #data info - datainfo = [] - - if history: - sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table - else: - sqlstmt = "SELECT T1.version, T1.pkgarch, T1.checksum, T1.value FROM %s as T1, \ - (SELECT version, pkgarch, max(value) as maxvalue FROM %s GROUP BY version, pkgarch) as T2 \ - WHERE T1.version=T2.version AND T1.pkgarch=T2.pkgarch AND T1.value=T2.maxvalue " % (self.table, self.table) - sqlarg = [] - where = "" - if version: - where += "AND T1.version=? " - sqlarg.append(str(version)) - if pkgarch: - where += "AND T1.pkgarch=? " - sqlarg.append(str(pkgarch)) - if checksum: - where += "AND T1.checksum=? " - sqlarg.append(str(checksum)) - - sqlstmt += where + ";" - - if len(sqlarg): - data = self._execute(sqlstmt, tuple(sqlarg)) - else: - data = self._execute(sqlstmt) - for row in data: - if row["version"]: - col = {} - col["version"] = row["version"] - col["pkgarch"] = row["pkgarch"] - col["checksum"] = row["checksum"] - col["value"] = row["value"] - datainfo.append(col) + if row["version"]: + col = {} + col["version"] = row["version"] + col["pkgarch"] = row["pkgarch"] + col["checksum"] = row["checksum"] + col["value"] = row["value"] + datainfo.append(col) return (metainfo, datainfo) def dump_db(self, fd): @@ -345,14 +323,15 @@ class PRData(object): raise e uri = "file:%s%s" % (self.filename, "?mode=ro" if self.read_only else "") logger.debug("Opening PRServ database '%s'" % (uri)) - self.connection=sqlite3.connect(uri, uri=True, isolation_level="EXCLUSIVE", check_same_thread = False) + self.connection=sqlite3.connect(uri, uri=True) self.connection.row_factory=sqlite3.Row - if not self.read_only: - self.connection.execute("pragma synchronous = off;") - self.connection.execute("PRAGMA journal_mode = MEMORY;") + self.connection.execute("PRAGMA synchronous = OFF;") + self.connection.execute("PRAGMA journal_mode = WAL;") + self.connection.commit() self._tables={} def disconnect(self): + self.connection.commit() self.connection.close() def __getitem__(self, tblname): @@ -370,3 +349,4 @@ class PRData(object): del self._tables[tblname] logger.info("drop table %s" % (tblname)) self.connection.execute("DROP TABLE IF EXISTS %s;" % tblname) + self.connection.commit() diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py index 05573d06cc..fd673b1851 100644 --- a/lib/prserv/serv.py +++ b/lib/prserv/serv.py @@ -44,8 +44,6 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): except: self.server.table.sync() raise - else: - self.server.table.sync_if_dirty() async def handle_test_pr(self, request): '''Finds the PR value corresponding to the request. If not found, returns None and doesn't insert a new value''' @@ -233,15 +231,9 @@ class PRServer(bb.asyncrpc.AsyncServer): return tasks async def stop(self): - self.table.sync_if_dirty() self.db.disconnect() await super().stop() - def signal_handler(self): - super().signal_handler() - if self.table: - self.table.sync() - class PRServSingleton(object): def __init__(self, dbfile, logfile, host, port, upstream): self.dbfile = dbfile From patchwork Tue Apr 30 17:15:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF03AC25B5F for ; Tue, 30 Apr 2024 17:15:41 +0000 (UTC) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by mx.groups.io with SMTP id smtpd.web11.21516.1714497336121594339 for ; Tue, 30 Apr 2024 10:15:36 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=ZPuFiYne; spf=pass (domain: bootlin.com, ip: 217.70.183.196, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 07BA6E0008; Tue, 30 Apr 2024 17:15:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497334; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8GtxRIPKaNXoiIt6N50JuWAbFzQ/N6lr3/u+WkEfw6M=; b=ZPuFiYnePT/ZWKDO+BjNWge6dxe4FZEZ/TrWDyZPJmM/uACNOwI9ZxtWs7IVa/X29mEDEB AjsTQ/Vo5XJIOnfoNoGJMTgh8MgoWxYTkLhha1Tddb2ioXnODDkQggU0fRgn5P7m3kOjsO CEggN1cb5OqVpT4fpJBPb3TXoyTsJorT33GBb9/CB2SyBSua21skNV3AZ5OGbSAcAzDXPy GJmRagDqRSLJUM3lPqYZ5OH+7PSO0sgYuqC660BbOJj3yrToUbx/QtcKKdJ4jx/pcyPODp R/PUHYKQKr2I8+l54W2FCrdUQABSfIYpT3RxWIg/DF14b96yK9C1hw1/kJaGag== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 5/8] prserv: avoid possible race condition in database code Date: Tue, 30 Apr 2024 19:15:09 +0200 Message-Id: <20240430171512.936371-6-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16170 From: Michael Opdenacker Remove a possible race condition by allowing a read-only server to create the PR table anyway. This avoids a failure if both a read-only and read-write server try to access an empty database at the same time. Signed-off-by: Michael Opdenacker Suggested-by: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni --- lib/prserv/db.py | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/lib/prserv/db.py b/lib/prserv/db.py index f430586d73..79c9001bf5 100644 --- a/lib/prserv/db.py +++ b/lib/prserv/db.py @@ -30,21 +30,18 @@ class PRTable(object): self.read_only = read_only self.table = table + # Creating the table even if the server is read-only. + # This avoids a race condition if a shared database + # is accessed by a read-only server first. + with closing(self.conn.cursor()) as cursor: - if self.read_only: - table_exists = cursor.execute( - "SELECT count(*) FROM sqlite_master \ - WHERE type='table' AND name='%s'" % (self.table)) - if not table_exists: - raise prserv.NotFoundError - else: - cursor.execute("CREATE TABLE IF NOT EXISTS %s \ - (version TEXT NOT NULL, \ - pkgarch TEXT NOT NULL, \ - checksum TEXT NOT NULL, \ - value TEXT, \ - PRIMARY KEY (version, pkgarch, checksum, value));" % self.table) - self.conn.commit() + cursor.execute("CREATE TABLE IF NOT EXISTS %s \ + (version TEXT NOT NULL, \ + pkgarch TEXT NOT NULL, \ + checksum TEXT NOT NULL, \ + value TEXT, \ + PRIMARY KEY (version, pkgarch, checksum, value));" % self.table) + self.conn.commit() def _extremum_value(self, rows, is_max): value = None From patchwork Tue Apr 30 17:15:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6B83C10F16 for ; Tue, 30 Apr 2024 17:15:41 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web11.21518.1714497336637762903 for ; Tue, 30 Apr 2024 10:15:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=GGEiGjbB; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 28B7AC0008; Tue, 30 Apr 2024 17:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FoQbZoRO+6uY2CILaBtuqnVI4zezx0rUWnRLlmd21v0=; b=GGEiGjbBc7gxZz7lmPO7xRaV4W46beFYyuQKhTqd1uZ2SaIT0BeEDcF9Y1yIxY3rS5DxBU 1xlju8+MMEFO7GdBIveToQR7iQ00lY7PPX15LbEznmiqcLEymEWA6ObD39cK7cNsKaMJ/3 bb2nVaA4cBf5ijDJEyDIujCi3Hlra9zhEcvVD0ADIcmj3zLhRqDdzUUETrulWACf4+hQPS gw7eOzzOzP5o5cv9pgZGZqEVbcPUIUGOoV7px6lVpeZw9i2JiPhtxUgMsCNp76XRnydG2e yODPtPDTHf3i2hdNrSeMMSp8Z5azWr95kv8IhF6SueCbMPRp0fcqKUa+d7tjkg== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 6/8] prserv: store_value() improvements Date: Tue, 30 Apr 2024 19:15:10 +0200 Message-Id: <20240430171512.936371-7-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16171 From: Michael Opdenacker Add a test_checksum_value() to test whether a (version, pkgarch, checksum, value) entry already exists in the database. This is used to protect the store_value() function from an error when trying to store a duplicate entry in the database. Also check whether the current database is open in read-only mode. Signed-off-by: Michael Opdenacker Cc: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni --- lib/prserv/db.py | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/lib/prserv/db.py b/lib/prserv/db.py index 79c9001bf5..88ed8e2125 100644 --- a/lib/prserv/db.py +++ b/lib/prserv/db.py @@ -78,6 +78,18 @@ class PRTable(object): else: return False + def test_checksum_value(self, version, pkgarch, checksum, value): + """Returns whether the specified value is found in the database for the specified package, architecture and checksum""" + + with closing(self.conn.cursor()) as cursor: + data=cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? and checksum=? and value=?;" % self.table, + (version, pkgarch, checksum, value)) + row=data.fetchone() + if row is not None: + return True + else: + return False + def test_value(self, version, pkgarch, value): """Returns whether the specified value is found in the database for the specified package and architecture""" @@ -143,15 +155,13 @@ class PRTable(object): return base + ".0" def store_value(self, version, pkgarch, checksum, value): - """Store new value in the database""" + """Store value in the database""" - with closing(self.conn.cursor()) as cursor: - try: + if not self.read_only and not self.test_checksum_value(version, pkgarch, checksum, value): + with closing(self.conn.cursor()) as cursor: cursor.execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), (version, pkgarch, checksum, value)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) - self.conn.commit() + self.conn.commit() def _get_value(self, version, pkgarch, checksum, history): From patchwork Tue Apr 30 17:15:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCCA5C25B10 for ; Tue, 30 Apr 2024 17:15:41 +0000 (UTC) Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by mx.groups.io with SMTP id smtpd.web10.21665.1714497337817769415 for ; Tue, 30 Apr 2024 10:15:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=SqxHn11k; spf=pass (domain: bootlin.com, ip: 217.70.183.199, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id 488DFFF804; Tue, 30 Apr 2024 17:15:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497336; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j3wQKXgZzP30OmOUXKmT7rHM/gRMcV/K5giGfMjDUDk=; b=SqxHn11khdE9w5V8HcBAYGtpFyk6cbHXgeOh2IUxmzjBG74K7HarNGFzSyX57DnQG/nFDi j4Or6Cxdk7++sizDf+hbr3h03CB2Eidl59HXVp3fNSH0zcxV8gWK5CcS+JsQcre3g/jDLQ 7UFe98Z1edXALJc2xswJ49OqDw8anYEDvgOh5XRah2JNIQ5vgTD3Pa9XQA0Ghz7Etm+3Pu 6z1w3sKxg2SHYeaoQ6nCX018TE/YtCasNfKlftMPuMiAp2A9wTqbh4w7LmHXKZCQHisfe2 BH07TP5AzIIlPfQ9dCRdFA96ghOV+4sfMa/ZrNdUVG6Kc6AqAvnmlU8A4B+bAA== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 7/8] prserv: import simplification Date: Tue, 30 Apr 2024 19:15:11 +0200 Message-Id: <20240430171512.936371-8-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16172 From: Michael Opdenacker Simplify the importone() hook: - to make it independent from the "history" mode which is client specific. - remove the "history" parameter - we want all values to be imported for binary reproducibility purposes. - using the store_value() function (which warrants you don't save the same value twice and doesn't write when you're using a read-only server) is enough. Signed-off-by: Michael Opdenacker Cc: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni --- lib/prserv/client.py | 4 +-- lib/prserv/db.py | 64 +++----------------------------------------- lib/prserv/serv.py | 3 +-- 3 files changed, 6 insertions(+), 65 deletions(-) diff --git a/lib/prserv/client.py b/lib/prserv/client.py index 565c6f3872..9f5794c433 100644 --- a/lib/prserv/client.py +++ b/lib/prserv/client.py @@ -42,9 +42,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): if response: return response["value"] - async def importone(self, version, pkgarch, checksum, value, history=False): + async def importone(self, version, pkgarch, checksum, value): response = await self.invoke( - {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value, "history": history}} + {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value}} ) if response: return response["value"] diff --git a/lib/prserv/db.py b/lib/prserv/db.py index 88ed8e2125..2da493ddf5 100644 --- a/lib/prserv/db.py +++ b/lib/prserv/db.py @@ -192,67 +192,9 @@ class PRTable(object): self.store_value(version, pkgarch, checksum, value) return value - def _import_hist(self, version, pkgarch, checksum, value): - if self.read_only: - return None - - val = None - with closing(self.conn.cursor()) as cursor: - data = cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row = data.fetchone() - if row is not None: - val=row[0] - else: - #no value found, try to insert - try: - cursor.execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), - (version, pkgarch, checksum, value)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) - - self.conn.commit() - - data = cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row = data.fetchone() - if row is not None: - val = row[0] - return val - - def _import_no_hist(self, version, pkgarch, checksum, value): - if self.read_only: - return None - - with closing(self.conn.cursor()) as cursor: - try: - #try to insert - cursor.execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), - (version, pkgarch, checksum, value)) - except sqlite3.IntegrityError as exc: - #already have the record, try to update - try: - cursor.execute("UPDATE %s SET value=? WHERE version=? AND pkgarch=? AND checksum=? AND value=?;" % self.table, - (version, pkgarch, checksum, value)) - row=data.fetchone() - if row is not None: - return row[0] - else: - return None - - def importone(self, version, pkgarch, checksum, value, history=False): - if history: - return self._import_hist(version, pkgarch, checksum, value) - else: - return self._import_no_hist(version, pkgarch, checksum, value) + def importone(self, version, pkgarch, checksum, value): + self.store_value(version, pkgarch, checksum, value) + return value def export(self, version, pkgarch, checksum, colinfo, history=False): metainfo = {} diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py index fd673b1851..d3ee43dcb5 100644 --- a/lib/prserv/serv.py +++ b/lib/prserv/serv.py @@ -180,9 +180,8 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): pkgarch = request["pkgarch"] checksum = request["checksum"] value = request["value"] - history = request["history"] - value = self.server.table.importone(version, pkgarch, checksum, value, history) + value = self.server.table.importone(version, pkgarch, checksum, value) if value is not None: response = {"value": value} From patchwork Tue Apr 30 17:15:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Opdenacker X-Patchwork-Id: 42985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D33DEC19F4F for ; Tue, 30 Apr 2024 17:15:41 +0000 (UTC) Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by mx.groups.io with SMTP id smtpd.web10.21666.1714497340366283765 for ; Tue, 30 Apr 2024 10:15:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=Vjnk81/T; spf=pass (domain: bootlin.com, ip: 217.70.183.193, mailfrom: michael.opdenacker@bootlin.com) Received: by mail.gandi.net (Postfix) with ESMTPSA id EE8EF24000D; Tue, 30 Apr 2024 17:15:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1714497338; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BuiggZbEnnKO9Pq0uX2zrt+ZfoJk0iGVD1iuwNJGA2Q=; b=Vjnk81/TBNd/uqlVG4yuRX9NQBv0CEbwsYnrWy4ff+adPyEYk+Lp4WQrHgX5HlhiRf7v1o IFkBh3OO3twlGAQOuDzoLj8G3q0GqLBRabB9fuvhagpAYZerxJc0vLiEY6ZDsHQLj9G5Fb YR506Sg/EZmGazWF5mBzXlV9rusFnwlIFHA+to+PpUZEVXd+n3AFaA0ceWRXwpsLHXYCUg PvnvXbmPVMo75HOurjMp2BuPFoNeaegLSLY92uzS7u2I2i5VGv821YSkRl2BR8arGGaVAu Qi+wfdB30hL0AI9+itBDV13niJPgNVx/78A1wps+7ruwZNaYkDz0E1OD19e93w== From: michael.opdenacker@bootlin.com To: bitbake-devel@lists.openembedded.org Cc: Michael Opdenacker , Joshua Watt , Tim Orling , Thomas Petazzoni Subject: [PATCH v6 8/8] prserv: add bitbake selftests Date: Tue, 30 Apr 2024 19:15:12 +0200 Message-Id: <20240430171512.936371-9-michael.opdenacker@bootlin.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com> References: <20240430171512.936371-1-michael.opdenacker@bootlin.com> MIME-Version: 1.0 X-GND-Sasl: michael.opdenacker@bootlin.com List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 30 Apr 2024 17:15:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16173 From: Michael Opdenacker Run them with "bitbake-selftest prserv.tests" Signed-off-by: Michael Opdenacker Cc: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni --- bin/bitbake-selftest | 2 + lib/prserv/tests.py | 386 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 388 insertions(+) create mode 100644 bitbake/lib/prserv/tests.py diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest index f25f23b1ae..ce901232fe 100755 --- a/bin/bitbake-selftest +++ b/bin/bitbake-selftest @@ -15,6 +15,7 @@ import unittest try: import bb import hashserv + import prserv import layerindexlib except RuntimeError as exc: sys.exit(str(exc)) @@ -33,6 +34,7 @@ tests = ["bb.tests.codeparser", "bb.tests.utils", "bb.tests.compression", "hashserv.tests", + "prserv.tests", "layerindexlib.tests.layerindexobj", "layerindexlib.tests.restapi", "layerindexlib.tests.cooker"] diff --git a/lib/prserv/tests.py b/lib/prserv/tests.py new file mode 100644 index 0000000000..8765b129f2 --- /dev/null +++ b/lib/prserv/tests.py @@ -0,0 +1,386 @@ +#! /usr/bin/env python3 +# +# Copyright (C) 2024 BitBake Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +from . import create_server, create_client, increase_revision, revision_greater, revision_smaller, _revision_greater_or_equal +import prserv.db as db +from bb.asyncrpc import InvokeError +import logging +import os +import sys +import tempfile +import unittest +import socket +import subprocess +from pathlib import Path + +THIS_DIR = Path(__file__).parent +BIN_DIR = THIS_DIR.parent.parent / "bin" + +version = "dummy-1.0-r0" +pkgarch = "core2-64" +other_arch = "aarch64" + +checksumX = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4f0" +checksum0 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a0" +checksum1 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a1" +checksum2 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a2" +checksum3 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a3" +checksum4 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a4" +checksum5 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a5" +checksum6 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a6" +checksum7 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a7" +checksum8 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a8" +checksum9 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a9" +checksum10 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4aa" + +def server_prefunc(server, name): + logging.basicConfig(level=logging.DEBUG, filename='prserv-%s.log' % name, filemode='w', + format='%(levelname)s %(filename)s:%(lineno)d %(message)s') + server.logger.debug("Running server %s" % name) + sys.stdout = open('prserv-stdout-%s.log' % name, 'w') + sys.stderr = sys.stdout + +class PRTestSetup(object): + + def start_server(self, name, dbfile, upstream=None, read_only=False, prefunc=server_prefunc): + + def cleanup_server(server): + if server.process.exitcode is not None: + return + server.process.terminate() + server.process.join() + + server = create_server(socket.gethostbyname("localhost") + ":0", + dbfile, + upstream=upstream, + read_only=read_only) + + server.serve_as_process(prefunc=prefunc, args=(name,)) + self.addCleanup(cleanup_server, server) + + return server + + def start_client(self, server_address): + def cleanup_client(client): + client.close() + + client = create_client(server_address) + self.addCleanup(cleanup_client, client) + + return client + +class FunctionTests(unittest.TestCase): + + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv') + self.addCleanup(self.temp_dir.cleanup) + + def test_increase_revision(self): + self.assertEqual(increase_revision("1"), "2") + self.assertEqual(increase_revision("1.0"), "1.1") + self.assertEqual(increase_revision("1.1.1"), "1.1.2") + self.assertEqual(increase_revision("1.1.1.3"), "1.1.1.4") + self.assertRaises(ValueError, increase_revision, "1.a") + self.assertRaises(ValueError, increase_revision, "1.") + self.assertRaises(ValueError, increase_revision, "") + + def test_revision_greater_or_equal(self): + self.assertTrue(_revision_greater_or_equal("2", "2")) + self.assertTrue(_revision_greater_or_equal("2", "1")) + self.assertTrue(_revision_greater_or_equal("10", "2")) + self.assertTrue(_revision_greater_or_equal("1.10", "1.2")) + self.assertFalse(_revision_greater_or_equal("1.2", "1.10")) + self.assertTrue(_revision_greater_or_equal("1.10", "1")) + self.assertTrue(_revision_greater_or_equal("1.10.1", "1.10")) + self.assertFalse(_revision_greater_or_equal("1.10.1", "1.10.2")) + self.assertTrue(_revision_greater_or_equal("1.10.1", "1.10.1")) + self.assertTrue(_revision_greater_or_equal("1.10.1", "1")) + self.assertTrue(revision_greater("1.20", "1.3")) + self.assertTrue(revision_smaller("1.3", "1.20")) + + # DB tests + + def test_db(self): + dbfile = os.path.join(self.temp_dir.name, "testtable.sqlite3") + + self.db = db.PRData(dbfile) + self.table = self.db["PRMAIN"] + + self.table.store_value(version, pkgarch, checksum0, "0") + self.table.store_value(version, pkgarch, checksum1, "1") + # "No history" mode supports multiple PRs for the same checksum + self.table.store_value(version, pkgarch, checksum0, "2") + self.table.store_value(version, pkgarch, checksum2, "1.0") + + self.assertTrue(self.table.test_package(version, pkgarch)) + self.assertFalse(self.table.test_package(version, other_arch)) + + self.assertTrue(self.table.test_value(version, pkgarch, "0")) + self.assertTrue(self.table.test_value(version, pkgarch, "1")) + self.assertTrue(self.table.test_value(version, pkgarch, "2")) + + self.assertEqual(self.table.find_package_max_value(version, pkgarch), "2") + + self.assertEqual(self.table.find_min_value(version, pkgarch, checksum0), "0") + self.assertEqual(self.table.find_max_value(version, pkgarch, checksum0), "2") + + # Test history modes + self.assertEqual(self.table.find_value(version, pkgarch, checksum0, True), "0") + self.assertEqual(self.table.find_value(version, pkgarch, checksum0, False), "2") + + self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "3"), "3.0") + self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "1"), "1.1") + + # Revision comparison tests + self.table.store_value(version, pkgarch, checksum1, "1.3") + self.table.store_value(version, pkgarch, checksum1, "1.20") + self.assertEqual(self.table.find_min_value(version, pkgarch, checksum1), "1") + self.assertEqual(self.table.find_max_value(version, pkgarch, checksum1), "1.20") + +class PRBasicTests(PRTestSetup, unittest.TestCase): + + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv') + self.addCleanup(self.temp_dir.cleanup) + + dbfile = os.path.join(self.temp_dir.name, "prtest-basic.sqlite3") + + self.server1 = self.start_server("basic", dbfile) + self.client1 = self.start_client(self.server1.address) + + def test_basic(self): + + # Checks on non existing configuration + + result = self.client1.test_pr(version, pkgarch, checksum0) + self.assertIsNone(result, "test_pr should return 'None' for a non existing PR") + + result = self.client1.test_package(version, pkgarch) + self.assertFalse(result, "test_package should return 'False' for a non existing PR") + + result = self.client1.max_package_pr(version, pkgarch) + self.assertIsNone(result, "max_package_pr should return 'None' for a non existing PR") + + # Add a first configuration + + result = self.client1.getPR(version, pkgarch, checksum0) + self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'") + + result = self.client1.test_pr(version, pkgarch, checksum0) + self.assertEqual(result, "0", "test_pr should return '0' here, matching the result of getPR") + + result = self.client1.test_package(version, pkgarch) + self.assertTrue(result, "test_package should return 'True' for an existing PR") + + result = self.client1.max_package_pr(version, pkgarch) + self.assertEqual(result, "0", "max_package_pr should return '0' in the current test series") + + # Check that the same request gets the same value + + result = self.client1.getPR(version, pkgarch, checksum0) + self.assertEqual(result, "0", "getPR: asking for the same PR a second time in a row should return the same value.") + + # Add new configurations + + result = self.client1.getPR(version, pkgarch, checksum1) + self.assertEqual(result, "1", "getPR: second PR of a package should be '1'") + + result = self.client1.test_pr(version, pkgarch, checksum1) + self.assertEqual(result, "1", "test_pr should return '1' here, matching the result of getPR") + + result = self.client1.max_package_pr(version, pkgarch) + self.assertEqual(result, "1", "max_package_pr should return '1' in the current test series") + + result = self.client1.getPR(version, pkgarch, checksum2) + self.assertEqual(result, "2", "getPR: second PR of a package should be '2'") + + result = self.client1.test_pr(version, pkgarch, checksum2) + self.assertEqual(result, "2", "test_pr should return '2' here, matching the result of getPR") + + result = self.client1.max_package_pr(version, pkgarch) + self.assertEqual(result, "2", "max_package_pr should return '2' in the current test series") + + result = self.client1.getPR(version, pkgarch, checksum3) + self.assertEqual(result, "3", "getPR: second PR of a package should be '3'") + + result = self.client1.test_pr(version, pkgarch, checksum3) + self.assertEqual(result, "3", "test_pr should return '3' here, matching the result of getPR") + + result = self.client1.max_package_pr(version, pkgarch) + self.assertEqual(result, "3", "max_package_pr should return '3' in the current test series") + + # Ask again for the first configuration + + result = self.client1.getPR(version, pkgarch, checksum0) + self.assertEqual(result, "4", "getPR: should return '4' in this configuration") + + # Ask again with explicit "no history" mode + + result = self.client1.getPR(version, pkgarch, checksum0, False) + self.assertEqual(result, "4", "getPR: should return '4' in this configuration") + + # Ask again with explicit "history" mode. This should return the first recorded PR for checksum0 + + result = self.client1.getPR(version, pkgarch, checksum0, True) + self.assertEqual(result, "0", "getPR: should return '0' in this configuration") + + # Check again that another pkgarg resets the counters + + result = self.client1.test_pr(version, other_arch, checksum0) + self.assertIsNone(result, "test_pr should return 'None' for a non existing PR") + + result = self.client1.test_package(version, other_arch) + self.assertFalse(result, "test_package should return 'False' for a non existing PR") + + result = self.client1.max_package_pr(version, other_arch) + self.assertIsNone(result, "max_package_pr should return 'None' for a non existing PR") + + # Now add the configuration + + result = self.client1.getPR(version, other_arch, checksum0) + self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'") + + result = self.client1.test_pr(version, other_arch, checksum0) + self.assertEqual(result, "0", "test_pr should return '0' here, matching the result of getPR") + + result = self.client1.test_package(version, other_arch) + self.assertTrue(result, "test_package should return 'True' for an existing PR") + + result = self.client1.max_package_pr(version, other_arch) + self.assertEqual(result, "0", "max_package_pr should return '0' in the current test series") + + result = self.client1.is_readonly() + self.assertFalse(result, "Server should not be described as 'read-only'") + +class PRUpstreamTests(PRTestSetup, unittest.TestCase): + + def setUp(self): + + self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv') + self.addCleanup(self.temp_dir.cleanup) + + dbfile2 = os.path.join(self.temp_dir.name, "prtest-upstream2.sqlite3") + self.server2 = self.start_server("upstream2", dbfile2) + self.client2 = self.start_client(self.server2.address) + + dbfile1 = os.path.join(self.temp_dir.name, "prtest-upstream1.sqlite3") + self.server1 = self.start_server("upstream1", dbfile1, upstream=self.server2.address) + self.client1 = self.start_client(self.server1.address) + + dbfile0 = os.path.join(self.temp_dir.name, "prtest-local.sqlite3") + self.server0 = self.start_server("local", dbfile0, upstream=self.server1.address) + self.client0 = self.start_client(self.server0.address) + self.shared_db = dbfile0 + + def test_upstream_and_readonly(self): + + # For identical checksums, all servers should return the same PR + + result = self.client2.getPR(version, pkgarch, checksum0) + self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'") + + result = self.client1.getPR(version, pkgarch, checksum0) + self.assertEqual(result, "0", "getPR: initial PR of a package should be '0' (same as upstream)") + + result = self.client0.getPR(version, pkgarch, checksum0) + self.assertEqual(result, "0", "getPR: initial PR of a package should be '0' (same as upstream)") + + # Now introduce new checksums on server1 for, same version + + result = self.client1.getPR(version, pkgarch, checksum1) + self.assertEqual(result, "0.0", "getPR: first PR of a package which has a different checksum upstream should be '0.0'") + + result = self.client1.getPR(version, pkgarch, checksum2) + self.assertEqual(result, "0.1", "getPR: second PR of a package that has a different checksum upstream should be '0.1'") + + # Now introduce checksums on server0 for, same version + + result = self.client1.getPR(version, pkgarch, checksum1) + self.assertEqual(result, "0.2", "getPR: can't decrease for known PR") + + result = self.client1.getPR(version, pkgarch, checksum2) + self.assertEqual(result, "0.3") + + result = self.client1.max_package_pr(version, pkgarch) + self.assertEqual(result, "0.3") + + result = self.client0.getPR(version, pkgarch, checksum3) + self.assertEqual(result, "0.3.0", "getPR: first PR of a package that doesn't exist upstream should be '0.3.0'") + + result = self.client0.getPR(version, pkgarch, checksum4) + self.assertEqual(result, "0.3.1", "getPR: second PR of a package that doesn't exist upstream should be '0.3.1'") + + result = self.client0.getPR(version, pkgarch, checksum3) + self.assertEqual(result, "0.3.2") + + # More upstream updates + # Here, we assume no communication between server2 and server0. server2 only impacts server0 + # after impacting server1 + + self.assertEqual(self.client2.getPR(version, pkgarch, checksum5), "1") + self.assertEqual(self.client1.getPR(version, pkgarch, checksum6), "1.0") + self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), "1.1") + self.assertEqual(self.client0.getPR(version, pkgarch, checksum8), "1.1.0") + self.assertEqual(self.client0.getPR(version, pkgarch, checksum9), "1.1.1") + + # "history" mode tests + + self.assertEqual(self.client2.getPR(version, pkgarch, checksum0, True), "0") + self.assertEqual(self.client1.getPR(version, pkgarch, checksum2, True), "0.1") + self.assertEqual(self.client0.getPR(version, pkgarch, checksum3, True), "0.3.0") + + # More "no history" mode tests + + self.assertEqual(self.client2.getPR(version, pkgarch, checksum0), "2") + self.assertEqual(self.client1.getPR(version, pkgarch, checksum0), "2") # Same as upstream + self.assertEqual(self.client0.getPR(version, pkgarch, checksum0), "2") # Same as upstream + self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), "3") # This could be surprising, but since the previous revision was "2", increasing it yields "3". + # We don't know how many upstream servers we have + # Start read-only server with server1 as upstream + self.server_ro = self.start_server("local-ro", self.shared_db, upstream=self.server1.address, read_only=True) + self.client_ro = self.start_client(self.server_ro.address) + + self.assertTrue(self.client_ro.is_readonly(), "Database should be described as 'read-only'") + + # Checks on non existing configurations + self.assertIsNone(self.client_ro.test_pr(version, pkgarch, checksumX)) + self.assertFalse(self.client_ro.test_package("unknown", pkgarch)) + + # Look up existing configurations + self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum0), "3") # "no history" mode + self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum0, True), "0") # "history" mode + self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum3), "3") + self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum3, True), "0.3.0") + self.assertEqual(self.client_ro.max_package_pr(version, pkgarch), "2") # normal as "3" was never saved + + # Try to insert a new value. Here this one is know upstream. + self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum7), "3") + # Try to insert a completely new value. As the max upstream value is already "3", it should be "3.0" + self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum10), "3.0") + # Same with another value which only exists in the upstream upstream server + # This time, as the upstream server doesn't know it, it will ask its upstream server. So that's a known one. + self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum9), "3") + +class ScriptTests(unittest.TestCase): + + def setUp(self): + + self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv') + self.addCleanup(self.temp_dir.cleanup) + self.dbfile = os.path.join(self.temp_dir.name, "prtest.sqlite3") + + def test_1_start_bitbake_prserv(self): + try: + subprocess.check_call([BIN_DIR / "bitbake-prserv", "--start", "-f", self.dbfile]) + except subprocess.CalledProcessError as e: + self.fail("Failed to start bitbake-prserv: %s" % e.returncode) + + def test_2_stop_bitbake_prserv(self): + try: + subprocess.check_call([BIN_DIR / "bitbake-prserv", "--stop"]) + except subprocess.CalledProcessError as e: + self.fail("Failed to stop bitbake-prserv: %s" % e.returncode)