From patchwork Fri Oct 6 15:36:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 31778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 310C1E81E0D for ; Fri, 6 Oct 2023 15:36:52 +0000 (UTC) Received: from mail-oa1-f53.google.com (mail-oa1-f53.google.com [209.85.160.53]) by mx.groups.io with SMTP id smtpd.web11.16219.1696606611272205649 for ; Fri, 06 Oct 2023 08:36:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=FdGTVD8o; spf=pass (domain: gmail.com, ip: 209.85.160.53, mailfrom: jpewhacker@gmail.com) Received: by mail-oa1-f53.google.com with SMTP id 586e51a60fabf-1dceb2b8823so1142170fac.1 for ; Fri, 06 Oct 2023 08:36:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696606610; x=1697211410; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3vClnEF6TsFaqvx7Nu15ggYBbXirYCuUUUmR4LIkyv0=; b=FdGTVD8oK98gbhI3PhH5/ushlpQ7UXRFxHgMBYC1TuDeC/C+ZTa2AgYz5chKlkBeWY bNm/XvpVbfRSNnjN1ebw7FeO56wvKqpn3Qffk+KafDbUKfbbyry4geoBnLL0eJPvaSnU hhNczj8c43em5KbhZvw3lM2O9teZp7bsBfp9i4Q77GrPpaVdnV9AR0yh7XWvWQH5yWb3 1azV6e7cMoFFgNoWm1M1zZXssTthGVoQMbn11y1zlKov1ySLRLPz2BQVHl1fzZMsSxVR UHnbpolrRvM1aFgHAH9RJHHpBsCF4UmLO1VuO6olBvloyXdaLtqu5KsV+UhohLThA1om Mp1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696606610; x=1697211410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3vClnEF6TsFaqvx7Nu15ggYBbXirYCuUUUmR4LIkyv0=; b=FYY0QisTwLQw2OiOy52i4nvDSDt0mSq3OGJ7knVWyrxvTbRE44fjfOY3KKAtZ3sca7 kxd90GHXDksn6K+Dyrif7qqtU+dbWxWyIaqw9L6r2uVUTY71FdTlpgyQ5KmnatY+7GQF zo3MN5QNIYmk6DVhbbyHeYEGpaJ+XZlyUXdA/NXIQo85VfXQ0fjodsNBcv1cwDk7fGNp vax7OPlgbOFE5szuwnrfgbhRKMIX6oe16YaGKcd68sYJArJDucIcwi2b/ZYZvrgvNyr+ LvWu0O7rmN1rnI+3s2xoGh4Mjl4vEIHgj7wyvvACI2bLuMjNngk0Nw7PYzhrJEV/Zwkx X9SQ== X-Gm-Message-State: AOJu0YxIOi9g81eFifOm5GZ619TKRqV5abc8ZDQyt5CReV9DRQT7Qjri r5Xpj7AAxPZXfbzc0ioXuCK3XEYFAaA= X-Google-Smtp-Source: AGHT+IHFq56CzkTlDXLYM7Qf6vxvLPU+Qx7/AI6jLbIFqIOEFjFzwrIujKHDtoCNGJg4i/gQH3Ib9w== X-Received: by 2002:a05:6870:231c:b0:1d4:eefb:3ae5 with SMTP id w28-20020a056870231c00b001d4eefb3ae5mr2693864oao.15.1696606609920; Fri, 06 Oct 2023 08:36:49 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::8282]) by smtp.gmail.com with ESMTPSA id 26-20020a056870135a00b001dd395339c3sm780426oac.19.2023.10.06.08.36.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Oct 2023 08:36:48 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 1/5] hashserv: Add remove API Date: Fri, 6 Oct 2023 09:36:41 -0600 Message-Id: <20231006153645.1609760-2-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231006153645.1609760-1-JPEWhacker@gmail.com> References: <20231006153645.1609760-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Oct 2023 15:36:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15188 Adds a `remove` API to the client and server that can be used to remove hash equivalence entries that match a particular critera Signed-off-by: Joshua Watt --- bitbake/lib/hashserv/client.py | 5 +++++ bitbake/lib/hashserv/server.py | 28 ++++++++++++++++++++++++++++ bitbake/lib/hashserv/tests.py | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+) diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.py index b2aa1026ac9..7446e4c9f67 100644 --- a/bitbake/lib/hashserv/client.py +++ b/bitbake/lib/hashserv/client.py @@ -101,6 +101,10 @@ class AsyncClient(bb.asyncrpc.AsyncClient): await self._set_mode(self.MODE_NORMAL) return (await self.send_message({"backfill-wait": None}))["tasks"] + async def remove(self, where): + await self._set_mode(self.MODE_NORMAL) + return await self.send_message({"remove": {"where": where}}) + class Client(bb.asyncrpc.Client): def __init__(self): @@ -115,6 +119,7 @@ class Client(bb.asyncrpc.Client): "get_stats", "reset_stats", "backfill_wait", + "remove", ) def _get_async_client(self): diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.py index d40a2ab8f88..daf1ffacbb9 100644 --- a/bitbake/lib/hashserv/server.py +++ b/bitbake/lib/hashserv/server.py @@ -186,6 +186,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection): 'report-equiv': self.handle_equivreport, 'reset-stats': self.handle_reset_stats, 'backfill-wait': self.handle_backfill_wait, + 'remove': self.handle_remove, }) def validate_proto_version(self): @@ -499,6 +500,33 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection): await self.backfill_queue.join() self.write_message(d) + async def handle_remove(self, request): + condition = request["where"] + if not isinstance(condition, dict): + raise TypeError("Bad condition type %s" % type(condition)) + + def do_remove(columns, table_name, cursor): + nonlocal condition + where = {} + for c in columns: + if c in condition and condition[c] is not None: + where[c] = condition[c] + + if where: + query = ('DELETE FROM %s WHERE ' % table_name) + ' AND '.join("%s=:%s" % (k, k) for k in where.keys()) + cursor.execute(query, where) + return cursor.rowcount + + return 0 + + count = 0 + with closing(self.db.cursor()) as cursor: + count += do_remove(OUTHASH_TABLE_COLUMNS, "outhashes_v2", cursor) + count += do_remove(UNIHASH_TABLE_COLUMNS, "unihashes_v2", cursor) + self.db.commit() + + self.write_message({"count": count}) + def query_equivalent(self, cursor, method, taskhash): # This is part of the inner loop and must be as fast as possible cursor.execute( diff --git a/bitbake/lib/hashserv/tests.py b/bitbake/lib/hashserv/tests.py index f6b85aed85a..a3e066406e3 100644 --- a/bitbake/lib/hashserv/tests.py +++ b/bitbake/lib/hashserv/tests.py @@ -84,6 +84,7 @@ class HashEquivalenceCommonTests(object): result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash) self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash') + return taskhash, outhash, unihash def test_create_equivalent(self): # Tests that a second reported task with the same outhash will be @@ -125,6 +126,38 @@ class HashEquivalenceCommonTests(object): self.assertClientGetHash(self.client, taskhash, unihash) + def test_remove_taskhash(self): + taskhash, outhash, unihash = self.test_create_hash() + result = self.client.remove({"taskhash": taskhash}) + self.assertGreater(result["count"], 0) + self.assertClientGetHash(self.client, taskhash, None) + + result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash) + self.assertIsNone(result_outhash) + + def test_remove_unihash(self): + taskhash, outhash, unihash = self.test_create_hash() + result = self.client.remove({"unihash": unihash}) + self.assertGreater(result["count"], 0) + self.assertClientGetHash(self.client, taskhash, None) + + def test_remove_outhash(self): + taskhash, outhash, unihash = self.test_create_hash() + result = self.client.remove({"outhash": outhash}) + self.assertGreater(result["count"], 0) + + result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash) + self.assertIsNone(result_outhash) + + def test_remove_method(self): + taskhash, outhash, unihash = self.test_create_hash() + result = self.client.remove({"method": self.METHOD}) + self.assertGreater(result["count"], 0) + self.assertClientGetHash(self.client, taskhash, None) + + result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash) + self.assertIsNone(result_outhash) + def test_huge_message(self): # Simple test that hashes can be created taskhash = 'c665584ee6817aa99edfc77a44dd853828279370' From patchwork Fri Oct 6 15:36:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 31779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B3AFE81E10 for ; Fri, 6 Oct 2023 15:37:02 +0000 (UTC) Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) by mx.groups.io with SMTP id smtpd.web10.16163.1696606612989330060 for ; Fri, 06 Oct 2023 08:36:53 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Mw6/56Pl; spf=pass (domain: gmail.com, ip: 209.85.160.50, mailfrom: jpewhacker@gmail.com) Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-1dd5b98d9aeso1148203fac.0 for ; Fri, 06 Oct 2023 08:36:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696606611; x=1697211411; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GUoWHkynR4Ogd1i339u7tNTcw45boZtbQCJfU0l7Hm0=; b=Mw6/56PlsZtZtQrcWjNbSQsymr1KA6CAqStB44uFi90sXpDw9UNq9ElAsIQviIkE9k /mXZtQKDHHZ2gGAGjupoe5aU7+JfXPWUYvniejQcjkRzw4AJQycsojfTyAHOqorBhS/O LqpmG0V3W5tRrGPcuQc1g20YmC3EnxjGlraZeODm1BJg2mEiZ5peQqmpPdbsKowSH/Kz BGW3jr92Rbj7C0ROONpU25/iMDwshpvBWndDZ5xdZKxOtsRF80DcCHfbk9YR7OAxhRcG 4RoPfYfgQsz84qsDe8zna9HrAegOvDfR9bDvHri3pFt8WQBaqG0dKdJWC8OuriCHvGmX 202g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696606611; x=1697211411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GUoWHkynR4Ogd1i339u7tNTcw45boZtbQCJfU0l7Hm0=; b=mvyVI267GabGwm7CDAN+TBY0fO/J+h+QgbxAJ9qHWdBR1gHtsGB3EyJwMfFVyHiW15 avZ8e1HcLh7MNBrelhrtJ8jXcUPEPyXexKjIhupCEUWJlTYuYJpXv7xtnS17NUTACNif Yb2TmVuqZYNr2PeFLNhm/1VthVQZ8GxxuttPpbVi+LNeroCw+RrU3aeg/FQTbXgjVrG6 bX8zHayFT2cJNRStZmr9plHu/jfYdlEmd3vzaAkcdF3uT8Po8WxyorTWVnv8DIpdh9Nf TcyXtycn/DnlGiQRrY7YNB0rvpdzTDEtVf4xi7jRaeWJ/puSp2prjraS9/yL/llk/GsV sYNg== X-Gm-Message-State: AOJu0YyA1aeDoQjuYt7e5Gl2/KwDMnpxKtuQGCWyFYq1v/fcNaRSvcZM d2pYcGw4jjmk0t9XV/kARU0dmfmSMY8= X-Google-Smtp-Source: AGHT+IHdqg3euQ50e4wpJofCu8nM8qLCUvXr9chlMOIxi3dhFv3LuUwqU/RLEeMxeByAom0BCfEiPg== X-Received: by 2002:a05:6870:1cb:b0:19f:e5a8:7525 with SMTP id n11-20020a05687001cb00b0019fe5a87525mr2588222oad.12.1696606611443; Fri, 06 Oct 2023 08:36:51 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::8282]) by smtp.gmail.com with ESMTPSA id 26-20020a056870135a00b001dd395339c3sm780426oac.19.2023.10.06.08.36.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Oct 2023 08:36:50 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 2/5] bitbake-hashclient: Add remove subcommand Date: Fri, 6 Oct 2023 09:36:42 -0600 Message-Id: <20231006153645.1609760-3-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231006153645.1609760-1-JPEWhacker@gmail.com> References: <20231006153645.1609760-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Oct 2023 15:37:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15189 Adds a subcommand to invoke the remove API on the server Signed-off-by: Joshua Watt --- bitbake/bin/bitbake-hashclient | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient index 494f17592ac..d09104336ab 100755 --- a/bitbake/bin/bitbake-hashclient +++ b/bitbake/bin/bitbake-hashclient @@ -113,6 +113,14 @@ def main(): with lock: pbar.update() + def handle_remove(args, client): + where = {k: v for k, v in args.where} + if where: + result = client.remove(where) + print("Removed %d row(s)" % (result["count"])) + else: + print("No query specified") + parser = argparse.ArgumentParser(description='Hash Equivalence Client') parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")') parser.add_argument('--log', default='WARNING', help='Set logging level') @@ -137,6 +145,11 @@ def main(): help='Include string in outhash') stress_parser.set_defaults(func=handle_stress) + remove_parser = subparsers.add_parser('remove', help="Remove hash entries") + remove_parser.add_argument("--where", "-w", metavar="KEY VALUE", nargs=2, action="append", default=[], + help="Remove entries from table where KEY == VALUE") + remove_parser.set_defaults(func=handle_remove) + args = parser.parse_args() logger = logging.getLogger('hashserv') From patchwork Fri Oct 6 15:36:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 31780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AF5DE81E0D for ; Fri, 6 Oct 2023 15:37:02 +0000 (UTC) Received: from mail-oa1-f41.google.com (mail-oa1-f41.google.com [209.85.160.41]) by mx.groups.io with SMTP id smtpd.web10.16164.1696606614498358669 for ; Fri, 06 Oct 2023 08:36:54 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YFTsvDMa; spf=pass (domain: gmail.com, ip: 209.85.160.41, mailfrom: jpewhacker@gmail.com) Received: by mail-oa1-f41.google.com with SMTP id 586e51a60fabf-1dd4eed487dso1414643fac.3 for ; Fri, 06 Oct 2023 08:36:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696606613; x=1697211413; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nqOKYLqDivxAAK9NCZFkN/LEiY4n1fpj5xSxK4ODOXc=; b=YFTsvDMaWSYDDxgwRn/dmfNY/2x+7ezjHlTiRYlhVfVdv7yyzXhIRtcreq7Xa2kyAz P0l9KEhoZT/6iH//uqW6o5gGcrECzGHxoCW5aXvieDIyZaTkKmjZncaLD9UHTdpbvcGM KfnpQFN5BGSn4gnBmyPo/jcyG5q9MrzHaD29rMJXMFfRAhtW7DKaYZg0fSzhha5qxXfG ibqdtmcVgMVKxxIX0pQjrJ6uAw+EazdSh7DbNntM99iDbCtT8pfBMrmWN4Euh3Uwb2TO 4zi0AtMHHDemtDiZUVDsv68xREb+lWXtqDVHLV1xUfZzUHipUfyVZLmimiqCgV2mSN1k lsMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696606613; x=1697211413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nqOKYLqDivxAAK9NCZFkN/LEiY4n1fpj5xSxK4ODOXc=; b=G/0qlXnTh5XMn2IOFPqMNvD0HOkHgYkUJ9v9FVLyeYge2fm+BvZ2hL6O4JlifzNIzh OpTYZiSmkSbeYe6rqMxnjDhkTQq9csqXhd3Ltk7UWB5uXhcjezvGNFbn9AYKT8l5PJuO qPz4COiW0G5tpUwLDni9omulIlV0Q33JYZCAe6kfDbKrK3lHt4XSp7/GGUQFZtwHTcp2 gNXiT9ZssSfNf0kQEpXRweKzQf9p7n8tLk/Dq2xCUTLn/7syJMT/2ZyIZGy1/3SLzTm9 x6tacRD93gfqZIAdEGRE4ivqVrRp2MuCqgTpR0MjHW2kJdnV1xU6uaic0CbyYf2TumhA OXVA== X-Gm-Message-State: AOJu0Yy4oXUTlNoTRM5YP+Ws3zLy8KbN7h0xLXH63ZCuoEU8i/pV41Mf 6vg/uqW94tdkuKOU8VzyqkejXvLsFWQ= X-Google-Smtp-Source: AGHT+IGAJqlXmifVkWoywHLSGqq7nnmXebxgA2PdtJsAzhCG8cpHchOSDVZXb7XBFuaKXET/rStTtw== X-Received: by 2002:a05:6870:65aa:b0:1d0:afe4:2155 with SMTP id fp42-20020a05687065aa00b001d0afe42155mr9801496oab.38.1696606613090; Fri, 06 Oct 2023 08:36:53 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::8282]) by smtp.gmail.com with ESMTPSA id 26-20020a056870135a00b001dd395339c3sm780426oac.19.2023.10.06.08.36.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Oct 2023 08:36:51 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 3/5] hashserv: Extend get_outhash API to optionally include unihash Date: Fri, 6 Oct 2023 09:36:43 -0600 Message-Id: <20231006153645.1609760-4-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231006153645.1609760-1-JPEWhacker@gmail.com> References: <20231006153645.1609760-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Oct 2023 15:37:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15190 Extends the get_outhash API with a flag indicating whether to include the unihash in the output. This is means that the query doesn't require the unihash entry to be present to return a result Signed-off-by: Joshua Watt --- bitbake/lib/hashserv/client.py | 4 +-- bitbake/lib/hashserv/server.py | 45 ++++++++++++++++++++++------------ 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.py index 7446e4c9f67..eeafeabda05 100644 --- a/bitbake/lib/hashserv/client.py +++ b/bitbake/lib/hashserv/client.py @@ -83,10 +83,10 @@ class AsyncClient(bb.asyncrpc.AsyncClient): {"get": {"taskhash": taskhash, "method": method, "all": all_properties}} ) - async def get_outhash(self, method, outhash, taskhash): + async def get_outhash(self, method, outhash, taskhash, with_unihash=True): await self._set_mode(self.MODE_NORMAL) return await self.send_message( - {"get-outhash": {"outhash": outhash, "taskhash": taskhash, "method": method}} + {"get-outhash": {"outhash": outhash, "taskhash": taskhash, "method": method, "with_unihash": with_unihash}} ) async def get_stats(self): diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.py index daf1ffacbb9..d52e1d46df5 100644 --- a/bitbake/lib/hashserv/server.py +++ b/bitbake/lib/hashserv/server.py @@ -270,27 +270,42 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection): method = request['method'] outhash = request['outhash'] taskhash = request['taskhash'] + with_unihash = request.get("with_unihash", True) with closing(self.db.cursor()) as cursor: - d = await self.get_outhash(cursor, method, outhash, taskhash) + d = await self.get_outhash(cursor, method, outhash, taskhash, with_unihash) self.write_message(d) - async def get_outhash(self, cursor, method, outhash, taskhash): + async def get_outhash(self, cursor, method, outhash, taskhash, with_unihash=True): d = None - cursor.execute( - ''' - SELECT *, unihashes_v2.unihash AS unihash FROM outhashes_v2 - INNER JOIN unihashes_v2 ON unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash - WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash - ORDER BY outhashes_v2.created ASC - LIMIT 1 - ''', - { - 'method': method, - 'outhash': outhash, - } - ) + if with_unihash: + cursor.execute( + ''' + SELECT *, unihashes_v2.unihash AS unihash FROM outhashes_v2 + INNER JOIN unihashes_v2 ON unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash + WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash + ORDER BY outhashes_v2.created ASC + LIMIT 1 + ''', + { + 'method': method, + 'outhash': outhash, + } + ) + else: + cursor.execute( + """ + SELECT * FROM outhashes_v2 + WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash + ORDER BY outhashes_v2.created ASC + LIMIT 1 + """, + { + 'method': method, + 'outhash': outhash, + } + ) row = cursor.fetchone() if row is not None: From patchwork Fri Oct 6 15:36:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 31782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3921AE81E0F for ; Fri, 6 Oct 2023 15:37:02 +0000 (UTC) Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) by mx.groups.io with SMTP id smtpd.web11.16221.1696606616146265124 for ; Fri, 06 Oct 2023 08:36:56 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=MgAuX4Po; spf=pass (domain: gmail.com, ip: 209.85.167.179, mailfrom: jpewhacker@gmail.com) Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3af64a4c97eso1375001b6e.2 for ; Fri, 06 Oct 2023 08:36:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696606615; x=1697211415; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lVW3VRzH/4UqT044Xh0YM0w1oLYVoESPuAAjcL81a/c=; b=MgAuX4PogqqyO2TEuYJv5Jq9bcNVdrEY29MCoziDsUErgMcoctfcPFsGV6oWKjM3xr Vh6J7FHbo3iyYBjKuLHDK4XRgacK2gewSNhVGaXgEAekugauUXd0VhyWzzhOH5lqXf5y BkSAjTCDUh/38DHp7jnGxiLow1z4vXTKnwm5HQXCCCThjfY085dLi0zTGckHcsNAfe4k UwkOnei39Y6yKJrXBmutnbGV0B8+2OUHVkdoprnBPWs1oWQeCaEWyXcLA5Hf27zDpMN4 efS9msNEIwDSUlt5i91+jufdH+2S+nlha7i98my+6cv1wbpSjo6UkpkAVCezjCDMKbNV xf9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696606615; x=1697211415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lVW3VRzH/4UqT044Xh0YM0w1oLYVoESPuAAjcL81a/c=; b=ACJVeRWtxonpzxBVJIuRS68+Wtujf5JpLcTV7Y6/W68fddhRPhHRQh0Lhl09lyNH/8 IIP9kuI2sPAKl4PMFVt4M8TjHGJy06dbiwh+PFgdpvzLIZvBCfxhfZSpwGg3GE6GMwoK 6LLhtBcmhU6s7SppQ+8Bvsu2GDSV91ZTORHsCabb44yEryp7JaMzHBxue0NNgnLu1Lg7 7UC6Ae3rLJSqrfb9XoPrdfoY3EYSHxk1c0OT7ZeguRxHQujltG6r1tsp40Y38D8xvI0H i/t6XgDHUKKESzVg16zge/V9pjXgyKnVdqH7gOuBRMSKZ9JtfArqyTQLMyjHkv1TO02E ZACA== X-Gm-Message-State: AOJu0Yx9dzIjEZ2CAFqnk/zFaWCGSc8SurqGBR5AeInpAI8HHiQ1lTMw lFQKdXWN0iVoSxzIUgngQjeVCgra+10= X-Google-Smtp-Source: AGHT+IHbmDpu2bya1vXrW0fMZ2RoJWiwyMfqN5LKoAsLL9QA4RyK7kbJIqO+YYiC0g2jD8KevOgGBg== X-Received: by 2002:a05:6870:8a0b:b0:1bf:5b4:4a51 with SMTP id p11-20020a0568708a0b00b001bf05b44a51mr8883625oaq.58.1696606614706; Fri, 06 Oct 2023 08:36:54 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::8282]) by smtp.gmail.com with ESMTPSA id 26-20020a056870135a00b001dd395339c3sm780426oac.19.2023.10.06.08.36.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Oct 2023 08:36:53 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 4/5] hashserv: Add API to clean unused entries Date: Fri, 6 Oct 2023 09:36:44 -0600 Message-Id: <20231006153645.1609760-5-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231006153645.1609760-1-JPEWhacker@gmail.com> References: <20231006153645.1609760-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Oct 2023 15:37:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15191 Adds an API to remove unused entries in the outhash database based on age and if they are referenced by any unihash Signed-off-by: Joshua Watt --- bitbake/lib/hashserv/client.py | 5 +++++ bitbake/lib/hashserv/server.py | 20 +++++++++++++++++++- bitbake/lib/hashserv/tests.py | 19 +++++++++++++++++++ 3 files changed, 43 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.py index eeafeabda05..d5c981864a2 100644 --- a/bitbake/lib/hashserv/client.py +++ b/bitbake/lib/hashserv/client.py @@ -105,6 +105,10 @@ class AsyncClient(bb.asyncrpc.AsyncClient): await self._set_mode(self.MODE_NORMAL) return await self.send_message({"remove": {"where": where}}) + async def clean_unused(self, max_age): + await self._set_mode(self.MODE_NORMAL) + return await self.send_message({"clean_unused": {"max_age_seconds": max_age}}) + class Client(bb.asyncrpc.Client): def __init__(self): @@ -120,6 +124,7 @@ class Client(bb.asyncrpc.Client): "reset_stats", "backfill_wait", "remove", + "clean_unused", ) def _get_async_client(self): diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.py index d52e1d46df5..b2ca357b2b1 100644 --- a/bitbake/lib/hashserv/server.py +++ b/bitbake/lib/hashserv/server.py @@ -4,7 +4,7 @@ # from contextlib import closing, contextmanager -from datetime import datetime +from datetime import datetime, timedelta import enum import asyncio import logging @@ -187,6 +187,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection): 'reset-stats': self.handle_reset_stats, 'backfill-wait': self.handle_backfill_wait, 'remove': self.handle_remove, + 'clean_unused': self.handle_clean_unused, }) def validate_proto_version(self): @@ -542,6 +543,23 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection): self.write_message({"count": count}) + async def handle_clean_unused(self, request): + max_age = request["max_age_seconds"] + with closing(self.db.cursor()) as cursor: + cursor.execute( + """ + DELETE FROM outhashes_v2 WHERE created<:oldest AND NOT EXISTS ( + SELECT unihashes_v2.id FROM unihashes_v2 WHERE unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash LIMIT 1 + ) + """, + { + "oldest": datetime.now() - timedelta(seconds=-max_age) + } + ) + count = cursor.rowcount + + self.write_message({"count": count}) + def query_equivalent(self, cursor, method, taskhash): # This is part of the inner loop and must be as fast as possible cursor.execute( diff --git a/bitbake/lib/hashserv/tests.py b/bitbake/lib/hashserv/tests.py index a3e066406e3..f343c586b5d 100644 --- a/bitbake/lib/hashserv/tests.py +++ b/bitbake/lib/hashserv/tests.py @@ -158,6 +158,25 @@ class HashEquivalenceCommonTests(object): result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash) self.assertIsNone(result_outhash) + def test_clean_unused(self): + taskhash, outhash, unihash = self.test_create_hash() + + # Clean the database, which should not remove anything because all hashes an in-use + result = self.client.clean_unused(0) + self.assertEqual(result["count"], 0) + self.assertClientGetHash(self.client, taskhash, unihash) + + # Remove the unihash. The row in the outhash table should still be present + self.client.remove({"unihash": unihash}) + result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash, False) + self.assertIsNotNone(result_outhash) + + # Now clean with no minimum age which will remove the outhash + result = self.client.clean_unused(0) + self.assertEqual(result["count"], 1) + result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash, False) + self.assertIsNone(result_outhash) + def test_huge_message(self): # Simple test that hashes can be created taskhash = 'c665584ee6817aa99edfc77a44dd853828279370' From patchwork Fri Oct 6 15:36:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 31781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39B0DE81E14 for ; Fri, 6 Oct 2023 15:37:02 +0000 (UTC) Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) by mx.groups.io with SMTP id smtpd.web11.16223.1696606617454019270 for ; Fri, 06 Oct 2023 08:36:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=jx/FtFTk; spf=pass (domain: gmail.com, ip: 209.85.210.48, mailfrom: jpewhacker@gmail.com) Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-6b9e478e122so1521189a34.1 for ; Fri, 06 Oct 2023 08:36:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696606616; x=1697211416; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3g97yjGXbXFbP5HGnu6VO3o1P55wxI9Ev0ppm6C2408=; b=jx/FtFTkIsGwbaJZ9csz0iqenHmKuL8vIgLqln0C8hLb4bZD1pZdz5ZO3Uk0dz6QsS c43CSAoA3Ls6Wp8FLJTuUGFM6Mr2AhCOIxi3lJhxvkwLB+/Ij4awha6DilIcTuY5JJ5n 8w8Ye+EoGns7VdnQDiFs0zZzcpi7AGFHBMkpc1RQhkxzwgFDub/qLccIv5Ztqlj6Pdt2 /NPgNId3vCc/n4LYT9381bw18Mlpl9Ex9htRCSwoZUJYZSEiCVp5zfp/PWOskP/F7Vjc SfLUth3/wDL2rcdaBAK4Kla2l9GUX1rr9Jz61LelXCpR7jCH+pZ7SwRRil+3DQdTSfpj ZVKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696606616; x=1697211416; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3g97yjGXbXFbP5HGnu6VO3o1P55wxI9Ev0ppm6C2408=; b=hPxSOhdDXiQ3hxPuo2Jra6KMfNsQs7fNiP/Cw/dQqUrzhfbTmZI4ordQlcSjo2tmB+ 3URld01eZuMC1N1vzTLtpZyslZD1wyjXMW99hWVO2a7tPmWcRxWcNA5PlYzMl5l6UrBp GndxlHij4lIR65okhf3AH6FYqtxgvAJ9Sph8hFcFbKmwJc8lkoJiV1wOk95C1teiUzII 7nKGSnjoGAJr4c15SFCrgHpjianocw3BSVU5Zn4jWvFKdItM4iiFSfyrmWjz9l4hVqiY UUpkltrZ++9HVFG2BeqgbLPJd6mrzw+88FrxHKjs6j5Rwoi3VpQ6GCygdPLsaND5I6lw 3mlw== X-Gm-Message-State: AOJu0YzdG7GT27+6o+YGKOc8kSHCda9BAs8rtdvv3m9CO1L6Cvaezpie 7yWjg6ClySRpe/nFDD88LcYRF14OnmE= X-Google-Smtp-Source: AGHT+IEY0X1Z6YKWR9gYbp1Mv0hHkvdAs4gCFXLviqs/w8PAbzQufVzklC+0e555RfATB7j3kZj7uw== X-Received: by 2002:a05:6871:54f:b0:1bb:c50d:7451 with SMTP id t15-20020a056871054f00b001bbc50d7451mr9475114oal.46.1696606616143; Fri, 06 Oct 2023 08:36:56 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::8282]) by smtp.gmail.com with ESMTPSA id 26-20020a056870135a00b001dd395339c3sm780426oac.19.2023.10.06.08.36.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Oct 2023 08:36:55 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 5/5] bitbake-hashclient: Add clean-unused subcommand Date: Fri, 6 Oct 2023 09:36:45 -0600 Message-Id: <20231006153645.1609760-6-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231006153645.1609760-1-JPEWhacker@gmail.com> References: <20231006153645.1609760-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Oct 2023 15:37:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15192 Adds a subcommand to clean unused outhash entries from the server based on age Signed-off-by: Joshua Watt --- bitbake/bin/bitbake-hashclient | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient index d09104336ab..3f265e8fa7b 100755 --- a/bitbake/bin/bitbake-hashclient +++ b/bitbake/bin/bitbake-hashclient @@ -121,6 +121,11 @@ def main(): else: print("No query specified") + def handle_clean_unused(args, client): + result = client.clean_unused(args.max_age) + print("Removed %d rows" % (result["count"])) + return 0 + parser = argparse.ArgumentParser(description='Hash Equivalence Client') parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")') parser.add_argument('--log', default='WARNING', help='Set logging level') @@ -150,6 +155,10 @@ def main(): help="Remove entries from table where KEY == VALUE") remove_parser.set_defaults(func=handle_remove) + clean_unused_parser = subparsers.add_parser('clean-unused', help="Remove unused database entries") + clean_unused_parser.add_argument("max_age", metavar="SECONDS", type=int, help="Remove unused entries older than SECONDS old") + clean_unused_parser.set_defaults(func=handle_clean_unused) + args = parser.parse_args() logger = logging.getLogger('hashserv')