From patchwork Fri Nov 3 14:26:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42017C4167B for ; Fri, 3 Nov 2023 14:26:56 +0000 (UTC) Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by mx.groups.io with SMTP id smtpd.web10.53485.1699021608869583076 for ; Fri, 03 Nov 2023 07:26:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=BvvoRbzM; spf=pass (domain: gmail.com, ip: 209.85.160.45, mailfrom: jpewhacker@gmail.com) Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-1ef36a04931so1162787fac.2 for ; Fri, 03 Nov 2023 07:26:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699021607; x=1699626407; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Oyoz0e78yMtHubWzNi4MStFo2x++8IC463/wBGtuJJk=; b=BvvoRbzMwHrGCU5Gktrb0rufWRsXktNarn3+p5zBqubZvKn7zFRjkKl5dSJq23acWl G9vmsuQczjlGYI+9hRR2Fpid9qsZvDBscxmERwxycwKhW6PEFp9yN9VW93lwue/quxtk pLnVu33ySi2na1RfiXYDBm9Eg8Q8gOqp9lWxogi+8Xz0ha3DyjoGOcalqGpTvdKoRtRU KZskhvYAWlE3BDMFGXZKjGQgHgqxQCE8ejLf2aqUlmRE1mrqIANoGqG61tmyorKDEDIv oKw8Ck6BRkujXxO/3bjZoHUPrbBffgRRP9rVbtpLi9WR4mGye8Fjtoy30VSl/7aRRvfl cIog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699021607; x=1699626407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Oyoz0e78yMtHubWzNi4MStFo2x++8IC463/wBGtuJJk=; b=j/sfqsyc6aFE42RJ3AyKWtNvCQIXuYZFXyyG+f9kraSiRb3xSEqwgQ45A5RMFSp8Fc SgRiFpV3RUGp727CFC5AscNgxzmwF8PvQ2bL9jqHvZLfth6FruW+MEE0ccibk+HYiuHZ ijz0Xqe3AHmmaa2VN6MuvPoP+NSYtPj8qCWUpQOI3X7ecpYW64/AsB4vYTOsyTWY3UBs SkYHXNImDL0A1kMD+SuAo44Dgxm4l7HepJ0HXP0ujYuH39XcZG/5n36+O1aOKXdJ17jC 1vhBmfRjPbEl9WlQNdWV+rRzmsVOwPnNfTxHtYb9nykNExcZlsHXz5PhtqhJ4StI+LCV NGzw== X-Gm-Message-State: AOJu0YxK+ELJKz1Iej7CwiFlqOb3/3I9pXX4YQJbfpHGHBqyx9gndlTW 9Azzv2dCnZF85mzdCCyepm6qG3nLLJg= X-Google-Smtp-Source: AGHT+IHKKPDtI3/7BWE278A8HOsjJp4c1yuwF1m3e0MPK3TQSxnPAKmI3XIscGSOgwx1NeZ7LasRXQ== X-Received: by 2002:a05:6870:1017:b0:1ef:9f6c:3dea with SMTP id 23-20020a056870101700b001ef9f6c3deamr16187132oai.39.1699021606900; Fri, 03 Nov 2023 07:26:46 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::2fe0]) by smtp.gmail.com with ESMTPSA id bb29-20020a056871b21d00b001dcde628a6fsm308272oac.42.2023.11.03.07.26.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 07:26:46 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH v6 00/22] Bitbake Hash Server WebSockets, Alternate Database Backend, and User Management Date: Fri, 3 Nov 2023 08:26:18 -0600 Message-Id: <20231103142640.1936827-1-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231031172138.3577199-1-JPEWhacker@gmail.com> References: <20231031172138.3577199-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 03 Nov 2023 14:26:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15421 This patch series reworks the bitbake asyncrpc API to add a WebSockets implementation for both the client and server. The hash equivalence server is updated to allow using this new API (the PR server can also be updated in the future if desired). In addition, the database backed for the hash equivalence server is abstracted so that sqlalchemy can optionally be used instead of sqlite. This allows using "big metal" databases as the backend, which allows the hash equivalence server to scale to a large number of queries. Note that both websockets and sqlalchemy require 3rd party python modules to function. However, these modules are optional unless the user desires to use the APIs. Also, user management is added. This allows user accounts to be registered with the server and users can be given permissions to do certain operations on the server. Users are not (necessarily) required to login to access the server, as permissions can granted to anonymous users. The default permissions will give anonymous users the same permissions that they would have before user accounts were added so as to retain backward compatibility, but server admins will likely want to change this. V3: Remove RFC status; patches are ready for review V4: Fixed protocol breakage with mixing older and newer clients/servers V5: Fixed compatibility with Python 3.8 V6: Fixed protocol incompatibility when exiting stream state that broke mixing older and new clients/servers Joshua Watt (22): asyncrpc: Abstract sockets hashserv: Add websocket connection implementation asyncrpc: Add context manager API hashserv: tests: Add external database tests asyncrpc: Prefix log messages with client info bitbake-hashserv: Allow arguments from environment hashserv: Abstract database hashserv: Add SQLalchemy backend hashserv: Implement read-only version of "report" RPC asyncrpc: Add InvokeError asyncrpc: client: Prevent double closing of loop asyncrpc: client: Add disconnect API hashserv: Add user permissions hashserv: Add become-user API hashserv: Add db-usage API hashserv: Add database column query API hashserv: test: Add bitbake-hashclient tests bitbake-hashclient: Output stats in JSON format bitbake-hashserver: Allow anonymous permissions to be space separated hashserv: tests: Allow authentication for external server tests hashserv: Allow self-service deletion hashserv: server: Add owner if user is logged in bin/bitbake-hashclient | 145 +++++- bin/bitbake-hashserv | 132 ++++- lib/bb/asyncrpc/__init__.py | 33 +- lib/bb/asyncrpc/client.py | 120 ++--- lib/bb/asyncrpc/connection.py | 146 ++++++ lib/bb/asyncrpc/exceptions.py | 21 + lib/bb/asyncrpc/serv.py | 365 ++++++++----- lib/hashserv/__init__.py | 190 +++---- lib/hashserv/client.py | 147 +++++- lib/hashserv/server.py | 952 +++++++++++++++++++++------------- lib/hashserv/sqlalchemy.py | 427 +++++++++++++++ lib/hashserv/sqlite.py | 408 +++++++++++++++ lib/hashserv/tests.py | 736 +++++++++++++++++++++++++- lib/prserv/client.py | 8 +- lib/prserv/serv.py | 37 +- 15 files changed, 3060 insertions(+), 807 deletions(-) create mode 100644 lib/bb/asyncrpc/connection.py create mode 100644 lib/bb/asyncrpc/exceptions.py create mode 100644 lib/hashserv/sqlalchemy.py create mode 100644 lib/hashserv/sqlite.py