From patchwork Thu Nov 23 15:12:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lussier-Cullen X-Patchwork-Id: 35138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45C41C61DF7 for ; Thu, 23 Nov 2023 15:12:28 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.92933.1700752342850339738 for ; Thu, 23 Nov 2023 07:12:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=hdi6U+Sz; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: alexander.lussier-cullen@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id DA9C89C352F for ; Thu, 23 Nov 2023 10:12:21 -0500 (EST) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id Cm74XLC-4QG3; Thu, 23 Nov 2023 10:12:21 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 603299C35A3; Thu, 23 Nov 2023 10:12:21 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 603299C35A3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1700752341; bh=ehnCDmw+ssA7V8H9dpY5nFoN00CbPWSPDreEycSDBXA=; h=From:To:Date:Message-Id:MIME-Version; b=hdi6U+Sz3hYjFP9dJVMc/oagiO2zcTjmI5z6DpM3kNNMM0N/TUjf1V6kixZo3T32s Toc5IhjXOlWS/JWhyXVbrx8j/yzZzqQHuRjEPFUFwExRlIEMoI6C+g25ErK7/dPb/d 8dML5WvhjfhLluXNZQXwYGyaMe9LMDu9IJxLWMSOj6hhr7Wn12OtR54rWSO4m3f5kx +FRjpnazysxNFVkp0l0wp9fwfJ+k3dawcIb6049nDa1EpczeIOfySPTxlfxtRQX8We WRrjv+uEwGPwRTewdD23/vMkb0QulqCn+pD8s+I9Q0A83QJUaao8gSyWVVbTPSzoDm h2C4y+yXWi6rQ== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id Bc4C2dUTc4jX; Thu, 23 Nov 2023 10:12:21 -0500 (EST) Received: from alex-GA-78LMT-USB3.home (bras-base-mtrlpq427kw-grc-24-184-147-255-35.dsl.bell.ca [184.147.255.35]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 4630E9C3506; Thu, 23 Nov 2023 10:12:21 -0500 (EST) From: Alexander Lussier-Cullen To: toaster@lists.yoctoproject.org Cc: Alexander Lussier-Cullen Subject: [PATCH 1/2] Toaster: tests: add passthroughs for relevant build environment variables Date: Thu, 23 Nov 2023 10:12:14 -0500 Message-Id: <20231123151215.4390-2-alexander.lussier-cullen@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231123151215.4390-1-alexander.lussier-cullen@savoirfairelinux.com> References: <20231123151215.4390-1-alexander.lussier-cullen@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 23 Nov 2023 15:12:28 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/toaster/message/6021 Toaster build tests create new build environments which do not utlize any existing build environment variables. In particular, DL_DIR and SSTATE_DIR from an existing environment can be passed in to allow for faster builds. Adding these as passthroughs specifically resolves slow builds related to the autobuilder integrations of the toaster test suite. Signed-off-by: Alexander Lussier-Cullen --- bitbake/lib/toaster/tests/builds/buildtest.py | 9 +++++++++ bitbake/lib/toaster/tox.ini | 3 +++ 2 files changed, 12 insertions(+) diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py b/bitbake/lib/toaster/tests/builds/buildtest.py index 13b51fb0d8..53cd7a9ffa 100644 --- a/bitbake/lib/toaster/tests/builds/buildtest.py +++ b/bitbake/lib/toaster/tests/builds/buildtest.py @@ -116,6 +116,15 @@ class BuildTest(unittest.TestCase): project = Project.objects.create_project(name=BuildTest.PROJECT_NAME, release=release) + passthrough_variable_names = ["SSTATE_DIR", "DL_DIR"] + for variable_name in passthrough_variable_names: + current_variable = os.environ.get(variable_name) + if current_variable: + ProjectVariable.objects.get_or_create( + name=variable_name, + value=current_variable, + project=project) + if os.environ.get("TOASTER_TEST_USE_SSTATE_MIRROR"): ProjectVariable.objects.get_or_create( name="SSTATE_MIRRORS", diff --git a/bitbake/lib/toaster/tox.ini b/bitbake/lib/toaster/tox.ini index d058558b05..9928657106 100644 --- a/bitbake/lib/toaster/tox.ini +++ b/bitbake/lib/toaster/tox.ini @@ -5,6 +5,9 @@ toxworkdir = {env:TOX_WORKDIR:.tox} passenv = * [testenv] +passenv = + SSTATE_DIR + DL_DIR setenv = DJANGO_SETTINGS_MODULE=toastermain.settings_test TOASTER_BUILDSERVER=1 From patchwork Thu Nov 23 15:12:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lussier-Cullen X-Patchwork-Id: 35137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 460C4C624B4 for ; Thu, 23 Nov 2023 15:12:28 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.92932.1700752342777540453 for ; Thu, 23 Nov 2023 07:12:22 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=mSQ+IQHr; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: alexander.lussier-cullen@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 14E8B9C3599 for ; Thu, 23 Nov 2023 10:12:22 -0500 (EST) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id EIBKKhawilE2; Thu, 23 Nov 2023 10:12:21 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 72EB79C3310; Thu, 23 Nov 2023 10:12:21 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 72EB79C3310 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1700752341; bh=UO/oY7iYXccou/mbe/PNwerTxLoQaJn9/07aq3vyuSA=; h=From:To:Date:Message-Id:MIME-Version; b=mSQ+IQHrgBlp3dseSRhWmHvLJVwk3Ps2Svgl1bYYR+Hwvw8mGdKMYznrqdywRn6XI TpWMN8J57NY8IwxsWJQWVF0eM8iAFvSgdeTGRVQKAMP4BcEZyOeM5kNW5OP/g+tYtc 0hUX/JRdLPLeUHli7QrO0iAkUC873Ur/fYOKPI8FPOWS1/YG0iIQUa1hfkaqXrfSnZ dWNhsWDf0kmAO6w5Yt9CyJENy/nHxdpexPyTnOCJM1yLGC2dtwYtZBhpvVsON8B4zF JkfxqgGvvV5hcVoiD3zESL8dBHWEbb1wdBlDRI4HPLW7HcWykVjSNb9QtBFeA9OYrc lQxS1tGNxAv6A== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id jtGVzB5pVT1C; Thu, 23 Nov 2023 10:12:21 -0500 (EST) Received: from alex-GA-78LMT-USB3.home (bras-base-mtrlpq427kw-grc-24-184-147-255-35.dsl.bell.ca [184.147.255.35]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 57B0C9C3599; Thu, 23 Nov 2023 10:12:21 -0500 (EST) From: Alexander Lussier-Cullen To: toaster@lists.yoctoproject.org Cc: Alexander Lussier-Cullen Subject: [PATCH 2/2] Toaster: make django temp directory configurable Date: Thu, 23 Nov 2023 10:12:15 -0500 Message-Id: <20231123151215.4390-3-alexander.lussier-cullen@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231123151215.4390-1-alexander.lussier-cullen@savoirfairelinux.com> References: <20231123151215.4390-1-alexander.lussier-cullen@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 23 Nov 2023 15:12:28 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/toaster/message/6020 Modify the django settings to make cache, logs and databases save to a configurable directory with TOASTER_DJANGO_TMPDIR instead of the machine root level /tmp directory. In addition, add this variable to the passthroughs for tox. This change is made to facilitate cleaning test files for runs on the autobuilder and avoid errors involving the persisted database. Signed-off-by: Alexander Lussier-Cullen --- bitbake/lib/toaster/toastermain/settings.py | 7 ++++--- bitbake/lib/toaster/toastermain/settings_test.py | 4 ++-- bitbake/lib/toaster/tox.ini | 1 + 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/bitbake/lib/toaster/toastermain/settings.py b/bitbake/lib/toaster/toastermain/settings.py index 69aa5a10c0..3c12359366 100644 --- a/bitbake/lib/toaster/toastermain/settings.py +++ b/bitbake/lib/toaster/toastermain/settings.py @@ -148,6 +148,8 @@ STATICFILES_FINDERS = ( # Make this unique, and don't share it with anybody. SECRET_KEY = 'NOT_SUITABLE_FOR_HOSTED_DEPLOYMENT' +TMPDIR = os.environ.get('TOASTER_DJANGO_TMPDIR', '/tmp') + class InvalidString(str): def __mod__(self, other): from django.template.base import TemplateSyntaxError @@ -214,7 +216,7 @@ CACHES = { # }, 'default': { 'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache', - 'LOCATION': '/tmp/toaster_cache_%d' % os.getuid(), + 'LOCATION': '%s/toaster_cache_%d' % (TMPDIR, os.getuid()), 'TIMEOUT': 1, } } @@ -312,7 +314,7 @@ for t in os.walk(os.path.dirname(currentdir)): LOGGING = LOGGING_SETTINGS # Build paths inside the project like this: BASE_DIR / 'subdir'. -BUILDDIR = os.environ.get("BUILDDIR", "/tmp") +BUILDDIR = os.environ.get("BUILDDIR", TMPDIR) # LOG VIEWER # https://pypi.org/project/django-log-viewer/ @@ -325,7 +327,6 @@ LOG_VIEWER_PATTERNS = ['INFO', 'DEBUG', 'WARNING', 'ERROR', 'CRITICAL'] # Optionally you can set the next variables in order to customize the admin: LOG_VIEWER_FILE_LIST_TITLE = "Logs list" - if DEBUG and SQL_DEBUG: LOGGING['loggers']['django.db.backends'] = { 'level': 'DEBUG', diff --git a/bitbake/lib/toaster/toastermain/settings_test.py b/bitbake/lib/toaster/toastermain/settings_test.py index 6538d9e453..74def2d240 100644 --- a/bitbake/lib/toaster/toastermain/settings_test.py +++ b/bitbake/lib/toaster/toastermain/settings_test.py @@ -19,10 +19,10 @@ TEMPLATE_DEBUG = DEBUG DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': '/tmp/toaster-test-db.sqlite', + 'NAME': '%s/toaster-test-db.sqlite' % TMPDIR, 'TEST': { 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': '/tmp/toaster-test-db.sqlite', + 'NAME': '%s/toaster-test-db.sqlite' % TMPDIR, } } } diff --git a/bitbake/lib/toaster/tox.ini b/bitbake/lib/toaster/tox.ini index 9928657106..1516a527ae 100644 --- a/bitbake/lib/toaster/tox.ini +++ b/bitbake/lib/toaster/tox.ini @@ -8,6 +8,7 @@ passenv = * passenv = SSTATE_DIR DL_DIR + TOASTER_DJANGO_TMPDIR setenv = DJANGO_SETTINGS_MODULE=toastermain.settings_test TOASTER_BUILDSERVER=1