From patchwork Mon Dec 11 03:07:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marlon Rodriguez Garcia X-Patchwork-Id: 36013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17826C10F07 for ; Mon, 11 Dec 2023 03:07:20 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web11.269.1702264035313381094 for ; Sun, 10 Dec 2023 19:07:15 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=v7lK1Ftq; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: marlon.rodriguez-garcia@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 80E749C3D77; Sun, 10 Dec 2023 22:07:14 -0500 (EST) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id ILYR8R5b0sRP; Sun, 10 Dec 2023 22:07:09 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 545409C08AD; Sun, 10 Dec 2023 22:07:09 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 545409C08AD DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1702264029; bh=XEBE9Cq5r2yuiPnz4NCmN5xvNSHIEPw7956pidcAmtI=; h=From:To:Date:Message-Id:MIME-Version; b=v7lK1Ftqyrkw/eW7oUpLjuG/0azJ8fTaoqc/JaPUEhbH5Rb1T6yVHRcEVI1JEECpz ylDliuQcFzf9BIotQrU5jRXRORn+Pd/tenhKaIerMweyRfFlDa4ng4DEB/fgrmejCP R7SFNGC5DeGBttwxRwb6mueFQz9L1jQpF6KN/xuCHAVRbxG3eD7vaa8DIY9vfgIqn1 K9o0ApstMlk89ccX+VJO2NItW8UOi6fg3M4lO63GFkyxXnRej7lbkTix24N4mSfpqm RZeLIXmqlrFCpUN2SwyH3DB6YbeHbYpRN9p/Bm7hqRwTRovBferSD06ZTTX2jK0OM4 ss/cdctE0bskA== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id AiqXgHOwxl7v; Sun, 10 Dec 2023 22:07:09 -0500 (EST) Received: from savoirfairelinux.ht.home (modemcable141.201-58-74.mc.videotron.ca [74.58.201.141]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 296899C27A2; Sun, 10 Dec 2023 22:07:09 -0500 (EST) From: Marlon Rodriguez Garcia To: bitbake-devel@lists.openembedded.org, toaster@lists.yoctoproject.org Cc: Marlon Rodriguez Garcia Subject: [toaster][PATCHv5 1/1] toaster: Added new feature to import eventlogs from command line into toaster using replay functionality Date: Sun, 10 Dec 2023 22:07:06 -0500 Message-Id: <20231211030706.108629-2-marlon.rodriguez-garcia@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231211030706.108629-1-marlon.rodriguez-garcia@savoirfairelinux.com> References: <20231211030706.108629-1-marlon.rodriguez-garcia@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 11 Dec 2023 03:07:20 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/toaster/message/6074 Added a new button on the base template to access a new template. Added a model register the information on the builds and generate access links Added a form to include the option to load specific files Added jquery and ajax functions to block screen and redirect to build page when import eventlogs is trigger Added a new button on landing page linked to import build page, and set min-height of buttons in landing page for uniformity Removed test assertion to check command line build in content, because new button contains text Updated toaster_eventreplay to use library Fix test in test_layerdetails_page This feature uses the value from the variable BB_DEFAULT_EVENTLOG to read the files created by bitbake Exclude listing of files that don't contain the allvariables definitions used to replay builds This part of the feature should be revisited. Over a long period of time, the BB_DEFAULT_EVENTLOG will exponentially increase the size of the log file and cause bottlenecks when importing. Signed-off-by: Marlon Rodriguez Garcia --- bin/toaster-eventreplay | 80 ++----- lib/bb/ui/eventreplay.py | 86 ++++++++ lib/bb/ui/toasterui.py | 2 +- .../orm/migrations/0021_eventlogsimports.py | 22 ++ lib/toaster/orm/models.py | 9 + .../tests/browser/test_landing_page.py | 2 - .../tests/browser/test_layerdetails_page.py | 9 +- lib/toaster/toastergui/forms.py | 14 ++ lib/toaster/toastergui/static/css/default.css | 28 +++ lib/toaster/toastergui/templates/base.html | 3 +- .../templates/command_line_builds.html | 198 ++++++++++++++++++ lib/toaster/toastergui/templates/landing.html | 10 +- lib/toaster/toastergui/urls.py | 1 + lib/toaster/toastergui/views.py | 173 ++++++++++++++- 14 files changed, 559 insertions(+), 78 deletions(-) create mode 100644 lib/bb/ui/eventreplay.py create mode 100644 lib/toaster/orm/migrations/0021_eventlogsimports.py create mode 100644 lib/toaster/toastergui/forms.py create mode 100644 lib/toaster/toastergui/templates/command_line_builds.html diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay index 404b61f5..74a31932 100755 --- a/bin/toaster-eventreplay +++ b/bin/toaster-eventreplay @@ -30,79 +30,23 @@ sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'lib')) import bb.cooker from bb.ui import toasterui - -class EventPlayer: - """Emulate a connection to a bitbake server.""" - - def __init__(self, eventfile, variables): - self.eventfile = eventfile - self.variables = variables - self.eventmask = [] - - def waitEvent(self, _timeout): - """Read event from the file.""" - line = self.eventfile.readline().strip() - if not line: - return - try: - event_str = json.loads(line)['vars'].encode('utf-8') - event = pickle.loads(codecs.decode(event_str, 'base64')) - event_name = "%s.%s" % (event.__module__, event.__class__.__name__) - if event_name not in self.eventmask: - return - return event - except ValueError as err: - print("Failed loading ", line) - raise err - - def runCommand(self, command_line): - """Emulate running a command on the server.""" - name = command_line[0] - - if name == "getVariable": - var_name = command_line[1] - variable = self.variables.get(var_name) - if variable: - return variable['v'], None - return None, "Missing variable %s" % var_name - - elif name == "getAllKeysWithFlags": - dump = {} - flaglist = command_line[1] - for key, val in self.variables.items(): - try: - if not key.startswith("__"): - dump[key] = { - 'v': val['v'], - 'history' : val['history'], - } - for flag in flaglist: - dump[key][flag] = val[flag] - except Exception as err: - print(err) - return (dump, None) - - elif name == 'setEventMask': - self.eventmask = command_line[-1] - return True, None - - else: - raise Exception("Command %s not implemented" % command_line[0]) - - def getEventHandle(self): - """ - This method is called by toasterui. - The return value is passed to self.runCommand but not used there. - """ - pass +from bb.ui import eventreplay def main(argv): with open(argv[-1]) as eventfile: # load variables from the first line - variables = json.loads(eventfile.readline().strip())['allvariables'] - + variables = None + while line := eventfile.readline().strip(): + try: + variables = json.loads(line)['allvariables'] + break + except (KeyError, json.JSONDecodeError): + continue + if not variables: + sys.exit("Cannot find allvariables entry in event log file %s" % argv[-1]) + eventfile.seek(0) params = namedtuple('ConfigParams', ['observe_only'])(True) - player = EventPlayer(eventfile, variables) + player = eventreplay.EventPlayer(eventfile, variables) return toasterui.main(player, player, params) diff --git a/lib/bb/ui/eventreplay.py b/lib/bb/ui/eventreplay.py new file mode 100644 index 00000000..d62ecbfa --- /dev/null +++ b/lib/bb/ui/eventreplay.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# +# This file re-uses code spread throughout other Bitbake source files. +# As such, all other copyrights belong to their own right holders. +# + + +import os +import sys +import json +import pickle +import codecs + + +class EventPlayer: + """Emulate a connection to a bitbake server.""" + + def __init__(self, eventfile, variables): + self.eventfile = eventfile + self.variables = variables + self.eventmask = [] + + def waitEvent(self, _timeout): + """Read event from the file.""" + line = self.eventfile.readline().strip() + if not line: + return + try: + decodedline = json.loads(line) + if 'allvariables' in decodedline: + self.variables = decodedline['allvariables'] + return + if not 'vars' in decodedline: + raise ValueError + event_str = decodedline['vars'].encode('utf-8') + event = pickle.loads(codecs.decode(event_str, 'base64')) + event_name = "%s.%s" % (event.__module__, event.__class__.__name__) + if event_name not in self.eventmask: + return + return event + except ValueError as err: + print("Failed loading ", line) + raise err + + def runCommand(self, command_line): + """Emulate running a command on the server.""" + name = command_line[0] + + if name == "getVariable": + var_name = command_line[1] + variable = self.variables.get(var_name) + if variable: + return variable['v'], None + return None, "Missing variable %s" % var_name + + elif name == "getAllKeysWithFlags": + dump = {} + flaglist = command_line[1] + for key, val in self.variables.items(): + try: + if not key.startswith("__"): + dump[key] = { + 'v': val['v'], + 'history' : val['history'], + } + for flag in flaglist: + dump[key][flag] = val[flag] + except Exception as err: + print(err) + return (dump, None) + + elif name == 'setEventMask': + self.eventmask = command_line[-1] + return True, None + + else: + raise Exception("Command %s not implemented" % command_line[0]) + + def getEventHandle(self): + """ + This method is called by toasterui. + The return value is passed to self.runCommand but not used there. + """ + pass diff --git a/lib/bb/ui/toasterui.py b/lib/bb/ui/toasterui.py index ec5bd4f1..6bd21f18 100644 --- a/lib/bb/ui/toasterui.py +++ b/lib/bb/ui/toasterui.py @@ -385,7 +385,7 @@ def main(server, eventHandler, params): main.shutdown = 1 logger.info("ToasterUI build done, brbe: %s", brbe) - continue + break if isinstance(event, (bb.command.CommandCompleted, bb.command.CommandFailed, diff --git a/lib/toaster/orm/migrations/0021_eventlogsimports.py b/lib/toaster/orm/migrations/0021_eventlogsimports.py new file mode 100644 index 00000000..328eb575 --- /dev/null +++ b/lib/toaster/orm/migrations/0021_eventlogsimports.py @@ -0,0 +1,22 @@ +# Generated by Django 4.2.5 on 2023-11-23 18:44 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('orm', '0020_models_bigautofield'), + ] + + operations = [ + migrations.CreateModel( + name='EventLogsImports', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('name', models.CharField(max_length=255)), + ('imported', models.BooleanField(default=False)), + ('build_id', models.IntegerField(blank=True, null=True)), + ], + ), + ] diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py index 1098ad3f..19c96862 100644 --- a/lib/toaster/orm/models.py +++ b/lib/toaster/orm/models.py @@ -1868,6 +1868,15 @@ class Distro(models.Model): def __unicode__(self): return "Distro " + self.name + "(" + self.description + ")" +class EventLogsImports(models.Model): + name = models.CharField(max_length=255) + imported = models.BooleanField(default=False) + build_id = models.IntegerField(blank=True, null=True) + + def __str__(self): + return self.name + + django.db.models.signals.post_save.connect(invalidate_cache) django.db.models.signals.post_delete.connect(invalidate_cache) django.db.models.signals.m2m_changed.connect(invalidate_cache) diff --git a/lib/toaster/tests/browser/test_landing_page.py b/lib/toaster/tests/browser/test_landing_page.py index 7ec52a4b..06cc0f5c 100644 --- a/lib/toaster/tests/browser/test_landing_page.py +++ b/lib/toaster/tests/browser/test_landing_page.py @@ -211,5 +211,3 @@ class TestLandingPage(SeleniumTestCase): content = self.get_page_source() self.assertTrue(self.PROJECT_NAME in content, 'should show builds for project %s' % self.PROJECT_NAME) - self.assertFalse(self.CLI_BUILDS_PROJECT_NAME in content, - 'should not show builds for cli project') diff --git a/lib/toaster/tests/browser/test_layerdetails_page.py b/lib/toaster/tests/browser/test_layerdetails_page.py index cb7b915b..05ee88b0 100644 --- a/lib/toaster/tests/browser/test_layerdetails_page.py +++ b/lib/toaster/tests/browser/test_layerdetails_page.py @@ -68,6 +68,7 @@ class TestLayerDetailsPage(SeleniumTestCase): check that the new values exist""" self.get(self.url) + self.wait_until_visible("#add-remove-layer-btn") self.click("#add-remove-layer-btn") self.click("#edit-layer-source") @@ -105,7 +106,9 @@ class TestLayerDetailsPage(SeleniumTestCase): for save_btn in self.find_all(".change-btn"): save_btn.click() - self.click("#save-changes-for-switch") + self.wait_until_visible("#save-changes-for-switch", poll=3) + btn_save_chg_for_switch = self.find("#save-changes-for-switch") + self.driver.execute_script("arguments[0].click();", btn_save_chg_for_switch) self.wait_until_visible("#edit-layer-source") # Refresh the page to see if the new values are returned @@ -134,7 +137,9 @@ class TestLayerDetailsPage(SeleniumTestCase): new_dir = "/home/test/my-meta-dir" dir_input.send_keys(new_dir) - self.click("#save-changes-for-switch") + self.wait_until_visible("#save-changes-for-switch", poll=3) + btn_save_chg_for_switch = self.find("#save-changes-for-switch") + btn_save_chg_for_switch.click() self.wait_until_visible("#edit-layer-source") # Refresh the page to see if the new values are returned diff --git a/lib/toaster/toastergui/forms.py b/lib/toaster/toastergui/forms.py new file mode 100644 index 00000000..10c7ac40 --- /dev/null +++ b/lib/toaster/toastergui/forms.py @@ -0,0 +1,14 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# BitBake Toaster UI tests implementation +# +# Copyright (C) 2023 Savoir-faire Linux +# +# SPDX-License-Identifier: GPL-2.0-only +# + +from django import forms +from django.core.validators import FileExtensionValidator + +class LoadFileForm(forms.Form): + eventlog_file = forms.FileField(widget=forms.FileInput(attrs={'accept': '.json'})) diff --git a/lib/toaster/toastergui/static/css/default.css b/lib/toaster/toastergui/static/css/default.css index 5cd7e211..284355e7 100644 --- a/lib/toaster/toastergui/static/css/default.css +++ b/lib/toaster/toastergui/static/css/default.css @@ -367,3 +367,31 @@ h2.panel-title { font-size: 30px; } } } /* End copied in from newer version of Font-Awesome 4.3.0 */ + + +#overlay { + display: flex; + position: fixed; + top: 0; + left: 0; + width: 100%; + height: 100%; + background-color: rgba(0, 0, 0, 0.7); + align-items: center; + justify-content: center; + z-index: 999; +} + +.spinner { + border: 6px solid rgba(255, 255, 255, 0.3); + border-radius: 50%; + border-top: 6px solid #3498db; + width: 50px; + height: 50px; + animation: spin 1s linear infinite; +} + +@keyframes spin { + 0% { transform: rotate(0deg); } + 100% { transform: rotate(360deg); } +} diff --git a/lib/toaster/toastergui/templates/base.html b/lib/toaster/toastergui/templates/base.html index 041448d1..e90be696 100644 --- a/lib/toaster/toastergui/templates/base.html +++ b/lib/toaster/toastergui/templates/base.html @@ -132,7 +132,8 @@ {% if project_enable %} New project {% endif %} - + Import command line builds + diff --git a/lib/toaster/toastergui/templates/command_line_builds.html b/lib/toaster/toastergui/templates/command_line_builds.html new file mode 100644 index 00000000..95944c74 --- /dev/null +++ b/lib/toaster/toastergui/templates/command_line_builds.html @@ -0,0 +1,198 @@ +{% extends "base.html" %} +{% load projecttags %} +{% load humanize %} + +{% block title %} Import Builds from eventlogs - Toaster {% endblock %} + +{% block pagecontent %} + +
+
+ +
+
+
+
+
+
+

Import command line builds

+
+ {% if import_all %} +
+ +
+ {% endif %} +
+
+ {% if messages %} +
+ {% for message in messages %} +
{{message}}
+ {%endfor%} +
+ {% endif %} +
+

Import eventlog file

+
+ {% csrf_token %} +
+
+ +
{{ form.eventlog_file}}
+
+
+
+ +
+
+
+
+
+ +
+
+

Eventlogs from existing build directory: + + + + + + +

+ {% if files %} +
+ + + + + + + + + + {% for file in files %} + + + + + + {% endfor%} + +
NameSizeAction
+ {{file.name}} + {{file.size|filesizeformat}} + {% if file.imported == True and file.build_id is not None %} + Build Details + {% elif request.session.file == file.name or request.session.all_builds %} + + + + {%else%} + + + + {%endif%} +
+
+ {% else %} +
+
Sorry - no files found
+
+ {%endif%} +
+
+
+
+
+ + + +{% endblock %} diff --git a/lib/toaster/toastergui/templates/landing.html b/lib/toaster/toastergui/templates/landing.html index 22bbed69..589ee226 100644 --- a/lib/toaster/toastergui/templates/landing.html +++ b/lib/toaster/toastergui/templates/landing.html @@ -15,7 +15,7 @@

A web interface to OpenEmbedded and BitBake, the Yocto Project build system.

- + Toaster is ready to capture your command line builds

@@ -23,7 +23,7 @@ {% if lvs_nos %} {% if project_enable %}

- + Create your first Toaster project to run manage builds

@@ -42,6 +42,12 @@ {% endif %} +

+ + Import command line event logs from build directory + +

+