From patchwork Wed Aug 31 11:13:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 12145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9D16ECAAD4 for ; Wed, 31 Aug 2022 11:14:20 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web08.23993.1661944451699092633 for ; Wed, 31 Aug 2022 04:14:12 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Bb9jyzL2; spf=pass (domain: gmail.com, ip: 209.85.128.50, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f50.google.com with SMTP id v7-20020a1cac07000000b003a6062a4f81so11496781wme.1 for ; Wed, 31 Aug 2022 04:14:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc; bh=aCfN+01A1GAEPeLvo+v+lKDT24CcTK8ssBznLi8zgZs=; b=Bb9jyzL24m3VTh5/kB2euX4IY5jZiGBDTOuc2uNl33KmaNOPa7VYUPEpqe3iO4tAxV V6T1gFZlAmjNyDjqGyLgIFpaMpofilFbvQvb5b4UeUNsoDlSmtP/5vHT2GuHqQKDwspo CQ6ybBqnAjrWfU4AHx9jHK69xBy7pGUK4RJEE9+ZFh2exhmvNNFfls6If3CZ9R6GkmF4 1R4yiBtKJjztZPWj2wfk/My7Qd6aDa+6gY20br3OOA6vQosowEh5dnMQv7FJObe1MjfN H/ZZgifdvvyYnT2uyIHq7M8hjChBGIsEFIuJvqYWMQpnBch/0cGDHf9gEL9ANGHhNBU5 5gZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc; bh=aCfN+01A1GAEPeLvo+v+lKDT24CcTK8ssBznLi8zgZs=; b=n+pCD4h7H3xL8z8pf7RcfRyozwJdBjjqBxB0qBc4vsSHRou8c5WETjP939nrOOY5Kb 9OS+68JMjPRSuPmfdE79E0xqaT7UVXw9/yCHBE5UKkiyTbHAAK/2oXabkQyX2coVgQdx lPv+uPRV1yAZM7ytd5RCiI9KYZ52em50LWY6nNWi4uvdUyS92Jeu4ZIJPjW7lNJOVH6K /DUiIs3t5I9Kr80UvpWxA4r4gX5/4+p7/VI9jp+wEzpvyxjVpRoxUJrHVfn3yENpfC+o DTmONJSzJSnCINio5r3bVYOi/1m+eoXj59pVr7Ku9XR5HslONeJqs20IM7kpvEjPjCfl X3Xw== X-Gm-Message-State: ACgBeo14t7+KGor3nXAomKs+aqt6MBDnAya40myx8VjS/yAa8k0m2Wxk SlbwyDbdI6H12yVLRTtCDVWSIVJ64Q4= X-Google-Smtp-Source: AA6agR4RV68DI4D/PEt27RT24f/7MDRyL0FXuXgPgO6lJrESHqgMQCs93oXVDF5Zayiuu0t6VeUUpQ== X-Received: by 2002:a05:600c:1e89:b0:3a6:38de:7024 with SMTP id be9-20020a05600c1e8900b003a638de7024mr1675236wmb.184.1661944450178; Wed, 31 Aug 2022 04:14:10 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003a60edc3a44sm2324753wmb.5.2022.08.31.04.14.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 04:14:09 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Date: Wed, 31 Aug 2022 13:13:55 +0200 Message-Id: <20220831111401.3330342-1-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 31 Aug 2022 11:14:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/170101 specifically that ../../layer.conf exists, and that second-from-last component in the path is 'templates'. This requires tweaking template.conf creation in eSDK bbclass, as we need to ensure that the path in it is valid, and exists (which may not be the case if the SDK is poky-based). Signed-off-by: Alexander Kanavin --- meta/classes-recipe/populate_sdk_ext.bbclass | 3 ++- scripts/oe-setup-builddir | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass index 56e24c4eed..925cb313fc 100644 --- a/meta/classes-recipe/populate_sdk_ext.bbclass +++ b/meta/classes-recipe/populate_sdk_ext.bbclass @@ -438,7 +438,8 @@ python copy_buildsystem () { else: # Write a templateconf.cfg with open(baseoutpath + '/conf/templateconf.cfg', 'w') as f: - f.write('meta/conf\n') + f.write('meta/conf/templates/default\n') + os.makedirs(os.path.join(baseoutpath, core_meta_subdir, 'conf/templates/default'), exist_ok=True) # Ensure any variables set from the external environment (by way of # BB_ENV_PASSTHROUGH_ADDITIONS) are set in the SDK's configuration diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir index 5d644168cb..bf832ee0ca 100755 --- a/scripts/oe-setup-builddir +++ b/scripts/oe-setup-builddir @@ -61,6 +61,11 @@ if [ -n "$TEMPLATECONF" ]; then echo >&2 "Error: TEMPLATECONF value points to nonexistent directory '$TEMPLATECONF'" exit 1 fi + templatesdir=$(python3 -c "import sys; print(sys.argv[1].strip('/').split('/')[-2])" $TEMPLATECONF) + if [ ! -f "$TEMPLATECONF/../../layer.conf" -o $templatesdir != "templates" ]; then + echo >&2 "Error: TEMPLATECONF value (which is $TEMPLATECONF) must point to meta-some-layer/conf/templates/template-name" + exit 1 + fi fi OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample" OECORELOCALCONF="$TEMPLATECONF/local.conf.sample" From patchwork Wed Aug 31 11:13:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 12149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D911EC64991 for ; Wed, 31 Aug 2022 11:14:20 +0000 (UTC) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mx.groups.io with SMTP id smtpd.web12.24175.1661944453176482679 for ; Wed, 31 Aug 2022 04:14:13 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=QV/aBa28; spf=pass (domain: gmail.com, ip: 209.85.128.49, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f49.google.com with SMTP id k17so7182309wmr.2 for ; Wed, 31 Aug 2022 04:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=oZSvd3uJQVS03oevdB0iJXaforqWH3cNpyDTcl0aEzQ=; b=QV/aBa28rwHcN1AyvQM67ZDm/ON+eJvJETxxzcOoOEfoMUahi6OaS7uEsKwgR2lYLO AmFngtX4fVkNandgl7Fw9u/tBlQ/TgrqnradnOJq2ilCs54Be3Vxd1ihQH4I2uMi9Kdl Dz/da0Q7eR6IuE8pSCVajHuD6FgEUpIcyELhX/QPYtbMPHQFIT97ggwjz79D+uPL/hSf AJOTMfXH8U+9gH7+nJOfnEYgaAbnoj02TLNR3cMSOWnpCTyThw3T2UUZh2MwB9d2zpie c+MSkdpWg8d/zgZtoV4Aw0coESfof4WKk9JwX8GqxKQzIiyML5VzMIxWVt2bGktLfyIN 0skQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=oZSvd3uJQVS03oevdB0iJXaforqWH3cNpyDTcl0aEzQ=; b=buo5cBGGXe7odb2AUg4JHw+GBfzNs+S1jpQC19vGSsIBEhTZwvCNQI+judblUFVc7B d7QGCKKBWwcRHbgSNmH/vT0a2NQ785e1uq21n+C8a/QzeAfXgruMLw9MUnlhSW8L4bFG MFfjeRjisRof5+zVbU1SUqrLisElnE9c/VYrUgDyEyM1ac1tXVtgEL/JAnBaICiFP88p 3L7A8FOYSSkTQd4B7Y8q0qoiDWc0en+2zFhbaGxNniD5Un7S/YkBSup4xLPsGUN4kcxI YpFrtbMLJzKXYuQN/yP1pD9D5LNXdmBQyeh++DfjRWKxgtoalyHUp9azIfFGxfOrmM9x E2Bg== X-Gm-Message-State: ACgBeo3nUd75knaQPXIoStTrDuQGRZ3yh4fQknbun7KyzasSQGUMeNKF NZRN4sKZ/tb+nrLTtFlE4L7f5RHrnd4= X-Google-Smtp-Source: AA6agR6qbnLCH4Kws20gFgsqzErlOX4d+7L5WaDaSiNAF3ULD+KzNJhbmMocLNkkLxUMzBkynQTvyA== X-Received: by 2002:a05:600c:206:b0:3a5:abe9:8a91 with SMTP id 6-20020a05600c020600b003a5abe98a91mr1664372wmi.155.1661944451622; Wed, 31 Aug 2022 04:14:11 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003a60edc3a44sm2324753wmb.5.2022.08.31.04.14.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 04:14:10 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 2/7] bitbake-layers: add a command to save the active build configuration as a template into a layer Date: Wed, 31 Aug 2022 13:13:56 +0200 Message-Id: <20220831111401.3330342-2-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220831111401.3330342-1-alex@linutronix.de> References: <20220831111401.3330342-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 31 Aug 2022 11:14:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/170102 This is the reverse of setting up a build by pointing TEMPLATECONF to a directory with a template and running '. oe-init-build-env': this takes the config files from build/conf, replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies the config files into a specified layer under a specified template name. In many or perhaps most cases such static prefabricated configurations (that require no further editing) are just enough, and I believe they should be offered by the official configuration management. On the other hand, generating build configurations with a sufficiently versatile tool is a far more complex problem, and one we should try to tackle once we see where and how static configs fall short. Tooling to discover and select these templates when setting up a build will be provided later on. How to use: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers save-build-conf ../../meta-alex/ test-1 NOTE: Starting bitbake server... NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt You can try out the configuration with TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1 alex@Zen2:/srv/work/alex/poky/build-layersetup$ Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/buildconf.py | 85 ++++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 5 ++ 2 files changed, 90 insertions(+) create mode 100644 meta/lib/bblayers/buildconf.py diff --git a/meta/lib/bblayers/buildconf.py b/meta/lib/bblayers/buildconf.py new file mode 100644 index 0000000000..e07fc534e1 --- /dev/null +++ b/meta/lib/bblayers/buildconf.py @@ -0,0 +1,85 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil +import json + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return BuildConfPlugin() + +class BuildConfPlugin(LayerPlugin): + notes_fixme = """FIXME: Please place here the description of this build configuration. +It will be shown to the users when they set up their builds via TEMPLATECONF. +""" + + def _save_conf(self, templatename, templatepath, oecorepath, relpaths_to_oecore): + confdir = os.path.join(os.environ["BBPATH"], "conf") + destdir = os.path.join(templatepath, "conf", "templates", templatename) + os.makedirs(destdir, exist_ok=True) + + with open(os.path.join(confdir, "local.conf")) as src: + with open(os.path.join(destdir, "local.conf.sample"), 'w') as dest: + dest.write(src.read()) + + with open(os.path.join(confdir, "bblayers.conf")) as src: + with open(os.path.join(destdir, "bblayers.conf.sample"), 'w') as dest: + bblayers_data = src.read() + + for (abspath, relpath) in relpaths_to_oecore: + bblayers_data = bblayers_data.replace(abspath, "##OEROOT##/" + relpath) + dest.write(bblayers_data) + + with open(os.path.join(destdir, "conf-notes.txt"), 'w') as dest: + dest.write(self.notes_fixme) + + logger.info("""Configuration template placed into {} +Please review the files in there, and particularly provide a configuration description in {} +You can try out the configuration with +TEMPLATECONF={} . {}/oe-init-build-env build-try-{}""" +.format(destdir, os.path.join(destdir, "conf-notes.txt"), destdir, oecorepath, templatename)) + + def do_save_build_conf(self, args): + """ Save the currently active build configuration (conf/local.conf, conf/bblayers.conf) as a template into a layer.\n This template can later be used for setting up builds via TEMPLATECONF. """ + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + targetlayer = None + oecore = None + + for l in layers: + if l[0] == os.path.abspath(args.layerpath): + targetlayer = l[0] + if l[1] == 'meta': + oecore = os.path.dirname(l[0]) + + if not targetlayer: + logger.error("Layer {} not in one of the currently enabled layers:\n{}".format(args.layerpath, "\n".join([l[0] for l in layers]))) + elif not oecore: + logger.error("Openembedded-core not in one of the currently enabled layers:\n{}".format("\n".join([l[0] for l in layers]))) + else: + relpaths_to_oecore = [(l[0], os.path.relpath(l[0], start=oecore)) for l in layers] + self._save_conf(args.templatename, targetlayer, oecore, relpaths_to_oecore) + + def register_commands(self, sp): + parser_build_conf = self.add_command(sp, 'save-build-conf', self.do_save_build_conf, parserecipes=False) + parser_build_conf.add_argument('layerpath', + help='The path to the layer where the configuration template should be saved.') + parser_build_conf.add_argument('templatename', + help='The name of the configuration template.') diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 494fa892a3..549abe7d10 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -113,6 +113,11 @@ class BitbakeLayers(OESelftestTestCase): self.assertEqual(bb_vars['BBFILE_PRIORITY_%s' % layername], str(priority), 'BBFILE_PRIORITY_%s != %d' % (layername, priority)) + result = runCmd('bitbake-layers save-build-conf {} {}'.format(layerpath, "buildconf-1")) + for f in ('local.conf.sample', 'bblayers.conf.sample', 'conf-notes.txt'): + fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f) + self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath)) + def get_recipe_basename(self, recipe): recipe_file = "" result = runCmd("bitbake-layers show-recipes -f %s" % recipe) From patchwork Wed Aug 31 11:13:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 12146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFD19C0502C for ; Wed, 31 Aug 2022 11:14:20 +0000 (UTC) Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by mx.groups.io with SMTP id smtpd.web10.24367.1661944454529131915 for ; Wed, 31 Aug 2022 04:14:14 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=j/q2xw2G; spf=pass (domain: gmail.com, ip: 209.85.128.46, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f46.google.com with SMTP id s23so7174599wmj.4 for ; Wed, 31 Aug 2022 04:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=0h0W42jtYp5tUyPgtSblhR+Kj6ub1/4/y+7ZtEGbYrA=; b=j/q2xw2GATsdX+X4kqGaeBLf2WgJlhdu5SzRpl9qUz4A9oWspCsqAh3gqgAHlJLUiq Ugci2oUVW6MeoHPRcP6j+Zn1M6Ve6bXc1hjqkig/upq0yXolA0uO+vV4jTqcWokcZ+4N 5dqLpiLCa/bj+VocrhSGxL4yhgbk84tBFIbemBRdZxEsAfmbRx8kFyHRx1N+qtaFmT8x 60I3kLRnfObFwAgMmyAdguDAAqVTV9ONpP5pEVDS2S36RllgJhpMoLIAK/2E4njycimV /3UHNhlfWeAybrvBrGgAnjgsLuzQZXuVyujN4Vu4q+Oz2eLQcxrSv3BF22htVLOPdXET 7IiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=0h0W42jtYp5tUyPgtSblhR+Kj6ub1/4/y+7ZtEGbYrA=; b=I+1RzgAsG5V7fTBH/4cDX1mWrS/dUMqPOUGJxfUvYEGnkchGtkT2Y8MlmcoTse+fxk dj1gCRHRItFNMrSNu3jZaSzLhVQUJpNsVmZqQJHfFpUodFuo18l7mqi6jREeap1xP6m9 dWZnAihXiPdHXV446GUpXlIkNTBmd4G9HPtOVH4ZAR1csxHPQ8iAm2YnwlefPnYsrOpj 4i+rcGdqMD9QxSnf02sFAL9Q44QpYuPu6VdvsROsZz5Clkajh2dOLf99uVQf9zQJ6zbL v71V2h0HDYXh+E1Pw5ZTbw6eBF3Uw/xx9xnGRsl9yZJKh0vNJN4OKBO7B3KXcwCyRonj qdVA== X-Gm-Message-State: ACgBeo18LHwr4bIgDS6LV4suxzy3dQFUc8kHKdQunX6Q1DTHrgmS9Jh/ jq9Cf9sNNU586icw3Q0swg01lXhewpY= X-Google-Smtp-Source: AA6agR4KbaF+zfOhef+4pdGkzEu8Fez3Jbfub500pLr55+SSmc0vRkkIhzwVQc486X/2kbjMUA2jSw== X-Received: by 2002:a05:600c:4e8c:b0:3a6:11e:cc08 with SMTP id f12-20020a05600c4e8c00b003a6011ecc08mr1674030wmq.198.1661944452787; Wed, 31 Aug 2022 04:14:12 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003a60edc3a44sm2324753wmb.5.2022.08.31.04.14.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 04:14:12 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Joshua Watt , Alexander Kanavin Subject: [PATCH 3/7] meta/files: add layer setup JSON schema and example Date: Wed, 31 Aug 2022 13:13:57 +0200 Message-Id: <20220831111401.3330342-3-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220831111401.3330342-1-alex@linutronix.de> References: <20220831111401.3330342-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 31 Aug 2022 11:14:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/170103 From: Joshua Watt Defines a common schema for layer setup that can be consumed by tools to know how to fetch and assemble layers for end users. Also includes an example of the layer setup that constructs poky/meta-intel/imaginary product layer for reference. The schema can be used to validate a layer setup file with the commands: $ python3 -m pip install jsonschema $ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json Signed-off-by: Joshua Watt Alex: I made the following modifications to Joshua's original commit: - moved the files from meta/lib to meta/files - the example json showcases a multi-repo, multi-layer setup instead of just poky - closer to a typical product - added oe-selftest that validates the example json against the schema using python3-jsonschema-native - the schema is modified so that: -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items. This allows using them as dicts inside Python, and makes the json more compact and readable. -- added 'contains_this_file' property to source object -- replaced 'remote' property with a 'oneOf' definition for git with a specific 'git-remote' property. 'oneOf' is problematic when schema validation fails: the diagnostic is only that none of oneOf variants matched, which is too non-specific. -- added 'describe' property to 'git-remote' object. -- removed description property for a layer source: it is not clear how to add that when auto-generating the json Signed-off-by: Alexander Kanavin --- meta/files/layers.example.json | 72 +++++++++++++++++++ meta/files/layers.schema.json | 91 ++++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 16 ++++- 3 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 meta/files/layers.example.json create mode 100644 meta/files/layers.schema.json diff --git a/meta/files/layers.example.json b/meta/files/layers.example.json new file mode 100644 index 0000000000..3772404ec9 --- /dev/null +++ b/meta/files/layers.example.json @@ -0,0 +1,72 @@ +{ + "sources": { + "meta-alex": { + "contains_this_file": true, + "git-remote": { + "branch": "master", + "describe": "", + "remotes": { + "remote-alex": { + "uri": "https://github.com/kanavin/meta-alex" + } + }, + "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5" + }, + "layers": { + "meta-alex": { + "subpath": "" + } + }, + "path": "meta-alex" + }, + "meta-intel": { + "git-remote": { + "branch": "master", + "describe": "15.0-hardknott-3.3-310-g0a96edae", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/meta-intel" + } + }, + "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a" + }, + "layers": { + "meta-intel": { + "subpath": "" + } + }, + "path": "meta-intel" + }, + "poky": { + "git-remote": { + "branch": "akanavin/setup-layers", + "describe": "4.1_M1-374-g9dda719b2a", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/poky" + }, + "poky-contrib": { + "uri": "ssh://git@push.yoctoproject.org/poky-contrib" + } + }, + "rev": "9dda719b2a4727a4d43a6ab8d9e23f8ca68790ec" + }, + "layers": { + "meta": { + "subpath": "meta" + }, + "meta-poky": { + "subpath": "meta-poky" + }, + "meta-selftest": { + "subpath": "meta-selftest" + }, + "meta-yocto-bsp": { + "subpath": "meta-yocto-bsp" + } + }, + "path": "poky" + } + }, + "version": "1.0" +} diff --git a/meta/files/layers.schema.json b/meta/files/layers.schema.json new file mode 100644 index 0000000000..cd4ddd3dcd --- /dev/null +++ b/meta/files/layers.schema.json @@ -0,0 +1,91 @@ +{ + "description": "OpenEmbedder Layer Setup Manifest", + "type": "object", + "additionalProperties": false, + "required": [ + "version" + ], + "properties": { + "version": { + "description": "The version of this document; currently '1.0'", + "enum": ["1.0"] + }, + "sources": { + "description": "The dict of layer sources", + "type": "object", + "patternProperties": { ".*" : { + "type": "object", + "description": "The upstream source from which a set of layers may be fetched", + "additionalProperties": false, + "required": [ + "path" + ], + "properties": { + "path": { + "description": "The path where this layer source will be placed when fetching", + "type": "string" + }, + "contains_this_file": { + "description": "Whether the directory with the layer source also contains this json description. Tools may want to skip the checkout of the source then.", + "type": "boolean" + }, + "layers": { + "description": "The dict of layers to be used from this upstream source", + "type": "object", + "patternProperties": { ".*" : { + "description": "A layer from the upstream source", + "type": "object", + "additionalProperties": false, + "properties": { + "subpath": { + "description": "The subpath (relative to the source root) for this layer. Omit if the source root is the layer path", + "type": "string" + } + } + }} + }, + "git-remote": { + "description": "A remote git source from which to fetch", + "type": "object", + "additionalProperties": false, + "required": [ + "rev" + ], + "properties": { + "branch": { + "description": "The git branch to fetch (optional)", + "type": "string" + }, + "rev": { + "description": "The git revision to checkout", + "type": "string" + }, + "describe": { + "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).", + "type": "string" + }, + "remotes": { + "description": "The dict of git remotes to add to this repository", + "type": "object", + "patternProperties": { ".*" : { + "description": "A git remote", + "type": "object", + "addtionalProperties": false, + "required": [ + "uri" + ], + "properties": { + "uri": { + "description": "The URI for the remote", + "type": "string" + } + } + }} + } + } + } + } + } + }} + } +} diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 549abe7d10..c753a7b795 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -8,12 +8,16 @@ import os import re import oeqa.utils.ftools as ftools -from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars +from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake from oeqa.selftest.case import OESelftestTestCase class BitbakeLayers(OESelftestTestCase): + def setUpLocal(self): + bitbake("python3-jsonschema-native") + bitbake("-c addto_recipe_sysroot python3-jsonschema-native") + def test_bitbakelayers_layerindexshowdepends(self): result = runCmd('bitbake-layers layerindex-show-depends meta-poky') find_in_contents = re.search("openembedded-core", result.output) @@ -128,3 +132,13 @@ class BitbakeLayers(OESelftestTestCase): self.assertTrue(os.path.isfile(recipe_file), msg = "Can't find recipe file for %s" % recipe) return os.path.basename(recipe_file) + + def validate_layersjson(self, json): + python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'nativepython3') + jsonvalidator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'jsonschema') + jsonschema = os.path.join(get_bb_var('COREBASE'), 'meta/files/layers.schema.json') + result = runCmd("{} {} -i {} {}".format(python, jsonvalidator, json, jsonschema)) + + def test_validate_examplelayersjson(self): + json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json") + self.validate_layersjson(json) From patchwork Wed Aug 31 11:13:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 12148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7A8BECAAD1 for ; Wed, 31 Aug 2022 11:14:20 +0000 (UTC) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx.groups.io with SMTP id smtpd.web08.23994.1661944455917032992 for ; Wed, 31 Aug 2022 04:14:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=A/imvrHo; spf=pass (domain: gmail.com, ip: 209.85.128.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f51.google.com with SMTP id m17-20020a7bce11000000b003a5bedec07bso11513876wmc.0 for ; Wed, 31 Aug 2022 04:14:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=OYvYNmf/x4P7vSFlY+ls2p8t4uEh4HhTGMr1xOrMvUs=; b=A/imvrHoLndMy9nCjcccxwFXMZXKPCzUZjoltuRKSl/Vb727xEmzr+4T6rl2VPlNQd SlsUfT7NJBxz+xbeV1bLfeSJS2YfFjyxXMSFdj+fxvuiDy5twhoahzK40O1EgD+G4AMa eDYu3HaLAuYX4xJNY5lWUlKtqN0BIZSAavznS50h4HxkrmS+e94O3sf8/7qmxj7MnGMi 1LKax6jtYPEqz0CqYiziq8L4SC50OToNp7/DWXzflTxqkIkEIoY5tAkApqL+uBZbk/r4 P/MxAqs6m7so13tj1PIi3jSUrNUeEViY7AvIY9A8Y7Wl44sRn5eM/DeAQXOdzRonetIG kD/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=OYvYNmf/x4P7vSFlY+ls2p8t4uEh4HhTGMr1xOrMvUs=; b=AZGWZ0aNt4UW6B/yDxjaquGMvZY6mOVbsor3gbWyJTrAx3xCTtgEeyTuQuqX77jtvm jvZPD+6odJYmmqSeam6FjbXiJHUTTFpDVB+DUXFmidIlEXhhv1NNEMK7M2FfRKAgpo4j 0Fb+aJQhPqc0VUNcu92sxupzflbaR3Yr+00Li8thmS3X0JKIorPp9tyGkVqcN2Xk0PG0 YSXhqdFpl5b4mseBMd3Dr9xXfeZV7CZ+8Yz+cz1badlpbRYvvDwFAlJQmptAi667i+T3 5IZ0uN8hRnpqrPTR+pL7r+3nxqjDiCei7pd/vLE4dmrmmOZqgpZ8y1NjTsXYWRrExG41 Hb1A== X-Gm-Message-State: ACgBeo1FgpzBSBXvwyyGXFG+qqMAfeO5JMutxlLvYRUlV3bCmyaJbig7 gSmjhtZPRMdCnrZCu9J7om+CNltSPrE= X-Google-Smtp-Source: AA6agR4QrI/goo0Ifa3I6QTrkZ/rp7BifF54kL3nFEtBOnBao2RJNC60ulxX+CvYR3X4h0m2jlu5CA== X-Received: by 2002:a7b:c453:0:b0:3a5:b42e:c4fb with SMTP id l19-20020a7bc453000000b003a5b42ec4fbmr1633990wmi.167.1661944454295; Wed, 31 Aug 2022 04:14:14 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003a60edc3a44sm2324753wmb.5.2022.08.31.04.14.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 04:14:13 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 4/7] bitbake-layers: add ability to save current layer repository configuration into a file Date: Wed, 31 Aug 2022 13:13:58 +0200 Message-Id: <20220831111401.3330342-4-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220831111401.3330342-1-alex@linutronix.de> References: <20220831111401.3330342-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 31 Aug 2022 11:14:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/170104 This addresses a long standing gap in the core offering: there is no tooling to capture the currently configured layers with their revisions, or restore the layers from a configuration file (without using external tools, some of which aren't particularly suitable for the task). This plugin addresses the 'capture' part. Note that the actual writing is performed by a sub-plugin; one such sub-plugin is provided (for the json + python script format), but more can be added (e.g. kas, repo, etc.). How to save a layer configuration: a) Running with default choices: $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/ NOTE: Starting bitbake server... NOTE: Created /srv/work/alex/meta-alex/setup-layers.json NOTE: Created /srv/work/alex/meta-alex/setup-layers b) Command line options: NOTE: Starting bitbake server... usage: bitbake-layers create-layers-setup [-h] [--output-prefix OUTPUT_PREFIX] [--writer {oe-setup-layers}] [--json-only] destdir Writes out a configuration file and/or a script that replicate the directory structure and revisions of the layers in a current build. positional arguments: destdir Directory where to write the output (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching). optional arguments: -h, --help show this help message and exit --output-prefix OUTPUT_PREFIX, -o OUTPUT_PREFIX File name prefix for the output files, if the default (setup-layers) is undesirable. --writer {oe-setup-layers}, -w {oe-setup-layers} Choose the output format (defaults to oe-setup-layers). Currently supported options are: oe-setup-layers - a self-contained python script and a json config for it. --json-only When using the oe-setup-layers writer, write only the layer configuruation in json format. Otherwise, also a copy of scripts/oe-setup-layers (from oe-core or poky) is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 108 ++++++++++++++++++ .../bblayers/setupwriters/oe-setup-layers.py | 50 ++++++++ 2 files changed, 158 insertions(+) create mode 100644 meta/lib/bblayers/makesetup.py create mode 100644 meta/lib/bblayers/setupwriters/oe-setup-layers.py diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py new file mode 100644 index 0000000000..bef6da0ea8 --- /dev/null +++ b/meta/lib/bblayers/makesetup.py @@ -0,0 +1,108 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return MakeSetupPlugin() + +class MakeSetupPlugin(LayerPlugin): + + def _get_repo_path(self, layer_path): + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) + return repo_path.strip() + + def _get_remotes(self, repo_path): + remotes = {} + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) + for r in remotes_list.split(): + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) + remotes[r] = {'uri':uri.strip()} + return remotes + + def _get_describe(self, repo_path): + try: + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) + except bb.process.ExecutionError: + return "" + return describe.strip() + + def make_repo_config(self, destdir): + """ This is a helper function for the writer plugins that discovers currently confugured layers. + The writers do not have to use it, but it can save a bit of work and avoid duplicated code, hence it is + available here. """ + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + try: + destdir_repo = self._get_repo_path(destdir) + except bb.process.ExecutionError: + destdir_repo = None + + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + if l_name == 'workspace': + continue + if l_ismodified: + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path)) + return + repo_path = self._get_repo_path(l_path) + if repo_path not in repos.keys(): + repos[repo_path] = {'path':os.path.basename(repo_path),'layers':{},'git-remote':{'rev':l_rev, 'branch':l_branch, 'remotes':self._get_remotes(repo_path), 'describe':self._get_describe(repo_path)}} + if repo_path == destdir_repo: + repos[repo_path]['contains_this_file'] = True + if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']: + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) + return + repos[repo_path]['layers'][l_name] = {'subpath':l_path.replace(repo_path,'')[1:]} + + top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()]) + + repos_nopaths = {} + for r in repos.keys(): + r_nopath = os.path.basename(r) + repos_nopaths[r_nopath] = repos[r] + r_relpath = os.path.relpath(r, top_path) + repos_nopaths[r_nopath]['path'] = r_relpath + return repos_nopaths + + def do_make_setup(self, args): + """ Writes out a configuration file and/or a script that replicate the directory structure and revisions of the layers in a current build. """ + for p in self.plugins: + if str(p) == args.writer: + p.do_write(self, args) + + def register_commands(self, sp): + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) + parser_setup_layers.add_argument('destdir', + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching).') + parser_setup_layers.add_argument('--output-prefix', '-o', + help='File name prefix for the output files, if the default (setup-layers) is undesirable.') + + self.plugins = [] + + for path in (self.tinfoil.config_data.getVar('BBPATH').split(':')): + pluginpath = os.path.join(path, 'lib', 'bblayers', 'setupwriters') + bb.utils.load_plugins(logger, self.plugins, pluginpath) + + parser_setup_layers.add_argument('--writer', '-w', choices=[str(p) for p in self.plugins], help="Choose the output format (defaults to oe-setup-layers).\n\nCurrently supported options are:\noe-setup-layers - a self-contained python script and a json config for it.\n\n", default="oe-setup-layers") + + for plugin in self.plugins: + if hasattr(plugin, 'register_arguments'): + plugin.register_arguments(parser_setup_layers) diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py new file mode 100644 index 0000000000..f6a484b766 --- /dev/null +++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py @@ -0,0 +1,50 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import json +import stat + +logger = logging.getLogger('bitbake-layers') + +def plugin_init(plugins): + return OeSetupLayersWriter() + +class OeSetupLayersWriter(): + + def __str__(self): + return "oe-setup-layers" + + def _write_python(self, input, output): + with open(input) as f: + script = f.read() + with open(output, 'w') as f: + f.write(script) + st = os.stat(output) + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + def _write_json(self, repos, output): + with open(output, 'w') as f: + json.dump(repos, f, sort_keys=True, indent=4) + + def do_write(self, parent, args): + """ Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. """ + repos = parent.make_repo_config(args.destdir) + json = {"version":"1.0","sources":repos} + if not repos: + raise Exception("Could not determine layer sources") + output = args.output_prefix or "setup-layers" + output = os.path.join(os.path.abspath(args.destdir),output) + self._write_json(json, output + ".json") + logger.info('Created {}.json'.format(output)) + if not args.json_only: + self._write_python(os.path.join(os.path.dirname(__file__),'../../../../scripts/oe-setup-layers'), output) + logger.info('Created {}'.format(output)) + + def register_arguments(self, parser): + parser.add_argument('--json-only', action='store_true', + help='When using the oe-setup-layers writer, write only the layer configuruation in json format. Otherwise, also a copy of scripts/oe-setup-layers (from oe-core or poky) is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json.') From patchwork Wed Aug 31 11:13:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 12147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9B08C54EE9 for ; Wed, 31 Aug 2022 11:14:20 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web08.23995.1661944456950656563 for ; Wed, 31 Aug 2022 04:14:17 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=C7apodfZ; spf=pass (domain: gmail.com, ip: 209.85.128.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-wm1-f42.google.com with SMTP id i188-20020a1c3bc5000000b003a7b6ae4eb2so6526870wma.4 for ; Wed, 31 Aug 2022 04:14:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=5InC7AJyE5N/2stlUrTiDCUrOP/UQAc5XX5BDLbVdoE=; b=C7apodfZ/I3+/h6+nqCwtWlK3DZHNGgufno97muw9oDASHAHmTu87RPDsPBt9MqJ0x QfcHLN47R5Tj4kS4PbGUAcr3Tbkr5x+x+w0BrMsxYrXMsGhpgh605e3w4p+VJrfR1HIU QJdmvaMHS5gCf63jQx3cXBFaERk4FXqWga33DbVICaQ1zSvmHo9lcR6oypX1xp53f/LL DCo8kDNNjGJG1GZEn6VCHHtyeWGlJw0aG8+yqsBo90XZDC6HDXGfv2XwrJ4yzXLnNb0w fsF/SaWU20+E02uDYG/745JFykS+7Q7n99dLbgCZFriYYLcLHa8/iGlgx35oU62VCmIe q82g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=5InC7AJyE5N/2stlUrTiDCUrOP/UQAc5XX5BDLbVdoE=; b=0ugQ+BJ9ZHTYXd/qwWmYNvuFWRuZ9frapNrF20ilOPLDl6VgZOC3ERSr8zeK87VSVO 2U064+ES9Z8JY6Y6eFr+dmhtSpCAGuE6yOnbMN8hNo5HQcvE+b8VoSU9i9KF8HC6o5CR JzivDShZZxezS67CbjIOEHgbi0C6xs1KyTkUIlIsrBflV61BX2k9F32ot3L0ugYhKaFR 8Csz3rghT+DC+a7x4X8A60VxbEeFBWJ7w55FmD4UD0p/+jw2VGS+Bcihfuz1gVunoUY5 xVBtWKIwzo0h4Cog/p49q7NmDwMM32FWA7vE9V8WyCoZ6ZyWUL/KzHjl9hORvaA//ITP 6Ocw== X-Gm-Message-State: ACgBeo2Kn6lsIeZPW+fIbeOC947w4QxJi85V+TCLMi2H0YVKZKqpKBwk y7Zek48/qL/uowER3YQagc/9Tf7glyk= X-Google-Smtp-Source: AA6agR4KQo0W2ovbUpYFnuAg0UpFCoW776xsYIld1yu7KOmKr4hgqLm5wsHhC2gkGOYweTnUHHeycw== X-Received: by 2002:a7b:ce89:0:b0:3a5:cefe:80f6 with SMTP id q9-20020a7bce89000000b003a5cefe80f6mr1761328wmj.113.1661944455379; Wed, 31 Aug 2022 04:14:15 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003a60edc3a44sm2324753wmb.5.2022.08.31.04.14.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 04:14:14 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 5/7] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file Date: Wed, 31 Aug 2022 13:13:59 +0200 Message-Id: <20220831111401.3330342-5-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220831111401.3330342-1-alex@linutronix.de> References: <20220831111401.3330342-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 31 Aug 2022 11:14:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/170105 This script can be used directly from poky or oe-core, or can be copied directly into alayer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --help show this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin --- scripts/oe-setup-layers | 88 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 0000000000..cbd2efb5c7 --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): + layers = json['sources'] + buildconfs = [] + oecorepath = "" + for l_name in layers: + l_data = layers[l_name] + layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + + for ll_name in l_data['layers']: + if ll_name == 'meta': + oecorepath = layerdir + ll_data = l_data['layers'][ll_name] + if 'buildconfigs' in ll_data: + for c in ll_data['buildconfigs']: + buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + + if 'contains_this_file' in l_data.keys(): + force_arg = 'force_bootstraplayer_checkout' + if not args[force_arg]: + print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) + continue + l_remote = l_data['git-remote'] + rev = l_remote['rev'] + desc = l_remote['describe'] + if not desc: + desc = rev[:10] + branch = l_remote['branch'] + remotes = l_remote['remotes'] + + print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) + cmd = 'git init -q {}'.format(layerdir) + print("Running '{}'".format(cmd)) + subprocess.check_output(cmd, shell=True) + + for remote in remotes: + cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri']) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + cmd = "git fetch -q {} || true".format(remote) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + cmd = 'git checkout -q {}'.format(rev) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + +parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") + +parser.add_argument('--force-bootstraplayer-checkout', action='store_true', + help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).') + +try: + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', universal_newlines=True, shell=True, cwd=os.path.dirname(__file__))) +except subprocess.CalledProcessError as e: + defaultdest = os.path.abspath(".") + +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) +parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) + +args = parser.parse_args() + +with open(args.jsondata) as f: + json = json.load(f) + +supported_versions = ["1.0"] +if json["version"] not in supported_versions: + raise Exception("File {} has version {}, which is not in supported versions: {}".format(args.jsondata, json["version"], supported_versions)) + +_do_checkout(vars(args), json) From patchwork Wed Aug 31 11:14:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 12144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BADE0ECAAD5 for ; Wed, 31 Aug 2022 11:14:20 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web08.23996.1661944458598369324 for ; Wed, 31 Aug 2022 04:14:19 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=KB6kcznC; spf=pass (domain: gmail.com, ip: 209.85.221.54, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f54.google.com with SMTP id c7so11134556wrp.11 for ; Wed, 31 Aug 2022 04:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=3srNYb6ND/psxvsL6kW7dKFqi29bJH6UWdvY60fmw0A=; b=KB6kcznCW2OuBv0CPE2UquQL8YMJGrw5Pgh055pCj5txsCZZ4nrnzEQQMsiW5/3I0I cV3O9ynxBxvMD83w7+6nDRG5szFeK2m0F/v4JbvND3iiOvCn+eL3mmIWyRX5xo0fz8bX dQPFygADCsluuDdvN/v05w8GeXs2IWqsRL925osTB4Z/V7DTTJ1JBW2uBUj2/mw67Qfg QxXGW/XZK8+msER1vzbe03V+bveMxUMn+8h1gf9dywetqXHYtsHt+3YzSxd0gWhKWXqA usrAxw0k5xCyyaD+njckP3JJxGQyDZnK5zb8zshB2GjSo6dAWVvosaK8j87M9h9EaZ/m edPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=3srNYb6ND/psxvsL6kW7dKFqi29bJH6UWdvY60fmw0A=; b=kRygNJP8++NICz6zW2fEIEkZVNczqaF+x4M9FXhSsybpFoCOj2++t5HjqOImfAc+mj JDD8GC8OISTKuDkHVOneatPqB9Z0DPIEZP3OwjWqChCL+uTob4WcAby9/NHIn1RRsbLb L3YQbO4G1hKtz7oXZ8nIpnw3ONm2QVW2dwFHJxbERp1NHDT+LAv5zMP+f6coMxSs6zqY W43d7I20o5L5XtbTn5KhslvQesa3HN7BGduNaj1zfLCUExYlGcI4Awq67SjPjJAVvGyl ZcPqbK5GZHMoQY3ZidhDMpgooMdgCLrc4FZQihGCJK83fAsEzdUsIUekahHaPF50IRgO U13Q== X-Gm-Message-State: ACgBeo2xI9g6FV1YJiQYdlk57K9aw9NKbf/T4CHmiS3VFZuDIj93xjGE 5qjP2jRN0JJkL5FQvIiaHMjQWKHYB18= X-Google-Smtp-Source: AA6agR7oNjbZTW9QNppy55GYfV392t8NqsjKgo1v0OTeXUF5yEkwXIfRIzkfbnWmPgRJbLdVzXDbbw== X-Received: by 2002:adf:9cca:0:b0:226:dfa0:3fb7 with SMTP id h10-20020adf9cca000000b00226dfa03fb7mr6188600wre.412.1661944456999; Wed, 31 Aug 2022 04:14:16 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003a60edc3a44sm2324753wmb.5.2022.08.31.04.14.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 04:14:16 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 6/7] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers Date: Wed, 31 Aug 2022 13:14:00 +0200 Message-Id: <20220831111401.3330342-6-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220831111401.3330342-1-alex@linutronix.de> References: <20220831111401.3330342-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 31 Aug 2022 11:14:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/170106 This does a basic run-through of the bitbake-layers plugin, and the resulting json layer config and the layer setup script that uses it. Only poky is actually fetched by the script. Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/bblayers.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index c753a7b795..18007764b3 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -142,3 +142,25 @@ class BitbakeLayers(OESelftestTestCase): def test_validate_examplelayersjson(self): json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json") self.validate_layersjson(json) + + def test_bitbakelayers_setup(self): + result = runCmd('bitbake-layers create-layers-setup {}'.format(self.testlayer_path)) + jsonfile = os.path.join(self.testlayer_path, "setup-layers.json") + self.validate_layersjson(jsonfile) + + # The revision-under-test may not necessarily be available on the remote server, + # so replace it with a stable release tag. + import json + with open(jsonfile) as f: + data = json.load(f) + for s in data['sources']: + data['sources'][s]['git-remote']['rev'] = 'yocto-4.0' + with open(jsonfile, 'w') as f: + json.dump(data, f) + + testcheckoutdir = os.path.join(self.builddir, 'test-layer-checkout') + result = runCmd('{}/setup-layers --destdir {}'.format(self.testlayer_path, testcheckoutdir)) + # May not necessarily be named 'poky' or 'openembedded-core' + oecoredir = os.listdir(testcheckoutdir)[0] + testcheckoutfile = os.path.join(testcheckoutdir, oecoredir, "oe-init-build-env") + self.assertTrue(os.path.exists(testcheckoutfile), "File {} not found in test layer checkout".format(testcheckoutfile)) From patchwork Wed Aug 31 11:14:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 12143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBCBDC3DA6B for ; Wed, 31 Aug 2022 11:14:20 +0000 (UTC) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mx.groups.io with SMTP id smtpd.web10.24370.1661944459645155651 for ; Wed, 31 Aug 2022 04:14:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=GDIkEmcx; spf=pass (domain: gmail.com, ip: 209.85.221.44, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f44.google.com with SMTP id u18so5126561wrq.10 for ; Wed, 31 Aug 2022 04:14:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=rXJSQ7ZwCs3657OxftYZuNacMZCLx7wshnYBTDQdLZA=; b=GDIkEmcxyiEb867p9L4UscvfLfJ752JyOKKH8+U+cREOV4jZhKBCUzUo/OTB62wtkc YvnzrP1YSbMnD6W3+RkC5ophWzw+p7MB3r1DjcruvVRizDbaeg2ZyUb+PJe8XLXYpKuu zeMT+kFLt0RSgmNtU0fjqJfJWhdQWSugxMBoDn0wqLFY9bqNEcvxSZ2NgdvjImIIUXSu KdmiZuaDtTVZv5v2dWgAdaLCjPmBe13vG0D6Y8+JDnDeohlinZld6Jn4hIk8S051Ce2m e40pGAgGKws6dG2/UrNlfeXBpzkCfmLDzjDSfCw5J9bfp8rbrsuZd5OymB6PkNHSH+XA ++zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=rXJSQ7ZwCs3657OxftYZuNacMZCLx7wshnYBTDQdLZA=; b=Xe1r3O1z/vlvHSxrDUc+0mqxSx3LiTITlY7qqzEoXm78Qg3nrAUBcR5xppVNVEE8sS zWhb4XmJADn1PJdLMO2Rkv+uplXSakt71STgCCC85DwL/kEzvU9iDwmKO2QMsklhICTF u0EzyRLZBMWuKvNh4fBzCMrIch6JHQ25oV2LPpRM7aIv9sbtyNuvwj2kV4SovZ8pseJq Vl/epW617CY5IUkp1TbPTmndPIqZM9AMoYa1uwkgI58Q3NSKzaIhbsfxZtA9GuBObpiR HGizoMAYSrBGz/rVNXNqsX+oIIWNj7A9dfUELhH6Q9WkjGursLY0hSpiSCQi9BZ2qOGP V/Ug== X-Gm-Message-State: ACgBeo1agl6/8Pv0MMNeCOIPIP3Xd/GRmGiEqU+PB676e5ziM4gc5RAu 3pDgWoliRo0YLmBT8MKd0s/YLpe8tCg= X-Google-Smtp-Source: AA6agR6f9jJAA3/VPvrEkO9035L8TgNLz4MQnKmwx/Dkaur+Zv6TF5QtePPHQfuddDibOtpXRyf/HQ== X-Received: by 2002:a05:6000:817:b0:226:3d89:ebb4 with SMTP id bt23-20020a056000081700b002263d89ebb4mr11932770wrb.699.1661944458179; Wed, 31 Aug 2022 04:14:18 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003a60edc3a44sm2324753wmb.5.2022.08.31.04.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 04:14:17 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 7/7] selftest/bblayers: adjust the revision for the layer setup test Date: Wed, 31 Aug 2022 13:14:01 +0200 Message-Id: <20220831111401.3330342-7-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220831111401.3330342-1-alex@linutronix.de> References: <20220831111401.3330342-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 31 Aug 2022 11:14:20 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/170107 Using a tag is not actually robust enough, e.g. poky-contrib checkouts do not come with any tags. So let's use a revision matching yocto-4.0, that ought to be present. Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/bblayers.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 18007764b3..c6bd5a1f6a 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -149,12 +149,12 @@ class BitbakeLayers(OESelftestTestCase): self.validate_layersjson(jsonfile) # The revision-under-test may not necessarily be available on the remote server, - # so replace it with a stable release tag. + # so replace it with a revision that has a yocto-4.0 tag. import json with open(jsonfile) as f: data = json.load(f) for s in data['sources']: - data['sources'][s]['git-remote']['rev'] = 'yocto-4.0' + data['sources'][s]['git-remote']['rev'] = '00cfdde791a0176c134f31e5a09eff725e75b905' with open(jsonfile, 'w') as f: json.dump(data, f)