From patchwork Thu Jan 4 12:33:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: jhatnedap@gmail.com X-Patchwork-Id: 37344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19E02C47073 for ; Thu, 4 Jan 2024 12:35:52 +0000 (UTC) Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) by mx.groups.io with SMTP id smtpd.web10.52896.1704371747416334148 for ; Thu, 04 Jan 2024 04:35:47 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Piz1FazH; spf=pass (domain: gmail.com, ip: 209.85.167.179, mailfrom: jhatnedap@gmail.com) Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3b9f727d94cso255111b6e.1 for ; Thu, 04 Jan 2024 04:35:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704371746; x=1704976546; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ScJG4WObXJZd+TUXnBOfXow32P8urGSxsyd6VTydXA4=; b=Piz1FazHHLlzmE49S9xOUc9EV1Q7yOvD//nQHfplVhHb5RfcXKOm554jDLQJxBredN 3sDRkpl7DhHtjHgRHuD8M3zDYvGibJjEV9XouFdC1SY71uWbygvhSJN/v8JRjPPVun3g y2BAPlBJjnMDMaWoGMDPxgzBu2o6OVgxWAEonnq+dAV3BgynrQ6W0kLNT+IrvK0KQ2hv B7NT2NiPDMIvlE2qugQud+Bnb/yD8dQc47Yv6bTS9I22PRN+juIrW/R6zvxvG4hhXkEt UTGP0sMbav7esvAcj8YjbpSDWFA46+NnqhYd/Ph4GWP3XwwFX5IZpwUDKtrT59Qc+L38 YE6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704371747; x=1704976547; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ScJG4WObXJZd+TUXnBOfXow32P8urGSxsyd6VTydXA4=; b=arlnuz3eKo+jXCQK4EZhGuQW9juj3fnA1WmvS85aQlapG8aXvOdPH6R6SAs3OWnKAr VJjrJ/LcTP+eA+bts/iVRIFvq8FfDBpITnLzVDFca9ezwIbJd122mZH7SdpSuQsPouG6 Tm7envjfUjk6QDxfbq3kQkR4tIM7fLWzxB9OiZvPZixbN4DHdUUNZmfxlkk+HZX/r6Jc U11LWs/yM/6UlYfpl/oZV0OKOM92XGjskEDC/REaCLm1P7D2zNeJ58C2z2DQe0BITUbs 5InXZn0Up361T4jopZcjDdQd+yRr2YMspoLq+28FgocMVtaVBGg5/9DClf6SpyslBgxi hjrg== X-Gm-Message-State: AOJu0YwnUxS+dK9SILm1I4rdqtoAe2f8R9Pfhflb9Ywrpnkak9CQlE0z dN/SHLQvu2RmhZmDLcGR3H/LthUV4DE= X-Google-Smtp-Source: AGHT+IGqYyFlytni8SnEIBMehCmE3pDKp5iKke0bCKuR8VaDLyMF8GXp948F8xF0FOjIluEXG5EP0w== X-Received: by 2002:a05:6359:28cb:b0:175:5b44:755f with SMTP id qb11-20020a05635928cb00b001755b44755fmr258935rwb.42.1704371746546; Thu, 04 Jan 2024 04:35:46 -0800 (PST) Received: from NVC4181.nedap.local ([149.3.168.10]) by smtp.gmail.com with ESMTPSA id i11-20020a05622a08cb00b004281ce041f6sm4516537qte.21.2024.01.04.04.35.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jan 2024 04:35:46 -0800 (PST) From: jhatnedap@gmail.com X-Google-Original-From: jermain.horsman@nedap.com To: openembedded-core@lists.openembedded.org Cc: Jermain Horsman Subject: [RFC v2 1/2] bitbake-layers: Add ability to update the reference of repositories Date: Thu, 4 Jan 2024 13:33:17 +0100 Message-ID: <20240104123420.932-3-jermain.horsman@nedap.com> X-Mailer: git-send-email 2.42.0.windows.2 In-Reply-To: <20240104123420.932-2-jermain.horsman@nedap.com> References: <20240104123420.932-2-jermain.horsman@nedap.com> Reply-To: jermain.horsman@nedap.com MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 04 Jan 2024 12:35:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193317 From: Jermain Horsman This uses an existing setup-layers configuration and modifies one or more repositories using a reference provided by the user. This is a very minimal implementation, no validation of this reference is done and it is left to the user to provide a valid value. Signed-off-by: Jermain Horsman --- meta/lib/bblayers/makesetup.py | 11 +++++ .../bblayers/setupwriters/oe-setup-layers.py | 43 ++++++++++++++++--- 2 files changed, 47 insertions(+), 7 deletions(-) diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py index 99d5973760..df266c645d 100644 --- a/meta/lib/bblayers/makesetup.py +++ b/meta/lib/bblayers/makesetup.py @@ -78,6 +78,10 @@ class MakeSetupPlugin(LayerPlugin): def do_make_setup(self, args): """ Writes out a configuration file and/or a script that replicate the directory structure and revisions of the layers in a current build. """ + if args.update is not None and args.reference is None: + logger.error("No reference specified. Please provide one using '--reference REFERENCE'.") + return + for p in self.plugins: if str(p) == args.writer: p.do_write(self, args) @@ -89,6 +93,13 @@ class MakeSetupPlugin(LayerPlugin): parser_setup_layers.add_argument('--output-prefix', '-o', help='File name prefix for the output files, if the default (setup-layers) is undesirable.') + parser_setup_layers.add_argument('--update', '-u', + action='append', + metavar='REPOSITORY', + help='Repository to update, this requires a reference to be specified.\nThis option can be used multiple times.') + parser_setup_layers.add_argument('--reference', '-r', + help="Reference to use when updating repositories.\nThe value can be any git reference, however it is up to the user to provide a valid value.\nThis option is only useful when using '--update'.") + self.plugins = [] for path in (self.tinfoil.config_data.getVar('BBPATH').split(':')): diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py index bd71ca1f51..fdc33d94d5 100644 --- a/meta/lib/bblayers/setupwriters/oe-setup-layers.py +++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py @@ -31,16 +31,45 @@ class OeSetupLayersWriter(): with open(output, 'w') as f: json.dump(repos, f, sort_keys=True, indent=4) + def _read_repo_config(self, json_path): + with open(json_path) as f: + json_config = json.load(f) + + supported_versions = ["1.0"] + if json_config["version"] not in supported_versions: + raise Exception("File {} has version {}, which is not in supported versions: {}".format(json_path, json_config["version"], supported_versions)) + + return json_config + + def _modify_repo_config(self, json_config, args): + sources = json_config['sources'] + for repo in args.update: + if not repo in sources.keys(): + raise Exception("Repository {} does not exist in setup-layers config".format(repo)) + + layer_remote = json_config['sources'][repo]['git-remote'] + layer_remote['rev'] = args.reference + # Clear describe + layer_remote['describe'] = '' + def do_write(self, parent, args): """ Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. """ - if not os.path.exists(args.destdir): - os.makedirs(args.destdir) - repos = parent.make_repo_config(args.destdir) - json = {"version":"1.0","sources":repos} - if not repos: - raise Exception("Could not determine layer sources") output = args.output_prefix or "setup-layers" - output = os.path.join(os.path.abspath(args.destdir),output) + output = os.path.join(os.path.abspath(args.destdir), output) + + if args.update is None: + if not os.path.exists(args.destdir): + os.makedirs(args.destdir) + repos = parent.make_repo_config(args.destdir) + json = {"version":"1.0","sources":repos} + if not repos: + raise Exception("Could not determine layer sources") + else: + json = self._read_repo_config(output + ".json") + if not 'sources' in json.keys(): + raise Exception("File {}.json does not contain valid layer sources.".format(output)) + self._modify_repo_config(json, args) + self._write_json(json, output + ".json") logger.info('Created {}.json'.format(output)) if not args.json_only: From patchwork Thu Jan 4 12:33:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: jhatnedap@gmail.com X-Patchwork-Id: 37345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3CD7C47073 for ; Thu, 4 Jan 2024 12:36:01 +0000 (UTC) Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by mx.groups.io with SMTP id smtpd.web10.52898.1704371752653587197 for ; Thu, 04 Jan 2024 04:35:52 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=LTEpkniL; spf=pass (domain: gmail.com, ip: 209.85.160.171, mailfrom: jhatnedap@gmail.com) Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-427e83601c4so2367461cf.0 for ; Thu, 04 Jan 2024 04:35:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704371752; x=1704976552; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=em3uFh53S6QsE1Z8Y+6Zlt7tibIr0cuA5cf0iljNz9k=; b=LTEpkniLPyqJEW9Eb79fiKubrXjlxydG6w2kG3zhAV1E1wnGNj9yH+WX2MSHVYZqWJ yUJnYTo8zTtF9aHGoZ+1nVkRy0alvlTIivBXmJdnkGp6e2atlFSCnRaNck/kQPrwO4Pj sE4cwCGXKcutN3diSylJhgQyhjAgwrTAdoHY2sqOrVHls3unx5mG78ORJVZHvvLSVZ6d Twyh+wlQhdXfvfBE3ivSqti82XH34n5OGFTaEzx9OcZ9TeVMoPGpnfpyzXirlzcxEjWH SjBQdtNvj0EMOjD0YJDoJAhFS6nyX1sBQqf/9Cz2wDHxFE0eLSwZJv7GaeQ9CyV2srdA otVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704371752; x=1704976552; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=em3uFh53S6QsE1Z8Y+6Zlt7tibIr0cuA5cf0iljNz9k=; b=cb0AhVunblBIs0sklnZu62BigegNcsM/rsz0+ZVu1wuDRIlZKxGou8wipU+X9fzeMV Mn4aG2dutGbwhImzVqXaz7SvMhaNX8fDl9Zb6N+B3TpSRbk0Z/x6c9MdDBPOfH63CHVb RYBU4TzNmF7CVsW0/sCeqhb8Khu7yPUSChBi4NQIoVSn8EGwy+QfrdUrKSAEtFMYwosS LzCA1a0Mzyq3jVWCTJ+Sp5U2ZGXgqyfFuOlr9cllEVlTcNTDEV502APPi4hlyLZQRp7w xW0nVNdR37AHmBcIUeRE9eKs53fxPyyduIy5z96T4XrzpNFjlJfffp1IZo6KsCYtNp7M p6Hg== X-Gm-Message-State: AOJu0Yy4Md4oiWj481HgGvWUdKwSBFHMYzO5BMcoaCMWm6qTs1MK6EOa kwdIlEriQDia0pXIeNvyA6fP5cDyxFY= X-Google-Smtp-Source: AGHT+IG8RGN67ycw/Za8N8fbjjcPtp/ufm4Zfv9Ii02VPsa28o8TaYxYqnX2HopRvoBzHODSH9se6g== X-Received: by 2002:ac8:4e82:0:b0:428:272c:a655 with SMTP id 2-20020ac84e82000000b00428272ca655mr496625qtp.2.1704371751764; Thu, 04 Jan 2024 04:35:51 -0800 (PST) Received: from NVC4181.nedap.local ([149.3.168.10]) by smtp.gmail.com with ESMTPSA id i11-20020a05622a08cb00b004281ce041f6sm4516537qte.21.2024.01.04.04.35.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jan 2024 04:35:51 -0800 (PST) From: jhatnedap@gmail.com X-Google-Original-From: jermain.horsman@nedap.com To: openembedded-core@lists.openembedded.org Cc: Jermain Horsman Subject: [RFC v2 2/2] bitbake-layers: Add test case layers setup update Date: Thu, 4 Jan 2024 13:33:18 +0100 Message-ID: <20240104123420.932-4-jermain.horsman@nedap.com> X-Mailer: git-send-email 2.42.0.windows.2 In-Reply-To: <20240104123420.932-2-jermain.horsman@nedap.com> References: <20240104123420.932-2-jermain.horsman@nedap.com> Reply-To: jermain.horsman@nedap.com MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 04 Jan 2024 12:36:01 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193318 From: Jermain Horsman This includes a simple test which creates a layer setup, modifies it using a test string and checks if the resulting layer setup is written correctly. Signed-off-by: Jermain Horsman --- meta/lib/oeqa/selftest/cases/bblayers.py | 26 ++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 8faa060234..0467fbb736 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -167,3 +167,29 @@ class BitbakeLayers(OESelftestTestCase): oecoredir = os.listdir(testcheckoutdir)[0] testcheckoutfile = os.path.join(testcheckoutdir, oecoredir, "oe-init-build-env") self.assertTrue(os.path.exists(testcheckoutfile), "File {} not found in test layer checkout".format(testcheckoutfile)) + + def test_bitbakelayers_updatelayer(self): + result = runCmd('bitbake-layers create-layers-setup {}'.format(self.testlayer_path)) + jsonfile = os.path.join(self.testlayer_path, "setup-layers.json") + self.validate_layersjson(jsonfile) + + import json + with open(jsonfile) as f: + data = json.load(f) + test_repo = None + for s in data['sources']: + if 'contains_this_file' not in data['sources'][s].keys(): + test_repo = s + result = runCmd('bitbake-layers create-layers-setup --update {} --reference {} {}'.format(test_repo, 'remote/test', self.testlayer_path)) + break + + self.assertTrue(test_repo, "No test repo found") + self.validate_layersjson(jsonfile) + + with open(jsonfile) as f: + data = json.load(f) + curr_rev = data['sources'][test_repo]['git-remote']['rev'] + curr_desc = data['sources'][test_repo]['git-remote']['describe'] + + self.assertEqual(curr_rev, 'remote/test', "Revision not set correctly: '{}'".format(curr_rev)) + self.assertEqual(curr_desc, '', "Describe not cleared: '{}'".format(curr_desc))