From patchwork Tue Jan 16 10:55:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: jhatnedap@gmail.com X-Patchwork-Id: 37914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33611C47077 for ; Tue, 16 Jan 2024 10:57:04 +0000 (UTC) Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by mx.groups.io with SMTP id smtpd.web10.10683.1705402619051926523 for ; Tue, 16 Jan 2024 02:56:59 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=W+GL3Wbd; spf=pass (domain: gmail.com, ip: 209.85.208.52, mailfrom: jhatnedap@gmail.com) Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-559bd483877so169265a12.0 for ; Tue, 16 Jan 2024 02:56:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705402617; x=1706007417; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Nwjb8TPJh9KTuqXluYnJRWLYD1vGo95ouZ0iggvSZSA=; b=W+GL3WbdP5Sy4idEQhHw+XfvcXElqM1Z/XiqQ/MTjCXaM3hlBPVlkDCxfJEHlqYTqU DLYeV+v8d0zqSzxxI47/alIUnwd/e9BpSAvfrW+osEQEz8vv/02seuAfAICzJf0p+kpu r5VIQO/yUsiv1+iFyl7ruoKqOzC+PHkwYgnjK7xuqqSBldLg6abVlJSm7hOUuR18eJzO QytbZ87S4mhW6On99WfGhpIbCRdDtyoJ5SCHicshiUCsJIJrV3uWCMitgk6qErhUVzoj qZ2Ij8yyrwNuLfKceVV/a+8146dD1ezMXHG3r2fdVUlDWLKMaqAs2bEhHORjaiuNb8g6 unYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705402617; x=1706007417; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Nwjb8TPJh9KTuqXluYnJRWLYD1vGo95ouZ0iggvSZSA=; b=qPgQ2+aJOlaSiHybPC0+gnjpXl6XEYMQRwPLxR8g+kCdAo5tdchxvZ+3pqu2ZosMrH iWMtv0Z+7L+3vMsPEFoNDHj24zuuRtxgcaqLTJ1w2SKC3riswKql/N/R96b95AF32V9+ DN5dh4lTCBgMXIBnbQiCtfVO0pysPQFfc0BrPaemB+NFQOwXuSSHvTqNFaOXLSjqvtVq 8Ya2rAI8h3ASkj+wamdwf/n1zJQuCQ4MtRH+SUT9+McDDb2kLPM7VpHUTVUy8Vsh+rkN edk2o7PpS9uFRQowvyxmPUKkFiuwPuQUbnm3H5fK92pw8/c3YmUggBEFzBvl3dJQnv71 uepw== X-Gm-Message-State: AOJu0Yx1xooH+hAefNM96Msb5nJWuGGFPyK8zEfeGtQ3qV2C1yhVHAmw Bi0GRSiXIR1hQJMZFy9bc98HXC2uxV8= X-Google-Smtp-Source: AGHT+IFK2PuDJl8nFZ+7zvwwEZHQ8NIBGQ2nuI/9yNsVUOt+FRqvARhDtgjjqyKouQplUV+GeDW33Q== X-Received: by 2002:a17:907:7f11:b0:a2b:29bc:f1de with SMTP id qf17-20020a1709077f1100b00a2b29bcf1demr3792149ejc.9.1705402617165; Tue, 16 Jan 2024 02:56:57 -0800 (PST) Received: from NVC4181.nedap.local ([149.3.168.10]) by smtp.gmail.com with ESMTPSA id tg14-20020a1709078dce00b00a2cb117050fsm5366969ejc.126.2024.01.16.02.56.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 02:56:56 -0800 (PST) From: jhatnedap@gmail.com X-Google-Original-From: jermain.horsman@nedap.com To: openembedded-core@lists.openembedded.org Cc: Jermain Horsman Subject: [RFC v4 1/2] bitbake-layers: Add ability to update the reference of repositories Date: Tue, 16 Jan 2024 11:55:37 +0100 Message-ID: <20240116105537.961-3-jermain.horsman@nedap.com> X-Mailer: git-send-email 2.42.0.windows.2 In-Reply-To: <20240116105537.961-2-jermain.horsman@nedap.com> References: <20240116105537.961-2-jermain.horsman@nedap.com> Reply-To: jermain.horsman@nedap.com MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 16 Jan 2024 10:57:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193851 From: Jermain Horsman This uses an existing setup-layers configuration and modifies one or more repositories using a reference provided by the user. This is a very minimal implementation, no validation of this reference is done and it is left to the user to provide a valid value. Signed-off-by: Jermain Horsman --- .../bblayers/setupwriters/oe-setup-layers.py | 79 +++++++++++++++++-- 1 file changed, 72 insertions(+), 7 deletions(-) diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py index bd71ca1f51..59ca968ff3 100644 --- a/meta/lib/bblayers/setupwriters/oe-setup-layers.py +++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py @@ -31,16 +31,69 @@ class OeSetupLayersWriter(): with open(output, 'w') as f: json.dump(repos, f, sort_keys=True, indent=4) + def _read_repo_config(self, json_path): + with open(json_path) as f: + json_config = json.load(f) + + supported_versions = ["1.0"] + if json_config["version"] not in supported_versions: + err = "File {} has version {}, which is not in supported versions: {}".format(json_path, json_config["version"], supported_versions) + logger.error(err) + raise Exception(err) + + return json_config + + def _modify_repo_config(self, json_config, args): + sources = json_config['sources'] + for pair in args.custom_references: + try: + repo, rev = pair.split(':', maxsplit=1) + except ValueError: + err = "Invalid custom reference specified: '{}'. Provide one using 'REPOSITORY:REFERENCE'.".format(pair) + logger.error(err) + raise Exception(err) + if not repo in sources.keys(): + err = "Repository {} does not exist in setup-layers config".format(repo) + logger.error(err) + raise Exception(err) + + layer_remote = json_config['sources'][repo]['git-remote'] + layer_remote['rev'] = rev + # Clear describe + layer_remote['describe'] = '' + def do_write(self, parent, args): """ Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. """ - if not os.path.exists(args.destdir): - os.makedirs(args.destdir) - repos = parent.make_repo_config(args.destdir) - json = {"version":"1.0","sources":repos} - if not repos: - raise Exception("Could not determine layer sources") output = args.output_prefix or "setup-layers" - output = os.path.join(os.path.abspath(args.destdir),output) + output = os.path.join(os.path.abspath(args.destdir), output) + + if args.update: + # Modify existing layers setup + if args.custom_references is None: + err = "No custom reference specified. Please provide one using '--use-custom-reference REPOSITORY:REFERENCE'." + logger.error(err) + raise Exception(err) + + json = self._read_repo_config(output + ".json") + if not 'sources' in json.keys(): + err = "File {}.json does not contain valid layer sources.".format(output) + logger.error(err) + raise Exception(err) + + else: + # Create new layers setup + if not os.path.exists(args.destdir): + os.makedirs(args.destdir) + repos = parent.make_repo_config(args.destdir) + json = {"version":"1.0","sources":repos} + if not repos: + err = "Could not determine layer sources" + logger.error(err) + raise Exception(err) + + if args.custom_references is not None: + self._modify_repo_config(json, args) + self._write_json(json, output + ".json") logger.info('Created {}.json'.format(output)) if not args.json_only: @@ -50,3 +103,15 @@ class OeSetupLayersWriter(): def register_arguments(self, parser): parser.add_argument('--json-only', action='store_true', help='When using the oe-setup-layers writer, write only the layer configuruation in json format. Otherwise, also a copy of scripts/oe-setup-layers (from oe-core or poky) is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json.') + + parser.add_argument('--update', '-u', + action='store_true', + help=("Instead of writing a new json file, update an existing layer setup json file with custom references provided via the '--use-custom-reference' option." + "\nThis will only update repositories for which a custom reference is specified, all other repositores will be left unchanged.")) + parser.add_argument('--use-custom-reference', '-r', + action='append', + dest='custom_references', + metavar='REPOSITORY:REFERENCE', + help=("A pair consisting of a repository and a custom reference to use for it (by default the currently checked out commit id would be written out)." + "\nThis value can be any reference that 'git checkout' would accept, and is not checked for validity." + "\nThis option can be used multiple times.")) From patchwork Tue Jan 16 10:55:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: jhatnedap@gmail.com X-Patchwork-Id: 37913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C92DC4706C for ; Tue, 16 Jan 2024 10:57:04 +0000 (UTC) Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by mx.groups.io with SMTP id smtpd.web10.10684.1705402621673460347 for ; Tue, 16 Jan 2024 02:57:01 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=QNafrVar; spf=pass (domain: gmail.com, ip: 209.85.218.49, mailfrom: jhatnedap@gmail.com) Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a277339dcf4so1085900466b.2 for ; Tue, 16 Jan 2024 02:57:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705402620; x=1706007420; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Ij+k4cMrUdDW8f+dfOLnGZ2C9znYtPpveKpFFUHO9qk=; b=QNafrVarci562GgTBe8GSoy+KU2kcfjGiHf13WNoJAKWutRVz2q4kU+koCCqY9Nlid WxffNrYVnRTDCN4pyoxiKeu7Xviopdo5NAggmrzQYl24TAu+jHtnKuy02trsKTcl0dTG QjNEcyinFX3i54dZIptpNqThYFEiGnrexsBmE4w2hQDdimMq7TygZEXyoLlWiGBe56EH z+/M2aQyuDhAxqLaF4t6MVm4SuMtrqzCXNvuQ9c9gY97naQxiIhq5U9beUrmBsX9Tq/2 35UeyIKNCUKEf9zAX22jfFTMssupjsw8HTunIXo6/sP0uvIp9vcm2Me5PtoOTN8TORem MVaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705402620; x=1706007420; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Ij+k4cMrUdDW8f+dfOLnGZ2C9znYtPpveKpFFUHO9qk=; b=H0bdpGoDf13OkXu0Ywe3D9XTah0faGriJ2X0bg5qDY2iQxH486Es20EchdvXkggJQt gEV9U7iixOWWmkrZXcMCSF9vp2YGvAhzm8MXqxLLxt8danP0OhDsTKWAHazxULrfINNz mSz6jqxU8qBOJJfFzZrNevOglOnsO92tBfrkOYsVARehMaORYGnqjH+cV+6GhPuAChwd HQKCILjewnkiCeD1ToU6wGWKNBVPJa46uxGKzCJPD7/UUD6D5kZlSqXWFlddjdsm9xV+ UiKoaHNByYWEbET0mcWMi7YqNpT8urLy4nwsWLLq1HeQ53vInPQPlhvnLPHrEWn4Zq3y RyiQ== X-Gm-Message-State: AOJu0YzmNLZfEyWx2OrvI0sPQU/87G53p6sduMwuax0DVw3nNfPn/TGk WnBtCLoIBpg2+RYk5caDonR/6U3G+FU= X-Google-Smtp-Source: AGHT+IEu9FEy+vZZfFtLSXMAFIsEsG6oAsvHvRjOO+exkewppNBKpsHieKDT0X7xFGDx1of9UJVX6Q== X-Received: by 2002:a17:907:1184:b0:a2d:5394:18fb with SMTP id uz4-20020a170907118400b00a2d539418fbmr2426453ejb.109.1705402619869; Tue, 16 Jan 2024 02:56:59 -0800 (PST) Received: from NVC4181.nedap.local ([149.3.168.10]) by smtp.gmail.com with ESMTPSA id tg14-20020a1709078dce00b00a2cb117050fsm5366969ejc.126.2024.01.16.02.56.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 02:56:59 -0800 (PST) From: jhatnedap@gmail.com X-Google-Original-From: jermain.horsman@nedap.com To: openembedded-core@lists.openembedded.org Cc: Jermain Horsman Subject: [RFC v4 2/2] bitbake-layers: Add test case layers setup update Date: Tue, 16 Jan 2024 11:55:38 +0100 Message-ID: <20240116105537.961-4-jermain.horsman@nedap.com> X-Mailer: git-send-email 2.42.0.windows.2 In-Reply-To: <20240116105537.961-2-jermain.horsman@nedap.com> References: <20240116105537.961-2-jermain.horsman@nedap.com> Reply-To: jermain.horsman@nedap.com MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 16 Jan 2024 10:57:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193852 From: Jermain Horsman This includes a simple test which creates a layer setup using custom references, and subsequently modifies the resulting layers setup using a different custom reference. Signed-off-by: Jermain Horsman --- meta/lib/oeqa/selftest/cases/bblayers.py | 53 ++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 8faa060234..bf807badb7 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -167,3 +167,56 @@ class BitbakeLayers(OESelftestTestCase): oecoredir = os.listdir(testcheckoutdir)[0] testcheckoutfile = os.path.join(testcheckoutdir, oecoredir, "oe-init-build-env") self.assertTrue(os.path.exists(testcheckoutfile), "File {} not found in test layer checkout".format(testcheckoutfile)) + + def test_bitbakelayers_updatelayer(self): + result = runCmd('bitbake-layers create-layers-setup {}'.format(self.testlayer_path)) + jsonfile = os.path.join(self.testlayer_path, "setup-layers.json") + self.validate_layersjson(jsonfile) + + import json + with open(jsonfile) as f: + data = json.load(f) + repos = [] + for s in data['sources']: + repos.append(s) + + self.assertTrue(len(repos) > 1, "Not enough repositories available") + self.validate_layersjson(jsonfile) + + test_ref_1 = 'ref_1' + test_ref_2 = 'ref_2' + + # Create a new layers setup using custom references + result = runCmd('bitbake-layers create-layers-setup --use-custom-reference {first_repo}:{test_ref} --use-custom-reference {second_repo}:{test_ref} {path}' + .format(first_repo=repos[0], second_repo=repos[1], test_ref=test_ref_1, path=self.testlayer_path)) + self.validate_layersjson(jsonfile) + + with open(jsonfile) as f: + data = json.load(f) + first_rev_1 = data['sources'][repos[0]]['git-remote']['rev'] + first_desc_1 = data['sources'][repos[0]]['git-remote']['describe'] + second_rev_1 = data['sources'][repos[1]]['git-remote']['rev'] + second_desc_1 = data['sources'][repos[1]]['git-remote']['describe'] + + self.assertEqual(first_rev_1, test_ref_1, "Revision not set correctly: '{}'".format(first_rev_1)) + self.assertEqual(first_desc_1, '', "Describe not cleared: '{}'".format(first_desc_1)) + self.assertEqual(second_rev_1, test_ref_1, "Revision not set correctly: '{}'".format(second_rev_1)) + self.assertEqual(second_desc_1, '', "Describe not cleared: '{}'".format(second_desc_1)) + + # Update one of the repositories in the layers setup using a different custom reference + # This should only update the selected repository, everything else should remain as is + result = runCmd('bitbake-layers create-layers-setup --update --use-custom-reference {first_repo}:{test_ref} {path}' + .format(first_repo=repos[0], test_ref=test_ref_2, path=self.testlayer_path)) + self.validate_layersjson(jsonfile) + + with open(jsonfile) as f: + data = json.load(f) + first_rev_2 = data['sources'][repos[0]]['git-remote']['rev'] + first_desc_2 = data['sources'][repos[0]]['git-remote']['describe'] + second_rev_2 = data['sources'][repos[1]]['git-remote']['rev'] + second_desc_2 = data['sources'][repos[1]]['git-remote']['describe'] + + self.assertEqual(first_rev_2, test_ref_2, "Revision not set correctly: '{}'".format(first_rev_2)) + self.assertEqual(first_desc_2, '', "Describe not cleared: '{}'".format(first_desc_2)) + self.assertEqual(second_rev_2, second_rev_1, "Revision should not be updated: '{}'".format(second_rev_2)) + self.assertEqual(second_desc_2, second_desc_1, "Describe should not be updated: '{}'".format(second_desc_2))