From patchwork Fri Feb 23 12:01:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CDAFC54798 for ; Fri, 23 Feb 2024 12:01:46 +0000 (UTC) Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by mx.groups.io with SMTP id smtpd.web11.8975.1708689704935869614 for ; Fri, 23 Feb 2024 04:01:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=al974W1k; spf=pass (domain: gmail.com, ip: 209.85.218.44, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a3f829cde6dso87659266b.0 for ; Fri, 23 Feb 2024 04:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689703; x=1709294503; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=hOfpcwtJPHLioc2iDM+BFjugBbNU3DfvXictnAj3QYY=; b=al974W1kSGgN4/NVouaF2MWV/EiXON1SZRY8Y5BuuKkjKOKG1Ul3PNuJJyy53yuoHW yi6nz/lV7HcSoBGhN6V1gVznaXN+pKDDlTlwVMFtfZuGatkSp+kWrrnYXyxwbz0DN7gi Vsze7aS5QqE/JECm4Pju+wQZv0FJvnd5CbQwCjdDTPKhv1RTxcKWEvbZ1Ocqh5K8bFle Tku10j2xCIZsXloK/G1XndvUl+OB/n+axRiuYDoxszEdsu9dm3mqURA0683wvhKV3VYD N9s8Xr1CEbawWzD7IYcE+oKxFFdn3ZM+duE3m6vavDKmwg9MKBhj+obzS3KjePcglwAU az0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689703; x=1709294503; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hOfpcwtJPHLioc2iDM+BFjugBbNU3DfvXictnAj3QYY=; b=hCqMsBZMwyggPyJmwPgAUvcH/6wIXbZiiuSHSiyWYIzAqY3AaUmFv4gHA3iK6aJ6Dm +VpkgaONP1lNlQJgI+5MbvYIzlhVDnkLoSaOh1Q4LnV5uzob/7GkkwL3iBT2Fd3Wycn8 +wCK9nGOKSKuBE5eEsmWFyBV4FsHCkG8gxRlpX1d9AgY2FG5ylCP4B9k502CykZAOGjL az1PFDdEnWkIeK/8/KQvLd3osSS/evo9rINHCV0XwQAiAjmHnlrbXKn+SVg01nlMNLBe zstIHIwBMv6GXP5zwNBm+wun0fXd8QKIXfEzgXNuRO84v70ZnhZv3obBGX/5BHFWo8FI t6wA== X-Gm-Message-State: AOJu0YxWxEiL5AhG6qEcb753Dp+nmXFHKNdTNaPybUbFgF+4vAvzQeu+ JDpdz+J24Ht37NHAIQUjEDY09bVqQYjNrk7lxBlLc+yr0CiVxklxNHXyBDRw X-Google-Smtp-Source: AGHT+IHliyIVP8Lvb0B0vh/kcstWRL5jV2T/N4MeAcD9g4oWlC7iANxCZuG8kZyuEzWOib/3xZ5BJg== X-Received: by 2002:a17:906:b792:b0:a3f:4eaf:ec66 with SMTP id dt18-20020a170906b79200b00a3f4eafec66mr1002071ejb.25.1708689702950; Fri, 23 Feb 2024 04:01:42 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:42 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 01/10] scripts/oe-setup-build: write a build environment initialization one-liner into the build directory Date: Fri, 23 Feb 2024 13:01:25 +0100 Message-Id: <20240223120134.3713127-1-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:46 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196061 This allows avoiding having to know where oe-init-build-env is relative to the build directory; that information is contained in the one liner and then it's possible to simply use that: . /path/to/build/init-build-env This will particularly help with initializing builds in unpacked build bundles, as users won't have to know where oe-init-build-env is in the bundle directory tree - similar to esdk initialization. Signed-off-by: Alexander Kanavin --- scripts/oe-setup-build | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/scripts/oe-setup-build b/scripts/oe-setup-build index 5364f2b4812..c0476992a2a 100755 --- a/scripts/oe-setup-build +++ b/scripts/oe-setup-build @@ -91,7 +91,16 @@ def setup_build_env(args): builddir = args.b if args.b else template["buildpath"] no_shell = args.no_shell coredir = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..')) - cmd = "TEMPLATECONF={} . {} {}".format(template["templatepath"], os.path.join(coredir, 'oe-init-build-env'), builddir) + cmd_base = ". {} {}".format(os.path.join(coredir, 'oe-init-build-env'), os.path.abspath(builddir)) + + initbuild = os.path.join(builddir, 'init-build-env') + if not os.path.exists(initbuild): + os.makedirs(builddir, exist_ok=True) + with open(initbuild, 'w') as f: + f.write(cmd_base) + print("\nRun '. {}' to initialize the build in a current shell session.\n".format(initbuild)) + + cmd = "TEMPLATECONF={} {}".format(template["templatepath"], cmd_base) if not no_shell: cmd = cmd + " && {}".format(os.environ['SHELL']) print("Running:", cmd) From patchwork Fri Feb 23 12:01:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E074C54E49 for ; Fri, 23 Feb 2024 12:01:46 +0000 (UTC) Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) by mx.groups.io with SMTP id smtpd.web10.9015.1708689705420209203 for ; Fri, 23 Feb 2024 04:01:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PS06JP4g; spf=pass (domain: gmail.com, ip: 209.85.218.45, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-a3ee69976c9so49508066b.0 for ; Fri, 23 Feb 2024 04:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689704; x=1709294504; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Oy9njXgd7UeS0Vdnklq95qp49wdar6iDZvYtG8lpUDw=; b=PS06JP4gMQmzL46sz3Zx3lJZZg//wFLeOjOrtrty6RPFIQiEGDdiAJ24UIgIOwh8DM LUj8LpIB49+L4IwuOZV+qKfG/wT1ZPjY+UXNq5u5Oh/kafrngatBI/fVtu71fnpcv4jY sN1kCDU8Mg/RJYaTnD7eJzaYRbHj/e9NB7Bbq4MjNkDMjaMrxIvoAoeuFCr3yFm7jbjG JWv6+ElUpkmuT0Mf2kdVM9GWQ+CkB2/vbmx9pDd09Tf9Q7tAg36UGG25AYNHcyaKZ0PR 0BUGQif1GIoFxII6S0GX/DdSmZ2dy2bMpS00peZ3zJw0BPxIvrS/ZrrK1OWdWUaFW0u5 e02w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689704; x=1709294504; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Oy9njXgd7UeS0Vdnklq95qp49wdar6iDZvYtG8lpUDw=; b=SFQPDL50AjGg+qsiQqbxDl7V7wDGW25NjCz/EK3zwTDCJG1q2t3x9+4XFRn3u38FKC oTMae6+fzwFnBspLhr914CFIOb1/QXjoawzMRt9mTXabbnCugXCB/jtXwMJS/11eeg4n fPYswyxQtRwWJgVw7GHWYu5TX1m3TK47X+0eRTDXcWQNdaOZMy5T0QFja2xoHBcH8BdG b4XJds9G3oI8mdTIJaugjEQ+ciQdDxmvz0x/ZdYYy0EIIs4t2e+bJgSwxN09hirLV98z gIsO6AwMtGLZuAwjDzwkzCqdAddt7XGdmVs3bQMZHMeEYgdmkRuY0YvWGJhg1I46EkV9 UdqA== X-Gm-Message-State: AOJu0Yx8S92YQhIYYj3271Pk8Rbkc+P9P+TRB/kgN7Fjc+M8HWXQ4WFw tyHwi1c4uGBFon1Kt+4amlNyMTQZ1VIAoHwo40TdSZqtbwQQuCu9ZJMTcJ/Z X-Google-Smtp-Source: AGHT+IEMw+6r3pWMPKK8h2f29tCBFFSv4ZMCFrgdy4BW/SgjfufkQAHZDGWR0JsUNK9auaag1oVyeA== X-Received: by 2002:a17:906:4813:b0:a3f:a3bc:1444 with SMTP id w19-20020a170906481300b00a3fa3bc1444mr1093537ejq.54.1708689703557; Fri, 23 Feb 2024 04:01:43 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:43 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 02/10] bblayers/makesetup: raise exceptions when errors happen Date: Fri, 23 Feb 2024 13:01:26 +0100 Message-Id: <20240223120134.3713127-2-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:46 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196062 Otherwise the calling code can only issue a generic, unhelpful erorr message, and it's difficult to tell what went wrong if logger.error output is obscured or redirected. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py index 4f27c565eeb..ecfa65adc7f 100644 --- a/meta/lib/bblayers/makesetup.py +++ b/meta/lib/bblayers/makesetup.py @@ -65,8 +65,9 @@ class MakeSetupPlugin(LayerPlugin): if l_name == 'workspace': continue if l_ismodified: - logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path)) - return + e = "Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path) + logger.error(e) + raise Exception(e) repo_path = self._get_repo_path(l_path) if self._is_submodule(repo_path): @@ -76,8 +77,9 @@ class MakeSetupPlugin(LayerPlugin): if repo_path == destdir_repo: repos[repo_path]['contains_this_file'] = True if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']: - logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) - return + e = "Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path) + logger.error(e) + raise Exception(e) top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()]) From patchwork Fri Feb 23 12:01:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAC30C5478C for ; Fri, 23 Feb 2024 12:01:46 +0000 (UTC) Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by mx.groups.io with SMTP id smtpd.web11.8977.1708689705787603901 for ; Fri, 23 Feb 2024 04:01:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=kp9uqRBf; spf=pass (domain: gmail.com, ip: 209.85.218.49, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a3e550ef31cso83291666b.3 for ; Fri, 23 Feb 2024 04:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689704; x=1709294504; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1z31RHaHiY8+IULtx4EM/dAvi6dGN1TMREQeo8g8IuI=; b=kp9uqRBfRB2xtRrN5HEOZw2hmVNB8pVceG0vKo7RxYkLU7V15VcaKFmJgw33R5h9Wk 8NgLfslvvJGUDMNT54LgxyLSams1ER7ZYpud3n0XlNKGbWr+YS1awZvi8nYpuYqfANMU n5ZzdV2ZKcbWxo5pBZYIlUghsdBXLg3zL9MlP+jIw8kudb3+L6BKaCc0EI2ZyeVBaRl4 9i91gt4q558sTokiDl4Xx2CFBUHyz+unL17zAQT564hZSwW94CtSBE1AR+cZkR1eDL9X 1kkzRHEEOU1AA3GHu9RYqrAlY1NSi0LAsEC0qCf0FNW7bDGrMU1lK33yM8WPdLg08jl3 NH7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689704; x=1709294504; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1z31RHaHiY8+IULtx4EM/dAvi6dGN1TMREQeo8g8IuI=; b=wqozKNXH4veX+huA79ndKtuUCujo7l4AfyTkKDk9T7Yub/dLTF9ITwZ9OCOlFhw2uf vH0OLRYhxyMkt6DCaZfHfKHeO+9nfFYR6sJS+zuyJIPn0ZSFpAx0Fb3kFSmsGsaOFrD7 4aLRL/7sllKLqmeqGHKkukWQbUdcnznw3Zc/WliVDmcArQzSQJcBPg3EAdd0TGk7f3lF /JsQkEitEGXOtz0dffZBldfjojP4Vyb/S1IQAiH1bXRgeDV81GsjhAmgG0AvxNtKQMxf Mi60EhITxle9phPjnKXN8PXj+W5dXFyrDj4TCsYooV0DHrHXP9EKQfr7AIYPDqlrIgv/ VddQ== X-Gm-Message-State: AOJu0YyoiOwtrzCF5YO2ZVxi6ttyVfTcHE6/nL+mL4+tZqAD7MOjNWsQ xwGjz2bYJlhT/XooPQmWtLNUt/He03CaR0LsnrYHt/iPLmnj8k2sEbLpzTcF X-Google-Smtp-Source: AGHT+IHtVUTgPt0vsWpeftTdavdPKfJCFKLnNZ0x6Z0NIp8TU2UptK0HRWIQISfh/bmCsWlOm26RaA== X-Received: by 2002:a17:906:6b8c:b0:a3e:4fc6:c881 with SMTP id l12-20020a1709066b8c00b00a3e4fc6c881mr1218284ejr.36.1708689704098; Fri, 23 Feb 2024 04:01:44 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:43 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 03/10] bblayers/makesetup: include local repository paths in discovered layer repo data Date: Fri, 23 Feb 2024 13:01:27 +0100 Message-Id: <20240223120134.3713127-3-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:46 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196063 This can be useful when the caller needs to know where the repos actually are on local disk (for example to make a local copy of them or obtain additional information about them). This field however should be removed from the json output in oe-setup-layers plugin, as it is host-specific and not part of the schema. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 1 + meta/lib/bblayers/setupwriters/oe-setup-layers.py | 2 ++ 2 files changed, 3 insertions(+) diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py index ecfa65adc7f..814cbb6357c 100644 --- a/meta/lib/bblayers/makesetup.py +++ b/meta/lib/bblayers/makesetup.py @@ -89,6 +89,7 @@ class MakeSetupPlugin(LayerPlugin): repos_nopaths[r_nopath] = repos[r] r_relpath = os.path.relpath(r, top_path) repos_nopaths[r_nopath]['path'] = r_relpath + repos_nopaths[r_nopath]['originpath'] = r return repos_nopaths def do_make_setup(self, args): diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py index 59ca968ff33..56c929471e6 100644 --- a/meta/lib/bblayers/setupwriters/oe-setup-layers.py +++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py @@ -85,6 +85,8 @@ class OeSetupLayersWriter(): if not os.path.exists(args.destdir): os.makedirs(args.destdir) repos = parent.make_repo_config(args.destdir) + for r in repos.values(): + del r['originpath'] json = {"version":"1.0","sources":repos} if not repos: err = "Could not determine layer sources" From patchwork Fri Feb 23 12:01:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87901C5478C for ; Fri, 23 Feb 2024 12:01:56 +0000 (UTC) Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by mx.groups.io with SMTP id smtpd.web10.9016.1708689706164187959 for ; Fri, 23 Feb 2024 04:01:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=NImkQiCo; spf=pass (domain: gmail.com, ip: 209.85.218.46, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a3fd9063261so45279966b.1 for ; Fri, 23 Feb 2024 04:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689704; x=1709294504; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q21Ze4Xo1FqnTNlgq9oxGbt7Xk4ojTxu3fTx4YyPq/c=; b=NImkQiCo9V+wVGlJdjsVy8Uf8GQm0UpI0e7AcRcyEagrUEHJP9iD0lUJFLaRUItbkj M7wnKgugheXa/7EbvMnAuakBHeOEoKVBSqljUSo3SJtPFW55Wt6rOGbfr5QWxfrXMbcF beylOlPGL8ZPEJyjT06p+9tfC2CyBJj5QiKhsDVpuFd+ib7DtP7rGKwy/AZYTqwxpJXj 7kHTZJBnuIXPkvs+KKQZBwuX6L2CoHZVhR+hn9nAkVKymjK8Cgwfb3usHmnqXRpRygC+ NxcKOreT7LSZFVNwXJVD0paqD8bk8uaZXjzmZc3NgFdw38+ChFpfK5GAhc86AWSRTZw6 rbXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689704; x=1709294504; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q21Ze4Xo1FqnTNlgq9oxGbt7Xk4ojTxu3fTx4YyPq/c=; b=Nr/9FyzVlhdnV06C+M44Bfoh95mDOKucipyvPtXwTEBhnwQkiTLogNBfV6zTzljQjS lfr1HzsA6Tnv6EWPA9izJJr5RCin+jAPV8gAZT2TmmPVA/ap4W676lnh259a3ScKu70B uemAZxLPbZkTS9+Pkl2fcMpC5FZy/70irmNmDUPBIreGae8d35Q642txJ+c9OHblJY41 v2Fdo26oOoBpQL8uT84HGcrvKH/mzrn41VQfYJm1OBcsijHjF8cZV+hOqQiBpe/cMZhC WxuAkQe7jsSu0BmV2vwGAFTkZTphCxKLB5fGy+L3khSXH6G1XO/2Wzx6boL2tYGdJLjw anBw== X-Gm-Message-State: AOJu0YzIQVQAABjX0/+VYvYO/ak5QL1qNdb2XSXSkr7eyWvBQFCcJ3sC L9TjJfTLiyqLbWZg8i8szYxIVrgpDPvvDrWcI5f5FjM0PUuVULP/YrSHfrBC X-Google-Smtp-Source: AGHT+IFcHIRPO/+oUhigzXuivFK94a5JtBMoQoDnfEIxkT8AIUQLWpvyuqUeqR1TpvSbKbK0u7ikjQ== X-Received: by 2002:a17:906:eb0d:b0:a41:309d:8a34 with SMTP id mb13-20020a170906eb0d00b00a41309d8a34mr246205ejb.56.1708689704597; Fri, 23 Feb 2024 04:01:44 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:44 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 04/10] bblayers/makesetup: move check for existence of git remotes to oe-setup-layers plugin Date: Fri, 23 Feb 2024 13:01:28 +0100 Message-Id: <20240223120134.3713127-4-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196064 This helps when making a local copy of all the layers: it's fine to not have any remotes configured in that case. Also correct the mistake in accessing 'contains_this_file' property: it may not necessarily exist, and if so, it should default to False. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 4 ---- meta/lib/bblayers/setupwriters/oe-setup-layers.py | 3 +++ 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py index 814cbb6357c..ebe6bd5440a 100644 --- a/meta/lib/bblayers/makesetup.py +++ b/meta/lib/bblayers/makesetup.py @@ -76,10 +76,6 @@ class MakeSetupPlugin(LayerPlugin): repos[repo_path] = {'path':os.path.basename(repo_path),'git-remote':{'rev':l_rev, 'branch':l_branch, 'remotes':self._get_remotes(repo_path), 'describe':self._get_describe(repo_path)}} if repo_path == destdir_repo: repos[repo_path]['contains_this_file'] = True - if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']: - e = "Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path) - logger.error(e) - raise Exception(e) top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()]) diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py index 56c929471e6..8faeabfabc8 100644 --- a/meta/lib/bblayers/setupwriters/oe-setup-layers.py +++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py @@ -86,6 +86,9 @@ class OeSetupLayersWriter(): os.makedirs(args.destdir) repos = parent.make_repo_config(args.destdir) for r in repos.values(): + if not r['git-remote']['remotes'] and not r.get('contains_this_file', False): + e = "Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=r['originpath']) + raise Exception(e) del r['originpath'] json = {"version":"1.0","sources":repos} if not repos: From patchwork Fri Feb 23 12:01:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEC3AC54E49 for ; Fri, 23 Feb 2024 12:01:56 +0000 (UTC) Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) by mx.groups.io with SMTP id smtpd.web11.8981.1708689706925111934 for ; Fri, 23 Feb 2024 04:01:47 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=QZ0yGIDw; spf=pass (domain: gmail.com, ip: 209.85.208.49, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-55f279dca99so1029227a12.3 for ; Fri, 23 Feb 2024 04:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689705; x=1709294505; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LQf6xTUaMPFjGWdEp3oX+VMcpuC/bNXJcVjctzut2RY=; b=QZ0yGIDw9q+4KE0ilQF6SzacQIfR+Ocqx9IjfCX+KmA1XJbjSqwLUOEAa9+AHzBBly Ugo1GlGImXrGOH4+4Og3iUYO35dzyAdX3K5P3drDuAE7YUh/0b7pfDAMX2Q5Oy/HrHD3 DSPAtUasXQbAGXrcUiUDNzTDCkIAzug6WJ5hng5Nn+RsRTwXvHYrd2C+WEOUwQ3/nyt9 pY6XXdT8JPQN6l8w+vSlTdtt9cmyjc54iYcB2a4YX2sW90EaFPJ6St98sWQ4i3WaXPff dBX/WEqR6RSREoF1MJVkcuTYkmA1Lpy8ElMWSzOA0/JlGsCGc3XqcPS/D9MD4cVeSE1j 2ViQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689705; x=1709294505; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LQf6xTUaMPFjGWdEp3oX+VMcpuC/bNXJcVjctzut2RY=; b=MlyIBqopG0vz9nUliFK+9vlOzjkW43U03noWEc2bYULWDHXLsMk2ebF11+BROdbFeq shjd1maHax8NZGidqYrX1zOGevVf7J7zKEYkwvXJ6qbmrO2lPgR/si7Ci623LDq5ykcD WCavQN+GSs85sgRguIedg/tLqJqtmE4CkjU/6K+Wju3M8YBtcblBNVCIMp1gNZZsorfQ htxP+myOQav2pilKOuuKXw+9UjgrS1UQ9qv+cfjZEmrN149/XKSNGCstjgGwndXMhMfK gD+F58ZejjND2tZu/QN4MJUa9eslYaSsRPjCJMH+e9d0Y1clRuC7OJ/nYp0TI2IFn6Hg zrWw== X-Gm-Message-State: AOJu0YxJOAG5bx0zVgbRHzlHhOxve66UbaXYegcaPVgBPoTnYftQmAwa 2PHuPJIi97iEJGguCFfBpUC+lNXV8RcPrsAQbFM2jgvvl/vpbwC/kzqLM1sQ X-Google-Smtp-Source: AGHT+IELQ6wBK9N5esREKZxpgsCc3oRutL8Senx1Hv/GPEGUGJoIO6BPtGNLAkQ6pzhKayHQTBCtJA== X-Received: by 2002:a17:906:688d:b0:a3f:c10:4355 with SMTP id n13-20020a170906688d00b00a3f0c104355mr1201691ejr.26.1708689705131; Fri, 23 Feb 2024 04:01:45 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:44 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 05/10] bblayers/setupwriters/oe-local-copy: add a 'local copy' plugin for bitbake-layers create-layer-setup Date: Fri, 23 Feb 2024 13:01:29 +0100 Message-Id: <20240223120134.3713127-5-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196065 This plugin copies all currently configured layer respositories into a dedicated location on local disk. This is useful for entirely offline layer replication, when the layers are packed and then unpacked from an archive, rather than fetched from git (there can be situations where fetching from git is undesirable or impossible). This plugin will also be used when replicating an entire yocto build (with build config and sstate cache), and is an element of that. This plugin is similar to what esdk does, and it provides that functionality outside of populate_sdk_ext task. It does not reuse esdk code, as that simply copies the layer tree; it's better to use 'git clone' with a path to original repo on local disk, as that will preserve commit history in the copy. Signed-off-by: Alexander Kanavin --- .../bblayers/setupwriters/oe-local-copy.py | 41 +++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 13 ++++++ 2 files changed, 54 insertions(+) create mode 100644 meta/lib/bblayers/setupwriters/oe-local-copy.py diff --git a/meta/lib/bblayers/setupwriters/oe-local-copy.py b/meta/lib/bblayers/setupwriters/oe-local-copy.py new file mode 100644 index 00000000000..8c1ccb67368 --- /dev/null +++ b/meta/lib/bblayers/setupwriters/oe-local-copy.py @@ -0,0 +1,41 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import json + +logger = logging.getLogger('bitbake-layers') + +def plugin_init(plugins): + return OeLocalCopyWriter() + +class OeLocalCopyWriter(): + + def __str__(self): + return "oe-local-copy" + + def do_write(self, parent, args): + """ Writes out a local copy of all the metadata layers (and bitbake) included in a current build. """ + if not os.path.exists(args.destdir): + os.makedirs(args.destdir) + repos = parent.make_repo_config(args.destdir) + if not repos: + raise Exception("Could not determine layer sources") + output = os.path.join(os.path.abspath(args.destdir), args.output_prefix or "layers") + json_f = os.path.join(os.path.abspath(args.destdir), "bundle-repos.json") + + for r in repos.values(): + r['git-remote']['remotes'] = {"origin":{"uri":r["originpath"]}} + + json_data = {"version":"1.0","sources":repos} + with open(json_f, 'w') as f: + json.dump(json_data, f, sort_keys=True, indent=4) + + logger.info("Cloning repositories into {}".format(output)) + bb.process.run('oe-setup-layers --force-bootstraplayer-checkout --destdir {} --jsondata {}'.format(output, json_f)) + + def register_arguments(self, parser): + pass diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 695d17377d4..8b2bc319d50 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -240,3 +240,16 @@ class BitbakeLayers(OESelftestTestCase): self.assertEqual(first_desc_2, '', "Describe not cleared: '{}'".format(first_desc_2)) self.assertEqual(second_rev_2, second_rev_1, "Revision should not be updated: '{}'".format(second_rev_2)) self.assertEqual(second_desc_2, second_desc_1, "Describe should not be updated: '{}'".format(second_desc_2)) + + def test_bitbakelayers_setup_localcopy(self): + testcopydir = os.path.join(self.builddir, 'test-layer-copy') + result = runCmd('bitbake-layers create-layers-setup --writer oe-local-copy {}'.format(testcopydir)) + oe_core_found = False + meta_selftest_found = False + for topdir, subdirs, files in os.walk(testcopydir): + if topdir.endswith('meta/conf') and 'layer.conf' in files: + oe_core_found = True + if topdir.endswith('meta-selftest/conf') and 'layer.conf' in files: + meta_selftest_found = True + self.assertTrue(oe_core_found, "meta/conf/layer.conf not found in {}".format(testcopydir)) + self.assertTrue(meta_selftest_found, "meta-selftest/conf/layer.conf not found in {}".format(testcopydir)) From patchwork Fri Feb 23 12:01:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAE41C54E51 for ; Fri, 23 Feb 2024 12:01:56 +0000 (UTC) Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by mx.groups.io with SMTP id smtpd.web11.8982.1708689707167090497 for ; Fri, 23 Feb 2024 04:01:47 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ZWenUI5v; spf=pass (domain: gmail.com, ip: 209.85.218.49, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a3e7f7b3d95so86625766b.3 for ; Fri, 23 Feb 2024 04:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689705; x=1709294505; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uykDh6tidqiAhfxLvh+iRdpFFA/dtrPIiixnnGiCnN8=; b=ZWenUI5vb8wKyqhsY4dg27YILOxz95SnAV4Ya0cZ0FKdeDd60emM1z6UIQoCBrDY2d mARCST/4+t2+YSPA690Ybfeh4amxLJX3z5u2PlmOFlmihgm0WoVIuG1YjxIhn2iKwK/z G66aRvBB0XREgBUFZswueO1Jwh+wcoa4lDaCZybe8ltBr41IKzH6FypP9DVTMuwnaT+w frX9P6JglM1sWnNnVkl75bDiBtTOPEBzzoTf6346HkBSBb8ggMsUKwL5jUsMkSeaNloQ eYM/tlp5LxrB9yPcD2mRLT9gntj366s89tWrKx33upnD5Zyvx05g+5r4YaeRf5vXUhnT XLXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689705; x=1709294505; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uykDh6tidqiAhfxLvh+iRdpFFA/dtrPIiixnnGiCnN8=; b=YlXacQoCxtSqmKq9otQ7DHiLa1S8PP52nTIP4icKvS43zZ8/RV7Czu95a8BLXg0K08 BeRzcn/VL/IdapDB22LlOC+UdZatsq4WCU3wsCIe+D5CovE4KVjLRpkwcUAw+0CXHZ8/ OnlndiKzw0u4l6GaGR35QIoV8Ly51KaqeLR/CMqJbicCwkJ5eu3C745nF6Yvp/sUpPGb kafSWNJfCl5/TH1zmxKMC7INOi3M6MT37lHn4Lv1QGNFuUKnsmv6iQzGdc7gRbv9J33+ ldEdOAg5bVwXsEF3dPkOa/lDw2x3UfktuVi+Yy9TG3y9oGiKIcueHmjs1Myrz8Hc0Lxu dGSA== X-Gm-Message-State: AOJu0YwEkrOAGLzlZySxJ6Ij31nRtx1Jb2UVSujq3GKSfBI62LewFqBs HY1xAxEluEiWY7+OgS2fMez/Gulx2Mx4sSMXXlKOQub+I6QD+Kj2Ld0n+nLV X-Google-Smtp-Source: AGHT+IGMNq4pVLfPjVKTMieSeWHklwYPU5lULRyQ1IQgJx+wfWgWyCDOdcxaPl1khgtLkw6aANcl/Q== X-Received: by 2002:a17:906:3413:b0:a3f:9971:bf73 with SMTP id c19-20020a170906341300b00a3f9971bf73mr493785ejb.55.1708689705628; Fri, 23 Feb 2024 04:01:45 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:45 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 06/10] lib/oe/sstatesig.py: add an option to make a unihash cache copy Date: Fri, 23 Feb 2024 13:01:30 +0100 Message-Id: <20240223120134.3713127-6-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196066 This is already performed in eSDK creation; the change allows also making a copy with a standalone 'bitbake -S unihash-cache-copy targets...' command, which is useful for replicating builds on a different build host (simply coping sstate objects is not enough if they need to be discovered through unihash redirections). Signed-off-by: Alexander Kanavin --- meta/lib/oe/sstatesig.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index aa891ecf0a0..e2582917fbd 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -146,6 +146,10 @@ class SignatureGeneratorOEBasicHashMixIn(object): sigfile = os.getcwd() + "/locked-sigs.inc" bb.plain("Writing locked sigs to %s" % sigfile) self.dump_lockedsigs(sigfile) + if 'unihash-cache-copy' in options: + cachefile = os.getcwd() + "/bb_unihashes.dat" + bb.plain("Writing unihash cache to %s" % cachefile) + bb.parse.siggen.copy_unitaskhashes(os.getcwd()) return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options) From patchwork Fri Feb 23 12:01:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7D5DC54E52 for ; Fri, 23 Feb 2024 12:01:56 +0000 (UTC) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mx.groups.io with SMTP id smtpd.web10.9018.1708689707816042054 for ; Fri, 23 Feb 2024 04:01:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=OwziZ5Vy; spf=pass (domain: gmail.com, ip: 209.85.218.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-a3f1bf03722so41285466b.1 for ; Fri, 23 Feb 2024 04:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689706; x=1709294506; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=10CoF3ztwnY21f99CLvzWe5fEhXGn21nDxJ5CDreXOE=; b=OwziZ5VyDrf+bOtFtqqiQlIE9AeqjEsgdOs5sjnGu7/2nrE/DJQlU6rCDeeTIXeaSJ lqUaUYZSpmKIzmgbUWIYRA2LfRrVodOK9irzxTaJB0RdpaIFN1kQo3Haz1sARQLGL/vr 6OxTiPjetZoL91eGqhgM6HrsQf/s9bQk07tP1yoCjapQv4TSqwZXXoJyyZ1wE8JlbycD YbWyq8NGKBBfNTryvGHZVtaO1HzA9zABP96pL4qT1aCbmyyLqsQ7HupM0eET8W8lr2bM Bzft5BsbzAV7eu8nQ2kagiIiHHF433cdSXi3yipZhJJ+7/dVe6pe0FcPWOksRgifdWJA XlyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689706; x=1709294506; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=10CoF3ztwnY21f99CLvzWe5fEhXGn21nDxJ5CDreXOE=; b=fTaie78Q4wtvcuGllZXF3qY55rK8d3gsHPT1Pq8nKxx0UP8GcdyEwUn2msAhuJcHkq jO/reMugnyoGDiB5ALJFIVaBHEfI3cDjf5eQ3n95o0LQZy8N4NBa4xgynUNd2+/yfN0X HwBlHfIYY3XbNKvGE0LORMsSJoyk6hOYZ361jXoTGNLbRBgqwmoY+rWA5fUpjC7u0ek2 CTpF77Qg68tfKvsbEX+dw/dmW9V8LYVc0uBK6Pf/iJdS+HUHF8ZycnzZhDn1ruc6o1PK A68rjlMewFycqwn1GkVAL9fP0DqWaGx/Zj7XK1Wm1FcbhXNXgtkGq6wueQwkd5VE+w0X eJjQ== X-Gm-Message-State: AOJu0YytiJC7o5bKRfKoIdKRN7nVSKNvaxEB2t1AuKTD/4KHtGPbkg08 ESw7d4bDNZIdcrsNRBKZlh0qYtRwouOdhbnb5EuLBT+efNgP/d6ZCZjU1jru X-Google-Smtp-Source: AGHT+IE7n18pjJxxzRJpLyz+N79zZIuBa+O2FDk2DF9m65UkLfilFkxa5jT05EI6Hf3t15pPuvqckw== X-Received: by 2002:a17:906:b88e:b0:a3f:c922:6ed7 with SMTP id hb14-20020a170906b88e00b00a3fc9226ed7mr851911ejb.21.1708689706125; Fri, 23 Feb 2024 04:01:46 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:45 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 07/10] sstatesig/dump_sigs: allow setting output directory explicitly from command line option Date: Fri, 23 Feb 2024 13:01:31 +0100 Message-Id: <20240223120134.3713127-7-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196067 The default (current directory) is unfortunately non-deterministic, as the operation is performed from bitbake-server and not from bitbake process directly. If the server was started in a different directory it will write the files there, and not where bitbake command requesting those files was started. Signed-off-by: Alexander Kanavin --- meta/lib/oe/sstatesig.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index e2582917fbd..4c5b168fb3f 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -142,14 +142,19 @@ class SignatureGeneratorOEBasicHashMixIn(object): super().set_taskdata(data[3:]) def dump_sigs(self, dataCache, options): + outdir = os.getcwd() + for o in options: + if o.startswith('outdir'): + outdir = o.split('=')[1] + if 'lockedsigs' in options: - sigfile = os.getcwd() + "/locked-sigs.inc" + sigfile = outdir + "/locked-sigs.inc" bb.plain("Writing locked sigs to %s" % sigfile) self.dump_lockedsigs(sigfile) if 'unihash-cache-copy' in options: - cachefile = os.getcwd() + "/bb_unihashes.dat" + cachefile = outdir + "/bb_unihashes.dat" bb.plain("Writing unihash cache to %s" % cachefile) - bb.parse.siggen.copy_unitaskhashes(os.getcwd()) + bb.parse.siggen.copy_unitaskhashes(outdir) return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options) From patchwork Fri Feb 23 12:01:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95295C54798 for ; Fri, 23 Feb 2024 12:01:56 +0000 (UTC) Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by mx.groups.io with SMTP id smtpd.web11.8983.1708689708260554441 for ; Fri, 23 Feb 2024 04:01:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PS3vVqql; spf=pass (domain: gmail.com, ip: 209.85.218.44, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a3d484a58f6so105299466b.3 for ; Fri, 23 Feb 2024 04:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689707; x=1709294507; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fdrlJHj/g07klK3ZoNzn0msgnWO21kpNTXnGgIls6lE=; b=PS3vVqql6aKoNdDpOABDAq44DioUT4+WCAiLNPt989hJT445IhXHDL29TiCgbPNcII L9e8did4IQdMjp/gx3valekzutmL9JKpJJOwCkK1SylZhduCigg/0A9jIcWwtUzZCQyi yF7P+TSx6QKWbgCjuh7MwcC644++YdHdglZx6KtCU8QPobyA35iew8iOMGYPbvZE/an4 RphJGSCkl+PMUEWt5L73/UIxy+3+P7IFwaeOIEtTov1oCO0COgwGuyH7asNBscvpMMEC GIwstW6n52RWCd8KcwOYADkdiYT8YGqU3/3Zfjrud2YA9bVparO5k88IbhW6ksAu5MTj S4sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689707; x=1709294507; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fdrlJHj/g07klK3ZoNzn0msgnWO21kpNTXnGgIls6lE=; b=qdpHa882EWz89CLpKVm7JYY7TfdOBYBOGvZVgiXFQblpq1Lw7iv/th0VSeWPfCkzia zQl9a2D8knsfJ/Uhmfez8IksqjpNdXVwZ7PpNiV/z01OULcSqC9xrY0LJDRqfM9cEtM1 TnckWI2wH4gNDlC9qjSKaOBhYuVv5EKSsqHCnu/Xodf+K+qCCtBLxvUEjur3aKjMX/EI DOXKMsxl/q6/siSz9s7dHKW3sopPUpeZQuSMpATT/+FvYb6pVcfl+prUdcG6hjbjRwSK dTGI4PI02Nrui+EGzz/kwtoTyAyve8G+PL9ePrloS0wrVWMJzgCMet88WdBJ8lTDR9SM yRbA== X-Gm-Message-State: AOJu0YzTzcw/lf+uIcRMjg3zqEyVr1hVKTY9N1wTCFKleXaqcGu+GJZL 166hRMnDorTcPpIFX0M2oyyNt7S/IkAq1lH8Z6AhgUogyIxanwimOP0iDcOn X-Google-Smtp-Source: AGHT+IF60POsEh3ymmVX/jg2iaSZp6Q3RCO0GmfD7tg/CToZ7AAZuUKMtb+VcoYK5HRbOPhYUzQTmA== X-Received: by 2002:a17:906:a10e:b0:a3f:3aa:7dcf with SMTP id t14-20020a170906a10e00b00a3f03aa7dcfmr1118782ejy.0.1708689706673; Fri, 23 Feb 2024 04:01:46 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:46 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 08/10] selftest/sstatetests: separate sstate presence check into its own class Date: Fri, 23 Feb 2024 13:01:32 +0100 Message-Id: <20240223120134.3713127-8-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196068 This allows reusing the check function in tests other than CDN/local mirror ones, such as sstate bundle testing. Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/sstatetests.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 56dfcdb0f39..8d41ead8abf 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -890,8 +890,7 @@ INHERIT += "base-do-configure-modified" """, expected_sametmp_output, expected_difftmp_output) -@OETestTag("yocto-mirrors") -class SStateMirrors(SStateBase): +class SStateCheckObjectPresence(SStateBase): def check_bb_output(self, output, exceptions, check_cdn): def is_exception(object, exceptions): for e in exceptions: @@ -933,6 +932,8 @@ class SStateMirrors(SStateBase): self.assertEqual(len(failed_urls), missing_objects, "Amount of reported missing objects does not match failed URLs: {}\nFailed URLs:\n{}\nFetcher diagnostics:\n{}".format(missing_objects, "\n".join(failed_urls), "\n".join(failed_urls_extrainfo))) self.assertEqual(len(failed_urls), 0, "Missing objects in the cache:\n{}\nFetcher diagnostics:\n{}".format("\n".join(failed_urls), "\n".join(failed_urls_extrainfo))) +@OETestTag("yocto-mirrors") +class SStateMirrors(SStateCheckObjectPresence): def run_test(self, machine, targets, exceptions, check_cdn = True): # sstate is checked for existence of these, but they never get written out to begin with exceptions += ["{}.*image_qa".format(t) for t in targets.split()] From patchwork Fri Feb 23 12:01:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39974 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0544C54E4C for ; Fri, 23 Feb 2024 12:01:56 +0000 (UTC) Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by mx.groups.io with SMTP id smtpd.web10.9019.1708689709696090317 for ; Fri, 23 Feb 2024 04:01:50 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=P0VEsv9S; spf=pass (domain: gmail.com, ip: 209.85.218.52, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-a3d5e77cfbeso141095766b.0 for ; Fri, 23 Feb 2024 04:01:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689708; x=1709294508; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BF2shshDxKBtCT7/MXQqYBl25A0ZFLNv5UbtYZIopXs=; b=P0VEsv9ShfRY9TvySRBs72zMsPMMsQN013nL8XaEGFb2hYuBA3+qI/EsD9jofD6loa QLyWMbQpAKb3Mrq+YpPqJVYCwXvytwRLfDmhwElzHAaQT1NVEsv6ZfdtVRNqb2fxi8IN CpC5DgboSNFlS+CJdTBs7mLDZp1SCy5Fd3jEZkXFwE3uLiP4ZLWQ3Dvo6EuP/9FJvNsh rqt68wwOISSdTdbG1R1V8exEwplBm+C/TeQjRCMrd9WPDQRX7BWQhXA5GIJQJCWWrAC8 qPmgfOmlHV5OATfIqJMn8nEB/64SLW3daRJnT6J1ksw/Uhx1WKHET4EmoB/eKoAeJBBb qL2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689708; x=1709294508; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BF2shshDxKBtCT7/MXQqYBl25A0ZFLNv5UbtYZIopXs=; b=M6qHkUyltr00+UWiDoYqZOawSouMoENjSNoRlKlp4h+YQpLh98kkkJc/NQQO+GMP63 4Dmc8e0Qg0CduujF1C1l+tzE4mUvjLckrNcgaIs3EEhQ2OiIVweDtQS2X4WyIw8uHUpI mIHZaTdSH/Zsl3MlsFtHtg4h1WyDERdL5ij5KN1mAM6QeWjzIBVmcvVTndhbGX63ne/X lqj8hAGysIL7Z4sNmkCZMKHw015WwVKNqCT4JhIYOlPVn8PO6BMzoLbMW7nATkGX+OSl YJNCYusGAQpCGHKXiN0CyL5MvMDcxL8tA/NEADZFbLDrV997YQHz5APXgsQU93B2G1qg cFAw== X-Gm-Message-State: AOJu0Yz0GJ9kMAnLffCUKLvGppnIQTVo1BsW+mLT+XBYPhRl4pNzKJfs juSy9SbHUJ9yvbrbyAkD932N7j9MHoUCDbPub72IiD/jfr9mqgKWMR+8GbnD X-Google-Smtp-Source: AGHT+IF8URTG1yxsIMHIDsxF+SA17Ko2xt/b0WrhgdjsTP40HIyw5Um2UQoLbCMtyLE8/myyf7CWEQ== X-Received: by 2002:a17:906:c354:b0:a3f:adad:9ce1 with SMTP id ci20-20020a170906c35400b00a3fadad9ce1mr2119516ejb.18.1708689707167; Fri, 23 Feb 2024 04:01:47 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:46 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 09/10] selftest/sstatetests: move exception list for cache objects to the parent class Date: Fri, 23 Feb 2024 13:01:33 +0100 Message-Id: <20240223120134.3713127-9-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196069 This is useful not just in cdn/local cache tests, but also in the build bundle test, and so should be applied by default. Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/sstatetests.py | 26 ++++++++++----------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 8d41ead8abf..bb2a1a2fb8c 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -891,13 +891,24 @@ INHERIT += "base-do-configure-modified" expected_sametmp_output, expected_difftmp_output) class SStateCheckObjectPresence(SStateBase): - def check_bb_output(self, output, exceptions, check_cdn): + def check_bb_output(self, output, targets, exceptions, check_cdn): def is_exception(object, exceptions): for e in exceptions: if re.search(e, object): return True return False + # sstate is checked for existence of these, but they never get written out to begin with + exceptions += ["{}.*image_qa".format(t) for t in targets.split()] + exceptions += ["{}.*deploy_source_date_epoch".format(t) for t in targets.split()] + exceptions += ["{}.*image_complete".format(t) for t in targets.split()] + exceptions += ["linux-yocto.*shared_workdir"] + # these get influnced by IMAGE_FSTYPES tweaks in yocto-autobuilder-helper's config.json (on x86-64) + # additionally, they depend on noexec (thus, absent stamps) package, install, etc. image tasks, + # which makes tracing other changes difficult + exceptions += ["{}.*create_spdx".format(t) for t in targets.split()] + exceptions += ["{}.*create_runtime_spdx".format(t) for t in targets.split()] + output_l = output.splitlines() for l in output_l: if l.startswith("Sstate summary"): @@ -935,17 +946,6 @@ class SStateCheckObjectPresence(SStateBase): @OETestTag("yocto-mirrors") class SStateMirrors(SStateCheckObjectPresence): def run_test(self, machine, targets, exceptions, check_cdn = True): - # sstate is checked for existence of these, but they never get written out to begin with - exceptions += ["{}.*image_qa".format(t) for t in targets.split()] - exceptions += ["{}.*deploy_source_date_epoch".format(t) for t in targets.split()] - exceptions += ["{}.*image_complete".format(t) for t in targets.split()] - exceptions += ["linux-yocto.*shared_workdir"] - # these get influnced by IMAGE_FSTYPES tweaks in yocto-autobuilder-helper's config.json (on x86-64) - # additionally, they depend on noexec (thus, absent stamps) package, install, etc. image tasks, - # which makes tracing other changes difficult - exceptions += ["{}.*create_spdx".format(t) for t in targets.split()] - exceptions += ["{}.*create_runtime_spdx".format(t) for t in targets.split()] - if check_cdn: self.config_sstate(True) self.append_config(""" @@ -959,7 +959,7 @@ MACHINE = "{}" """.format(machine)) result = bitbake("-DD -n {}".format(targets)) bitbake("-S none {}".format(targets)) - self.check_bb_output(result.output, exceptions, check_cdn) + self.check_bb_output(result.output, targets, exceptions, check_cdn) def test_cdn_mirror_qemux86_64(self): exceptions = [] From patchwork Fri Feb 23 12:01:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 39979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEFA5C54E50 for ; Fri, 23 Feb 2024 12:01:56 +0000 (UTC) Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by mx.groups.io with SMTP id smtpd.web11.8985.1708689710244015321 for ; Fri, 23 Feb 2024 04:01:50 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=iPZqK5l9; spf=pass (domain: gmail.com, ip: 209.85.218.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a3e706f50beso94807266b.0 for ; Fri, 23 Feb 2024 04:01:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708689709; x=1709294509; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2LO02WLe4Qp122fJFeCxZrC8MHySBci5Vsgg3LB+XIQ=; b=iPZqK5l9k8N8HfrfXHm7Pn08c6Vx9BlZw4G+F1ewwq4UsQa37TRrqw19Iyk3AM8oL2 Pq3G+dH44r2BbxA8t/2d+i5pQ/3ClceOylVWPDl9yP4vH2xmWOpJeMKSJ/yXG598TrTP w35ImJb4UFsKLQ9mH+qYvambi56BQPs/6pNnHDR3u6rZ9OvGMb6VwKRCqm7rZvPWE+BZ 7mrwmiJb94Wv+FCVm2QibcKBVXWBja13mj1y6jHYR7KcvSnj5M4DUCHqRpjEVYjUXokw d23FMMd/4JKnV7IMDlHfENzZ9N8iIO4M+udPpC0u/bjANALK1J/h1XS0yJXTuTImisi6 mI8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708689709; x=1709294509; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2LO02WLe4Qp122fJFeCxZrC8MHySBci5Vsgg3LB+XIQ=; b=gHbA0OkQkGTIKgWToAtHX/2URB2y3C8XBqTqDJH1oZp81jF2umwYXjA0j9GhC9FSAe 6M41PGQeDUR6OOe9VsjhhMMyfaAUY49ZsHk/hfTGBozdYCKzLfzyb6DQK943wOSk1WG+ BgQRz+l4MckK5wrh66GrpOgFj5rdahErKb89fp2Rolm7A6zTl6P5GEIHrsP+xz/VHI76 pa+iud7fP/4kxbh/mr9rWKW8/eRvJkUbeAHUqgO12tU8J5w0qaltYIIgO2MAhawEEOm3 eXEhob4G6Xko25o4Vbabm+qMPzhS1WOec1XRrBfkUzjPHz14NHrD+ZewkalSwFDfpqrW CyWQ== X-Gm-Message-State: AOJu0YzthlzBO4GsbRT8K7oTZfM7RtR9RFd4FYJf0Z3kPVwPIjFlCDI3 HDcyL7r94J/k4X/DNrkNsU0E9kzAU+NRwl3mnSbAdpwTfNSCFwRO0EF5epRf X-Google-Smtp-Source: AGHT+IEbD5XCG+0tBDadfmUvSjK2aARD5W2wpZfD3IoP8i11ixK/nZstvuuNtpVlvZvMqyKUjIBPcw== X-Received: by 2002:a17:906:7192:b0:a3e:a951:4087 with SMTP id h18-20020a170906719200b00a3ea9514087mr1100764ejk.76.1708689708620; Fri, 23 Feb 2024 04:01:48 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id d17-20020a170906345100b00a3d125b9c0asm220330ejb.81.2024.02.23.04.01.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 04:01:48 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 10/10] scripts/oe-replicate-build: add a script that packages and replicates a yocto build elsewhere Date: Fri, 23 Feb 2024 13:01:34 +0100 Message-Id: <20240223120134.3713127-10-alex@linutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240223120134.3713127-1-alex@linutronix.de> References: <20240223120134.3713127-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 23 Feb 2024 12:01:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/196070 This scripts takes an existing, active yocto build, and puts pieces of it into a bundle directory, which is then placed in a tarball, and that goes into a self-extracting shell archive. This allows moving the bundle to another machine, or placing it on a network server for downloads. It's like esdk bundles, except inside there is a regular, plain, direct yocto build. The bundle includes: - copies of all layers used in a build, with their git histories - build configuration as a template in a special layer private to the bundle - (this is the best bit) all the sstate needed for the specified target, but *only* that sstate and not a complete copy of the local cache. When someone runs the self-extracting shell archive, everything should 'just work': the layers are restored, and the build set up to use them, the bundled sstate cache, and the template configuration. Alternatively, it's possible to not include the cache, and presumably it will come from a configured sstate mirror. This enables at least two*scratch*three interesting use cases: - moving builds around in single-step fashion: run the script, and give the artifact to someone else via file sharing or usb sticks. - publishing builds on the network, perhaps right next to the sstate cache that can fulfil them. This allows an alternative, much smoother entry point into 'doing yocto', as this guarantees availability of needed cache items, and so builds will be a lot faster than the typical 'compile everything from scratch' yocto experience. Sstate is guaranteed to be available, as there's a record of needed items in each bundle, and that record can be used to prevent pruning of still-needed cache items. - this, in turn, can enable 'binary Yocto distro' implemented with sstate, if there's a higher level tool that creates bundles in a systematic, structured fashion, and another tool that iterates over available bundles, and lets users pick them for building locally. Signed-off-by: Alexander Kanavin --- meta/files/bundle-shar-extract.sh | 55 +++++++++++++++ meta/lib/oeqa/selftest/cases/sstatetests.py | 13 ++++ scripts/oe-replicate-build | 75 +++++++++++++++++++++ 3 files changed, 143 insertions(+) create mode 100644 meta/files/bundle-shar-extract.sh create mode 100755 scripts/oe-replicate-build diff --git a/meta/files/bundle-shar-extract.sh b/meta/files/bundle-shar-extract.sh new file mode 100644 index 00000000000..ae10dbaf065 --- /dev/null +++ b/meta/files/bundle-shar-extract.sh @@ -0,0 +1,55 @@ +#!/bin/bash + +verbose=0 +listcontents=0 +prepare_buildsystem="yes" +while getopts "Dln" OPT; do + case $OPT in + D) + verbose=1 + ;; + l) + listcontents=1 + ;; + n) + prepare_buildsystem="no" + ;; + *) + echo "Usage: $(basename "$0") [-D] [-l] [-n]" + echo " -D Use set -x to see what is going on" + echo " -l list files that will be extracted" + echo " -n Extract files but do not prepare the build system" + exit 1 + ;; + esac +done + +if [ $verbose = 1 ] ; then + set -x +fi + +payload_offset=$(($(grep -na -m1 "^MARKER:$" "$0"|cut -d':' -f1) + 1)) + +if [ "$listcontents" = "1" ] ; then + tail -n +$payload_offset "$0"| tar tv || exit 1 + exit +fi + + +tail -n +$payload_offset "$0"| tar mx --zstd --checkpoint=.2500 || exit 1 + +if [ $prepare_buildsystem = "no" ] ; then + exit +fi + +target_dir=$(basename "$0" .sh) +pushd $target_dir +layers/setup-build setup -c build-config-default -b build --no-shell > setup-build.log +popd + +echo "Each time you wish to use this build in a new shell session, you need to source the environment setup script:" +echo " \$ . $target_dir/build/init-build-env" + +exit 0 + +MARKER: diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index bb2a1a2fb8c..d5e9ed099f8 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -976,3 +976,16 @@ MACHINE = "{}" def test_local_cache_qemuarm64(self): exceptions = [] self.run_test("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions, check_cdn = False) + +class SStateBundles(SStateCheckObjectPresence): + def test_minimal_bundle(self): + targets = "core-image-minimal" + machine = get_bb_var('MACHINE') + bitbake("--runall build {}".format(targets)) + runCmd("oe-replicate-build --targets {}".format(targets)) + extractedbundledir = tempfile.mkdtemp(prefix='bundle-extracted-', dir=self.topdir) + runCmd("../build-bundle.sh", cwd=extractedbundledir) + result = runCmd(". build-bundle/build/init-build-env && MACHINE={} bitbake -DD -n {}".format(machine,targets), cwd=extractedbundledir, shell=True, executable='/bin/bash') + + exceptions = [] + self.check_bb_output(result.output, targets, exceptions, check_cdn=False) diff --git a/scripts/oe-replicate-build b/scripts/oe-replicate-build new file mode 100755 index 00000000000..bb465f0372d --- /dev/null +++ b/scripts/oe-replicate-build @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import argparse +import json +import os +import subprocess +import shutil + +def _do_bundle(args): + bundledir = args.output_prefix or "build-bundle" + print("Making a self-extracting bundle archive in {}.sh ...".format(bundledir)) + os.mkdir(bundledir) + + subprocess.check_output("bitbake -S outdir={} -S lockedsigs {}".format(os.path.abspath(bundledir),args.targets), shell=True) + sstate_dir = subprocess.check_output("bitbake-getvar --value SSTATE_DIR", shell=True).decode().strip() + nativelsbstring = subprocess.check_output("bitbake-getvar --value NATIVELSBSTRING", shell=True).decode().strip() + if not args.no_sstate: + subprocess.check_output("gen-lockedsig-cache {} {} {} {}".format("locked-sigs.inc", sstate_dir, "build/sstate-cache", nativelsbstring), shell=True, cwd=bundledir) + cachedir = bundledir + "/build/cache" + os.mkdir(cachedir) + subprocess.check_output("bitbake -S outdir={} -S unihash-cache-copy {}".format(os.path.abspath(cachedir),args.targets), shell=True) + + layerdir = "meta-build-config" + subprocess.check_output("bitbake-layers create-layer --add-layer {}".format(layerdir), shell=True, cwd=bundledir) + subprocess.check_output("bitbake-layers save-build-conf {} default".format(layerdir), shell=True, cwd=bundledir) + shutil.copy(os.path.join(os.environ["BUILDDIR"],'conf',"conf-summary.txt"), os.path.join(bundledir,layerdir,'conf/templates/default')) + shutil.copy(os.path.join(os.environ["BUILDDIR"],'conf',"conf-notes.txt"), os.path.join(bundledir,layerdir,'conf/templates/default')) + subprocess.check_output("bitbake-layers remove-layer {}".format(layerdir), shell=True, cwd=bundledir) + + # meta-build-config is then in bblayers.conf.sample, and should be removed from it as it wasn't in the actual build + bblayers = os.path.join(bundledir, layerdir, 'conf/templates/default/bblayers.conf.sample') + with open(bblayers) as f: + lines = f.readlines() + lines = [l for l in lines if os.path.join(bundledir, layerdir) not in l] + with open(bblayers,'w') as f: + f.write(''.join(lines)) + + subprocess.check_output("bitbake-layers create-layers-setup --writer oe-local-copy {}".format(bundledir), shell=True) + + # meta-build-config should however be present in .oe-layers.json, as otherwise oe-setup-build won't be able to discover + # the config template in it + oelayers = os.path.join(bundledir, 'layers', '.oe-layers.json') + with open(oelayers) as f: + json_f = json.load(f) + json_f["layers"].append("../meta-build-config") + with open(oelayers,'w') as f: + json.dump(json_f, f, sort_keys=True, indent=4) + + subprocess.check_output("tar caf {}.tar.zst {}".format(bundledir, bundledir), shell=True) + + corebase = subprocess.check_output("bitbake-getvar --value COREBASE", shell=True).decode().strip() + subprocess.check_output("cp {}/meta/files/bundle-shar-extract.sh {}.sh".format(corebase, bundledir), shell=True) + subprocess.check_output("cat {}.tar.zst >> {}.sh".format(bundledir, bundledir), shell=True) + subprocess.check_output("chmod +x {}.sh".format(bundledir), shell=True) + + if not args.keep_tmp: + shutil.rmtree(bundledir) + os.remove("{}.tar.zst".format(bundledir)) + +parser = argparse.ArgumentParser(description="A script that bundles up everything needed to replicate a yocto build elsewhere (including appropriate portions of sstate cache) into a self-contained shell archive.") + +parser.add_argument('--targets', required=True, help="Bitbake targets that the bundle should be made for.") +parser.add_argument('--output-prefix', help='File name prefix for the output files, if the default (build-bundle) is undesirable.') +parser.add_argument('--no-sstate', action='store_true', help='Do not include sstate cache into the bundle.') +parser.add_argument('--keep-tmp', action='store_true', help='Keep intermediate output: unpacked bundle directory, and compressed tarball (in addition to the final self-extracting shell archuve.') + +args = parser.parse_args() + +_do_bundle(args) +