From patchwork Mon Jan 22 13:58:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E006C47DD9 for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by mx.groups.io with SMTP id smtpd.web10.74257.1705931963388870001 for ; Mon, 22 Jan 2024 05:59:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=NUUHrx9h; spf=pass (domain: gmail.com, ip: 209.85.218.50, mailfrom: adrian.freihofer@gmail.com) Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a2d7e2e7fe0so541156366b.1 for ; Mon, 22 Jan 2024 05:59:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931961; x=1706536761; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3HlAeRAPp3I4urAm+keorks1nYc9ZvWV4HesjIU5poc=; b=NUUHrx9hquYXB3XKoiw2ceFttIe5/0FDMHPTqjwvgAUxV3X26Uifqzt7lIDF7ZCUxc WxQmiKPsYTFzDQXUXAQRkQmOCVBKIIOjdHWxBoNGGD9LCLVlr4CgKYeH82m9OFJjUojE QeQZ3MvBuQ0VPvha5rL2nif5oP32w+GyAhTcZdtuOP8eHCEsBAFSMeUeG6MCVQi5lQME Z0bEN6dwRHRil3ypn1m6LMv6IpBLZ0HzLHrJD/w1kPuJ60GD6eiahxPwyzoXeqEpKI2Z kuJuD1Yx0x1QP4Mijuk8s2hZbjD8+oHjPSJ1kQsxAdiibOQlp5ulNQVO8ZZIRdjDwSzR Zbqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931961; x=1706536761; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3HlAeRAPp3I4urAm+keorks1nYc9ZvWV4HesjIU5poc=; b=Qb6GXatypPcJ9jhCoG6HePlbDBJFZOuamWN7V32pNPYTZ5G1qwcIX7wUTR65SeRzQO NsLWlTHkE4OcMG/by36T0nkr0DlqY9e2qe/F3iPPVQ2q16VCaOZ7CJFKAUYTJYnkEkf1 DCnsjhJX7YcOU/9q3yRJ2cpt24Rm1Axb/Z/u5siDRLZz/Vbo4TdFCCH0xDqwcZiYET6b DiFFxwIiE7gFA2UTvsms/DOvjhRrc7LpSONR0z23GWkAtW1rKVMjfPvRU6tuZzoO/0yz GI2RnsUMxpJ/5Vx6RhNom+tyOa1xDtyhlTenP1CcLxlCnmo+HqGkMlMWiAmmabz6rNDp zgIQ== X-Gm-Message-State: AOJu0YyKqvV8fwkpNnjhPmh3X9BYA/5n8H5ZU/RAu2zwd+vEUi6Oe2Zu 8Iou1Z3HVwM/buudFAlaWf3AbV06z0HS0YKttlcylloRYGB7ofTSPmT2xZc7 X-Google-Smtp-Source: AGHT+IHOcwMAFiZiGTs6J9JWrZlihXoXOCqc3nrWsbvNOhJGysArflgE8Z6b+/rlNyGKGBgB+I92Gw== X-Received: by 2002:a17:906:3283:b0:a2f:1546:aadc with SMTP id 3-20020a170906328300b00a2f1546aadcmr4884694ejw.20.1705931961382; Mon, 22 Jan 2024 05:59:21 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:21 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 1/9] cmake.bbclass: use --install Date: Mon, 22 Jan 2024 14:58:19 +0100 Message-ID: <20240122135901.171596-2-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194150 Since version 3.15 CMake provides a command-line signature to install an already-generated project binary tree. This may be used after building a project to run installation without using the generated build system or the native build tool. This is a small improvement, for regular bitbake calls. CMake does not check the dependencies again which is expected to be faster. The main motivation for this change is using CMake from an SDK context. With this change it is possible to initiate the compile step from an IDE and later on initiating the install step via bitbake which runs the install step on pseudo. This is also what the meson.bbclass already does with the --no-rebuild option. Signed-off-by: Adrian Freihofer --- meta/classes-recipe/cmake.bbclass | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/cmake.bbclass b/meta/classes-recipe/cmake.bbclass index 1e353f660f7..3d3781ef333 100644 --- a/meta/classes-recipe/cmake.bbclass +++ b/meta/classes-recipe/cmake.bbclass @@ -224,12 +224,24 @@ cmake_runcmake_build() { eval ${DESTDIR:+DESTDIR=${DESTDIR} }${CMAKE_VERBOSE} cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD} } +# Install an already-generated project binary tree. Not checking the compile +# dependencies again is particularly important for SDK use cases. +cmake_runcmake_install() { + bbnote ${DESTDIR:+DESTDIR=${DESTDIR} }${CMAKE_VERBOSE} cmake --install '${B}' + eval ${DESTDIR:+DESTDIR=${DESTDIR} }${CMAKE_VERBOSE} cmake --install '${B}' +} + cmake_do_compile() { cmake_runcmake_build --target ${OECMAKE_TARGET_COMPILE} } cmake_do_install() { - DESTDIR='${D}' cmake_runcmake_build --target ${OECMAKE_TARGET_INSTALL} + if [ "${OECMAKE_TARGET_INSTALL}" = "install" ]; then + DESTDIR='${D}' cmake_runcmake_install + else + # Legacy path which supports also custom install targets + DESTDIR='${D}' cmake_runcmake_build --target ${OECMAKE_TARGET_INSTALL} + fi } EXPORT_FUNCTIONS do_configure do_compile do_install do_generate_toolchain_file From patchwork Mon Jan 22 13:58:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78E04C48260 for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) by mx.groups.io with SMTP id smtpd.web10.74258.1705931964159893909 for ; Mon, 22 Jan 2024 05:59:24 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=LjTcY6dT; spf=pass (domain: gmail.com, ip: 209.85.218.45, mailfrom: adrian.freihofer@gmail.com) Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-a26ed1e05c7so345295566b.2 for ; Mon, 22 Jan 2024 05:59:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931962; x=1706536762; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R52fLyrsmSD3iAQqdDTFrY+4XPuWr/xbBfQ6SmMMg+o=; b=LjTcY6dTjjpd/1Yx5UYhNTWi/F0TurN1PfdcHKIur389KZFkDYPHimyHKvx6NHlW1f zQvXu0q4YWvRQAi6tyASGdIPUYPfxVyYGHdOWHIcRYed/3cWxyJf6rpGDN8mEr1R8F68 WErC2f4wE5F3OvT+/ciUI6YSG4Id9N+V7ZxfN3K+eT72z6/VVA50vzksSX4sXelbQx+V fmxYcs7vtck7woBmAyLFHJl7EzCxRcDjl5jYt8M34wZBYY9IUcKoXLmeDAW/C+16iilG innTsggM/Yop0JrvWGqavOzLe1UrhgiVvVXqXYbahXVYXfwxTkeyVBsowiaKU5+ullJ7 h9QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931962; x=1706536762; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R52fLyrsmSD3iAQqdDTFrY+4XPuWr/xbBfQ6SmMMg+o=; b=owa5YKL3+WfZBvUMhkPf01lgO4SX+Y4v9lIAXzeufd7YxyRRHFSzHWv6QknDu5Fy9y qjQMXUJyYfEf4u46sNy7pkF6Ob3xMU3WpnpU8xTm+g2shVDU6HTfzeWAFXE/GMw+Oj/w OiT8WMGasTPVEcXL6zP3gkKCeHz3sd9MqnAw7WVzsWX/akCqAX6KTtD9OgbB8CPJcKxu jEfnI4yKGgqHbMpaOGQcJ5nVT0760caejwFk9vCkC3D8GMVzA2DTAz/+U4jAFn/MXNXg 9RmNMDH9grNlIlFKCmksSB+Jidqi+tYQjfR+OlSnQL5GfEkXl/BPKcU+4hn3nzaTCxZ+ y1oA== X-Gm-Message-State: AOJu0YxGhc0EyBtK5n5E18v6YKUGS0PjQ/3jCIustfxXY3rRg9/t80Ga OQIsxMxepZfcvGTRd3qo/rbufAlQoW7JjMHAkKrzPayLlvH5wHQ6oSHaGibP X-Google-Smtp-Source: AGHT+IHYJLWu0a4xERQe979U56uVjgC2tl2C2OVGW+Caexamv92hX6/ldHYuWCN5AtsCWqrfOeuzVw== X-Received: by 2002:a17:906:3b44:b0:a2c:29ee:fb7d with SMTP id h4-20020a1709063b4400b00a2c29eefb7dmr1355844ejf.58.1705931962017; Mon, 22 Jan 2024 05:59:22 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:21 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 2/9] devtool: support plugins with plugins Date: Mon, 22 Jan 2024 14:58:20 +0100 Message-ID: <20240122135901.171596-3-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194151 Pass the plugins search path to plugins via context. This allows plugins to search for their own plugins at the same paths. Signed-off-by: Adrian Freihofer --- scripts/devtool | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/devtool b/scripts/devtool index 3aae7b93b4b..60ea3e82985 100755 --- a/scripts/devtool +++ b/scripts/devtool @@ -299,8 +299,9 @@ def main(): return 2 # Search BBPATH first to allow layers to override plugins in scripts_path - for path in global_args.bbpath.split(':') + [scripts_path]: - pluginpath = os.path.join(path, 'lib', 'devtool') + pluginpaths = [os.path.join(path, 'lib', 'devtool') for path in global_args.bbpath.split(':') + [scripts_path]] + context.pluginpaths = pluginpaths + for pluginpath in pluginpaths: scriptutils.load_plugins(logger, plugins, pluginpath) subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='') From patchwork Mon Jan 22 13:58:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 688A0C47E49 for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) by mx.groups.io with SMTP id smtpd.web10.74259.1705931965920853794 for ; Mon, 22 Jan 2024 05:59:26 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=EpWDc70H; spf=pass (domain: gmail.com, ip: 209.85.167.49, mailfrom: adrian.freihofer@gmail.com) Received: by mail-lf1-f49.google.com with SMTP id 2adb3069b0e04-50ea9daac4cso3245078e87.3 for ; Mon, 22 Jan 2024 05:59:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931964; x=1706536764; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V7XTgybMWftqedAh5FAuFqlAgW8OImiM4FYuZj/PsUI=; b=EpWDc70Hbhzxk+4EJX2ohzxSS/qeYPE7BMHhwE1aeKYavubmrI53dmzYkt3OMg3Yxo eUC58gJpNSDnmIIIBcNp7ZZ5b24hU+wkNb7LOx9joU5Wh/fHEtmHsDo1D4Yxs87cCPLH DPc00TDOHZe8h56+LoIHShvqgh/Jds3OBhxFB+bOxTMrlg4Cwu+TEmU/Fbl6DsXdoLa6 PK4LmthVUCo4086C8jBvxMLEocdhkksp7QqI+IHsL5OEoBZQhlXpFrqDYSaAFG6/5zvY YNEf/2MndSFsfcG146l66mYKACRmKc0TUEmhjdR7XdCxuqXg5n9z2u9wPjWBsQMc554a /OKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931964; x=1706536764; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V7XTgybMWftqedAh5FAuFqlAgW8OImiM4FYuZj/PsUI=; b=RLzBoWFJw5Z4fSwJXU/jtDpxl2Og5pzYtcz1d2jXoEpFTksJZJyZb2yw7p9fNNxAvw oykR5OPkAZqT0pCtHVrnHlWmrAm/Se7tVaCkgENDL3YkAqs2+S3VGj4SN2lstuBlYkQR UArC2/nVF6OJLEI0WTotve6gppeP3Dgb7yZz+ifCLzw7/zX+aq60xi9aqmdtVTwCFOHP 8OkcmsS03RR4sFlYTvLT7AHjVBGHzFa5orwWRXCvyZf3xljo2r7VIvUytdbc0AUM9MPB aaZ/JrnGjL3gk7gaMjLCFmHjBEhUMO+IZpKcwrM2xmOQpdiF2iVgBFbXbqVoQfk3pwJl 7Zog== X-Gm-Message-State: AOJu0YwMAnP5F+xqyf3rqOBwguz8yQcEpRvKzKVb3uIsbPxOFTyQQVn6 HvS4VV57LTpiMtskwYGNnImxq+koaxtx/kmaIk0K5TlW8LfjUd1WLU6th/18 X-Google-Smtp-Source: AGHT+IEpNjltTHuXHqgQSkIg7KOmada0NXm2aeyp1reNxQzRuI8QyXuYBEBELPoweb6SgEo0JqM6aw== X-Received: by 2002:a05:6512:3095:b0:50e:74f0:810d with SMTP id z21-20020a056512309500b0050e74f0810dmr1919151lfd.101.1705931963058; Mon, 22 Jan 2024 05:59:23 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:22 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 3/9] devtool: new ide-sdk plugin Date: Mon, 22 Jan 2024 14:58:21 +0100 Message-ID: <20240122135901.171596-4-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194153 The new devtool ide plugin provides the eSDK and configures an IDE to work with the eSDK. In doing so, bitbake should be used to generate the IDE configuration and update the SDK, but it should no longer play a role when working on the source code. The work on the source code should take place exclusively with the IDE, which, for example, calls cmake directly to compile the code and execute the unit tests from the IDE. The plugin works for recipes inheriting the cmake or the meson bbclass. Support for more programming languages and build tools may be added in the future. There are various IDEs that can be used for the development of embedded Linux applications. Therefore, devtool ide-sdk, like devtool itself, supports plugins to support IDEs. VSCode is the default IDE for this first implementation. Additionally, some generic helper scripts can be generated with --ide none instead of a specific IDE configuration. This can be used for any IDE that supports calling some scripts. There are two different modes supported: - devtool modify mode (default): devtool ide-sdk configures the IDE to manage the build-tool used by the recipe (e.g. cmake or meson). The workflow looks like: $ devtool modify a-recipe $ devtool ide-sdk a-recipe a-image $ code "$BUILDDIR/workspace/sources/a-recipe" Work in VSCode, after installing the proposed plugins Deploying the artifacts to the target device and running a remote debugging session is supported as well. This first implementation still calls bitbake and devtool to copy the binary artifacts to the target device. In contrast to compiling, installation and copying must be performed with the file rights of the target device. The pseudo tool must be used for this. Therefore bitbake -c install a-recipe && devtool deploy-target a-recipe are called by the IDE for the deployment. This might be improved later on. Executing the unit tests out of the IDE is supported via Qemu user if the build tool supports that. CMake (if cmake-qemu.bbclass is inherited) and Meson support Qemu usermode. - Shared sysroots mode: bootstraps the eSDK with shared sysroots for all the recipes passed to devtool ide-sdk. This is basically a wrapper for bitbake meta-ide-support && bitbake build-sysroots. The workflow looks like: $ devtool ide-sdk --share-sysroots a-recipe another-recipe vscode where/the/sources/are If the IDE and the build tool support it, the IDE gets configured to offer the cross tool-chain provided by the eSDK. In case of VSCode and cmake a cmake-kit is generated. This offers to use the cross tool-chain from the UI of the IDE. Many thanks to Enguerrand de Ribaucourt for testing and bug fixing. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/ide_plugins/__init__.py | 267 +++++ scripts/lib/devtool/ide_plugins/ide_code.py | 438 ++++++++ scripts/lib/devtool/ide_plugins/ide_none.py | 53 + scripts/lib/devtool/ide_sdk.py | 1032 +++++++++++++++++++ 4 files changed, 1790 insertions(+) create mode 100644 scripts/lib/devtool/ide_plugins/__init__.py create mode 100644 scripts/lib/devtool/ide_plugins/ide_code.py create mode 100644 scripts/lib/devtool/ide_plugins/ide_none.py create mode 100755 scripts/lib/devtool/ide_sdk.py diff --git a/scripts/lib/devtool/ide_plugins/__init__.py b/scripts/lib/devtool/ide_plugins/__init__.py new file mode 100644 index 00000000000..3371b242640 --- /dev/null +++ b/scripts/lib/devtool/ide_plugins/__init__.py @@ -0,0 +1,267 @@ +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk IDE plugin interface definition and helper functions""" + +import errno +import json +import logging +import os +import stat +from enum import Enum, auto +from devtool import DevtoolError +from bb.utils import mkdirhier + +logger = logging.getLogger('devtool') + + +class BuildTool(Enum): + UNDEFINED = auto() + CMAKE = auto() + MESON = auto() + + @property + def is_c_ccp(self): + if self is BuildTool.CMAKE: + return True + if self is BuildTool.MESON: + return True + return False + + +class GdbCrossConfig: + """Base class defining the GDB configuration generator interface + + Generate a GDB configuration for a binary on the target device. + Only one instance per binary is allowed. This allows to assign unique port + numbers for all gdbserver instances. + """ + _gdbserver_port_next = 1234 + _binaries = [] + + def __init__(self, image_recipe, modified_recipe, binary, gdbserver_multi=True): + self.image_recipe = image_recipe + self.modified_recipe = modified_recipe + self.gdb_cross = modified_recipe.gdb_cross + self.binary = binary + if binary in GdbCrossConfig._binaries: + raise DevtoolError( + "gdbserver config for binary %s is already generated" % binary) + GdbCrossConfig._binaries.append(binary) + self.script_dir = modified_recipe.ide_sdk_scripts_dir + self.gdbinit_dir = os.path.join(self.script_dir, 'gdbinit') + self.gdbserver_multi = gdbserver_multi + self.binary_pretty = self.binary.replace(os.sep, '-').lstrip('-') + self.gdbserver_port = GdbCrossConfig._gdbserver_port_next + GdbCrossConfig._gdbserver_port_next += 1 + self.id_pretty = "%d_%s" % (self.gdbserver_port, self.binary_pretty) + # gdbserver start script + gdbserver_script_file = 'gdbserver_' + self.id_pretty + if self.gdbserver_multi: + gdbserver_script_file += "_m" + self.gdbserver_script = os.path.join( + self.script_dir, gdbserver_script_file) + # gdbinit file + self.gdbinit = os.path.join( + self.gdbinit_dir, 'gdbinit_' + self.id_pretty) + # gdb start script + self.gdb_script = os.path.join( + self.script_dir, 'gdb_' + self.id_pretty) + + def _gen_gdbserver_start_script(self): + """Generate a shell command starting the gdbserver on the remote device via ssh + + GDB supports two modes: + multi: gdbserver remains running over several debug sessions + once: gdbserver terminates after the debugged process terminates + """ + cmd_lines = ['#!/bin/sh'] + if self.gdbserver_multi: + temp_dir = "TEMP_DIR=/tmp/gdbserver_%s; " % self.id_pretty + gdbserver_cmd_start = temp_dir + gdbserver_cmd_start += "test -f \$TEMP_DIR/pid && exit 0; " + gdbserver_cmd_start += "mkdir -p \$TEMP_DIR; " + gdbserver_cmd_start += "%s --multi :%s > \$TEMP_DIR/log 2>&1 & " % ( + self.gdb_cross.gdbserver_path, self.gdbserver_port) + gdbserver_cmd_start += "echo \$! > \$TEMP_DIR/pid;" + + gdbserver_cmd_stop = temp_dir + gdbserver_cmd_stop += "test -f \$TEMP_DIR/pid && kill \$(cat \$TEMP_DIR/pid); " + gdbserver_cmd_stop += "rm -rf \$TEMP_DIR; " + + gdbserver_cmd_l = [] + gdbserver_cmd_l.append('if [ "$1" = "stop" ]; then') + gdbserver_cmd_l.append(' shift') + gdbserver_cmd_l.append(" %s %s %s %s 'sh -c \"%s\"'" % ( + self.gdb_cross.target_device.ssh_sshexec, self.gdb_cross.target_device.ssh_port, self.gdb_cross.target_device.extraoptions, self.gdb_cross.target_device.target, gdbserver_cmd_stop)) + gdbserver_cmd_l.append('else') + gdbserver_cmd_l.append(" %s %s %s %s 'sh -c \"%s\"'" % ( + self.gdb_cross.target_device.ssh_sshexec, self.gdb_cross.target_device.ssh_port, self.gdb_cross.target_device.extraoptions, self.gdb_cross.target_device.target, gdbserver_cmd_start)) + gdbserver_cmd_l.append('fi') + gdbserver_cmd = os.linesep.join(gdbserver_cmd_l) + else: + gdbserver_cmd_start = "%s --once :%s %s" % ( + self.gdb_cross.gdbserver_path, self.gdbserver_port, self.binary) + gdbserver_cmd = "%s %s %s %s 'sh -c \"%s\"'" % ( + self.gdb_cross.target_device.ssh_sshexec, self.gdb_cross.target_device.ssh_port, self.gdb_cross.target_device.extraoptions, self.gdb_cross.target_device.target, gdbserver_cmd_start) + cmd_lines.append(gdbserver_cmd) + GdbCrossConfig.write_file(self.gdbserver_script, cmd_lines, True) + + def _gen_gdbinit_config(self): + """Generate a gdbinit file for this binary and the corresponding gdbserver configuration""" + gdbinit_lines = ['# This file is generated by devtool ide-sdk'] + if self.gdbserver_multi: + target_help = '# gdbserver --multi :%d' % self.gdbserver_port + remote_cmd = 'target extended-remote' + else: + target_help = '# gdbserver :%d %s' % ( + self.gdbserver_port, self.binary) + remote_cmd = 'target remote' + gdbinit_lines.append('# On the remote target:') + gdbinit_lines.append(target_help) + gdbinit_lines.append('# On the build machine:') + gdbinit_lines.append('# cd ' + self.modified_recipe.real_srctree) + gdbinit_lines.append( + '# ' + self.gdb_cross.gdb + ' -ix ' + self.gdbinit) + + gdbinit_lines.append('set sysroot ' + self.modified_recipe.d) + gdbinit_lines.append('set substitute-path "/usr/include" "' + + os.path.join(self.modified_recipe.recipe_sysroot, 'usr', 'include') + '"') + # Disable debuginfod for now, the IDE configuration uses rootfs-dbg from the image workdir. + gdbinit_lines.append('set debuginfod enabled off') + if self.image_recipe.rootfs_dbg: + gdbinit_lines.append( + 'set solib-search-path "' + self.modified_recipe.solib_search_path_str(self.image_recipe) + '"') + gdbinit_lines.append('set substitute-path "/usr/src/debug" "' + os.path.join( + self.image_recipe.rootfs_dbg, 'usr', 'src', 'debug') + '"') + gdbinit_lines.append( + '%s %s:%d' % (remote_cmd, self.gdb_cross.host, self.gdbserver_port)) + gdbinit_lines.append('set remote exec-file ' + self.binary) + gdbinit_lines.append( + 'run ' + os.path.join(self.modified_recipe.d, self.binary)) + + GdbCrossConfig.write_file(self.gdbinit, gdbinit_lines) + + def _gen_gdb_start_script(self): + """Generate a script starting GDB with the corresponding gdbinit configuration.""" + cmd_lines = ['#!/bin/sh'] + cmd_lines.append('cd ' + self.modified_recipe.real_srctree) + cmd_lines.append(self.gdb_cross.gdb + ' -ix ' + + self.gdbinit + ' "$@"') + GdbCrossConfig.write_file(self.gdb_script, cmd_lines, True) + + def initialize(self): + self._gen_gdbserver_start_script() + self._gen_gdbinit_config() + self._gen_gdb_start_script() + + @staticmethod + def write_file(script_file, cmd_lines, executable=False): + script_dir = os.path.dirname(script_file) + mkdirhier(script_dir) + with open(script_file, 'w') as script_f: + script_f.write(os.linesep.join(cmd_lines)) + script_f.write(os.linesep) + if executable: + st = os.stat(script_file) + os.chmod(script_file, st.st_mode | stat.S_IEXEC) + logger.info("Created: %s" % script_file) + + +class IdeBase: + """Base class defining the interface for IDE plugins""" + + def __init__(self): + self.ide_name = 'undefined' + self.gdb_cross_configs = [] + + @classmethod + def ide_plugin_priority(cls): + """Used to find the default ide handler if --ide is not passed""" + return 10 + + def setup_shared_sysroots(self, shared_env): + logger.warn("Shared sysroot mode is not supported for IDE %s" % + self.ide_name) + + def setup_modified_recipe(self, args, image_recipe, modified_recipe): + logger.warn("Modified recipe mode is not supported for IDE %s" % + self.ide_name) + + def initialize_gdb_cross_configs(self, image_recipe, modified_recipe, gdb_cross_config_class=GdbCrossConfig): + binaries = modified_recipe.find_installed_binaries() + for binary in binaries: + gdb_cross_config = gdb_cross_config_class( + image_recipe, modified_recipe, binary) + gdb_cross_config.initialize() + self.gdb_cross_configs.append(gdb_cross_config) + + @staticmethod + def gen_oe_scrtips_sym_link(modified_recipe): + # create a sym-link from sources to the scripts directory + if os.path.isdir(modified_recipe.ide_sdk_scripts_dir): + IdeBase.symlink_force(modified_recipe.ide_sdk_scripts_dir, + os.path.join(modified_recipe.real_srctree, 'oe-scripts')) + + @staticmethod + def update_json_file(json_dir, json_file, update_dict): + """Update a json file + + By default it uses the dict.update function. If this is not sutiable + the update function might be passed via update_func parameter. + """ + json_path = os.path.join(json_dir, json_file) + logger.info("Updating IDE config file: %s (%s)" % + (json_file, json_path)) + if not os.path.exists(json_dir): + os.makedirs(json_dir) + try: + with open(json_path) as f: + orig_dict = json.load(f) + except json.decoder.JSONDecodeError: + logger.info( + "Decoding %s failed. Probably because of comments in the json file" % json_path) + orig_dict = {} + except FileNotFoundError: + orig_dict = {} + orig_dict.update(update_dict) + with open(json_path, 'w') as f: + json.dump(orig_dict, f, indent=4) + + @staticmethod + def symlink_force(tgt, dst): + try: + os.symlink(tgt, dst) + except OSError as err: + if err.errno == errno.EEXIST: + if os.readlink(dst) != tgt: + os.remove(dst) + os.symlink(tgt, dst) + else: + raise err + + +def get_devtool_deploy_opts(args): + """Filter args for devtool deploy-target args""" + if not args.target: + return None + devtool_deploy_opts = [args.target] + if args.no_host_check: + devtool_deploy_opts += ["-c"] + if args.show_status: + devtool_deploy_opts += ["-s"] + if args.no_preserve: + devtool_deploy_opts += ["-p"] + if args.no_check_space: + devtool_deploy_opts += ["--no-check-space"] + if args.ssh_exec: + devtool_deploy_opts += ["-e", args.ssh.exec] + if args.port: + devtool_deploy_opts += ["-P", args.port] + if args.key: + devtool_deploy_opts += ["-I", args.key] + if args.strip is False: + devtool_deploy_opts += ["--no-strip"] + return devtool_deploy_opts diff --git a/scripts/lib/devtool/ide_plugins/ide_code.py b/scripts/lib/devtool/ide_plugins/ide_code.py new file mode 100644 index 00000000000..b2193130d2e --- /dev/null +++ b/scripts/lib/devtool/ide_plugins/ide_code.py @@ -0,0 +1,438 @@ +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk IDE plugin for VSCode and VSCodium""" + +import json +import logging +import os +import shutil +from devtool.ide_plugins import BuildTool, IdeBase, GdbCrossConfig, get_devtool_deploy_opts + +logger = logging.getLogger('devtool') + + +class GdbCrossConfigVSCode(GdbCrossConfig): + def __init__(self, image_recipe, modified_recipe, binary): + super().__init__(image_recipe, modified_recipe, binary, False) + + def initialize(self): + self._gen_gdbserver_start_script() + + +class IdeVSCode(IdeBase): + """Manage IDE configurations for VSCode + + Modified recipe mode: + - cmake: use the cmake-preset generated by devtool ide-sdk + - meson: meson is called via a wrapper script generated by devtool ide-sdk + + Shared sysroot mode: + In shared sysroot mode, the cross tool-chain is exported to the user's global configuration. + A workspace cannot be created because there is no recipe that defines how a workspace could + be set up. + - cmake: adds a cmake-kit to .local/share/CMakeTools/cmake-tools-kits.json + The cmake-kit uses the environment script and the tool-chain file + generated by meta-ide-support. + - meson: Meson needs manual workspace configuration. + """ + + @classmethod + def ide_plugin_priority(cls): + """If --ide is not passed this is the default plugin""" + if shutil.which('code'): + return 100 + return 0 + + def setup_shared_sysroots(self, shared_env): + """Expose the toolchain of the shared sysroots SDK""" + datadir = shared_env.ide_support.datadir + deploy_dir_image = shared_env.ide_support.deploy_dir_image + real_multimach_target_sys = shared_env.ide_support.real_multimach_target_sys + standalone_sysroot_native = shared_env.build_sysroots.standalone_sysroot_native + vscode_ws_path = os.path.join( + os.environ['HOME'], '.local', 'share', 'CMakeTools') + cmake_kits_path = os.path.join(vscode_ws_path, 'cmake-tools-kits.json') + oecmake_generator = "Ninja" + env_script = os.path.join( + deploy_dir_image, 'environment-setup-' + real_multimach_target_sys) + + if not os.path.isdir(vscode_ws_path): + os.makedirs(vscode_ws_path) + cmake_kits_old = [] + if os.path.exists(cmake_kits_path): + with open(cmake_kits_path, 'r', encoding='utf-8') as cmake_kits_file: + cmake_kits_old = json.load(cmake_kits_file) + cmake_kits = cmake_kits_old.copy() + + cmake_kit_new = { + "name": "OE " + real_multimach_target_sys, + "environmentSetupScript": env_script, + "toolchainFile": standalone_sysroot_native + datadir + "/cmake/OEToolchainConfig.cmake", + "preferredGenerator": { + "name": oecmake_generator + } + } + + def merge_kit(cmake_kits, cmake_kit_new): + i = 0 + while i < len(cmake_kits): + if 'environmentSetupScript' in cmake_kits[i] and \ + cmake_kits[i]['environmentSetupScript'] == cmake_kit_new['environmentSetupScript']: + cmake_kits[i] = cmake_kit_new + return + i += 1 + cmake_kits.append(cmake_kit_new) + merge_kit(cmake_kits, cmake_kit_new) + + if cmake_kits != cmake_kits_old: + logger.info("Updating: %s" % cmake_kits_path) + with open(cmake_kits_path, 'w', encoding='utf-8') as cmake_kits_file: + json.dump(cmake_kits, cmake_kits_file, indent=4) + else: + logger.info("Already up to date: %s" % cmake_kits_path) + + cmake_native = os.path.join( + shared_env.build_sysroots.standalone_sysroot_native, 'usr', 'bin', 'cmake') + if os.path.isfile(cmake_native): + logger.info('cmake-kits call cmake by default. If the cmake provided by this SDK should be used, please add the following line to ".vscode/settings.json" file: "cmake.cmakePath": "%s"' % cmake_native) + else: + logger.error("Cannot find cmake native at: %s" % cmake_native) + + def dot_code_dir(self, modified_recipe): + return os.path.join(modified_recipe.srctree, '.vscode') + + def __vscode_settings_meson(self, settings_dict, modified_recipe): + if modified_recipe.build_tool is not BuildTool.MESON: + return + settings_dict["mesonbuild.mesonPath"] = modified_recipe.meson_wrapper + + confopts = modified_recipe.mesonopts.split() + confopts += modified_recipe.meson_cross_file.split() + confopts += modified_recipe.extra_oemeson.split() + settings_dict["mesonbuild.configureOptions"] = confopts + settings_dict["mesonbuild.buildFolder"] = modified_recipe.b + + def __vscode_settings_cmake(self, settings_dict, modified_recipe): + """Add cmake specific settings to settings.json. + + Note: most settings are passed to the cmake preset. + """ + if modified_recipe.build_tool is not BuildTool.CMAKE: + return + settings_dict["cmake.configureOnOpen"] = True + settings_dict["cmake.sourceDirectory"] = modified_recipe.real_srctree + + def vscode_settings(self, modified_recipe): + files_excludes = { + "**/.git/**": True, + "**/oe-logs/**": True, + "**/oe-workdir/**": True, + "**/source-date-epoch/**": True + } + python_exclude = [ + "**/.git/**", + "**/oe-logs/**", + "**/oe-workdir/**", + "**/source-date-epoch/**" + ] + settings_dict = { + "files.watcherExclude": files_excludes, + "files.exclude": files_excludes, + "python.analysis.exclude": python_exclude + } + self.__vscode_settings_cmake(settings_dict, modified_recipe) + self.__vscode_settings_meson(settings_dict, modified_recipe) + + settings_file = 'settings.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), settings_file, settings_dict) + + def __vscode_extensions_cmake(self, modified_recipe, recommendations): + if modified_recipe.build_tool is not BuildTool.CMAKE: + return + recommendations += [ + "twxs.cmake", + "ms-vscode.cmake-tools", + "ms-vscode.cpptools", + "ms-vscode.cpptools-extension-pack", + "ms-vscode.cpptools-themes" + ] + + def __vscode_extensions_meson(self, modified_recipe, recommendations): + if modified_recipe.build_tool is not BuildTool.MESON: + return + recommendations += [ + 'mesonbuild.mesonbuild', + "ms-vscode.cpptools", + "ms-vscode.cpptools-extension-pack", + "ms-vscode.cpptools-themes" + ] + + def vscode_extensions(self, modified_recipe): + recommendations = [] + self.__vscode_extensions_cmake(modified_recipe, recommendations) + self.__vscode_extensions_meson(modified_recipe, recommendations) + extensions_file = 'extensions.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), extensions_file, {"recommendations": recommendations}) + + def vscode_c_cpp_properties(self, modified_recipe): + properties_dict = { + "name": modified_recipe.recipe_id_pretty, + } + if modified_recipe.build_tool is BuildTool.CMAKE: + properties_dict["configurationProvider"] = "ms-vscode.cmake-tools" + elif modified_recipe.build_tool is BuildTool.MESON: + properties_dict["configurationProvider"] = "mesonbuild.mesonbuild" + else: # no C/C++ build + return + + properties_dicts = { + "configurations": [ + properties_dict + ], + "version": 4 + } + prop_file = 'c_cpp_properties.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), prop_file, properties_dicts) + + def vscode_launch_bin_dbg(self, gdb_cross_config): + modified_recipe = gdb_cross_config.modified_recipe + + launch_config = { + "name": gdb_cross_config.id_pretty, + "type": "cppdbg", + "request": "launch", + "program": os.path.join(modified_recipe.d, gdb_cross_config.binary.lstrip('/')), + "stopAtEntry": True, + "cwd": "${workspaceFolder}", + "environment": [], + "externalConsole": False, + "MIMode": "gdb", + "preLaunchTask": gdb_cross_config.id_pretty, + "miDebuggerPath": modified_recipe.gdb_cross.gdb, + "miDebuggerServerAddress": "%s:%d" % (modified_recipe.gdb_cross.host, gdb_cross_config.gdbserver_port) + } + + # Search for header files in recipe-sysroot. + src_file_map = { + "/usr/include": os.path.join(modified_recipe.recipe_sysroot, "usr", "include") + } + # First of all search for not stripped binaries in the image folder. + # These binaries are copied (and optionally stripped) by deploy-target + setup_commands = [ + { + "description": "sysroot", + "text": "set sysroot " + modified_recipe.d + } + ] + + if gdb_cross_config.image_recipe.rootfs_dbg: + launch_config['additionalSOLibSearchPath'] = modified_recipe.solib_search_path_str( + gdb_cross_config.image_recipe) + src_file_map["/usr/src/debug"] = os.path.join( + gdb_cross_config.image_recipe.rootfs_dbg, "usr", "src", "debug") + else: + logger.warning( + "Cannot setup debug symbols configuration for GDB. IMAGE_GEN_DEBUGFS is not enabled.") + + launch_config['sourceFileMap'] = src_file_map + launch_config['setupCommands'] = setup_commands + return launch_config + + def vscode_launch(self, modified_recipe): + """GDB Launch configuration for binaries (elf files)""" + + configurations = [self.vscode_launch_bin_dbg( + gdb_cross_config) for gdb_cross_config in self.gdb_cross_configs] + launch_dict = { + "version": "0.2.0", + "configurations": configurations + } + launch_file = 'launch.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), launch_file, launch_dict) + + def vscode_tasks_cpp(self, args, modified_recipe): + run_install_deploy = modified_recipe.gen_install_deploy_script(args) + install_task_name = "install && deploy-target %s" % modified_recipe.recipe_id_pretty + tasks_dict = { + "version": "2.0.0", + "tasks": [ + { + "label": install_task_name, + "type": "shell", + "command": run_install_deploy, + "problemMatcher": [] + } + ] + } + for gdb_cross_config in self.gdb_cross_configs: + tasks_dict['tasks'].append( + { + "label": gdb_cross_config.id_pretty, + "type": "shell", + "isBackground": True, + "dependsOn": [ + install_task_name + ], + "command": gdb_cross_config.gdbserver_script, + "problemMatcher": [ + { + "pattern": [ + { + "regexp": ".", + "file": 1, + "location": 2, + "message": 3 + } + ], + "background": { + "activeOnStart": True, + "beginsPattern": ".", + "endsPattern": ".", + } + } + ] + }) + tasks_file = 'tasks.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), tasks_file, tasks_dict) + + def vscode_tasks_fallback(self, args, modified_recipe): + oe_init_dir = modified_recipe.oe_init_dir + oe_init = ". %s %s > /dev/null && " % (modified_recipe.oe_init_build_env, modified_recipe.topdir) + dt_build = "devtool build " + dt_build_label = dt_build + modified_recipe.recipe_id_pretty + dt_build_cmd = dt_build + modified_recipe.bpn + clean_opt = " --clean" + dt_build_clean_label = dt_build + modified_recipe.recipe_id_pretty + clean_opt + dt_build_clean_cmd = dt_build + modified_recipe.bpn + clean_opt + dt_deploy = "devtool deploy-target " + dt_deploy_label = dt_deploy + modified_recipe.recipe_id_pretty + dt_deploy_cmd = dt_deploy + modified_recipe.bpn + dt_build_deploy_label = "devtool build & deploy-target %s" % modified_recipe.recipe_id_pretty + deploy_opts = ' '.join(get_devtool_deploy_opts(args)) + tasks_dict = { + "version": "2.0.0", + "tasks": [ + { + "label": dt_build_label, + "type": "shell", + "command": "bash", + "linux": { + "options": { + "cwd": oe_init_dir + } + }, + "args": [ + "--login", + "-c", + "%s%s" % (oe_init, dt_build_cmd) + ], + "problemMatcher": [] + }, + { + "label": dt_deploy_label, + "type": "shell", + "command": "bash", + "linux": { + "options": { + "cwd": oe_init_dir + } + }, + "args": [ + "--login", + "-c", + "%s%s %s" % ( + oe_init, dt_deploy_cmd, deploy_opts) + ], + "problemMatcher": [] + }, + { + "label": dt_build_deploy_label, + "dependsOrder": "sequence", + "dependsOn": [ + dt_build_label, + dt_deploy_label + ], + "problemMatcher": [], + "group": { + "kind": "build", + "isDefault": True + } + }, + { + "label": dt_build_clean_label, + "type": "shell", + "command": "bash", + "linux": { + "options": { + "cwd": oe_init_dir + } + }, + "args": [ + "--login", + "-c", + "%s%s" % (oe_init, dt_build_clean_cmd) + ], + "problemMatcher": [] + } + ] + } + if modified_recipe.gdb_cross: + for gdb_cross_config in self.gdb_cross_configs: + tasks_dict['tasks'].append( + { + "label": gdb_cross_config.id_pretty, + "type": "shell", + "isBackground": True, + "dependsOn": [ + dt_build_deploy_label + ], + "command": gdb_cross_config.gdbserver_script, + "problemMatcher": [ + { + "pattern": [ + { + "regexp": ".", + "file": 1, + "location": 2, + "message": 3 + } + ], + "background": { + "activeOnStart": True, + "beginsPattern": ".", + "endsPattern": ".", + } + } + ] + }) + tasks_file = 'tasks.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), tasks_file, tasks_dict) + + def vscode_tasks(self, args, modified_recipe): + if modified_recipe.build_tool.is_c_ccp: + self.vscode_tasks_cpp(args, modified_recipe) + else: + self.vscode_tasks_fallback(args, modified_recipe) + + def setup_modified_recipe(self, args, image_recipe, modified_recipe): + self.vscode_settings(modified_recipe) + self.vscode_extensions(modified_recipe) + self.vscode_c_cpp_properties(modified_recipe) + if args.target: + self.initialize_gdb_cross_configs( + image_recipe, modified_recipe, gdb_cross_config_class=GdbCrossConfigVSCode) + self.vscode_launch(modified_recipe) + self.vscode_tasks(args, modified_recipe) + + +def register_ide_plugin(ide_plugins): + ide_plugins['code'] = IdeVSCode diff --git a/scripts/lib/devtool/ide_plugins/ide_none.py b/scripts/lib/devtool/ide_plugins/ide_none.py new file mode 100644 index 00000000000..f106c5a0269 --- /dev/null +++ b/scripts/lib/devtool/ide_plugins/ide_none.py @@ -0,0 +1,53 @@ +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk generic IDE plugin""" + +import os +import logging +from devtool.ide_plugins import IdeBase, GdbCrossConfig + +logger = logging.getLogger('devtool') + + +class IdeNone(IdeBase): + """Generate some generic helpers for other IDEs + + Modified recipe mode: + Generate some helper scripts for remote debugging with GDB + + Shared sysroot mode: + A wrapper for bitbake meta-ide-support and bitbake build-sysroots + """ + + def __init__(self): + super().__init__() + + def setup_shared_sysroots(self, shared_env): + real_multimach_target_sys = shared_env.ide_support.real_multimach_target_sys + deploy_dir_image = shared_env.ide_support.deploy_dir_image + env_script = os.path.join( + deploy_dir_image, 'environment-setup-' + real_multimach_target_sys) + logger.info( + "To use this SDK please source this: %s" % env_script) + + def setup_modified_recipe(self, args, image_recipe, modified_recipe): + """generate some helper scripts and config files + + - Execute the do_install task + - Execute devtool deploy-target + - Generate a gdbinit file per executable + - Generate the oe-scripts sym-link + """ + script_path = modified_recipe.gen_install_deploy_script(args) + logger.info("Created: %s" % script_path) + + self.initialize_gdb_cross_configs(image_recipe, modified_recipe) + + IdeBase.gen_oe_scrtips_sym_link(modified_recipe) + + +def register_ide_plugin(ide_plugins): + ide_plugins['none'] = IdeNone diff --git a/scripts/lib/devtool/ide_sdk.py b/scripts/lib/devtool/ide_sdk.py new file mode 100755 index 00000000000..27389026be3 --- /dev/null +++ b/scripts/lib/devtool/ide_sdk.py @@ -0,0 +1,1032 @@ +# Development tool - ide-sdk command plugin +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk plugin""" + +import json +import logging +import os +import re +import shutil +import stat +import subprocess +from argparse import RawTextHelpFormatter +from enum import Enum + +import scriptutils +import bb +from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError, parse_recipe +from devtool.standard import get_real_srctree +from devtool.ide_plugins import BuildTool, get_devtool_deploy_opts + + +logger = logging.getLogger('devtool') + +# dict of classes derived from IdeBase +ide_plugins = {} + + +class DevtoolIdeMode(Enum): + """Different modes are supported by the ide-sdk plugin. + + The enum might be extended by more advanced modes in the future. Some ideas: + - auto: modified if all recipes are modified, shared if none of the recipes is modified. + - mixed: modified mode for modified recipes, shared mode for all other recipes. + """ + + modified = 'modified' + shared = 'shared' + + +class TargetDevice: + """SSH remote login parameters""" + + def __init__(self, args): + self.extraoptions = '' + if args.no_host_check: + self.extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' + self.ssh_sshexec = 'ssh' + if args.ssh_exec: + self.ssh_sshexec = args.ssh_exec + self.ssh_port = '' + if args.port: + self.ssh_port = "-p %s" % args.port + if args.key: + self.extraoptions += ' -i %s' % args.key + + self.target = args.target + target_sp = args.target.split('@') + if len(target_sp) == 1: + self.login = "" + self.host = target_sp[0] + elif len(target_sp) == 2: + self.login = target_sp[0] + self.host = target_sp[1] + else: + logger.error("Invalid target argument: %s" % args.target) + + +class RecipeNative: + """Base class for calling bitbake to provide a -native recipe""" + + def __init__(self, name, target_arch=None): + self.name = name + self.target_arch = target_arch + self.bootstrap_tasks = [self.name + ':do_addto_recipe_sysroot'] + self.staging_bindir_native = None + self.target_sys = None + self.__native_bin = None + + def _initialize(self, config, workspace, tinfoil): + """Get the parsed recipe""" + recipe_d = parse_recipe( + config, tinfoil, self.name, appends=True, filter_workspace=False) + if not recipe_d: + raise DevtoolError("Parsing %s recipe failed" % self.name) + self.staging_bindir_native = os.path.realpath( + recipe_d.getVar('STAGING_BINDIR_NATIVE')) + self.target_sys = recipe_d.getVar('TARGET_SYS') + return recipe_d + + def initialize(self, config, workspace, tinfoil): + """Basic initialization that can be overridden by a derived class""" + self._initialize(config, workspace, tinfoil) + + @property + def native_bin(self): + if not self.__native_bin: + raise DevtoolError("native binary name is not defined.") + return self.__native_bin + + +class RecipeGdbCross(RecipeNative): + """Handle handle gdb-cross on the host and the gdbserver on the target device""" + + def __init__(self, args, target_arch, target_device): + super().__init__('gdb-cross-' + target_arch, target_arch) + self.target_device = target_device + self.gdb = None + self.gdbserver_port_next = int(args.gdbserver_port_start) + self.config_db = {} + + def __find_gdbserver(self, config, tinfoil): + """Absolute path of the gdbserver""" + recipe_d_gdb = parse_recipe( + config, tinfoil, 'gdb', appends=True, filter_workspace=False) + if not recipe_d_gdb: + raise DevtoolError("Parsing gdb recipe failed") + return os.path.join(recipe_d_gdb.getVar('bindir'), 'gdbserver') + + def initialize(self, config, workspace, tinfoil): + super()._initialize(config, workspace, tinfoil) + gdb_bin = self.target_sys + '-gdb' + gdb_path = os.path.join( + self.staging_bindir_native, self.target_sys, gdb_bin) + self.gdb = gdb_path + self.gdbserver_path = self.__find_gdbserver(config, tinfoil) + + @property + def host(self): + return self.target_device.host + + +class RecipeImage: + """Handle some image recipe related properties + + Most workflows require firmware that runs on the target device. + This firmware must be consistent with the setup of the host system. + In particular, the debug symbols must be compatible. For this, the + rootfs must be created as part of the SDK. + """ + + def __init__(self, name): + self.combine_dbg_image = False + self.gdbserver_missing = False + self.name = name + self.rootfs = None + self.__rootfs_dbg = None + self.bootstrap_tasks = [self.name + ':do_build'] + + def initialize(self, config, tinfoil): + image_d = parse_recipe( + config, tinfoil, self.name, appends=True, filter_workspace=False) + if not image_d: + raise DevtoolError( + "Parsing image recipe %s failed" % self.name) + + self.combine_dbg_image = bb.data.inherits_class( + 'image-combined-dbg', image_d) + + workdir = image_d.getVar('WORKDIR') + self.rootfs = os.path.join(workdir, 'rootfs') + if image_d.getVar('IMAGE_GEN_DEBUGFS') == "1": + self.__rootfs_dbg = os.path.join(workdir, 'rootfs-dbg') + + self.gdbserver_missing = 'gdbserver' not in image_d.getVar( + 'IMAGE_INSTALL') + + @property + def debug_support(self): + return bool(self.rootfs_dbg) + + @property + def rootfs_dbg(self): + if self.__rootfs_dbg and os.path.isdir(self.__rootfs_dbg): + return self.__rootfs_dbg + return None + + +class RecipeMetaIdeSupport: + """For the shared sysroots mode meta-ide-support is needed + + For use cases where just a cross tool-chain is required but + no recipe is used, devtool ide-sdk abstracts calling bitbake meta-ide-support + and bitbake build-sysroots. This also allows to expose the cross-toolchains + to IDEs. For example VSCode support different tool-chains with e.g. cmake-kits. + """ + + def __init__(self): + self.bootstrap_tasks = ['meta-ide-support:do_build'] + self.topdir = None + self.datadir = None + self.deploy_dir_image = None + self.build_sys = None + # From toolchain-scripts + self.real_multimach_target_sys = None + + def initialize(self, config, tinfoil): + meta_ide_support_d = parse_recipe( + config, tinfoil, 'meta-ide-support', appends=True, filter_workspace=False) + if not meta_ide_support_d: + raise DevtoolError("Parsing meta-ide-support recipe failed") + + self.topdir = meta_ide_support_d.getVar('TOPDIR') + self.datadir = meta_ide_support_d.getVar('datadir') + self.deploy_dir_image = meta_ide_support_d.getVar( + 'DEPLOY_DIR_IMAGE') + self.build_sys = meta_ide_support_d.getVar('BUILD_SYS') + self.real_multimach_target_sys = meta_ide_support_d.getVar( + 'REAL_MULTIMACH_TARGET_SYS') + + +class RecipeBuildSysroots: + """For the shared sysroots mode build-sysroots is needed""" + + def __init__(self): + self.standalone_sysroot = None + self.standalone_sysroot_native = None + self.bootstrap_tasks = [ + 'build-sysroots:do_build_target_sysroot', + 'build-sysroots:do_build_native_sysroot' + ] + + def initialize(self, config, tinfoil): + build_sysroots_d = parse_recipe( + config, tinfoil, 'build-sysroots', appends=True, filter_workspace=False) + if not build_sysroots_d: + raise DevtoolError("Parsing build-sysroots recipe failed") + self.standalone_sysroot = build_sysroots_d.getVar( + 'STANDALONE_SYSROOT') + self.standalone_sysroot_native = build_sysroots_d.getVar( + 'STANDALONE_SYSROOT_NATIVE') + + +class SharedSysrootsEnv: + """Handle the shared sysroots based workflow + + Support the workflow with just a tool-chain without a recipe. + It's basically like: + bitbake some-dependencies + bitbake meta-ide-support + bitbake build-sysroots + Use the environment-* file found in the deploy folder + """ + + def __init__(self): + self.ide_support = None + self.build_sysroots = None + + def initialize(self, ide_support, build_sysroots): + self.ide_support = ide_support + self.build_sysroots = build_sysroots + + def setup_ide(self, ide): + ide.setup(self) + + +class RecipeNotModified: + """Handling of recipes added to the Direct DSK shared sysroots.""" + + def __init__(self, name): + self.name = name + self.bootstrap_tasks = [name + ':do_populate_sysroot'] + + +class RecipeModified: + """Handling of recipes in the workspace created by devtool modify""" + OE_INIT_BUILD_ENV = 'oe-init-build-env' + + VALID_BASH_ENV_NAME_CHARS = re.compile(r"^[a-zA-Z0-9_]*$") + + def __init__(self, name): + self.name = name + self.bootstrap_tasks = [name + ':do_install'] + self.gdb_cross = None + # workspace + self.real_srctree = None + self.srctree = None + self.ide_sdk_dir = None + self.ide_sdk_scripts_dir = None + self.bbappend = None + # recipe variables from d.getVar + self.b = None + self.base_libdir = None + self.bblayers = None + self.bpn = None + self.d = None + self.fakerootcmd = None + self.fakerootenv = None + self.libdir = None + self.max_process = None + self.package_arch = None + self.package_debug_split_style = None + self.path = None + self.pn = None + self.recipe_sysroot = None + self.recipe_sysroot_native = None + self.staging_incdir = None + self.strip_cmd = None + self.target_arch = None + self.topdir = None + self.workdir = None + self.recipe_id = None + # replicate bitbake build environment + self.exported_vars = None + self.cmd_compile = None + self.__oe_init_dir = None + # main build tool used by this recipe + self.build_tool = BuildTool.UNDEFINED + # build_tool = cmake + self.oecmake_generator = None + self.cmake_cache_vars = None + # build_tool = meson + self.meson_buildtype = None + self.meson_wrapper = None + self.mesonopts = None + self.extra_oemeson = None + self.meson_cross_file = None + + def initialize(self, config, workspace, tinfoil): + recipe_d = parse_recipe( + config, tinfoil, self.name, appends=True, filter_workspace=False) + if not recipe_d: + raise DevtoolError("Parsing %s recipe failed" % self.name) + + # Verify this recipe is built as externalsrc setup by devtool modify + workspacepn = check_workspace_recipe( + workspace, self.name, bbclassextend=True) + self.srctree = workspace[workspacepn]['srctree'] + # Need to grab this here in case the source is within a subdirectory + self.real_srctree = get_real_srctree( + self.srctree, recipe_d.getVar('S'), recipe_d.getVar('WORKDIR')) + self.bbappend = workspace[workspacepn]['bbappend'] + + self.ide_sdk_dir = os.path.join( + config.workspace_path, 'ide-sdk', self.name) + if os.path.exists(self.ide_sdk_dir): + shutil.rmtree(self.ide_sdk_dir) + self.ide_sdk_scripts_dir = os.path.join(self.ide_sdk_dir, 'scripts') + + self.b = recipe_d.getVar('B') + self.base_libdir = recipe_d.getVar('base_libdir') + self.bblayers = recipe_d.getVar('BBLAYERS').split() + self.bpn = recipe_d.getVar('BPN') + self.d = recipe_d.getVar('D') + self.fakerootcmd = recipe_d.getVar('FAKEROOTCMD') + self.fakerootenv = recipe_d.getVar('FAKEROOTENV') + self.libdir = recipe_d.getVar('libdir') + self.max_process = int(recipe_d.getVar( + "BB_NUMBER_THREADS") or os.cpu_count() or 1) + self.package_arch = recipe_d.getVar('PACKAGE_ARCH') + self.package_debug_split_style = recipe_d.getVar( + 'PACKAGE_DEBUG_SPLIT_STYLE') + self.path = recipe_d.getVar('PATH') + self.pn = recipe_d.getVar('PN') + self.recipe_sysroot = os.path.realpath( + recipe_d.getVar('RECIPE_SYSROOT')) + self.recipe_sysroot_native = os.path.realpath( + recipe_d.getVar('RECIPE_SYSROOT_NATIVE')) + self.staging_incdir = os.path.realpath( + recipe_d.getVar('STAGING_INCDIR')) + self.strip_cmd = recipe_d.getVar('STRIP') + self.target_arch = recipe_d.getVar('TARGET_ARCH') + self.topdir = recipe_d.getVar('TOPDIR') + self.workdir = os.path.realpath(recipe_d.getVar('WORKDIR')) + + self.__init_exported_variables(recipe_d) + + if bb.data.inherits_class('cmake', recipe_d): + self.oecmake_generator = recipe_d.getVar('OECMAKE_GENERATOR') + self.__init_cmake_preset_cache(recipe_d) + self.build_tool = BuildTool.CMAKE + elif bb.data.inherits_class('meson', recipe_d): + self.meson_buildtype = recipe_d.getVar('MESON_BUILDTYPE') + self.mesonopts = recipe_d.getVar('MESONOPTS') + self.extra_oemeson = recipe_d.getVar('EXTRA_OEMESON') + self.meson_cross_file = recipe_d.getVar('MESON_CROSS_FILE') + self.build_tool = BuildTool.MESON + + # Recipe ID is the identifier for IDE config sections + self.recipe_id = self.bpn + "-" + self.package_arch + self.recipe_id_pretty = self.bpn + ": " + self.package_arch + + def append_to_bbappend(self, append_text): + with open(self.bbappend, 'a') as bbap: + bbap.write(append_text) + + def remove_from_bbappend(self, append_text): + with open(self.bbappend, 'r') as bbap: + text = bbap.read() + new_text = text.replace(append_text, '') + with open(self.bbappend, 'w') as bbap: + bbap.write(new_text) + + @staticmethod + def is_valid_shell_variable(var): + """Skip strange shell variables like systemd + + prevent from strange bugs because of strange variables which + are not used in this context but break various tools. + """ + if RecipeModified.VALID_BASH_ENV_NAME_CHARS.match(var): + bb.debug(1, "ignoring variable: %s" % var) + return True + return False + + def debug_build_config(self, args): + """Explicitely set for example CMAKE_BUILD_TYPE to Debug if not defined otherwise""" + if self.build_tool is BuildTool.CMAKE: + append_text = os.linesep + \ + 'OECMAKE_ARGS:append = " -DCMAKE_BUILD_TYPE:STRING=Debug"' + os.linesep + if args.debug_build_config and not 'CMAKE_BUILD_TYPE' in self.cmake_cache_vars: + self.cmake_cache_vars['CMAKE_BUILD_TYPE'] = { + "type": "STRING", + "value": "Debug", + } + self.append_to_bbappend(append_text) + elif 'CMAKE_BUILD_TYPE' in self.cmake_cache_vars: + del self.cmake_cache_vars['CMAKE_BUILD_TYPE'] + self.remove_from_bbappend(append_text) + elif self.build_tool is BuildTool.MESON: + append_text = os.linesep + 'MESON_BUILDTYPE = "debug"' + os.linesep + if args.debug_build_config and self.meson_buildtype != "debug": + self.mesonopts.replace( + '--buildtype ' + self.meson_buildtype, '--buildtype debug') + self.append_to_bbappend(append_text) + elif self.meson_buildtype == "debug": + self.mesonopts.replace( + '--buildtype debug', '--buildtype plain') + self.remove_from_bbappend(append_text) + elif args.debug_build_config: + logger.warn( + "--debug-build-config is not implemented for this build tool yet.") + + def solib_search_path(self, image): + """Search for debug symbols in the rootfs and rootfs-dbg + + The debug symbols of shared libraries which are provided by other packages + are grabbed from the -dbg packages in the rootfs-dbg. + + But most cross debugging tools like gdb, perf, and systemtap need to find + executable/library first and through it debuglink note find corresponding + symbols file. Therefore the library paths from the rootfs are added as well. + + Note: For the devtool modified recipe compiled from the IDE, the debug + symbols are taken from the unstripped binaries in the image folder. + Also, devtool deploy-target takes the files from the image folder. + debug symbols in the image folder refer to the corresponding source files + with absolute paths of the build machine. Debug symbols found in the + rootfs-dbg are relocated and contain paths which refer to the source files + installed on the target device e.g. /usr/src/... + """ + base_libdir = self.base_libdir.lstrip('/') + libdir = self.libdir.lstrip('/') + so_paths = [ + # debug symbols for package_debug_split_style: debug-with-srcpkg or .debug + os.path.join(image.rootfs_dbg, base_libdir, ".debug"), + os.path.join(image.rootfs_dbg, libdir, ".debug"), + # debug symbols for package_debug_split_style: debug-file-directory + os.path.join(image.rootfs_dbg, "usr", "lib", "debug"), + + # The binaries are required as well, the debug packages are not enough + # With image-combined-dbg.bbclass the binaries are copied into rootfs-dbg + os.path.join(image.rootfs_dbg, base_libdir), + os.path.join(image.rootfs_dbg, libdir), + # Without image-combined-dbg.bbclass the binaries are only in rootfs. + # Note: Stepping into source files located in rootfs-dbg does not + # work without image-combined-dbg.bbclass yet. + os.path.join(image.rootfs, base_libdir), + os.path.join(image.rootfs, libdir) + ] + return so_paths + + def solib_search_path_str(self, image): + """Return a : separated list of paths usable by GDB's set solib-search-path""" + return ':'.join(self.solib_search_path(image)) + + def __init_exported_variables(self, d): + """Find all variables with export flag set. + + This allows to generate IDE configurations which compile with the same + environment as bitbake does. That's at least a reasonable default behavior. + """ + exported_vars = {} + + vars = (key for key in d.keys() if not key.startswith( + "__") and not d.getVarFlag(key, "func", False)) + for var in vars: + func = d.getVarFlag(var, "func", False) + if d.getVarFlag(var, 'python', False) and func: + continue + export = d.getVarFlag(var, "export", False) + unexport = d.getVarFlag(var, "unexport", False) + if not export and not unexport and not func: + continue + if unexport: + continue + + val = d.getVar(var) + if val is None: + continue + if set(var) & set("-.{}+"): + logger.warn( + "Warning: Found invalid character in variable name %s", str(var)) + continue + varExpanded = d.expand(var) + val = str(val) + + if not RecipeModified.is_valid_shell_variable(varExpanded): + continue + + if func: + code_line = "line: {0}, file: {1}\n".format( + d.getVarFlag(var, "lineno", False), + d.getVarFlag(var, "filename", False)) + val = val.rstrip('\n') + logger.warn("Warning: exported shell function %s() is not exported (%s)" % + (varExpanded, code_line)) + continue + + if export: + exported_vars[varExpanded] = val.strip() + continue + + self.exported_vars = exported_vars + + def __init_cmake_preset_cache(self, d): + """Get the arguments passed to cmake + + Replicate the cmake configure arguments with all details to + share on build folder between bitbake and SDK. + """ + site_file = os.path.join(self.workdir, 'site-file.cmake') + if os.path.exists(site_file): + print("Warning: site-file.cmake is not supported") + + cache_vars = {} + oecmake_args = d.getVar('OECMAKE_ARGS').split() + extra_oecmake = d.getVar('EXTRA_OECMAKE').split() + for param in oecmake_args + extra_oecmake: + d_pref = "-D" + if param.startswith(d_pref): + param = param[len(d_pref):] + else: + print("Error: expected a -D") + param_s = param.split('=', 1) + param_nt = param_s[0].split(':', 1) + + def handle_undefined_variable(var): + if var.startswith('${') and var.endswith('}'): + return '' + else: + return var + # Example: FOO=ON + if len(param_nt) == 1: + cache_vars[param_s[0]] = handle_undefined_variable(param_s[1]) + # Example: FOO:PATH=/tmp + elif len(param_nt) == 2: + cache_vars[param_nt[0]] = { + "type": param_nt[1], + "value": handle_undefined_variable(param_s[1]), + } + else: + print("Error: cannot parse %s" % param) + self.cmake_cache_vars = cache_vars + + def cmake_preset(self): + """Create a preset for cmake that mimics how bitbake calls cmake""" + toolchain_file = os.path.join(self.workdir, 'toolchain.cmake') + cmake_executable = os.path.join( + self.recipe_sysroot_native, 'usr', 'bin', 'cmake') + self.cmd_compile = cmake_executable + " --build --preset " + self.recipe_id + + preset_dict_configure = { + "name": self.recipe_id, + "displayName": self.recipe_id_pretty, + "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch), + "binaryDir": self.b, + "generator": self.oecmake_generator, + "toolchainFile": toolchain_file, + "cacheVariables": self.cmake_cache_vars, + "environment": self.exported_vars, + "cmakeExecutable": cmake_executable + } + + preset_dict_build = { + "name": self.recipe_id, + "displayName": self.recipe_id_pretty, + "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch), + "configurePreset": self.recipe_id, + "inheritConfigureEnvironment": True + } + + preset_dict_test = { + "name": self.recipe_id, + "displayName": self.recipe_id_pretty, + "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch), + "configurePreset": self.recipe_id, + "inheritConfigureEnvironment": True + } + + preset_dict = { + "version": 3, # cmake 3.21, backward compatible with kirkstone + "configurePresets": [preset_dict_configure], + "buildPresets": [preset_dict_build], + "testPresets": [preset_dict_test] + } + + # Finally write the json file + json_file = 'CMakeUserPresets.json' + json_path = os.path.join(self.real_srctree, json_file) + logger.info("Updating CMake preset: %s (%s)" % (json_file, json_path)) + if not os.path.exists(self.real_srctree): + os.makedirs(self.real_srctree) + try: + with open(json_path) as f: + orig_dict = json.load(f) + except json.decoder.JSONDecodeError: + logger.info( + "Decoding %s failed. Probably because of comments in the json file" % json_path) + orig_dict = {} + except FileNotFoundError: + orig_dict = {} + + # Add or update the presets for the recipe and keep other presets + for k, v in preset_dict.items(): + if isinstance(v, list): + update_preset = v[0] + preset_added = False + if k in orig_dict: + for index, orig_preset in enumerate(orig_dict[k]): + if 'name' in orig_preset: + if orig_preset['name'] == update_preset['name']: + logger.debug("Updating preset: %s" % + orig_preset['name']) + orig_dict[k][index] = update_preset + preset_added = True + break + else: + logger.debug("keeping preset: %s" % + orig_preset['name']) + else: + logger.warn("preset without a name found") + if not preset_added: + if not k in orig_dict: + orig_dict[k] = [] + orig_dict[k].append(update_preset) + logger.debug("Added preset: %s" % + update_preset['name']) + else: + orig_dict[k] = v + + with open(json_path, 'w') as f: + json.dump(orig_dict, f, indent=4) + + def gen_meson_wrapper(self): + """Generate a wrapper script to call meson with the cross environment""" + bb.utils.mkdirhier(self.ide_sdk_scripts_dir) + meson_wrapper = os.path.join(self.ide_sdk_scripts_dir, 'meson') + meson_real = os.path.join( + self.recipe_sysroot_native, 'usr', 'bin', 'meson.real') + with open(meson_wrapper, 'w') as mwrap: + mwrap.write("#!/bin/sh" + os.linesep) + for var, val in self.exported_vars.items(): + mwrap.write('export %s="%s"' % (var, val) + os.linesep) + mwrap.write("unset CC CXX CPP LD AR NM STRIP" + os.linesep) + private_temp = os.path.join(self.b, "meson-private", "tmp") + mwrap.write('mkdir -p "%s"' % private_temp + os.linesep) + mwrap.write('export TMPDIR="%s"' % private_temp + os.linesep) + mwrap.write('exec "%s" "$@"' % meson_real + os.linesep) + st = os.stat(meson_wrapper) + os.chmod(meson_wrapper, st.st_mode | stat.S_IEXEC) + self.meson_wrapper = meson_wrapper + self.cmd_compile = meson_wrapper + " compile -C " + self.b + + def which(self, executable): + bin_path = shutil.which(executable, path=self.path) + if not bin_path: + raise DevtoolError( + 'Cannot find %s. Probably the recipe %s is not built yet.' % (executable, self.bpn)) + return bin_path + + @staticmethod + def is_elf_file(file_path): + with open(file_path, "rb") as f: + data = f.read(4) + if data == b'\x7fELF': + return True + return False + + def find_installed_binaries(self): + """find all executable elf files in the image directory""" + binaries = [] + d_len = len(self.d) + re_so = re.compile('.*\.so[.0-9]*$') + for root, _, files in os.walk(self.d, followlinks=False): + for file in files: + if os.path.islink(file): + continue + if re_so.match(file): + continue + abs_name = os.path.join(root, file) + if os.access(abs_name, os.X_OK) and RecipeModified.is_elf_file(abs_name): + binaries.append(abs_name[d_len:]) + return sorted(binaries) + + def gen_delete_package_dirs(self): + """delete folders of package tasks + + This is a workaround for and issue with recipes having their sources + downloaded as file:// + This likely breaks pseudo like: + path mismatch [3 links]: ino 79147802 db + .../build/tmp/.../cmake-example/1.0/package/usr/src/debug/ + cmake-example/1.0-r0/oe-local-files/cpp-example-lib.cpp + .../build/workspace/sources/cmake-example/oe-local-files/cpp-example-lib.cpp + Since the files are anyway outdated lets deleted them (also from pseudo's db) to workaround this issue. + """ + cmd_lines = ['#!/bin/sh'] + + # Set up the appropriate environment + newenv = dict(os.environ) + for varvalue in self.fakerootenv.split(): + if '=' in varvalue: + splitval = varvalue.split('=', 1) + newenv[splitval[0]] = splitval[1] + + # Replicate the environment variables from bitbake + for var, val in newenv.items(): + if not RecipeModified.is_valid_shell_variable(var): + continue + cmd_lines.append('%s="%s"' % (var, val)) + cmd_lines.append('export %s' % var) + + # Delete the folders + pkg_dirs = ' '.join([os.path.join(self.workdir, d) for d in [ + "package", "packages-split", "pkgdata", "sstate-install-package", "debugsources.list", "*.spec"]]) + cmd = "%s rm -rf %s" % (self.fakerootcmd, pkg_dirs) + cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd)) + + return self.write_script(cmd_lines, 'delete_package_dirs') + + def gen_install_deploy_script(self, args): + """Generate a script which does install and deploy""" + cmd_lines = ['#!/bin/bash'] + + cmd_lines.append(self.gen_delete_package_dirs()) + + # . oe-init-build-env $BUILDDIR + # Note: Sourcing scripts with arguments requires bash + cmd_lines.append('cd "%s" || { echo "cd %s failed"; exit 1; }' % ( + self.oe_init_dir, self.oe_init_dir)) + cmd_lines.append('. "%s" "%s" || { echo ". %s %s failed"; exit 1; }' % ( + self.oe_init_build_env, self.topdir, self.oe_init_build_env, self.topdir)) + + # bitbake -c install + cmd_lines.append( + 'bitbake %s -c install --force || { echo "bitbake %s -c install --force failed"; exit 1; }' % (self.bpn, self.bpn)) + + # devtool deploy-target + deploy_opts = ' '.join(get_devtool_deploy_opts(args)) + cmd_lines.append("devtool deploy-target %s %s" % + (self.bpn, deploy_opts)) + return self.write_script(cmd_lines, 'install_and_deploy') + + def write_script(self, cmd_lines, script_name): + bb.utils.mkdirhier(self.ide_sdk_scripts_dir) + script_name_arch = script_name + '_' + self.recipe_id + script_file = os.path.join(self.ide_sdk_scripts_dir, script_name_arch) + with open(script_file, 'w') as script_f: + script_f.write(os.linesep.join(cmd_lines)) + st = os.stat(script_file) + os.chmod(script_file, st.st_mode | stat.S_IEXEC) + return script_file + + @property + def oe_init_build_env(self): + """Find the oe-init-build-env used for this setup""" + oe_init_dir = self.oe_init_dir + if oe_init_dir: + return os.path.join(oe_init_dir, RecipeModified.OE_INIT_BUILD_ENV) + return None + + @property + def oe_init_dir(self): + """Find the directory where the oe-init-build-env is located + + Assumption: There might be a layer with higher priority than poky + which provides to oe-init-build-env in the layer's toplevel folder. + """ + if not self.__oe_init_dir: + for layer in reversed(self.bblayers): + result = subprocess.run( + ['git', 'rev-parse', '--show-toplevel'], cwd=layer, capture_output=True) + if result.returncode == 0: + oe_init_dir = result.stdout.decode('utf-8').strip() + oe_init_path = os.path.join( + oe_init_dir, RecipeModified.OE_INIT_BUILD_ENV) + if os.path.exists(oe_init_path): + logger.debug("Using %s from: %s" % ( + RecipeModified.OE_INIT_BUILD_ENV, oe_init_path)) + self.__oe_init_dir = oe_init_dir + break + if not self.__oe_init_dir: + logger.error("Cannot find the bitbake top level folder") + return self.__oe_init_dir + + +def ide_setup(args, config, basepath, workspace): + """Generate the IDE configuration for the workspace""" + + # Explicitely passing some special recipes does not make sense + for recipe in args.recipenames: + if recipe in ['meta-ide-support', 'build-sysroots']: + raise DevtoolError("Invalid recipe: %s." % recipe) + + # Collect information about tasks which need to be bitbaked + bootstrap_tasks = [] + bootstrap_tasks_late = [] + tinfoil = setup_tinfoil(config_only=False, basepath=basepath) + try: + # define mode depending on recipes which need to be processed + recipes_image_names = [] + recipes_modified_names = [] + recipes_other_names = [] + for recipe in args.recipenames: + try: + check_workspace_recipe( + workspace, recipe, bbclassextend=True) + recipes_modified_names.append(recipe) + except DevtoolError: + recipe_d = parse_recipe( + config, tinfoil, recipe, appends=True, filter_workspace=False) + if not recipe_d: + raise DevtoolError("Parsing recipe %s failed" % recipe) + if bb.data.inherits_class('image', recipe_d): + recipes_image_names.append(recipe) + else: + recipes_other_names.append(recipe) + + invalid_params = False + if args.mode == DevtoolIdeMode.shared: + if len(recipes_modified_names): + logger.error("In shared sysroots mode modified recipes %s cannot be handled." % str( + recipes_modified_names)) + invalid_params = True + if args.mode == DevtoolIdeMode.modified: + if len(recipes_other_names): + logger.error("Only in shared sysroots mode not modified recipes %s can be handled." % str( + recipes_other_names)) + invalid_params = True + if len(recipes_image_names) != 1: + logger.error( + "One image recipe is required as the rootfs for the remote development.") + invalid_params = True + for modified_recipe_name in recipes_modified_names: + if modified_recipe_name.startswith('nativesdk-') or modified_recipe_name.endswith('-native'): + logger.error( + "Only cross compiled recipes are support. %s is not cross." % modified_recipe_name) + invalid_params = True + + if invalid_params: + raise DevtoolError("Invalid parameters are passed.") + + # For the shared sysroots mode, add all dependencies of all the images to the sysroots + # For the modified mode provide one rootfs and the corresponding debug symbols via rootfs-dbg + recipes_images = [] + for recipes_image_name in recipes_image_names: + logger.info("Using image: %s" % recipes_image_name) + recipe_image = RecipeImage(recipes_image_name) + recipe_image.initialize(config, tinfoil) + bootstrap_tasks += recipe_image.bootstrap_tasks + recipes_images.append(recipe_image) + + # Provide a Direct SDK with shared sysroots + recipes_not_modified = [] + if args.mode == DevtoolIdeMode.shared: + ide_support = RecipeMetaIdeSupport() + ide_support.initialize(config, tinfoil) + bootstrap_tasks += ide_support.bootstrap_tasks + + logger.info("Adding %s to the Direct SDK sysroots." % + str(recipes_other_names)) + for recipe_name in recipes_other_names: + recipe_not_modified = RecipeNotModified(recipe_name) + bootstrap_tasks += recipe_not_modified.bootstrap_tasks + recipes_not_modified.append(recipe_not_modified) + + build_sysroots = RecipeBuildSysroots() + build_sysroots.initialize(config, tinfoil) + bootstrap_tasks_late += build_sysroots.bootstrap_tasks + shared_env = SharedSysrootsEnv() + shared_env.initialize(ide_support, build_sysroots) + + recipes_modified = [] + if args.mode == DevtoolIdeMode.modified: + logger.info("Setting up workspaces for modified recipe: %s" % + str(recipes_modified_names)) + gdbs_cross = {} + for recipe_name in recipes_modified_names: + recipe_modified = RecipeModified(recipe_name) + recipe_modified.initialize(config, workspace, tinfoil) + bootstrap_tasks += recipe_modified.bootstrap_tasks + recipes_modified.append(recipe_modified) + + if recipe_modified.target_arch not in gdbs_cross: + target_device = TargetDevice(args) + gdb_cross = RecipeGdbCross( + args, recipe_modified.target_arch, target_device) + gdb_cross.initialize(config, workspace, tinfoil) + bootstrap_tasks += gdb_cross.bootstrap_tasks + gdbs_cross[recipe_modified.target_arch] = gdb_cross + recipe_modified.gdb_cross = gdbs_cross[recipe_modified.target_arch] + + finally: + tinfoil.shutdown() + + if not args.skip_bitbake: + bb_cmd = 'bitbake ' + if args.bitbake_k: + bb_cmd += "-k " + bb_cmd_early = bb_cmd + ' '.join(bootstrap_tasks) + exec_build_env_command( + config.init_path, basepath, bb_cmd_early, watch=True) + if bootstrap_tasks_late: + bb_cmd_late = bb_cmd + ' '.join(bootstrap_tasks_late) + exec_build_env_command( + config.init_path, basepath, bb_cmd_late, watch=True) + + for recipe_image in recipes_images: + if (recipe_image.gdbserver_missing): + logger.warning( + "gdbserver not installed in image %s. Remote debugging will not be available" % recipe_image) + + if recipe_image.combine_dbg_image is False: + logger.warning( + 'IMAGE_CLASSES += "image-combined-dbg" is missing for image %s. Remote debugging will not find debug symbols from rootfs-dbg.' % recipe_image) + + # Instantiate the active IDE plugin + ide = ide_plugins[args.ide]() + if args.mode == DevtoolIdeMode.shared: + ide.setup_shared_sysroots(shared_env) + elif args.mode == DevtoolIdeMode.modified: + for recipe_modified in recipes_modified: + if recipe_modified.build_tool is BuildTool.CMAKE: + recipe_modified.cmake_preset() + if recipe_modified.build_tool is BuildTool.MESON: + recipe_modified.gen_meson_wrapper() + ide.setup_modified_recipe( + args, recipe_image, recipe_modified) + else: + raise DevtoolError("Must not end up here.") + + +def register_commands(subparsers, context): + """Register devtool subcommands from this plugin""" + + global ide_plugins + + # Search for IDE plugins in all sub-folders named ide_plugins where devtool seraches for plugins. + pluginpaths = [os.path.join(path, 'ide_plugins') + for path in context.pluginpaths] + ide_plugin_modules = [] + for pluginpath in pluginpaths: + scriptutils.load_plugins(logger, ide_plugin_modules, pluginpath) + + for ide_plugin_module in ide_plugin_modules: + if hasattr(ide_plugin_module, 'register_ide_plugin'): + ide_plugin_module.register_ide_plugin(ide_plugins) + # Sort plugins according to their priority. The first entry is the default IDE plugin. + ide_plugins = dict(sorted(ide_plugins.items(), + key=lambda p: p[1].ide_plugin_priority(), reverse=True)) + + parser_ide_sdk = subparsers.add_parser('ide-sdk', group='working', order=50, formatter_class=RawTextHelpFormatter, + help='Setup the SDK and configure the IDE') + parser_ide_sdk.add_argument( + 'recipenames', nargs='+', help='Generate an IDE configuration suitable to work on the given recipes.\n' + 'Depending on the --mode paramter different types of SDKs and IDE configurations are generated.') + parser_ide_sdk.add_argument( + '-m', '--mode', type=DevtoolIdeMode, default=DevtoolIdeMode.modified, + help='Different SDK types are supported:\n' + '- "' + DevtoolIdeMode.modified.name + '" (default):\n' + ' devtool modify creates a workspace to work on the source code of a recipe.\n' + ' devtool ide-sdk builds the SDK and generates the IDE configuration(s) in the workspace directorie(s)\n' + ' Usage example:\n' + ' devtool modify cmake-example\n' + ' devtool ide-sdk cmake-example core-image-minimal\n' + ' Start the IDE in the workspace folder\n' + ' At least one devtool modified recipe plus one image recipe are required:\n' + ' The image recipe is used to generate the target image and the remote debug configuration.\n' + '- "' + DevtoolIdeMode.shared.name + '":\n' + ' Usage example:\n' + ' devtool ide-sdk -m ' + DevtoolIdeMode.shared.name + ' recipe(s)\n' + ' This command generates a cross-toolchain as well as the corresponding shared sysroot directories.\n' + ' To use this tool-chain the environment-* file found in the deploy..image folder needs to be sourced into a shell.\n' + ' In case of VSCode and cmake the tool-chain is also exposed as a cmake-kit') + default_ide = list(ide_plugins.keys())[0] + parser_ide_sdk.add_argument( + '-i', '--ide', choices=ide_plugins.keys(), default=default_ide, + help='Setup the configuration for this IDE (default: %s)' % default_ide) + parser_ide_sdk.add_argument( + '-t', '--target', default='root@192.168.7.2', + help='Live target machine running an ssh server: user@hostname.') + parser_ide_sdk.add_argument( + '-G', '--gdbserver-port-start', default="1234", help='port where gdbserver is listening.') + parser_ide_sdk.add_argument( + '-c', '--no-host-check', help='Disable ssh host key checking', action='store_true') + parser_ide_sdk.add_argument( + '-e', '--ssh-exec', help='Executable to use in place of ssh') + parser_ide_sdk.add_argument( + '-P', '--port', help='Specify ssh port to use for connection to the target') + parser_ide_sdk.add_argument( + '-I', '--key', help='Specify ssh private key for connection to the target') + parser_ide_sdk.add_argument( + '--skip-bitbake', help='Generate IDE configuration but skip calling bibtake to update the SDK.', action='store_true') + parser_ide_sdk.add_argument( + '-k', '--bitbake-k', help='Pass -k parameter to bitbake', action='store_true') + parser_ide_sdk.add_argument( + '--no-strip', help='Do not strip executables prior to deploy', dest='strip', action='store_false') + parser_ide_sdk.add_argument( + '-n', '--dry-run', help='List files to be undeployed only', action='store_true') + parser_ide_sdk.add_argument( + '-s', '--show-status', help='Show progress/status output', action='store_true') + parser_ide_sdk.add_argument( + '-p', '--no-preserve', help='Do not preserve existing files', action='store_true') + parser_ide_sdk.add_argument( + '--no-check-space', help='Do not check for available space before deploying', action='store_true') + parser_ide_sdk.add_argument( + '--debug-build-config', help='Use debug build flags, for example set CMAKE_BUILD_TYPE=Debug', action='store_true') + parser_ide_sdk.set_defaults(func=ide_setup) From patchwork Mon Jan 22 13:58:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CB47C47DDF for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) by mx.groups.io with SMTP id smtpd.web11.74366.1705931965978086434 for ; Mon, 22 Jan 2024 05:59:26 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=cyRhBEtH; spf=pass (domain: gmail.com, ip: 209.85.208.43, mailfrom: adrian.freihofer@gmail.com) Received: by mail-ed1-f43.google.com with SMTP id 4fb4d7f45d1cf-55a684acf92so3242931a12.0 for ; Mon, 22 Jan 2024 05:59:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931964; x=1706536764; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EBNBj0nu8ui88glkGOzRhtBoQZsl8ckqJKa7AYI7LMQ=; b=cyRhBEtH4L1N1Y3kLBj0dMArBAO2lnEcl7+xVE7w+HQj91cBAV2K4MLdT0hxZ8Rp5K Rf+/q+YMLx3R298lV4fJBJZ0A6+DrG18e8k2l5IbtqtuIDvaH7Ejl2sKLce0aRMfdJL/ x4wDHcWsbbBY0Eimxt6+CPcuBYcqaIVIXH0BS249OPjHcNGrNFmFQ1yy4uM012iBMfHu Q24qmNVc01DGofGH1Y/H4Fsi+ueTWAMFDDlQGRwVCBuiZFMf8evngVaZ4PUMTiD8oopS WCZMRWJ6wDWpQBvKO+XW5Nvc/TCUsUIM8tLswl3g58/p1SIU20/e+UirZ/2srKcLLc+c aCfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931964; x=1706536764; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EBNBj0nu8ui88glkGOzRhtBoQZsl8ckqJKa7AYI7LMQ=; b=V33wzLKXNmsD3vYhjg7KK8eL7+vBKeWlkg5R1WqJpLP9ntkwOgI3tkeTTwDFAa8zaL YIZcLoNu1hYvICpnnZyN8m6G4mjqqYqYxW81XNQIAvcJw/HnOsRXJaLQVNoF0Ha+SSG3 oCYdMpemdpuvk/3bXBOh3jP2yn/4apOspEjtdKxrofS3qYpMZZSMvOYjeOyGkg02zYRV 7zmvu6QEQcFpTngq3FF7dKfv4CMMP8HPlRicQSq5a6yn54W+E7CFZZrbaPYm1rf7i4oc 9odxzodp0vrKyjBa7+SxtY4RtS+amO/xFNBmV4V4b/Q5zKQNO97o8WpHXLG7bK23TiI5 nBeQ== X-Gm-Message-State: AOJu0YyJCs9QRG4/oCySqs8kVmGUdry6KxV4Phk3SxMbVbsIgAakmsIU EVoItc+IzSR8vUItrLZbA1cOtCLCDMmk2gGvwMXE9KcTnuHqTDx4bP3BP0Ov X-Google-Smtp-Source: AGHT+IFNP2ZVRWR8Bj6yrId8d8ogPdMajP+xIaAQfIwcHz2oU5wOVryRQX6yk/97qHMJP2kFTIyZ0w== X-Received: by 2002:a17:907:2d9f:b0:a2f:617b:f8a4 with SMTP id gt31-20020a1709072d9f00b00a2f617bf8a4mr1878458ejc.19.1705931963897; Mon, 22 Jan 2024 05:59:23 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:23 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 4/9] oe-selftest devtool: ide-sdk tests Date: Mon, 22 Jan 2024 14:58:22 +0100 Message-ID: <20240122135901.171596-5-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194152 Add some oe-selftests for the new devtool ide-sdk plugin. Most of the workflows are covered. Many thanks to Enguerrand de Ribaucourt for testing and bug fixing. Signed-off-by: Adrian Freihofer --- meta/lib/oeqa/selftest/cases/devtool.py | 492 ++++++++++++++++++++++++ 1 file changed, 492 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index a8777207694..006c846438d 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -12,6 +12,7 @@ import tempfile import glob import fnmatch import unittest +import json from oeqa.selftest.case import OESelftestTestCase from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer @@ -2314,3 +2315,494 @@ class DevtoolUpgradeTests(DevtoolBase): #Step 4.5 runCmd("grep %s %s" % (modconfopt, codeconfigfile)) + + +class DevtoolIdeSdkTests(DevtoolBase): + def _write_bb_config(self, recipe_names): + """Helper to write the bitbake local.conf file""" + conf_lines = [ + 'IMAGE_CLASSES += "image-combined-dbg"', + 'IMAGE_GEN_DEBUGFS = "1"', + 'IMAGE_INSTALL:append = " gdbserver %s"' % ' '.join( + [r + '-ptest' for r in recipe_names]) + ] + self.write_config("\n".join(conf_lines)) + + def _check_workspace(self): + """Check if a workspace directory is available and setup the cleanup""" + self.assertTrue(not os.path.exists(self.workspacedir), + 'This test cannot be run with a workspace directory under the build directory') + self.track_for_cleanup(self.workspacedir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + + def _workspace_scripts_dir(self, recipe_name): + return os.path.realpath(os.path.join(self.builddir, 'workspace', 'ide-sdk', recipe_name, 'scripts')) + + def _sources_scripts_dir(self, src_dir): + return os.path.realpath(os.path.join(src_dir, 'oe-scripts')) + + def _workspace_gdbinit_dir(self, recipe_name): + return os.path.realpath(os.path.join(self.builddir, 'workspace', 'ide-sdk', recipe_name, 'scripts', 'gdbinit')) + + def _sources_gdbinit_dir(self, src_dir): + return os.path.realpath(os.path.join(src_dir, 'oe-gdbinit')) + + def _devtool_ide_sdk_recipe(self, recipe_name, build_file, testimage): + """Setup a recipe for working with devtool ide-sdk + + Basically devtool modify -x followed by some tests + """ + tempdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir) + self.add_command_to_tearDown('bitbake -c clean %s' % recipe_name) + + result = runCmd('devtool modify %s -x %s' % (recipe_name, tempdir)) + self.assertExists(os.path.join(tempdir, build_file), + 'Extracted source could not be found') + self.assertExists(os.path.join(self.workspacedir, 'conf', + 'layer.conf'), 'Workspace directory not created') + matches = glob.glob(os.path.join(self.workspacedir, + 'appends', recipe_name + '.bbappend')) + self.assertTrue(matches, 'bbappend not created %s' % result.output) + + # Test devtool status + result = runCmd('devtool status') + self.assertIn(recipe_name, result.output) + self.assertIn(tempdir, result.output) + self._check_src_repo(tempdir) + + # Usually devtool ide-sdk would initiate the build of the SDK. + # But there is a circular dependency with starting Qemu and passing the IP of runqemu to devtool ide-sdk. + if testimage: + bitbake("%s qemu-native qemu-helper-native" % testimage) + deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE') + self.add_command_to_tearDown('bitbake -c clean %s' % testimage) + self.add_command_to_tearDown( + 'rm -f %s/%s*' % (deploy_dir_image, testimage)) + + return tempdir + + def _get_recipe_ids(self, recipe_name): + """IDs needed to write recipe specific config entries into IDE config files""" + package_arch = get_bb_var('PACKAGE_ARCH', recipe_name) + recipe_id = recipe_name + "-" + package_arch + recipe_id_pretty = recipe_name + ": " + package_arch + return (recipe_id, recipe_id_pretty) + + def _verify_install_script_code(self, tempdir, recipe_name): + """Verify the scripts referred by the tasks.json file are fine. + + This function does not depend on Qemu. Therefore it verifies the scripts + exists and the delete step works as expected. But it does not try to + deploy to Qemu. + """ + recipe_id, recipe_id_pretty = self._get_recipe_ids(recipe_name) + with open(os.path.join(tempdir, '.vscode', 'tasks.json')) as tasks_j: + tasks_d = json.load(tasks_j) + tasks = tasks_d["tasks"] + task_install = next( + (task for task in tasks if task["label"] == "install && deploy-target %s" % recipe_id_pretty), None) + self.assertIsNot(task_install, None) + # execute only the bb_run_do_install script since the deploy would require e.g. Qemu running. + i_and_d_script = "install_and_deploy_" + recipe_id + i_and_d_script_path = os.path.join( + self._workspace_scripts_dir(recipe_name), i_and_d_script) + self.assertExists(i_and_d_script_path) + del_script = "delete_package_dirs_" + recipe_id + del_script_path = os.path.join( + self._workspace_scripts_dir(recipe_name), del_script) + self.assertExists(del_script_path) + runCmd(del_script_path, cwd=tempdir) + + def _devtool_ide_sdk_qemu(self, tempdir, qemu, recipe_name, example_exe): + """Verify deployment and execution in Qemu system work for one recipe. + + This function checks the entire SDK workflow: changing the code, recompiling + it and deploying it back to Qemu, and checking that the changes have been + incorporated into the provided binaries. It also runs the tests of the recipe. + """ + recipe_id, _ = self._get_recipe_ids(recipe_name) + i_and_d_script = "install_and_deploy_" + recipe_id + install_deploy_cmd = os.path.join( + self._workspace_scripts_dir(recipe_name), i_and_d_script) + self.assertExists(install_deploy_cmd, + '%s script not found' % install_deploy_cmd) + runCmd(install_deploy_cmd) + + MAGIC_STRING_ORIG = "Magic: 123456789" + MAGIC_STRING_NEW = "Magic: 987654321" + ptest_cmd = "ptest-runner " + recipe_name + + # validate that SSH is working + status, _ = qemu.run("uname") + self.assertEqual( + status, 0, msg="Failed to connect to the SSH server on Qemu") + + # Verify the unmodified example prints the magic string + status, output = qemu.run(example_exe) + self.assertEqual(status, 0, msg="%s failed: %s" % + (example_exe, output)) + self.assertIn(MAGIC_STRING_ORIG, output) + + # Verify the unmodified ptests work + status, output = qemu.run(ptest_cmd) + self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output)) + self.assertIn("PASS: cpp-example-lib", output) + + # Replace the Magic String in the code, compile and deploy to Qemu + cpp_example_lib_hpp = os.path.join(tempdir, 'cpp-example-lib.hpp') + with open(cpp_example_lib_hpp, 'r') as file: + cpp_code = file.read() + cpp_code = cpp_code.replace(MAGIC_STRING_ORIG, MAGIC_STRING_NEW) + with open(cpp_example_lib_hpp, 'w') as file: + file.write(cpp_code) + runCmd(install_deploy_cmd, cwd=tempdir) + + # Verify the modified example prints the modified magic string + status, output = qemu.run(example_exe) + self.assertEqual(status, 0, msg="%s failed: %s" % + (example_exe, output)) + self.assertNotIn(MAGIC_STRING_ORIG, output) + self.assertIn(MAGIC_STRING_NEW, output) + + # Verify the modified example ptests work + status, output = qemu.run(ptest_cmd) + self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output)) + self.assertIn("PASS: cpp-example-lib", output) + + def _gdb_cross(self): + """Verify gdb-cross is provided by devtool ide-sdk""" + target_arch = self.td["TARGET_ARCH"] + target_sys = self.td["TARGET_SYS"] + gdb_recipe = "gdb-cross-" + target_arch + gdb_binary = target_sys + "-gdb" + + native_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", gdb_recipe) + r = runCmd("%s --version" % gdb_binary, + native_sysroot=native_sysroot, target_sys=target_sys) + self.assertEqual(r.status, 0) + self.assertIn("GNU gdb", r.output) + + def _gdb_cross_debugging(self, qemu, recipe_name, example_exe): + """Verify gdb-cross is working + + Test remote debugging: + break main + run + continue + """ + sshargs = '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' + gdbserver_script = os.path.join(self._workspace_scripts_dir( + recipe_name), 'gdbserver_1234_usr-bin-' + example_exe + '_m') + gdb_script = os.path.join(self._workspace_scripts_dir( + recipe_name), 'gdb_1234_usr-bin-' + example_exe) + + # Start a gdbserver + r = runCmd(gdbserver_script) + self.assertEqual(r.status, 0) + + # Check there is a gdbserver running + r = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, 'ps')) + self.assertEqual(r.status, 0) + self.assertIn("gdbserver ", r.output) + + # Check the pid file is correct + test_cmd = "cat /proc/$(cat /tmp/gdbserver_1234_usr-bin-" + \ + example_exe + "/pid)/cmdline" + r = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, test_cmd)) + self.assertEqual(r.status, 0) + self.assertIn("gdbserver", r.output) + + # Test remote debugging works + r = runCmd( + gdb_script + " --batch -ex 'break main' --ex 'run' -ex 'continue'") + self.assertEqual(r.status, 0) + self.assertIn("Breakpoint 1, main", r.output) + self.assertIn("exited normally", r.output) + + # Stop the gdbserver + r = runCmd(gdbserver_script + ' stop') + self.assertEqual(r.status, 0) + + # Check there is no gdbserver running + r = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, 'ps')) + self.assertEqual(r.status, 0) + self.assertNotIn("gdbserver ", r.output) + + def _verify_cmake_preset(self, tempdir): + """Verify the generated cmake preset works as expected + + Check if compiling works + Check if unit tests can be executed in qemu (not qemu-system) + """ + with open(os.path.join(tempdir, 'CMakeUserPresets.json')) as cmake_preset_j: + cmake_preset_d = json.load(cmake_preset_j) + config_presets = cmake_preset_d["configurePresets"] + self.assertEqual(len(config_presets), 1) + cmake_exe = config_presets[0]["cmakeExecutable"] + preset_name = config_presets[0]["name"] + + # Verify the wrapper for cmake native is available + self.assertExists(cmake_exe) + + # Verify the cmake preset generated by devtool ide-sdk is available + result = runCmd('%s --list-presets' % cmake_exe, cwd=tempdir) + self.assertIn(preset_name, result.output) + + # Verify cmake re-uses the o files compiled by bitbake + result = runCmd('%s --build --preset %s' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("ninja: no work to do.", result.output) + + # Verify the unit tests work (in Qemu user mode) + result = runCmd('%s --build --preset %s --target test' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("100% tests passed", result.output) + + # Verify re-building and testing works again + result = runCmd('%s --build --preset %s --target clean' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("Cleaning", result.output) + result = runCmd('%s --build --preset %s' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("Building", result.output) + self.assertIn("Linking", result.output) + result = runCmd('%s --build --preset %s --target test' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("Running tests...", result.output) + self.assertIn("100% tests passed", result.output) + + @OETestTag("runqemu") + def test_devtool_ide_sdk_none_qemu(self): + """Start qemu-system and run tests for multiple recipes. ide=none is used.""" + recipe_names = ["cmake-example", "meson-example"] + testimage = "oe-selftest-image" + + self._check_workspace() + self._write_bb_config(recipe_names) + self._check_runqemu_prerequisites() + + # Verify deployment to Qemu (system mode) works + bitbake(testimage) + with runqemu(testimage, runqemuparams="nographic") as qemu: + # cmake-example recipe + recipe_name = "cmake-example" + example_exe = "cmake-example" + build_file = "CMakeLists.txt" + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@%s -c --ide=none' % ( + recipe_name, testimage, qemu.ip) + runCmd(bitbake_sdk_cmd) + self._verify_cmake_preset(tempdir) + self._devtool_ide_sdk_qemu(tempdir, qemu, recipe_name, example_exe) + # Verify the oe-scripts sym-link is valid + self.assertEqual(self._workspace_scripts_dir( + recipe_name), self._sources_scripts_dir(tempdir)) + # Verify GDB is working after devtool ide-sdk + self._gdb_cross() + self._gdb_cross_debugging(qemu, recipe_name, example_exe) + + # meson-example recipe + recipe_name = "meson-example" + example_exe = "mesonex" + build_file = "meson.build" + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@%s -c --ide=none' % ( + recipe_name, testimage, qemu.ip) + runCmd(bitbake_sdk_cmd) + self._devtool_ide_sdk_qemu(tempdir, qemu, recipe_name, example_exe) + # Verify the oe-scripts sym-link is valid + self.assertEqual(self._workspace_scripts_dir( + recipe_name), self._sources_scripts_dir(tempdir)) + # Verify GDB is working after devtool ide-sdk + self._gdb_cross() + self._gdb_cross_debugging(qemu, recipe_name, example_exe) + + def test_devtool_ide_sdk_code_cmake(self): + """Verify a cmake recipe works with ide=code mode""" + recipe_name = "cmake-example" + build_file = "CMakeLists.txt" + testimage = "oe-selftest-image" + + self._check_workspace() + self._write_bb_config([recipe_name]) + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@192.168.17.17 -c --ide=code' % ( + recipe_name, testimage) + runCmd(bitbake_sdk_cmd) + self._verify_cmake_preset(tempdir) + self._verify_install_script_code(tempdir, recipe_name) + self._gdb_cross() + + def test_devtool_ide_sdk_code_meson(self): + """Verify a meson recipe works with ide=code mode""" + recipe_name = "meson-example" + build_file = "meson.build" + testimage = "oe-selftest-image" + + self._check_workspace() + self._write_bb_config([recipe_name]) + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@192.168.17.17 -c --ide=code' % ( + recipe_name, testimage) + runCmd(bitbake_sdk_cmd) + + with open(os.path.join(tempdir, '.vscode', 'settings.json')) as settings_j: + settings_d = json.load(settings_j) + meson_exe = settings_d["mesonbuild.mesonPath"] + meson_build_folder = settings_d["mesonbuild.buildFolder"] + + # Verify the wrapper for meson native is available + self.assertExists(meson_exe) + + # Verify meson re-uses the o files compiled by bitbake + result = runCmd('%s compile -C %s' % + (meson_exe, meson_build_folder), cwd=tempdir) + self.assertIn("ninja: no work to do.", result.output) + + # Verify the unit tests work (in Qemu) + runCmd('%s test -C %s' % (meson_exe, meson_build_folder), cwd=tempdir) + + # Verify re-building and testing works again + result = runCmd('%s compile -C %s --clean' % + (meson_exe, meson_build_folder), cwd=tempdir) + self.assertIn("Cleaning...", result.output) + result = runCmd('%s compile -C %s' % + (meson_exe, meson_build_folder), cwd=tempdir) + self.assertIn("Linking target", result.output) + runCmd('%s test -C %s' % (meson_exe, meson_build_folder), cwd=tempdir) + + self._verify_install_script_code(tempdir, recipe_name) + self._gdb_cross() + + def test_devtool_ide_sdk_shared_sysroots(self): + """Verify the shared sysroot SDK""" + + # Handle the workspace (which is not needed by this test case) + self._check_workspace() + + result_init = runCmd( + 'devtool ide-sdk -m shared oe-selftest-image cmake-example meson-example --ide=code') + bb_vars = get_bb_vars( + ['REAL_MULTIMACH_TARGET_SYS', 'DEPLOY_DIR_IMAGE', 'COREBASE'], "meta-ide-support") + environment_script = 'environment-setup-%s' % bb_vars['REAL_MULTIMACH_TARGET_SYS'] + deploydir = bb_vars['DEPLOY_DIR_IMAGE'] + environment_script_path = os.path.join(deploydir, environment_script) + cpp_example_src = os.path.join( + bb_vars['COREBASE'], 'meta-selftest', 'recipes-test', 'cpp', 'files') + + # Verify the cross environment script is available + self.assertExists(environment_script_path) + + def runCmdEnv(cmd, cwd): + cmd = '/bin/sh -c ". %s > /dev/null && %s"' % ( + environment_script_path, cmd) + return runCmd(cmd, cwd) + + # Verify building the C++ example works with CMake + tempdir_cmake = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir_cmake) + + result_cmake = runCmdEnv("which cmake", cwd=tempdir_cmake) + cmake_native = os.path.normpath(result_cmake.output.strip()) + self.assertExists(cmake_native) + + runCmdEnv('cmake %s' % cpp_example_src, cwd=tempdir_cmake) + runCmdEnv('cmake --build %s' % tempdir_cmake, cwd=tempdir_cmake) + + # Verify the printed note really referres to a cmake executable + cmake_native_code = "" + for line in result_init.output.splitlines(): + m = re.search(r'"cmake.cmakePath": "(.*)"', line) + if m: + cmake_native_code = m.group(1) + break + self.assertExists(cmake_native_code) + self.assertEqual(cmake_native, cmake_native_code) + + # Verify building the C++ example works with Meson + tempdir_meson = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir_meson) + + result_cmake = runCmdEnv("which meson", cwd=tempdir_meson) + meson_native = os.path.normpath(result_cmake.output.strip()) + self.assertExists(meson_native) + + runCmdEnv('meson setup %s' % tempdir_meson, cwd=cpp_example_src) + runCmdEnv('meson compile', cwd=tempdir_meson) + + def test_devtool_ide_sdk_plugins(self): + """Test that devtool ide-sdk can use plugins from other layers.""" + + # We need a workspace layer and a modified recipe (but no image) + modified_recipe_name = "meson-example" + modified_build_file = "meson.build" + testimage = "oe-selftest-image" + shared_recipe_name = "cmake-example" + + self._check_workspace() + self._write_bb_config([modified_recipe_name]) + tempdir = self._devtool_ide_sdk_recipe( + modified_recipe_name, modified_build_file, None) + + IDE_RE = re.compile(r'.*--ide \{(.*)\}.*') + + def get_ides_from_help(help_str): + m = IDE_RE.search(help_str) + return m.group(1).split(',') + + # verify the default plugins are available but the foo plugin is not + result = runCmd('devtool ide-sdk -h') + found_ides = get_ides_from_help(result.output) + self.assertIn('code', found_ides) + self.assertIn('none', found_ides) + self.assertNotIn('foo', found_ides) + + shared_config_file = os.path.join(tempdir, 'shared-config.txt') + shared_config_str = 'Dummy shared IDE config' + modified_config_file = os.path.join(tempdir, 'modified-config.txt') + modified_config_str = 'Dummy modified IDE config' + + # Generate a foo plugin in the workspace layer + plugin_dir = os.path.join( + self.workspacedir, 'lib', 'devtool', 'ide_plugins') + os.makedirs(plugin_dir) + plugin_code = 'from devtool.ide_plugins import IdeBase\n\n' + plugin_code += 'class IdeFoo(IdeBase):\n' + plugin_code += ' def setup_shared_sysroots(self, shared_env):\n' + plugin_code += ' with open("%s", "w") as config_file:\n' % shared_config_file + plugin_code += ' config_file.write("%s")\n\n' % shared_config_str + plugin_code += ' def setup_modified_recipe(self, args, image_recipe, modified_recipe):\n' + plugin_code += ' with open("%s", "w") as config_file:\n' % modified_config_file + plugin_code += ' config_file.write("%s")\n\n' % modified_config_str + plugin_code += 'def register_ide_plugin(ide_plugins):\n' + plugin_code += ' ide_plugins["foo"] = IdeFoo\n' + + plugin_py = os.path.join(plugin_dir, 'ide_foo.py') + with open(plugin_py, 'w') as plugin_file: + plugin_file.write(plugin_code) + + # Verify the foo plugin is available as well + result = runCmd('devtool ide-sdk -h') + found_ides = get_ides_from_help(result.output) + self.assertIn('code', found_ides) + self.assertIn('none', found_ides) + self.assertIn('foo', found_ides) + + # Verify the foo plugin generates a shared config + result = runCmd( + 'devtool ide-sdk -m shared --skip-bitbake --ide foo %s' % shared_recipe_name) + with open(shared_config_file) as shared_config: + shared_config_new = shared_config.read() + self.assertEqual(shared_config_str, shared_config_new) + + # Verify the foo plugin generates a modified config + result = runCmd('devtool ide-sdk --skip-bitbake --ide foo %s %s' % + (modified_recipe_name, testimage)) + with open(modified_config_file) as modified_config: + modified_config_new = modified_config.read() + self.assertEqual(modified_config_str, modified_config_new) From patchwork Mon Jan 22 13:58:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54B44C47DDE for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by mx.groups.io with SMTP id smtpd.web11.74367.1705931966967662372 for ; Mon, 22 Jan 2024 05:59:27 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=fAnmpjk8; spf=pass (domain: gmail.com, ip: 209.85.167.41, mailfrom: adrian.freihofer@gmail.com) Received: by mail-lf1-f41.google.com with SMTP id 2adb3069b0e04-50eaaf2c7deso3440644e87.2 for ; Mon, 22 Jan 2024 05:59:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931965; x=1706536765; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Sz9hf9rJQjPVy4Q9SbgvcNjWvjydRtS609XVC+jICFg=; b=fAnmpjk8WiqCmqCSvblTlXzFz5q42/8uxzqMs3xjuuUZ+9OC7RysVBHRj0YeuBTXgh AnCPqSq/elF/V9qHLLOYJ21X6L5oLO63w4Os4ICun3b76X3gRrC//igIIY5bdkkpbeXJ Zb8qd5JQbWURBvzBUFFYrIvsxpZOX6BLxorF15mPe3zcHOp6ZMW3leJakI+7oRaB48CK lTEkUNLZopcbvO5d0y7Fq3AoTyPMv8t5enZK7xe4XkrfqzFr1hgpG9r3YMw2RLjS3ND0 SBbjMZg9romBhGcxHYGyCUwXV7QmTpgc5wpM4APIIAutGijcm+Ol6RT5NE3ukw019FiX vZ2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931965; x=1706536765; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sz9hf9rJQjPVy4Q9SbgvcNjWvjydRtS609XVC+jICFg=; b=c6OP5aS8k23BiQyLWov3hV+8MyLWG+cumZF49MtorsmOTxwk2EjrYaE2O8okKolOA7 LrR22Ue/lf0zM/2BKEdjSH6ERGLBijy8hTNyzd3bm9D8N1iaW9VhvtRDMrAtJvng2L/R YW/k1YGgzUJhWOjsOElU0ti9yLToJ27PrfYaRobAqSUxw7hgK5xIaxdoLdrpWlR7Dnul iSCTy+E4WhbIP8zbt1m9EdERPLO+30on+/2pp2d7RtB/CipDABQT37NipSP+VV0Oig4m TmwfDuuOP4JJKa+gy9vzdTHf/ZtgFhEp6kBJo1BNev318MXfV33WiSjOaJXLEwCszyRH ty2A== X-Gm-Message-State: AOJu0Yy5MplvqxKpyM4Gj489Ru05giH3vbZZaVPuQKZSohllukMcuaTj 42yFgdIQI77EnZNICK5EPGJbm9IRzh55zuNxYuLn3I7Rho/sIxjxLsw9MqjD X-Google-Smtp-Source: AGHT+IGvmd+9wRU3dwkJjQ/DPo7iKy7AI5CPdOBBtmo5zsb1m+7H+97E/ypbkKmFFRrB9osmxGvcMg== X-Received: by 2002:a05:6512:b20:b0:50e:8100:c0c3 with SMTP id w32-20020a0565120b2000b0050e8100c0c3mr2038668lfu.2.1705931964680; Mon, 22 Jan 2024 05:59:24 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:24 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 5/9] devtool: refactor exec_fakeroot Date: Mon, 22 Jan 2024 14:58:23 +0100 Message-ID: <20240122135901.171596-6-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194154 Provide a function exec_fakeroot_no_d which does the same like exec_fakeroot does, but is usable independenlty from bitbake. This allows to use the fanction from scripts where the d variable is not available. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py index b4f998a5bc4..c7fc3cfc410 100644 --- a/scripts/lib/devtool/__init__.py +++ b/scripts/lib/devtool/__init__.py @@ -78,12 +78,15 @@ def exec_fakeroot(d, cmd, **kwargs): """Run a command under fakeroot (pseudo, in fact) so that it picks up the appropriate file permissions""" # Grab the command and check it actually exists fakerootcmd = d.getVar('FAKEROOTCMD') + fakerootenv = d.getVar('FAKEROOTENV') + exec_fakeroot_no_d(fakerootcmd, fakerootenv, cmd, kwargs) + +def exec_fakeroot_no_d(fakerootcmd, fakerootenv, cmd, **kwargs): if not os.path.exists(fakerootcmd): logger.error('pseudo executable %s could not be found - have you run a build yet? pseudo-native should install this and if you have run any build then that should have been built') return 2 # Set up the appropriate environment newenv = dict(os.environ) - fakerootenv = d.getVar('FAKEROOTENV') for varvalue in fakerootenv.split(): if '=' in varvalue: splitval = varvalue.split('=', 1) From patchwork Mon Jan 22 13:58:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C627C47DDB for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) by mx.groups.io with SMTP id smtpd.web11.74368.1705931967665846752 for ; Mon, 22 Jan 2024 05:59:27 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PzfdAxe9; spf=pass (domain: gmail.com, ip: 209.85.208.42, mailfrom: adrian.freihofer@gmail.com) Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-55c1ac8d2f2so1504272a12.2 for ; Mon, 22 Jan 2024 05:59:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931966; x=1706536766; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JwRA+OhhW5FIBT2i7TxbtL1+itpsmXTPUTEbpdriySI=; b=PzfdAxe9i1n3MI0KaeaMJU3UP6Vy0A3PMVOcGHZaH0tJdG87H4j3VIYQxwbbMDOhIw Mt5wHDU8pWlA7tGjZuUXWw1ojbqe1ta1Rkqh0WNWF3rzN4LTDNDnufVJtpdcjR0cIu8V VWdRi7sPD/aWf+lggUJThkUM9QAtb8fInEfsUNl1hlFZagiK8RNFFkChtJ7029ian1hX /9IPgFDf5kEw97PKfMgFjDgU323glAnYJIoZBEHT1uxPFAg3OCOImfwJ57YDr3VV32mV Dk7CeM30rbtVPyyg3WYACLmB6rS/0RvLKW4NVH0wEVYu14pQcJZen98vZpxaNt/L/6Vo uHTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931966; x=1706536766; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JwRA+OhhW5FIBT2i7TxbtL1+itpsmXTPUTEbpdriySI=; b=gAV7Z3yuzx6pqjBTcutESu5JtHZOJ1S7AZcS4umhLsxXjLegmlqOUIs2d3Ajr51NeL oZ4blHJ2axlgdZHA2F1Ng6XQ3AYUHbh74Cj+NJgLwtmRefuNogRDdcU8dZXLe9L44roH EOe4TrlkhJRMbjRbPce4kyiZlJkXT4+D7IBfvAXT5RjQgH/99EwvfBWHp49DAFpTmxkN 0UZy/u794UBILb7cEnAMMhhPxYdWUI7cWUCK/AMJg8sFmVBfJWFb8acK8KcwfqBRDrFO Q53QjvBGQWCu7Ds3qaNwLLZzZbM1jMuOALevoMKN4JcTAoAdjWXUJV5MnhbfB+uoQHoK dp3Q== X-Gm-Message-State: AOJu0YzCEBVTXud0236VDx9Lhp38MNwbupOtA/VVluCkRgBm5hGJVGOC zysgycNKAliCCqUAkGNSl7qJpssOR+AQXJ5UVHt6nd3bQsS2aSs1/kIsJ2jp X-Google-Smtp-Source: AGHT+IECfutax2e6Z2DtpE8LuiGD8kFB8Sa801+9i6I12xXJkKfRbgqnIhfG/o1/qxlRlsdy8Kly4Q== X-Received: by 2002:a17:906:493:b0:a2f:de3b:1307 with SMTP id f19-20020a170906049300b00a2fde3b1307mr1862689eja.135.1705931965877; Mon, 22 Jan 2024 05:59:25 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:24 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 6/9] devtool: refactor deploy to use exec_fakeroot_no_d Date: Mon, 22 Jan 2024 14:58:24 +0100 Message-ID: <20240122135901.171596-7-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194155 This is a step towards a deploy function which can be called without passing the d variable. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/deploy.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py index eadf6e15214..b71d55e2312 100644 --- a/scripts/lib/devtool/deploy.py +++ b/scripts/lib/devtool/deploy.py @@ -16,7 +16,7 @@ import bb.utils import argparse_oe import oe.types -from devtool import exec_fakeroot, setup_tinfoil, check_workspace_recipe, DevtoolError +from devtool import exec_fakeroot_no_d, setup_tinfoil, check_workspace_recipe, DevtoolError logger = logging.getLogger('devtool') @@ -160,19 +160,22 @@ def deploy(args, config, basepath, workspace): except Exception as e: raise DevtoolError('Exception parsing recipe %s: %s' % (args.recipename, e)) + recipe_outdir = rd.getVar('D') if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir): raise DevtoolError('No files to deploy - have you built the %s ' 'recipe? If so, the install step has not installed ' 'any files.' % args.recipename) + fakerootcmd = rd.getVar('FAKEROOTCMD') + fakerootenv = rd.getVar('FAKEROOTENV') if args.strip and not args.dry_run: # Fakeroot copy to new destination srcdir = recipe_outdir recipe_outdir = os.path.join(rd.getVar('WORKDIR'), 'devtool-deploy-target-stripped') if os.path.isdir(recipe_outdir): - exec_fakeroot(rd, "rm -rf %s" % recipe_outdir, shell=True) - exec_fakeroot(rd, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "rm -rf %s" % recipe_outdir, shell=True) + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) os.environ['PATH'] = ':'.join([os.environ['PATH'], rd.getVar('PATH') or '']) oe.package.strip_execs(args.recipename, recipe_outdir, rd.getVar('STRIP'), rd.getVar('libdir'), rd.getVar('base_libdir'), oe.utils.get_bb_number_threads(rd), rd) @@ -251,7 +254,7 @@ def deploy(args, config, basepath, workspace): shutil.rmtree(tmpdir) # Now run the script - ret = exec_fakeroot(rd, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) + ret = exec_fakeroot_no_d(fakerootcmd, fakerootenv, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) if ret != 0: raise DevtoolError('Deploy failed - rerun with -s to get a complete ' 'error message') From patchwork Mon Jan 22 13:58:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 832A0C48277 for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) by mx.groups.io with SMTP id smtpd.web10.74262.1705931968604177761 for ; Mon, 22 Jan 2024 05:59:28 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=aVtO5Q58; spf=pass (domain: gmail.com, ip: 209.85.218.53, mailfrom: adrian.freihofer@gmail.com) Received: by mail-ej1-f53.google.com with SMTP id a640c23a62f3a-a2cea0563cbso476982966b.3 for ; Mon, 22 Jan 2024 05:59:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931967; x=1706536767; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ia6d/hhX/iexjlFxgXuruNl1zuEtuCTVFnk98oKC1o4=; b=aVtO5Q58gPZH26owTp8SlLTxL6QKz+lkuqL66Jltk59qRwhnBxhKSsUWkI/aUI8fQP sJ8A/B2VtpNO9CYdhj/2q6gXksvjx8kWkaDFUcYmmNCqaXUFEYNxoW/Nv49RMxdQ2BMp J3FK2MazY+4ExfZXSx9O8amctMdc7BXcnxEAS2rqETXSSyrcQqwPTXAl7QU5/WuR1i+C 48Xa8vnz1eMP4xier1ZB9qejR1Ok6IR9VPBSkH2SiHnOgmwhznobxSYl5cUwX1IcG90Q NABKAYBjhRXUEY24T+ImCV6kTUqZ1l2aR3VYiHYde1aazn0DeupfWXBWekqPZ0GI928m fL1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931967; x=1706536767; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ia6d/hhX/iexjlFxgXuruNl1zuEtuCTVFnk98oKC1o4=; b=h/kA0F6nYQiMrOdm7Ne8n5jeEzWNYdjE/QtocmmX7/eRt02/aQehtbDxIxJd3V7J5L ZAUVXTk+8tUOlWeTplA5A3Q5GtmYgsLVf+DB/Fwk6wWP6gpsB0ZAh9KzZ4anhKRqS8hB 1iYjE9l8T5KVa1nvlPoAJSU1Gxo6w1eVocn60YuqZdOsuLdnUPe9i28f5dVcW3EZBvXt 04rZKlx5FYoAavIBww4EpiCsIE6FkPXizhGc3jzcVKNjWhfsiUMc5PHcBsuPxRzRXJpb T7EM2KgN+4HI/cxQeSbf4jOd/59zVDOoooXZAx5TvZRODSPbOIoZADNH210Cv7qdKfEE zjIA== X-Gm-Message-State: AOJu0Yzp3LFClI3IY/FHaoKfQVjUClj6yPZnADRKzYzrbToSEy852jow JaDsqy0CvfUljQNPY43BnuIi7TS8DJxQMV2G4kYyGTfTTA25V07kpz7SjeWq X-Google-Smtp-Source: AGHT+IHModj4cx40ma8VpjRGL8eHwlynswaLcQQo7Xpnm4cCzLFhfUEOkcz8zcvQed+/We5CIkpqeg== X-Received: by 2002:a17:907:d409:b0:a2f:ca46:fc8d with SMTP id vi9-20020a170907d40900b00a2fca46fc8dmr1916614ejc.17.1705931966666; Mon, 22 Jan 2024 05:59:26 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:26 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 7/9] devtool: refactor deploy-target Date: Mon, 22 Jan 2024 14:58:25 +0100 Message-ID: <20240122135901.171596-8-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194156 Make the deploy function independent from d. This allows to call the function also from Python code not running in bitbake. This is needed to for the devtool ide plugin which will call the do_install task and the code from devtool deploy-target independently from a bitbake server. This allows a much quicker workflow. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/deploy.py | 236 ++++++++++++++++++---------------- 1 file changed, 122 insertions(+), 114 deletions(-) diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py index b71d55e2312..b5ca8f2c2f1 100644 --- a/scripts/lib/devtool/deploy.py +++ b/scripts/lib/devtool/deploy.py @@ -133,17 +133,38 @@ def _prepare_remote_script(deploy, verbose=False, dryrun=False, undeployall=Fals return '\n'.join(lines) - - def deploy(args, config, basepath, workspace): """Entry point for the devtool 'deploy' subcommand""" - import math - import oe.recipeutils - import oe.package import oe.utils check_workspace_recipe(workspace, args.recipename, checksrc=False) + tinfoil = setup_tinfoil(basepath=basepath) + try: + try: + rd = tinfoil.parse_recipe(args.recipename) + except Exception as e: + raise DevtoolError('Exception parsing recipe %s: %s' % + (args.recipename, e)) + + srcdir = rd.getVar('D') + workdir = rd.getVar('WORKDIR') + path = rd.getVar('PATH') + strip_cmd = rd.getVar('STRIP') + libdir = rd.getVar('libdir') + base_libdir = rd.getVar('base_libdir') + max_process = oe.utils.get_bb_number_threads(rd) + fakerootcmd = rd.getVar('FAKEROOTCMD') + fakerootenv = rd.getVar('FAKEROOTENV') + finally: + tinfoil.shutdown() + + return deploy_no_d(srcdir, workdir, path, strip_cmd, libdir, base_libdir, max_process, fakerootcmd, fakerootenv, args) + +def deploy_no_d(srcdir, workdir, path, strip_cmd, libdir, base_libdir, max_process, fakerootcmd, fakerootenv, args): + import math + import oe.package + try: host, destdir = args.target.split(':') except ValueError: @@ -153,121 +174,108 @@ def deploy(args, config, basepath, workspace): if not destdir.endswith('/'): destdir += '/' - tinfoil = setup_tinfoil(basepath=basepath) - try: - try: - rd = tinfoil.parse_recipe(args.recipename) - except Exception as e: - raise DevtoolError('Exception parsing recipe %s: %s' % - (args.recipename, e)) + recipe_outdir = srcdir + if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir): + raise DevtoolError('No files to deploy - have you built the %s ' + 'recipe? If so, the install step has not installed ' + 'any files.' % args.recipename) + + if args.strip and not args.dry_run: + # Fakeroot copy to new destination + srcdir = recipe_outdir + recipe_outdir = os.path.join(workdir, 'devtool-deploy-target-stripped') + if os.path.isdir(recipe_outdir): + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "rm -rf %s" % recipe_outdir, shell=True) + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) + os.environ['PATH'] = ':'.join([os.environ['PATH'], path or '']) + oe.package.strip_execs(args.recipename, recipe_outdir, strip_cmd, libdir, base_libdir, max_process) + + filelist = [] + inodes = set({}) + ftotalsize = 0 + for root, _, files in os.walk(recipe_outdir): + for fn in files: + fstat = os.lstat(os.path.join(root, fn)) + # Get the size in kiB (since we'll be comparing it to the output of du -k) + # MUST use lstat() here not stat() or getfilesize() since we don't want to + # dereference symlinks + if fstat.st_ino in inodes: + fsize = 0 + else: + fsize = int(math.ceil(float(fstat.st_size)/1024)) + inodes.add(fstat.st_ino) + ftotalsize += fsize + # The path as it would appear on the target + fpath = os.path.join(destdir, os.path.relpath(root, recipe_outdir), fn) + filelist.append((fpath, fsize)) + + if args.dry_run: + print('Files to be deployed for %s on target %s:' % (args.recipename, args.target)) + for item, _ in filelist: + print(' %s' % item) + return 0 - recipe_outdir = rd.getVar('D') - if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir): - raise DevtoolError('No files to deploy - have you built the %s ' - 'recipe? If so, the install step has not installed ' - 'any files.' % args.recipename) + extraoptions = '' + if args.no_host_check: + extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' + if not args.show_status: + extraoptions += ' -q' - fakerootcmd = rd.getVar('FAKEROOTCMD') - fakerootenv = rd.getVar('FAKEROOTENV') - if args.strip and not args.dry_run: - # Fakeroot copy to new destination - srcdir = recipe_outdir - recipe_outdir = os.path.join(rd.getVar('WORKDIR'), 'devtool-deploy-target-stripped') - if os.path.isdir(recipe_outdir): - exec_fakeroot_no_d(fakerootcmd, fakerootenv, "rm -rf %s" % recipe_outdir, shell=True) - exec_fakeroot_no_d(fakerootcmd, fakerootenv, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) - os.environ['PATH'] = ':'.join([os.environ['PATH'], rd.getVar('PATH') or '']) - oe.package.strip_execs(args.recipename, recipe_outdir, rd.getVar('STRIP'), rd.getVar('libdir'), - rd.getVar('base_libdir'), oe.utils.get_bb_number_threads(rd), rd) - - filelist = [] - inodes = set({}) - ftotalsize = 0 - for root, _, files in os.walk(recipe_outdir): - for fn in files: - fstat = os.lstat(os.path.join(root, fn)) - # Get the size in kiB (since we'll be comparing it to the output of du -k) - # MUST use lstat() here not stat() or getfilesize() since we don't want to - # dereference symlinks - if fstat.st_ino in inodes: - fsize = 0 - else: - fsize = int(math.ceil(float(fstat.st_size)/1024)) - inodes.add(fstat.st_ino) - ftotalsize += fsize - # The path as it would appear on the target - fpath = os.path.join(destdir, os.path.relpath(root, recipe_outdir), fn) - filelist.append((fpath, fsize)) - - if args.dry_run: - print('Files to be deployed for %s on target %s:' % (args.recipename, args.target)) - for item, _ in filelist: - print(' %s' % item) - return 0 - - extraoptions = '' - if args.no_host_check: - extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' - if not args.show_status: - extraoptions += ' -q' - - scp_sshexec = '' - ssh_sshexec = 'ssh' - if args.ssh_exec: - scp_sshexec = "-S %s" % args.ssh_exec - ssh_sshexec = args.ssh_exec - scp_port = '' - ssh_port = '' - if args.port: - scp_port = "-P %s" % args.port - ssh_port = "-p %s" % args.port - - if args.key: - extraoptions += ' -i %s' % args.key - - # In order to delete previously deployed files and have the manifest file on - # the target, we write out a shell script and then copy it to the target - # so we can then run it (piping tar output to it). - # (We cannot use scp here, because it doesn't preserve symlinks.) - tmpdir = tempfile.mkdtemp(prefix='devtool') - try: - tmpscript = '/tmp/devtool_deploy.sh' - tmpfilelist = os.path.join(os.path.dirname(tmpscript), 'devtool_deploy.list') - shellscript = _prepare_remote_script(deploy=True, - verbose=args.show_status, - nopreserve=args.no_preserve, - nocheckspace=args.no_check_space) - # Write out the script to a file - with open(os.path.join(tmpdir, os.path.basename(tmpscript)), 'w') as f: - f.write(shellscript) - # Write out the file list - with open(os.path.join(tmpdir, os.path.basename(tmpfilelist)), 'w') as f: - f.write('%d\n' % ftotalsize) - for fpath, fsize in filelist: - f.write('%s %d\n' % (fpath, fsize)) - # Copy them to the target - ret = subprocess.call("scp %s %s %s %s/* %s:%s" % (scp_sshexec, scp_port, extraoptions, tmpdir, args.target, os.path.dirname(tmpscript)), shell=True) - if ret != 0: - raise DevtoolError('Failed to copy script to %s - rerun with -s to ' - 'get a complete error message' % args.target) - finally: - shutil.rmtree(tmpdir) + scp_sshexec = '' + ssh_sshexec = 'ssh' + if args.ssh_exec: + scp_sshexec = "-S %s" % args.ssh_exec + ssh_sshexec = args.ssh_exec + scp_port = '' + ssh_port = '' + if args.port: + scp_port = "-P %s" % args.port + ssh_port = "-p %s" % args.port - # Now run the script - ret = exec_fakeroot_no_d(fakerootcmd, fakerootenv, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) + if args.key: + extraoptions += ' -i %s' % args.key + + # In order to delete previously deployed files and have the manifest file on + # the target, we write out a shell script and then copy it to the target + # so we can then run it (piping tar output to it). + # (We cannot use scp here, because it doesn't preserve symlinks.) + tmpdir = tempfile.mkdtemp(prefix='devtool') + try: + tmpscript = '/tmp/devtool_deploy.sh' + tmpfilelist = os.path.join(os.path.dirname(tmpscript), 'devtool_deploy.list') + shellscript = _prepare_remote_script(deploy=True, + verbose=args.show_status, + nopreserve=args.no_preserve, + nocheckspace=args.no_check_space) + # Write out the script to a file + with open(os.path.join(tmpdir, os.path.basename(tmpscript)), 'w') as f: + f.write(shellscript) + # Write out the file list + with open(os.path.join(tmpdir, os.path.basename(tmpfilelist)), 'w') as f: + f.write('%d\n' % ftotalsize) + for fpath, fsize in filelist: + f.write('%s %d\n' % (fpath, fsize)) + # Copy them to the target + ret = subprocess.call("scp %s %s %s %s/* %s:%s" % (scp_sshexec, scp_port, extraoptions, tmpdir, args.target, os.path.dirname(tmpscript)), shell=True) if ret != 0: - raise DevtoolError('Deploy failed - rerun with -s to get a complete ' - 'error message') + raise DevtoolError('Failed to copy script to %s - rerun with -s to ' + 'get a complete error message' % args.target) + finally: + shutil.rmtree(tmpdir) - logger.info('Successfully deployed %s' % recipe_outdir) + # Now run the script + ret = exec_fakeroot_no_d(fakerootcmd, fakerootenv, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) + if ret != 0: + raise DevtoolError('Deploy failed - rerun with -s to get a complete ' + 'error message') - files_list = [] - for root, _, files in os.walk(recipe_outdir): - for filename in files: - filename = os.path.relpath(os.path.join(root, filename), recipe_outdir) - files_list.append(os.path.join(destdir, filename)) - finally: - tinfoil.shutdown() + logger.info('Successfully deployed %s' % recipe_outdir) + + files_list = [] + for root, _, files in os.walk(recipe_outdir): + for filename in files: + filename = os.path.relpath(os.path.join(root, filename), recipe_outdir) + files_list.append(os.path.join(destdir, filename)) return 0 From patchwork Mon Jan 22 13:58:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 134BCC47DAF for ; Mon, 22 Jan 2024 13:59:31 +0000 (UTC) Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by mx.groups.io with SMTP id smtpd.web11.74370.1705931969660714168 for ; Mon, 22 Jan 2024 05:59:29 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=SGduewfj; spf=pass (domain: gmail.com, ip: 209.85.218.50, mailfrom: adrian.freihofer@gmail.com) Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a2d7e2e7fe0so541174666b.1 for ; Mon, 22 Jan 2024 05:59:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931968; x=1706536768; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q7PlUfy88FjNzANHDrqYCMCgrQGLLMgHZVn/zengsgA=; b=SGduewfjbFNPJmkZqYwZxfx06CGVMb7O/CeQQclxftHoY9kJb51G+j8jJnt28OccjV loP/cdk9h+NVCJ0CN8cL6d3MWkjk7XGdC9QgIjH0dAuz2lM2viAjedajXMsBbgMoVFdW VOPGrJTg8YtYvKbIgy0JA4qrxLCLG3zt1j8Qyu2XHvYIXEGgSnL6ycVGuMvcTdmm5xxZ qrxNQ1Jb2DlSiNqLlPpk5p9KbPUKoBa2PKad0DHxR2CJkGHTUGiCnqT58RH1qPyv1PBz lJ1S2jDd/M/xVK8QRjfU1EsGMIgiDxekYeJmcnhzhT2L0yBDZj10MTv2LrWN/GQaDaNA PahQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931968; x=1706536768; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q7PlUfy88FjNzANHDrqYCMCgrQGLLMgHZVn/zengsgA=; b=Kqi5i4RZZwMm5dtiwfCaFEN9beBK3g/FxnUsag7wlppdpssK3ewiqVBNniwlM322M+ X9pmLTvciD4+TAGFqmB9ERS1Id4fhgX7wEri62PglVk8umZjPFWCrC1IwM1kr9wIaYB1 t5xr9FkeLlCw2R4XbUJwGOj0meO37kMPpVf4CXVtNlP4HEIt8N6zw1wau93oGKAM++Xf 4jxuVYzu7GZ4c4RLuWmVyCsFa+WiqB+nmV2QeTmwqMI3D9sivraPvGJoX/ku490V8nhV 5Ak6vsTiTRWQcCAkXkEdqgo4OXz/6M+YbMSytdDT9F3Pyd7aAfkN9W3yQbivdDMJBPeR 74jg== X-Gm-Message-State: AOJu0YzVoS4aX8xU1VPBhOjLKRcU3p45xqbPfqZwqGcWAMjdJk3XykVy 6ViJROn88csS5O9yxRB81lyqdZc79KgvLGYx9Jrk+XPoNYD9/qZOwAEFivla X-Google-Smtp-Source: AGHT+IHf7A0RogfaTiJjLqaJ9vZw67xiosqReHUxrUCWHAtOBRZskmRn49zkAgAu/HjfIqizvSnrQw== X-Received: by 2002:a17:906:30db:b0:a27:5fd1:791f with SMTP id b27-20020a17090630db00b00a275fd1791fmr5576136ejb.0.1705931967954; Mon, 22 Jan 2024 05:59:27 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:27 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 8/9] devtool: ide-sdk make deploy-target quicker Date: Mon, 22 Jan 2024 14:58:26 +0100 Message-ID: <20240122135901.171596-9-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194157 Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/ide_sdk.py | 43 ++++++++++++++++++++++++++++++---- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/scripts/lib/devtool/ide_sdk.py b/scripts/lib/devtool/ide_sdk.py index 27389026be3..3986dc1436a 100755 --- a/scripts/lib/devtool/ide_sdk.py +++ b/scripts/lib/devtool/ide_sdk.py @@ -13,6 +13,7 @@ import re import shutil import stat import subprocess +import sys from argparse import RawTextHelpFormatter from enum import Enum @@ -20,7 +21,7 @@ import scriptutils import bb from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError, parse_recipe from devtool.standard import get_real_srctree -from devtool.ide_plugins import BuildTool, get_devtool_deploy_opts +from devtool.ide_plugins import BuildTool logger = logging.getLogger('devtool') @@ -742,6 +743,39 @@ class RecipeModified: return self.write_script(cmd_lines, 'delete_package_dirs') + def gen_deploy_target_script(self, args): + """Generate a script which does what devtool deploy-target does + + This script is much quicker than devtool target-deploy. Because it + does not need to start a bitbake server. All information from tinfoil + is hard-coded in the generated script. + """ + cmd_lines = ['#!/usr/bin/env python3'] + cmd_lines.append('import sys') + cmd_lines.append('devtool_sys_path = %s' % str(sys.path)) + cmd_lines.append('devtool_sys_path.reverse()') + cmd_lines.append('for p in devtool_sys_path:') + cmd_lines.append(' if p not in sys.path:') + cmd_lines.append(' sys.path.insert(0, p)') + cmd_lines.append('from devtool.deploy import deploy_no_d') + args_filter = ['debug', 'dry_run', 'key', 'no_check_space', 'no_host_check', + 'no_preserve', 'port', 'show_status', 'ssh_exec', 'strip', 'target'] + filtered_args_dict = {key: value for key, value in vars( + args).items() if key in args_filter} + cmd_lines.append('filtered_args_dict = %s' % str(filtered_args_dict)) + cmd_lines.append('class Dict2Class(object):') + cmd_lines.append(' def __init__(self, my_dict):') + cmd_lines.append(' for key in my_dict:') + cmd_lines.append(' setattr(self, key, my_dict[key])') + cmd_lines.append('filtered_args = Dict2Class(filtered_args_dict)') + cmd_lines.append( + 'setattr(filtered_args, "recipename", "%s")' % self.bpn) + cmd_lines.append('deploy_no_d("%s", "%s", "%s", "%s", "%s", "%s", %d, "%s", "%s", filtered_args)' % + (self.d, self.workdir, self.path, self.strip_cmd, + self.libdir, self.base_libdir, self.max_process, + self.fakerootcmd, self.fakerootenv)) + return self.write_script(cmd_lines, 'deploy_target') + def gen_install_deploy_script(self, args): """Generate a script which does install and deploy""" cmd_lines = ['#!/bin/bash'] @@ -759,10 +793,9 @@ class RecipeModified: cmd_lines.append( 'bitbake %s -c install --force || { echo "bitbake %s -c install --force failed"; exit 1; }' % (self.bpn, self.bpn)) - # devtool deploy-target - deploy_opts = ' '.join(get_devtool_deploy_opts(args)) - cmd_lines.append("devtool deploy-target %s %s" % - (self.bpn, deploy_opts)) + # Self contained devtool deploy-target + cmd_lines.append(self.gen_deploy_target_script(args)) + return self.write_script(cmd_lines, 'install_and_deploy') def write_script(self, cmd_lines, script_name): From patchwork Mon Jan 22 13:58:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 38133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80394C4725D for ; Mon, 22 Jan 2024 13:59:41 +0000 (UTC) Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) by mx.groups.io with SMTP id smtpd.web10.74264.1705931971316804621 for ; Mon, 22 Jan 2024 05:59:31 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YrxNFNQl; spf=pass (domain: gmail.com, ip: 209.85.167.45, mailfrom: adrian.freihofer@gmail.com) Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-50edf4f478eso4066259e87.3 for ; Mon, 22 Jan 2024 05:59:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705931969; x=1706536769; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vkA8JmIqu+tEZQPINCgxYk+1SogLmP0MfuItZDG+1ho=; b=YrxNFNQl0TMhkAIulnzcES9q5OKbfApTyvUw7PL+Jno3a9k5uczT1OXfG1k14pat4W Ih+pDIP/rcLaVcXfYNDIKlWmQt/9veVRC0EdCtX9nxXRuGiGb5XU50EIgS1vi7zIdWlL v4zvvbA8yPGtB7jBGbD4i4tCJVfFJekYtUGT2fXGsOoYhRrTtBeU5z04+x/UwEUn7sKI udojDMwUpKy7n1k6+vsOYqugsRAvCekDQs/N2lk5Qx29aeBSAGozB3d80aJGAObF7Sps y5i/vLcRbr4rXwV6A+w1mssmtw8yEK6rGkBCVHQxXn8ZrPyYRAPSGey4lTrBjYKUZPzx 5BRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705931969; x=1706536769; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vkA8JmIqu+tEZQPINCgxYk+1SogLmP0MfuItZDG+1ho=; b=xQ26UdRs4RoMoQPk9ZylujC6GyH+YTooUHC+A1sujjG6bx3ShMNCSamRVUyhvGQr0b i3IFx9jpTiQfeopcNDAXIPi0yvB1QVdVbdT5pe1ADcVNlj4iOKLDZs5tv39eyKk4hmSz ADpH0baLUUQ5OB1J809/pe4aZbuPiUmoEoaA0MeNjbYmQTb1cikvD4tuR245yz7k3iUw xpZ2N+oVELJ8PkjdR6YodEGm1Y5hOEFNxhaASkP1vQyWpaxIPo6dBIOLbnecqKUy9y5X Y1pNTZ5wK4f/8oAH8F7f9efdnw+FLcV7OmgtTB1eAsc93ZN/8IERtdurVrdglaHvUZBJ rdHQ== X-Gm-Message-State: AOJu0Yyi+Vqh4n0LU2wyMLC2H/Gfe6lubYwkobWDlKwC3ZIQaVYG9YYx rGKeL/bHM1ZIey2QjG+K4jUAOcs9qyXYdqPMJ6hk+7ccaidkDACpdf2FpzrJ X-Google-Smtp-Source: AGHT+IHG+SalTJnBNqzLJpRQ2+fvqSBlQl56AHx39s8Lm+j52Wai674hmgt1AkE5+UCUhnas7b/rSQ== X-Received: by 2002:ac2:5dee:0:b0:50e:44a4:d439 with SMTP id z14-20020ac25dee000000b0050e44a4d439mr1556377lfq.56.1705931968978; Mon, 22 Jan 2024 05:59:28 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id vw15-20020a170907a70f00b00a2d0595cd0dsm11568513ejc.86.2024.01.22.05.59.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 05:59:28 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v10 9/9] WIP: sdk-manual: extensible.rst: cover devtool ide-sdk Date: Mon, 22 Jan 2024 14:58:27 +0100 Message-ID: <20240122135901.171596-10-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240122135901.171596-1-adrian.freihofer@siemens.com> References: <20240122135901.171596-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Jan 2024 13:59:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194158 Cover the new devtool ide plugin in the extensible sdk section. Many thanks to Enguerrand de Ribaucourt for his re-view and contributions. Signed-off-by: Adrian Freihofer --- documentation/sdk-manual/extensible.rst | 257 +++++++++++++++++++++++- 1 file changed, 256 insertions(+), 1 deletion(-) diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst index 355c6cb0e4a..7a80f94a38b 100644 --- a/documentation/sdk-manual/extensible.rst +++ b/documentation/sdk-manual/extensible.rst @@ -63,6 +63,8 @@ their own pros and cons: need to provide a well-functioning binary artefact cache over the network for developers with underpowered laptops. +.. _setting_up_ext_sdk_in_build: + Setting up the Extensible SDK environment directly in a Yocto build ------------------------------------------------------------------- @@ -168,6 +170,8 @@ architecture. The example assumes the SDK installer is located in that case, set up the proper permissions in the directory and run the installer again. +.. _running_the_ext_sdk_env: + Running the Extensible SDK Environment Setup Script =================================================== @@ -205,6 +209,8 @@ use the SDK (e.g. ``PATH``, :term:`CC`, :term:`LD`, and so forth). If you want to see all the environment variables the script exports, examine the installation file itself. +.. _using_devtool: + Using ``devtool`` in Your SDK Workflow ====================================== @@ -230,13 +236,15 @@ all the commands. See the ":doc:`/ref-manual/devtool-reference`" section in the Yocto Project Reference Manual. -Three ``devtool`` subcommands provide entry-points into development: +``devtool`` subcommands provide entry-points into development: - *devtool add*: Assists in adding new software to be built. - *devtool modify*: Sets up an environment to enable you to modify the source of an existing component. +- *devtool ide-sdk*: Generates a configuration for an IDE. + - *devtool upgrade*: Updates an existing recipe so that you can build it for an updated set of source files. @@ -614,6 +622,253 @@ command: decide you do not want to proceed with your work. If you do use this command, realize that the source tree is preserved. +``devtool ide-sdk`` configures IDEs for the extensible SDK +---------------------------------------------------------- + +``devtool ide-sdk`` automatically configures IDEs to use the extensible SDK. +To make sure that all parts of the extensible SDK required by the generated IDE configuration are +available, ``devtool ide-sdk`` uses bitbake in the background to bootstrap the extensible SDK. + +The extensible SDK supports two different development modes. +``devtool ide-sdk`` supports both of them: + +#. *Modified mode*: + + By default ``devtool ide-sdk`` generates IDE configurations for recipes in workspaces + created by ``devtool modify`` or ``devtool add`` as described in :ref:`using_devtool`. + This mode creates IDE configurations with support for advanced features, such as deploying + the binaries to the remote target device and performing remote debugging sessions. + The generated IDE configurations use the per recipe sysroots as Bitbake does internally. + + In order to use the tool, a few settings must be made. + As a starting example, the following lines of code can be added to the ``local.conf`` file:: + + # Build the companion debug file system + IMAGE_GEN_DEBUGFS = "1" + # Optimize build time: with devtool ide-sdk the dbg tar is not needed + IMAGE_FSTYPES_DEBUGFS = "" + # Without copying the binaries into roofs-dbg, GDB does not find all source files. + IMAGE_CLASSES += "image-combined-dbg" + + # ssh is mandatory, no password simplifies the usage + EXTRA_IMAGE_FEATURES += "\ + ssh-server-openssh \ + debug-tweaks \ + " + + # Remote debugging needs the gdbserver on the target device + IMAGE_INSTALL:append = " gdbserver" + + # Add the recipes which should be modified to the image + # Otherwise some dependencies might be missing. + IMAGE_INSTALL:append = " my-recipe" + + Assuming the bitbake environment is set up correctly and a workspace has been created + for the recipe using ``devtool modify my-recipe``, the following command can create the + SDK and the configuration for VSCode in the recipe workspace:: + + $ devtool ide-sdk my-recipe core-image-minimal --target root@192.168.7.2 + + The command requires an image recipe (``core-image-minimal`` for this example) + that is used to create the SDK. + This firmware image should also be installed on the target device. + It is possible to pass multiple package recipes. + ``devtool ide-sdk`` tries to create an IDE configuration for all package recipes. + + Exactly what this command does depends on the recipe respectively on the build tool used by + the recipe. The basic idea is to configure the IDE so that it calls the build tool exactly + as ``bitbake`` does. + + For example, a CMake preset is created for a recipe that inherits :ref:`ref-classes-cmake`. + In the case of VSCode, CMake presets are supported by the CMake Tools plugin. + This is an example of how the build configuration used by ``bitbake`` is exported to an IDE + configuration that gives exactly the same build results. + + Support for remote debugging with seamless integration into the IDE is important for a cross-SDK. + ``devtool ide-sdk`` automatically generates the necessary helper scripts for deploying the compiled + artifacts to the target device as well as the necessary configuration for the debugger and the IDE. + + .. note:: + + To ensure that the debug symbols on the build machine match the binaries running on the target device, + it is essential that the image built by ``devtool ide-sdk`` is running on the target device. + + ``devtool ide-sdk`` aims to support multiple programming languages and multiple IDEs natively. + Native means that the IDE is configured to call the build tool (e.g. CMake or Meson) directly. + This has several advantages. First of all, it is much faster than ``devtool build``. + But it also allows to use the very good integration of tools like CMake or GDB directly with VSCode or other IDEs. + However, supporting many programming languages and multiple IDEs is quite an elaborate and constantly evolving thing. + Support for IDEs is therefore implemented as plugins. + Plugins can also be provided by optional layers. + + The default IDE is VSCode. Some hints about using VSCode: + + - To work on the source code of a recipe an instance of VSCode is started in the recipe's workspace. + Example:: + + code build/workspace/sources/my-recipe + + - To work with CMake press ``Ctrl + Shift + p``, type ``cmake``. + This will show some possible commands like selecting a CMake preset, compiling or running CTest. + + For recipes inheriting :ref:`ref-classes-cmake-qemu` rather than :ref:`ref-classes-cmake` executing + cross-compiled unit tests on the host can be supported transparently with QEMU user-mode. + + - To work with Meson press ``Ctrl + Shift + p``, type ``meson``. + This will show some possible commands like compiling or executing the unit tests. + + A note on running cross-compiled unit tests on the host: Meson enables support for QEMU user-mode by default. + It is expected that the execution of the unit tests from the IDE will work easily without any additional steps, + provided that the code is suitable for execution on the host machine. + + - For the deployment to the target device, just press ``Ctrl + Shift + p``, type ``task``. + Select the ``install && deploy-target``. + + - For remote debugging, switch to the debugging view by pressing the "play" button with the ``bug icon`` on the left side. + This will provide a green play button with a drop-down list where a debug configuration can be selected. + After selecting one of the generated configurations, press the "play" button. + + Starting a remote debugging session automatically initiates the deployment to the target device. + If this is not desired, the ``"dependsOn": ["install && deploy-target...]`` parameter of the tasks + with ``"label": "gdbserver start...`` can be removed from the ``tasks.json`` file. + + VSCode supports GDB with many different setups and configurations for many different use cases. + However, most of these setups have some limitations when it comes to cross-development, support only a few target + architectures or require a high performance target device. + Therefore ``devtool ide-sdk`` supports the classic, generic setup with GDB on the development host and gdbserver + on the target device. + Roughly summarized, this means: + + - The binaries are copied via ssh to the remote target device by a script referred by ``tasks.json``. + - gdbserver is started on the remote target device via ssh by a script referred by ``tasks.json``. + + Changing the parameters that are passed to the debugging executable requires changing the generated script. + The scipts is located at ``oe-scripts/gdbserver_*``. + The definition of the parameters in the ``args`` field in the ``launch.json`` file does not work. + + - VSCode connects to the gdbserver as documented in + `Remote debugging or debugging with a local debugger server `__. + + Additionally ``--ide=none`` is supported. + With the none IDE parameter some generic configurations files like ``gdbinit`` files and some helper scripts + starting the gdbserver remotely on the target device as well as the gdb client on the host are generated. + + Usage example for the cmake-example recipe from the meta-selftest layer + which inherits :ref:`ref-classes-cmake-qemu`: + + .. code-block:: sh + + # Create the SDK + devtool modify cmake-example + devtool ide-sdk cmake-example core-image-minimal -c --debug-build-config --ide=none + + # Install the firmware on a target device or start QEMU + runqemu + + # From a Navigate into the workspace of cmake-example + cd build/workspace/sources/cmake-example + + # Find cmake-native and save the path into a variable + # Note: using just cmake instead of $CMAKE_NATIVE would work in many cases + CMAKE_NATIVE="$(jq -r '.configurePresets[0] | "\(.cmakeExecutable)"' CMakeUserPresets.json)" + + # List available CMake presets + "$CMAKE_NATIVE" --list-presets + Available configure presets: + + "cmake-example-cortexa57" - cmake-example: cortexa57 + + # Re-compile the already compiled sources + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 + ninja: no work to do. + # Do a clean re-build + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target clean + [1/1] Cleaning all built files... + Cleaning... 8 files. + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target all + [7/7] Linking CXX executable cmake-example + + # Run the cross-compiled unit tests with QEMU user-mode + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target test + [0/1] Running tests... + Test project .../build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0 + Start 1: test-cmake-example + 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec + + 100% tests passed, 0 tests failed out of 1 + + Total Test time (real) = 0.03 sec + + # Using CTest directly is possible as well + CTEST_NATIVE="$(dirname "$CMAKE_NATIVE")/ctest" + + # List available CMake presets + "$CTEST_NATIVE" --list-presets + Available test presets: + + "cmake-example-cortexa57" - cmake-example: cortexa57 + + # Run the cross-compiled unit tests with QEMU user-mode + "$CTEST_NATIVE" --preset "cmake-example-cortexa57" + Test project ...build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0 + Start 1: test-cmake-example + 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec + + 100% tests passed, 0 tests failed out of 1 + + Total Test time (real) = 0.03 sec + + # Deploying the new build to the target device (default is QEUM at 192.168.7.2) + oe-scripts/install_and_deploy_cmake-example-cortexa57 + + # Start a remote debugging session with gdbserver on the target ang GDB on the host + oe-scripts/gdbserver_1234_usr-bin-cmake-example_m + oe-scripts/gdb_1234_usr-bin-cmake-example + break main + run + step + stepi + continue + quit + + # Stop the gdbserver on the target device + oe-scripts/gdbserver_1234_usr-bin-cmake-example_m stop + +#. *Shared sysroots mode* + + For some recipes and use cases a per-recipe sysroot based SDK is not suitable. + Optionally ``devtool ide-sdk`` configures the IDE to use the tool-chain provided by the + extensible SDK as described in :ref:`running_the_ext_sdk_env`. + ``devtool ide-sdk --mode=shared`` is basically a wrapper for the setup of the extensible SDK + as described in :ref:`setting_up_ext_sdk_in_build`. + The IDE gets configuration to use the shared sysroots. + + Creating a SDK with shared sysroots that contains all the dependencies needed to work with my-recipe is possible + with the following example command:: + + $ devtool ide-sdk --mode=shared my-recipe + + For VSCode the cross-tool-chain is exposed as a CMake kit. CMake kits are defined in + ``~/.local/share/CMakeTools/cmake-tools-kits.json``. + The following example shows how the cross-toolchain can be selected in VSCode. + Fist of all we need a folder containing a CMake project. + For this example lets create a CMake project and start VSCode:: + + mkdir kit-test + echo "project(foo VERSION 1.0)" > kit-test/CMakeLists.txt + code kit-test + + If there is a CMake project in the workspace cross-compilation is supported: + + - Press ``Ctrl + Shift + P``, type ``CMake: Scan for Kits`` + - Press ``Ctrl + Shift + P``, type ``CMake: Select a Kit`` + + Finally most of the features provided by CMake and the IDE should be available. + + Other IDEs than VSCode are supported as well. + However, ``devtool ide-sdk --mode=shared --ide=none my-recipe`` is currently just a simple wrapper + for the setup of the extensible SDK as described in :ref:`setting_up_ext_sdk_in_build`. + Use ``devtool upgrade`` to Create a Version of the Recipe that Supports a Newer Version of the Software -------------------------------------------------------------------------------------------------------