From patchwork Sun Jan 14 22:14:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D557C47DA9 for ; Sun, 14 Jan 2024 22:14:51 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web11.51506.1705270483822562154 for ; Sun, 14 Jan 2024 14:14:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=QZ3F08DQ; spf=pass (domain: gmail.com, ip: 209.85.128.50, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-40e775695c6so3928895e9.3 for ; Sun, 14 Jan 2024 14:14:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270482; x=1705875282; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v9nEmZRmXAXgJYwk+3uizUuvD61oxAZpP1cypbq4aKU=; b=QZ3F08DQwLKZKwQqT+5V75dtenjEpWZ26spthuAJ0eDGs7MXDO4xv0hraZbI2wpE1/ wBHazNYfdBwm9R1GAsZHxvPaRTmpOvSYZXtx6ibPbmZBt2W9l+KwGzQXgMIpkVAM98J/ QbgAAZe9/TtRSyA1gEgMJeii6V3f7CYRrdU/IMAxIpBOfrRQqBolXsYpFbRpCtIeDyje 5hCql4hhlIJP1ycISkwiwh7TS+6p4121GKh59ze0IcOViUKh8trRBeqOmOK1izcKszlR CEUt/QM/EgAROM06psteblxRG7HFk682EMUVW3rsrby8p7Y2gZMUe/Vxhym9A+B0AsaS fJ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270482; x=1705875282; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v9nEmZRmXAXgJYwk+3uizUuvD61oxAZpP1cypbq4aKU=; b=hHHH8qkSQGqIbqSzuXojmxQkcgejrzpxqVzWEqk/slTXWl3McJL6NlwJd5RtC/b2iW 2yCxGiil/bg91EMudPeycRSCLRNyuc78eP2mG6lfu6VeCCHAofQgcOZpTw40Le8zQrY0 dgGCAeaZ1Cqt3SaYe581+7FlXUXhyhCCLRr3yH64npemGZfhr+V1g5I9WIHFlrBeiuqm lkSWp4aAqWcE824VztUgXb+Hq7O1AuPU+vlgj/BXTbWmilMmNwtKdSgTgdkldo5qncO8 qVhgwD9iatt9VEM4diKJacLzeMFqr4qjh5DDfjE13S10woOuHH+iAVVsgmsAzqHD4qea VLmA== X-Gm-Message-State: AOJu0Yy6e6As3ckDH84SJ19PnBVZD77nvrxGaudxhv2njWD2BiGokPLD tkPm1D4PVA2cjEN/DWnRQs7RNaO4QpA= X-Google-Smtp-Source: AGHT+IGXrWNyF9wLL+KVI4e/Hqznju4aGxV/xaIn5WS9ZQ+rV8RjzOvJ+B1VJhes9nZEe1MSQNVYYQ== X-Received: by 2002:a05:600c:3147:b0:40e:4338:79cc with SMTP id h7-20020a05600c314700b0040e433879ccmr2845770wmo.8.1705270481878; Sun, 14 Jan 2024 14:14:41 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:41 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 1/9] cmake.bbclass: use --install Date: Sun, 14 Jan 2024 23:14:13 +0100 Message-ID: <20240114221437.1255866-2-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193619 Since version 3.15 CMake provides a command-line signature to install an already-generated project binary tree. This may be used after building a project to run installation without using the generated build system or the native build tool. This is a small improvement, for regular bitbake calls. CMake does not check the dependencies again which is expected to be faster. The main motivation for this change is using CMake from an SDK context. With this change it is possible to initiate the compile step from an IDE and later on initiating the install step via bitbake which runs the install step on pseudo. This is also what the meson.bbclass already does with the --no-rebuild option. Signed-off-by: Adrian Freihofer --- meta/classes-recipe/cmake.bbclass | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/cmake.bbclass b/meta/classes-recipe/cmake.bbclass index d978b889440..ff96f61c0eb 100644 --- a/meta/classes-recipe/cmake.bbclass +++ b/meta/classes-recipe/cmake.bbclass @@ -222,12 +222,24 @@ cmake_runcmake_build() { eval ${DESTDIR:+DESTDIR=${DESTDIR} }${CMAKE_VERBOSE} cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD} } +# Install an already-generated project binary tree. Not checking the compile +# dependencies again is particularly important for SDK use cases. +cmake_runcmake_install() { + bbnote ${DESTDIR:+DESTDIR=${DESTDIR} }${CMAKE_VERBOSE} cmake --install '${B}' + eval ${DESTDIR:+DESTDIR=${DESTDIR} }${CMAKE_VERBOSE} cmake --install '${B}' +} + cmake_do_compile() { cmake_runcmake_build --target ${OECMAKE_TARGET_COMPILE} } cmake_do_install() { - DESTDIR='${D}' cmake_runcmake_build --target ${OECMAKE_TARGET_INSTALL} + if [ "${OECMAKE_TARGET_INSTALL}" = "install" ]; then + DESTDIR='${D}' cmake_runcmake_install + else + # Legacy path which supports also custom install targets + DESTDIR='${D}' cmake_runcmake_build --target ${OECMAKE_TARGET_INSTALL} + fi } EXPORT_FUNCTIONS do_configure do_compile do_install do_generate_toolchain_file From patchwork Sun Jan 14 22:14:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37736 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FEF6C47DAC for ; Sun, 14 Jan 2024 22:14:51 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web10.52014.1705270484458795038 for ; Sun, 14 Jan 2024 14:14:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=T136c0jF; spf=pass (domain: gmail.com, ip: 209.85.128.50, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-40e7065b7bdso11145795e9.3 for ; Sun, 14 Jan 2024 14:14:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270483; x=1705875283; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R52fLyrsmSD3iAQqdDTFrY+4XPuWr/xbBfQ6SmMMg+o=; b=T136c0jF1Qhn3IdUCSYKdU570AkkLxR1iRtr259gDcIKLSmNLiLPAFOvUzQumjNQuA dotCHP36KNfVzKPtxqRg5rzg64CCmlYUf8Vr6DEu6SVtMJLVMQ1Sy2jIy06dl/wo/fXz F+IsotPNMgWG+szn5DRHd/2bQTcm2m+brQxNBssdAvIhq8K3y4JgcSEHA2+s27wvb23H licpgdrEDlUgYWzDQ5zsTopRG5S1pn9ZCjdwd0u8TZFCZpp+QcfJ6A8pJITOUzK2qPqI lmFbzv9cOjLNOjtwRSqodvKEdyXXrWOxn9ZmvSpOZX6zd3UeUxcwdZdOxI47mDaHJzTL bSxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270483; x=1705875283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R52fLyrsmSD3iAQqdDTFrY+4XPuWr/xbBfQ6SmMMg+o=; b=MSa5wgIFWb3OixW83kAbBAKSRGpgB8Q4Udmp2Q0q6RIBsO4tWlYTwh65L8qLevAu6V 0nyWkb8ktdcduRMJJ2TGTBT3u5IDUcO69U++ST0gxN0GzoM+f8n4XgXUlgH0uCZORdPB 7cvERLpUYGHOjD+jV9vlp0XEmrjsNGwkTOpKdhuTjOFbXqK9f0BCCTRG+e28o+wrcP8j nlf2n3ow6O7VYDwVqLdH8cuymmnM8e62JaVb+EdWThA4dL63BTxm4d0TjvcfKYQJbG3f /0qnQ7kAxf9NekX7PQt7N6/+e15YcDLOyjmD1RkizIhACaH1LlKbRVOLYKEDOrO7kLAG qsWA== X-Gm-Message-State: AOJu0YwYKIpy2u56RJM9AFEralTZzy3ydGHO8+48fv0GkFfJPYoUVMBJ 3WcnJqHehY2404J5wENlx/4Ggxhgvn0= X-Google-Smtp-Source: AGHT+IHdNt3gcuqU3aQ/EAiD9j5Hb8da7KQYjiLjnyb+E4zprRAMNln+ZM78xeShRsqoEifW40/ndg== X-Received: by 2002:a05:600c:a06:b0:40e:4181:a549 with SMTP id z6-20020a05600c0a0600b0040e4181a549mr1596241wmp.163.1705270482737; Sun, 14 Jan 2024 14:14:42 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:42 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 2/9] devtool: support plugins with plugins Date: Sun, 14 Jan 2024 23:14:14 +0100 Message-ID: <20240114221437.1255866-3-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193620 Pass the plugins search path to plugins via context. This allows plugins to search for their own plugins at the same paths. Signed-off-by: Adrian Freihofer --- scripts/devtool | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/devtool b/scripts/devtool index 3aae7b93b4b..60ea3e82985 100755 --- a/scripts/devtool +++ b/scripts/devtool @@ -299,8 +299,9 @@ def main(): return 2 # Search BBPATH first to allow layers to override plugins in scripts_path - for path in global_args.bbpath.split(':') + [scripts_path]: - pluginpath = os.path.join(path, 'lib', 'devtool') + pluginpaths = [os.path.join(path, 'lib', 'devtool') for path in global_args.bbpath.split(':') + [scripts_path]] + context.pluginpaths = pluginpaths + for pluginpath in pluginpaths: scriptutils.load_plugins(logger, plugins, pluginpath) subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='') From patchwork Sun Jan 14 22:14:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27A05C47DA7 for ; Sun, 14 Jan 2024 22:14:51 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web11.51508.1705270486075835650 for ; Sun, 14 Jan 2024 14:14:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=dKXS8dTO; spf=pass (domain: gmail.com, ip: 209.85.128.54, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-40e60e137aaso35329545e9.0 for ; Sun, 14 Jan 2024 14:14:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270484; x=1705875284; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hWpSKiM2z7T+3u9XNZ4QTSqaetmXbL+edh16bwGqz20=; b=dKXS8dTO2FHHR1JW1xlAMJNbWc4IBdYJpegwt8SkuGjnXLNpxlLCVHCS/2TizwPo5k uPhnsSR39gFKY5uLl7o7XrfMtbUItAZQDZI1X9W52eF8A1guC1tFtkJFy9FfJzJMBFlD NXaEXs2bg4f+DjiygbyrsMlxtuQ1WZAXTPIaX8aUUyzdCnr4sMvz2TevUVkSRQ45kCwh jMogVUdHRxhNxZ+DKDZ2i6kddcAzljx52o4KjvJVN+s1brCCfePgDXt/UzhNeWAWcU0X +XORCvZbDnr15hHJLZdXY0DjWEOd6ZHxV2hzE3unt+iIaM7qcnHvq7oPgOAQPNVtJyJj RIAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270484; x=1705875284; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hWpSKiM2z7T+3u9XNZ4QTSqaetmXbL+edh16bwGqz20=; b=H+pfz7uIOHp0DxQWRm24FQzIDVHy5tQN93AzjocpQBmN0Zx+wI9ivN5gNx9COOFYs/ 3ZXF+76mS+obHquoBSH4f+fDBUPFRI2D+o7twkyFjVjSnMrJmj77HtoNmUiuBYJEBbid Eti/AAa704MQgLei+2qfoxr20Oib5APCDIIznckAyHl3VxkCGPlB27Ec2FDB5e1+JSZR m6NPNRZwzIyBRsAeaGf3xO5067wSFFjo+UILT7cTIuPWKkYc9e2VZeUxh8ldPzhiDMfh tvODgzFwV5Wk9JTgWkWjpe85B1XlstOAvZLO2rC/d8s7GwGckOf4jF1D2bt16U14EoyQ LDxA== X-Gm-Message-State: AOJu0YzP8mWkoshBFHAnwkDbzvUx0KTedCqykP2XluAfsdw7/iXRRaew UTidLn1lnu4hQEXElyZhNWEj5e8+kG8= X-Google-Smtp-Source: AGHT+IEiK0towhTHnB1ftd+q7jl24ogrjGj0qv2fDkoLjfJNNJ2CT+6BA6ySuNsGaEqOGat+iHVXiQ== X-Received: by 2002:a05:600c:ad3:b0:40e:4572:57ea with SMTP id c19-20020a05600c0ad300b0040e457257eamr2751836wmr.65.1705270483818; Sun, 14 Jan 2024 14:14:43 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:43 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 3/9] devtool: new ide-sdk plugin Date: Sun, 14 Jan 2024 23:14:15 +0100 Message-ID: <20240114221437.1255866-4-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193621 The new devtool ide plugin provides the eSDK and configures an IDE to work with the eSDK. In doing so, bitbake should be used to generate the IDE configuration and update the SDK, but it should no longer play a role when working on the source code. The work on the source code should take place exclusively with the IDE, which, for example, calls cmake directly to compile the code and execute the unit tests from the IDE. The plugin works for recipes inheriting the cmake or the meson bbclass. Support for more programming languages and build tools may be added in the future. There are various IDEs that can be used for the development of embedded Linux applications. Therefore, devtool ide-sdk, like devtool itself, supports plugins to support IDEs. VSCode is the default IDE for this first implementation. Additionally, some generic helper scripts can be generated with --ide none instead of a specific IDE configuration. This can be used for any IDE that supports calling some scripts. There are two different modes supported: - devtool modify mode (default): devtool ide-sdk configures the IDE to manage the build-tool used by the recipe (e.g. cmake or meson). The workflow looks like: $ devtool modify a-recipe $ devtool ide-sdk a-recipe a-image $ code "$BUILDDIR/workspace/sources/a-recipe" Work in VSCode, after installing the proposed plugins Deploying the artifacts to the target device and running a remote debugging session is supported as well. This first implementation still calls bitbake and devtool to copy the binary artifacts to the target device. In contrast to compiling, installation and copying must be performed with the file rights of the target device. The pseudo tool must be used for this. Therefore bitbake -c install a-recipe && devtool deploy-target a-recipe are called by the IDE for the deployment. This might be improved later on. Executing the unit tests out of the IDE is supported via Qemu user if the build tool supports that. CMake (if cmake-qemu.bbclass is inherited) and Meson support Qemu usermode. - Shared sysroots mode: bootstraps the eSDK with shared sysroots for all the recipes passed to devtool ide-sdk. This is basically a wrapper for bitbake meta-ide-support && bitbake build-sysroots. The workflow looks like: $ devtool ide-sdk --share-sysroots a-recipe another-recipe vscode where/the/sources/are If the IDE and the build tool support it, the IDE gets configured to offer the cross tool-chain provided by the eSDK. In case of VSCode and cmake a cmake-kit is generated. This offers to use the cross tool-chain from the UI of the IDE. Many thanks to Enguerrand de Ribaucourt for testing and bug fixing. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/ide_plugins/__init__.py | 267 +++++ scripts/lib/devtool/ide_plugins/ide_code.py | 432 ++++++++ scripts/lib/devtool/ide_plugins/ide_none.py | 53 + scripts/lib/devtool/ide_sdk.py | 1030 +++++++++++++++++++ 4 files changed, 1782 insertions(+) create mode 100644 scripts/lib/devtool/ide_plugins/__init__.py create mode 100644 scripts/lib/devtool/ide_plugins/ide_code.py create mode 100644 scripts/lib/devtool/ide_plugins/ide_none.py create mode 100755 scripts/lib/devtool/ide_sdk.py diff --git a/scripts/lib/devtool/ide_plugins/__init__.py b/scripts/lib/devtool/ide_plugins/__init__.py new file mode 100644 index 00000000000..3371b242640 --- /dev/null +++ b/scripts/lib/devtool/ide_plugins/__init__.py @@ -0,0 +1,267 @@ +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk IDE plugin interface definition and helper functions""" + +import errno +import json +import logging +import os +import stat +from enum import Enum, auto +from devtool import DevtoolError +from bb.utils import mkdirhier + +logger = logging.getLogger('devtool') + + +class BuildTool(Enum): + UNDEFINED = auto() + CMAKE = auto() + MESON = auto() + + @property + def is_c_ccp(self): + if self is BuildTool.CMAKE: + return True + if self is BuildTool.MESON: + return True + return False + + +class GdbCrossConfig: + """Base class defining the GDB configuration generator interface + + Generate a GDB configuration for a binary on the target device. + Only one instance per binary is allowed. This allows to assign unique port + numbers for all gdbserver instances. + """ + _gdbserver_port_next = 1234 + _binaries = [] + + def __init__(self, image_recipe, modified_recipe, binary, gdbserver_multi=True): + self.image_recipe = image_recipe + self.modified_recipe = modified_recipe + self.gdb_cross = modified_recipe.gdb_cross + self.binary = binary + if binary in GdbCrossConfig._binaries: + raise DevtoolError( + "gdbserver config for binary %s is already generated" % binary) + GdbCrossConfig._binaries.append(binary) + self.script_dir = modified_recipe.ide_sdk_scripts_dir + self.gdbinit_dir = os.path.join(self.script_dir, 'gdbinit') + self.gdbserver_multi = gdbserver_multi + self.binary_pretty = self.binary.replace(os.sep, '-').lstrip('-') + self.gdbserver_port = GdbCrossConfig._gdbserver_port_next + GdbCrossConfig._gdbserver_port_next += 1 + self.id_pretty = "%d_%s" % (self.gdbserver_port, self.binary_pretty) + # gdbserver start script + gdbserver_script_file = 'gdbserver_' + self.id_pretty + if self.gdbserver_multi: + gdbserver_script_file += "_m" + self.gdbserver_script = os.path.join( + self.script_dir, gdbserver_script_file) + # gdbinit file + self.gdbinit = os.path.join( + self.gdbinit_dir, 'gdbinit_' + self.id_pretty) + # gdb start script + self.gdb_script = os.path.join( + self.script_dir, 'gdb_' + self.id_pretty) + + def _gen_gdbserver_start_script(self): + """Generate a shell command starting the gdbserver on the remote device via ssh + + GDB supports two modes: + multi: gdbserver remains running over several debug sessions + once: gdbserver terminates after the debugged process terminates + """ + cmd_lines = ['#!/bin/sh'] + if self.gdbserver_multi: + temp_dir = "TEMP_DIR=/tmp/gdbserver_%s; " % self.id_pretty + gdbserver_cmd_start = temp_dir + gdbserver_cmd_start += "test -f \$TEMP_DIR/pid && exit 0; " + gdbserver_cmd_start += "mkdir -p \$TEMP_DIR; " + gdbserver_cmd_start += "%s --multi :%s > \$TEMP_DIR/log 2>&1 & " % ( + self.gdb_cross.gdbserver_path, self.gdbserver_port) + gdbserver_cmd_start += "echo \$! > \$TEMP_DIR/pid;" + + gdbserver_cmd_stop = temp_dir + gdbserver_cmd_stop += "test -f \$TEMP_DIR/pid && kill \$(cat \$TEMP_DIR/pid); " + gdbserver_cmd_stop += "rm -rf \$TEMP_DIR; " + + gdbserver_cmd_l = [] + gdbserver_cmd_l.append('if [ "$1" = "stop" ]; then') + gdbserver_cmd_l.append(' shift') + gdbserver_cmd_l.append(" %s %s %s %s 'sh -c \"%s\"'" % ( + self.gdb_cross.target_device.ssh_sshexec, self.gdb_cross.target_device.ssh_port, self.gdb_cross.target_device.extraoptions, self.gdb_cross.target_device.target, gdbserver_cmd_stop)) + gdbserver_cmd_l.append('else') + gdbserver_cmd_l.append(" %s %s %s %s 'sh -c \"%s\"'" % ( + self.gdb_cross.target_device.ssh_sshexec, self.gdb_cross.target_device.ssh_port, self.gdb_cross.target_device.extraoptions, self.gdb_cross.target_device.target, gdbserver_cmd_start)) + gdbserver_cmd_l.append('fi') + gdbserver_cmd = os.linesep.join(gdbserver_cmd_l) + else: + gdbserver_cmd_start = "%s --once :%s %s" % ( + self.gdb_cross.gdbserver_path, self.gdbserver_port, self.binary) + gdbserver_cmd = "%s %s %s %s 'sh -c \"%s\"'" % ( + self.gdb_cross.target_device.ssh_sshexec, self.gdb_cross.target_device.ssh_port, self.gdb_cross.target_device.extraoptions, self.gdb_cross.target_device.target, gdbserver_cmd_start) + cmd_lines.append(gdbserver_cmd) + GdbCrossConfig.write_file(self.gdbserver_script, cmd_lines, True) + + def _gen_gdbinit_config(self): + """Generate a gdbinit file for this binary and the corresponding gdbserver configuration""" + gdbinit_lines = ['# This file is generated by devtool ide-sdk'] + if self.gdbserver_multi: + target_help = '# gdbserver --multi :%d' % self.gdbserver_port + remote_cmd = 'target extended-remote' + else: + target_help = '# gdbserver :%d %s' % ( + self.gdbserver_port, self.binary) + remote_cmd = 'target remote' + gdbinit_lines.append('# On the remote target:') + gdbinit_lines.append(target_help) + gdbinit_lines.append('# On the build machine:') + gdbinit_lines.append('# cd ' + self.modified_recipe.real_srctree) + gdbinit_lines.append( + '# ' + self.gdb_cross.gdb + ' -ix ' + self.gdbinit) + + gdbinit_lines.append('set sysroot ' + self.modified_recipe.d) + gdbinit_lines.append('set substitute-path "/usr/include" "' + + os.path.join(self.modified_recipe.recipe_sysroot, 'usr', 'include') + '"') + # Disable debuginfod for now, the IDE configuration uses rootfs-dbg from the image workdir. + gdbinit_lines.append('set debuginfod enabled off') + if self.image_recipe.rootfs_dbg: + gdbinit_lines.append( + 'set solib-search-path "' + self.modified_recipe.solib_search_path_str(self.image_recipe) + '"') + gdbinit_lines.append('set substitute-path "/usr/src/debug" "' + os.path.join( + self.image_recipe.rootfs_dbg, 'usr', 'src', 'debug') + '"') + gdbinit_lines.append( + '%s %s:%d' % (remote_cmd, self.gdb_cross.host, self.gdbserver_port)) + gdbinit_lines.append('set remote exec-file ' + self.binary) + gdbinit_lines.append( + 'run ' + os.path.join(self.modified_recipe.d, self.binary)) + + GdbCrossConfig.write_file(self.gdbinit, gdbinit_lines) + + def _gen_gdb_start_script(self): + """Generate a script starting GDB with the corresponding gdbinit configuration.""" + cmd_lines = ['#!/bin/sh'] + cmd_lines.append('cd ' + self.modified_recipe.real_srctree) + cmd_lines.append(self.gdb_cross.gdb + ' -ix ' + + self.gdbinit + ' "$@"') + GdbCrossConfig.write_file(self.gdb_script, cmd_lines, True) + + def initialize(self): + self._gen_gdbserver_start_script() + self._gen_gdbinit_config() + self._gen_gdb_start_script() + + @staticmethod + def write_file(script_file, cmd_lines, executable=False): + script_dir = os.path.dirname(script_file) + mkdirhier(script_dir) + with open(script_file, 'w') as script_f: + script_f.write(os.linesep.join(cmd_lines)) + script_f.write(os.linesep) + if executable: + st = os.stat(script_file) + os.chmod(script_file, st.st_mode | stat.S_IEXEC) + logger.info("Created: %s" % script_file) + + +class IdeBase: + """Base class defining the interface for IDE plugins""" + + def __init__(self): + self.ide_name = 'undefined' + self.gdb_cross_configs = [] + + @classmethod + def ide_plugin_priority(cls): + """Used to find the default ide handler if --ide is not passed""" + return 10 + + def setup_shared_sysroots(self, shared_env): + logger.warn("Shared sysroot mode is not supported for IDE %s" % + self.ide_name) + + def setup_modified_recipe(self, args, image_recipe, modified_recipe): + logger.warn("Modified recipe mode is not supported for IDE %s" % + self.ide_name) + + def initialize_gdb_cross_configs(self, image_recipe, modified_recipe, gdb_cross_config_class=GdbCrossConfig): + binaries = modified_recipe.find_installed_binaries() + for binary in binaries: + gdb_cross_config = gdb_cross_config_class( + image_recipe, modified_recipe, binary) + gdb_cross_config.initialize() + self.gdb_cross_configs.append(gdb_cross_config) + + @staticmethod + def gen_oe_scrtips_sym_link(modified_recipe): + # create a sym-link from sources to the scripts directory + if os.path.isdir(modified_recipe.ide_sdk_scripts_dir): + IdeBase.symlink_force(modified_recipe.ide_sdk_scripts_dir, + os.path.join(modified_recipe.real_srctree, 'oe-scripts')) + + @staticmethod + def update_json_file(json_dir, json_file, update_dict): + """Update a json file + + By default it uses the dict.update function. If this is not sutiable + the update function might be passed via update_func parameter. + """ + json_path = os.path.join(json_dir, json_file) + logger.info("Updating IDE config file: %s (%s)" % + (json_file, json_path)) + if not os.path.exists(json_dir): + os.makedirs(json_dir) + try: + with open(json_path) as f: + orig_dict = json.load(f) + except json.decoder.JSONDecodeError: + logger.info( + "Decoding %s failed. Probably because of comments in the json file" % json_path) + orig_dict = {} + except FileNotFoundError: + orig_dict = {} + orig_dict.update(update_dict) + with open(json_path, 'w') as f: + json.dump(orig_dict, f, indent=4) + + @staticmethod + def symlink_force(tgt, dst): + try: + os.symlink(tgt, dst) + except OSError as err: + if err.errno == errno.EEXIST: + if os.readlink(dst) != tgt: + os.remove(dst) + os.symlink(tgt, dst) + else: + raise err + + +def get_devtool_deploy_opts(args): + """Filter args for devtool deploy-target args""" + if not args.target: + return None + devtool_deploy_opts = [args.target] + if args.no_host_check: + devtool_deploy_opts += ["-c"] + if args.show_status: + devtool_deploy_opts += ["-s"] + if args.no_preserve: + devtool_deploy_opts += ["-p"] + if args.no_check_space: + devtool_deploy_opts += ["--no-check-space"] + if args.ssh_exec: + devtool_deploy_opts += ["-e", args.ssh.exec] + if args.port: + devtool_deploy_opts += ["-P", args.port] + if args.key: + devtool_deploy_opts += ["-I", args.key] + if args.strip is False: + devtool_deploy_opts += ["--no-strip"] + return devtool_deploy_opts diff --git a/scripts/lib/devtool/ide_plugins/ide_code.py b/scripts/lib/devtool/ide_plugins/ide_code.py new file mode 100644 index 00000000000..a8aa3ed07e4 --- /dev/null +++ b/scripts/lib/devtool/ide_plugins/ide_code.py @@ -0,0 +1,432 @@ +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk IDE plugin for VSCode and VSCodium""" + +import json +import logging +import os +import shutil +from devtool.ide_plugins import BuildTool, IdeBase, GdbCrossConfig, get_devtool_deploy_opts + +logger = logging.getLogger('devtool') + + +class GdbCrossConfigVSCode(GdbCrossConfig): + def __init__(self, image_recipe, modified_recipe, binary): + super().__init__(image_recipe, modified_recipe, binary, False) + + def initialize(self): + self._gen_gdbserver_start_script() + + +class IdeVSCode(IdeBase): + """Manage IDE configurations for VSCode + + Modified recipe mode: + - cmake: use the cmake-preset generated by devtool ide-sdk + - meson: meson is called via a wrapper script generated by devtool ide-sdk + + Shared sysroot mode: + In shared sysroot mode, the cross tool-chain is exported to the user's global configuration. + A workspace cannot be created because there is no recipe that defines how a workspace could + be set up. + - cmake: adds a cmake-kit to .local/share/CMakeTools/cmake-tools-kits.json + The cmake-kit uses the environment script and the tool-chain file + generated by meta-ide-support. + - meson: Meson needs manual workspace configuration. + """ + + @classmethod + def ide_plugin_priority(cls): + """If --ide is not passed this is the default plugin""" + if shutil.which('code'): + return 100 + return 0 + + def setup_shared_sysroots(self, shared_env): + """Expose the toolchain of the shared sysroots SDK""" + datadir = shared_env.ide_support.datadir + deploy_dir_image = shared_env.ide_support.deploy_dir_image + real_multimach_target_sys = shared_env.ide_support.real_multimach_target_sys + standalone_sysroot_native = shared_env.build_sysroots.standalone_sysroot_native + vscode_ws_path = os.path.join( + os.environ['HOME'], '.local', 'share', 'CMakeTools') + cmake_kits_path = os.path.join(vscode_ws_path, 'cmake-tools-kits.json') + oecmake_generator = "Ninja" + env_script = os.path.join( + deploy_dir_image, 'environment-setup-' + real_multimach_target_sys) + + if not os.path.isdir(vscode_ws_path): + os.makedirs(vscode_ws_path) + cmake_kits_old = [] + if os.path.exists(cmake_kits_path): + with open(cmake_kits_path, 'r', encoding='utf-8') as cmake_kits_file: + cmake_kits_old = json.load(cmake_kits_file) + cmake_kits = cmake_kits_old.copy() + + cmake_kit_new = { + "name": "OE " + real_multimach_target_sys, + "environmentSetupScript": env_script, + "toolchainFile": standalone_sysroot_native + datadir + "/cmake/OEToolchainConfig.cmake", + "preferredGenerator": { + "name": oecmake_generator + } + } + + def merge_kit(cmake_kits, cmake_kit_new): + i = 0 + while i < len(cmake_kits): + if 'environmentSetupScript' in cmake_kits[i] and \ + cmake_kits[i]['environmentSetupScript'] == cmake_kit_new['environmentSetupScript']: + cmake_kits[i] = cmake_kit_new + return + i += 1 + cmake_kits.append(cmake_kit_new) + merge_kit(cmake_kits, cmake_kit_new) + + if cmake_kits != cmake_kits_old: + logger.info("Updating: %s" % cmake_kits_path) + with open(cmake_kits_path, 'w', encoding='utf-8') as cmake_kits_file: + json.dump(cmake_kits, cmake_kits_file, indent=4) + else: + logger.info("Already up to date: %s" % cmake_kits_path) + + cmake_native = os.path.join( + shared_env.build_sysroots.standalone_sysroot_native, 'usr', 'bin', 'cmake') + if os.path.isfile(cmake_native): + logger.info('cmake-kits call cmake by default. If the cmake provided by this SDK should be used, please add the following line to ".vscode/settings.json" file: "cmake.cmakePath": "%s"' % cmake_native) + else: + logger.error("Cannot find cmake native at: %s" % cmake_native) + + def dot_code_dir(self, modified_recipe): + return os.path.join(modified_recipe.srctree, '.vscode') + + def __vscode_settings_meson(self, settings_dict, modified_recipe): + if modified_recipe.build_tool is not BuildTool.MESON: + return + settings_dict["mesonbuild.mesonPath"] = modified_recipe.meson_wrapper + + confopts = modified_recipe.mesonopts.split() + confopts += modified_recipe.meson_cross_file.split() + confopts += modified_recipe.extra_oemeson.split() + settings_dict["mesonbuild.configureOptions"] = confopts + settings_dict["mesonbuild.buildFolder"] = modified_recipe.b + + def __vscode_settings_cmake(self, settings_dict, modified_recipe): + """Add cmake specific settings to settings.json. + + Note: most settings are passed to the cmake preset. + """ + if modified_recipe.build_tool is not BuildTool.CMAKE: + return + settings_dict["cmake.configureOnOpen"] = True + settings_dict["cmake.sourceDirectory"] = modified_recipe.real_srctree + + def vscode_settings(self, modified_recipe): + files_excludes = { + "**/.git/**": True, + "**/oe-logs/**": True, + "**/oe-workdir/**": True, + "**/source-date-epoch/**": True + } + python_exclude = [ + "**/.git/**", + "**/oe-logs/**", + "**/oe-workdir/**", + "**/source-date-epoch/**" + ] + settings_dict = { + "files.watcherExclude": files_excludes, + "files.exclude": files_excludes, + "python.analysis.exclude": python_exclude + } + self.__vscode_settings_cmake(settings_dict, modified_recipe) + self.__vscode_settings_meson(settings_dict, modified_recipe) + + settings_file = 'settings.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), settings_file, settings_dict) + + def __vscode_extensions_cmake(self, modified_recipe, recommendations): + if modified_recipe.build_tool is not BuildTool.CMAKE: + return + recommendations += [ + "twxs.cmake", + "ms-vscode.cmake-tools", + "ms-vscode.cpptools", + "ms-vscode.cpptools-extension-pack", + "ms-vscode.cpptools-themes" + ] + + def __vscode_extensions_meson(self, modified_recipe, recommendations): + if modified_recipe.build_tool is not BuildTool.MESON: + return + recommendations += [ + 'mesonbuild.mesonbuild', + "ms-vscode.cpptools", + "ms-vscode.cpptools-extension-pack", + "ms-vscode.cpptools-themes" + ] + + def vscode_extensions(self, modified_recipe): + recommendations = [] + self.__vscode_extensions_cmake(modified_recipe, recommendations) + self.__vscode_extensions_meson(modified_recipe, recommendations) + extensions_file = 'extensions.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), extensions_file, {"recommendations": recommendations}) + + def vscode_c_cpp_properties(self, modified_recipe): + properties_dict = { + "name": modified_recipe.recipe_id_pretty, + } + if modified_recipe.build_tool is BuildTool.CMAKE: + properties_dict["configurationProvider"] = "ms-vscode.cmake-tools" + elif modified_recipe.build_tool is BuildTool.MESON: + properties_dict["configurationProvider"] = "mesonbuild.mesonbuild" + else: # no C/C++ build + return + + properties_dicts = { + "configurations": [ + properties_dict + ], + "version": 4 + } + prop_file = 'c_cpp_properties.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), prop_file, properties_dicts) + + def vscode_launch_bin_dbg(self, gdb_cross_config): + modified_recipe = gdb_cross_config.modified_recipe + + launch_config = { + "name": gdb_cross_config.id_pretty, + "type": "cppdbg", + "request": "launch", + "program": os.path.join(modified_recipe.d, gdb_cross_config.binary.lstrip('/')), + "stopAtEntry": True, + "cwd": "${workspaceFolder}", + "environment": [], + "externalConsole": False, + "MIMode": "gdb", + "preLaunchTask": gdb_cross_config.id_pretty, + "miDebuggerPath": modified_recipe.gdb_cross.gdb, + "miDebuggerServerAddress": "%s:%d" % (modified_recipe.gdb_cross.host, gdb_cross_config.gdbserver_port) + } + + # Search for header files in recipe-sysroot. + src_file_map = { + "/usr/include": os.path.join(modified_recipe.recipe_sysroot, "usr", "include") + } + # First of all search for not stripped binaries in the image folder. + # These binaries are copied (and optionally stripped) by deploy-target + setup_commands = [ + { + "description": "sysroot", + "text": "set sysroot " + modified_recipe.d + } + ] + + if gdb_cross_config.image_recipe.rootfs_dbg: + launch_config['additionalSOLibSearchPath'] = modified_recipe.solib_search_path_str( + gdb_cross_config.image_recipe) + src_file_map["/usr/src/debug"] = os.path.join( + gdb_cross_config.image_recipe.rootfs_dbg, "usr", "src", "debug") + else: + logger.warning( + "Cannot setup debug symbols configuration for GDB. IMAGE_GEN_DEBUGFS is not enabled.") + + launch_config['sourceFileMap'] = src_file_map + launch_config['setupCommands'] = setup_commands + return launch_config + + def vscode_launch(self, modified_recipe): + """GDB Launch configuration for binaries (elf files)""" + + configurations = [self.vscode_launch_bin_dbg( + gdb_cross_config) for gdb_cross_config in self.gdb_cross_configs] + launch_dict = { + "version": "0.2.0", + "configurations": configurations + } + launch_file = 'launch.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), launch_file, launch_dict) + + def vscode_tasks_cpp(self, args, modified_recipe): + run_install_deploy = modified_recipe.gen_install_deploy_script(args) + install_task_name = "install && deploy-target %s" % modified_recipe.recipe_id_pretty + tasks_dict = { + "version": "2.0.0", + "tasks": [ + { + "label": install_task_name, + "type": "shell", + "command": run_install_deploy, + "problemMatcher": [] + } + ] + } + for gdb_cross_config in self.gdb_cross_configs: + tasks_dict['tasks'].append( + { + "label": gdb_cross_config.id_pretty, + "type": "shell", + "isBackground": True, + "dependsOn": [ + install_task_name + ], + "command": gdb_cross_config.gdbserver_script, + "problemMatcher": [ + { + "pattern": [ + { + "regexp": ".", + "file": 1, + "location": 2, + "message": 3 + } + ], + "background": { + "activeOnStart": True, + "beginsPattern": ".", + "endsPattern": ".", + } + } + ] + }) + tasks_file = 'tasks.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), tasks_file, tasks_dict) + + def vscode_tasks_fallback(self, args, modified_recipe): + oe_init_dir = modified_recipe.oe_init_dir + oe_init = ". %s > /dev/null && " % modified_recipe.oe_init_build_env + task_devtool_build = "devtool build %s" % modified_recipe.recipe_id_pretty + task_devtool_build_clean = "devtool build %s --clean" % modified_recipe.recipe_id_pretty + task_devtool_deploy = "devtool deploy-target %s" % modified_recipe.recipe_id_pretty + task_devtool_build_deploy = "devtool build & deploy-target %s" % modified_recipe.recipe_id_pretty + deploy_opts = ' '.join(get_devtool_deploy_opts(args)) + tasks_dict = { + "version": "2.0.0", + "tasks": [ + { + "label": task_devtool_build, + "type": "shell", + "command": "bash", + "linux": { + "options": { + "cwd": oe_init_dir + } + }, + "args": [ + "--login", + "-c", + "%s%s" % (oe_init, task_devtool_build) + ], + "problemMatcher": [] + }, + { + "label": task_devtool_deploy, + "type": "shell", + "command": "bash", + "linux": { + "options": { + "cwd": oe_init_dir + } + }, + "args": [ + "--login", + "-c", + "%s%s %s" % ( + oe_init, task_devtool_deploy, deploy_opts) + ], + "problemMatcher": [] + }, + { + "label": task_devtool_build_deploy, + "dependsOrder": "sequence", + "dependsOn": [ + task_devtool_build, + task_devtool_deploy + ], + "problemMatcher": [], + "group": { + "kind": "build", + "isDefault": True + } + }, + { + "label": task_devtool_build_clean, + "type": "shell", + "command": "bash", + "linux": { + "options": { + "cwd": oe_init_dir + } + }, + "args": [ + "--login", + "-c", + "%s%s" % (oe_init, task_devtool_build_clean) + ], + "problemMatcher": [] + } + ] + } + if modified_recipe.gdb_cross: + for gdb_cross_config in self.gdb_cross_configs: + tasks_dict['tasks'].append( + { + "label": gdb_cross_config.id_pretty, + "type": "shell", + "isBackground": True, + "dependsOn": [ + task_devtool_build_deploy + ], + "command": gdb_cross_config.gdbserver_script, + "problemMatcher": [ + { + "pattern": [ + { + "regexp": ".", + "file": 1, + "location": 2, + "message": 3 + } + ], + "background": { + "activeOnStart": True, + "beginsPattern": ".", + "endsPattern": ".", + } + } + ] + }) + tasks_file = 'tasks.json' + IdeBase.update_json_file( + self.dot_code_dir(modified_recipe), tasks_file, tasks_dict) + + def vscode_tasks(self, args, modified_recipe): + if modified_recipe.build_tool.is_c_ccp: + self.vscode_tasks_cpp(args, modified_recipe) + else: + self.vscode_tasks_fallback(args, modified_recipe) + + def setup_modified_recipe(self, args, image_recipe, modified_recipe): + self.vscode_settings(modified_recipe) + self.vscode_extensions(modified_recipe) + self.vscode_c_cpp_properties(modified_recipe) + if args.target: + self.initialize_gdb_cross_configs( + image_recipe, modified_recipe, gdb_cross_config_class=GdbCrossConfigVSCode) + self.vscode_launch(modified_recipe) + self.vscode_tasks(args, modified_recipe) + + +def register_ide_plugin(ide_plugins): + ide_plugins['code'] = IdeVSCode diff --git a/scripts/lib/devtool/ide_plugins/ide_none.py b/scripts/lib/devtool/ide_plugins/ide_none.py new file mode 100644 index 00000000000..f106c5a0269 --- /dev/null +++ b/scripts/lib/devtool/ide_plugins/ide_none.py @@ -0,0 +1,53 @@ +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk generic IDE plugin""" + +import os +import logging +from devtool.ide_plugins import IdeBase, GdbCrossConfig + +logger = logging.getLogger('devtool') + + +class IdeNone(IdeBase): + """Generate some generic helpers for other IDEs + + Modified recipe mode: + Generate some helper scripts for remote debugging with GDB + + Shared sysroot mode: + A wrapper for bitbake meta-ide-support and bitbake build-sysroots + """ + + def __init__(self): + super().__init__() + + def setup_shared_sysroots(self, shared_env): + real_multimach_target_sys = shared_env.ide_support.real_multimach_target_sys + deploy_dir_image = shared_env.ide_support.deploy_dir_image + env_script = os.path.join( + deploy_dir_image, 'environment-setup-' + real_multimach_target_sys) + logger.info( + "To use this SDK please source this: %s" % env_script) + + def setup_modified_recipe(self, args, image_recipe, modified_recipe): + """generate some helper scripts and config files + + - Execute the do_install task + - Execute devtool deploy-target + - Generate a gdbinit file per executable + - Generate the oe-scripts sym-link + """ + script_path = modified_recipe.gen_install_deploy_script(args) + logger.info("Created: %s" % script_path) + + self.initialize_gdb_cross_configs(image_recipe, modified_recipe) + + IdeBase.gen_oe_scrtips_sym_link(modified_recipe) + + +def register_ide_plugin(ide_plugins): + ide_plugins['none'] = IdeNone diff --git a/scripts/lib/devtool/ide_sdk.py b/scripts/lib/devtool/ide_sdk.py new file mode 100755 index 00000000000..8c9ed3a2217 --- /dev/null +++ b/scripts/lib/devtool/ide_sdk.py @@ -0,0 +1,1030 @@ +# Development tool - ide-sdk command plugin +# +# Copyright (C) 2023-2024 Siemens AG +# +# SPDX-License-Identifier: GPL-2.0-only +# +"""Devtool ide-sdk plugin""" + +import json +import logging +import os +import re +import shutil +import stat +import subprocess +from argparse import RawTextHelpFormatter +from enum import Enum + +import scriptutils +import bb +from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError, parse_recipe +from devtool.standard import get_real_srctree +from devtool.ide_plugins import BuildTool, get_devtool_deploy_opts + + +logger = logging.getLogger('devtool') + +# dict of classes derived from IdeBase +ide_plugins = {} + + +class DevtoolIdeMode(Enum): + """Different modes are supported by the ide-sdk plugin. + + The enum might be extended by more advanced modes in the future. Some ideas: + - auto: modified if all recipes are modified, shared if none of the recipes is modified. + - mixed: modified mode for modified recipes, shared mode for all other recipes. + """ + + modified = 'modified' + shared = 'shared' + + +class TargetDevice: + """SSH remote login parameters""" + + def __init__(self, args): + self.extraoptions = '' + if args.no_host_check: + self.extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' + self.ssh_sshexec = 'ssh' + if args.ssh_exec: + self.ssh_sshexec = args.ssh_exec + self.ssh_port = '' + if args.port: + self.ssh_port = "-p %s" % args.port + if args.key: + self.extraoptions += ' -i %s' % args.key + + self.target = args.target + target_sp = args.target.split('@') + if len(target_sp) == 1: + self.login = "" + self.host = target_sp[0] + elif len(target_sp) == 2: + self.login = target_sp[0] + self.host = target_sp[1] + else: + logger.error("Invalid target argument: %s" % args.target) + + +class RecipeNative: + """Base class for calling bitbake to provide a -native recipe""" + + def __init__(self, name, target_arch=None): + self.name = name + self.target_arch = target_arch + self.bootstrap_tasks = [self.name + ':do_addto_recipe_sysroot'] + self.staging_bindir_native = None + self.target_sys = None + self.__native_bin = None + + def _initialize(self, config, workspace, tinfoil): + """Get the parsed recipe""" + recipe_d = parse_recipe( + config, tinfoil, self.name, appends=True, filter_workspace=False) + if not recipe_d: + raise DevtoolError("Parsing %s recipe failed" % self.name) + self.staging_bindir_native = os.path.realpath( + recipe_d.getVar('STAGING_BINDIR_NATIVE')) + self.target_sys = recipe_d.getVar('TARGET_SYS') + return recipe_d + + def initialize(self, config, workspace, tinfoil): + """Basic initialization that can be overridden by a derived class""" + self._initialize(config, workspace, tinfoil) + + @property + def native_bin(self): + if not self.__native_bin: + raise DevtoolError("native binary name is not defined.") + return self.__native_bin + + +class RecipeGdbCross(RecipeNative): + """Handle handle gdb-cross on the host and the gdbserver on the target device""" + + def __init__(self, args, target_arch, target_device): + super().__init__('gdb-cross-' + target_arch, target_arch) + self.target_device = target_device + self.gdb = None + self.gdbserver_port_next = int(args.gdbserver_port_start) + self.config_db = {} + + def __find_gdbserver(self, config, tinfoil): + """Absolute path of the gdbserver""" + recipe_d_gdb = parse_recipe( + config, tinfoil, 'gdb', appends=True, filter_workspace=False) + if not recipe_d_gdb: + raise DevtoolError("Parsing gdb recipe failed") + return os.path.join(recipe_d_gdb.getVar('bindir'), 'gdbserver') + + def initialize(self, config, workspace, tinfoil): + super()._initialize(config, workspace, tinfoil) + gdb_bin = self.target_sys + '-gdb' + gdb_path = os.path.join( + self.staging_bindir_native, self.target_sys, gdb_bin) + self.gdb = gdb_path + self.gdbserver_path = self.__find_gdbserver(config, tinfoil) + + @property + def host(self): + return self.target_device.host + + +class RecipeImage: + """Handle some image recipe related properties + + Most workflows require firmware that runs on the target device. + This firmware must be consistent with the setup of the host system. + In particular, the debug symbols must be compatible. For this, the + rootfs must be created as part of the SDK. + """ + + def __init__(self, name): + self.combine_dbg_image = False + self.gdbserver_missing = False + self.name = name + self.rootfs = None + self.__rootfs_dbg = None + self.bootstrap_tasks = [self.name + ':do_build'] + + def initialize(self, config, tinfoil): + image_d = parse_recipe( + config, tinfoil, self.name, appends=True, filter_workspace=False) + if not image_d: + raise DevtoolError( + "Parsing image recipe %s failed" % self.name) + + self.combine_dbg_image = bb.data.inherits_class( + 'image-combined-dbg', image_d) + + workdir = image_d.getVar('WORKDIR') + self.rootfs = os.path.join(workdir, 'rootfs') + if image_d.getVar('IMAGE_GEN_DEBUGFS') == "1": + self.__rootfs_dbg = os.path.join(workdir, 'rootfs-dbg') + + self.gdbserver_missing = 'gdbserver' not in image_d.getVar( + 'IMAGE_INSTALL') + + @property + def debug_support(self): + return bool(self.rootfs_dbg) + + @property + def rootfs_dbg(self): + if self.__rootfs_dbg and os.path.isdir(self.__rootfs_dbg): + return self.__rootfs_dbg + return None + + +class RecipeMetaIdeSupport: + """For the shared sysroots mode meta-ide-support is needed + + For use cases where just a cross tool-chain is required but + no recipe is used, devtool ide-sdk abstracts calling bitbake meta-ide-support + and bitbake build-sysroots. This also allows to expose the cross-toolchains + to IDEs. For example VSCode support different tool-chains with e.g. cmake-kits. + """ + + def __init__(self): + self.bootstrap_tasks = ['meta-ide-support:do_build'] + self.topdir = None + self.datadir = None + self.deploy_dir_image = None + self.build_sys = None + # From toolchain-scripts + self.real_multimach_target_sys = None + + def initialize(self, config, tinfoil): + meta_ide_support_d = parse_recipe( + config, tinfoil, 'meta-ide-support', appends=True, filter_workspace=False) + if not meta_ide_support_d: + raise DevtoolError("Parsing meta-ide-support recipe failed") + + self.topdir = meta_ide_support_d.getVar('TOPDIR') + self.datadir = meta_ide_support_d.getVar('datadir') + self.deploy_dir_image = meta_ide_support_d.getVar( + 'DEPLOY_DIR_IMAGE') + self.build_sys = meta_ide_support_d.getVar('BUILD_SYS') + self.real_multimach_target_sys = meta_ide_support_d.getVar( + 'REAL_MULTIMACH_TARGET_SYS') + + +class RecipeBuildSysroots: + """For the shared sysroots mode build-sysroots is needed""" + + def __init__(self): + self.standalone_sysroot = None + self.standalone_sysroot_native = None + self.bootstrap_tasks = [ + 'build-sysroots:do_build_target_sysroot', + 'build-sysroots:do_build_native_sysroot' + ] + + def initialize(self, config, tinfoil): + build_sysroots_d = parse_recipe( + config, tinfoil, 'build-sysroots', appends=True, filter_workspace=False) + if not build_sysroots_d: + raise DevtoolError("Parsing build-sysroots recipe failed") + self.standalone_sysroot = build_sysroots_d.getVar( + 'STANDALONE_SYSROOT') + self.standalone_sysroot_native = build_sysroots_d.getVar( + 'STANDALONE_SYSROOT_NATIVE') + + +class SharedSysrootsEnv: + """Handle the shared sysroots based workflow + + Support the workflow with just a tool-chain without a recipe. + It's basically like: + bitbake some-dependencies + bitbake meta-ide-support + bitbake build-sysroots + Use the environment-* file found in the deploy folder + """ + + def __init__(self): + self.ide_support = None + self.build_sysroots = None + + def initialize(self, ide_support, build_sysroots): + self.ide_support = ide_support + self.build_sysroots = build_sysroots + + def setup_ide(self, ide): + ide.setup(self) + + +class RecipeNotModified: + """Handling of recipes added to the Direct DSK shared sysroots.""" + + def __init__(self, name): + self.name = name + self.bootstrap_tasks = [name + ':do_populate_sysroot'] + + +class RecipeModified: + """Handling of recipes in the workspace created by devtool modify""" + OE_INIT_BUILD_ENV = 'oe-init-build-env' + + VALID_BASH_ENV_NAME_CHARS = re.compile(r"^[a-zA-Z0-9_]*$") + + def __init__(self, name): + self.name = name + self.bootstrap_tasks = [name + ':do_install'] + self.gdb_cross = None + # workspace + self.real_srctree = None + self.srctree = None + self.ide_sdk_dir = None + self.ide_sdk_scripts_dir = None + self.bbappend = None + # recipe variables from d.getVar + self.b = None + self.base_libdir = None + self.bblayers = None + self.bpn = None + self.d = None + self.fakerootcmd = None + self.fakerootenv = None + self.libdir = None + self.max_process = None + self.package_arch = None + self.package_debug_split_style = None + self.path = None + self.pn = None + self.recipe_sysroot = None + self.recipe_sysroot_native = None + self.staging_incdir = None + self.strip_cmd = None + self.target_arch = None + self.topdir = None + self.workdir = None + self.recipe_id = None + # replicate bitbake build environment + self.exported_vars = None + self.cmd_compile = None + self.__oe_init_dir = None + # main build tool used by this recipe + self.build_tool = BuildTool.UNDEFINED + # build_tool = cmake + self.oecmake_generator = None + self.cmake_cache_vars = None + # build_tool = meson + self.meson_buildtype = None + self.meson_wrapper = None + self.mesonopts = None + self.extra_oemeson = None + self.meson_cross_file = None + + def initialize(self, config, workspace, tinfoil): + recipe_d = parse_recipe( + config, tinfoil, self.name, appends=True, filter_workspace=False) + if not recipe_d: + raise DevtoolError("Parsing %s recipe failed" % self.name) + + # Verify this recipe is built as externalsrc setup by devtool modify + workspacepn = check_workspace_recipe( + workspace, self.name, bbclassextend=True) + self.srctree = workspace[workspacepn]['srctree'] + # Need to grab this here in case the source is within a subdirectory + self.real_srctree = get_real_srctree( + self.srctree, recipe_d.getVar('S'), recipe_d.getVar('WORKDIR')) + self.bbappend = workspace[workspacepn]['bbappend'] + + self.ide_sdk_dir = os.path.join( + config.workspace_path, 'ide-sdk', self.name) + if os.path.exists(self.ide_sdk_dir): + shutil.rmtree(self.ide_sdk_dir) + self.ide_sdk_scripts_dir = os.path.join(self.ide_sdk_dir, 'scripts') + + self.b = recipe_d.getVar('B') + self.base_libdir = recipe_d.getVar('base_libdir') + self.bblayers = recipe_d.getVar('BBLAYERS').split() + self.bpn = recipe_d.getVar('BPN') + self.d = recipe_d.getVar('D') + self.fakerootcmd = recipe_d.getVar('FAKEROOTCMD') + self.fakerootenv = recipe_d.getVar('FAKEROOTENV') + self.libdir = recipe_d.getVar('libdir') + self.max_process = int(recipe_d.getVar( + "BB_NUMBER_THREADS") or os.cpu_count() or 1) + self.package_arch = recipe_d.getVar('PACKAGE_ARCH') + self.package_debug_split_style = recipe_d.getVar( + 'PACKAGE_DEBUG_SPLIT_STYLE') + self.path = recipe_d.getVar('PATH') + self.pn = recipe_d.getVar('PN') + self.recipe_sysroot = os.path.realpath( + recipe_d.getVar('RECIPE_SYSROOT')) + self.recipe_sysroot_native = os.path.realpath( + recipe_d.getVar('RECIPE_SYSROOT_NATIVE')) + self.staging_incdir = os.path.realpath( + recipe_d.getVar('STAGING_INCDIR')) + self.strip_cmd = recipe_d.getVar('STRIP') + self.target_arch = recipe_d.getVar('TARGET_ARCH') + self.topdir = recipe_d.getVar('TOPDIR') + self.workdir = os.path.realpath(recipe_d.getVar('WORKDIR')) + + self.__init_exported_variables(recipe_d) + + if bb.data.inherits_class('cmake', recipe_d): + self.oecmake_generator = recipe_d.getVar('OECMAKE_GENERATOR') + self.__init_cmake_preset_cache(recipe_d) + self.build_tool = BuildTool.CMAKE + elif bb.data.inherits_class('meson', recipe_d): + self.meson_buildtype = recipe_d.getVar('MESON_BUILDTYPE') + self.mesonopts = recipe_d.getVar('MESONOPTS') + self.extra_oemeson = recipe_d.getVar('EXTRA_OEMESON') + self.meson_cross_file = recipe_d.getVar('MESON_CROSS_FILE') + self.build_tool = BuildTool.MESON + + # Recipe ID is the identifier for IDE config sections + self.recipe_id = self.bpn + "-" + self.package_arch + self.recipe_id_pretty = self.bpn + ": " + self.package_arch + + def append_to_bbappend(self, append_text): + with open(self.bbappend, 'a') as bbap: + bbap.write(append_text) + + def remove_from_bbappend(self, append_text): + with open(self.bbappend, 'r') as bbap: + text = bbap.read() + new_text = text.replace(append_text, '') + with open(self.bbappend, 'w') as bbap: + bbap.write(new_text) + + @staticmethod + def is_valid_shell_variable(var): + """Skip strange shell variables like systemd + + prevent from strange bugs because of strange variables which + are not used in this context but break various tools. + """ + if RecipeModified.VALID_BASH_ENV_NAME_CHARS.match(var): + bb.debug(1, "ignoring variable: %s" % var) + return True + return False + + def debug_build_config(self, args): + """Explicitely set for example CMAKE_BUILD_TYPE to Debug if not defined otherwise""" + if self.build_tool is BuildTool.CMAKE: + append_text = os.linesep + \ + 'OECMAKE_ARGS:append = " -DCMAKE_BUILD_TYPE:STRING=Debug"' + os.linesep + if args.debug_build_config and not 'CMAKE_BUILD_TYPE' in self.cmake_cache_vars: + self.cmake_cache_vars['CMAKE_BUILD_TYPE'] = { + "type": "STRING", + "value": "Debug", + } + self.append_to_bbappend(append_text) + elif 'CMAKE_BUILD_TYPE' in self.cmake_cache_vars: + del self.cmake_cache_vars['CMAKE_BUILD_TYPE'] + self.remove_from_bbappend(append_text) + elif self.build_tool is BuildTool.MESON: + append_text = os.linesep + 'MESON_BUILDTYPE = "debug"' + os.linesep + if args.debug_build_config and self.meson_buildtype != "debug": + self.mesonopts.replace( + '--buildtype ' + self.meson_buildtype, '--buildtype debug') + self.append_to_bbappend(append_text) + elif self.meson_buildtype == "debug": + self.mesonopts.replace( + '--buildtype debug', '--buildtype plain') + self.remove_from_bbappend(append_text) + elif args.debug_build_config: + logger.warn( + "--debug-build-config is not implemented for this build tool yet.") + + def solib_search_path(self, image): + """Search for debug symbols in the rootfs and rootfs-dbg + + The debug symbols of shared libraries which are provided by other packages + are grabbed from the -dbg packages in the rootfs-dbg. + + But most cross debugging tools like gdb, perf, and systemtap need to find + executable/library first and through it debuglink note find corresponding + symbols file. Therefore the library paths from the rootfs are added as well. + + Note: For the devtool modified recipe compiled from the IDE, the debug + symbols are taken from the unstripped binaries in the image folder. + Also, devtool deploy-target takes the files from the image folder. + debug symbols in the image folder refer to the corresponding source files + with absolute paths of the build machine. Debug symbols found in the + rootfs-dbg are relocated and contain paths which refer to the source files + installed on the target device e.g. /usr/src/... + """ + base_libdir = self.base_libdir.lstrip('/') + libdir = self.libdir.lstrip('/') + so_paths = [ + # debug symbols for package_debug_split_style: debug-with-srcpkg or .debug + os.path.join(image.rootfs_dbg, base_libdir, ".debug"), + os.path.join(image.rootfs_dbg, libdir, ".debug"), + # debug symbols for package_debug_split_style: debug-file-directory + os.path.join(image.rootfs_dbg, "usr", "lib", "debug"), + + # The binaries are required as well, the debug packages are not enough + # With image-combined-dbg.bbclass the binaries are copied into rootfs-dbg + os.path.join(image.rootfs_dbg, base_libdir), + os.path.join(image.rootfs_dbg, libdir), + # Without image-combined-dbg.bbclass the binaries are only in rootfs. + # Note: Stepping into source files located in rootfs-dbg does not + # work without image-combined-dbg.bbclass yet. + os.path.join(image.rootfs, base_libdir), + os.path.join(image.rootfs, libdir) + ] + return so_paths + + def solib_search_path_str(self, image): + """Return a : separated list of paths usable by GDB's set solib-search-path""" + return ':'.join(self.solib_search_path(image)) + + def __init_exported_variables(self, d): + """Find all variables with export flag set. + + This allows to generate IDE configurations which compile with the same + environment as bitbake does. That's at least a reasonable default behavior. + """ + exported_vars = {} + + vars = (key for key in d.keys() if not key.startswith( + "__") and not d.getVarFlag(key, "func", False)) + for var in vars: + func = d.getVarFlag(var, "func", False) + if d.getVarFlag(var, 'python', False) and func: + continue + export = d.getVarFlag(var, "export", False) + unexport = d.getVarFlag(var, "unexport", False) + if not export and not unexport and not func: + continue + if unexport: + continue + + val = d.getVar(var) + if val is None: + continue + if set(var) & set("-.{}+"): + logger.warn( + "Warning: Found invalid character in variable name %s", str(var)) + continue + varExpanded = d.expand(var) + val = str(val) + + if not RecipeModified.is_valid_shell_variable(varExpanded): + continue + + if func: + code_line = "line: {0}, file: {1}\n".format( + d.getVarFlag(var, "lineno", False), + d.getVarFlag(var, "filename", False)) + val = val.rstrip('\n') + logger.warn("Warning: exported shell function %s() is not exported (%s)" % + (varExpanded, code_line)) + continue + + if export: + exported_vars[varExpanded] = val.strip() + continue + + self.exported_vars = exported_vars + + def __init_cmake_preset_cache(self, d): + """Get the arguments passed to cmake + + Replicate the cmake configure arguments with all details to + share on build folder between bitbake and SDK. + """ + site_file = os.path.join(self.workdir, 'site-file.cmake') + if os.path.exists(site_file): + print("Warning: site-file.cmake is not supported") + + cache_vars = {} + oecmake_args = d.getVar('OECMAKE_ARGS').split() + extra_oecmake = d.getVar('EXTRA_OECMAKE').split() + for param in oecmake_args + extra_oecmake: + d_pref = "-D" + if param.startswith(d_pref): + param = param[len(d_pref):] + else: + print("Error: expected a -D") + param_s = param.split('=', 1) + param_nt = param_s[0].split(':', 1) + + def handle_undefined_variable(var): + if var.startswith('${') and var.endswith('}'): + return '' + else: + return var + # Example: FOO=ON + if len(param_nt) == 1: + cache_vars[param_s[0]] = handle_undefined_variable(param_s[1]) + # Example: FOO:PATH=/tmp + elif len(param_nt) == 2: + cache_vars[param_nt[0]] = { + "type": param_nt[1], + "value": handle_undefined_variable(param_s[1]), + } + else: + print("Error: cannot parse %s" % param) + self.cmake_cache_vars = cache_vars + + def cmake_preset(self): + """Create a preset for cmake that mimics how bitbake calls cmake""" + toolchain_file = os.path.join(self.workdir, 'toolchain.cmake') + cmake_executable = os.path.join( + self.recipe_sysroot_native, 'usr', 'bin', 'cmake') + self.cmd_compile = cmake_executable + " --build --preset " + self.recipe_id + + preset_dict_configure = { + "name": self.recipe_id, + "displayName": self.recipe_id_pretty, + "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch), + "binaryDir": self.b, + "generator": self.oecmake_generator, + "toolchainFile": toolchain_file, + "cacheVariables": self.cmake_cache_vars, + "environment": self.exported_vars, + "cmakeExecutable": cmake_executable + } + + preset_dict_build = { + "name": self.recipe_id, + "displayName": self.recipe_id_pretty, + "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch), + "configurePreset": self.recipe_id, + "inheritConfigureEnvironment": True + } + + preset_dict_test = { + "name": self.recipe_id, + "displayName": self.recipe_id_pretty, + "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch), + "configurePreset": self.recipe_id, + "inheritConfigureEnvironment": True + } + + preset_dict = { + "version": 3, # cmake 3.21, backward compatible with kirkstone + "configurePresets": [preset_dict_configure], + "buildPresets": [preset_dict_build], + "testPresets": [preset_dict_test] + } + + # Finally write the json file + json_file = 'CMakeUserPresets.json' + json_path = os.path.join(self.real_srctree, json_file) + logger.info("Updating CMake preset: %s (%s)" % (json_file, json_path)) + if not os.path.exists(self.real_srctree): + os.makedirs(self.real_srctree) + try: + with open(json_path) as f: + orig_dict = json.load(f) + except json.decoder.JSONDecodeError: + logger.info( + "Decoding %s failed. Probably because of comments in the json file" % json_path) + orig_dict = {} + except FileNotFoundError: + orig_dict = {} + + # Add or update the presets for the recipe and keep other presets + for k, v in preset_dict.items(): + if isinstance(v, list): + update_preset = v[0] + preset_added = False + if k in orig_dict: + for index, orig_preset in enumerate(orig_dict[k]): + if 'name' in orig_preset: + if orig_preset['name'] == update_preset['name']: + logger.debug("Updating preset: %s" % + orig_preset['name']) + orig_dict[k][index] = update_preset + preset_added = True + break + else: + logger.debug("keeping preset: %s" % + orig_preset['name']) + else: + logger.warn("preset without a name found") + if not preset_added: + if not k in orig_dict: + orig_dict[k] = [] + orig_dict[k].append(update_preset) + logger.debug("Added preset: %s" % + update_preset['name']) + else: + orig_dict[k] = v + + with open(json_path, 'w') as f: + json.dump(orig_dict, f, indent=4) + + def gen_meson_wrapper(self): + """Generate a wrapper script to call meson with the cross environment""" + bb.utils.mkdirhier(self.ide_sdk_scripts_dir) + meson_wrapper = os.path.join(self.ide_sdk_scripts_dir, 'meson') + meson_real = os.path.join( + self.recipe_sysroot_native, 'usr', 'bin', 'meson.real') + with open(meson_wrapper, 'w') as mwrap: + mwrap.write("#!/bin/sh" + os.linesep) + for var, val in self.exported_vars.items(): + mwrap.write('export %s="%s"' % (var, val) + os.linesep) + mwrap.write("unset CC CXX CPP LD AR NM STRIP" + os.linesep) + private_temp = os.path.join(self.b, "meson-private", "tmp") + mwrap.write('mkdir -p "%s"' % private_temp + os.linesep) + mwrap.write('export TMPDIR="%s"' % private_temp + os.linesep) + mwrap.write('exec "%s" "$@"' % meson_real + os.linesep) + st = os.stat(meson_wrapper) + os.chmod(meson_wrapper, st.st_mode | stat.S_IEXEC) + self.meson_wrapper = meson_wrapper + self.cmd_compile = meson_wrapper + " compile -C " + self.b + + def which(self, executable): + bin_path = shutil.which(executable, path=self.path) + if not bin_path: + raise DevtoolError( + 'Cannot find %s. Probably the recipe %s is not built yet.' % (executable, self.bpn)) + return bin_path + + @staticmethod + def is_elf_file(file_path): + with open(file_path, "rb") as f: + data = f.read(4) + if data == b'\x7fELF': + return True + return False + + def find_installed_binaries(self): + """find all executable elf files in the image directory""" + binaries = [] + d_len = len(self.d) + re_so = re.compile('.*\.so[.0-9]*$') + for root, _, files in os.walk(self.d, followlinks=False): + for file in files: + if os.path.islink(file): + continue + if re_so.match(file): + continue + abs_name = os.path.join(root, file) + if os.access(abs_name, os.X_OK) and RecipeModified.is_elf_file(abs_name): + binaries.append(abs_name[d_len:]) + return binaries + + def gen_delete_package_dirs(self): + """delete folders of package tasks + + This is a workaround for and issue with recipes having their sources + downloaded as file:// + This likely breaks pseudo like: + path mismatch [3 links]: ino 79147802 db + .../build/tmp/.../cmake-example/1.0/package/usr/src/debug/ + cmake-example/1.0-r0/oe-local-files/cpp-example-lib.cpp + .../build/workspace/sources/cmake-example/oe-local-files/cpp-example-lib.cpp + Since the files are anyway outdated lets deleted them (also from pseudo's db) to workaround this issue. + """ + cmd_lines = ['#!/bin/sh'] + + # Set up the appropriate environment + newenv = dict(os.environ) + for varvalue in self.fakerootenv.split(): + if '=' in varvalue: + splitval = varvalue.split('=', 1) + newenv[splitval[0]] = splitval[1] + + # Replicate the environment variables from bitbake + for var, val in newenv.items(): + if not RecipeModified.is_valid_shell_variable(var): + continue + cmd_lines.append('%s="%s"' % (var, val)) + cmd_lines.append('export %s' % var) + + # Delete the folders + pkg_dirs = ' '.join([os.path.join(self.workdir, d) for d in [ + "package", "packages-split", "pkgdata", "sstate-install-package", "debugsources.list", "*.spec"]]) + cmd = "%s rm -rf %s" % (self.fakerootcmd, pkg_dirs) + cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd)) + + return self.write_script(cmd_lines, 'delete_package_dirs') + + def gen_install_deploy_script(self, args): + """Generate a script which does install and deploy""" + cmd_lines = ['#!/bin/sh -e'] + + cmd_lines.append(self.gen_delete_package_dirs()) + + # . oe-init-build-env + cmd_lines.append('cd "%s" || { echo "cd %s failed"; exit 1; }' % ( + self.oe_init_dir, self.oe_init_dir)) + cmd_lines.append('. "%s" "%s" || { echo ". %s %s failed"; exit 1; }' % ( + self.oe_init_build_env, self.topdir, self.oe_init_build_env, self.topdir)) + + # bitbake -c install + cmd_lines.append("bitbake %s -c install" % self.bpn) + + # devtool deploy-target + deploy_opts = ' '.join(get_devtool_deploy_opts(args)) + cmd_lines.append("devtool deploy-target %s %s" % + (self.bpn, deploy_opts)) + return self.write_script(cmd_lines, 'install_and_deploy') + + def write_script(self, cmd_lines, script_name): + bb.utils.mkdirhier(self.ide_sdk_scripts_dir) + script_name_arch = script_name + '_' + self.recipe_id + script_file = os.path.join(self.ide_sdk_scripts_dir, script_name_arch) + with open(script_file, 'w') as script_f: + script_f.write(os.linesep.join(cmd_lines)) + st = os.stat(script_file) + os.chmod(script_file, st.st_mode | stat.S_IEXEC) + return script_file + + @property + def oe_init_build_env(self): + """Find the oe-init-build-env used for this setup""" + oe_init_dir = self.oe_init_dir + if oe_init_dir: + return os.path.join(oe_init_dir, RecipeModified.OE_INIT_BUILD_ENV) + return None + + @property + def oe_init_dir(self): + """Find the directory where the oe-init-build-env is located + + Assumption: There might be a layer with higher priority than poky + which provides to oe-init-build-env in the layer's toplevel folder. + """ + if not self.__oe_init_dir: + for layer in reversed(self.bblayers): + result = subprocess.run( + ['git', 'rev-parse', '--show-toplevel'], cwd=layer, capture_output=True) + if result.returncode == 0: + oe_init_dir = result.stdout.decode('utf-8').strip() + oe_init_path = os.path.join( + oe_init_dir, RecipeModified.OE_INIT_BUILD_ENV) + if os.path.exists(oe_init_path): + logger.debug("Using %s from: %s" % ( + RecipeModified.OE_INIT_BUILD_ENV, oe_init_path)) + self.__oe_init_dir = oe_init_dir + break + if not self.__oe_init_dir: + logger.error("Cannot find the bitbake top level folder") + return self.__oe_init_dir + + +def ide_setup(args, config, basepath, workspace): + """Generate the IDE configuration for the workspace""" + + # Explicitely passing some special recipes does not make sense + for recipe in args.recipenames: + if recipe in ['meta-ide-support', 'build-sysroots']: + raise DevtoolError("Invalid recipe: %s." % recipe) + + # Collect information about tasks which need to be bitbaked + bootstrap_tasks = [] + bootstrap_tasks_late = [] + tinfoil = setup_tinfoil(config_only=False, basepath=basepath) + try: + # define mode depending on recipes which need to be processed + recipes_image_names = [] + recipes_modified_names = [] + recipes_other_names = [] + for recipe in args.recipenames: + try: + check_workspace_recipe( + workspace, recipe, bbclassextend=True) + recipes_modified_names.append(recipe) + except DevtoolError: + recipe_d = parse_recipe( + config, tinfoil, recipe, appends=True, filter_workspace=False) + if not recipe_d: + raise DevtoolError("Parsing recipe %s failed" % recipe) + if bb.data.inherits_class('image', recipe_d): + recipes_image_names.append(recipe) + else: + recipes_other_names.append(recipe) + + invalid_params = False + if args.mode == DevtoolIdeMode.shared: + if len(recipes_modified_names): + logger.error("In shared sysroots mode modified recipes %s cannot be handled." % str( + recipes_modified_names)) + invalid_params = True + if args.mode == DevtoolIdeMode.modified: + if len(recipes_other_names): + logger.error("Only in shared sysroots mode not modified recipes %s can be handled." % str( + recipes_other_names)) + invalid_params = True + if len(recipes_image_names) != 1: + logger.error( + "One image recipe is required as the rootfs for the remote development.") + invalid_params = True + for modified_recipe_name in recipes_modified_names: + if modified_recipe_name.startswith('nativesdk-') or modified_recipe_name.endswith('-native'): + logger.error( + "Only cross compiled recipes are support. %s is not cross." % modified_recipe_name) + invalid_params = True + + if invalid_params: + raise DevtoolError("Invalid parameters are passed.") + + # For the shared sysroots mode, add all dependencies of all the images to the sysroots + # For the modified mode provide one rootfs and the corresponding debug symbols via rootfs-dbg + recipes_images = [] + for recipes_image_name in recipes_image_names: + logger.info("Using image: %s" % recipes_image_name) + recipe_image = RecipeImage(recipes_image_name) + recipe_image.initialize(config, tinfoil) + bootstrap_tasks += recipe_image.bootstrap_tasks + recipes_images.append(recipe_image) + + # Provide a Direct SDK with shared sysroots + recipes_not_modified = [] + if args.mode == DevtoolIdeMode.shared: + ide_support = RecipeMetaIdeSupport() + ide_support.initialize(config, tinfoil) + bootstrap_tasks += ide_support.bootstrap_tasks + + logger.info("Adding %s to the Direct SDK sysroots." % + str(recipes_other_names)) + for recipe_name in recipes_other_names: + recipe_not_modified = RecipeNotModified(recipe_name) + bootstrap_tasks += recipe_not_modified.bootstrap_tasks + recipes_not_modified.append(recipe_not_modified) + + build_sysroots = RecipeBuildSysroots() + build_sysroots.initialize(config, tinfoil) + bootstrap_tasks_late += build_sysroots.bootstrap_tasks + shared_env = SharedSysrootsEnv() + shared_env.initialize(ide_support, build_sysroots) + + recipes_modified = [] + if args.mode == DevtoolIdeMode.modified: + logger.info("Setting up workspaces for modified recipe: %s" % + str(recipes_modified_names)) + gdbs_cross = {} + for recipe_name in recipes_modified_names: + recipe_modified = RecipeModified(recipe_name) + recipe_modified.initialize(config, workspace, tinfoil) + bootstrap_tasks += recipe_modified.bootstrap_tasks + recipes_modified.append(recipe_modified) + + if recipe_modified.target_arch not in gdbs_cross: + target_device = TargetDevice(args) + gdb_cross = RecipeGdbCross( + args, recipe_modified.target_arch, target_device) + gdb_cross.initialize(config, workspace, tinfoil) + bootstrap_tasks += gdb_cross.bootstrap_tasks + gdbs_cross[recipe_modified.target_arch] = gdb_cross + recipe_modified.gdb_cross = gdbs_cross[recipe_modified.target_arch] + + finally: + tinfoil.shutdown() + + if not args.skip_bitbake: + bb_cmd = 'bitbake ' + if args.bitbake_k: + bb_cmd += "-k " + bb_cmd_early = bb_cmd + ' '.join(bootstrap_tasks) + exec_build_env_command( + config.init_path, basepath, bb_cmd_early, watch=True) + if bootstrap_tasks_late: + bb_cmd_late = bb_cmd + ' '.join(bootstrap_tasks_late) + exec_build_env_command( + config.init_path, basepath, bb_cmd_late, watch=True) + + for recipe_image in recipes_images: + if (recipe_image.gdbserver_missing): + logger.warning( + "gdbserver not installed in image %s. Remote debugging will not be available" % recipe_image) + + if recipe_image.combine_dbg_image is False: + logger.warning( + 'IMAGE_CLASSES += "image-combined-dbg" is missing for image %s. Remote debugging will not find debug symbols from rootfs-dbg.' % recipe_image) + + # Instantiate the active IDE plugin + ide = ide_plugins[args.ide]() + if args.mode == DevtoolIdeMode.shared: + ide.setup_shared_sysroots(shared_env) + elif args.mode == DevtoolIdeMode.modified: + for recipe_modified in recipes_modified: + if recipe_modified.build_tool is BuildTool.CMAKE: + recipe_modified.cmake_preset() + if recipe_modified.build_tool is BuildTool.MESON: + recipe_modified.gen_meson_wrapper() + ide.setup_modified_recipe( + args, recipe_image, recipe_modified) + else: + raise DevtoolError("Must not end up here.") + + +def register_commands(subparsers, context): + """Register devtool subcommands from this plugin""" + + global ide_plugins + + # Search for IDE plugins in all sub-folders named ide_plugins where devtool seraches for plugins. + pluginpaths = [os.path.join(path, 'ide_plugins') + for path in context.pluginpaths] + ide_plugin_modules = [] + for pluginpath in pluginpaths: + scriptutils.load_plugins(logger, ide_plugin_modules, pluginpath) + + for ide_plugin_module in ide_plugin_modules: + if hasattr(ide_plugin_module, 'register_ide_plugin'): + ide_plugin_module.register_ide_plugin(ide_plugins) + # Sort plugins according to their priority. The first entry is the default IDE plugin. + ide_plugins = dict(sorted(ide_plugins.items(), + key=lambda p: p[1].ide_plugin_priority(), reverse=True)) + + parser_ide_sdk = subparsers.add_parser('ide-sdk', group='working', order=50, formatter_class=RawTextHelpFormatter, + help='Setup the SDK and configure the IDE') + parser_ide_sdk.add_argument( + 'recipenames', nargs='+', help='Generate an IDE configuration suitable to work on the given recipes.\n' + 'Depending on the --mode paramter different types of SDKs and IDE configurations are generated.') + parser_ide_sdk.add_argument( + '-m', '--mode', type=DevtoolIdeMode, default=DevtoolIdeMode.modified, + help='Different SDK types are supported:\n' + '- "' + DevtoolIdeMode.modified.name + '" (default):\n' + ' devtool modify creates a workspace to work on the source code of a recipe.\n' + ' devtool ide-sdk builds the SDK and generates the IDE configuration(s) in the workspace directorie(s)\n' + ' Usage example:\n' + ' devtool modify cmake-example\n' + ' devtool ide-sdk cmake-example core-image-minimal\n' + ' Start the IDE in the workspace folder\n' + ' At least one devtool modified recipe plus one image recipe are required:\n' + ' The image recipe is used to generate the target image and the remote debug configuration.\n' + '- "' + DevtoolIdeMode.shared.name + '":\n' + ' Usage example:\n' + ' devtool ide-sdk -m ' + DevtoolIdeMode.shared.name + ' recipe(s)\n' + ' This command generates a cross-toolchain as well as the corresponding shared sysroot directories.\n' + ' To use this tool-chain the environment-* file found in the deploy..image folder needs to be sourced into a shell.\n' + ' In case of VSCode and cmake the tool-chain is also exposed as a cmake-kit') + default_ide = list(ide_plugins.keys())[0] + parser_ide_sdk.add_argument( + '-i', '--ide', choices=ide_plugins.keys(), default=default_ide, + help='Setup the configuration for this IDE (default: %s)' % default_ide) + parser_ide_sdk.add_argument( + '-t', '--target', default='root@192.168.7.2', + help='Live target machine running an ssh server: user@hostname.') + parser_ide_sdk.add_argument( + '-G', '--gdbserver-port-start', default="1234", help='port where gdbserver is listening.') + parser_ide_sdk.add_argument( + '-c', '--no-host-check', help='Disable ssh host key checking', action='store_true') + parser_ide_sdk.add_argument( + '-e', '--ssh-exec', help='Executable to use in place of ssh') + parser_ide_sdk.add_argument( + '-P', '--port', help='Specify ssh port to use for connection to the target') + parser_ide_sdk.add_argument( + '-I', '--key', help='Specify ssh private key for connection to the target') + parser_ide_sdk.add_argument( + '--skip-bitbake', help='Generate IDE configuration but skip calling bibtake to update the SDK.', action='store_true') + parser_ide_sdk.add_argument( + '-k', '--bitbake-k', help='Pass -k parameter to bitbake', action='store_true') + parser_ide_sdk.add_argument( + '--no-strip', help='Do not strip executables prior to deploy', dest='strip', action='store_false') + parser_ide_sdk.add_argument( + '-n', '--dry-run', help='List files to be undeployed only', action='store_true') + parser_ide_sdk.add_argument( + '-s', '--show-status', help='Show progress/status output', action='store_true') + parser_ide_sdk.add_argument( + '-p', '--no-preserve', help='Do not preserve existing files', action='store_true') + parser_ide_sdk.add_argument( + '--no-check-space', help='Do not check for available space before deploying', action='store_true') + parser_ide_sdk.add_argument( + '--debug-build-config', help='Use debug build flags, for example set CMAKE_BUILD_TYPE=Debug', action='store_true') + parser_ide_sdk.set_defaults(func=ide_setup) From patchwork Sun Jan 14 22:14:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2174FC47DA2 for ; Sun, 14 Jan 2024 22:14:51 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx.groups.io with SMTP id smtpd.web10.52017.1705270486392715897 for ; Sun, 14 Jan 2024 14:14:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=R3JC/WcN; spf=pass (domain: gmail.com, ip: 209.85.128.45, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-40e636fd3d2so27408005e9.1 for ; Sun, 14 Jan 2024 14:14:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270484; x=1705875284; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EBNBj0nu8ui88glkGOzRhtBoQZsl8ckqJKa7AYI7LMQ=; b=R3JC/WcN+mECYFa3Tkg7ZdVsUFDaLKMe6a61SSOXZ1ZM1O6NOoeCuZCwaYCV04oBol WJfPwp24mZ2eeA2nSZxhB7JllI+w0xkweNwiz+yWd8vWFy4+gWAKlDO1mX0fns73RRWY zegNQMau66B2ne36/0PEF1Jb1GZuCkdCcsNdra5ABLNlDsnoqcCpqJarbMwrwJUsmV6S MAbniU8aknafV+NakK8Inx7hgjp46e4bWXfIlZY9llkS0fTj92Wbf3Bz6ShabItv3RUo 2ysNyvXaiVef2eIbN7Gk+NgLynvEnNFIPpcqFrJxO9EIM26D7HdsaqIcVcQHQdEQdy6n Uzvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270484; x=1705875284; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EBNBj0nu8ui88glkGOzRhtBoQZsl8ckqJKa7AYI7LMQ=; b=bO8ZQiaHYkS8/rtkmX0fodXVy1lzKvp4h4Yo0B/tWmMEmk9b9oVhv59dPrE9co7vxz ivtfyfcQY3QwgOmhQAh7/QU4Nd1vEGY3CqecxGXtH9qww0gy58O4ZoYlES4Lr+etOCFR ugB0T98Ax24ozcIRSK7a5/zaVLevISVXrj4+z61EWO45K4mJJ5o8S4Ug8j1+qWsMDXvt +O6X8PTnqroQ8chUkljt5IPAQHThpHLJDJ/shrHIvBU5NWcGPGjW3XikiNwZcdZiHmdZ w0HYKepOn993XE/LyGJqGWHtfiTMJjv3G9SAda94fIFiWTALRRDwv9ZhcTFWGPBob0ao xGGg== X-Gm-Message-State: AOJu0YxaXGwN4x4QRMoJpggAgyKZ8IVmK9Zukx3UdHEJ1DWUzQIOkaoz D9ovgKRX6Z16+G2IYPAfdzbndvWiS5o= X-Google-Smtp-Source: AGHT+IEbgh+hOcYcx3SIMtS3/S8+zGjnkpURD67OvVDFLoEKwVM/eOQIUJXODxkI0vB3FyZxK8ZPlw== X-Received: by 2002:a05:600c:3508:b0:40e:62ad:4627 with SMTP id h8-20020a05600c350800b0040e62ad4627mr2410608wmq.170.1705270484495; Sun, 14 Jan 2024 14:14:44 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:44 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 4/9] oe-selftest devtool: ide-sdk tests Date: Sun, 14 Jan 2024 23:14:16 +0100 Message-ID: <20240114221437.1255866-5-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193622 Add some oe-selftests for the new devtool ide-sdk plugin. Most of the workflows are covered. Many thanks to Enguerrand de Ribaucourt for testing and bug fixing. Signed-off-by: Adrian Freihofer --- meta/lib/oeqa/selftest/cases/devtool.py | 492 ++++++++++++++++++++++++ 1 file changed, 492 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index a8777207694..006c846438d 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -12,6 +12,7 @@ import tempfile import glob import fnmatch import unittest +import json from oeqa.selftest.case import OESelftestTestCase from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer @@ -2314,3 +2315,494 @@ class DevtoolUpgradeTests(DevtoolBase): #Step 4.5 runCmd("grep %s %s" % (modconfopt, codeconfigfile)) + + +class DevtoolIdeSdkTests(DevtoolBase): + def _write_bb_config(self, recipe_names): + """Helper to write the bitbake local.conf file""" + conf_lines = [ + 'IMAGE_CLASSES += "image-combined-dbg"', + 'IMAGE_GEN_DEBUGFS = "1"', + 'IMAGE_INSTALL:append = " gdbserver %s"' % ' '.join( + [r + '-ptest' for r in recipe_names]) + ] + self.write_config("\n".join(conf_lines)) + + def _check_workspace(self): + """Check if a workspace directory is available and setup the cleanup""" + self.assertTrue(not os.path.exists(self.workspacedir), + 'This test cannot be run with a workspace directory under the build directory') + self.track_for_cleanup(self.workspacedir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + + def _workspace_scripts_dir(self, recipe_name): + return os.path.realpath(os.path.join(self.builddir, 'workspace', 'ide-sdk', recipe_name, 'scripts')) + + def _sources_scripts_dir(self, src_dir): + return os.path.realpath(os.path.join(src_dir, 'oe-scripts')) + + def _workspace_gdbinit_dir(self, recipe_name): + return os.path.realpath(os.path.join(self.builddir, 'workspace', 'ide-sdk', recipe_name, 'scripts', 'gdbinit')) + + def _sources_gdbinit_dir(self, src_dir): + return os.path.realpath(os.path.join(src_dir, 'oe-gdbinit')) + + def _devtool_ide_sdk_recipe(self, recipe_name, build_file, testimage): + """Setup a recipe for working with devtool ide-sdk + + Basically devtool modify -x followed by some tests + """ + tempdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir) + self.add_command_to_tearDown('bitbake -c clean %s' % recipe_name) + + result = runCmd('devtool modify %s -x %s' % (recipe_name, tempdir)) + self.assertExists(os.path.join(tempdir, build_file), + 'Extracted source could not be found') + self.assertExists(os.path.join(self.workspacedir, 'conf', + 'layer.conf'), 'Workspace directory not created') + matches = glob.glob(os.path.join(self.workspacedir, + 'appends', recipe_name + '.bbappend')) + self.assertTrue(matches, 'bbappend not created %s' % result.output) + + # Test devtool status + result = runCmd('devtool status') + self.assertIn(recipe_name, result.output) + self.assertIn(tempdir, result.output) + self._check_src_repo(tempdir) + + # Usually devtool ide-sdk would initiate the build of the SDK. + # But there is a circular dependency with starting Qemu and passing the IP of runqemu to devtool ide-sdk. + if testimage: + bitbake("%s qemu-native qemu-helper-native" % testimage) + deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE') + self.add_command_to_tearDown('bitbake -c clean %s' % testimage) + self.add_command_to_tearDown( + 'rm -f %s/%s*' % (deploy_dir_image, testimage)) + + return tempdir + + def _get_recipe_ids(self, recipe_name): + """IDs needed to write recipe specific config entries into IDE config files""" + package_arch = get_bb_var('PACKAGE_ARCH', recipe_name) + recipe_id = recipe_name + "-" + package_arch + recipe_id_pretty = recipe_name + ": " + package_arch + return (recipe_id, recipe_id_pretty) + + def _verify_install_script_code(self, tempdir, recipe_name): + """Verify the scripts referred by the tasks.json file are fine. + + This function does not depend on Qemu. Therefore it verifies the scripts + exists and the delete step works as expected. But it does not try to + deploy to Qemu. + """ + recipe_id, recipe_id_pretty = self._get_recipe_ids(recipe_name) + with open(os.path.join(tempdir, '.vscode', 'tasks.json')) as tasks_j: + tasks_d = json.load(tasks_j) + tasks = tasks_d["tasks"] + task_install = next( + (task for task in tasks if task["label"] == "install && deploy-target %s" % recipe_id_pretty), None) + self.assertIsNot(task_install, None) + # execute only the bb_run_do_install script since the deploy would require e.g. Qemu running. + i_and_d_script = "install_and_deploy_" + recipe_id + i_and_d_script_path = os.path.join( + self._workspace_scripts_dir(recipe_name), i_and_d_script) + self.assertExists(i_and_d_script_path) + del_script = "delete_package_dirs_" + recipe_id + del_script_path = os.path.join( + self._workspace_scripts_dir(recipe_name), del_script) + self.assertExists(del_script_path) + runCmd(del_script_path, cwd=tempdir) + + def _devtool_ide_sdk_qemu(self, tempdir, qemu, recipe_name, example_exe): + """Verify deployment and execution in Qemu system work for one recipe. + + This function checks the entire SDK workflow: changing the code, recompiling + it and deploying it back to Qemu, and checking that the changes have been + incorporated into the provided binaries. It also runs the tests of the recipe. + """ + recipe_id, _ = self._get_recipe_ids(recipe_name) + i_and_d_script = "install_and_deploy_" + recipe_id + install_deploy_cmd = os.path.join( + self._workspace_scripts_dir(recipe_name), i_and_d_script) + self.assertExists(install_deploy_cmd, + '%s script not found' % install_deploy_cmd) + runCmd(install_deploy_cmd) + + MAGIC_STRING_ORIG = "Magic: 123456789" + MAGIC_STRING_NEW = "Magic: 987654321" + ptest_cmd = "ptest-runner " + recipe_name + + # validate that SSH is working + status, _ = qemu.run("uname") + self.assertEqual( + status, 0, msg="Failed to connect to the SSH server on Qemu") + + # Verify the unmodified example prints the magic string + status, output = qemu.run(example_exe) + self.assertEqual(status, 0, msg="%s failed: %s" % + (example_exe, output)) + self.assertIn(MAGIC_STRING_ORIG, output) + + # Verify the unmodified ptests work + status, output = qemu.run(ptest_cmd) + self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output)) + self.assertIn("PASS: cpp-example-lib", output) + + # Replace the Magic String in the code, compile and deploy to Qemu + cpp_example_lib_hpp = os.path.join(tempdir, 'cpp-example-lib.hpp') + with open(cpp_example_lib_hpp, 'r') as file: + cpp_code = file.read() + cpp_code = cpp_code.replace(MAGIC_STRING_ORIG, MAGIC_STRING_NEW) + with open(cpp_example_lib_hpp, 'w') as file: + file.write(cpp_code) + runCmd(install_deploy_cmd, cwd=tempdir) + + # Verify the modified example prints the modified magic string + status, output = qemu.run(example_exe) + self.assertEqual(status, 0, msg="%s failed: %s" % + (example_exe, output)) + self.assertNotIn(MAGIC_STRING_ORIG, output) + self.assertIn(MAGIC_STRING_NEW, output) + + # Verify the modified example ptests work + status, output = qemu.run(ptest_cmd) + self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output)) + self.assertIn("PASS: cpp-example-lib", output) + + def _gdb_cross(self): + """Verify gdb-cross is provided by devtool ide-sdk""" + target_arch = self.td["TARGET_ARCH"] + target_sys = self.td["TARGET_SYS"] + gdb_recipe = "gdb-cross-" + target_arch + gdb_binary = target_sys + "-gdb" + + native_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", gdb_recipe) + r = runCmd("%s --version" % gdb_binary, + native_sysroot=native_sysroot, target_sys=target_sys) + self.assertEqual(r.status, 0) + self.assertIn("GNU gdb", r.output) + + def _gdb_cross_debugging(self, qemu, recipe_name, example_exe): + """Verify gdb-cross is working + + Test remote debugging: + break main + run + continue + """ + sshargs = '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' + gdbserver_script = os.path.join(self._workspace_scripts_dir( + recipe_name), 'gdbserver_1234_usr-bin-' + example_exe + '_m') + gdb_script = os.path.join(self._workspace_scripts_dir( + recipe_name), 'gdb_1234_usr-bin-' + example_exe) + + # Start a gdbserver + r = runCmd(gdbserver_script) + self.assertEqual(r.status, 0) + + # Check there is a gdbserver running + r = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, 'ps')) + self.assertEqual(r.status, 0) + self.assertIn("gdbserver ", r.output) + + # Check the pid file is correct + test_cmd = "cat /proc/$(cat /tmp/gdbserver_1234_usr-bin-" + \ + example_exe + "/pid)/cmdline" + r = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, test_cmd)) + self.assertEqual(r.status, 0) + self.assertIn("gdbserver", r.output) + + # Test remote debugging works + r = runCmd( + gdb_script + " --batch -ex 'break main' --ex 'run' -ex 'continue'") + self.assertEqual(r.status, 0) + self.assertIn("Breakpoint 1, main", r.output) + self.assertIn("exited normally", r.output) + + # Stop the gdbserver + r = runCmd(gdbserver_script + ' stop') + self.assertEqual(r.status, 0) + + # Check there is no gdbserver running + r = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, 'ps')) + self.assertEqual(r.status, 0) + self.assertNotIn("gdbserver ", r.output) + + def _verify_cmake_preset(self, tempdir): + """Verify the generated cmake preset works as expected + + Check if compiling works + Check if unit tests can be executed in qemu (not qemu-system) + """ + with open(os.path.join(tempdir, 'CMakeUserPresets.json')) as cmake_preset_j: + cmake_preset_d = json.load(cmake_preset_j) + config_presets = cmake_preset_d["configurePresets"] + self.assertEqual(len(config_presets), 1) + cmake_exe = config_presets[0]["cmakeExecutable"] + preset_name = config_presets[0]["name"] + + # Verify the wrapper for cmake native is available + self.assertExists(cmake_exe) + + # Verify the cmake preset generated by devtool ide-sdk is available + result = runCmd('%s --list-presets' % cmake_exe, cwd=tempdir) + self.assertIn(preset_name, result.output) + + # Verify cmake re-uses the o files compiled by bitbake + result = runCmd('%s --build --preset %s' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("ninja: no work to do.", result.output) + + # Verify the unit tests work (in Qemu user mode) + result = runCmd('%s --build --preset %s --target test' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("100% tests passed", result.output) + + # Verify re-building and testing works again + result = runCmd('%s --build --preset %s --target clean' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("Cleaning", result.output) + result = runCmd('%s --build --preset %s' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("Building", result.output) + self.assertIn("Linking", result.output) + result = runCmd('%s --build --preset %s --target test' % + (cmake_exe, preset_name), cwd=tempdir) + self.assertIn("Running tests...", result.output) + self.assertIn("100% tests passed", result.output) + + @OETestTag("runqemu") + def test_devtool_ide_sdk_none_qemu(self): + """Start qemu-system and run tests for multiple recipes. ide=none is used.""" + recipe_names = ["cmake-example", "meson-example"] + testimage = "oe-selftest-image" + + self._check_workspace() + self._write_bb_config(recipe_names) + self._check_runqemu_prerequisites() + + # Verify deployment to Qemu (system mode) works + bitbake(testimage) + with runqemu(testimage, runqemuparams="nographic") as qemu: + # cmake-example recipe + recipe_name = "cmake-example" + example_exe = "cmake-example" + build_file = "CMakeLists.txt" + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@%s -c --ide=none' % ( + recipe_name, testimage, qemu.ip) + runCmd(bitbake_sdk_cmd) + self._verify_cmake_preset(tempdir) + self._devtool_ide_sdk_qemu(tempdir, qemu, recipe_name, example_exe) + # Verify the oe-scripts sym-link is valid + self.assertEqual(self._workspace_scripts_dir( + recipe_name), self._sources_scripts_dir(tempdir)) + # Verify GDB is working after devtool ide-sdk + self._gdb_cross() + self._gdb_cross_debugging(qemu, recipe_name, example_exe) + + # meson-example recipe + recipe_name = "meson-example" + example_exe = "mesonex" + build_file = "meson.build" + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@%s -c --ide=none' % ( + recipe_name, testimage, qemu.ip) + runCmd(bitbake_sdk_cmd) + self._devtool_ide_sdk_qemu(tempdir, qemu, recipe_name, example_exe) + # Verify the oe-scripts sym-link is valid + self.assertEqual(self._workspace_scripts_dir( + recipe_name), self._sources_scripts_dir(tempdir)) + # Verify GDB is working after devtool ide-sdk + self._gdb_cross() + self._gdb_cross_debugging(qemu, recipe_name, example_exe) + + def test_devtool_ide_sdk_code_cmake(self): + """Verify a cmake recipe works with ide=code mode""" + recipe_name = "cmake-example" + build_file = "CMakeLists.txt" + testimage = "oe-selftest-image" + + self._check_workspace() + self._write_bb_config([recipe_name]) + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@192.168.17.17 -c --ide=code' % ( + recipe_name, testimage) + runCmd(bitbake_sdk_cmd) + self._verify_cmake_preset(tempdir) + self._verify_install_script_code(tempdir, recipe_name) + self._gdb_cross() + + def test_devtool_ide_sdk_code_meson(self): + """Verify a meson recipe works with ide=code mode""" + recipe_name = "meson-example" + build_file = "meson.build" + testimage = "oe-selftest-image" + + self._check_workspace() + self._write_bb_config([recipe_name]) + tempdir = self._devtool_ide_sdk_recipe( + recipe_name, build_file, testimage) + bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@192.168.17.17 -c --ide=code' % ( + recipe_name, testimage) + runCmd(bitbake_sdk_cmd) + + with open(os.path.join(tempdir, '.vscode', 'settings.json')) as settings_j: + settings_d = json.load(settings_j) + meson_exe = settings_d["mesonbuild.mesonPath"] + meson_build_folder = settings_d["mesonbuild.buildFolder"] + + # Verify the wrapper for meson native is available + self.assertExists(meson_exe) + + # Verify meson re-uses the o files compiled by bitbake + result = runCmd('%s compile -C %s' % + (meson_exe, meson_build_folder), cwd=tempdir) + self.assertIn("ninja: no work to do.", result.output) + + # Verify the unit tests work (in Qemu) + runCmd('%s test -C %s' % (meson_exe, meson_build_folder), cwd=tempdir) + + # Verify re-building and testing works again + result = runCmd('%s compile -C %s --clean' % + (meson_exe, meson_build_folder), cwd=tempdir) + self.assertIn("Cleaning...", result.output) + result = runCmd('%s compile -C %s' % + (meson_exe, meson_build_folder), cwd=tempdir) + self.assertIn("Linking target", result.output) + runCmd('%s test -C %s' % (meson_exe, meson_build_folder), cwd=tempdir) + + self._verify_install_script_code(tempdir, recipe_name) + self._gdb_cross() + + def test_devtool_ide_sdk_shared_sysroots(self): + """Verify the shared sysroot SDK""" + + # Handle the workspace (which is not needed by this test case) + self._check_workspace() + + result_init = runCmd( + 'devtool ide-sdk -m shared oe-selftest-image cmake-example meson-example --ide=code') + bb_vars = get_bb_vars( + ['REAL_MULTIMACH_TARGET_SYS', 'DEPLOY_DIR_IMAGE', 'COREBASE'], "meta-ide-support") + environment_script = 'environment-setup-%s' % bb_vars['REAL_MULTIMACH_TARGET_SYS'] + deploydir = bb_vars['DEPLOY_DIR_IMAGE'] + environment_script_path = os.path.join(deploydir, environment_script) + cpp_example_src = os.path.join( + bb_vars['COREBASE'], 'meta-selftest', 'recipes-test', 'cpp', 'files') + + # Verify the cross environment script is available + self.assertExists(environment_script_path) + + def runCmdEnv(cmd, cwd): + cmd = '/bin/sh -c ". %s > /dev/null && %s"' % ( + environment_script_path, cmd) + return runCmd(cmd, cwd) + + # Verify building the C++ example works with CMake + tempdir_cmake = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir_cmake) + + result_cmake = runCmdEnv("which cmake", cwd=tempdir_cmake) + cmake_native = os.path.normpath(result_cmake.output.strip()) + self.assertExists(cmake_native) + + runCmdEnv('cmake %s' % cpp_example_src, cwd=tempdir_cmake) + runCmdEnv('cmake --build %s' % tempdir_cmake, cwd=tempdir_cmake) + + # Verify the printed note really referres to a cmake executable + cmake_native_code = "" + for line in result_init.output.splitlines(): + m = re.search(r'"cmake.cmakePath": "(.*)"', line) + if m: + cmake_native_code = m.group(1) + break + self.assertExists(cmake_native_code) + self.assertEqual(cmake_native, cmake_native_code) + + # Verify building the C++ example works with Meson + tempdir_meson = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir_meson) + + result_cmake = runCmdEnv("which meson", cwd=tempdir_meson) + meson_native = os.path.normpath(result_cmake.output.strip()) + self.assertExists(meson_native) + + runCmdEnv('meson setup %s' % tempdir_meson, cwd=cpp_example_src) + runCmdEnv('meson compile', cwd=tempdir_meson) + + def test_devtool_ide_sdk_plugins(self): + """Test that devtool ide-sdk can use plugins from other layers.""" + + # We need a workspace layer and a modified recipe (but no image) + modified_recipe_name = "meson-example" + modified_build_file = "meson.build" + testimage = "oe-selftest-image" + shared_recipe_name = "cmake-example" + + self._check_workspace() + self._write_bb_config([modified_recipe_name]) + tempdir = self._devtool_ide_sdk_recipe( + modified_recipe_name, modified_build_file, None) + + IDE_RE = re.compile(r'.*--ide \{(.*)\}.*') + + def get_ides_from_help(help_str): + m = IDE_RE.search(help_str) + return m.group(1).split(',') + + # verify the default plugins are available but the foo plugin is not + result = runCmd('devtool ide-sdk -h') + found_ides = get_ides_from_help(result.output) + self.assertIn('code', found_ides) + self.assertIn('none', found_ides) + self.assertNotIn('foo', found_ides) + + shared_config_file = os.path.join(tempdir, 'shared-config.txt') + shared_config_str = 'Dummy shared IDE config' + modified_config_file = os.path.join(tempdir, 'modified-config.txt') + modified_config_str = 'Dummy modified IDE config' + + # Generate a foo plugin in the workspace layer + plugin_dir = os.path.join( + self.workspacedir, 'lib', 'devtool', 'ide_plugins') + os.makedirs(plugin_dir) + plugin_code = 'from devtool.ide_plugins import IdeBase\n\n' + plugin_code += 'class IdeFoo(IdeBase):\n' + plugin_code += ' def setup_shared_sysroots(self, shared_env):\n' + plugin_code += ' with open("%s", "w") as config_file:\n' % shared_config_file + plugin_code += ' config_file.write("%s")\n\n' % shared_config_str + plugin_code += ' def setup_modified_recipe(self, args, image_recipe, modified_recipe):\n' + plugin_code += ' with open("%s", "w") as config_file:\n' % modified_config_file + plugin_code += ' config_file.write("%s")\n\n' % modified_config_str + plugin_code += 'def register_ide_plugin(ide_plugins):\n' + plugin_code += ' ide_plugins["foo"] = IdeFoo\n' + + plugin_py = os.path.join(plugin_dir, 'ide_foo.py') + with open(plugin_py, 'w') as plugin_file: + plugin_file.write(plugin_code) + + # Verify the foo plugin is available as well + result = runCmd('devtool ide-sdk -h') + found_ides = get_ides_from_help(result.output) + self.assertIn('code', found_ides) + self.assertIn('none', found_ides) + self.assertIn('foo', found_ides) + + # Verify the foo plugin generates a shared config + result = runCmd( + 'devtool ide-sdk -m shared --skip-bitbake --ide foo %s' % shared_recipe_name) + with open(shared_config_file) as shared_config: + shared_config_new = shared_config.read() + self.assertEqual(shared_config_str, shared_config_new) + + # Verify the foo plugin generates a modified config + result = runCmd('devtool ide-sdk --skip-bitbake --ide foo %s %s' % + (modified_recipe_name, testimage)) + with open(modified_config_file) as modified_config: + modified_config_new = modified_config.read() + self.assertEqual(modified_config_str, modified_config_new) From patchwork Sun Jan 14 22:14:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09B97C47DA6 for ; Sun, 14 Jan 2024 22:14:51 +0000 (UTC) Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by mx.groups.io with SMTP id smtpd.web10.52018.1705270487277294657 for ; Sun, 14 Jan 2024 14:14:47 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=XMFmQTyN; spf=pass (domain: gmail.com, ip: 209.85.128.52, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-40e586a62f7so56960525e9.2 for ; Sun, 14 Jan 2024 14:14:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270485; x=1705875285; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Sz9hf9rJQjPVy4Q9SbgvcNjWvjydRtS609XVC+jICFg=; b=XMFmQTyNVWYw2hlkd59BKqC+X9lFn4HVcw0dnWdqaQ7gywohl/P3NVmxvkF3jlk8/E e95U6LeEDDERDrqhU7Qm7hX4mKA03wGISnSMdjg6U4QqxBbiT74cKcrhyNpVdesmD9oA 3zZT8M31Sy5rgWj8GlpuWj6JR4FSlRXjUAAw5lLeiHhhA/5PYAwP9yDthFe7fyKOwhEb 4Q/6b0YzGpOuUb/tLNheTbJxAIiglweM4FI9I3hWCK7IAID+XBYnYgoVPDxGZm4eCcbV 7WUXdKuxIsb58C2mAU2sEGn9Mk52EEa/IEpN9hoGRF4zVyMasm7RyYT8KaXgb77+Mk9T ThkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270485; x=1705875285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sz9hf9rJQjPVy4Q9SbgvcNjWvjydRtS609XVC+jICFg=; b=Y/gsD1Dd8/jVkBomTRXJTovXQIRSPUYMLliAUN5kxIjna4VJV2PZTfOQDF7HX8AK6B 6nBC5OFD4huUIDIP/MLGzsZAZWIK/VHx4pyW0HYc7B8eZ2v7Nm/7/Xci+SQmEXZalOKE Ha2UG4yQze5vgLmotfGmc6mbPHBi6aqtNMQJJU3Rhr9QJgBQo/Tvsw+M2n89Hq+1cBAg 5j/2QxGcjXVFrWaC8A5WwAYNFzBD+Y2akfCMxM8J1l4Zohy3Yy+zTL0ZVONwhC76DXC9 1lZC/Rcxy9KOOId4V+ofdtCNnOp0fp3y9+C85R5zeDHDvIoYS+IMSHShlqt2U28ELawQ hexg== X-Gm-Message-State: AOJu0YyBuXvXPXeSwJ14Cu/and6RcTN4kvJRIbeWw52kBrHHctyrX2wX nt/OzAwgYVS2DW9Y9NIK5Od2H2yuYwY= X-Google-Smtp-Source: AGHT+IEMb5913epyxmJmzqikEZL1nzuytZdfI0NOGMY0dzpw328dtsBtEcUwj38ceqF69FFDiFHMrA== X-Received: by 2002:a05:600c:2e05:b0:40e:6ea4:bf57 with SMTP id o5-20020a05600c2e0500b0040e6ea4bf57mr1312316wmf.126.1705270485581; Sun, 14 Jan 2024 14:14:45 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:44 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 5/9] devtool: refactor exec_fakeroot Date: Sun, 14 Jan 2024 23:14:17 +0100 Message-ID: <20240114221437.1255866-6-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193623 Provide a function exec_fakeroot_no_d which does the same like exec_fakeroot does, but is usable independenlty from bitbake. This allows to use the fanction from scripts where the d variable is not available. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py index b4f998a5bc4..c7fc3cfc410 100644 --- a/scripts/lib/devtool/__init__.py +++ b/scripts/lib/devtool/__init__.py @@ -78,12 +78,15 @@ def exec_fakeroot(d, cmd, **kwargs): """Run a command under fakeroot (pseudo, in fact) so that it picks up the appropriate file permissions""" # Grab the command and check it actually exists fakerootcmd = d.getVar('FAKEROOTCMD') + fakerootenv = d.getVar('FAKEROOTENV') + exec_fakeroot_no_d(fakerootcmd, fakerootenv, cmd, kwargs) + +def exec_fakeroot_no_d(fakerootcmd, fakerootenv, cmd, **kwargs): if not os.path.exists(fakerootcmd): logger.error('pseudo executable %s could not be found - have you run a build yet? pseudo-native should install this and if you have run any build then that should have been built') return 2 # Set up the appropriate environment newenv = dict(os.environ) - fakerootenv = d.getVar('FAKEROOTENV') for varvalue in fakerootenv.split(): if '=' in varvalue: splitval = varvalue.split('=', 1) From patchwork Sun Jan 14 22:14:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 099A8C47422 for ; Sun, 14 Jan 2024 22:14:51 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web11.51510.1705270488482104649 for ; Sun, 14 Jan 2024 14:14:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YNayfaff; spf=pass (domain: gmail.com, ip: 209.85.128.54, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-40e72a567eeso8720895e9.0 for ; Sun, 14 Jan 2024 14:14:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270487; x=1705875287; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JwRA+OhhW5FIBT2i7TxbtL1+itpsmXTPUTEbpdriySI=; b=YNayfaffdxkqohuG9pZ1kdBIu95JZc8Rir/z8TVOhnbK6zJiw7WvYQ3oPud/LfvMdQ lnr5fqPplwcvyEDL0srgjI2lZrUuJ7FsPEGkQfV0ZI9X/gJFViiybIXji0TtPDtuxz22 B7Ug8WAveHtreneCslUH+RhD6wiYOJFz1+6bKoWYhVLJdThogdQ997o8AOTRQhEJYME8 HF9OQ5P7c1+26VISBpe1KTsQ34vsmBqyn1IdFsn0nfzLFLLMuOYpqOh9T9FZ3hJ3S50/ ZuqCpLmnt1SEYhZyrENTpaFOcItxNuLHPvUfO4enkGI8m8G67yJ7GfF9nO8YVeTyyguZ beng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270487; x=1705875287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JwRA+OhhW5FIBT2i7TxbtL1+itpsmXTPUTEbpdriySI=; b=HxpBtK0YUi/1Xzl823Qim5V1puI4bq8HaprdlMGxYCAw3u2rswXfTR8jYXzP8C4L07 G5KTO5jqZNjo3owwRguvNV36el9INMf51f0Xi98ESvsel8zy1Ok6bOxtIZVPF35ybX6z lgTozJ4gf7UWYd+9z/6FXI1PtPzSUglUGMyuLPD3HqTs04MNPz8xSJaB/T8ijdzJxnik E902Lm1KuYOWivZHQBUqX2U/BHizS/iKmzz615pPn3rGgO5TAqYaA6GijxxN5lcvlnR4 CpTALG8JlRVFpnZAZhv+wrs6CxgQwZh+A1x4gUcO7gwN+q/YJEORCFFkjrBgbTGYkO33 rILQ== X-Gm-Message-State: AOJu0Yww6ezsmIj17kYb2cISiVix5EPm/qbL/UECyj7DvkwTWe9awhGy c+lSVh58mP2p5FkVrEaAJ3rMavSm8Us= X-Google-Smtp-Source: AGHT+IFPG4395T0k/ldwLHc/QubdmrdrY1cnD6wScs4bXmOYzTdy/pBPE5pkQm/v9Bi4ONQtlyEv3A== X-Received: by 2002:a05:600c:2e51:b0:40e:4267:5738 with SMTP id q17-20020a05600c2e5100b0040e42675738mr1332809wmf.96.1705270486692; Sun, 14 Jan 2024 14:14:46 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:45 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 6/9] devtool: refactor deploy to use exec_fakeroot_no_d Date: Sun, 14 Jan 2024 23:14:18 +0100 Message-ID: <20240114221437.1255866-7-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193624 This is a step towards a deploy function which can be called without passing the d variable. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/deploy.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py index eadf6e15214..b71d55e2312 100644 --- a/scripts/lib/devtool/deploy.py +++ b/scripts/lib/devtool/deploy.py @@ -16,7 +16,7 @@ import bb.utils import argparse_oe import oe.types -from devtool import exec_fakeroot, setup_tinfoil, check_workspace_recipe, DevtoolError +from devtool import exec_fakeroot_no_d, setup_tinfoil, check_workspace_recipe, DevtoolError logger = logging.getLogger('devtool') @@ -160,19 +160,22 @@ def deploy(args, config, basepath, workspace): except Exception as e: raise DevtoolError('Exception parsing recipe %s: %s' % (args.recipename, e)) + recipe_outdir = rd.getVar('D') if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir): raise DevtoolError('No files to deploy - have you built the %s ' 'recipe? If so, the install step has not installed ' 'any files.' % args.recipename) + fakerootcmd = rd.getVar('FAKEROOTCMD') + fakerootenv = rd.getVar('FAKEROOTENV') if args.strip and not args.dry_run: # Fakeroot copy to new destination srcdir = recipe_outdir recipe_outdir = os.path.join(rd.getVar('WORKDIR'), 'devtool-deploy-target-stripped') if os.path.isdir(recipe_outdir): - exec_fakeroot(rd, "rm -rf %s" % recipe_outdir, shell=True) - exec_fakeroot(rd, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "rm -rf %s" % recipe_outdir, shell=True) + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) os.environ['PATH'] = ':'.join([os.environ['PATH'], rd.getVar('PATH') or '']) oe.package.strip_execs(args.recipename, recipe_outdir, rd.getVar('STRIP'), rd.getVar('libdir'), rd.getVar('base_libdir'), oe.utils.get_bb_number_threads(rd), rd) @@ -251,7 +254,7 @@ def deploy(args, config, basepath, workspace): shutil.rmtree(tmpdir) # Now run the script - ret = exec_fakeroot(rd, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) + ret = exec_fakeroot_no_d(fakerootcmd, fakerootenv, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) if ret != 0: raise DevtoolError('Deploy failed - rerun with -s to get a complete ' 'error message') From patchwork Sun Jan 14 22:14:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37734 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFA91C4707B for ; Sun, 14 Jan 2024 22:14:50 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx.groups.io with SMTP id smtpd.web11.51511.1705270489275006074 for ; Sun, 14 Jan 2024 14:14:49 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=bKiUh6J4; spf=pass (domain: gmail.com, ip: 209.85.128.45, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-40e76626170so5834315e9.2 for ; Sun, 14 Jan 2024 14:14:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270487; x=1705875287; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ia6d/hhX/iexjlFxgXuruNl1zuEtuCTVFnk98oKC1o4=; b=bKiUh6J4IBGmLPMaiFgWBG4ZMstxRwOascRendapmK41PI3wG3smsqZK8CmJSKzTsv AldRucWSWuC/YiCMrqBnDRw4cLtVPOQRViwD/vUi9qLI9o+i3586VjYs/jarkfUH85Lg DOyBMZ/HQAM2007ZDsSc7iDDupNJPIJ9wmD/WaEEkpmfJyeMmlAW0D1y80gF6tVjIQpx XZvj/5DtFmCRw27DCDoonC9BN5MmAG5oE6kkadbo0azAWaKnj0cKN7OFLvEGTQO3y2RB Aqf6JZYxpqEjbf94bM/h/KLYWKqWnu11YJ0EDO/WYxqvdAoOagHMTWX/kZIdGHVI8SjF kC8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270487; x=1705875287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ia6d/hhX/iexjlFxgXuruNl1zuEtuCTVFnk98oKC1o4=; b=wjQgh2mGcHJa7gbEhEnSSE4ZaAHGcLocQm5f69NBHSE2fJt+RKKCXMVNtHaJhimDG1 Det7e1KykKmqvAC10upyBJ3isGYuMwybXxifxNTGbQht84q4Me9i2rjSJq/WUaT1Ki5z OwFVkZDGI1LmTimL0AQouhZia5GZFB/H7smo2nhLV7jj9sO2ORz96DuEC0Ezp0yST75u HxftOQ44RGMd+Axf4LX0AHeQj9VqTbZLSQLar7AO9BgaYANOd0gzVbWMNxVhCge4Sk7G yjyx5reKVBgpmjQRlkOzEmY+xEwgqQfSIAYuxQnbNFCEwxdoIQZu91c6PFMLzJ84rT9P fd/g== X-Gm-Message-State: AOJu0YwyvaixFMyYRLkE9bpB1XaCkUhFsSe6XeqmX6xI3oEAfUNceX0b AuJOdvosfcNuH5vVvnjjAflGH6pBDQk= X-Google-Smtp-Source: AGHT+IEA0AVtz+oXlTdkFAGO9jbZiuYMh/42Cz/JUCrFTpDs/l3byZWwYllxuv/wiDS0HEzADOLP5w== X-Received: by 2002:a7b:ce98:0:b0:40e:74fd:51f2 with SMTP id q24-20020a7bce98000000b0040e74fd51f2mr596484wmj.23.1705270487544; Sun, 14 Jan 2024 14:14:47 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:47 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 7/9] devtool: refactor deploy-target Date: Sun, 14 Jan 2024 23:14:19 +0100 Message-ID: <20240114221437.1255866-8-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193625 Make the deploy function independent from d. This allows to call the function also from Python code not running in bitbake. This is needed to for the devtool ide plugin which will call the do_install task and the code from devtool deploy-target independently from a bitbake server. This allows a much quicker workflow. Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/deploy.py | 236 ++++++++++++++++++---------------- 1 file changed, 122 insertions(+), 114 deletions(-) diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py index b71d55e2312..b5ca8f2c2f1 100644 --- a/scripts/lib/devtool/deploy.py +++ b/scripts/lib/devtool/deploy.py @@ -133,17 +133,38 @@ def _prepare_remote_script(deploy, verbose=False, dryrun=False, undeployall=Fals return '\n'.join(lines) - - def deploy(args, config, basepath, workspace): """Entry point for the devtool 'deploy' subcommand""" - import math - import oe.recipeutils - import oe.package import oe.utils check_workspace_recipe(workspace, args.recipename, checksrc=False) + tinfoil = setup_tinfoil(basepath=basepath) + try: + try: + rd = tinfoil.parse_recipe(args.recipename) + except Exception as e: + raise DevtoolError('Exception parsing recipe %s: %s' % + (args.recipename, e)) + + srcdir = rd.getVar('D') + workdir = rd.getVar('WORKDIR') + path = rd.getVar('PATH') + strip_cmd = rd.getVar('STRIP') + libdir = rd.getVar('libdir') + base_libdir = rd.getVar('base_libdir') + max_process = oe.utils.get_bb_number_threads(rd) + fakerootcmd = rd.getVar('FAKEROOTCMD') + fakerootenv = rd.getVar('FAKEROOTENV') + finally: + tinfoil.shutdown() + + return deploy_no_d(srcdir, workdir, path, strip_cmd, libdir, base_libdir, max_process, fakerootcmd, fakerootenv, args) + +def deploy_no_d(srcdir, workdir, path, strip_cmd, libdir, base_libdir, max_process, fakerootcmd, fakerootenv, args): + import math + import oe.package + try: host, destdir = args.target.split(':') except ValueError: @@ -153,121 +174,108 @@ def deploy(args, config, basepath, workspace): if not destdir.endswith('/'): destdir += '/' - tinfoil = setup_tinfoil(basepath=basepath) - try: - try: - rd = tinfoil.parse_recipe(args.recipename) - except Exception as e: - raise DevtoolError('Exception parsing recipe %s: %s' % - (args.recipename, e)) + recipe_outdir = srcdir + if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir): + raise DevtoolError('No files to deploy - have you built the %s ' + 'recipe? If so, the install step has not installed ' + 'any files.' % args.recipename) + + if args.strip and not args.dry_run: + # Fakeroot copy to new destination + srcdir = recipe_outdir + recipe_outdir = os.path.join(workdir, 'devtool-deploy-target-stripped') + if os.path.isdir(recipe_outdir): + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "rm -rf %s" % recipe_outdir, shell=True) + exec_fakeroot_no_d(fakerootcmd, fakerootenv, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) + os.environ['PATH'] = ':'.join([os.environ['PATH'], path or '']) + oe.package.strip_execs(args.recipename, recipe_outdir, strip_cmd, libdir, base_libdir, max_process) + + filelist = [] + inodes = set({}) + ftotalsize = 0 + for root, _, files in os.walk(recipe_outdir): + for fn in files: + fstat = os.lstat(os.path.join(root, fn)) + # Get the size in kiB (since we'll be comparing it to the output of du -k) + # MUST use lstat() here not stat() or getfilesize() since we don't want to + # dereference symlinks + if fstat.st_ino in inodes: + fsize = 0 + else: + fsize = int(math.ceil(float(fstat.st_size)/1024)) + inodes.add(fstat.st_ino) + ftotalsize += fsize + # The path as it would appear on the target + fpath = os.path.join(destdir, os.path.relpath(root, recipe_outdir), fn) + filelist.append((fpath, fsize)) + + if args.dry_run: + print('Files to be deployed for %s on target %s:' % (args.recipename, args.target)) + for item, _ in filelist: + print(' %s' % item) + return 0 - recipe_outdir = rd.getVar('D') - if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir): - raise DevtoolError('No files to deploy - have you built the %s ' - 'recipe? If so, the install step has not installed ' - 'any files.' % args.recipename) + extraoptions = '' + if args.no_host_check: + extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' + if not args.show_status: + extraoptions += ' -q' - fakerootcmd = rd.getVar('FAKEROOTCMD') - fakerootenv = rd.getVar('FAKEROOTENV') - if args.strip and not args.dry_run: - # Fakeroot copy to new destination - srcdir = recipe_outdir - recipe_outdir = os.path.join(rd.getVar('WORKDIR'), 'devtool-deploy-target-stripped') - if os.path.isdir(recipe_outdir): - exec_fakeroot_no_d(fakerootcmd, fakerootenv, "rm -rf %s" % recipe_outdir, shell=True) - exec_fakeroot_no_d(fakerootcmd, fakerootenv, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True) - os.environ['PATH'] = ':'.join([os.environ['PATH'], rd.getVar('PATH') or '']) - oe.package.strip_execs(args.recipename, recipe_outdir, rd.getVar('STRIP'), rd.getVar('libdir'), - rd.getVar('base_libdir'), oe.utils.get_bb_number_threads(rd), rd) - - filelist = [] - inodes = set({}) - ftotalsize = 0 - for root, _, files in os.walk(recipe_outdir): - for fn in files: - fstat = os.lstat(os.path.join(root, fn)) - # Get the size in kiB (since we'll be comparing it to the output of du -k) - # MUST use lstat() here not stat() or getfilesize() since we don't want to - # dereference symlinks - if fstat.st_ino in inodes: - fsize = 0 - else: - fsize = int(math.ceil(float(fstat.st_size)/1024)) - inodes.add(fstat.st_ino) - ftotalsize += fsize - # The path as it would appear on the target - fpath = os.path.join(destdir, os.path.relpath(root, recipe_outdir), fn) - filelist.append((fpath, fsize)) - - if args.dry_run: - print('Files to be deployed for %s on target %s:' % (args.recipename, args.target)) - for item, _ in filelist: - print(' %s' % item) - return 0 - - extraoptions = '' - if args.no_host_check: - extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' - if not args.show_status: - extraoptions += ' -q' - - scp_sshexec = '' - ssh_sshexec = 'ssh' - if args.ssh_exec: - scp_sshexec = "-S %s" % args.ssh_exec - ssh_sshexec = args.ssh_exec - scp_port = '' - ssh_port = '' - if args.port: - scp_port = "-P %s" % args.port - ssh_port = "-p %s" % args.port - - if args.key: - extraoptions += ' -i %s' % args.key - - # In order to delete previously deployed files and have the manifest file on - # the target, we write out a shell script and then copy it to the target - # so we can then run it (piping tar output to it). - # (We cannot use scp here, because it doesn't preserve symlinks.) - tmpdir = tempfile.mkdtemp(prefix='devtool') - try: - tmpscript = '/tmp/devtool_deploy.sh' - tmpfilelist = os.path.join(os.path.dirname(tmpscript), 'devtool_deploy.list') - shellscript = _prepare_remote_script(deploy=True, - verbose=args.show_status, - nopreserve=args.no_preserve, - nocheckspace=args.no_check_space) - # Write out the script to a file - with open(os.path.join(tmpdir, os.path.basename(tmpscript)), 'w') as f: - f.write(shellscript) - # Write out the file list - with open(os.path.join(tmpdir, os.path.basename(tmpfilelist)), 'w') as f: - f.write('%d\n' % ftotalsize) - for fpath, fsize in filelist: - f.write('%s %d\n' % (fpath, fsize)) - # Copy them to the target - ret = subprocess.call("scp %s %s %s %s/* %s:%s" % (scp_sshexec, scp_port, extraoptions, tmpdir, args.target, os.path.dirname(tmpscript)), shell=True) - if ret != 0: - raise DevtoolError('Failed to copy script to %s - rerun with -s to ' - 'get a complete error message' % args.target) - finally: - shutil.rmtree(tmpdir) + scp_sshexec = '' + ssh_sshexec = 'ssh' + if args.ssh_exec: + scp_sshexec = "-S %s" % args.ssh_exec + ssh_sshexec = args.ssh_exec + scp_port = '' + ssh_port = '' + if args.port: + scp_port = "-P %s" % args.port + ssh_port = "-p %s" % args.port - # Now run the script - ret = exec_fakeroot_no_d(fakerootcmd, fakerootenv, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) + if args.key: + extraoptions += ' -i %s' % args.key + + # In order to delete previously deployed files and have the manifest file on + # the target, we write out a shell script and then copy it to the target + # so we can then run it (piping tar output to it). + # (We cannot use scp here, because it doesn't preserve symlinks.) + tmpdir = tempfile.mkdtemp(prefix='devtool') + try: + tmpscript = '/tmp/devtool_deploy.sh' + tmpfilelist = os.path.join(os.path.dirname(tmpscript), 'devtool_deploy.list') + shellscript = _prepare_remote_script(deploy=True, + verbose=args.show_status, + nopreserve=args.no_preserve, + nocheckspace=args.no_check_space) + # Write out the script to a file + with open(os.path.join(tmpdir, os.path.basename(tmpscript)), 'w') as f: + f.write(shellscript) + # Write out the file list + with open(os.path.join(tmpdir, os.path.basename(tmpfilelist)), 'w') as f: + f.write('%d\n' % ftotalsize) + for fpath, fsize in filelist: + f.write('%s %d\n' % (fpath, fsize)) + # Copy them to the target + ret = subprocess.call("scp %s %s %s %s/* %s:%s" % (scp_sshexec, scp_port, extraoptions, tmpdir, args.target, os.path.dirname(tmpscript)), shell=True) if ret != 0: - raise DevtoolError('Deploy failed - rerun with -s to get a complete ' - 'error message') + raise DevtoolError('Failed to copy script to %s - rerun with -s to ' + 'get a complete error message' % args.target) + finally: + shutil.rmtree(tmpdir) - logger.info('Successfully deployed %s' % recipe_outdir) + # Now run the script + ret = exec_fakeroot_no_d(fakerootcmd, fakerootenv, 'tar cf - . | %s %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True) + if ret != 0: + raise DevtoolError('Deploy failed - rerun with -s to get a complete ' + 'error message') - files_list = [] - for root, _, files in os.walk(recipe_outdir): - for filename in files: - filename = os.path.relpath(os.path.join(root, filename), recipe_outdir) - files_list.append(os.path.join(destdir, filename)) - finally: - tinfoil.shutdown() + logger.info('Successfully deployed %s' % recipe_outdir) + + files_list = [] + for root, _, files in os.walk(recipe_outdir): + for filename in files: + filename = os.path.relpath(os.path.join(root, filename), recipe_outdir) + files_list.append(os.path.join(destdir, filename)) return 0 From patchwork Sun Jan 14 22:14:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BADFC47DAF for ; Sun, 14 Jan 2024 22:14:51 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web10.52021.1705270490474785856 for ; Sun, 14 Jan 2024 14:14:50 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=OBoq55A6; spf=pass (domain: gmail.com, ip: 209.85.128.43, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-40e7c09f0fcso687585e9.0 for ; Sun, 14 Jan 2024 14:14:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270489; x=1705875289; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s8VEseN/IW3LjI0L9LLd+eKLJ2pFK4iV91KX/GdvnF0=; b=OBoq55A6fjsv9m2JG1NfPH85ijkpKPXm4h8Jc8zy3Uq8w4rYa6ksoFpEfU5aeOPigu 27MgVnoXcuqpAG533vw8UnFymFJW8CeXaODbXR7yOnDY5POetes6jINu+MMsRaBAGF98 YcFDXdK0G04AmcjUw+9tynEOK7mMBCAutPnnxRomwSmZlvd560udu0PAL2QT5KwCyEmy I4prNBvvNmmCkJA5LftkJueLuhh4a9efQb+LShRzevlgqFhf5EXywrbqWnVH+uBlTtbQ Iee6SU48dvKE5lJZ9VMl4Qlk/DP7J+D+JpE66i97oDTvieWmA6QJgTJb1gsrunZgehLV P4GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270489; x=1705875289; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s8VEseN/IW3LjI0L9LLd+eKLJ2pFK4iV91KX/GdvnF0=; b=wUZT54yN8T7hw+WPxajnpBfbf4mnF1sRm0vooGIIS6sPOcZ+69Rhut/Qyj5YrKvGGT 8HyFL5BKMjYTeuhas224txA79vGOr6LnhZzrq5XejXokQGo1CEFKMiPuxciVbH0x1PT2 6MTmTN8VzSjzxujV2vPLVC/H2Nk29+aHfC3DsLpkl4646I0z1w26af7O/8sCQJ9yhdXL 5uD6VgqJlL3SNi6PAsMv5HavZ83aEviLF2MIoaCWHDWnkSOJCDVy73TtUIDj1mKY5KRP f8CXPsiSJ2ImiLEjHEMNpXH2H2WDwjBCZyK6jxE7kUkJl0tV3EEvatj0/Pu0FMcFqH8w XkgA== X-Gm-Message-State: AOJu0YwtyaLUjxzVNOfSJkKIBILHK/ij9iOebCGQRjlD03lmkjtnHvwX 4yHjGMlGEHCcvUMOwVjZdMGbNnJBUSU= X-Google-Smtp-Source: AGHT+IEFaOKgrShbgNU8iFjSj2NUUARZEOpSxeRpaa/MAP/fjovSYytuG6tJlNKmoDcZIg71ioerCw== X-Received: by 2002:a05:600c:81b:b0:40e:44a9:5664 with SMTP id k27-20020a05600c081b00b0040e44a95664mr2300412wmp.231.1705270488540; Sun, 14 Jan 2024 14:14:48 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:47 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 8/9] devtool: ide-sdk make deploy-target quicker Date: Sun, 14 Jan 2024 23:14:20 +0100 Message-ID: <20240114221437.1255866-9-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:14:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193626 Signed-off-by: Adrian Freihofer --- scripts/lib/devtool/ide_sdk.py | 43 ++++++++++++++++++++++++++++++---- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/scripts/lib/devtool/ide_sdk.py b/scripts/lib/devtool/ide_sdk.py index 8c9ed3a2217..5eea039395c 100755 --- a/scripts/lib/devtool/ide_sdk.py +++ b/scripts/lib/devtool/ide_sdk.py @@ -13,6 +13,7 @@ import re import shutil import stat import subprocess +import sys from argparse import RawTextHelpFormatter from enum import Enum @@ -20,7 +21,7 @@ import scriptutils import bb from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError, parse_recipe from devtool.standard import get_real_srctree -from devtool.ide_plugins import BuildTool, get_devtool_deploy_opts +from devtool.ide_plugins import BuildTool logger = logging.getLogger('devtool') @@ -742,6 +743,39 @@ class RecipeModified: return self.write_script(cmd_lines, 'delete_package_dirs') + def gen_deploy_target_script(self, args): + """Generate a script which does what devtool deploy-target does + + This script is much quicker than devtool target-deploy. Because it + does not need to start a bitbake server. All information from tinfoil + is hard-coded in the generated script. + """ + cmd_lines = ['#!/usr/bin/env python3'] + cmd_lines.append('import sys') + cmd_lines.append('devtool_sys_path = %s' % str(sys.path)) + cmd_lines.append('devtool_sys_path.reverse()') + cmd_lines.append('for p in devtool_sys_path:') + cmd_lines.append(' if p not in sys.path:') + cmd_lines.append(' sys.path.insert(0, p)') + cmd_lines.append('from devtool.deploy import deploy_no_d') + args_filter = ['debug', 'dry_run', 'key', 'no_check_space', 'no_host_check', + 'no_preserve', 'port', 'show_status', 'ssh_exec', 'strip', 'target'] + filtered_args_dict = {key: value for key, value in vars( + args).items() if key in args_filter} + cmd_lines.append('filtered_args_dict = %s' % str(filtered_args_dict)) + cmd_lines.append('class Dict2Class(object):') + cmd_lines.append(' def __init__(self, my_dict):') + cmd_lines.append(' for key in my_dict:') + cmd_lines.append(' setattr(self, key, my_dict[key])') + cmd_lines.append('filtered_args = Dict2Class(filtered_args_dict)') + cmd_lines.append( + 'setattr(filtered_args, "recipename", "%s")' % self.bpn) + cmd_lines.append('deploy_no_d("%s", "%s", "%s", "%s", "%s", "%s", %d, "%s", "%s", filtered_args)' % + (self.d, self.workdir, self.path, self.strip_cmd, + self.libdir, self.base_libdir, self.max_process, + self.fakerootcmd, self.fakerootenv)) + return self.write_script(cmd_lines, 'deploy_target') + def gen_install_deploy_script(self, args): """Generate a script which does install and deploy""" cmd_lines = ['#!/bin/sh -e'] @@ -757,10 +791,9 @@ class RecipeModified: # bitbake -c install cmd_lines.append("bitbake %s -c install" % self.bpn) - # devtool deploy-target - deploy_opts = ' '.join(get_devtool_deploy_opts(args)) - cmd_lines.append("devtool deploy-target %s %s" % - (self.bpn, deploy_opts)) + # Self contained devtool deploy-target + cmd_lines.append(self.gen_deploy_target_script(args)) + return self.write_script(cmd_lines, 'install_and_deploy') def write_script(self, cmd_lines, script_name): From patchwork Sun Jan 14 22:14:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Freihofer X-Patchwork-Id: 37742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46AE2C47077 for ; Sun, 14 Jan 2024 22:15:01 +0000 (UTC) Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by mx.groups.io with SMTP id smtpd.web10.52022.1705270491674183926 for ; Sun, 14 Jan 2024 14:14:52 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=O9TFGtPc; spf=pass (domain: gmail.com, ip: 209.85.221.50, mailfrom: adrian.freihofer@gmail.com) Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-336897b6bd6so7964357f8f.2 for ; Sun, 14 Jan 2024 14:14:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705270490; x=1705875290; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vkA8JmIqu+tEZQPINCgxYk+1SogLmP0MfuItZDG+1ho=; b=O9TFGtPcHKIepZ7s6nXrYLRP4vnrqm1Na89Pzg00dnS+NyYy4HQ9aOsDihvgzWLkpR H4han1s7U+yT6lYl9k4EiZ/oSfbRmyf75ycKhgxnFz8/n8zRe5Tj7P4YiXlLjeGHkykH m6CJJuTJkltH7Dar/jFSZOr2XdgvNIRpNc3syU//6W33Kuepq9jfyj8b6+wFt/1dTAx6 ByDacgjZ/wCEcSRhlC/GYqt7wKGyQFpGYMXUVd19BM+yUwhUDhL5ouo8MdZBSnf4/QVL 3clZ+yB8KBtlvISjngRKE9SbeFREH2FEsQ2cVUnxCB/SC0jESiiyls3sKGmXPztvfEtQ a6tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705270490; x=1705875290; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vkA8JmIqu+tEZQPINCgxYk+1SogLmP0MfuItZDG+1ho=; b=PSB2kkSLcdQha1HzO94TxG85AbYi9A2dNEwizE3FJO7yPk2aYR37cPsUufjEc7baAi LqsM//TCI8nCRGiMwaYdW1nJff5j7AxfDjGv8c1JGw/WmkOfhlOr0EoDkOp9NehXB9Qb 7CDEinyRHD7mRfgzC+BZmSszgH3/VyMHvVUUQdNmMBLM9K5tEKyWAHE5MeT9N0/duCC/ Fy0SU+8fDEBe1DN5T3HEA8eOb/vk61obvvFh003Ly8/jDaPRkHrilgqi1/It9DEnrvBe 9YOEog2c5udX5w5iG6KWktpb2lHj3mh9Ymt8QphqqWIYN/aJn0rfpNUSuw/7p5rpBtVU TEKQ== X-Gm-Message-State: AOJu0YyO89vcNgUlPC+mtZzJOZ2B0UKAGhVR/+o2tQ1W47Ud2+QnlgX7 p27E/kv7tCzIb+ozrL4WNhWF19t7zHM= X-Google-Smtp-Source: AGHT+IFHaKsSCkxKx1zFx+KtxxIlQHQkbSUh0k7lms0LWB3a5FoCE8vwLmo4UHQOkHWb309RYdqVkQ== X-Received: by 2002:a05:600c:a01c:b0:40e:5a02:7820 with SMTP id jg28-20020a05600ca01c00b0040e5a027820mr2357155wmb.13.1705270489763; Sun, 14 Jan 2024 14:14:49 -0800 (PST) Received: from wsadrian16.fritz.box ([2a02:169:59a6:0:55c4:f628:91f3:4287]) by smtp.gmail.com with ESMTPSA id t21-20020a05600c451500b0040e3ac9f4c8sm17451943wmo.28.2024.01.14.14.14.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 14:14:48 -0800 (PST) From: Adrian Freihofer X-Google-Original-From: Adrian Freihofer To: openembedded-core@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v9 9/9] WIP: sdk-manual: extensible.rst: cover devtool ide-sdk Date: Sun, 14 Jan 2024 23:14:21 +0100 Message-ID: <20240114221437.1255866-10-adrian.freihofer@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240114221437.1255866-1-adrian.freihofer@siemens.com> References: <20240114221437.1255866-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 14 Jan 2024 22:15:01 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/193627 Cover the new devtool ide plugin in the extensible sdk section. Many thanks to Enguerrand de Ribaucourt for his re-view and contributions. Signed-off-by: Adrian Freihofer --- documentation/sdk-manual/extensible.rst | 257 +++++++++++++++++++++++- 1 file changed, 256 insertions(+), 1 deletion(-) diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst index 355c6cb0e4a..7a80f94a38b 100644 --- a/documentation/sdk-manual/extensible.rst +++ b/documentation/sdk-manual/extensible.rst @@ -63,6 +63,8 @@ their own pros and cons: need to provide a well-functioning binary artefact cache over the network for developers with underpowered laptops. +.. _setting_up_ext_sdk_in_build: + Setting up the Extensible SDK environment directly in a Yocto build ------------------------------------------------------------------- @@ -168,6 +170,8 @@ architecture. The example assumes the SDK installer is located in that case, set up the proper permissions in the directory and run the installer again. +.. _running_the_ext_sdk_env: + Running the Extensible SDK Environment Setup Script =================================================== @@ -205,6 +209,8 @@ use the SDK (e.g. ``PATH``, :term:`CC`, :term:`LD`, and so forth). If you want to see all the environment variables the script exports, examine the installation file itself. +.. _using_devtool: + Using ``devtool`` in Your SDK Workflow ====================================== @@ -230,13 +236,15 @@ all the commands. See the ":doc:`/ref-manual/devtool-reference`" section in the Yocto Project Reference Manual. -Three ``devtool`` subcommands provide entry-points into development: +``devtool`` subcommands provide entry-points into development: - *devtool add*: Assists in adding new software to be built. - *devtool modify*: Sets up an environment to enable you to modify the source of an existing component. +- *devtool ide-sdk*: Generates a configuration for an IDE. + - *devtool upgrade*: Updates an existing recipe so that you can build it for an updated set of source files. @@ -614,6 +622,253 @@ command: decide you do not want to proceed with your work. If you do use this command, realize that the source tree is preserved. +``devtool ide-sdk`` configures IDEs for the extensible SDK +---------------------------------------------------------- + +``devtool ide-sdk`` automatically configures IDEs to use the extensible SDK. +To make sure that all parts of the extensible SDK required by the generated IDE configuration are +available, ``devtool ide-sdk`` uses bitbake in the background to bootstrap the extensible SDK. + +The extensible SDK supports two different development modes. +``devtool ide-sdk`` supports both of them: + +#. *Modified mode*: + + By default ``devtool ide-sdk`` generates IDE configurations for recipes in workspaces + created by ``devtool modify`` or ``devtool add`` as described in :ref:`using_devtool`. + This mode creates IDE configurations with support for advanced features, such as deploying + the binaries to the remote target device and performing remote debugging sessions. + The generated IDE configurations use the per recipe sysroots as Bitbake does internally. + + In order to use the tool, a few settings must be made. + As a starting example, the following lines of code can be added to the ``local.conf`` file:: + + # Build the companion debug file system + IMAGE_GEN_DEBUGFS = "1" + # Optimize build time: with devtool ide-sdk the dbg tar is not needed + IMAGE_FSTYPES_DEBUGFS = "" + # Without copying the binaries into roofs-dbg, GDB does not find all source files. + IMAGE_CLASSES += "image-combined-dbg" + + # ssh is mandatory, no password simplifies the usage + EXTRA_IMAGE_FEATURES += "\ + ssh-server-openssh \ + debug-tweaks \ + " + + # Remote debugging needs the gdbserver on the target device + IMAGE_INSTALL:append = " gdbserver" + + # Add the recipes which should be modified to the image + # Otherwise some dependencies might be missing. + IMAGE_INSTALL:append = " my-recipe" + + Assuming the bitbake environment is set up correctly and a workspace has been created + for the recipe using ``devtool modify my-recipe``, the following command can create the + SDK and the configuration for VSCode in the recipe workspace:: + + $ devtool ide-sdk my-recipe core-image-minimal --target root@192.168.7.2 + + The command requires an image recipe (``core-image-minimal`` for this example) + that is used to create the SDK. + This firmware image should also be installed on the target device. + It is possible to pass multiple package recipes. + ``devtool ide-sdk`` tries to create an IDE configuration for all package recipes. + + Exactly what this command does depends on the recipe respectively on the build tool used by + the recipe. The basic idea is to configure the IDE so that it calls the build tool exactly + as ``bitbake`` does. + + For example, a CMake preset is created for a recipe that inherits :ref:`ref-classes-cmake`. + In the case of VSCode, CMake presets are supported by the CMake Tools plugin. + This is an example of how the build configuration used by ``bitbake`` is exported to an IDE + configuration that gives exactly the same build results. + + Support for remote debugging with seamless integration into the IDE is important for a cross-SDK. + ``devtool ide-sdk`` automatically generates the necessary helper scripts for deploying the compiled + artifacts to the target device as well as the necessary configuration for the debugger and the IDE. + + .. note:: + + To ensure that the debug symbols on the build machine match the binaries running on the target device, + it is essential that the image built by ``devtool ide-sdk`` is running on the target device. + + ``devtool ide-sdk`` aims to support multiple programming languages and multiple IDEs natively. + Native means that the IDE is configured to call the build tool (e.g. CMake or Meson) directly. + This has several advantages. First of all, it is much faster than ``devtool build``. + But it also allows to use the very good integration of tools like CMake or GDB directly with VSCode or other IDEs. + However, supporting many programming languages and multiple IDEs is quite an elaborate and constantly evolving thing. + Support for IDEs is therefore implemented as plugins. + Plugins can also be provided by optional layers. + + The default IDE is VSCode. Some hints about using VSCode: + + - To work on the source code of a recipe an instance of VSCode is started in the recipe's workspace. + Example:: + + code build/workspace/sources/my-recipe + + - To work with CMake press ``Ctrl + Shift + p``, type ``cmake``. + This will show some possible commands like selecting a CMake preset, compiling or running CTest. + + For recipes inheriting :ref:`ref-classes-cmake-qemu` rather than :ref:`ref-classes-cmake` executing + cross-compiled unit tests on the host can be supported transparently with QEMU user-mode. + + - To work with Meson press ``Ctrl + Shift + p``, type ``meson``. + This will show some possible commands like compiling or executing the unit tests. + + A note on running cross-compiled unit tests on the host: Meson enables support for QEMU user-mode by default. + It is expected that the execution of the unit tests from the IDE will work easily without any additional steps, + provided that the code is suitable for execution on the host machine. + + - For the deployment to the target device, just press ``Ctrl + Shift + p``, type ``task``. + Select the ``install && deploy-target``. + + - For remote debugging, switch to the debugging view by pressing the "play" button with the ``bug icon`` on the left side. + This will provide a green play button with a drop-down list where a debug configuration can be selected. + After selecting one of the generated configurations, press the "play" button. + + Starting a remote debugging session automatically initiates the deployment to the target device. + If this is not desired, the ``"dependsOn": ["install && deploy-target...]`` parameter of the tasks + with ``"label": "gdbserver start...`` can be removed from the ``tasks.json`` file. + + VSCode supports GDB with many different setups and configurations for many different use cases. + However, most of these setups have some limitations when it comes to cross-development, support only a few target + architectures or require a high performance target device. + Therefore ``devtool ide-sdk`` supports the classic, generic setup with GDB on the development host and gdbserver + on the target device. + Roughly summarized, this means: + + - The binaries are copied via ssh to the remote target device by a script referred by ``tasks.json``. + - gdbserver is started on the remote target device via ssh by a script referred by ``tasks.json``. + + Changing the parameters that are passed to the debugging executable requires changing the generated script. + The scipts is located at ``oe-scripts/gdbserver_*``. + The definition of the parameters in the ``args`` field in the ``launch.json`` file does not work. + + - VSCode connects to the gdbserver as documented in + `Remote debugging or debugging with a local debugger server `__. + + Additionally ``--ide=none`` is supported. + With the none IDE parameter some generic configurations files like ``gdbinit`` files and some helper scripts + starting the gdbserver remotely on the target device as well as the gdb client on the host are generated. + + Usage example for the cmake-example recipe from the meta-selftest layer + which inherits :ref:`ref-classes-cmake-qemu`: + + .. code-block:: sh + + # Create the SDK + devtool modify cmake-example + devtool ide-sdk cmake-example core-image-minimal -c --debug-build-config --ide=none + + # Install the firmware on a target device or start QEMU + runqemu + + # From a Navigate into the workspace of cmake-example + cd build/workspace/sources/cmake-example + + # Find cmake-native and save the path into a variable + # Note: using just cmake instead of $CMAKE_NATIVE would work in many cases + CMAKE_NATIVE="$(jq -r '.configurePresets[0] | "\(.cmakeExecutable)"' CMakeUserPresets.json)" + + # List available CMake presets + "$CMAKE_NATIVE" --list-presets + Available configure presets: + + "cmake-example-cortexa57" - cmake-example: cortexa57 + + # Re-compile the already compiled sources + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 + ninja: no work to do. + # Do a clean re-build + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target clean + [1/1] Cleaning all built files... + Cleaning... 8 files. + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target all + [7/7] Linking CXX executable cmake-example + + # Run the cross-compiled unit tests with QEMU user-mode + "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target test + [0/1] Running tests... + Test project .../build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0 + Start 1: test-cmake-example + 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec + + 100% tests passed, 0 tests failed out of 1 + + Total Test time (real) = 0.03 sec + + # Using CTest directly is possible as well + CTEST_NATIVE="$(dirname "$CMAKE_NATIVE")/ctest" + + # List available CMake presets + "$CTEST_NATIVE" --list-presets + Available test presets: + + "cmake-example-cortexa57" - cmake-example: cortexa57 + + # Run the cross-compiled unit tests with QEMU user-mode + "$CTEST_NATIVE" --preset "cmake-example-cortexa57" + Test project ...build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0 + Start 1: test-cmake-example + 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec + + 100% tests passed, 0 tests failed out of 1 + + Total Test time (real) = 0.03 sec + + # Deploying the new build to the target device (default is QEUM at 192.168.7.2) + oe-scripts/install_and_deploy_cmake-example-cortexa57 + + # Start a remote debugging session with gdbserver on the target ang GDB on the host + oe-scripts/gdbserver_1234_usr-bin-cmake-example_m + oe-scripts/gdb_1234_usr-bin-cmake-example + break main + run + step + stepi + continue + quit + + # Stop the gdbserver on the target device + oe-scripts/gdbserver_1234_usr-bin-cmake-example_m stop + +#. *Shared sysroots mode* + + For some recipes and use cases a per-recipe sysroot based SDK is not suitable. + Optionally ``devtool ide-sdk`` configures the IDE to use the tool-chain provided by the + extensible SDK as described in :ref:`running_the_ext_sdk_env`. + ``devtool ide-sdk --mode=shared`` is basically a wrapper for the setup of the extensible SDK + as described in :ref:`setting_up_ext_sdk_in_build`. + The IDE gets configuration to use the shared sysroots. + + Creating a SDK with shared sysroots that contains all the dependencies needed to work with my-recipe is possible + with the following example command:: + + $ devtool ide-sdk --mode=shared my-recipe + + For VSCode the cross-tool-chain is exposed as a CMake kit. CMake kits are defined in + ``~/.local/share/CMakeTools/cmake-tools-kits.json``. + The following example shows how the cross-toolchain can be selected in VSCode. + Fist of all we need a folder containing a CMake project. + For this example lets create a CMake project and start VSCode:: + + mkdir kit-test + echo "project(foo VERSION 1.0)" > kit-test/CMakeLists.txt + code kit-test + + If there is a CMake project in the workspace cross-compilation is supported: + + - Press ``Ctrl + Shift + P``, type ``CMake: Scan for Kits`` + - Press ``Ctrl + Shift + P``, type ``CMake: Select a Kit`` + + Finally most of the features provided by CMake and the IDE should be available. + + Other IDEs than VSCode are supported as well. + However, ``devtool ide-sdk --mode=shared --ide=none my-recipe`` is currently just a simple wrapper + for the setup of the extensible SDK as described in :ref:`setting_up_ext_sdk_in_build`. + Use ``devtool upgrade`` to Create a Version of the Recipe that Supports a Newer Version of the Software -------------------------------------------------------------------------------------------------------