From patchwork Mon Apr 15 06:02:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SCHNEIDER Johannes X-Patchwork-Id: 42339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3F4CC04FFF for ; Mon, 15 Apr 2024 06:02:15 +0000 (UTC) Received: from EUR02-AM0-obe.outbound.protection.outlook.com (EUR02-AM0-obe.outbound.protection.outlook.com [40.107.247.58]) by mx.groups.io with SMTP id smtpd.web10.14554.1713160930950673491 for ; Sun, 14 Apr 2024 23:02:11 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@leica-geosystems.com header.s=selector1 header.b=UZ8hjS2K; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: leica-geosystems.com, ip: 40.107.247.58, mailfrom: johannes.schneider@leica-geosystems.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LxNkACip4W7JOCpHLmShy+j5h2dajWcfqgNqr2gRpRnnkVC0MJFjkMxfOYfZndVkZLumfdCdiwvNImeOY1yrYUntaCExrXuNvrfnhDjM98DF0DMGmsT9KxZSUjx/ZqgdIH5uVlFLfLZ+vllb0B3gK24DmNAIW4At1qESMgcFQcnT4u0pp8dlE1wMPlrTIcELK2Xf8WUV9512eFRT6VUEbqTcmz68/NcS2oZtBzClHEWuELHICDK52lFSDwS/gz6WdaQxBqOac92+rCQFbr0/RCWa2+R16hN5nTvfFmLOY3EusgJmAIkB4C1oWdKA5KHhAcBN0xtxeYz1g+AaAujJFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GRItkBsFQSl4E7+3wALB3IGXKJywchs7t/WA1hOwkBM=; b=lbARWf1t8QHouRquNzPL4MgHgPJCiRbT0wCVYx9zC+YVJ/L+z0tOoEYHylsYm7SJUOPyA2GB3w5kTJywxmpnMeiKtffbqIvroSpVOgt1mb/7FirFFiIUyfB2uYeQee4b/d6xm4bvO0egUTGs6Ojc2ybQflqTYj6hR/8FBfpN6zt6vrtUO+N6GfZIUmIt7lDDDvlanJ8jcCIswOxS9nJbV13v6Tgo48INO4E7NHWVFV6ubquM8U9Hq0dWs1G1+7mzQdba1jN9OVQ3sBY3/id1kJ38HBPOTBrsdt/6E0QPoi9MUyV/cRzoauu8Sx8eNbJ9HasB5e9fi+Ut1m/wUJt5/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 193.8.40.94) smtp.rcpttodomain=lists.openembedded.org smtp.mailfrom=leica-geosystems.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=leica-geosystems.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leica-geosystems.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GRItkBsFQSl4E7+3wALB3IGXKJywchs7t/WA1hOwkBM=; b=UZ8hjS2KAt1WbyWibe06o3oL1YKY1JXUnhJMrsOkrVu/+3hAQOuNccTGmWfqVfgTXVQAPrw5Wzs2Q9fITG9+9yJlgtikouanaYxnUiw0QXogiF1vETB8oMvKMsgegMA83+OSlFVS425E5WEJ6R9YpMVSC9Dy8fB5gRdTL14K0PM= Received: from AM0PR10CA0011.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:17c::21) by DBAPR06MB7013.eurprd06.prod.outlook.com (2603:10a6:10:1b1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.31; Mon, 15 Apr 2024 06:02:07 +0000 Received: from AM4PEPF00025F99.EURPRD83.prod.outlook.com (2603:10a6:208:17c:cafe::7a) by AM0PR10CA0011.outlook.office365.com (2603:10a6:208:17c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.28 via Frontend Transport; Mon, 15 Apr 2024 06:02:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 193.8.40.94) smtp.mailfrom=leica-geosystems.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=leica-geosystems.com; Received-SPF: Pass (protection.outlook.com: domain of leica-geosystems.com designates 193.8.40.94 as permitted sender) receiver=protection.outlook.com; client-ip=193.8.40.94; helo=hexagon.com; pr=C Received: from hexagon.com (193.8.40.94) by AM4PEPF00025F99.mail.protection.outlook.com (10.167.16.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.0 via Frontend Transport; Mon, 15 Apr 2024 06:02:07 +0000 Received: from aherlnxbspsrv01.lgs-net.com ([10.60.34.116]) by hexagon.com with Microsoft SMTPSVC(10.0.17763.1697); Mon, 15 Apr 2024 08:02:05 +0200 From: Johannes Schneider To: openembedded-core@lists.openembedded.org, richard.purdie@linuxfoundation.org, alex.kanavin@gmail.com, alexandre.belloni@bootlin.com CC: Johannes Schneider Subject: [PATCH v6 1/3] image.bbclass/rootfs: archive and deploy package database Date: Mon, 15 Apr 2024 08:02:01 +0200 Message-ID: <20240415060203.2961-2-johannes.schneider@leica-geosystems.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240415060203.2961-1-johannes.schneider@leica-geosystems.com> References: <20240415060203.2961-1-johannes.schneider@leica-geosystems.com> MIME-Version: 1.0 X-OriginalArrivalTime: 15 Apr 2024 06:02:05.0886 (UTC) FILETIME=[718C9DE0:01DA8EFA] X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM4PEPF00025F99:EE_|DBAPR06MB7013:EE_ X-MS-Office365-Filtering-Correlation-Id: 07fb3441-7f39-4e72-82bd-08dc5d1194d2 X-SET-LOWER-SCL-SCANNER: YES X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: N3fD1+G9SGqw16rq4K5DY4w/xlH6+hjT89OG5NuwCkYMQrnamgGq7XrkzOyxatJl6qhFDnSuKe6u1iAzwnkJN2V9sEaMu81mj/OrWATnvpElMF4vUB67FMuntaTp9H8WRaL3GC4t7y1ebqZJ8YIWexuNdG8J8uPSIRyiG8I5NVi+Dlhw3jBGA+x5PjPKAqiWUHQqz7bkTAmq8bDdVfr0ic7ckazaUXw4SaI5hs6QbjqJL1KLsZlInJ4ikCisgBAuetMRUYbgUtY44YvocOk6tHGdqzufIH/o3ggcWkZkuyXABGm5C/UgW15Ut2AEbqztARjdJ2ho+Twvts9M9+VJAj0nadbprfEiYy7BQ2L7zidCRUbmLXQbPkRDi7sdwl1QkV8CQOnYmS2sc+tiJu0JZqgZNd6T4pKkvDXFIr3QOZhcjnWDB271A7Iwp9YSfID86/kWwjXgIcolE1WeomfrAOMK0ks4nnEdFu1kr1pEHD8E2wNcTUeo71GQ/la8lbb2Gbh4qMQutxv8ijflXwzKS+7nI9HNR4tdrp9oWIalz5dH3hzgb1C5opvGtYNbzFwd73Mu3ZEd8Cg1x/QZ6vc2oh1wsSPn+oX69LNR9R/bCTBK3pQtm234ksK1emI3tn04WO5+Q42OGOvRzb8obhbPwi8sLZ+K5+gZ/C8tB+6NSzAkaxEWFDggQA2Y71Sh7NmzGJuVQO6Os/JZv+8BpHsh0Q== X-Forefront-Antispam-Report: CIP:193.8.40.94;CTRY:CH;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:hexagon.com;PTR:ahersrvdom50.leica-geosystems.com;CAT:NONE;SFS:(13230031)(36860700004)(82310400014)(376005)(1800799015);DIR:OUT;SFP:1101; X-OriginatorOrg: leica-geosystems.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2024 06:02:07.2106 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 07fb3441-7f39-4e72-82bd-08dc5d1194d2 X-MS-Exchange-CrossTenant-Id: 1b16ab3e-b8f6-4fe3-9f3e-2db7fe549f6a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=1b16ab3e-b8f6-4fe3-9f3e-2db7fe549f6a;Ip=[193.8.40.94];Helo=[hexagon.com] X-MS-Exchange-CrossTenant-AuthSource: AM4PEPF00025F99.EURPRD83.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBAPR06MB7013 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 15 Apr 2024 06:02:15 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/198218 archive the package database after the rootfs has been put together as *rootfs-pkdbfs.tar.gz, and put it into the deploy folder. This creates a snapshot of the package mangers state at the point in time when all dependencies have been resolved and installed; which could be used by "extension images" to built upon. Signed-off-by: Johannes Schneider --- meta/classes-recipe/image.bbclass | 44 +++++++++++++- meta/classes-recipe/image_types.bbclass | 1 + meta/conf/documentation.conf | 1 + meta/lib/oe/package_manager/deb/rootfs.py | 1 + meta/lib/oe/package_manager/ipk/rootfs.py | 1 + meta/lib/oe/package_manager/rpm/rootfs.py | 1 + meta/lib/oe/rootfs.py | 20 +++++++ meta/lib/oeqa/selftest/cases/imagefeatures.py | 60 +++++++++++++++++++ 8 files changed, 128 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass index 28be6c6362..3ccaaa17b8 100644 --- a/meta/classes-recipe/image.bbclass +++ b/meta/classes-recipe/image.bbclass @@ -42,6 +42,9 @@ IMAGE_FEATURES ?= "" IMAGE_FEATURES[type] = "list" IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login serial-autologin-root post-install-logging overlayfs-etc" +# Generate snapshot of the package database? +IMAGE_GEN_PKGDBFS ?= "0" + # Generate companion debugfs? IMAGE_GEN_DEBUGFS ?= "0" @@ -131,7 +134,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_INSTALL_DEBUGFS'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -337,6 +341,17 @@ python do_image_qa_setscene () { } addtask do_image_qa_setscene +def setup_pkgdbfs_variables(d): + d.appendVar('IMAGE_ROOTFS', '-pkgdb') + if d.getVar('IMAGE_LINK_NAME'): + d.appendVar('IMAGE_LINK_NAME', '-pkgdb') + d.appendVar('IMAGE_NAME','-pkgdb') + d.setVar('IMAGE_FSTYPES', 'tar.gz') + +python setup_pkgdbfs () { + setup_pkgdbfs_variables(d) +} + def setup_debugfs_variables(d): d.appendVar('IMAGE_ROOTFS', '-dbg') if d.getVar('IMAGE_LINK_NAME'): @@ -381,6 +396,11 @@ python () { alltypes = d.getVar('IMAGE_FSTYPES').split() typedeps = {} + if d.getVar('IMAGE_GEN_PKGDBFS') == "1": + pkgdbfs_fstypes = ['tar.gz'] + for t in pkgdbfs_fstypes: + alltypes.append("pkgdbfs_" + t) + if d.getVar('IMAGE_GEN_DEBUGFS') == "1": debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split() for t in debugfs_fstypes: @@ -393,6 +413,10 @@ python () { basetypes[baset]= [] if t not in basetypes[baset]: basetypes[baset].append(t) + pkgdb = "" + if t.startswith("pkgdbfs_"): + t = t[8:] + pkgdb = "pkgdbfs_" debug = "" if t.startswith("debugfs_"): t = t[8:] @@ -401,6 +425,13 @@ python () { vardeps.add('IMAGE_TYPEDEP:' + t) if baset not in typedeps: typedeps[baset] = set() + deps = [pkgdb + dep for dep in deps] + for dep in deps: + if dep not in alltypes: + alltypes.append(dep) + _add_type(dep) + basedep = _image_base_type(dep) + typedeps[baset].add(basedep) deps = [debug + dep for dep in deps] for dep in deps: if dep not in alltypes: @@ -419,6 +450,7 @@ python () { maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split() maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")] + maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("", "pkgdbfs_")] for t in basetypes: vardeps = set() @@ -430,6 +462,11 @@ python () { continue localdata = bb.data.createCopy(d) + pkgdb = "" + if t.startswith("pkgdbfs_"): + setup_pkgdbfs_variables(localdata) + pkgdb = "setup_pkgdbfs " + realt = t[8:] debug = "" if t.startswith("debugfs_"): setup_debugfs_variables(localdata) @@ -468,6 +505,8 @@ python () { for ctype in sorted(ctypes): if bt.endswith("." + ctype): type = bt[0:-len(ctype) - 1] + if type.startswith("pkgdbfs_"): + type = type[8:] if type.startswith("debugfs_"): type = type[8:] # Create input image first. @@ -509,6 +548,9 @@ python () { d.setVarFlag(task, 'fakeroot', '1') d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + if pkgdb: + d.appendVarFlag(task, 'prefuncs', ' ' + pkgdb + ' set_image_size') + d.prependVarFlag(task, 'postfuncs', 'create_symlinks ') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass index 913cb8788c..dbb39f9bd0 100644 --- a/meta/classes-recipe/image_types.bbclass +++ b/meta/classes-recipe/image_types.bbclass @@ -25,6 +25,7 @@ def imagetypes_getdepends(d): fstypes = set((d.getVar('IMAGE_FSTYPES') or "").split()) fstypes |= set((d.getVar('IMAGE_FSTYPES_DEBUGFS') or "").split()) + fstypes |= set('tar.gz') # hardcoded for 'pkgdbfs' deprecated = set() deps = set() diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf index b0591881ba..36aebb59ab 100644 --- a/meta/conf/documentation.conf +++ b/meta/conf/documentation.conf @@ -215,6 +215,7 @@ IMAGE_FEATURES[doc] = "The primary list of features to include in an image. Conf IMAGE_FSTYPES[doc] = "Formats of root filesystem images that you want to have created." IMAGE_FSTYPES_DEBUGFS[doc] = "Formats of the debug root filesystem images that you want to have created." IMAGE_GEN_DEBUGFS[doc] = "When set to '1', generate a companion debug object/source filesystem image." +IMAGE_GEN_PKGDBFS[doc] = "When set to '1', create a snapshot of the package-manager state after the rootfs has been assembled." IMAGE_INSTALL[doc] = "Specifies the packages to install into an image. Image recipes set IMAGE_INSTALL to specify the packages to install into an image through image.bbclass." IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image during the root filesystem construction process." IMAGE_NAME[doc] = "The name of the output image files minus the extension." diff --git a/meta/lib/oe/package_manager/deb/rootfs.py b/meta/lib/oe/package_manager/deb/rootfs.py index 1e25b64ed9..43107c8663 100644 --- a/meta/lib/oe/package_manager/deb/rootfs.py +++ b/meta/lib/oe/package_manager/deb/rootfs.py @@ -178,6 +178,7 @@ class PkgRootfs(DpkgOpkgRootfs): if self.progress_reporter: self.progress_reporter.next_stage() + self._setup_pkg_db_rootfs(['/var/lib/dpkg']) self._setup_dbg_rootfs(['/var/lib/dpkg']) self.pm.fix_broken_dependencies() diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py b/meta/lib/oe/package_manager/ipk/rootfs.py index ba93eb62ea..64d9bc7969 100644 --- a/meta/lib/oe/package_manager/ipk/rootfs.py +++ b/meta/lib/oe/package_manager/ipk/rootfs.py @@ -319,6 +319,7 @@ class PkgRootfs(DpkgOpkgRootfs): opkg_lib_dir = self.d.getVar('OPKGLIBDIR') opkg_dir = os.path.join(opkg_lib_dir, 'opkg') + self._setup_pkg_db_rootfs([opkg_dir]) self._setup_dbg_rootfs([opkg_dir]) execute_pre_post_process(self.d, opkg_post_process_cmds) diff --git a/meta/lib/oe/package_manager/rpm/rootfs.py b/meta/lib/oe/package_manager/rpm/rootfs.py index 3ba5396320..673006c131 100644 --- a/meta/lib/oe/package_manager/rpm/rootfs.py +++ b/meta/lib/oe/package_manager/rpm/rootfs.py @@ -110,6 +110,7 @@ class PkgRootfs(Rootfs): if self.progress_reporter: self.progress_reporter.next_stage() + self._setup_pkg_db_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf']) self._setup_dbg_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf']) execute_pre_post_process(self.d, rpm_post_process_cmds) diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index 8cd48f9450..6d6e888a30 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -106,6 +106,26 @@ class Rootfs(object, metaclass=ABCMeta): def _cleanup(self): pass + def _setup_pkg_db_rootfs(self, package_paths): + gen_pkg_db_fs = bb.utils.to_boolean(self.d.getVar('IMAGE_GEN_PKGDBFS')) + if not gen_pkg_db_fs: + return + + bb.note(" Creating pkg-db rootfs...") + try: + shutil.rmtree(self.image_rootfs + '-pkgdb') + except (FileNotFoundError, NotADirectoryError): + pass + bb.utils.mkdirhier(self.image_rootfs + "-pkgdb") + + bb.note(" Copying back package database...") + for path in package_paths: + bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path)) + if os.path.isdir(self.image_rootfs + path): + shutil.copytree(self.image_rootfs + path, self.image_rootfs + '-pkgdb' + path, symlinks=True) + elif os.path.isfile(self.image_rootfs + path): + shutil.copyfile(self.image_rootfs + path, self.image_rootfs + '-pkgdb' + path) + def _setup_dbg_rootfs(self, package_paths): gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0' if gen_debugfs != '1': diff --git a/meta/lib/oeqa/selftest/cases/imagefeatures.py b/meta/lib/oeqa/selftest/cases/imagefeatures.py index dc88c222bd..72cf177b6f 100644 --- a/meta/lib/oeqa/selftest/cases/imagefeatures.py +++ b/meta/lib/oeqa/selftest/cases/imagefeatures.py @@ -9,6 +9,7 @@ from oeqa.core.decorator import OETestTag from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu from oeqa.utils.sshcontrol import SSHControl import glob +import oe.path import os import json @@ -17,6 +18,20 @@ class ImageFeatures(OESelftestTestCase): test_user = 'tester' root_user = 'root' + @classmethod + def setUpClass(cls): + super().setUpClass() + + # Build the tools we need and populate a sysroot + bitbake("dpkg-native opkg-native rpm-native python3-native") + bitbake("build-sysroots -c build_native_sysroot") + + # Get the paths so we can point into the sysroot correctly + vars = get_bb_vars(["STAGING_DIR", "BUILD_ARCH", "bindir_native", "libdir_native"]) + cls.staging = oe.path.join(vars["STAGING_DIR"], vars["BUILD_ARCH"]) + cls.bindir = oe.path.join(cls.staging, vars["bindir_native"]) + cls.libdir = oe.path.join(cls.staging, vars["libdir_native"]) + @OETestTag("runqemu") def test_non_root_user_can_connect_via_ssh_without_password(self): """ @@ -302,6 +317,51 @@ SKIP_RECIPE[busybox] = "Don't build this" result = runCmd('objdump --syms %s | grep debug' % t) self.assertTrue("debug" in result.output, msg='Failed to find debug symbol: %s' % result.output) + def test_image_gen_pkgdbfs(self): + """ + Summary: Check pkgdb snapshot + Expected: 1. core-image-minimal can be build with IMAGE_GEN_PKGDBFS variable set + 2. *rootfs-pkgdb.tar.gz snapshot of the package manager state is created, + after the rootfs of the core-image-minimal is put together + 3. check that the restored package index is usable by the default packagemanager + PACKAGE_CLASS=package_ipk -> opkg in this case. + """ + + image = 'core-image-minimal' + features = 'IMAGE_GEN_PKGDBFS = "1"\n' + self.write_config(features) + + bitbake(image) + + img_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME', 'MACHINE', 'DEFAULTTUNE'], image) + + tar_file = os.path.join(img_vars['DEPLOY_DIR_IMAGE'], "%s-pkgdb.tar.gz" % (img_vars['IMAGE_LINK_NAME'])) + self.assertTrue(os.path.exists(tar_file), 'pkgdb filesystem not generated at %s' % tar_file) + result = runCmd('cd %s; tar xvf %s' % (img_vars['DEPLOY_DIR_IMAGE'], tar_file)) + self.assertEqual(result.status, 0, msg='Failed to extract %s: %s' % (tar_file, result.output)) + self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/opkg/status')), 'opkg\'s status file was not present in: %s' % tar_file) + + # some 'add-arch' have '-' replaced by '_', some not; adding both does not hurt + cmd = os.path.join(self.bindir, 'opkg') + \ + ' --volatile-cache --offline-root=%s --add-arch %s:11 --add-arch %s:12 list-installed' \ + % (img_vars['DEPLOY_DIR_IMAGE'], img_vars["MACHINE"], img_vars["MACHINE"].replace('-','_')) + self.logger.debug('running cmd: %s' % cmd) + result = runCmd(cmd) + self.logger.debug("list-installed:\n%s" % result.output) + self.assertEqual(result.status, 0, msg='Failed to run package manager on unpacked pkgdb %s: %s' % (tar_file, result.output)) + # check for a 'random, very architecture specific' package + self.assertTrue("kernel-image" in result.output, msg='Failed query installed packages') + + cmd = os.path.join(self.bindir, 'opkg') + \ + ' --volatile-cache --offline-root=%s --add-arch %s:11 --add-arch %s:12 list-installed' \ + % (img_vars['DEPLOY_DIR_IMAGE'], img_vars["DEFAULTTUNE"], img_vars["DEFAULTTUNE"].replace('-','_')) + self.logger.debug('running cmd: %s' % cmd) + result = runCmd(cmd) + self.logger.debug("list-installed:\n%s" % result.output) + self.assertEqual(result.status, 0, msg='Failed to run package manager on unpacked pkgdb %s: %s' % (tar_file, result.output)) + # check for a common base package, a random library for example: + self.assertTrue("libc6" in result.output, msg='Failed query installed packages') + def test_empty_image(self): """Test creation of image with no packages""" image = 'test-empty-image' From patchwork Mon Apr 15 06:02:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SCHNEIDER Johannes X-Patchwork-Id: 42338 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3D13C4345F for ; Mon, 15 Apr 2024 06:02:15 +0000 (UTC) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (EUR04-VI1-obe.outbound.protection.outlook.com [40.107.8.81]) by mx.groups.io with SMTP id smtpd.web11.14608.1713160932352722644 for ; Sun, 14 Apr 2024 23:02:12 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@leica-geosystems.com header.s=selector1 header.b=Rfh1UI89; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: leica-geosystems.com, ip: 40.107.8.81, mailfrom: johannes.schneider@leica-geosystems.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ToQNxliujXR5UHImzWelauR1pG8xgJro5Z7dW/d6QckReiI4nkP8NTJcHjc1NgVKNULNdy5wNb+PVDrhWXYAVw6Yd/LPQlJrjn7jgUgQpDbgKkcxUuL7kcvxrudHffn+0YoVYTTSzLXgsG4p1HN5ewIFo6D5YXu5NP8Ar4LQjwp8HKhkbhB6WILLY58ltB6iSAP8GKfPpwU4Me6X9myD2KWFFEV4DXm/2TFkdikaDZHERCFaF+WwXYdzoRP2asG7Y0Azjfw6vZQfO+vpNRxFwbCJykaxg1P1+JHIGCnkxWVBh9doZmhiH1ctjGoFfqJ84TjKwhvy5MSBEUsq1qys1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KDDLc+6vzEy9KlJsCfmMkQGGnm/AypHebQ4QhG116pQ=; b=LPBrpCdROduuTz8nyvZdRQdwh/DAA37185lHba2T7TOxDQHXkbKD19u++yYuSzXhKP8qLcWHxHVxrw0FqnOLCnmWPrVdUZcWfrRPWhERWWjc5bzjZCffq9vdosgU6kfEEqot3M/j8R4tSrC+xw8036KBw+uQOqbW0TbForuo2VD+UqekKWUAkRpeiXgBj4XJWAYZhLDtuEA2yU1CAWZRtTQCX1z8tDhK7Ipsz9smfRVk7iP0jmSlEdbFWXQkMp7QNlqmTYOZB6iODUXlc9wL7TcJN7TvAQvVBfW6RU7S2QY0dxl1A3t6VOu6MM3S/8LRg6rnduOXR4TmzA7lQApuDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 193.8.40.94) smtp.rcpttodomain=lists.openembedded.org smtp.mailfrom=leica-geosystems.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=leica-geosystems.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leica-geosystems.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KDDLc+6vzEy9KlJsCfmMkQGGnm/AypHebQ4QhG116pQ=; b=Rfh1UI89S/k52Zpt0sMH+Yb550tGJXRYtJbZc4/q8rdgn9f0D6fgpWuDVQWyUzsijjICj9jCZctVNinIJhDn4KsrYpEncmCOVw2JJGIRpSdCHvb8Cw/OIRn6/V3KnnPS8VRMmGNajr+11fa3O9reE8ifRzycx7bOfbdwIdhZHsY= Received: from AM0PR10CA0019.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:17c::29) by PA4PR06MB8427.eurprd06.prod.outlook.com (2603:10a6:102:2a2::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.9; Mon, 15 Apr 2024 06:02:07 +0000 Received: from AM4PEPF00025F99.EURPRD83.prod.outlook.com (2603:10a6:208:17c:cafe::f1) by AM0PR10CA0019.outlook.office365.com (2603:10a6:208:17c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.30 via Frontend Transport; Mon, 15 Apr 2024 06:02:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 193.8.40.94) smtp.mailfrom=leica-geosystems.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=leica-geosystems.com; Received-SPF: Pass (protection.outlook.com: domain of leica-geosystems.com designates 193.8.40.94 as permitted sender) receiver=protection.outlook.com; client-ip=193.8.40.94; helo=hexagon.com; pr=C Received: from hexagon.com (193.8.40.94) by AM4PEPF00025F99.mail.protection.outlook.com (10.167.16.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.0 via Frontend Transport; Mon, 15 Apr 2024 06:02:07 +0000 Received: from aherlnxbspsrv01.lgs-net.com ([10.60.34.116]) by hexagon.com with Microsoft SMTPSVC(10.0.17763.1697); Mon, 15 Apr 2024 08:02:05 +0200 From: Johannes Schneider To: openembedded-core@lists.openembedded.org, richard.purdie@linuxfoundation.org, alex.kanavin@gmail.com, alexandre.belloni@bootlin.com CC: Johannes Schneider Subject: [PATCH v6 2/3] image.bbclass/rootfs: set and unpack package-database Date: Mon, 15 Apr 2024 08:02:02 +0200 Message-ID: <20240415060203.2961-3-johannes.schneider@leica-geosystems.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240415060203.2961-1-johannes.schneider@leica-geosystems.com> References: <20240415060203.2961-1-johannes.schneider@leica-geosystems.com> MIME-Version: 1.0 X-OriginalArrivalTime: 15 Apr 2024 06:02:05.0933 (UTC) FILETIME=[7193C9D0:01DA8EFA] X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM4PEPF00025F99:EE_|PA4PR06MB8427:EE_ X-MS-Office365-Filtering-Correlation-Id: 2138fe89-6ab7-4937-cfd2-08dc5d1194ff X-SET-LOWER-SCL-SCANNER: YES X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eh2SPNZvonjG4GBdfXws0TvSNdevHU4/RIsy/GA8ErXgt7CNGQg1KjkdbkUbOAnLUIHBQ1H1eqwxB/2ya9EkzebTfjOW071syc+IfT0RM+BosrmOwPDGBwA3t066mNZ9c08ZfdN9pmIWn/lXHsVv9e06dV2QzSPxhwVr/WA0+jvq3FuVdUukYvWak+niBfsfhx1m1OyM5qQA/pM68u1eE5/W6E4MWjAzXj2D7i8lo2XRpTRYDWcD1y33mF0dRAAwAApkMXA8ro5ftN45VHVBrJ0qCMygIUGOEnKe/dSnAFBy4baK1IisVIOHU6fWGiqk5WTN2yPNHnTqAI43ckKLnJvLfzsiBnZOBAwd4oJtRVgiV9oI5iS0wlKxrRvrtJa/WLtAJg0M9GV4Lphl6MF0mBtn5CphD6ko1aEwme8vk7GCK094xpo0XN1iMBlSYWzjTSQpMI3LDbx3kfvJm/gyTkmQuILf+EsMLEhKVCnljjR6NZ5ScAhCfpXdcrSmfqsH6zGoJG0k232b2LUtZLyxEOTWRZ22Wb096vEkT4Jm1tFvLckgfAZ9QeL8rL/b/qL2vW+qVBtTlxSHwXdpEOTVyR6dxSCtP2uTRkWBuNF/boY+4yc9swwJ5Lmf7UkqCpmu8evoBwliIZedC4XZG8G2c4MIJZODrOMBDmZvpcluP8StzViTNh3UbA758D1lCdbnCSV/NvZ98zfWfw4ts2pTMw== X-Forefront-Antispam-Report: CIP:193.8.40.94;CTRY:CH;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:hexagon.com;PTR:ahersrvdom50.leica-geosystems.com;CAT:NONE;SFS:(13230031)(82310400014)(1800799015)(36860700004)(376005);DIR:OUT;SFP:1101; X-OriginatorOrg: leica-geosystems.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2024 06:02:07.4919 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2138fe89-6ab7-4937-cfd2-08dc5d1194ff X-MS-Exchange-CrossTenant-Id: 1b16ab3e-b8f6-4fe3-9f3e-2db7fe549f6a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=1b16ab3e-b8f6-4fe3-9f3e-2db7fe549f6a;Ip=[193.8.40.94];Helo=[hexagon.com] X-MS-Exchange-CrossTenant-AuthSource: AM4PEPF00025F99.EURPRD83.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR06MB8427 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 15 Apr 2024 06:02:15 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/198220 set the package-database of a "lower image" to unpack and build upon when installing packages for the current image. This way a lean image will be created, which only holds the packages that are not already present in the lower image. An image build such could then be used with overlayfs or systemd- sysext to extend the "lower image" on demand; for development purposes on a device running the "lower image" in RO mode for example. A configuration could look as follows: some-core-image.bb inherit image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image IMAGE_BASE_PKGDB = "some-core-image" Signed-off-by: Johannes Schneider --- meta/classes-recipe/image.bbclass | 23 ++++++++-- meta/conf/documentation.conf | 3 +- meta/lib/oe/package_manager/deb/rootfs.py | 2 + meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++- meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++- meta/lib/oe/rootfs.py | 18 ++++++++ meta/lib/oeqa/selftest/cases/imagefeatures.py | 46 +++++++++++++++++++ 7 files changed, 97 insertions(+), 8 deletions(-) diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass index 3ccaaa17b8..c573c37cd8 100644 --- a/meta/classes-recipe/image.bbclass +++ b/meta/classes-recipe/image.bbclass @@ -42,8 +42,16 @@ IMAGE_FEATURES ?= "" IMAGE_FEATURES[type] = "list" IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login serial-autologin-root post-install-logging overlayfs-etc" -# Generate snapshot of the package database? +# Image layering: +# a "base image" would create a snapshot of the package-database after the +# installation of all packages into the rootfs is done. The next/other image +# "layered on-top" of the former would then import that database and install +# further packages; without reinstalling packages/dependencies that are already +# installed in the layer below. +# Set to '1' in a "base image" recipe, to preserve a snapshot of the package database. IMAGE_GEN_PKGDBFS ?= "0" +# "PN" of a "base image", upon which the current image is to be built upon. +IMAGE_BASE_PKGDB ?= "" # Generate companion debugfs? IMAGE_GEN_DEBUGFS ?= "0" @@ -118,6 +126,15 @@ do_rootfs[depends] += " \ " do_rootfs[recrdeptask] += "do_packagedata" +python () { + # make sure that the 'base image' has been queued in before this + # image wants to unpack and build upon the formers pgkdb + base_image = d.getVar('IMAGE_BASE_PKGDB') + pn = d.getVar('PN') + if base_image and base_image != pn: + d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image + ':do_image_complete') +} + def rootfs_command_variables(d): return ['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND', 'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS'] @@ -134,8 +151,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', - 'IMAGE_INSTALL_DEBUGFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', + 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf index 36aebb59ab..9f493cfe96 100644 --- a/meta/conf/documentation.conf +++ b/meta/conf/documentation.conf @@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary." ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the Icecream distributed compile support to consider." ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the Icecream distributed compile support to consider." ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use an empty PARALLEL_MAKE variable that you want to force remote distributed compilation on using the Icecream distributed compile support." +IMAGE_BASE_PKGDB[doc] = "Set to the PN of a base-image name, which was built with IMAGE_GEN_PKGDBFS enabled; Then this image will only add/install packages into its rootfs that are not already in the base-image." IMAGE_BASENAME[doc] = "The base name of image output files." IMAGE_BOOT_FILES[doc] = "Whitespace separated list of files from ${DEPLOY_DIR_IMAGE} to place in boot partition. Entries will be installed under a same name as the source file. To change the destination file name, pass a desired name after a semicolon (eg. u-boot.img;uboot)." IMAGE_CLASSES[doc] = "A list of classes that all images should inherit." @@ -215,7 +216,7 @@ IMAGE_FEATURES[doc] = "The primary list of features to include in an image. Conf IMAGE_FSTYPES[doc] = "Formats of root filesystem images that you want to have created." IMAGE_FSTYPES_DEBUGFS[doc] = "Formats of the debug root filesystem images that you want to have created." IMAGE_GEN_DEBUGFS[doc] = "When set to '1', generate a companion debug object/source filesystem image." -IMAGE_GEN_PKGDBFS[doc] = "When set to '1', create a snapshot of the package-manager state after the rootfs has been assembled." +IMAGE_GEN_PKGDBFS[doc] = "When set to '1', create a snapshot of the package-manager state after the rootfs has been assembled; This snapshot can then be used in a different image recipe, that sets IMAGE_BASE_PKGDB to this images PN." IMAGE_INSTALL[doc] = "Specifies the packages to install into an image. Image recipes set IMAGE_INSTALL to specify the packages to install into an image through image.bbclass." IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image during the root filesystem construction process." IMAGE_NAME[doc] = "The name of the output image files minus the extension." diff --git a/meta/lib/oe/package_manager/deb/rootfs.py b/meta/lib/oe/package_manager/deb/rootfs.py index 43107c8663..71a21df09b 100644 --- a/meta/lib/oe/package_manager/deb/rootfs.py +++ b/meta/lib/oe/package_manager/deb/rootfs.py @@ -152,6 +152,8 @@ class PkgRootfs(DpkgOpkgRootfs): execute_pre_post_process(self.d, deb_pre_process_cmds) + self._unpack_pkg_db_rootfs(['/var/lib/dpkg']) + if self.progress_reporter: self.progress_reporter.next_stage() # Don't support incremental, so skip that diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py b/meta/lib/oe/package_manager/ipk/rootfs.py index 64d9bc7969..408faa8030 100644 --- a/meta/lib/oe/package_manager/ipk/rootfs.py +++ b/meta/lib/oe/package_manager/ipk/rootfs.py @@ -276,12 +276,16 @@ class PkgRootfs(DpkgOpkgRootfs): pkgs_to_install = self.manifest.parse_initial_manifest() opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS') opkg_post_process_cmds = self.d.getVar('OPKG_POSTPROCESS_COMMANDS') + opkg_lib_dir = self.d.getVar('OPKGLIBDIR') + opkg_dir = os.path.join(opkg_lib_dir, 'opkg') # update PM index files self.pm.write_index() execute_pre_post_process(self.d, opkg_pre_process_cmds) + self._unpack_pkg_db_rootfs([opkg_dir]) + if self.progress_reporter: self.progress_reporter.next_stage() # Steps are a bit different in order, skip next @@ -317,8 +321,6 @@ class PkgRootfs(DpkgOpkgRootfs): if self.progress_reporter: self.progress_reporter.next_stage() - opkg_lib_dir = self.d.getVar('OPKGLIBDIR') - opkg_dir = os.path.join(opkg_lib_dir, 'opkg') self._setup_pkg_db_rootfs([opkg_dir]) self._setup_dbg_rootfs([opkg_dir]) diff --git a/meta/lib/oe/package_manager/rpm/rootfs.py b/meta/lib/oe/package_manager/rpm/rootfs.py index 673006c131..9986b13b5f 100644 --- a/meta/lib/oe/package_manager/rpm/rootfs.py +++ b/meta/lib/oe/package_manager/rpm/rootfs.py @@ -67,12 +67,15 @@ class PkgRootfs(Rootfs): pkgs_to_install = self.manifest.parse_initial_manifest() rpm_pre_process_cmds = self.d.getVar('RPM_PREPROCESS_COMMANDS') rpm_post_process_cmds = self.d.getVar('RPM_POSTPROCESS_COMMANDS') + package_paths = ['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'] # update PM index files self.pm.write_index() execute_pre_post_process(self.d, rpm_pre_process_cmds) + self._unpack_pkg_db_rootfs(package_paths) + if self.progress_reporter: self.progress_reporter.next_stage() @@ -110,8 +113,8 @@ class PkgRootfs(Rootfs): if self.progress_reporter: self.progress_reporter.next_stage() - self._setup_pkg_db_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf']) - self._setup_dbg_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf']) + self._setup_pkg_db_rootfs(package_paths) + self._setup_dbg_rootfs(package_paths) execute_pre_post_process(self.d, rpm_post_process_cmds) diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index 6d6e888a30..9d691c58cd 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -106,6 +106,24 @@ class Rootfs(object, metaclass=ABCMeta): def _cleanup(self): pass + def _unpack_pkg_db_rootfs(self, package_paths): + import tarfile + pn = self.d.getVar('PN') + base_pkgdb = self.d.getVar('IMAGE_BASE_PKGDB') + if not base_pkgdb or pn == base_pkgdb: + return + + fname = self.d.getVar('DEPLOY_DIR_IMAGE') + '/' + self.d.getVar('IMAGE_BASE_PKGDB') + '-' + self.d.getVar('MACHINE') + '.rootfs-pkgdb.tar.gz' + if not os.path.exists(fname): + bb.fatal("PKGDB for '%s' does not exist,\n\texpected: %s\n\tthe recipe for '%s' should set: IMAGE_GEN_PKGDBFS='1'" % (base_pkgdb, fname, base_pkgdb)) + return + + bb.note(" unpacking base image package database...") + if fname.endswith("tar.gz"): + tar = tarfile.open(fname, "r:gz") + tar.extractall(path=self.image_rootfs) + tar.close() + def _setup_pkg_db_rootfs(self, package_paths): gen_pkg_db_fs = bb.utils.to_boolean(self.d.getVar('IMAGE_GEN_PKGDBFS')) if not gen_pkg_db_fs: diff --git a/meta/lib/oeqa/selftest/cases/imagefeatures.py b/meta/lib/oeqa/selftest/cases/imagefeatures.py index 72cf177b6f..2e86e76735 100644 --- a/meta/lib/oeqa/selftest/cases/imagefeatures.py +++ b/meta/lib/oeqa/selftest/cases/imagefeatures.py @@ -362,6 +362,52 @@ SKIP_RECIPE[busybox] = "Don't build this" # check for a common base package, a random library for example: self.assertTrue("libc6" in result.output, msg='Failed query installed packages') + def test_image_use_pkgdbfs(self): + """ + Summary: Check pkgdb snapshot + Expected: 1. core-image-minimal is built with IMAGE_GEN_PKGDBFS variable set + 2. core-image-minimal-mtdutils is built with: + IMAGE_BASE_PKGDB pointing to the former, to build upon it + 3. the rootfs of both images is checked, and there should be no file + from any of the installed packages that shows up in both images + """ + + base_image = 'core-image-minimal' + extension_image = 'core-image-minimal-mtdutils' + + features = 'IMAGE_FSTYPES = "tar.bz2"\n' + # enable usrmerge so that all relevant pieces are in one place = /usr + features += 'DISTRO_FEATURES += "usrmerge"\n' + features += 'IMAGE_GEN_PKGDBFS = "1"\n' + features += 'IMAGE_BASE_PKGDB = "%s"\n' % base_image + self.write_config(features) + + # through task dependencies the base_image should automagically be built + bitbake(extension_image) + + # collect image content of both images + img_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'], base_image) + img_file = os.path.join(img_vars['DEPLOY_DIR_IMAGE'], "%s.%s" % (img_vars['IMAGE_LINK_NAME'], 'tar.bz2')) + self.logger.debug("base image: %s" % img_file) + img_content = runCmd('tar -tf %s' % img_file) + + ext_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'], extension_image) + ext_file = os.path.join(ext_vars['DEPLOY_DIR_IMAGE'], "%s.%s" % (ext_vars['IMAGE_LINK_NAME'], 'tar.bz2')) + self.logger.debug("extension image: %s" % ext_file) + ext_content = runCmd('tar -tf %s' % ext_file) + + # filter out folders and anything outside of /usr + img_files = [ x for x in img_content.output.splitlines() if not x.endswith('/') and x.startswith('./usr/') ] + self.logger.debug("image files:\n%s" % img_files) + ext_files = [ x for x in ext_content.output.splitlines() if not x.endswith('/') and x.startswith('./usr/') ] + + # check mutual exclusive content listing + self.logger.debug("extension files:\n%s" % ext_files) + for i in img_files: + self.assertTrue(i not in ext_files, "found %s in extension" % i) + for e in ext_files: + self.assertTrue(e not in img_files, "found %s in base-image" % e) + def test_empty_image(self): """Test creation of image with no packages""" image = 'test-empty-image' From patchwork Mon Apr 15 06:02:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SCHNEIDER Johannes X-Patchwork-Id: 42337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4F49C00A94 for ; Mon, 15 Apr 2024 06:02:15 +0000 (UTC) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (EUR01-DB5-obe.outbound.protection.outlook.com [40.107.15.85]) by mx.groups.io with SMTP id smtpd.web11.14607.1713160930691139183 for ; Sun, 14 Apr 2024 23:02:11 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@leica-geosystems.com header.s=selector1 header.b=nzswLFwl; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: leica-geosystems.com, ip: 40.107.15.85, mailfrom: johannes.schneider@leica-geosystems.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kjnpvYpzJsH4+8wtzmeoM9LldZs8N7/8SEuflVxlhFPYgyHPDOAOtemxkdDwBsV4fBwYCWbjSyCnKmQ1eD8IdcG8uS9pjd7HpLBZbIawXt4VVXhE5QpJ4pTV+F77ZFsyeJOdaYpzM9Ay9WhPS/YQZA12RS2uVvjfHKA5hwWpDyo3JHkmRSCdWrRHUNr1Zt6tLcWKGHSRuQ5lJMb8N+de262++H+XgQOTasII82Oe1rdHXVJiBdp3WUNOURnG2FNekbHNXJZ3EsF3kZMJbJcTeqwiNjQmNPQ+kbb2AJbNhb3qIIgrcXZnUA0+geHFafsAeYrDV3LGB6/8yTaiBLIWBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZKtblxxRrANVU8PaI2XIgyTLDwMaOc+7yM+MzGuoNPg=; b=Pxgxyrl6S+iEKIV/F1nj7Y9dvadrWhlK75KjhHcgpdOgmlUr0YBdI9I/9clkTu+Ciwy1QGSYxK7ocX7V4SKXIY8VA4+T77UsTRC/hrtvm2HXLwboScwG5QnrsSSF3yBBv0jGgQ1YrG9U571KRW/tQb4rSQLnzehv0G9dTJBgD5jj3/k4sPhBJrhnTWlt+tzqZ1PVkT1BVHWZtpUzX3YTYUzcm1IY7tgmlxlPTHe33HGEBXEdWiy0h468bhPKV7d6h5gz/O69+DS79gloW8wfcvTIurcASDMx5jONETvDgLv39ZAa+bEsC8/nCpNFDGhifSCu9EVXJln8NtWbOnmS1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 193.8.40.94) smtp.rcpttodomain=lists.openembedded.org smtp.mailfrom=leica-geosystems.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=leica-geosystems.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leica-geosystems.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZKtblxxRrANVU8PaI2XIgyTLDwMaOc+7yM+MzGuoNPg=; b=nzswLFwlv61IKNq2o7I5BzcWqpS42x3Eb/QbDz6ViEJOtEbaW7EMtD56spIlKvlpw+s96aOJ2g0L1uhhNyQODRt+BM35U16z+RTbm/UowR1uzBVkmJQMAlW2ODBM9sIvG/LyoOta7ulJflr4v5kJSxn0qgxFz3Qf5/Lr1gCkHUU= Received: from AM0PR10CA0029.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:17c::39) by AM8PR06MB7025.eurprd06.prod.outlook.com (2603:10a6:20b:1dc::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.26; Mon, 15 Apr 2024 06:02:07 +0000 Received: from AM4PEPF00025F99.EURPRD83.prod.outlook.com (2603:10a6:208:17c:cafe::9b) by AM0PR10CA0029.outlook.office365.com (2603:10a6:208:17c::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.30 via Frontend Transport; Mon, 15 Apr 2024 06:02:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 193.8.40.94) smtp.mailfrom=leica-geosystems.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=leica-geosystems.com; Received-SPF: Pass (protection.outlook.com: domain of leica-geosystems.com designates 193.8.40.94 as permitted sender) receiver=protection.outlook.com; client-ip=193.8.40.94; helo=hexagon.com; pr=C Received: from hexagon.com (193.8.40.94) by AM4PEPF00025F99.mail.protection.outlook.com (10.167.16.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.0 via Frontend Transport; Mon, 15 Apr 2024 06:02:07 +0000 Received: from aherlnxbspsrv01.lgs-net.com ([10.60.34.116]) by hexagon.com with Microsoft SMTPSVC(10.0.17763.1697); Mon, 15 Apr 2024 08:02:06 +0200 From: Johannes Schneider To: openembedded-core@lists.openembedded.org, richard.purdie@linuxfoundation.org, alex.kanavin@gmail.com, alexandre.belloni@bootlin.com CC: Johannes Schneider Subject: [PATCH v6 3/3] classes: add a systemd-sysext image class Date: Mon, 15 Apr 2024 08:02:03 +0200 Message-ID: <20240415060203.2961-4-johannes.schneider@leica-geosystems.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240415060203.2961-1-johannes.schneider@leica-geosystems.com> References: <20240415060203.2961-1-johannes.schneider@leica-geosystems.com> MIME-Version: 1.0 X-OriginalArrivalTime: 15 Apr 2024 06:02:06.0011 (UTC) FILETIME=[719FB0B0:01DA8EFA] X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM4PEPF00025F99:EE_|AM8PR06MB7025:EE_ X-MS-Office365-Filtering-Correlation-Id: 0479f1ba-70e7-443a-a5d9-08dc5d119536 X-SET-LOWER-SCL-SCANNER: YES X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: T48EbrsT4UnSfi1/ostpz60wbWKXsQ29LHKipQURmD5Fx1MoAj4oCiQxIYFcpvs2A5goZM/p0yNyC7AG8cMAeqIh4sYmjlYHX+mz2lNPdn2nmwevPqaaLvfSD1tWK4i/EZh2FSrMzaF2xT3kyKqfGtdo9UKCa3zXo6ziYNk0R+t9+S5/f6E0hUW85NZ+w0RnRWVLQzVdvr6BV6pmlDhgO5zCzKJ7S+d42QNrz3ilZdNzHwPRkCd+uJIvLgrrebD+Y95GBMLALBe2f5SRrbwFwIjpLtyUA7AuySW714FvskgwtP6/29WOc1gJOi7nHEq8niw2t5ca8i9wi0ZtYcl53J3EnzSUia4/pLOpOTRNLYUgfBFDSoibQcFpfJP4U56vgaJbuDiaTkRDaMY0/oDC0BrGQoPW0gDqkSOToZi6O0+JCcpIEeu5UT43ZZKQs3t0NsiUPZHd/HtqCl8o3UbyCrVVDDG8O/dlqkKzNE/AYeRH3thB+8IzBeCRCSNwk4TDG7OERmJW+nmosSGzYf1Qi7ZthHHoOZ8r3WVxYCYUXp2M4JiaLi6hWkNBBi7KRbqpNiN6uAmWqhAUe4lV4pKnd0z9ApFXEdgYFffUarGeQ46VdsN8cp91bdpRmWnhyXFSZ7iLJGwxSV75oC5ii2SgdN5OaRjbsuWLz4iWhNAUoYVrjm3JUmV2xnCw55cJ9Po9fg7xnyTab344RW+LObEhGw== X-Forefront-Antispam-Report: CIP:193.8.40.94;CTRY:CH;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:hexagon.com;PTR:ahersrvdom50.leica-geosystems.com;CAT:NONE;SFS:(13230031)(36860700004)(376005)(1800799015)(82310400014);DIR:OUT;SFP:1101; X-OriginatorOrg: leica-geosystems.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2024 06:02:07.8669 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0479f1ba-70e7-443a-a5d9-08dc5d119536 X-MS-Exchange-CrossTenant-Id: 1b16ab3e-b8f6-4fe3-9f3e-2db7fe549f6a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=1b16ab3e-b8f6-4fe3-9f3e-2db7fe549f6a;Ip=[193.8.40.94];Helo=[hexagon.com] X-MS-Exchange-CrossTenant-AuthSource: AM4PEPF00025F99.EURPRD83.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR06MB7025 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 15 Apr 2024 06:02:15 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/198217 systemd-sysext can load a raw-image containing usr/ and opt/ folders to mount them as RO overlay over the rootfs, to "extend" the systems. This class provides the necessary changes/additions to the enclosed file-system so that systemd-sysext accepts the extension for "merge" into the rootfs. With such an created image, placed into the correct folder (see [1]), `systemd-sysext list` should be able to list the "extension" and `systemd-sysext merge` should enable the overlay. On both commands a preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is amiss. The strict name checking systemd-sysext does against the name of extension-release.NAME file, is disabled, as there is only one such in the resulting image. This is done to allow a user to freely rename the resulting image file. Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y Link: https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html Link: https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html Signed-off-by: Johannes Schneider --- meta/classes-recipe/image-sysext.bbclass | 43 ++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 meta/classes-recipe/image-sysext.bbclass diff --git a/meta/classes-recipe/image-sysext.bbclass b/meta/classes-recipe/image-sysext.bbclass new file mode 100644 index 0000000000..bc3e4d52b5 --- /dev/null +++ b/meta/classes-recipe/image-sysext.bbclass @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: MIT +# +# Copyright Leica Geosystems AG +# + +# systemd-sysext [1] has a simple mechanism for version compatibility: +# the extension to be loaded has to contain a +# /usr/lib/extension-release.d/extension-release.NAME +# with "NAME" *exactly* matching the filename of the extensions +# raw-device filename/ +# +# from the extension-release file the "ID" and "VERSION_ID" fields are +# matched against the etc/os-release and the extension is only "merged" +# if no mismatches between NAME, ID, and VERSION_ID. +# +# Link: https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html + +inherit image + +IMAGE_NAME_SUFFIX = ".sysext" +EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}" +IMAGE_LINK_NAME:append = ".sysext" + +DEPENDS += " os-release" + +sysext_image_mangle_rootfs() { + R=${IMAGE_ROOTFS} + + # pull a copy of the rootfs version information, which systemd-sysext matches against + cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release ${WORKDIR}/extension-release.base + + echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base + + install -d $R${nonarch_libdir}/extension-release.d + install -m 0644 ${WORKDIR}/extension-release.base \ + $R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME} + + # disable systemd-sysext's strict name checking, so that the image file can be renamed, while still being 'merge'-able + setfattr -n user.extension-release.strict -v false \ + $R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME} +} + +ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "