From patchwork Wed Jan 4 15:47:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E1CBC54EF0 for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web11.15526.1672847238412947408 for ; Wed, 04 Jan 2023 07:47:19 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Y6HRHEg9; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f54.google.com with SMTP id d4so25441610wrw.6 for ; Wed, 04 Jan 2023 07:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=UW51WbzVfAhGWGA4Tgcjqke+4Xb/iVHosWMWga/AoC0=; b=Y6HRHEg9nm7RfwGcKtn9WF6LcjnoXzXXi9LTC2vtzXG3RITPk5W7xJpGY8oMtd4+JU siria8EDA9RXqAGYwv2q6HhTqucfxHLFIuW2T4DKDN53bLskokqk6r/ItLjsQ+EiuJnI idJ8Svee2aUvM6TWLUx4iI+laUE409aUrsTSI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UW51WbzVfAhGWGA4Tgcjqke+4Xb/iVHosWMWga/AoC0=; b=bwJP6LFkB2TulpnlXuq2gW3bxK6QFGX4PQxhEMMTI0OgIMd0/pD7rRso2FnJeptG2P v7qXAEyBlOn6XM7gc6LV5z3oL/sXBLW8wazxgQ0Y8tyKqmkcQnKL62LZx+701nfRCqqe PF+8sDkRM41Hjwm1zT8eAYEwVCjVu2WW+xGwybC4ylBb3rn6d4Bjdec7IiohSmQyF0J7 k4mOfRlYLEUewLXS5oDozUX3JFdFjtfcoNiDgpFrvedr9JSGvtpwLLapg0Ov477JLSHm RwYL4UEs8+q2qdsQHVxm5aPHWzwjwTf/SrlJVluMhVywosQNsBay0yLQ/980+lNtAyFL 0oxA== X-Gm-Message-State: AFqh2kqV7+soBEjP36dtaq6uvpT6dpXPsOl7FkinpNnhUBVVFh/qNzZO iiwesea3+3GVwwnlsQi5YhOPLc+Tx+nGSaEs X-Google-Smtp-Source: AMrXdXvZu1UfS1Ves/41lfbqdrM3Xy+IuPtryeIxyhHfkKtIez0IbTL6JGJFsyrSmm63EEjZJXzuLQ== X-Received: by 2002:a5d:6f03:0:b0:28f:31d2:be38 with SMTP id ay3-20020a5d6f03000000b0028f31d2be38mr13766084wrb.43.1672847236109; Wed, 04 Jan 2023 07:47:16 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:15 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 01/10] package: Move fixup_perms function to bb function library Date: Wed, 4 Jan 2023 15:47:05 +0000 Message-Id: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175490 To avoid reparsing the bbclass code all the time, move the function to the python function library code which is more efficient. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 238 +-------------------------- meta/lib/oe/package.py | 242 ++++++++++++++++++++++++++++ 2 files changed, 243 insertions(+), 237 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index 7a0a428b304..29f0c80abdc 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -861,244 +861,8 @@ python perform_packagecopy () { perform_packagecopy[cleandirs] = "${PKGD}" perform_packagecopy[dirs] = "${PKGD}" -# We generate a master list of directories to process, we start by -# seeding this list with reasonable defaults, then load from -# the fs-perms.txt files python fixup_perms () { - import pwd, grp - - # init using a string with the same format as a line as documented in - # the fs-perms.txt file - # - # link - # - # __str__ can be used to print out an entry in the input format - # - # if fs_perms_entry.path is None: - # an error occurred - # if fs_perms_entry.link, you can retrieve: - # fs_perms_entry.path = path - # fs_perms_entry.link = target of link - # if not fs_perms_entry.link, you can retrieve: - # fs_perms_entry.path = path - # fs_perms_entry.mode = expected dir mode or None - # fs_perms_entry.uid = expected uid or -1 - # fs_perms_entry.gid = expected gid or -1 - # fs_perms_entry.walk = 'true' or something else - # fs_perms_entry.fmode = expected file mode or None - # fs_perms_entry.fuid = expected file uid or -1 - # fs_perms_entry_fgid = expected file gid or -1 - class fs_perms_entry(): - def __init__(self, line): - lsplit = line.split() - if len(lsplit) == 3 and lsplit[1].lower() == "link": - self._setlink(lsplit[0], lsplit[2]) - elif len(lsplit) == 8: - self._setdir(lsplit[0], lsplit[1], lsplit[2], lsplit[3], lsplit[4], lsplit[5], lsplit[6], lsplit[7]) - else: - msg = "Fixup Perms: invalid config line %s" % line - oe.qa.handle_error("perm-config", msg, d) - self.path = None - self.link = None - - def _setdir(self, path, mode, uid, gid, walk, fmode, fuid, fgid): - self.path = os.path.normpath(path) - self.link = None - self.mode = self._procmode(mode) - self.uid = self._procuid(uid) - self.gid = self._procgid(gid) - self.walk = walk.lower() - self.fmode = self._procmode(fmode) - self.fuid = self._procuid(fuid) - self.fgid = self._procgid(fgid) - - def _setlink(self, path, link): - self.path = os.path.normpath(path) - self.link = link - - def _procmode(self, mode): - if not mode or (mode and mode == "-"): - return None - else: - return int(mode,8) - - # Note uid/gid -1 has special significance in os.lchown - def _procuid(self, uid): - if uid is None or uid == "-": - return -1 - elif uid.isdigit(): - return int(uid) - else: - return pwd.getpwnam(uid).pw_uid - - def _procgid(self, gid): - if gid is None or gid == "-": - return -1 - elif gid.isdigit(): - return int(gid) - else: - return grp.getgrnam(gid).gr_gid - - # Use for debugging the entries - def __str__(self): - if self.link: - return "%s link %s" % (self.path, self.link) - else: - mode = "-" - if self.mode: - mode = "0%o" % self.mode - fmode = "-" - if self.fmode: - fmode = "0%o" % self.fmode - uid = self._mapugid(self.uid) - gid = self._mapugid(self.gid) - fuid = self._mapugid(self.fuid) - fgid = self._mapugid(self.fgid) - return "%s %s %s %s %s %s %s %s" % (self.path, mode, uid, gid, self.walk, fmode, fuid, fgid) - - def _mapugid(self, id): - if id is None or id == -1: - return "-" - else: - return "%d" % id - - # Fix the permission, owner and group of path - def fix_perms(path, mode, uid, gid, dir): - if mode and not os.path.islink(path): - #bb.note("Fixup Perms: chmod 0%o %s" % (mode, dir)) - os.chmod(path, mode) - # -1 is a special value that means don't change the uid/gid - # if they are BOTH -1, don't bother to lchown - if not (uid == -1 and gid == -1): - #bb.note("Fixup Perms: lchown %d:%d %s" % (uid, gid, dir)) - os.lchown(path, uid, gid) - - # Return a list of configuration files based on either the default - # files/fs-perms.txt or the contents of FILESYSTEM_PERMS_TABLES - # paths are resolved via BBPATH - def get_fs_perms_list(d): - str = "" - bbpath = d.getVar('BBPATH') - fs_perms_tables = d.getVar('FILESYSTEM_PERMS_TABLES') or "" - for conf_file in fs_perms_tables.split(): - confpath = bb.utils.which(bbpath, conf_file) - if confpath: - str += " %s" % bb.utils.which(bbpath, conf_file) - else: - bb.warn("cannot find %s specified in FILESYSTEM_PERMS_TABLES" % conf_file) - return str - - - - dvar = d.getVar('PKGD') - - fs_perms_table = {} - fs_link_table = {} - - # By default all of the standard directories specified in - # bitbake.conf will get 0755 root:root. - target_path_vars = [ 'base_prefix', - 'prefix', - 'exec_prefix', - 'base_bindir', - 'base_sbindir', - 'base_libdir', - 'datadir', - 'sysconfdir', - 'servicedir', - 'sharedstatedir', - 'localstatedir', - 'infodir', - 'mandir', - 'docdir', - 'bindir', - 'sbindir', - 'libexecdir', - 'libdir', - 'includedir', - 'oldincludedir' ] - - for path in target_path_vars: - dir = d.getVar(path) or "" - if dir == "": - continue - fs_perms_table[dir] = fs_perms_entry(d.expand("%s 0755 root root false - - -" % (dir))) - - # Now we actually load from the configuration files - for conf in get_fs_perms_list(d).split(): - if not os.path.exists(conf): - continue - with open(conf) as f: - for line in f: - if line.startswith('#'): - continue - lsplit = line.split() - if len(lsplit) == 0: - continue - if len(lsplit) != 8 and not (len(lsplit) == 3 and lsplit[1].lower() == "link"): - msg = "Fixup perms: %s invalid line: %s" % (conf, line) - oe.qa.handle_error("perm-line", msg, d) - continue - entry = fs_perms_entry(d.expand(line)) - if entry and entry.path: - if entry.link: - fs_link_table[entry.path] = entry - if entry.path in fs_perms_table: - fs_perms_table.pop(entry.path) - else: - fs_perms_table[entry.path] = entry - if entry.path in fs_link_table: - fs_link_table.pop(entry.path) - - # Debug -- list out in-memory table - #for dir in fs_perms_table: - # bb.note("Fixup Perms: %s: %s" % (dir, str(fs_perms_table[dir]))) - #for link in fs_link_table: - # bb.note("Fixup Perms: %s: %s" % (link, str(fs_link_table[link]))) - - # We process links first, so we can go back and fixup directory ownership - # for any newly created directories - # Process in sorted order so /run gets created before /run/lock, etc. - for entry in sorted(fs_link_table.values(), key=lambda x: x.link): - link = entry.link - dir = entry.path - origin = dvar + dir - if not (cpath.exists(origin) and cpath.isdir(origin) and not cpath.islink(origin)): - continue - - if link[0] == "/": - target = dvar + link - ptarget = link - else: - target = os.path.join(os.path.dirname(origin), link) - ptarget = os.path.join(os.path.dirname(dir), link) - if os.path.exists(target): - msg = "Fixup Perms: Unable to correct directory link, target already exists: %s -> %s" % (dir, ptarget) - oe.qa.handle_error("perm-link", msg, d) - continue - - # Create path to move directory to, move it, and then setup the symlink - bb.utils.mkdirhier(os.path.dirname(target)) - #bb.note("Fixup Perms: Rename %s -> %s" % (dir, ptarget)) - bb.utils.rename(origin, target) - #bb.note("Fixup Perms: Link %s -> %s" % (dir, link)) - os.symlink(link, origin) - - for dir in fs_perms_table: - origin = dvar + dir - if not (cpath.exists(origin) and cpath.isdir(origin)): - continue - - fix_perms(origin, fs_perms_table[dir].mode, fs_perms_table[dir].uid, fs_perms_table[dir].gid, dir) - - if fs_perms_table[dir].walk == 'true': - for root, dirs, files in os.walk(origin): - for dr in dirs: - each_dir = os.path.join(root, dr) - fix_perms(each_dir, fs_perms_table[dir].mode, fs_perms_table[dir].uid, fs_perms_table[dir].gid, dir) - for f in files: - each_file = os.path.join(root, f) - fix_perms(each_file, fs_perms_table[dir].fmode, fs_perms_table[dir].fuid, fs_perms_table[dir].fgid, dir) + oe.package.fixup_perms(d) } def package_debug_vars(d): diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index 4aa40d70eec..05447f88084 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -292,3 +292,245 @@ def read_shlib_providers(d): shlib_provider[s[0]] = {} shlib_provider[s[0]][s[1]] = (dep_pkg, s[2]) return shlib_provider + +# We generate a master list of directories to process, we start by +# seeding this list with reasonable defaults, then load from +# the fs-perms.txt files +def fixup_perms(d): + import pwd, grp + import oe.cachedpath + + cpath = oe.cachedpath.CachedPath() + dvar = d.getVar('PKGD') + + # init using a string with the same format as a line as documented in + # the fs-perms.txt file + # + # link + # + # __str__ can be used to print out an entry in the input format + # + # if fs_perms_entry.path is None: + # an error occurred + # if fs_perms_entry.link, you can retrieve: + # fs_perms_entry.path = path + # fs_perms_entry.link = target of link + # if not fs_perms_entry.link, you can retrieve: + # fs_perms_entry.path = path + # fs_perms_entry.mode = expected dir mode or None + # fs_perms_entry.uid = expected uid or -1 + # fs_perms_entry.gid = expected gid or -1 + # fs_perms_entry.walk = 'true' or something else + # fs_perms_entry.fmode = expected file mode or None + # fs_perms_entry.fuid = expected file uid or -1 + # fs_perms_entry_fgid = expected file gid or -1 + class fs_perms_entry(): + def __init__(self, line): + lsplit = line.split() + if len(lsplit) == 3 and lsplit[1].lower() == "link": + self._setlink(lsplit[0], lsplit[2]) + elif len(lsplit) == 8: + self._setdir(lsplit[0], lsplit[1], lsplit[2], lsplit[3], lsplit[4], lsplit[5], lsplit[6], lsplit[7]) + else: + msg = "Fixup Perms: invalid config line %s" % line + oe.qa.handle_error("perm-config", msg, d) + self.path = None + self.link = None + + def _setdir(self, path, mode, uid, gid, walk, fmode, fuid, fgid): + self.path = os.path.normpath(path) + self.link = None + self.mode = self._procmode(mode) + self.uid = self._procuid(uid) + self.gid = self._procgid(gid) + self.walk = walk.lower() + self.fmode = self._procmode(fmode) + self.fuid = self._procuid(fuid) + self.fgid = self._procgid(fgid) + + def _setlink(self, path, link): + self.path = os.path.normpath(path) + self.link = link + + def _procmode(self, mode): + if not mode or (mode and mode == "-"): + return None + else: + return int(mode,8) + + # Note uid/gid -1 has special significance in os.lchown + def _procuid(self, uid): + if uid is None or uid == "-": + return -1 + elif uid.isdigit(): + return int(uid) + else: + return pwd.getpwnam(uid).pw_uid + + def _procgid(self, gid): + if gid is None or gid == "-": + return -1 + elif gid.isdigit(): + return int(gid) + else: + return grp.getgrnam(gid).gr_gid + + # Use for debugging the entries + def __str__(self): + if self.link: + return "%s link %s" % (self.path, self.link) + else: + mode = "-" + if self.mode: + mode = "0%o" % self.mode + fmode = "-" + if self.fmode: + fmode = "0%o" % self.fmode + uid = self._mapugid(self.uid) + gid = self._mapugid(self.gid) + fuid = self._mapugid(self.fuid) + fgid = self._mapugid(self.fgid) + return "%s %s %s %s %s %s %s %s" % (self.path, mode, uid, gid, self.walk, fmode, fuid, fgid) + + def _mapugid(self, id): + if id is None or id == -1: + return "-" + else: + return "%d" % id + + # Fix the permission, owner and group of path + def fix_perms(path, mode, uid, gid, dir): + if mode and not os.path.islink(path): + #bb.note("Fixup Perms: chmod 0%o %s" % (mode, dir)) + os.chmod(path, mode) + # -1 is a special value that means don't change the uid/gid + # if they are BOTH -1, don't bother to lchown + if not (uid == -1 and gid == -1): + #bb.note("Fixup Perms: lchown %d:%d %s" % (uid, gid, dir)) + os.lchown(path, uid, gid) + + # Return a list of configuration files based on either the default + # files/fs-perms.txt or the contents of FILESYSTEM_PERMS_TABLES + # paths are resolved via BBPATH + def get_fs_perms_list(d): + str = "" + bbpath = d.getVar('BBPATH') + fs_perms_tables = d.getVar('FILESYSTEM_PERMS_TABLES') or "" + for conf_file in fs_perms_tables.split(): + confpath = bb.utils.which(bbpath, conf_file) + if confpath: + str += " %s" % bb.utils.which(bbpath, conf_file) + else: + bb.warn("cannot find %s specified in FILESYSTEM_PERMS_TABLES" % conf_file) + return str + + fs_perms_table = {} + fs_link_table = {} + + # By default all of the standard directories specified in + # bitbake.conf will get 0755 root:root. + target_path_vars = [ 'base_prefix', + 'prefix', + 'exec_prefix', + 'base_bindir', + 'base_sbindir', + 'base_libdir', + 'datadir', + 'sysconfdir', + 'servicedir', + 'sharedstatedir', + 'localstatedir', + 'infodir', + 'mandir', + 'docdir', + 'bindir', + 'sbindir', + 'libexecdir', + 'libdir', + 'includedir', + 'oldincludedir' ] + + for path in target_path_vars: + dir = d.getVar(path) or "" + if dir == "": + continue + fs_perms_table[dir] = fs_perms_entry(d.expand("%s 0755 root root false - - -" % (dir))) + + # Now we actually load from the configuration files + for conf in get_fs_perms_list(d).split(): + if not os.path.exists(conf): + continue + with open(conf) as f: + for line in f: + if line.startswith('#'): + continue + lsplit = line.split() + if len(lsplit) == 0: + continue + if len(lsplit) != 8 and not (len(lsplit) == 3 and lsplit[1].lower() == "link"): + msg = "Fixup perms: %s invalid line: %s" % (conf, line) + oe.qa.handle_error("perm-line", msg, d) + continue + entry = fs_perms_entry(d.expand(line)) + if entry and entry.path: + if entry.link: + fs_link_table[entry.path] = entry + if entry.path in fs_perms_table: + fs_perms_table.pop(entry.path) + else: + fs_perms_table[entry.path] = entry + if entry.path in fs_link_table: + fs_link_table.pop(entry.path) + + # Debug -- list out in-memory table + #for dir in fs_perms_table: + # bb.note("Fixup Perms: %s: %s" % (dir, str(fs_perms_table[dir]))) + #for link in fs_link_table: + # bb.note("Fixup Perms: %s: %s" % (link, str(fs_link_table[link]))) + + # We process links first, so we can go back and fixup directory ownership + # for any newly created directories + # Process in sorted order so /run gets created before /run/lock, etc. + for entry in sorted(fs_link_table.values(), key=lambda x: x.link): + link = entry.link + dir = entry.path + origin = dvar + dir + if not (cpath.exists(origin) and cpath.isdir(origin) and not cpath.islink(origin)): + continue + + if link[0] == "/": + target = dvar + link + ptarget = link + else: + target = os.path.join(os.path.dirname(origin), link) + ptarget = os.path.join(os.path.dirname(dir), link) + if os.path.exists(target): + msg = "Fixup Perms: Unable to correct directory link, target already exists: %s -> %s" % (dir, ptarget) + oe.qa.handle_error("perm-link", msg, d) + continue + + # Create path to move directory to, move it, and then setup the symlink + bb.utils.mkdirhier(os.path.dirname(target)) + #bb.note("Fixup Perms: Rename %s -> %s" % (dir, ptarget)) + bb.utils.rename(origin, target) + #bb.note("Fixup Perms: Link %s -> %s" % (dir, link)) + os.symlink(link, origin) + + for dir in fs_perms_table: + origin = dvar + dir + if not (cpath.exists(origin) and cpath.isdir(origin)): + continue + + fix_perms(origin, fs_perms_table[dir].mode, fs_perms_table[dir].uid, fs_perms_table[dir].gid, dir) + + if fs_perms_table[dir].walk == 'true': + for root, dirs, files in os.walk(origin): + for dr in dirs: + each_dir = os.path.join(root, dr) + fix_perms(each_dir, fs_perms_table[dir].mode, fs_perms_table[dir].uid, fs_perms_table[dir].gid, dir) + for f in files: + each_file = os.path.join(root, f) + fix_perms(each_file, fs_perms_table[dir].fmode, fs_perms_table[dir].fuid, fs_perms_table[dir].fgid, dir) + + + From patchwork Wed Jan 4 15:47:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CCC2C46467 for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web11.15527.1672847238552390024 for ; Wed, 04 Jan 2023 07:47:19 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Xpm/MKOT; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f46.google.com with SMTP id bk16so20209399wrb.11 for ; Wed, 04 Jan 2023 07:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=skdl6gNpUQ9ciw0mB29SiclOCu/J2E/hyZVd9RQXMCM=; b=Xpm/MKOTRIA9KBSm3YL2Bgvy3E97u3FvKVmi2WbbA3CyBUwCnAOw2ew92LBCM1ZURb R5EUzSVPoE/gwZcsesGITonUaGqHSs3j61/bV+haJy9dglcL1yF49MOTO8zCHxJNxoRf 5MJkWpCcCIjnuh35CADjYfvSSP3Lys7aZpytc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=skdl6gNpUQ9ciw0mB29SiclOCu/J2E/hyZVd9RQXMCM=; b=h4C3a4XH/wGzYPEqSzEpj1iCez/ETEQG7SZogSQA1hg2vXR5wuOokJaqT7LU0IWXZ9 PDQzsaMnow7+Cgc2ghMCgmZecVG16HR/xdQ+E32H2o/01IfM7u7JGsb1M/syFBckPmTf cEDcyPA+aIteuq/WSeldX3cckllGPa1Kznxox9QpaPMC4CGE6563Qc9KU3bne2m0dbxh BL5aEqb2GEE+LbOEzPtoDltslcp8zG00nSNQLgwkqeYCfgvzhkhj+CG/N+8N0FRTqKez Rrm41aRfprslUnPE1gvZCUZ+J68nfJZN2+OSFq/IU6vrfQKjUkNxZWnBKIKQyg9jqnol 36KA== X-Gm-Message-State: AFqh2kqXm98ve2bophAB8cjaV2Ry7qxXgxvaDKba0quPcl2kl1GuFphZ ns3O6zDpMfkEPv01+iUWehwdt0efzsmNDUY0 X-Google-Smtp-Source: AMrXdXs/ixniSCqsi5Ou2oYmPH3lXW/hH9/eBFEjfXvS1ut+iPBy+lQoFSWUzL5/OjvTfEhYwk9N+A== X-Received: by 2002:adf:e6ce:0:b0:293:1089:d6a with SMTP id y14-20020adfe6ce000000b0029310890d6amr10179766wrm.39.1672847236735; Wed, 04 Jan 2023 07:47:16 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:16 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 02/10] package: Move get_conffiles/files_from_filevars functions to lib Date: Wed, 4 Jan 2023 15:47:06 +0000 Message-Id: <20230104154714.1168535-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175489 To avoid reparsing the bbclass code all the time, move the functions to the python function library code which is more efficient. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 78 +------------------------ meta/classes-global/package_deb.bbclass | 2 +- meta/classes-global/package_ipk.bbclass | 2 +- meta/classes-global/package_rpm.bbclass | 2 +- meta/lib/oe/package.py | 77 ++++++++++++++++++++++++ 5 files changed, 81 insertions(+), 80 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index 29f0c80abdc..a31224f243b 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -257,82 +257,6 @@ python () { d.appendVarFlag('do_package', 'deptask', " do_packagedata") } -# Get a list of files from file vars by searching files under current working directory -# The list contains symlinks, directories and normal files. -def files_from_filevars(filevars): - import os,glob - cpath = oe.cachedpath.CachedPath() - files = [] - for f in filevars: - if os.path.isabs(f): - f = '.' + f - if not f.startswith("./"): - f = './' + f - globbed = glob.glob(f) - if globbed: - if [ f ] != globbed: - files += globbed - continue - files.append(f) - - symlink_paths = [] - for ind, f in enumerate(files): - # Handle directory symlinks. Truncate path to the lowest level symlink - parent = '' - for dirname in f.split('/')[:-1]: - parent = os.path.join(parent, dirname) - if dirname == '.': - continue - if cpath.islink(parent): - bb.warn("FILES contains file '%s' which resides under a " - "directory symlink. Please fix the recipe and use the " - "real path for the file." % f[1:]) - symlink_paths.append(f) - files[ind] = parent - f = parent - break - - if not cpath.islink(f): - if cpath.isdir(f): - newfiles = [ os.path.join(f,x) for x in os.listdir(f) ] - if newfiles: - files += newfiles - - return files, symlink_paths - -# Called in package_.bbclass to get the correct list of configuration files -def get_conffiles(pkg, d): - pkgdest = d.getVar('PKGDEST') - root = os.path.join(pkgdest, pkg) - cwd = os.getcwd() - os.chdir(root) - - conffiles = d.getVar('CONFFILES:%s' % pkg); - if conffiles == None: - conffiles = d.getVar('CONFFILES') - if conffiles == None: - conffiles = "" - conffiles = conffiles.split() - conf_orig_list = files_from_filevars(conffiles)[0] - - # Remove links and directories from conf_orig_list to get conf_list which only contains normal files - conf_list = [] - for f in conf_orig_list: - if os.path.isdir(f): - continue - if os.path.islink(f): - continue - if not os.path.exists(f): - continue - conf_list.append(f) - - # Remove the leading './' - for i in range(0, len(conf_list)): - conf_list[i] = conf_list[i][1:] - - os.chdir(cwd) - return conf_list - def checkbuildpath(file, d): tmpdir = d.getVar('TMPDIR') with open(file) as f: @@ -1209,7 +1133,7 @@ python populate_packages () { filesvar.replace("//", "/") origfiles = filesvar.split() - files, symlink_paths = files_from_filevars(origfiles) + files, symlink_paths = oe.package.files_from_filevars(origfiles) if autodebug and pkg.endswith("-dbg"): files.extend(debug) diff --git a/meta/classes-global/package_deb.bbclass b/meta/classes-global/package_deb.bbclass index ec7e10dbc99..c3ae7d574dc 100644 --- a/meta/classes-global/package_deb.bbclass +++ b/meta/classes-global/package_deb.bbclass @@ -269,7 +269,7 @@ def deb_write_pkg(pkg, d): scriptfile.close() os.chmod(os.path.join(controldir, script), 0o755) - conffiles_str = ' '.join(get_conffiles(pkg, d)) + conffiles_str = ' '.join(oe.package.get_conffiles(pkg, d)) if conffiles_str: conffiles = open(os.path.join(controldir, 'conffiles'), 'w') for f in conffiles_str.split(): diff --git a/meta/classes-global/package_ipk.bbclass b/meta/classes-global/package_ipk.bbclass index c43592af7e3..0207ea874b1 100644 --- a/meta/classes-global/package_ipk.bbclass +++ b/meta/classes-global/package_ipk.bbclass @@ -226,7 +226,7 @@ def ipk_write_pkg(pkg, d): scriptfile.close() os.chmod(os.path.join(controldir, script), 0o755) - conffiles_str = ' '.join(get_conffiles(pkg, d)) + conffiles_str = ' '.join(oe.package.get_conffiles(pkg, d)) if conffiles_str: conffiles = open(os.path.join(controldir, 'conffiles'), 'w') for f in conffiles_str.split(): diff --git a/meta/classes-global/package_rpm.bbclass b/meta/classes-global/package_rpm.bbclass index 39efcc328eb..7ba73f48e7d 100644 --- a/meta/classes-global/package_rpm.bbclass +++ b/meta/classes-global/package_rpm.bbclass @@ -341,7 +341,7 @@ python write_specfile () { localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg) - conffiles = get_conffiles(pkg, d) + conffiles = oe.package.get_conffiles(pkg, d) dirfiles = localdata.getVar('DIRFILES') if dirfiles is not None: dirfiles = dirfiles.split() diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index 05447f88084..b4c8ab7222a 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -4,6 +4,8 @@ # SPDX-License-Identifier: GPL-2.0-only # +import os +import glob import stat import mmap import subprocess @@ -532,5 +534,80 @@ def fixup_perms(d): each_file = os.path.join(root, f) fix_perms(each_file, fs_perms_table[dir].fmode, fs_perms_table[dir].fuid, fs_perms_table[dir].fgid, dir) +# Get a list of files from file vars by searching files under current working directory +# The list contains symlinks, directories and normal files. +def files_from_filevars(filevars): + import oe.cachedpath + + cpath = oe.cachedpath.CachedPath() + files = [] + for f in filevars: + if os.path.isabs(f): + f = '.' + f + if not f.startswith("./"): + f = './' + f + globbed = glob.glob(f) + if globbed: + if [ f ] != globbed: + files += globbed + continue + files.append(f) + + symlink_paths = [] + for ind, f in enumerate(files): + # Handle directory symlinks. Truncate path to the lowest level symlink + parent = '' + for dirname in f.split('/')[:-1]: + parent = os.path.join(parent, dirname) + if dirname == '.': + continue + if cpath.islink(parent): + bb.warn("FILES contains file '%s' which resides under a " + "directory symlink. Please fix the recipe and use the " + "real path for the file." % f[1:]) + symlink_paths.append(f) + files[ind] = parent + f = parent + break + + if not cpath.islink(f): + if cpath.isdir(f): + newfiles = [ os.path.join(f,x) for x in os.listdir(f) ] + if newfiles: + files += newfiles + + return files, symlink_paths + +# Called in package_.bbclass to get the correct list of configuration files +def get_conffiles(pkg, d): + pkgdest = d.getVar('PKGDEST') + root = os.path.join(pkgdest, pkg) + cwd = os.getcwd() + os.chdir(root) + + conffiles = d.getVar('CONFFILES:%s' % pkg); + if conffiles == None: + conffiles = d.getVar('CONFFILES') + if conffiles == None: + conffiles = "" + conffiles = conffiles.split() + conf_orig_list = files_from_filevars(conffiles)[0] + + # Remove links and directories from conf_orig_list to get conf_list which only contains normal files + conf_list = [] + for f in conf_orig_list: + if os.path.isdir(f): + continue + if os.path.islink(f): + continue + if not os.path.exists(f): + continue + conf_list.append(f) + + # Remove the leading './' + for i in range(0, len(conf_list)): + conf_list[i] = conf_list[i][1:] + os.chdir(cwd) + return conf_list From patchwork Wed Jan 4 15:47:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B2D2C61DB3 for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web11.15528.1672847239246825140 for ; Wed, 04 Jan 2023 07:47:19 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=GRAmzawg; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f46.google.com with SMTP id z16so16935299wrw.1 for ; Wed, 04 Jan 2023 07:47:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=SFKVS7G7a+yXqTyIpx7g+1ERBej2nN2g3k3O4GuhM40=; b=GRAmzawgIqup/jFtsoYvuQD+fjvqKXKvlbirK001/aJQG6v4RNi6kzkbmJL2i9UvS1 OPGqAjfRtNvJ2qvcXbU787GbH9Dpf6CZsgpIGA2thIwemV8OeWhLE208VUc/TR9FB6Wc b203nXJcJ6+CGdh8TzREIzuq2NsJ1wd80Eq/s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFKVS7G7a+yXqTyIpx7g+1ERBej2nN2g3k3O4GuhM40=; b=TEGcsO+GJUJxbNSXhJhPxz+kRcgy/nDPAeLe9MjbGvLQKmCpkHRGQ4nAB/pnzQpoUp kXU1VgWM5Tsyn6YWfuOp21ahD0KIrMSao7VgQZ9ZuEmDvoKTD+5++zNahOAXuZgLrKvX EVIIk61zhUSHzNTNuEkyAFtTn2rRtSFFlyXW5UBBKWOh7lhpV6sfcOCxQunZvGCadL+y 08Dd6vZhdnZbbldtaQO2t8rOiJhzBY22Eq+WY8pCPhC8g+6J8gy0MJLfGhhcVmuv8CEx mJazMNzjNlFn8jowvGJKnyBfGIXHaGxlLJ4ADEtJhn1sgKlOQyXuQKsFU/gIK8xfj+IL Qovw== X-Gm-Message-State: AFqh2kp2T1+WIZP/Rd9nax77oKtqpOnXUCHB8pV1fs/72feuGezpRQKR JJIrlYY7Bp++JnZLGh1Ku03jdTA4QDESx3gt X-Google-Smtp-Source: AMrXdXvcv2Bosem92u7TQZRZ65PIF29cD3ImESKnv1zk0DuLGmwPKWRm5itTO4VMoCvNw8u1WA0JHQ== X-Received: by 2002:a05:6000:1b0e:b0:28c:44b4:d8ec with SMTP id f14-20020a0560001b0e00b0028c44b4d8ecmr12848416wrz.7.1672847237474; Wed, 04 Jan 2023 07:47:17 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:16 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 03/10] package: Move pkgdata handling functions to oe.packagedata Date: Wed, 4 Jan 2023 15:47:07 +0000 Message-Id: <20230104154714.1168535-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175491 To avoid reparsing the bbclass code all the time, move the functions to the packagedata python function library code which is more efficient. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 61 +------------------ meta/classes-global/package_deb.bbclass | 2 +- meta/classes-global/package_ipk.bbclass | 2 +- meta/classes-global/package_rpm.bbclass | 4 +- meta/classes-recipe/image.bbclass | 6 +- meta/classes-recipe/populate_sdk_base.bbclass | 8 +-- meta/lib/oe/packagedata.py | 53 ++++++++++++++++ 7 files changed, 67 insertions(+), 69 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index a31224f243b..72e39f69149 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -566,61 +566,6 @@ def copydebugsources(debugsrcdir, sources, d): if os.path.exists(p) and not os.listdir(p): os.rmdir(p) -# -# Package data handling routines -# - -def get_package_mapping (pkg, basepkg, d, depversions=None): - import oe.packagedata - - data = oe.packagedata.read_subpkgdata(pkg, d) - key = "PKG:%s" % pkg - - if key in data: - if bb.data.inherits_class('allarch', d) and bb.data.inherits_class('packagegroup', d) and pkg != data[key]: - bb.error("An allarch packagegroup shouldn't depend on packages which are dynamically renamed (%s to %s)" % (pkg, data[key])) - # Have to avoid undoing the write_extra_pkgs(global_variants...) - if bb.data.inherits_class('allarch', d) and not d.getVar('MULTILIB_VARIANTS') \ - and data[key] == basepkg: - return pkg - if depversions == []: - # Avoid returning a mapping if the renamed package rprovides its original name - rprovkey = "RPROVIDES:%s" % pkg - if rprovkey in data: - if pkg in bb.utils.explode_dep_versions2(data[rprovkey]): - bb.note("%s rprovides %s, not replacing the latter" % (data[key], pkg)) - return pkg - # Do map to rewritten package name - return data[key] - - return pkg - -def get_package_additional_metadata (pkg_type, d): - base_key = "PACKAGE_ADD_METADATA" - for key in ("%s_%s" % (base_key, pkg_type.upper()), base_key): - if d.getVar(key, False) is None: - continue - d.setVarFlag(key, "type", "list") - if d.getVarFlag(key, "separator") is None: - d.setVarFlag(key, "separator", "\\n") - metadata_fields = [field.strip() for field in oe.data.typed_value(key, d)] - return "\n".join(metadata_fields).strip() - -def runtime_mapping_rename (varname, pkg, d): - #bb.note("%s before: %s" % (varname, d.getVar(varname))) - - new_depends = {} - deps = bb.utils.explode_dep_versions2(d.getVar(varname) or "") - for depend, depversions in deps.items(): - new_depend = get_package_mapping(depend, pkg, d, depversions) - if depend != new_depend: - bb.note("package name mapping done: %s -> %s" % (depend, new_depend)) - new_depends[new_depend] = deps[depend] - - d.setVar(varname, bb.utils.join_deps(new_depends, commasep=False)) - - #bb.note("%s after: %s" % (varname, d.getVar(varname))) - # # Used by do_packagedata (and possibly other routines post do_package) # @@ -2244,6 +2189,6 @@ def mapping_rename_hook(d): like debian.bbclass or manual PKG variable name changes """ pkg = d.getVar("PKG") - runtime_mapping_rename("RDEPENDS", pkg, d) - runtime_mapping_rename("RRECOMMENDS", pkg, d) - runtime_mapping_rename("RSUGGESTS", pkg, d) + oe.packagedata.runtime_mapping_rename("RDEPENDS", pkg, d) + oe.packagedata.runtime_mapping_rename("RRECOMMENDS", pkg, d) + oe.packagedata.runtime_mapping_rename("RSUGGESTS", pkg, d) diff --git a/meta/classes-global/package_deb.bbclass b/meta/classes-global/package_deb.bbclass index c3ae7d574dc..99e377b4551 100644 --- a/meta/classes-global/package_deb.bbclass +++ b/meta/classes-global/package_deb.bbclass @@ -169,7 +169,7 @@ def deb_write_pkg(pkg, d): # more fields - custom_fields_chunk = get_package_additional_metadata("deb", localdata) + custom_fields_chunk = oe.packagedata.get_package_additional_metadata("deb", localdata) if custom_fields_chunk: ctrlfile.write(custom_fields_chunk) ctrlfile.write("\n") diff --git a/meta/classes-global/package_ipk.bbclass b/meta/classes-global/package_ipk.bbclass index 0207ea874b1..9b75f5cf1c3 100644 --- a/meta/classes-global/package_ipk.bbclass +++ b/meta/classes-global/package_ipk.bbclass @@ -162,7 +162,7 @@ def ipk_write_pkg(pkg, d): else: ctrlfile.write(c % tuple(pullData(fs, localdata))) - custom_fields_chunk = get_package_additional_metadata("ipk", localdata) + custom_fields_chunk = oe.packagedata.get_package_additional_metadata("ipk", localdata) if custom_fields_chunk is not None: ctrlfile.write(custom_fields_chunk) ctrlfile.write("\n") diff --git a/meta/classes-global/package_rpm.bbclass b/meta/classes-global/package_rpm.bbclass index 7ba73f48e7d..16631688b10 100644 --- a/meta/classes-global/package_rpm.bbclass +++ b/meta/classes-global/package_rpm.bbclass @@ -299,7 +299,7 @@ python write_specfile () { srcmaintainer = localdata.getVar('MAINTAINER') srchomepage = localdata.getVar('HOMEPAGE') srcdescription = localdata.getVar('DESCRIPTION') or "." - srccustomtagschunk = get_package_additional_metadata("rpm", localdata) + srccustomtagschunk = oe.packagedata.get_package_additional_metadata("rpm", localdata) srcdepends = d.getVar('DEPENDS') srcrdepends = "" @@ -355,7 +355,7 @@ python write_specfile () { splitlicense = (localdata.getVar('LICENSE') or "") splitsection = (localdata.getVar('SECTION') or "") splitdescription = (localdata.getVar('DESCRIPTION') or ".") - splitcustomtagschunk = get_package_additional_metadata("rpm", localdata) + splitcustomtagschunk = oe.packagedata.get_package_additional_metadata("rpm", localdata) translate_vers('RDEPENDS', localdata) translate_vers('RRECOMMENDS', localdata) diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass index e387645503a..5980ca481a3 100644 --- a/meta/classes-recipe/image.bbclass +++ b/meta/classes-recipe/image.bbclass @@ -247,9 +247,9 @@ fakeroot python do_rootfs () { # otherwise, the multilib renaming could step in and squash any fixups that # may have occurred. pn = d.getVar('PN') - runtime_mapping_rename("PACKAGE_INSTALL", pn, d) - runtime_mapping_rename("PACKAGE_INSTALL_ATTEMPTONLY", pn, d) - runtime_mapping_rename("BAD_RECOMMENDATIONS", pn, d) + oe.packagedata.runtime_mapping_rename("PACKAGE_INSTALL", pn, d) + oe.packagedata.runtime_mapping_rename("PACKAGE_INSTALL_ATTEMPTONLY", pn, d) + oe.packagedata.runtime_mapping_rename("BAD_RECOMMENDATIONS", pn, d) # Generate the initial manifest create_manifest(d) diff --git a/meta/classes-recipe/populate_sdk_base.bbclass b/meta/classes-recipe/populate_sdk_base.bbclass index 917619962c3..c1fabddcf21 100644 --- a/meta/classes-recipe/populate_sdk_base.bbclass +++ b/meta/classes-recipe/populate_sdk_base.bbclass @@ -184,13 +184,13 @@ def populate_sdk_common(d): d.setVar("PACKAGE_INSTALL_ATTEMPTONLY", ' '.join(inst_attempt_pkgs)) pn = d.getVar('PN') - runtime_mapping_rename("TOOLCHAIN_TARGET_TASK", pn, d) - runtime_mapping_rename("TOOLCHAIN_TARGET_TASK_ATTEMPTONLY", pn, d) + oe.packagedata.runtime_mapping_rename("TOOLCHAIN_TARGET_TASK", pn, d) + oe.packagedata.runtime_mapping_rename("TOOLCHAIN_TARGET_TASK_ATTEMPTONLY", pn, d) ld = bb.data.createCopy(d) ld.setVar("PKGDATA_DIR", "${STAGING_DIR}/${SDK_ARCH}-${SDKPKGSUFFIX}${SDK_VENDOR}-${SDK_OS}/pkgdata") - runtime_mapping_rename("TOOLCHAIN_HOST_TASK", pn, ld) - runtime_mapping_rename("TOOLCHAIN_HOST_TASK_ATTEMPTONLY", pn, ld) + oe.packagedata.runtime_mapping_rename("TOOLCHAIN_HOST_TASK", pn, ld) + oe.packagedata.runtime_mapping_rename("TOOLCHAIN_HOST_TASK_ATTEMPTONLY", pn, ld) d.setVar("TOOLCHAIN_HOST_TASK", ld.getVar("TOOLCHAIN_HOST_TASK")) d.setVar("TOOLCHAIN_HOST_TASK_ATTEMPTONLY", ld.getVar("TOOLCHAIN_HOST_TASK_ATTEMPTONLY")) diff --git a/meta/lib/oe/packagedata.py b/meta/lib/oe/packagedata.py index b2ed8b5a3d8..ff260f405ca 100644 --- a/meta/lib/oe/packagedata.py +++ b/meta/lib/oe/packagedata.py @@ -110,3 +110,56 @@ def recipename(pkg, d): """Return the recipe name for the given binary package name.""" return pkgmap(d).get(pkg) + +def get_package_mapping(pkg, basepkg, d, depversions=None): + import oe.packagedata + + data = oe.packagedata.read_subpkgdata(pkg, d) + key = "PKG:%s" % pkg + + if key in data: + if bb.data.inherits_class('allarch', d) and bb.data.inherits_class('packagegroup', d) and pkg != data[key]: + bb.error("An allarch packagegroup shouldn't depend on packages which are dynamically renamed (%s to %s)" % (pkg, data[key])) + # Have to avoid undoing the write_extra_pkgs(global_variants...) + if bb.data.inherits_class('allarch', d) and not d.getVar('MULTILIB_VARIANTS') \ + and data[key] == basepkg: + return pkg + if depversions == []: + # Avoid returning a mapping if the renamed package rprovides its original name + rprovkey = "RPROVIDES:%s" % pkg + if rprovkey in data: + if pkg in bb.utils.explode_dep_versions2(data[rprovkey]): + bb.note("%s rprovides %s, not replacing the latter" % (data[key], pkg)) + return pkg + # Do map to rewritten package name + return data[key] + + return pkg + +def get_package_additional_metadata(pkg_type, d): + base_key = "PACKAGE_ADD_METADATA" + for key in ("%s_%s" % (base_key, pkg_type.upper()), base_key): + if d.getVar(key, False) is None: + continue + d.setVarFlag(key, "type", "list") + if d.getVarFlag(key, "separator") is None: + d.setVarFlag(key, "separator", "\\n") + metadata_fields = [field.strip() for field in oe.data.typed_value(key, d)] + return "\n".join(metadata_fields).strip() + +def runtime_mapping_rename(varname, pkg, d): + #bb.note("%s before: %s" % (varname, d.getVar(varname))) + + new_depends = {} + deps = bb.utils.explode_dep_versions2(d.getVar(varname) or "") + for depend, depversions in deps.items(): + new_depend = get_package_mapping(depend, pkg, d, depversions) + if depend != new_depend: + bb.note("package name mapping done: %s -> %s" % (depend, new_depend)) + new_depends[new_depend] = deps[depend] + + d.setVar(varname, bb.utils.join_deps(new_depends, commasep=False)) + + #bb.note("%s after: %s" % (varname, d.getVar(varname))) + + From patchwork Wed Jan 4 15:47:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37CCCC53210 for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web10.15377.1672847240031413440 for ; Wed, 04 Jan 2023 07:47:20 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=K9dXw5Ue; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f46.google.com with SMTP id d9so4843652wrp.10 for ; Wed, 04 Jan 2023 07:47:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=zZurcP8raGMksJL4jCEZMb6wMDTuvWVtIv6qPB8G1sE=; b=K9dXw5UejJ8WOhCyhHBcX9Vu0NAgTrT+MaK+xnqgwQQNmxn4ghFShFOxSg78ZbyFxy ovt9JBRAxLIx4zMbwIjBYPphqYZfFUqh1c5QqSCYACqnrosNwOQJrZAIZFijEvW6otQ5 w6B3iq/+DRf41pT5xhQ9gHl+M268KhfBfZ85I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zZurcP8raGMksJL4jCEZMb6wMDTuvWVtIv6qPB8G1sE=; b=t1mBK+nGSM1nteFzUorrg6bT//0d1QpPeCxspea87JiQUfgQmxCVdvz75CBE2VLBx5 FafIDEdbAvxS0AvXinCcHrvag2M/EPuTLS1IUOmYPyUx1nDibUO5bBK2tzgelsclpaSt LvNs3eVXRl1vXUCm1zuGVY0SvchIcdyDy7q0+Q29HQ+l6IRm3gUuAOBuU/z/WVmWcT+z NLW5V2xYqDgMK+MG9yQFVFjWcLnmJ5jn6rUIi3vPayPNXFUsy5XGFDcfFX03iAhWQWad mxX7AG1rkkMRZ/X7lZRsTfw6BBwiZ9qqBg/bDlK3A+Ee7tGUVA9OcLV8XyuSFFxlRmub jJIg== X-Gm-Message-State: AFqh2kr7d/GyCcjAnlSRHQFnu0yXqPcSCqisXAngmD1EV3CEpsm5DoWa BAP4b/ohuMxTA7Y1ZIHn6nJ7wVqe23WDdQ1D X-Google-Smtp-Source: AMrXdXv5JLT4nT+WCrs9lQXHf90FJf2SIFzKDiHqYvkKNvOvVuT9uZVyjWvnt/EGbFwJ2MJavbMIWA== X-Received: by 2002:a05:6000:136b:b0:27f:85ce:bc9e with SMTP id q11-20020a056000136b00b0027f85cebc9emr24758543wrz.24.1672847238199; Wed, 04 Jan 2023 07:47:18 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:17 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 04/10] package: Move emit_pkgdata to packagedata.py Date: Wed, 4 Jan 2023 15:47:08 +0000 Message-Id: <20230104154714.1168535-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175492 Move one of the PACKAGEFUNCS from the package bbclass to packagedata library code for parsing efficiency. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 179 +--------------------------- meta/conf/bitbake.conf | 1 + meta/lib/oe/packagedata.py | 178 +++++++++++++++++++++++++++ 3 files changed, 181 insertions(+), 177 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index 72e39f69149..389c8578040 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -1232,185 +1232,10 @@ PKGDESTWORK = "${WORKDIR}/pkgdata" PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY RDEPENDS RPROVIDES RRECOMMENDS RSUGGESTS RREPLACES RCONFLICTS SECTION PKG ALLOW_EMPTY FILES CONFFILES FILES_INFO PACKAGE_ADD_METADATA pkg_postinst pkg_postrm pkg_preinst pkg_prerm" python emit_pkgdata() { - from glob import glob - import json - import bb.compress.zstd - - def process_postinst_on_target(pkg, mlprefix): - pkgval = d.getVar('PKG:%s' % pkg) - if pkgval is None: - pkgval = pkg - - defer_fragment = """ -if [ -n "$D" ]; then - $INTERCEPT_DIR/postinst_intercept delay_to_first_boot %s mlprefix=%s - exit 0 -fi -""" % (pkgval, mlprefix) - - postinst = d.getVar('pkg_postinst:%s' % pkg) - postinst_ontarget = d.getVar('pkg_postinst_ontarget:%s' % pkg) - - if postinst_ontarget: - bb.debug(1, 'adding deferred pkg_postinst_ontarget() to pkg_postinst() for %s' % pkg) - if not postinst: - postinst = '#!/bin/sh\n' - postinst += defer_fragment - postinst += postinst_ontarget - d.setVar('pkg_postinst:%s' % pkg, postinst) - - def add_set_e_to_scriptlets(pkg): - for scriptlet_name in ('pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm'): - scriptlet = d.getVar('%s:%s' % (scriptlet_name, pkg)) - if scriptlet: - scriptlet_split = scriptlet.split('\n') - if scriptlet_split[0].startswith("#!"): - scriptlet = scriptlet_split[0] + "\nset -e\n" + "\n".join(scriptlet_split[1:]) - else: - scriptlet = "set -e\n" + "\n".join(scriptlet_split[0:]) - d.setVar('%s:%s' % (scriptlet_name, pkg), scriptlet) - - def write_if_exists(f, pkg, var): - def encode(str): - import codecs - c = codecs.getencoder("unicode_escape") - return c(str)[0].decode("latin1") - - val = d.getVar('%s:%s' % (var, pkg)) - if val: - f.write('%s:%s: %s\n' % (var, pkg, encode(val))) - return val - val = d.getVar('%s' % (var)) - if val: - f.write('%s: %s\n' % (var, encode(val))) - return val - - def write_extra_pkgs(variants, pn, packages, pkgdatadir): - for variant in variants: - with open("%s/%s-%s" % (pkgdatadir, variant, pn), 'w') as fd: - fd.write("PACKAGES: %s\n" % ' '.join( - map(lambda pkg: '%s-%s' % (variant, pkg), packages.split()))) - - def write_extra_runtime_pkgs(variants, packages, pkgdatadir): - for variant in variants: - for pkg in packages.split(): - ml_pkg = "%s-%s" % (variant, pkg) - subdata_file = "%s/runtime/%s" % (pkgdatadir, ml_pkg) - with open(subdata_file, 'w') as fd: - fd.write("PKG:%s: %s" % (ml_pkg, pkg)) - - packages = d.getVar('PACKAGES') - pkgdest = d.getVar('PKGDEST') - pkgdatadir = d.getVar('PKGDESTWORK') - - data_file = pkgdatadir + d.expand("/${PN}") - with open(data_file, 'w') as fd: - fd.write("PACKAGES: %s\n" % packages) - - pkgdebugsource = d.getVar("PKGDEBUGSOURCES") or [] - - pn = d.getVar('PN') - global_variants = (d.getVar('MULTILIB_GLOBAL_VARIANTS') or "").split() - variants = (d.getVar('MULTILIB_VARIANTS') or "").split() - - if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d): - write_extra_pkgs(variants, pn, packages, pkgdatadir) - - if bb.data.inherits_class('allarch', d) and not variants \ - and not bb.data.inherits_class('packagegroup', d): - write_extra_pkgs(global_variants, pn, packages, pkgdatadir) - - workdir = d.getVar('WORKDIR') - - for pkg in packages.split(): - pkgval = d.getVar('PKG:%s' % pkg) - if pkgval is None: - pkgval = pkg - d.setVar('PKG:%s' % pkg, pkg) - - extended_data = { - "files_info": {} - } - - pkgdestpkg = os.path.join(pkgdest, pkg) - files = {} - files_extra = {} - total_size = 0 - seen = set() - for f in pkgfiles[pkg]: - fpath = os.sep + os.path.relpath(f, pkgdestpkg) - - fstat = os.lstat(f) - files[fpath] = fstat.st_size - - extended_data["files_info"].setdefault(fpath, {}) - extended_data["files_info"][fpath]['size'] = fstat.st_size - - if fstat.st_ino not in seen: - seen.add(fstat.st_ino) - total_size += fstat.st_size - - if fpath in pkgdebugsource: - extended_data["files_info"][fpath]['debugsrc'] = pkgdebugsource[fpath] - del pkgdebugsource[fpath] - - d.setVar('FILES_INFO:' + pkg , json.dumps(files, sort_keys=True)) - - process_postinst_on_target(pkg, d.getVar("MLPREFIX")) - add_set_e_to_scriptlets(pkg) - - subdata_file = pkgdatadir + "/runtime/%s" % pkg - with open(subdata_file, 'w') as sf: - for var in (d.getVar('PKGDATA_VARS') or "").split(): - val = write_if_exists(sf, pkg, var) - - write_if_exists(sf, pkg, 'FILERPROVIDESFLIST') - for dfile in sorted((d.getVar('FILERPROVIDESFLIST:' + pkg) or "").split()): - write_if_exists(sf, pkg, 'FILERPROVIDES:' + dfile) - - write_if_exists(sf, pkg, 'FILERDEPENDSFLIST') - for dfile in sorted((d.getVar('FILERDEPENDSFLIST:' + pkg) or "").split()): - write_if_exists(sf, pkg, 'FILERDEPENDS:' + dfile) - - sf.write('%s:%s: %d\n' % ('PKGSIZE', pkg, total_size)) - - subdata_extended_file = pkgdatadir + "/extended/%s.json.zstd" % pkg - num_threads = int(d.getVar("BB_NUMBER_THREADS")) - with bb.compress.zstd.open(subdata_extended_file, "wt", encoding="utf-8", num_threads=num_threads) as f: - json.dump(extended_data, f, sort_keys=True, separators=(",", ":")) - - # Symlinks needed for rprovides lookup - rprov = d.getVar('RPROVIDES:%s' % pkg) or d.getVar('RPROVIDES') - if rprov: - for p in bb.utils.explode_deps(rprov): - subdata_sym = pkgdatadir + "/runtime-rprovides/%s/%s" % (p, pkg) - bb.utils.mkdirhier(os.path.dirname(subdata_sym)) - oe.path.symlink("../../runtime/%s" % pkg, subdata_sym, True) - - allow_empty = d.getVar('ALLOW_EMPTY:%s' % pkg) - if not allow_empty: - allow_empty = d.getVar('ALLOW_EMPTY') - root = "%s/%s" % (pkgdest, pkg) - os.chdir(root) - g = glob('*') - if g or allow_empty == "1": - # Symlinks needed for reverse lookups (from the final package name) - subdata_sym = pkgdatadir + "/runtime-reverse/%s" % pkgval - oe.path.symlink("../runtime/%s" % pkg, subdata_sym, True) - - packagedfile = pkgdatadir + '/runtime/%s.packaged' % pkg - open(packagedfile, 'w').close() - - if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d): - write_extra_runtime_pkgs(variants, packages, pkgdatadir) - - if bb.data.inherits_class('allarch', d) and not variants \ - and not bb.data.inherits_class('packagegroup', d): - write_extra_runtime_pkgs(global_variants, packages, pkgdatadir) - + import oe.packagedata + oe.packagedata.emit_pkgdata(pkgfiles, d) } emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides ${PKGDESTWORK}/extended" -emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" ldconfig_postinst_fragment() { if [ x"$D" = "x" ]; then diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index bfa28d3bc9b..05d5a844de2 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -973,3 +973,4 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.multiprocess_launch[vardepsexclude] = "BB_NUMBER_THREADS" +oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" diff --git a/meta/lib/oe/packagedata.py b/meta/lib/oe/packagedata.py index ff260f405ca..efa600ddcd3 100644 --- a/meta/lib/oe/packagedata.py +++ b/meta/lib/oe/packagedata.py @@ -6,6 +6,11 @@ import codecs import os +import json +import bb.compress.zstd +import oe.path + +from glob import glob def packaged(pkg, d): return os.access(get_subpkgedata_fn(pkg, d) + '.packaged', os.R_OK) @@ -162,4 +167,177 @@ def runtime_mapping_rename(varname, pkg, d): #bb.note("%s after: %s" % (varname, d.getVar(varname))) +def emit_pkgdata(pkgfiles, d): + def process_postinst_on_target(pkg, mlprefix): + pkgval = d.getVar('PKG:%s' % pkg) + if pkgval is None: + pkgval = pkg + + defer_fragment = """ +if [ -n "$D" ]; then + $INTERCEPT_DIR/postinst_intercept delay_to_first_boot %s mlprefix=%s + exit 0 +fi +""" % (pkgval, mlprefix) + + postinst = d.getVar('pkg_postinst:%s' % pkg) + postinst_ontarget = d.getVar('pkg_postinst_ontarget:%s' % pkg) + + if postinst_ontarget: + bb.debug(1, 'adding deferred pkg_postinst_ontarget() to pkg_postinst() for %s' % pkg) + if not postinst: + postinst = '#!/bin/sh\n' + postinst += defer_fragment + postinst += postinst_ontarget + d.setVar('pkg_postinst:%s' % pkg, postinst) + + def add_set_e_to_scriptlets(pkg): + for scriptlet_name in ('pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm'): + scriptlet = d.getVar('%s:%s' % (scriptlet_name, pkg)) + if scriptlet: + scriptlet_split = scriptlet.split('\n') + if scriptlet_split[0].startswith("#!"): + scriptlet = scriptlet_split[0] + "\nset -e\n" + "\n".join(scriptlet_split[1:]) + else: + scriptlet = "set -e\n" + "\n".join(scriptlet_split[0:]) + d.setVar('%s:%s' % (scriptlet_name, pkg), scriptlet) + + def write_if_exists(f, pkg, var): + def encode(str): + import codecs + c = codecs.getencoder("unicode_escape") + return c(str)[0].decode("latin1") + + val = d.getVar('%s:%s' % (var, pkg)) + if val: + f.write('%s:%s: %s\n' % (var, pkg, encode(val))) + return val + val = d.getVar('%s' % (var)) + if val: + f.write('%s: %s\n' % (var, encode(val))) + return val + + def write_extra_pkgs(variants, pn, packages, pkgdatadir): + for variant in variants: + with open("%s/%s-%s" % (pkgdatadir, variant, pn), 'w') as fd: + fd.write("PACKAGES: %s\n" % ' '.join( + map(lambda pkg: '%s-%s' % (variant, pkg), packages.split()))) + + def write_extra_runtime_pkgs(variants, packages, pkgdatadir): + for variant in variants: + for pkg in packages.split(): + ml_pkg = "%s-%s" % (variant, pkg) + subdata_file = "%s/runtime/%s" % (pkgdatadir, ml_pkg) + with open(subdata_file, 'w') as fd: + fd.write("PKG:%s: %s" % (ml_pkg, pkg)) + + packages = d.getVar('PACKAGES') + pkgdest = d.getVar('PKGDEST') + pkgdatadir = d.getVar('PKGDESTWORK') + + data_file = pkgdatadir + d.expand("/${PN}") + with open(data_file, 'w') as fd: + fd.write("PACKAGES: %s\n" % packages) + + pkgdebugsource = d.getVar("PKGDEBUGSOURCES") or [] + + pn = d.getVar('PN') + global_variants = (d.getVar('MULTILIB_GLOBAL_VARIANTS') or "").split() + variants = (d.getVar('MULTILIB_VARIANTS') or "").split() + + if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d): + write_extra_pkgs(variants, pn, packages, pkgdatadir) + + if bb.data.inherits_class('allarch', d) and not variants \ + and not bb.data.inherits_class('packagegroup', d): + write_extra_pkgs(global_variants, pn, packages, pkgdatadir) + + workdir = d.getVar('WORKDIR') + + for pkg in packages.split(): + pkgval = d.getVar('PKG:%s' % pkg) + if pkgval is None: + pkgval = pkg + d.setVar('PKG:%s' % pkg, pkg) + + extended_data = { + "files_info": {} + } + + pkgdestpkg = os.path.join(pkgdest, pkg) + files = {} + files_extra = {} + total_size = 0 + seen = set() + for f in pkgfiles[pkg]: + fpath = os.sep + os.path.relpath(f, pkgdestpkg) + + fstat = os.lstat(f) + files[fpath] = fstat.st_size + + extended_data["files_info"].setdefault(fpath, {}) + extended_data["files_info"][fpath]['size'] = fstat.st_size + + if fstat.st_ino not in seen: + seen.add(fstat.st_ino) + total_size += fstat.st_size + + if fpath in pkgdebugsource: + extended_data["files_info"][fpath]['debugsrc'] = pkgdebugsource[fpath] + del pkgdebugsource[fpath] + + d.setVar('FILES_INFO:' + pkg , json.dumps(files, sort_keys=True)) + + process_postinst_on_target(pkg, d.getVar("MLPREFIX")) + add_set_e_to_scriptlets(pkg) + + subdata_file = pkgdatadir + "/runtime/%s" % pkg + with open(subdata_file, 'w') as sf: + for var in (d.getVar('PKGDATA_VARS') or "").split(): + val = write_if_exists(sf, pkg, var) + + write_if_exists(sf, pkg, 'FILERPROVIDESFLIST') + for dfile in sorted((d.getVar('FILERPROVIDESFLIST:' + pkg) or "").split()): + write_if_exists(sf, pkg, 'FILERPROVIDES:' + dfile) + + write_if_exists(sf, pkg, 'FILERDEPENDSFLIST') + for dfile in sorted((d.getVar('FILERDEPENDSFLIST:' + pkg) or "").split()): + write_if_exists(sf, pkg, 'FILERDEPENDS:' + dfile) + + sf.write('%s:%s: %d\n' % ('PKGSIZE', pkg, total_size)) + + subdata_extended_file = pkgdatadir + "/extended/%s.json.zstd" % pkg + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(subdata_extended_file, "wt", encoding="utf-8", num_threads=num_threads) as f: + json.dump(extended_data, f, sort_keys=True, separators=(",", ":")) + + # Symlinks needed for rprovides lookup + rprov = d.getVar('RPROVIDES:%s' % pkg) or d.getVar('RPROVIDES') + if rprov: + for p in bb.utils.explode_deps(rprov): + subdata_sym = pkgdatadir + "/runtime-rprovides/%s/%s" % (p, pkg) + bb.utils.mkdirhier(os.path.dirname(subdata_sym)) + oe.path.symlink("../../runtime/%s" % pkg, subdata_sym, True) + + allow_empty = d.getVar('ALLOW_EMPTY:%s' % pkg) + if not allow_empty: + allow_empty = d.getVar('ALLOW_EMPTY') + root = "%s/%s" % (pkgdest, pkg) + os.chdir(root) + g = glob('*') + if g or allow_empty == "1": + # Symlinks needed for reverse lookups (from the final package name) + subdata_sym = pkgdatadir + "/runtime-reverse/%s" % pkgval + oe.path.symlink("../runtime/%s" % pkg, subdata_sym, True) + + packagedfile = pkgdatadir + '/runtime/%s.packaged' % pkg + open(packagedfile, 'w').close() + + if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d): + write_extra_runtime_pkgs(variants, packages, pkgdatadir) + + if bb.data.inherits_class('allarch', d) and not variants \ + and not bb.data.inherits_class('packagegroup', d): + write_extra_runtime_pkgs(global_variants, packages, pkgdatadir) + From patchwork Wed Jan 4 15:47:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F0BDC54EBE for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web11.15531.1672847242160520083 for ; Wed, 04 Jan 2023 07:47:22 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=F/0UC5Rv; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id z8-20020a05600c220800b003d33b0bda11so1096900wml.0 for ; Wed, 04 Jan 2023 07:47:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Jn5a26K3ZMm2VFWTIIFE+FGjWQV02tQN5kwLLFmU9dI=; b=F/0UC5Rv8pe3NseyoOMyn/o4rGiFu29Fz4Oqu9Th6uEtsbFJ6cYbaPCUgbmKZKeMtk D6jdClVEnC2ZVDtefNPBbsy91oRSX/SJcKCIY64HSoX2T8ZL0DXkR4fjNciBwqYCG5tD sOL/gSNBADtqUag0wDpeOI1fVYb6flPLx2h5w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jn5a26K3ZMm2VFWTIIFE+FGjWQV02tQN5kwLLFmU9dI=; b=IrOoLEl3mBEPb58KhucAph2IWKH2JIBAA0tEC5vFZ68ykPeVUKEBwZLxD6nUd6ml/j 7U3CmPLp8Po3rVi7iCEXG3/bBERRovTjdJqdbuhERsQiu8huMLKrkpXm/+InMVlb8JTb OrQohynU0cXlTG6k6n+8N9sGbUyT030qpPlG+vD89BdDCKMXSQRJFLBTFCgkMn1HOi5N 12Sl5LNWFUL8TbqZ3/KRHyrdrBN2k6cwPuGn11p+D+/49yJLafABbfROf0kDgezdR3vm l/4VYhUrvAl4kXRpOVC80J5lPH+FBPMd3zN0RXFszVfyvu/xF4HpBk/0aijmHsNvzONv TRXA== X-Gm-Message-State: AFqh2koIPUsGPQf6iqh5cNreciDTpBW3h+nrA8fPUeM/vPYQA5tdh2tE /jzAH9oAL6D7GMrPy8YvzvLt9P12Zdy2Y4HO X-Google-Smtp-Source: AMrXdXtKC9TLjL7B7sMywjuPpF+VLSHSzp0WwYOJezbhn2TIqYBe8T8N3w+pc8vFbqz12uThaEC/xQ== X-Received: by 2002:a05:600c:3485:b0:3d0:761b:f86 with SMTP id a5-20020a05600c348500b003d0761b0f86mr33600032wmq.28.1672847239096; Wed, 04 Jan 2023 07:47:19 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:18 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 05/10] package: Move package functions to function library Date: Wed, 4 Jan 2023 15:47:09 +0000 Message-Id: <20230104154714.1168535-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175496 Move the bulk of the remaining package "processing" functions over to the package function library for parsing efficiency. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 1407 +------------------------- meta/lib/oe/package.py | 1408 ++++++++++++++++++++++++++- 2 files changed, 1416 insertions(+), 1399 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index 389c8578040..21a50bbb45e 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -69,21 +69,7 @@ PACKAGE_DEPENDS += "rpm-native dwarfsrcfiles-native" PACKAGE_WRITE_DEPS ??= "" def legitimize_package_name(s): - """ - Make sure package names are legitimate strings - """ - import re - - def fixutf(m): - cp = m.group(1) - if cp: - return ('\\u%s' % cp).encode('latin-1').decode('unicode_escape') - - # Handle unicode codepoints encoded as , as in glibc locale files. - s = re.sub(r'', fixutf, s) - - # Remaining package name validity fixes - return s.lower().replace('_', '-').replace('@', '+').replace(',', '+').replace('/', '-') + return oe.package.legitimize_package_name(s) def do_split_packages(d, root, file_regex, output_pattern, description, postinst=None, recursive=False, hook=None, extra_depends=None, aux_files_pattern=None, postrm=None, allow_dirs=False, prepend=False, match_path=False, aux_files_pattern_verbatim=None, allow_links=False, summary=None): """ @@ -195,7 +181,7 @@ def do_split_packages(d, root, file_regex, output_pattern, description, postinst mode = os.lstat(f).st_mode if not (stat.S_ISREG(mode) or (allow_links and stat.S_ISLNK(mode)) or (allow_dirs and stat.S_ISDIR(mode))): continue - on = legitimize_package_name(m.group(1)) + on = oe.package.legitimize_package_name(m.group(1)) pkg = output_pattern % on split_packages.add(pkg) if not pkg in packages: @@ -266,306 +252,6 @@ def checkbuildpath(file, d): return False -def parse_debugsources_from_dwarfsrcfiles_output(dwarfsrcfiles_output): - debugfiles = {} - - for line in dwarfsrcfiles_output.splitlines(): - if line.startswith("\t"): - debugfiles[os.path.normpath(line.split()[0])] = "" - - return debugfiles.keys() - -def source_info(file, d, fatal=True): - import subprocess - - cmd = ["dwarfsrcfiles", file] - try: - output = subprocess.check_output(cmd, universal_newlines=True, stderr=subprocess.STDOUT) - retval = 0 - except subprocess.CalledProcessError as exc: - output = exc.output - retval = exc.returncode - - # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure - if retval != 0 and retval != 255: - msg = "dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else "") - if fatal: - bb.fatal(msg) - bb.note(msg) - - debugsources = parse_debugsources_from_dwarfsrcfiles_output(output) - - return list(debugsources) - -def splitdebuginfo(file, dvar, dv, d): - # Function to split a single file into two components, one is the stripped - # target system binary, the other contains any debugging information. The - # two files are linked to reference each other. - # - # return a mapping of files:debugsources - - import stat - import subprocess - - src = file[len(dvar):] - dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"] - debugfile = dvar + dest - sources = [] - - if file.endswith(".ko") and file.find("/lib/modules/") != -1: - if oe.package.is_kernel_module_signed(file): - bb.debug(1, "Skip strip on signed module %s" % file) - return (file, sources) - - # Split the file... - bb.utils.mkdirhier(os.path.dirname(debugfile)) - #bb.note("Split %s -> %s" % (file, debugfile)) - # Only store off the hard link reference if we successfully split! - - dvar = d.getVar('PKGD') - objcopy = d.getVar("OBJCOPY") - - newmode = None - if not os.access(file, os.W_OK) or os.access(file, os.R_OK): - origmode = os.stat(file)[stat.ST_MODE] - newmode = origmode | stat.S_IWRITE | stat.S_IREAD - os.chmod(file, newmode) - - # We need to extract the debug src information here... - if dv["srcdir"]: - sources = source_info(file, d) - - bb.utils.mkdirhier(os.path.dirname(debugfile)) - - subprocess.check_output([objcopy, '--only-keep-debug', file, debugfile], stderr=subprocess.STDOUT) - - # Set the debuglink to have the view of the file path on the target - subprocess.check_output([objcopy, '--add-gnu-debuglink', debugfile, file], stderr=subprocess.STDOUT) - - if newmode: - os.chmod(file, origmode) - - return (file, sources) - -def splitstaticdebuginfo(file, dvar, dv, d): - # Unlike the function above, there is no way to split a static library - # two components. So to get similar results we will copy the unmodified - # static library (containing the debug symbols) into a new directory. - # We will then strip (preserving symbols) the static library in the - # typical location. - # - # return a mapping of files:debugsources - - import stat - - src = file[len(dvar):] - dest = dv["staticlibdir"] + os.path.dirname(src) + dv["staticdir"] + "/" + os.path.basename(src) + dv["staticappend"] - debugfile = dvar + dest - sources = [] - - # Copy the file... - bb.utils.mkdirhier(os.path.dirname(debugfile)) - #bb.note("Copy %s -> %s" % (file, debugfile)) - - dvar = d.getVar('PKGD') - - newmode = None - if not os.access(file, os.W_OK) or os.access(file, os.R_OK): - origmode = os.stat(file)[stat.ST_MODE] - newmode = origmode | stat.S_IWRITE | stat.S_IREAD - os.chmod(file, newmode) - - # We need to extract the debug src information here... - if dv["srcdir"]: - sources = source_info(file, d) - - bb.utils.mkdirhier(os.path.dirname(debugfile)) - - # Copy the unmodified item to the debug directory - shutil.copy2(file, debugfile) - - if newmode: - os.chmod(file, origmode) - - return (file, sources) - -def inject_minidebuginfo(file, dvar, dv, d): - # Extract just the symbols from debuginfo into minidebuginfo, - # compress it with xz and inject it back into the binary in a .gnu_debugdata section. - # https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html - - import subprocess - - readelf = d.getVar('READELF') - nm = d.getVar('NM') - objcopy = d.getVar('OBJCOPY') - - minidebuginfodir = d.expand('${WORKDIR}/minidebuginfo') - - src = file[len(dvar):] - dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"] - debugfile = dvar + dest - minidebugfile = minidebuginfodir + src + '.minidebug' - bb.utils.mkdirhier(os.path.dirname(minidebugfile)) - - # If we didn't produce debuginfo for any reason, we can't produce minidebuginfo either - # so skip it. - if not os.path.exists(debugfile): - bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file)) - return - - # minidebuginfo does not make sense to apply to ELF objects other than - # executables and shared libraries, skip applying the minidebuginfo - # generation for objects like kernel modules. - for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines(): - if not line.strip().startswith("Type:"): - continue - elftype = line.split(":")[1].strip() - if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]): - bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file)) - return - break - - # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo. - # We will exclude all of these from minidebuginfo to save space. - remove_section_names = [] - for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines(): - # strip the leading " [ 1]" section index to allow splitting on space - if ']' not in line: - continue - fields = line[line.index(']') + 1:].split() - if len(fields) < 7: - continue - name = fields[0] - type = fields[1] - flags = fields[6] - # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them - if name.startswith('.debug_'): - continue - if 'A' not in flags and type in ['PROGBITS', 'NOTE', 'NOBITS']: - remove_section_names.append(name) - - # List dynamic symbols in the binary. We can exclude these from minidebuginfo - # because they are always present in the binary. - dynsyms = set() - for line in subprocess.check_output([nm, '-D', file, '--format=posix', '--defined-only'], universal_newlines=True).splitlines(): - dynsyms.add(line.split()[0]) - - # Find all function symbols from debuginfo which aren't in the dynamic symbols table. - # These are the ones we want to keep in minidebuginfo. - keep_symbols_file = minidebugfile + '.symlist' - found_any_symbols = False - with open(keep_symbols_file, 'w') as f: - for line in subprocess.check_output([nm, debugfile, '--format=sysv', '--defined-only'], universal_newlines=True).splitlines(): - fields = line.split('|') - if len(fields) < 7: - continue - name = fields[0].strip() - type = fields[3].strip() - if type == 'FUNC' and name not in dynsyms: - f.write('{}\n'.format(name)) - found_any_symbols = True - - if not found_any_symbols: - bb.debug(1, 'ELF file {} contains no symbols, skipping minidebuginfo injection'.format(file)) - return - - bb.utils.remove(minidebugfile) - bb.utils.remove(minidebugfile + '.xz') - - subprocess.check_call([objcopy, '-S'] + - ['--remove-section={}'.format(s) for s in remove_section_names] + - ['--keep-symbols={}'.format(keep_symbols_file), debugfile, minidebugfile]) - - subprocess.check_call(['xz', '--keep', minidebugfile]) - - subprocess.check_call([objcopy, '--add-section', '.gnu_debugdata={}.xz'.format(minidebugfile), file]) - -def copydebugsources(debugsrcdir, sources, d): - # The debug src information written out to sourcefile is further processed - # and copied to the destination here. - - import stat - import subprocess - - if debugsrcdir and sources: - sourcefile = d.expand("${WORKDIR}/debugsources.list") - bb.utils.remove(sourcefile) - - # filenames are null-separated - this is an artefact of the previous use - # of rpm's debugedit, which was writing them out that way, and the code elsewhere - # is still assuming that. - debuglistoutput = '\0'.join(sources) + '\0' - with open(sourcefile, 'a') as sf: - sf.write(debuglistoutput) - - dvar = d.getVar('PKGD') - strip = d.getVar("STRIP") - objcopy = d.getVar("OBJCOPY") - workdir = d.getVar("WORKDIR") - sdir = d.getVar("S") - cflags = d.expand("${CFLAGS}") - - prefixmap = {} - for flag in cflags.split(): - if not flag.startswith("-fdebug-prefix-map"): - continue - if "recipe-sysroot" in flag: - continue - flag = flag.split("=") - prefixmap[flag[1]] = flag[2] - - nosuchdir = [] - basepath = dvar - for p in debugsrcdir.split("/"): - basepath = basepath + "/" + p - if not cpath.exists(basepath): - nosuchdir.append(basepath) - bb.utils.mkdirhier(basepath) - cpath.updatecache(basepath) - - for pmap in prefixmap: - # Ignore files from the recipe sysroots (target and native) - cmd = "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '((|)$|/.*recipe-sysroot.*/)' | " % sourcefile - # We need to ignore files that are not actually ours - # we do this by only paying attention to items from this package - cmd += "fgrep -zw '%s' | " % prefixmap[pmap] - # Remove prefix in the source paths - cmd += "sed 's#%s/##g' | " % (prefixmap[pmap]) - cmd += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s' 2>/dev/null)" % (pmap, dvar, prefixmap[pmap]) - - try: - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) - except subprocess.CalledProcessError: - # Can "fail" if internal headers/transient sources are attempted - pass - # cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced. - # Work around this by manually finding and copying any symbolic links that made it through. - cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd '%s' ; cpio -pd0mL --no-preserve-owner '%s%s')" % \ - (dvar, prefixmap[pmap], dvar, prefixmap[pmap], pmap, dvar, prefixmap[pmap]) - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) - - # debugsources.list may be polluted from the host if we used externalsrc, - # cpio uses copy-pass and may have just created a directory structure - # matching the one from the host, if thats the case move those files to - # debugsrcdir to avoid host contamination. - # Empty dir structure will be deleted in the next step. - - # Same check as above for externalsrc - if workdir not in sdir: - if os.path.exists(dvar + debugsrcdir + sdir): - cmd = "mv %s%s%s/* %s%s" % (dvar, debugsrcdir, sdir, dvar,debugsrcdir) - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) - - # The copy by cpio may have resulted in some empty directories! Remove these - cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir) - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) - - # Also remove debugsrcdir if its empty - for p in nosuchdir[::-1]: - if os.path.exists(p) and not os.listdir(p): - os.rmdir(p) - # # Used by do_packagedata (and possibly other routines post do_package) # @@ -656,58 +342,7 @@ python package_convert_pr_autoinc() { LOCALEBASEPN ??= "${PN}" python package_do_split_locales() { - if (d.getVar('PACKAGE_NO_LOCALE') == '1'): - bb.debug(1, "package requested not splitting locales") - return - - packages = (d.getVar('PACKAGES') or "").split() - - datadir = d.getVar('datadir') - if not datadir: - bb.note("datadir not defined") - return - - dvar = d.getVar('PKGD') - pn = d.getVar('LOCALEBASEPN') - - if pn + '-locale' in packages: - packages.remove(pn + '-locale') - - localedir = os.path.join(dvar + datadir, 'locale') - - if not cpath.isdir(localedir): - bb.debug(1, "No locale files in this package") - return - - locales = os.listdir(localedir) - - summary = d.getVar('SUMMARY') or pn - description = d.getVar('DESCRIPTION') or "" - locale_section = d.getVar('LOCALE_SECTION') - mlprefix = d.getVar('MLPREFIX') or "" - for l in sorted(locales): - ln = legitimize_package_name(l) - pkg = pn + '-locale-' + ln - packages.append(pkg) - d.setVar('FILES:' + pkg, os.path.join(datadir, 'locale', l)) - d.setVar('RRECOMMENDS:' + pkg, '%svirtual-locale-%s' % (mlprefix, ln)) - d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln)) - d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l)) - d.setVar('DESCRIPTION:' + pkg, '%s This package contains language translation files for the %s locale.' % (description, l)) - if locale_section: - d.setVar('SECTION:' + pkg, locale_section) - - d.setVar('PACKAGES', ' '.join(packages)) - - # Disabled by RP 18/06/07 - # Wildcards aren't supported in debian - # They break with ipkg since glibc-locale* will mean that - # glibc-localedata-translit* won't install as a dependency - # for some other package which breaks meta-toolchain - # Probably breaks since virtual-locale- isn't provided anywhere - #rdep = (d.getVar('RDEPENDS:%s' % pn) or "").split() - #rdep.append('%s-locale*' % pn) - #d.setVar('RDEPENDS:%s' % pn, ' '.join(rdep)) + oe.package.split_locales(d) } python perform_packagecopy () { @@ -734,488 +369,19 @@ python fixup_perms () { oe.package.fixup_perms(d) } -def package_debug_vars(d): - # We default to '.debug' style - if d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-file-directory': - # Single debug-file-directory style debug info - debug_vars = { - "append": ".debug", - "staticappend": "", - "dir": "", - "staticdir": "", - "libdir": "/usr/lib/debug", - "staticlibdir": "/usr/lib/debug-static", - "srcdir": "/usr/src/debug", - } - elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-without-src': - # Original OE-core, a.k.a. ".debug", style debug info, but without sources in /usr/src/debug - debug_vars = { - "append": "", - "staticappend": "", - "dir": "/.debug", - "staticdir": "/.debug-static", - "libdir": "", - "staticlibdir": "", - "srcdir": "", - } - elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-with-srcpkg': - debug_vars = { - "append": "", - "staticappend": "", - "dir": "/.debug", - "staticdir": "/.debug-static", - "libdir": "", - "staticlibdir": "", - "srcdir": "/usr/src/debug", - } - else: - # Original OE-core, a.k.a. ".debug", style debug info - debug_vars = { - "append": "", - "staticappend": "", - "dir": "/.debug", - "staticdir": "/.debug-static", - "libdir": "", - "staticlibdir": "", - "srcdir": "/usr/src/debug", - } - - return debug_vars - python split_and_strip_files () { - import stat, errno - import subprocess - - dvar = d.getVar('PKGD') - pn = d.getVar('PN') - hostos = d.getVar('HOST_OS') - - oldcwd = os.getcwd() - os.chdir(dvar) - - dv = package_debug_vars(d) - - # - # First lets figure out all of the files we may have to process ... do this only once! - # - elffiles = {} - symlinks = {} - staticlibs = [] - inodes = {} - libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir")) - baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir")) - skipfiles = (d.getVar("INHIBIT_PACKAGE_STRIP_FILES") or "").split() - if (d.getVar('INHIBIT_PACKAGE_STRIP') != '1' or \ - d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT') != '1'): - checkelf = {} - checkelflinks = {} - for root, dirs, files in cpath.walk(dvar): - for f in files: - file = os.path.join(root, f) - - # Skip debug files - if dv["append"] and file.endswith(dv["append"]): - continue - if dv["dir"] and dv["dir"] in os.path.dirname(file[len(dvar):]): - continue - - if file in skipfiles: - continue - - if oe.package.is_static_lib(file): - staticlibs.append(file) - continue - - try: - ltarget = cpath.realpath(file, dvar, False) - s = cpath.lstat(ltarget) - except OSError as e: - (err, strerror) = e.args - if err != errno.ENOENT: - raise - # Skip broken symlinks - continue - if not s: - continue - # Check its an executable - if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) \ - or (s[stat.ST_MODE] & stat.S_IXOTH) \ - or ((file.startswith(libdir) or file.startswith(baselibdir)) \ - and (".so" in f or ".node" in f)) \ - or (f.startswith('vmlinux') or ".ko" in f): - - if cpath.islink(file): - checkelflinks[file] = ltarget - continue - # Use a reference of device ID and inode number to identify files - file_reference = "%d_%d" % (s.st_dev, s.st_ino) - checkelf[file] = (file, file_reference) - - results = oe.utils.multiprocess_launch(oe.package.is_elf, checkelflinks.values(), d) - results_map = {} - for (ltarget, elf_file) in results: - results_map[ltarget] = elf_file - for file in checkelflinks: - ltarget = checkelflinks[file] - # If it's a symlink, and points to an ELF file, we capture the readlink target - if results_map[ltarget]: - target = os.readlink(file) - #bb.note("Sym: %s (%d)" % (ltarget, results_map[ltarget])) - symlinks[file] = target - - results = oe.utils.multiprocess_launch(oe.package.is_elf, checkelf.keys(), d) - - # Sort results by file path. This ensures that the files are always - # processed in the same order, which is important to make sure builds - # are reproducible when dealing with hardlinks - results.sort(key=lambda x: x[0]) - - for (file, elf_file) in results: - # It's a file (or hardlink), not a link - # ...but is it ELF, and is it already stripped? - if elf_file & 1: - if elf_file & 2: - if 'already-stripped' in (d.getVar('INSANE_SKIP:' + pn) or "").split(): - bb.note("Skipping file %s from %s for already-stripped QA test" % (file[len(dvar):], pn)) - else: - msg = "File '%s' from %s was already stripped, this will prevent future debugging!" % (file[len(dvar):], pn) - oe.qa.handle_error("already-stripped", msg, d) - continue - - # At this point we have an unstripped elf file. We need to: - # a) Make sure any file we strip is not hardlinked to anything else outside this tree - # b) Only strip any hardlinked file once (no races) - # c) Track any hardlinks between files so that we can reconstruct matching debug file hardlinks - - # Use a reference of device ID and inode number to identify files - file_reference = checkelf[file][1] - if file_reference in inodes: - os.unlink(file) - os.link(inodes[file_reference][0], file) - inodes[file_reference].append(file) - else: - inodes[file_reference] = [file] - # break hardlink - bb.utils.break_hardlinks(file) - elffiles[file] = elf_file - # Modified the file so clear the cache - cpath.updatecache(file) - - def strip_pkgd_prefix(f): - nonlocal dvar - - if f.startswith(dvar): - return f[len(dvar):] - - return f - - # - # First lets process debug splitting - # - if (d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT') != '1'): - results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, dv, d)) - - if dv["srcdir"] and not hostos.startswith("mingw"): - if (d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): - results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, dv, d)) - else: - for file in staticlibs: - results.append( (file,source_info(file, d)) ) - - d.setVar("PKGDEBUGSOURCES", {strip_pkgd_prefix(f): sorted(s) for f, s in results}) - - sources = set() - for r in results: - sources.update(r[1]) - - # Hardlink our debug symbols to the other hardlink copies - for ref in inodes: - if len(inodes[ref]) == 1: - continue - - target = inodes[ref][0][len(dvar):] - for file in inodes[ref][1:]: - src = file[len(dvar):] - dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(target) + dv["append"] - fpath = dvar + dest - ftarget = dvar + dv["libdir"] + os.path.dirname(target) + dv["dir"] + "/" + os.path.basename(target) + dv["append"] - bb.utils.mkdirhier(os.path.dirname(fpath)) - # Only one hardlink of separated debug info file in each directory - if not os.access(fpath, os.R_OK): - #bb.note("Link %s -> %s" % (fpath, ftarget)) - os.link(ftarget, fpath) - - # Create symlinks for all cases we were able to split symbols - for file in symlinks: - src = file[len(dvar):] - dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"] - fpath = dvar + dest - # Skip it if the target doesn't exist - try: - s = os.stat(fpath) - except OSError as e: - (err, strerror) = e.args - if err != errno.ENOENT: - raise - continue - - ltarget = symlinks[file] - lpath = os.path.dirname(ltarget) - lbase = os.path.basename(ltarget) - ftarget = "" - if lpath and lpath != ".": - ftarget += lpath + dv["dir"] + "/" - ftarget += lbase + dv["append"] - if lpath.startswith(".."): - ftarget = os.path.join("..", ftarget) - bb.utils.mkdirhier(os.path.dirname(fpath)) - #bb.note("Symlink %s -> %s" % (fpath, ftarget)) - os.symlink(ftarget, fpath) - - # Process the dv["srcdir"] if requested... - # This copies and places the referenced sources for later debugging... - copydebugsources(dv["srcdir"], sources, d) - # - # End of debug splitting - # - - # - # Now lets go back over things and strip them - # - if (d.getVar('INHIBIT_PACKAGE_STRIP') != '1'): - strip = d.getVar("STRIP") - sfiles = [] - for file in elffiles: - elf_file = int(elffiles[file]) - #bb.note("Strip %s" % file) - sfiles.append((file, elf_file, strip)) - if (d.getVar('PACKAGE_STRIP_STATIC') == '1' or d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): - for f in staticlibs: - sfiles.append((f, 16, strip)) - - oe.utils.multiprocess_launch(oe.package.runstrip, sfiles, d) - - # Build "minidebuginfo" and reinject it back into the stripped binaries - if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': - oe.utils.multiprocess_launch(inject_minidebuginfo, list(elffiles), d, - extraargs=(dvar, dv, d)) - - # - # End of strip - # - os.chdir(oldcwd) + oe.package.process_split_and_strip_files(d) } python populate_packages () { - import glob, re - - workdir = d.getVar('WORKDIR') - outdir = d.getVar('DEPLOY_DIR') - dvar = d.getVar('PKGD') - packages = d.getVar('PACKAGES').split() - pn = d.getVar('PN') - - bb.utils.mkdirhier(outdir) - os.chdir(dvar) - - autodebug = not (d.getVar("NOAUTOPACKAGEDEBUG") or False) - - split_source_package = (d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-with-srcpkg') - - # If debug-with-srcpkg mode is enabled then add the source package if it - # doesn't exist and add the source file contents to the source package. - if split_source_package: - src_package_name = ('%s-src' % d.getVar('PN')) - if not src_package_name in packages: - packages.append(src_package_name) - d.setVar('FILES:%s' % src_package_name, '/usr/src/debug') - - # Sanity check PACKAGES for duplicates - # Sanity should be moved to sanity.bbclass once we have the infrastructure - package_dict = {} - - for i, pkg in enumerate(packages): - if pkg in package_dict: - msg = "%s is listed in PACKAGES multiple times, this leads to packaging errors." % pkg - oe.qa.handle_error("packages-list", msg, d) - # Ensure the source package gets the chance to pick up the source files - # before the debug package by ordering it first in PACKAGES. Whether it - # actually picks up any source files is controlled by - # PACKAGE_DEBUG_SPLIT_STYLE. - elif pkg.endswith("-src"): - package_dict[pkg] = (10, i) - elif autodebug and pkg.endswith("-dbg"): - package_dict[pkg] = (30, i) - else: - package_dict[pkg] = (50, i) - packages = sorted(package_dict.keys(), key=package_dict.get) - d.setVar('PACKAGES', ' '.join(packages)) - pkgdest = d.getVar('PKGDEST') - - seen = [] - - # os.mkdir masks the permissions with umask so we have to unset it first - oldumask = os.umask(0) - - debug = [] - for root, dirs, files in cpath.walk(dvar): - dir = root[len(dvar):] - if not dir: - dir = os.sep - for f in (files + dirs): - path = "." + os.path.join(dir, f) - if "/.debug/" in path or "/.debug-static/" in path or path.endswith("/.debug"): - debug.append(path) - - for pkg in packages: - root = os.path.join(pkgdest, pkg) - bb.utils.mkdirhier(root) - - filesvar = d.getVar('FILES:%s' % pkg) or "" - if "//" in filesvar: - msg = "FILES variable for package %s contains '//' which is invalid. Attempting to fix this but you should correct the metadata.\n" % pkg - oe.qa.handle_error("files-invalid", msg, d) - filesvar.replace("//", "/") - - origfiles = filesvar.split() - files, symlink_paths = oe.package.files_from_filevars(origfiles) - - if autodebug and pkg.endswith("-dbg"): - files.extend(debug) - - for file in files: - if (not cpath.islink(file)) and (not cpath.exists(file)): - continue - if file in seen: - continue - seen.append(file) - - def mkdir(src, dest, p): - src = os.path.join(src, p) - dest = os.path.join(dest, p) - fstat = cpath.stat(src) - os.mkdir(dest) - os.chmod(dest, fstat.st_mode) - os.chown(dest, fstat.st_uid, fstat.st_gid) - if p not in seen: - seen.append(p) - cpath.updatecache(dest) - - def mkdir_recurse(src, dest, paths): - if cpath.exists(dest + '/' + paths): - return - while paths.startswith("./"): - paths = paths[2:] - p = "." - for c in paths.split("/"): - p = os.path.join(p, c) - if not cpath.exists(os.path.join(dest, p)): - mkdir(src, dest, p) - - if cpath.isdir(file) and not cpath.islink(file): - mkdir_recurse(dvar, root, file) - continue - - mkdir_recurse(dvar, root, os.path.dirname(file)) - fpath = os.path.join(root,file) - if not cpath.islink(file): - os.link(file, fpath) - continue - ret = bb.utils.copyfile(file, fpath) - if ret is False or ret == 0: - bb.fatal("File population failed") - - # Check if symlink paths exist - for file in symlink_paths: - if not os.path.exists(os.path.join(root,file)): - bb.fatal("File '%s' cannot be packaged into '%s' because its " - "parent directory structure does not exist. One of " - "its parent directories is a symlink whose target " - "directory is not included in the package." % - (file, pkg)) - - os.umask(oldumask) - os.chdir(workdir) - - # Handle excluding packages with incompatible licenses - package_list = [] - for pkg in packages: - licenses = d.getVar('_exclude_incompatible-' + pkg) - if licenses: - msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses) - oe.qa.handle_error("incompatible-license", msg, d) - else: - package_list.append(pkg) - d.setVar('PACKAGES', ' '.join(package_list)) - - unshipped = [] - for root, dirs, files in cpath.walk(dvar): - dir = root[len(dvar):] - if not dir: - dir = os.sep - for f in (files + dirs): - path = os.path.join(dir, f) - if ('.' + path) not in seen: - unshipped.append(path) - - if unshipped != []: - msg = pn + ": Files/directories were installed but not shipped in any package:" - if "installed-vs-shipped" in (d.getVar('INSANE_SKIP:' + pn) or "").split(): - bb.note("Package %s skipping QA tests: installed-vs-shipped" % pn) - else: - for f in unshipped: - msg = msg + "\n " + f - msg = msg + "\nPlease set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.\n" - msg = msg + "%s: %d installed and not shipped files." % (pn, len(unshipped)) - oe.qa.handle_error("installed-vs-shipped", msg, d) + oe.package.populate_packages(d) } populate_packages[dirs] = "${D}" python package_fixsymlinks () { - import errno - pkgdest = d.getVar('PKGDEST') - packages = d.getVar("PACKAGES", False).split() - - dangling_links = {} - pkg_files = {} - for pkg in packages: - dangling_links[pkg] = [] - pkg_files[pkg] = [] - inst_root = os.path.join(pkgdest, pkg) - for path in pkgfiles[pkg]: - rpath = path[len(inst_root):] - pkg_files[pkg].append(rpath) - rtarget = cpath.realpath(path, inst_root, True, assume_dir = True) - if not cpath.lexists(rtarget): - dangling_links[pkg].append(os.path.normpath(rtarget[len(inst_root):])) - - newrdepends = {} - for pkg in dangling_links: - for l in dangling_links[pkg]: - found = False - bb.debug(1, "%s contains dangling link %s" % (pkg, l)) - for p in packages: - if l in pkg_files[p]: - found = True - bb.debug(1, "target found in %s" % p) - if p == pkg: - break - if pkg not in newrdepends: - newrdepends[pkg] = [] - newrdepends[pkg].append(p) - break - if found == False: - bb.note("%s contains dangling symlink to %s" % (pkg, l)) - - for pkg in newrdepends: - rdepends = bb.utils.explode_dep_versions2(d.getVar('RDEPENDS:' + pkg) or "") - for p in newrdepends[pkg]: - if p not in rdepends: - rdepends[p] = [] - d.setVar('RDEPENDS:' + pkg, bb.utils.join_deps(rdepends, commasep=False)) + oe.package.process_fixsymlinks(pkgfiles, d) } - python package_package_name_hook() { """ A package_name_hook function can be used to rewrite the package names by @@ -1245,456 +411,23 @@ fi RPMDEPS = "${STAGING_LIBDIR_NATIVE}/rpm/rpmdeps --alldeps --define '__font_provides %{nil}'" -# Collect perfile run-time dependency metadata -# Output: -# FILERPROVIDESFLIST:pkg - list of all files w/ deps -# FILERPROVIDES:filepath:pkg - per file dep -# -# FILERDEPENDSFLIST:pkg - list of all files w/ deps -# FILERDEPENDS:filepath:pkg - per file dep - python package_do_filedeps() { - if d.getVar('SKIP_FILEDEPS') == '1': - return - - pkgdest = d.getVar('PKGDEST') - packages = d.getVar('PACKAGES') - rpmdeps = d.getVar('RPMDEPS') - - def chunks(files, n): - return [files[i:i+n] for i in range(0, len(files), n)] - - pkglist = [] - for pkg in packages.split(): - if d.getVar('SKIP_FILEDEPS:' + pkg) == '1': - continue - if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-') or pkg.endswith('-src'): - continue - for files in chunks(pkgfiles[pkg], 100): - pkglist.append((pkg, files, rpmdeps, pkgdest)) - - processed = oe.utils.multiprocess_launch(oe.package.filedeprunner, pkglist, d) - - provides_files = {} - requires_files = {} - - for result in processed: - (pkg, provides, requires) = result - - if pkg not in provides_files: - provides_files[pkg] = [] - if pkg not in requires_files: - requires_files[pkg] = [] - - for file in sorted(provides): - provides_files[pkg].append(file) - key = "FILERPROVIDES:" + file + ":" + pkg - d.appendVar(key, " " + " ".join(provides[file])) - - for file in sorted(requires): - requires_files[pkg].append(file) - key = "FILERDEPENDS:" + file + ":" + pkg - d.appendVar(key, " " + " ".join(requires[file])) - - for pkg in requires_files: - d.setVar("FILERDEPENDSFLIST:" + pkg, " ".join(sorted(requires_files[pkg]))) - for pkg in provides_files: - d.setVar("FILERPROVIDESFLIST:" + pkg, " ".join(sorted(provides_files[pkg]))) + oe.package.process_filedeps(pkgfiles, d) } SHLIBSDIRS = "${WORKDIR_PKGDATA}/${MLPREFIX}shlibs2" SHLIBSWORKDIR = "${PKGDESTWORK}/${MLPREFIX}shlibs2" python package_do_shlibs() { - import itertools - import re, pipes - import subprocess - - exclude_shlibs = d.getVar('EXCLUDE_FROM_SHLIBS', False) - if exclude_shlibs: - bb.note("not generating shlibs") - return - - lib_re = re.compile(r"^.*\.so") - libdir_re = re.compile(r".*/%s$" % d.getVar('baselib')) - - packages = d.getVar('PACKAGES') - - shlib_pkgs = [] - exclusion_list = d.getVar("EXCLUDE_PACKAGES_FROM_SHLIBS") - if exclusion_list: - for pkg in packages.split(): - if pkg not in exclusion_list.split(): - shlib_pkgs.append(pkg) - else: - bb.note("not generating shlibs for %s" % pkg) - else: - shlib_pkgs = packages.split() - - hostos = d.getVar('HOST_OS') - - workdir = d.getVar('WORKDIR') - - ver = d.getVar('PKGV') - if not ver: - msg = "PKGV not defined" - oe.qa.handle_error("pkgv-undefined", msg, d) - return - - pkgdest = d.getVar('PKGDEST') - - shlibswork_dir = d.getVar('SHLIBSWORKDIR') - - def linux_so(file, pkg, pkgver, d): - needs_ldconfig = False - needed = set() - sonames = set() - renames = [] - ldir = os.path.dirname(file).replace(pkgdest + "/" + pkg, '') - cmd = d.getVar('OBJDUMP') + " -p " + pipes.quote(file) + " 2>/dev/null" - fd = os.popen(cmd) - lines = fd.readlines() - fd.close() - rpath = tuple() - for l in lines: - m = re.match(r"\s+RPATH\s+([^\s]*)", l) - if m: - rpaths = m.group(1).replace("$ORIGIN", ldir).split(":") - rpath = tuple(map(os.path.normpath, rpaths)) - for l in lines: - m = re.match(r"\s+NEEDED\s+([^\s]*)", l) - if m: - dep = m.group(1) - if dep not in needed: - needed.add((dep, file, rpath)) - m = re.match(r"\s+SONAME\s+([^\s]*)", l) - if m: - this_soname = m.group(1) - prov = (this_soname, ldir, pkgver) - if not prov in sonames: - # if library is private (only used by package) then do not build shlib for it - import fnmatch - if not private_libs or len([i for i in private_libs if fnmatch.fnmatch(this_soname, i)]) == 0: - sonames.add(prov) - if libdir_re.match(os.path.dirname(file)): - needs_ldconfig = True - if needs_ldconfig and snap_symlinks and (os.path.basename(file) != this_soname): - renames.append((file, os.path.join(os.path.dirname(file), this_soname))) - return (needs_ldconfig, needed, sonames, renames) - - def darwin_so(file, needed, sonames, renames, pkgver): - if not os.path.exists(file): - return - ldir = os.path.dirname(file).replace(pkgdest + "/" + pkg, '') - - def get_combinations(base): - # - # Given a base library name, find all combinations of this split by "." and "-" - # - combos = [] - options = base.split(".") - for i in range(1, len(options) + 1): - combos.append(".".join(options[0:i])) - options = base.split("-") - for i in range(1, len(options) + 1): - combos.append("-".join(options[0:i])) - return combos - - if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg') and not pkg.endswith('-src'): - # Drop suffix - name = os.path.basename(file).rsplit(".",1)[0] - # Find all combinations - combos = get_combinations(name) - for combo in combos: - if not combo in sonames: - prov = (combo, ldir, pkgver) - sonames.add(prov) - if file.endswith('.dylib') or file.endswith('.so'): - rpath = [] - p = subprocess.Popen([d.expand("${HOST_PREFIX}otool"), '-l', file], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - out, err = p.communicate() - # If returned successfully, process stdout for results - if p.returncode == 0: - for l in out.split("\n"): - l = l.strip() - if l.startswith('path '): - rpath.append(l.split()[1]) - - p = subprocess.Popen([d.expand("${HOST_PREFIX}otool"), '-L', file], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - out, err = p.communicate() - # If returned successfully, process stdout for results - if p.returncode == 0: - for l in out.split("\n"): - l = l.strip() - if not l or l.endswith(":"): - continue - if "is not an object file" in l: - continue - name = os.path.basename(l.split()[0]).rsplit(".", 1)[0] - if name and name not in needed[pkg]: - needed[pkg].add((name, file, tuple())) - - def mingw_dll(file, needed, sonames, renames, pkgver): - if not os.path.exists(file): - return - - if file.endswith(".dll"): - # assume all dlls are shared objects provided by the package - sonames.add((os.path.basename(file), os.path.dirname(file).replace(pkgdest + "/" + pkg, ''), pkgver)) - - if (file.endswith(".dll") or file.endswith(".exe")): - # use objdump to search for "DLL Name: .*\.dll" - p = subprocess.Popen([d.expand("${HOST_PREFIX}objdump"), "-p", file], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - out, err = p.communicate() - # process the output, grabbing all .dll names - if p.returncode == 0: - for m in re.finditer(r"DLL Name: (.*?\.dll)$", out.decode(), re.MULTILINE | re.IGNORECASE): - dllname = m.group(1) - if dllname: - needed[pkg].add((dllname, file, tuple())) - - if d.getVar('PACKAGE_SNAP_LIB_SYMLINKS') == "1": - snap_symlinks = True - else: - snap_symlinks = False - - needed = {} - - shlib_provider = oe.package.read_shlib_providers(d) - - for pkg in shlib_pkgs: - private_libs = d.getVar('PRIVATE_LIBS:' + pkg) or d.getVar('PRIVATE_LIBS') or "" - private_libs = private_libs.split() - needs_ldconfig = False - bb.debug(2, "calculating shlib provides for %s" % pkg) - - pkgver = d.getVar('PKGV:' + pkg) - if not pkgver: - pkgver = d.getVar('PV_' + pkg) - if not pkgver: - pkgver = ver - - needed[pkg] = set() - sonames = set() - renames = [] - linuxlist = [] - for file in pkgfiles[pkg]: - soname = None - if cpath.islink(file): - continue - if hostos == "darwin" or hostos == "darwin8": - darwin_so(file, needed, sonames, renames, pkgver) - elif hostos.startswith("mingw"): - mingw_dll(file, needed, sonames, renames, pkgver) - elif os.access(file, os.X_OK) or lib_re.match(file): - linuxlist.append(file) - - if linuxlist: - results = oe.utils.multiprocess_launch(linux_so, linuxlist, d, extraargs=(pkg, pkgver, d)) - for r in results: - ldconfig = r[0] - needed[pkg] |= r[1] - sonames |= r[2] - renames.extend(r[3]) - needs_ldconfig = needs_ldconfig or ldconfig - - for (old, new) in renames: - bb.note("Renaming %s to %s" % (old, new)) - bb.utils.rename(old, new) - pkgfiles[pkg].remove(old) - - shlibs_file = os.path.join(shlibswork_dir, pkg + ".list") - if len(sonames): - with open(shlibs_file, 'w') as fd: - for s in sorted(sonames): - if s[0] in shlib_provider and s[1] in shlib_provider[s[0]]: - (old_pkg, old_pkgver) = shlib_provider[s[0]][s[1]] - if old_pkg != pkg: - bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s[0], pkg, pkgver)) - bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s[0])) - fd.write(s[0] + ':' + s[1] + ':' + s[2] + '\n') - if s[0] not in shlib_provider: - shlib_provider[s[0]] = {} - shlib_provider[s[0]][s[1]] = (pkg, pkgver) - if needs_ldconfig: - bb.debug(1, 'adding ldconfig call to postinst for %s' % pkg) - postinst = d.getVar('pkg_postinst:%s' % pkg) - if not postinst: - postinst = '#!/bin/sh\n' - postinst += d.getVar('ldconfig_postinst_fragment') - d.setVar('pkg_postinst:%s' % pkg, postinst) - bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames)) - - assumed_libs = d.getVar('ASSUME_SHLIBS') - if assumed_libs: - libdir = d.getVar("libdir") - for e in assumed_libs.split(): - l, dep_pkg = e.split(":") - lib_ver = None - dep_pkg = dep_pkg.rsplit("_", 1) - if len(dep_pkg) == 2: - lib_ver = dep_pkg[1] - dep_pkg = dep_pkg[0] - if l not in shlib_provider: - shlib_provider[l] = {} - shlib_provider[l][libdir] = (dep_pkg, lib_ver) - - libsearchpath = [d.getVar('libdir'), d.getVar('base_libdir')] - - for pkg in shlib_pkgs: - bb.debug(2, "calculating shlib requirements for %s" % pkg) - - private_libs = d.getVar('PRIVATE_LIBS:' + pkg) or d.getVar('PRIVATE_LIBS') or "" - private_libs = private_libs.split() - - deps = list() - for n in needed[pkg]: - # if n is in private libraries, don't try to search provider for it - # this could cause problem in case some abc.bb provides private - # /opt/abc/lib/libfoo.so.1 and contains /usr/bin/abc depending on system library libfoo.so.1 - # but skipping it is still better alternative than providing own - # version and then adding runtime dependency for the same system library - import fnmatch - if private_libs and len([i for i in private_libs if fnmatch.fnmatch(n[0], i)]) > 0: - bb.debug(2, '%s: Dependency %s covered by PRIVATE_LIBS' % (pkg, n[0])) - continue - if n[0] in shlib_provider.keys(): - shlib_provider_map = shlib_provider[n[0]] - matches = set() - for p in itertools.chain(list(n[2]), sorted(shlib_provider_map.keys()), libsearchpath): - if p in shlib_provider_map: - matches.add(p) - if len(matches) > 1: - matchpkgs = ', '.join([shlib_provider_map[match][0] for match in matches]) - bb.error("%s: Multiple shlib providers for %s: %s (used by files: %s)" % (pkg, n[0], matchpkgs, n[1])) - elif len(matches) == 1: - (dep_pkg, ver_needed) = shlib_provider_map[matches.pop()] - - bb.debug(2, '%s: Dependency %s requires package %s (used by files: %s)' % (pkg, n[0], dep_pkg, n[1])) - - if dep_pkg == pkg: - continue - - if ver_needed: - dep = "%s (>= %s)" % (dep_pkg, ver_needed) - else: - dep = dep_pkg - if not dep in deps: - deps.append(dep) - continue - bb.note("Couldn't find shared library provider for %s, used by files: %s" % (n[0], n[1])) - - deps_file = os.path.join(pkgdest, pkg + ".shlibdeps") - if os.path.exists(deps_file): - os.remove(deps_file) - if deps: - with open(deps_file, 'w') as fd: - for dep in sorted(deps): - fd.write(dep + '\n') + oe.package.process_shlibs(pkgfiles, d) } python package_do_pkgconfig () { - import re - - packages = d.getVar('PACKAGES') - workdir = d.getVar('WORKDIR') - pkgdest = d.getVar('PKGDEST') - - shlibs_dirs = d.getVar('SHLIBSDIRS').split() - shlibswork_dir = d.getVar('SHLIBSWORKDIR') - - pc_re = re.compile(r'(.*)\.pc$') - var_re = re.compile(r'(.*)=(.*)') - field_re = re.compile(r'(.*): (.*)') - - pkgconfig_provided = {} - pkgconfig_needed = {} - for pkg in packages.split(): - pkgconfig_provided[pkg] = [] - pkgconfig_needed[pkg] = [] - for file in sorted(pkgfiles[pkg]): - m = pc_re.match(file) - if m: - pd = bb.data.init() - name = m.group(1) - pkgconfig_provided[pkg].append(os.path.basename(name)) - if not os.access(file, os.R_OK): - continue - with open(file, 'r') as f: - lines = f.readlines() - for l in lines: - m = var_re.match(l) - if m: - name = m.group(1) - val = m.group(2) - pd.setVar(name, pd.expand(val)) - continue - m = field_re.match(l) - if m: - hdr = m.group(1) - exp = pd.expand(m.group(2)) - if hdr == 'Requires': - pkgconfig_needed[pkg] += exp.replace(',', ' ').split() - - for pkg in packages.split(): - pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist") - if pkgconfig_provided[pkg] != []: - with open(pkgs_file, 'w') as f: - for p in sorted(pkgconfig_provided[pkg]): - f.write('%s\n' % p) - - # Go from least to most specific since the last one found wins - for dir in reversed(shlibs_dirs): - if not os.path.exists(dir): - continue - for file in sorted(os.listdir(dir)): - m = re.match(r'^(.*)\.pclist$', file) - if m: - pkg = m.group(1) - with open(os.path.join(dir, file)) as fd: - lines = fd.readlines() - pkgconfig_provided[pkg] = [] - for l in lines: - pkgconfig_provided[pkg].append(l.rstrip()) - - for pkg in packages.split(): - deps = [] - for n in pkgconfig_needed[pkg]: - found = False - for k in pkgconfig_provided.keys(): - if n in pkgconfig_provided[k]: - if k != pkg and not (k in deps): - deps.append(k) - found = True - if found == False: - bb.note("couldn't find pkgconfig module '%s' in any package" % n) - deps_file = os.path.join(pkgdest, pkg + ".pcdeps") - if len(deps): - with open(deps_file, 'w') as fd: - for dep in deps: - fd.write(dep + '\n') + oe.package.process_pkgconfig(pkgfiles, d) } -def read_libdep_files(d): - pkglibdeps = {} - packages = d.getVar('PACKAGES').split() - for pkg in packages: - pkglibdeps[pkg] = {} - for extension in ".shlibdeps", ".pcdeps", ".clilibdeps": - depsfile = d.expand("${PKGDEST}/" + pkg + extension) - if os.access(depsfile, os.R_OK): - with open(depsfile) as fd: - lines = fd.readlines() - for l in lines: - l.rstrip() - deps = bb.utils.explode_dep_versions2(l) - for dep in deps: - if not dep in pkglibdeps[pkg]: - pkglibdeps[pkg][dep] = deps[dep] - return pkglibdeps - python read_shlibdeps () { - pkglibdeps = read_libdep_files(d) + pkglibdeps = oe.package.read_libdep_files(d) packages = d.getVar('PACKAGES').split() for pkg in packages: @@ -1710,125 +443,7 @@ python read_shlibdeps () { } python package_depchains() { - """ - For a given set of prefix and postfix modifiers, make those packages - RRECOMMENDS on the corresponding packages for its RDEPENDS. - - Example: If package A depends upon package B, and A's .bb emits an - A-dev package, this would make A-dev Recommends: B-dev. - - If only one of a given suffix is specified, it will take the RRECOMMENDS - based on the RDEPENDS of *all* other packages. If more than one of a given - suffix is specified, its will only use the RDEPENDS of the single parent - package. - """ - - packages = d.getVar('PACKAGES') - postfixes = (d.getVar('DEPCHAIN_POST') or '').split() - prefixes = (d.getVar('DEPCHAIN_PRE') or '').split() - - def pkg_adddeprrecs(pkg, base, suffix, getname, depends, d): - - #bb.note('depends for %s is %s' % (base, depends)) - rreclist = bb.utils.explode_dep_versions2(d.getVar('RRECOMMENDS:' + pkg) or "") - - for depend in sorted(depends): - if depend.find('-native') != -1 or depend.find('-cross') != -1 or depend.startswith('virtual/'): - #bb.note("Skipping %s" % depend) - continue - if depend.endswith('-dev'): - depend = depend[:-4] - if depend.endswith('-dbg'): - depend = depend[:-4] - pkgname = getname(depend, suffix) - #bb.note("Adding %s for %s" % (pkgname, depend)) - if pkgname not in rreclist and pkgname != pkg: - rreclist[pkgname] = [] - - #bb.note('setting: RRECOMMENDS:%s=%s' % (pkg, ' '.join(rreclist))) - d.setVar('RRECOMMENDS:%s' % pkg, bb.utils.join_deps(rreclist, commasep=False)) - - def pkg_addrrecs(pkg, base, suffix, getname, rdepends, d): - - #bb.note('rdepends for %s is %s' % (base, rdepends)) - rreclist = bb.utils.explode_dep_versions2(d.getVar('RRECOMMENDS:' + pkg) or "") - - for depend in sorted(rdepends): - if depend.find('virtual-locale-') != -1: - #bb.note("Skipping %s" % depend) - continue - if depend.endswith('-dev'): - depend = depend[:-4] - if depend.endswith('-dbg'): - depend = depend[:-4] - pkgname = getname(depend, suffix) - #bb.note("Adding %s for %s" % (pkgname, depend)) - if pkgname not in rreclist and pkgname != pkg: - rreclist[pkgname] = [] - - #bb.note('setting: RRECOMMENDS:%s=%s' % (pkg, ' '.join(rreclist))) - d.setVar('RRECOMMENDS:%s' % pkg, bb.utils.join_deps(rreclist, commasep=False)) - - def add_dep(list, dep): - if dep not in list: - list.append(dep) - - depends = [] - for dep in bb.utils.explode_deps(d.getVar('DEPENDS') or ""): - add_dep(depends, dep) - - rdepends = [] - for pkg in packages.split(): - for dep in bb.utils.explode_deps(d.getVar('RDEPENDS:' + pkg) or ""): - add_dep(rdepends, dep) - - #bb.note('rdepends is %s' % rdepends) - - def post_getname(name, suffix): - return '%s%s' % (name, suffix) - def pre_getname(name, suffix): - return '%s%s' % (suffix, name) - - pkgs = {} - for pkg in packages.split(): - for postfix in postfixes: - if pkg.endswith(postfix): - if not postfix in pkgs: - pkgs[postfix] = {} - pkgs[postfix][pkg] = (pkg[:-len(postfix)], post_getname) - - for prefix in prefixes: - if pkg.startswith(prefix): - if not prefix in pkgs: - pkgs[prefix] = {} - pkgs[prefix][pkg] = (pkg[:-len(prefix)], pre_getname) - - if "-dbg" in pkgs: - pkglibdeps = read_libdep_files(d) - pkglibdeplist = [] - for pkg in pkglibdeps: - for k in pkglibdeps[pkg]: - add_dep(pkglibdeplist, k) - dbgdefaultdeps = ((d.getVar('DEPCHAIN_DBGDEFAULTDEPS') == '1') or (bb.data.inherits_class('packagegroup', d))) - - for suffix in pkgs: - for pkg in pkgs[suffix]: - if d.getVarFlag('RRECOMMENDS:' + pkg, 'nodeprrecs'): - continue - (base, func) = pkgs[suffix][pkg] - if suffix == "-dev": - pkg_adddeprrecs(pkg, base, suffix, func, depends, d) - elif suffix == "-dbg": - if not dbgdefaultdeps: - pkg_addrrecs(pkg, base, suffix, func, pkglibdeplist, d) - continue - if len(pkgs[suffix]) == 1: - pkg_addrrecs(pkg, base, suffix, func, rdepends, d) - else: - rdeps = [] - for dep in bb.utils.explode_deps(d.getVar('RDEPENDS:' + base) or ""): - add_dep(rdeps, dep) - pkg_addrrecs(pkg, base, suffix, func, rdeps, d) + oe.package.process_depchains(pkgfiles, d) } # Since bitbake can't determine which variables are accessed during package diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index b4c8ab7222a..c9eb75d8523 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -4,12 +4,19 @@ # SPDX-License-Identifier: GPL-2.0-only # +import errno +import fnmatch +import itertools import os +import pipes +import re import glob import stat import mmap import subprocess +import oe.cachedpath + def runstrip(arg): # Function to strip a single file, called from split_and_strip_files below # A working 'file' (one which works on the target architecture) @@ -300,7 +307,6 @@ def read_shlib_providers(d): # the fs-perms.txt files def fixup_perms(d): import pwd, grp - import oe.cachedpath cpath = oe.cachedpath.CachedPath() dvar = d.getVar('PKGD') @@ -537,8 +543,6 @@ def fixup_perms(d): # Get a list of files from file vars by searching files under current working directory # The list contains symlinks, directories and normal files. def files_from_filevars(filevars): - import oe.cachedpath - cpath = oe.cachedpath.CachedPath() files = [] for f in filevars: @@ -611,3 +615,1401 @@ def get_conffiles(pkg, d): os.chdir(cwd) return conf_list +def legitimize_package_name(s): + """ + Make sure package names are legitimate strings + """ + + def fixutf(m): + cp = m.group(1) + if cp: + return ('\\u%s' % cp).encode('latin-1').decode('unicode_escape') + + # Handle unicode codepoints encoded as , as in glibc locale files. + s = re.sub(r'', fixutf, s) + + # Remaining package name validity fixes + return s.lower().replace('_', '-').replace('@', '+').replace(',', '+').replace('/', '-') + +def split_locales(d): + cpath = oe.cachedpath.CachedPath() + if (d.getVar('PACKAGE_NO_LOCALE') == '1'): + bb.debug(1, "package requested not splitting locales") + return + + packages = (d.getVar('PACKAGES') or "").split() + + datadir = d.getVar('datadir') + if not datadir: + bb.note("datadir not defined") + return + + dvar = d.getVar('PKGD') + pn = d.getVar('LOCALEBASEPN') + + if pn + '-locale' in packages: + packages.remove(pn + '-locale') + + localedir = os.path.join(dvar + datadir, 'locale') + + if not cpath.isdir(localedir): + bb.debug(1, "No locale files in this package") + return + + locales = os.listdir(localedir) + + summary = d.getVar('SUMMARY') or pn + description = d.getVar('DESCRIPTION') or "" + locale_section = d.getVar('LOCALE_SECTION') + mlprefix = d.getVar('MLPREFIX') or "" + for l in sorted(locales): + ln = legitimize_package_name(l) + pkg = pn + '-locale-' + ln + packages.append(pkg) + d.setVar('FILES:' + pkg, os.path.join(datadir, 'locale', l)) + d.setVar('RRECOMMENDS:' + pkg, '%svirtual-locale-%s' % (mlprefix, ln)) + d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln)) + d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l)) + d.setVar('DESCRIPTION:' + pkg, '%s This package contains language translation files for the %s locale.' % (description, l)) + if locale_section: + d.setVar('SECTION:' + pkg, locale_section) + + d.setVar('PACKAGES', ' '.join(packages)) + + # Disabled by RP 18/06/07 + # Wildcards aren't supported in debian + # They break with ipkg since glibc-locale* will mean that + # glibc-localedata-translit* won't install as a dependency + # for some other package which breaks meta-toolchain + # Probably breaks since virtual-locale- isn't provided anywhere + #rdep = (d.getVar('RDEPENDS:%s' % pn) or "").split() + #rdep.append('%s-locale*' % pn) + #d.setVar('RDEPENDS:%s' % pn, ' '.join(rdep)) + +def package_debug_vars(d): + # We default to '.debug' style + if d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-file-directory': + # Single debug-file-directory style debug info + debug_vars = { + "append": ".debug", + "staticappend": "", + "dir": "", + "staticdir": "", + "libdir": "/usr/lib/debug", + "staticlibdir": "/usr/lib/debug-static", + "srcdir": "/usr/src/debug", + } + elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-without-src': + # Original OE-core, a.k.a. ".debug", style debug info, but without sources in /usr/src/debug + debug_vars = { + "append": "", + "staticappend": "", + "dir": "/.debug", + "staticdir": "/.debug-static", + "libdir": "", + "staticlibdir": "", + "srcdir": "", + } + elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-with-srcpkg': + debug_vars = { + "append": "", + "staticappend": "", + "dir": "/.debug", + "staticdir": "/.debug-static", + "libdir": "", + "staticlibdir": "", + "srcdir": "/usr/src/debug", + } + else: + # Original OE-core, a.k.a. ".debug", style debug info + debug_vars = { + "append": "", + "staticappend": "", + "dir": "/.debug", + "staticdir": "/.debug-static", + "libdir": "", + "staticlibdir": "", + "srcdir": "/usr/src/debug", + } + + return debug_vars + + +def parse_debugsources_from_dwarfsrcfiles_output(dwarfsrcfiles_output): + debugfiles = {} + + for line in dwarfsrcfiles_output.splitlines(): + if line.startswith("\t"): + debugfiles[os.path.normpath(line.split()[0])] = "" + + return debugfiles.keys() + +def source_info(file, d, fatal=True): + cmd = ["dwarfsrcfiles", file] + try: + output = subprocess.check_output(cmd, universal_newlines=True, stderr=subprocess.STDOUT) + retval = 0 + except subprocess.CalledProcessError as exc: + output = exc.output + retval = exc.returncode + + # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure + if retval != 0 and retval != 255: + msg = "dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else "") + if fatal: + bb.fatal(msg) + bb.note(msg) + + debugsources = parse_debugsources_from_dwarfsrcfiles_output(output) + + return list(debugsources) + +def splitdebuginfo(file, dvar, dv, d): + # Function to split a single file into two components, one is the stripped + # target system binary, the other contains any debugging information. The + # two files are linked to reference each other. + # + # return a mapping of files:debugsources + + src = file[len(dvar):] + dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"] + debugfile = dvar + dest + sources = [] + + if file.endswith(".ko") and file.find("/lib/modules/") != -1: + if oe.package.is_kernel_module_signed(file): + bb.debug(1, "Skip strip on signed module %s" % file) + return (file, sources) + + # Split the file... + bb.utils.mkdirhier(os.path.dirname(debugfile)) + #bb.note("Split %s -> %s" % (file, debugfile)) + # Only store off the hard link reference if we successfully split! + + dvar = d.getVar('PKGD') + objcopy = d.getVar("OBJCOPY") + + newmode = None + if not os.access(file, os.W_OK) or os.access(file, os.R_OK): + origmode = os.stat(file)[stat.ST_MODE] + newmode = origmode | stat.S_IWRITE | stat.S_IREAD + os.chmod(file, newmode) + + # We need to extract the debug src information here... + if dv["srcdir"]: + sources = source_info(file, d) + + bb.utils.mkdirhier(os.path.dirname(debugfile)) + + subprocess.check_output([objcopy, '--only-keep-debug', file, debugfile], stderr=subprocess.STDOUT) + + # Set the debuglink to have the view of the file path on the target + subprocess.check_output([objcopy, '--add-gnu-debuglink', debugfile, file], stderr=subprocess.STDOUT) + + if newmode: + os.chmod(file, origmode) + + return (file, sources) + +def splitstaticdebuginfo(file, dvar, dv, d): + # Unlike the function above, there is no way to split a static library + # two components. So to get similar results we will copy the unmodified + # static library (containing the debug symbols) into a new directory. + # We will then strip (preserving symbols) the static library in the + # typical location. + # + # return a mapping of files:debugsources + + src = file[len(dvar):] + dest = dv["staticlibdir"] + os.path.dirname(src) + dv["staticdir"] + "/" + os.path.basename(src) + dv["staticappend"] + debugfile = dvar + dest + sources = [] + + # Copy the file... + bb.utils.mkdirhier(os.path.dirname(debugfile)) + #bb.note("Copy %s -> %s" % (file, debugfile)) + + dvar = d.getVar('PKGD') + + newmode = None + if not os.access(file, os.W_OK) or os.access(file, os.R_OK): + origmode = os.stat(file)[stat.ST_MODE] + newmode = origmode | stat.S_IWRITE | stat.S_IREAD + os.chmod(file, newmode) + + # We need to extract the debug src information here... + if dv["srcdir"]: + sources = source_info(file, d) + + bb.utils.mkdirhier(os.path.dirname(debugfile)) + + # Copy the unmodified item to the debug directory + shutil.copy2(file, debugfile) + + if newmode: + os.chmod(file, origmode) + + return (file, sources) + +def inject_minidebuginfo(file, dvar, dv, d): + # Extract just the symbols from debuginfo into minidebuginfo, + # compress it with xz and inject it back into the binary in a .gnu_debugdata section. + # https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html + + readelf = d.getVar('READELF') + nm = d.getVar('NM') + objcopy = d.getVar('OBJCOPY') + + minidebuginfodir = d.expand('${WORKDIR}/minidebuginfo') + + src = file[len(dvar):] + dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"] + debugfile = dvar + dest + minidebugfile = minidebuginfodir + src + '.minidebug' + bb.utils.mkdirhier(os.path.dirname(minidebugfile)) + + # If we didn't produce debuginfo for any reason, we can't produce minidebuginfo either + # so skip it. + if not os.path.exists(debugfile): + bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file)) + return + + # minidebuginfo does not make sense to apply to ELF objects other than + # executables and shared libraries, skip applying the minidebuginfo + # generation for objects like kernel modules. + for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines(): + if not line.strip().startswith("Type:"): + continue + elftype = line.split(":")[1].strip() + if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]): + bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file)) + return + break + + # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo. + # We will exclude all of these from minidebuginfo to save space. + remove_section_names = [] + for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines(): + # strip the leading " [ 1]" section index to allow splitting on space + if ']' not in line: + continue + fields = line[line.index(']') + 1:].split() + if len(fields) < 7: + continue + name = fields[0] + type = fields[1] + flags = fields[6] + # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them + if name.startswith('.debug_'): + continue + if 'A' not in flags and type in ['PROGBITS', 'NOTE', 'NOBITS']: + remove_section_names.append(name) + + # List dynamic symbols in the binary. We can exclude these from minidebuginfo + # because they are always present in the binary. + dynsyms = set() + for line in subprocess.check_output([nm, '-D', file, '--format=posix', '--defined-only'], universal_newlines=True).splitlines(): + dynsyms.add(line.split()[0]) + + # Find all function symbols from debuginfo which aren't in the dynamic symbols table. + # These are the ones we want to keep in minidebuginfo. + keep_symbols_file = minidebugfile + '.symlist' + found_any_symbols = False + with open(keep_symbols_file, 'w') as f: + for line in subprocess.check_output([nm, debugfile, '--format=sysv', '--defined-only'], universal_newlines=True).splitlines(): + fields = line.split('|') + if len(fields) < 7: + continue + name = fields[0].strip() + type = fields[3].strip() + if type == 'FUNC' and name not in dynsyms: + f.write('{}\n'.format(name)) + found_any_symbols = True + + if not found_any_symbols: + bb.debug(1, 'ELF file {} contains no symbols, skipping minidebuginfo injection'.format(file)) + return + + bb.utils.remove(minidebugfile) + bb.utils.remove(minidebugfile + '.xz') + + subprocess.check_call([objcopy, '-S'] + + ['--remove-section={}'.format(s) for s in remove_section_names] + + ['--keep-symbols={}'.format(keep_symbols_file), debugfile, minidebugfile]) + + subprocess.check_call(['xz', '--keep', minidebugfile]) + + subprocess.check_call([objcopy, '--add-section', '.gnu_debugdata={}.xz'.format(minidebugfile), file]) + +def copydebugsources(debugsrcdir, sources, d): + # The debug src information written out to sourcefile is further processed + # and copied to the destination here. + + cpath = oe.cachedpath.CachedPath() + + if debugsrcdir and sources: + sourcefile = d.expand("${WORKDIR}/debugsources.list") + bb.utils.remove(sourcefile) + + # filenames are null-separated - this is an artefact of the previous use + # of rpm's debugedit, which was writing them out that way, and the code elsewhere + # is still assuming that. + debuglistoutput = '\0'.join(sources) + '\0' + with open(sourcefile, 'a') as sf: + sf.write(debuglistoutput) + + dvar = d.getVar('PKGD') + strip = d.getVar("STRIP") + objcopy = d.getVar("OBJCOPY") + workdir = d.getVar("WORKDIR") + sdir = d.getVar("S") + cflags = d.expand("${CFLAGS}") + + prefixmap = {} + for flag in cflags.split(): + if not flag.startswith("-fdebug-prefix-map"): + continue + if "recipe-sysroot" in flag: + continue + flag = flag.split("=") + prefixmap[flag[1]] = flag[2] + + nosuchdir = [] + basepath = dvar + for p in debugsrcdir.split("/"): + basepath = basepath + "/" + p + if not cpath.exists(basepath): + nosuchdir.append(basepath) + bb.utils.mkdirhier(basepath) + cpath.updatecache(basepath) + + for pmap in prefixmap: + # Ignore files from the recipe sysroots (target and native) + cmd = "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '((|)$|/.*recipe-sysroot.*/)' | " % sourcefile + # We need to ignore files that are not actually ours + # we do this by only paying attention to items from this package + cmd += "fgrep -zw '%s' | " % prefixmap[pmap] + # Remove prefix in the source paths + cmd += "sed 's#%s/##g' | " % (prefixmap[pmap]) + cmd += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s' 2>/dev/null)" % (pmap, dvar, prefixmap[pmap]) + + try: + subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + except subprocess.CalledProcessError: + # Can "fail" if internal headers/transient sources are attempted + pass + # cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced. + # Work around this by manually finding and copying any symbolic links that made it through. + cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd '%s' ; cpio -pd0mL --no-preserve-owner '%s%s')" % \ + (dvar, prefixmap[pmap], dvar, prefixmap[pmap], pmap, dvar, prefixmap[pmap]) + subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + + # debugsources.list may be polluted from the host if we used externalsrc, + # cpio uses copy-pass and may have just created a directory structure + # matching the one from the host, if thats the case move those files to + # debugsrcdir to avoid host contamination. + # Empty dir structure will be deleted in the next step. + + # Same check as above for externalsrc + if workdir not in sdir: + if os.path.exists(dvar + debugsrcdir + sdir): + cmd = "mv %s%s%s/* %s%s" % (dvar, debugsrcdir, sdir, dvar,debugsrcdir) + subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + + # The copy by cpio may have resulted in some empty directories! Remove these + cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir) + subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + + # Also remove debugsrcdir if its empty + for p in nosuchdir[::-1]: + if os.path.exists(p) and not os.listdir(p): + os.rmdir(p) + + +def process_split_and_strip_files(d): + cpath = oe.cachedpath.CachedPath() + + dvar = d.getVar('PKGD') + pn = d.getVar('PN') + hostos = d.getVar('HOST_OS') + + oldcwd = os.getcwd() + os.chdir(dvar) + + dv = package_debug_vars(d) + + # + # First lets figure out all of the files we may have to process ... do this only once! + # + elffiles = {} + symlinks = {} + staticlibs = [] + inodes = {} + libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir")) + baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir")) + skipfiles = (d.getVar("INHIBIT_PACKAGE_STRIP_FILES") or "").split() + if (d.getVar('INHIBIT_PACKAGE_STRIP') != '1' or \ + d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT') != '1'): + checkelf = {} + checkelflinks = {} + for root, dirs, files in cpath.walk(dvar): + for f in files: + file = os.path.join(root, f) + + # Skip debug files + if dv["append"] and file.endswith(dv["append"]): + continue + if dv["dir"] and dv["dir"] in os.path.dirname(file[len(dvar):]): + continue + + if file in skipfiles: + continue + + if oe.package.is_static_lib(file): + staticlibs.append(file) + continue + + try: + ltarget = cpath.realpath(file, dvar, False) + s = cpath.lstat(ltarget) + except OSError as e: + (err, strerror) = e.args + if err != errno.ENOENT: + raise + # Skip broken symlinks + continue + if not s: + continue + # Check its an executable + if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) \ + or (s[stat.ST_MODE] & stat.S_IXOTH) \ + or ((file.startswith(libdir) or file.startswith(baselibdir)) \ + and (".so" in f or ".node" in f)) \ + or (f.startswith('vmlinux') or ".ko" in f): + + if cpath.islink(file): + checkelflinks[file] = ltarget + continue + # Use a reference of device ID and inode number to identify files + file_reference = "%d_%d" % (s.st_dev, s.st_ino) + checkelf[file] = (file, file_reference) + + results = oe.utils.multiprocess_launch(oe.package.is_elf, checkelflinks.values(), d) + results_map = {} + for (ltarget, elf_file) in results: + results_map[ltarget] = elf_file + for file in checkelflinks: + ltarget = checkelflinks[file] + # If it's a symlink, and points to an ELF file, we capture the readlink target + if results_map[ltarget]: + target = os.readlink(file) + #bb.note("Sym: %s (%d)" % (ltarget, results_map[ltarget])) + symlinks[file] = target + + results = oe.utils.multiprocess_launch(oe.package.is_elf, checkelf.keys(), d) + + # Sort results by file path. This ensures that the files are always + # processed in the same order, which is important to make sure builds + # are reproducible when dealing with hardlinks + results.sort(key=lambda x: x[0]) + + for (file, elf_file) in results: + # It's a file (or hardlink), not a link + # ...but is it ELF, and is it already stripped? + if elf_file & 1: + if elf_file & 2: + if 'already-stripped' in (d.getVar('INSANE_SKIP:' + pn) or "").split(): + bb.note("Skipping file %s from %s for already-stripped QA test" % (file[len(dvar):], pn)) + else: + msg = "File '%s' from %s was already stripped, this will prevent future debugging!" % (file[len(dvar):], pn) + oe.qa.handle_error("already-stripped", msg, d) + continue + + # At this point we have an unstripped elf file. We need to: + # a) Make sure any file we strip is not hardlinked to anything else outside this tree + # b) Only strip any hardlinked file once (no races) + # c) Track any hardlinks between files so that we can reconstruct matching debug file hardlinks + + # Use a reference of device ID and inode number to identify files + file_reference = checkelf[file][1] + if file_reference in inodes: + os.unlink(file) + os.link(inodes[file_reference][0], file) + inodes[file_reference].append(file) + else: + inodes[file_reference] = [file] + # break hardlink + bb.utils.break_hardlinks(file) + elffiles[file] = elf_file + # Modified the file so clear the cache + cpath.updatecache(file) + + def strip_pkgd_prefix(f): + nonlocal dvar + + if f.startswith(dvar): + return f[len(dvar):] + + return f + + # + # First lets process debug splitting + # + if (d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT') != '1'): + results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, dv, d)) + + if dv["srcdir"] and not hostos.startswith("mingw"): + if (d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): + results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, dv, d)) + else: + for file in staticlibs: + results.append( (file,source_info(file, d)) ) + + d.setVar("PKGDEBUGSOURCES", {strip_pkgd_prefix(f): sorted(s) for f, s in results}) + + sources = set() + for r in results: + sources.update(r[1]) + + # Hardlink our debug symbols to the other hardlink copies + for ref in inodes: + if len(inodes[ref]) == 1: + continue + + target = inodes[ref][0][len(dvar):] + for file in inodes[ref][1:]: + src = file[len(dvar):] + dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(target) + dv["append"] + fpath = dvar + dest + ftarget = dvar + dv["libdir"] + os.path.dirname(target) + dv["dir"] + "/" + os.path.basename(target) + dv["append"] + bb.utils.mkdirhier(os.path.dirname(fpath)) + # Only one hardlink of separated debug info file in each directory + if not os.access(fpath, os.R_OK): + #bb.note("Link %s -> %s" % (fpath, ftarget)) + os.link(ftarget, fpath) + + # Create symlinks for all cases we were able to split symbols + for file in symlinks: + src = file[len(dvar):] + dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"] + fpath = dvar + dest + # Skip it if the target doesn't exist + try: + s = os.stat(fpath) + except OSError as e: + (err, strerror) = e.args + if err != errno.ENOENT: + raise + continue + + ltarget = symlinks[file] + lpath = os.path.dirname(ltarget) + lbase = os.path.basename(ltarget) + ftarget = "" + if lpath and lpath != ".": + ftarget += lpath + dv["dir"] + "/" + ftarget += lbase + dv["append"] + if lpath.startswith(".."): + ftarget = os.path.join("..", ftarget) + bb.utils.mkdirhier(os.path.dirname(fpath)) + #bb.note("Symlink %s -> %s" % (fpath, ftarget)) + os.symlink(ftarget, fpath) + + # Process the dv["srcdir"] if requested... + # This copies and places the referenced sources for later debugging... + copydebugsources(dv["srcdir"], sources, d) + # + # End of debug splitting + # + + # + # Now lets go back over things and strip them + # + if (d.getVar('INHIBIT_PACKAGE_STRIP') != '1'): + strip = d.getVar("STRIP") + sfiles = [] + for file in elffiles: + elf_file = int(elffiles[file]) + #bb.note("Strip %s" % file) + sfiles.append((file, elf_file, strip)) + if (d.getVar('PACKAGE_STRIP_STATIC') == '1' or d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): + for f in staticlibs: + sfiles.append((f, 16, strip)) + + oe.utils.multiprocess_launch(oe.package.runstrip, sfiles, d) + + # Build "minidebuginfo" and reinject it back into the stripped binaries + if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': + oe.utils.multiprocess_launch(inject_minidebuginfo, list(elffiles), d, + extraargs=(dvar, dv, d)) + + # + # End of strip + # + os.chdir(oldcwd) + + +def populate_packages(d): + cpath = oe.cachedpath.CachedPath() + + workdir = d.getVar('WORKDIR') + outdir = d.getVar('DEPLOY_DIR') + dvar = d.getVar('PKGD') + packages = d.getVar('PACKAGES').split() + pn = d.getVar('PN') + + bb.utils.mkdirhier(outdir) + os.chdir(dvar) + + autodebug = not (d.getVar("NOAUTOPACKAGEDEBUG") or False) + + split_source_package = (d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-with-srcpkg') + + # If debug-with-srcpkg mode is enabled then add the source package if it + # doesn't exist and add the source file contents to the source package. + if split_source_package: + src_package_name = ('%s-src' % d.getVar('PN')) + if not src_package_name in packages: + packages.append(src_package_name) + d.setVar('FILES:%s' % src_package_name, '/usr/src/debug') + + # Sanity check PACKAGES for duplicates + # Sanity should be moved to sanity.bbclass once we have the infrastructure + package_dict = {} + + for i, pkg in enumerate(packages): + if pkg in package_dict: + msg = "%s is listed in PACKAGES multiple times, this leads to packaging errors." % pkg + oe.qa.handle_error("packages-list", msg, d) + # Ensure the source package gets the chance to pick up the source files + # before the debug package by ordering it first in PACKAGES. Whether it + # actually picks up any source files is controlled by + # PACKAGE_DEBUG_SPLIT_STYLE. + elif pkg.endswith("-src"): + package_dict[pkg] = (10, i) + elif autodebug and pkg.endswith("-dbg"): + package_dict[pkg] = (30, i) + else: + package_dict[pkg] = (50, i) + packages = sorted(package_dict.keys(), key=package_dict.get) + d.setVar('PACKAGES', ' '.join(packages)) + pkgdest = d.getVar('PKGDEST') + + seen = [] + + # os.mkdir masks the permissions with umask so we have to unset it first + oldumask = os.umask(0) + + debug = [] + for root, dirs, files in cpath.walk(dvar): + dir = root[len(dvar):] + if not dir: + dir = os.sep + for f in (files + dirs): + path = "." + os.path.join(dir, f) + if "/.debug/" in path or "/.debug-static/" in path or path.endswith("/.debug"): + debug.append(path) + + for pkg in packages: + root = os.path.join(pkgdest, pkg) + bb.utils.mkdirhier(root) + + filesvar = d.getVar('FILES:%s' % pkg) or "" + if "//" in filesvar: + msg = "FILES variable for package %s contains '//' which is invalid. Attempting to fix this but you should correct the metadata.\n" % pkg + oe.qa.handle_error("files-invalid", msg, d) + filesvar.replace("//", "/") + + origfiles = filesvar.split() + files, symlink_paths = oe.package.files_from_filevars(origfiles) + + if autodebug and pkg.endswith("-dbg"): + files.extend(debug) + + for file in files: + if (not cpath.islink(file)) and (not cpath.exists(file)): + continue + if file in seen: + continue + seen.append(file) + + def mkdir(src, dest, p): + src = os.path.join(src, p) + dest = os.path.join(dest, p) + fstat = cpath.stat(src) + os.mkdir(dest) + os.chmod(dest, fstat.st_mode) + os.chown(dest, fstat.st_uid, fstat.st_gid) + if p not in seen: + seen.append(p) + cpath.updatecache(dest) + + def mkdir_recurse(src, dest, paths): + if cpath.exists(dest + '/' + paths): + return + while paths.startswith("./"): + paths = paths[2:] + p = "." + for c in paths.split("/"): + p = os.path.join(p, c) + if not cpath.exists(os.path.join(dest, p)): + mkdir(src, dest, p) + + if cpath.isdir(file) and not cpath.islink(file): + mkdir_recurse(dvar, root, file) + continue + + mkdir_recurse(dvar, root, os.path.dirname(file)) + fpath = os.path.join(root,file) + if not cpath.islink(file): + os.link(file, fpath) + continue + ret = bb.utils.copyfile(file, fpath) + if ret is False or ret == 0: + bb.fatal("File population failed") + + # Check if symlink paths exist + for file in symlink_paths: + if not os.path.exists(os.path.join(root,file)): + bb.fatal("File '%s' cannot be packaged into '%s' because its " + "parent directory structure does not exist. One of " + "its parent directories is a symlink whose target " + "directory is not included in the package." % + (file, pkg)) + + os.umask(oldumask) + os.chdir(workdir) + + # Handle excluding packages with incompatible licenses + package_list = [] + for pkg in packages: + licenses = d.getVar('_exclude_incompatible-' + pkg) + if licenses: + msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses) + oe.qa.handle_error("incompatible-license", msg, d) + else: + package_list.append(pkg) + d.setVar('PACKAGES', ' '.join(package_list)) + + unshipped = [] + for root, dirs, files in cpath.walk(dvar): + dir = root[len(dvar):] + if not dir: + dir = os.sep + for f in (files + dirs): + path = os.path.join(dir, f) + if ('.' + path) not in seen: + unshipped.append(path) + + if unshipped != []: + msg = pn + ": Files/directories were installed but not shipped in any package:" + if "installed-vs-shipped" in (d.getVar('INSANE_SKIP:' + pn) or "").split(): + bb.note("Package %s skipping QA tests: installed-vs-shipped" % pn) + else: + for f in unshipped: + msg = msg + "\n " + f + msg = msg + "\nPlease set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.\n" + msg = msg + "%s: %d installed and not shipped files." % (pn, len(unshipped)) + oe.qa.handle_error("installed-vs-shipped", msg, d) + +def process_fixsymlinks(pkgfiles, d): + cpath = oe.cachedpath.CachedPath() + pkgdest = d.getVar('PKGDEST') + packages = d.getVar("PACKAGES", False).split() + + dangling_links = {} + pkg_files = {} + for pkg in packages: + dangling_links[pkg] = [] + pkg_files[pkg] = [] + inst_root = os.path.join(pkgdest, pkg) + for path in pkgfiles[pkg]: + rpath = path[len(inst_root):] + pkg_files[pkg].append(rpath) + rtarget = cpath.realpath(path, inst_root, True, assume_dir = True) + if not cpath.lexists(rtarget): + dangling_links[pkg].append(os.path.normpath(rtarget[len(inst_root):])) + + newrdepends = {} + for pkg in dangling_links: + for l in dangling_links[pkg]: + found = False + bb.debug(1, "%s contains dangling link %s" % (pkg, l)) + for p in packages: + if l in pkg_files[p]: + found = True + bb.debug(1, "target found in %s" % p) + if p == pkg: + break + if pkg not in newrdepends: + newrdepends[pkg] = [] + newrdepends[pkg].append(p) + break + if found == False: + bb.note("%s contains dangling symlink to %s" % (pkg, l)) + + for pkg in newrdepends: + rdepends = bb.utils.explode_dep_versions2(d.getVar('RDEPENDS:' + pkg) or "") + for p in newrdepends[pkg]: + if p not in rdepends: + rdepends[p] = [] + d.setVar('RDEPENDS:' + pkg, bb.utils.join_deps(rdepends, commasep=False)) + +def process_filedeps(pkgfiles, d): + """ + Collect perfile run-time dependency metadata + Output: + FILERPROVIDESFLIST:pkg - list of all files w/ deps + FILERPROVIDES:filepath:pkg - per file dep + + FILERDEPENDSFLIST:pkg - list of all files w/ deps + FILERDEPENDS:filepath:pkg - per file dep + """ + if d.getVar('SKIP_FILEDEPS') == '1': + return + + pkgdest = d.getVar('PKGDEST') + packages = d.getVar('PACKAGES') + rpmdeps = d.getVar('RPMDEPS') + + def chunks(files, n): + return [files[i:i+n] for i in range(0, len(files), n)] + + pkglist = [] + for pkg in packages.split(): + if d.getVar('SKIP_FILEDEPS:' + pkg) == '1': + continue + if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-') or pkg.endswith('-src'): + continue + for files in chunks(pkgfiles[pkg], 100): + pkglist.append((pkg, files, rpmdeps, pkgdest)) + + processed = oe.utils.multiprocess_launch(oe.package.filedeprunner, pkglist, d) + + provides_files = {} + requires_files = {} + + for result in processed: + (pkg, provides, requires) = result + + if pkg not in provides_files: + provides_files[pkg] = [] + if pkg not in requires_files: + requires_files[pkg] = [] + + for file in sorted(provides): + provides_files[pkg].append(file) + key = "FILERPROVIDES:" + file + ":" + pkg + d.appendVar(key, " " + " ".join(provides[file])) + + for file in sorted(requires): + requires_files[pkg].append(file) + key = "FILERDEPENDS:" + file + ":" + pkg + d.appendVar(key, " " + " ".join(requires[file])) + + for pkg in requires_files: + d.setVar("FILERDEPENDSFLIST:" + pkg, " ".join(sorted(requires_files[pkg]))) + for pkg in provides_files: + d.setVar("FILERPROVIDESFLIST:" + pkg, " ".join(sorted(provides_files[pkg]))) + +def process_shlibs(pkgfiles, d): + cpath = oe.cachedpath.CachedPath() + + exclude_shlibs = d.getVar('EXCLUDE_FROM_SHLIBS', False) + if exclude_shlibs: + bb.note("not generating shlibs") + return + + lib_re = re.compile(r"^.*\.so") + libdir_re = re.compile(r".*/%s$" % d.getVar('baselib')) + + packages = d.getVar('PACKAGES') + + shlib_pkgs = [] + exclusion_list = d.getVar("EXCLUDE_PACKAGES_FROM_SHLIBS") + if exclusion_list: + for pkg in packages.split(): + if pkg not in exclusion_list.split(): + shlib_pkgs.append(pkg) + else: + bb.note("not generating shlibs for %s" % pkg) + else: + shlib_pkgs = packages.split() + + hostos = d.getVar('HOST_OS') + + workdir = d.getVar('WORKDIR') + + ver = d.getVar('PKGV') + if not ver: + msg = "PKGV not defined" + oe.qa.handle_error("pkgv-undefined", msg, d) + return + + pkgdest = d.getVar('PKGDEST') + + shlibswork_dir = d.getVar('SHLIBSWORKDIR') + + def linux_so(file, pkg, pkgver, d): + needs_ldconfig = False + needed = set() + sonames = set() + renames = [] + ldir = os.path.dirname(file).replace(pkgdest + "/" + pkg, '') + cmd = d.getVar('OBJDUMP') + " -p " + pipes.quote(file) + " 2>/dev/null" + fd = os.popen(cmd) + lines = fd.readlines() + fd.close() + rpath = tuple() + for l in lines: + m = re.match(r"\s+RPATH\s+([^\s]*)", l) + if m: + rpaths = m.group(1).replace("$ORIGIN", ldir).split(":") + rpath = tuple(map(os.path.normpath, rpaths)) + for l in lines: + m = re.match(r"\s+NEEDED\s+([^\s]*)", l) + if m: + dep = m.group(1) + if dep not in needed: + needed.add((dep, file, rpath)) + m = re.match(r"\s+SONAME\s+([^\s]*)", l) + if m: + this_soname = m.group(1) + prov = (this_soname, ldir, pkgver) + if not prov in sonames: + # if library is private (only used by package) then do not build shlib for it + if not private_libs or len([i for i in private_libs if fnmatch.fnmatch(this_soname, i)]) == 0: + sonames.add(prov) + if libdir_re.match(os.path.dirname(file)): + needs_ldconfig = True + if needs_ldconfig and snap_symlinks and (os.path.basename(file) != this_soname): + renames.append((file, os.path.join(os.path.dirname(file), this_soname))) + return (needs_ldconfig, needed, sonames, renames) + + def darwin_so(file, needed, sonames, renames, pkgver): + if not os.path.exists(file): + return + ldir = os.path.dirname(file).replace(pkgdest + "/" + pkg, '') + + def get_combinations(base): + # + # Given a base library name, find all combinations of this split by "." and "-" + # + combos = [] + options = base.split(".") + for i in range(1, len(options) + 1): + combos.append(".".join(options[0:i])) + options = base.split("-") + for i in range(1, len(options) + 1): + combos.append("-".join(options[0:i])) + return combos + + if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg') and not pkg.endswith('-src'): + # Drop suffix + name = os.path.basename(file).rsplit(".",1)[0] + # Find all combinations + combos = get_combinations(name) + for combo in combos: + if not combo in sonames: + prov = (combo, ldir, pkgver) + sonames.add(prov) + if file.endswith('.dylib') or file.endswith('.so'): + rpath = [] + p = subprocess.Popen([d.expand("${HOST_PREFIX}otool"), '-l', file], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + out, err = p.communicate() + # If returned successfully, process stdout for results + if p.returncode == 0: + for l in out.split("\n"): + l = l.strip() + if l.startswith('path '): + rpath.append(l.split()[1]) + + p = subprocess.Popen([d.expand("${HOST_PREFIX}otool"), '-L', file], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + out, err = p.communicate() + # If returned successfully, process stdout for results + if p.returncode == 0: + for l in out.split("\n"): + l = l.strip() + if not l or l.endswith(":"): + continue + if "is not an object file" in l: + continue + name = os.path.basename(l.split()[0]).rsplit(".", 1)[0] + if name and name not in needed[pkg]: + needed[pkg].add((name, file, tuple())) + + def mingw_dll(file, needed, sonames, renames, pkgver): + if not os.path.exists(file): + return + + if file.endswith(".dll"): + # assume all dlls are shared objects provided by the package + sonames.add((os.path.basename(file), os.path.dirname(file).replace(pkgdest + "/" + pkg, ''), pkgver)) + + if (file.endswith(".dll") or file.endswith(".exe")): + # use objdump to search for "DLL Name: .*\.dll" + p = subprocess.Popen([d.expand("${HOST_PREFIX}objdump"), "-p", file], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + out, err = p.communicate() + # process the output, grabbing all .dll names + if p.returncode == 0: + for m in re.finditer(r"DLL Name: (.*?\.dll)$", out.decode(), re.MULTILINE | re.IGNORECASE): + dllname = m.group(1) + if dllname: + needed[pkg].add((dllname, file, tuple())) + + if d.getVar('PACKAGE_SNAP_LIB_SYMLINKS') == "1": + snap_symlinks = True + else: + snap_symlinks = False + + needed = {} + + shlib_provider = oe.package.read_shlib_providers(d) + + for pkg in shlib_pkgs: + private_libs = d.getVar('PRIVATE_LIBS:' + pkg) or d.getVar('PRIVATE_LIBS') or "" + private_libs = private_libs.split() + needs_ldconfig = False + bb.debug(2, "calculating shlib provides for %s" % pkg) + + pkgver = d.getVar('PKGV:' + pkg) + if not pkgver: + pkgver = d.getVar('PV_' + pkg) + if not pkgver: + pkgver = ver + + needed[pkg] = set() + sonames = set() + renames = [] + linuxlist = [] + for file in pkgfiles[pkg]: + soname = None + if cpath.islink(file): + continue + if hostos == "darwin" or hostos == "darwin8": + darwin_so(file, needed, sonames, renames, pkgver) + elif hostos.startswith("mingw"): + mingw_dll(file, needed, sonames, renames, pkgver) + elif os.access(file, os.X_OK) or lib_re.match(file): + linuxlist.append(file) + + if linuxlist: + results = oe.utils.multiprocess_launch(linux_so, linuxlist, d, extraargs=(pkg, pkgver, d)) + for r in results: + ldconfig = r[0] + needed[pkg] |= r[1] + sonames |= r[2] + renames.extend(r[3]) + needs_ldconfig = needs_ldconfig or ldconfig + + for (old, new) in renames: + bb.note("Renaming %s to %s" % (old, new)) + bb.utils.rename(old, new) + pkgfiles[pkg].remove(old) + + shlibs_file = os.path.join(shlibswork_dir, pkg + ".list") + if len(sonames): + with open(shlibs_file, 'w') as fd: + for s in sorted(sonames): + if s[0] in shlib_provider and s[1] in shlib_provider[s[0]]: + (old_pkg, old_pkgver) = shlib_provider[s[0]][s[1]] + if old_pkg != pkg: + bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s[0], pkg, pkgver)) + bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s[0])) + fd.write(s[0] + ':' + s[1] + ':' + s[2] + '\n') + if s[0] not in shlib_provider: + shlib_provider[s[0]] = {} + shlib_provider[s[0]][s[1]] = (pkg, pkgver) + if needs_ldconfig: + bb.debug(1, 'adding ldconfig call to postinst for %s' % pkg) + postinst = d.getVar('pkg_postinst:%s' % pkg) + if not postinst: + postinst = '#!/bin/sh\n' + postinst += d.getVar('ldconfig_postinst_fragment') + d.setVar('pkg_postinst:%s' % pkg, postinst) + bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames)) + + assumed_libs = d.getVar('ASSUME_SHLIBS') + if assumed_libs: + libdir = d.getVar("libdir") + for e in assumed_libs.split(): + l, dep_pkg = e.split(":") + lib_ver = None + dep_pkg = dep_pkg.rsplit("_", 1) + if len(dep_pkg) == 2: + lib_ver = dep_pkg[1] + dep_pkg = dep_pkg[0] + if l not in shlib_provider: + shlib_provider[l] = {} + shlib_provider[l][libdir] = (dep_pkg, lib_ver) + + libsearchpath = [d.getVar('libdir'), d.getVar('base_libdir')] + + for pkg in shlib_pkgs: + bb.debug(2, "calculating shlib requirements for %s" % pkg) + + private_libs = d.getVar('PRIVATE_LIBS:' + pkg) or d.getVar('PRIVATE_LIBS') or "" + private_libs = private_libs.split() + + deps = list() + for n in needed[pkg]: + # if n is in private libraries, don't try to search provider for it + # this could cause problem in case some abc.bb provides private + # /opt/abc/lib/libfoo.so.1 and contains /usr/bin/abc depending on system library libfoo.so.1 + # but skipping it is still better alternative than providing own + # version and then adding runtime dependency for the same system library + if private_libs and len([i for i in private_libs if fnmatch.fnmatch(n[0], i)]) > 0: + bb.debug(2, '%s: Dependency %s covered by PRIVATE_LIBS' % (pkg, n[0])) + continue + if n[0] in shlib_provider.keys(): + shlib_provider_map = shlib_provider[n[0]] + matches = set() + for p in itertools.chain(list(n[2]), sorted(shlib_provider_map.keys()), libsearchpath): + if p in shlib_provider_map: + matches.add(p) + if len(matches) > 1: + matchpkgs = ', '.join([shlib_provider_map[match][0] for match in matches]) + bb.error("%s: Multiple shlib providers for %s: %s (used by files: %s)" % (pkg, n[0], matchpkgs, n[1])) + elif len(matches) == 1: + (dep_pkg, ver_needed) = shlib_provider_map[matches.pop()] + + bb.debug(2, '%s: Dependency %s requires package %s (used by files: %s)' % (pkg, n[0], dep_pkg, n[1])) + + if dep_pkg == pkg: + continue + + if ver_needed: + dep = "%s (>= %s)" % (dep_pkg, ver_needed) + else: + dep = dep_pkg + if not dep in deps: + deps.append(dep) + continue + bb.note("Couldn't find shared library provider for %s, used by files: %s" % (n[0], n[1])) + + deps_file = os.path.join(pkgdest, pkg + ".shlibdeps") + if os.path.exists(deps_file): + os.remove(deps_file) + if deps: + with open(deps_file, 'w') as fd: + for dep in sorted(deps): + fd.write(dep + '\n') + +def process_pkgconfig(pkgfiles, d): + packages = d.getVar('PACKAGES') + workdir = d.getVar('WORKDIR') + pkgdest = d.getVar('PKGDEST') + + shlibs_dirs = d.getVar('SHLIBSDIRS').split() + shlibswork_dir = d.getVar('SHLIBSWORKDIR') + + pc_re = re.compile(r'(.*)\.pc$') + var_re = re.compile(r'(.*)=(.*)') + field_re = re.compile(r'(.*): (.*)') + + pkgconfig_provided = {} + pkgconfig_needed = {} + for pkg in packages.split(): + pkgconfig_provided[pkg] = [] + pkgconfig_needed[pkg] = [] + for file in sorted(pkgfiles[pkg]): + m = pc_re.match(file) + if m: + pd = bb.data.init() + name = m.group(1) + pkgconfig_provided[pkg].append(os.path.basename(name)) + if not os.access(file, os.R_OK): + continue + with open(file, 'r') as f: + lines = f.readlines() + for l in lines: + m = var_re.match(l) + if m: + name = m.group(1) + val = m.group(2) + pd.setVar(name, pd.expand(val)) + continue + m = field_re.match(l) + if m: + hdr = m.group(1) + exp = pd.expand(m.group(2)) + if hdr == 'Requires': + pkgconfig_needed[pkg] += exp.replace(',', ' ').split() + + for pkg in packages.split(): + pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist") + if pkgconfig_provided[pkg] != []: + with open(pkgs_file, 'w') as f: + for p in sorted(pkgconfig_provided[pkg]): + f.write('%s\n' % p) + + # Go from least to most specific since the last one found wins + for dir in reversed(shlibs_dirs): + if not os.path.exists(dir): + continue + for file in sorted(os.listdir(dir)): + m = re.match(r'^(.*)\.pclist$', file) + if m: + pkg = m.group(1) + with open(os.path.join(dir, file)) as fd: + lines = fd.readlines() + pkgconfig_provided[pkg] = [] + for l in lines: + pkgconfig_provided[pkg].append(l.rstrip()) + + for pkg in packages.split(): + deps = [] + for n in pkgconfig_needed[pkg]: + found = False + for k in pkgconfig_provided.keys(): + if n in pkgconfig_provided[k]: + if k != pkg and not (k in deps): + deps.append(k) + found = True + if found == False: + bb.note("couldn't find pkgconfig module '%s' in any package" % n) + deps_file = os.path.join(pkgdest, pkg + ".pcdeps") + if len(deps): + with open(deps_file, 'w') as fd: + for dep in deps: + fd.write(dep + '\n') + +def read_libdep_files(d): + pkglibdeps = {} + packages = d.getVar('PACKAGES').split() + for pkg in packages: + pkglibdeps[pkg] = {} + for extension in ".shlibdeps", ".pcdeps", ".clilibdeps": + depsfile = d.expand("${PKGDEST}/" + pkg + extension) + if os.access(depsfile, os.R_OK): + with open(depsfile) as fd: + lines = fd.readlines() + for l in lines: + l.rstrip() + deps = bb.utils.explode_dep_versions2(l) + for dep in deps: + if not dep in pkglibdeps[pkg]: + pkglibdeps[pkg][dep] = deps[dep] + return pkglibdeps + +def process_depchains(pkgfiles, d): + """ + For a given set of prefix and postfix modifiers, make those packages + RRECOMMENDS on the corresponding packages for its RDEPENDS. + + Example: If package A depends upon package B, and A's .bb emits an + A-dev package, this would make A-dev Recommends: B-dev. + + If only one of a given suffix is specified, it will take the RRECOMMENDS + based on the RDEPENDS of *all* other packages. If more than one of a given + suffix is specified, its will only use the RDEPENDS of the single parent + package. + """ + + packages = d.getVar('PACKAGES') + postfixes = (d.getVar('DEPCHAIN_POST') or '').split() + prefixes = (d.getVar('DEPCHAIN_PRE') or '').split() + + def pkg_adddeprrecs(pkg, base, suffix, getname, depends, d): + + #bb.note('depends for %s is %s' % (base, depends)) + rreclist = bb.utils.explode_dep_versions2(d.getVar('RRECOMMENDS:' + pkg) or "") + + for depend in sorted(depends): + if depend.find('-native') != -1 or depend.find('-cross') != -1 or depend.startswith('virtual/'): + #bb.note("Skipping %s" % depend) + continue + if depend.endswith('-dev'): + depend = depend[:-4] + if depend.endswith('-dbg'): + depend = depend[:-4] + pkgname = getname(depend, suffix) + #bb.note("Adding %s for %s" % (pkgname, depend)) + if pkgname not in rreclist and pkgname != pkg: + rreclist[pkgname] = [] + + #bb.note('setting: RRECOMMENDS:%s=%s' % (pkg, ' '.join(rreclist))) + d.setVar('RRECOMMENDS:%s' % pkg, bb.utils.join_deps(rreclist, commasep=False)) + + def pkg_addrrecs(pkg, base, suffix, getname, rdepends, d): + + #bb.note('rdepends for %s is %s' % (base, rdepends)) + rreclist = bb.utils.explode_dep_versions2(d.getVar('RRECOMMENDS:' + pkg) or "") + + for depend in sorted(rdepends): + if depend.find('virtual-locale-') != -1: + #bb.note("Skipping %s" % depend) + continue + if depend.endswith('-dev'): + depend = depend[:-4] + if depend.endswith('-dbg'): + depend = depend[:-4] + pkgname = getname(depend, suffix) + #bb.note("Adding %s for %s" % (pkgname, depend)) + if pkgname not in rreclist and pkgname != pkg: + rreclist[pkgname] = [] + + #bb.note('setting: RRECOMMENDS:%s=%s' % (pkg, ' '.join(rreclist))) + d.setVar('RRECOMMENDS:%s' % pkg, bb.utils.join_deps(rreclist, commasep=False)) + + def add_dep(list, dep): + if dep not in list: + list.append(dep) + + depends = [] + for dep in bb.utils.explode_deps(d.getVar('DEPENDS') or ""): + add_dep(depends, dep) + + rdepends = [] + for pkg in packages.split(): + for dep in bb.utils.explode_deps(d.getVar('RDEPENDS:' + pkg) or ""): + add_dep(rdepends, dep) + + #bb.note('rdepends is %s' % rdepends) + + def post_getname(name, suffix): + return '%s%s' % (name, suffix) + def pre_getname(name, suffix): + return '%s%s' % (suffix, name) + + pkgs = {} + for pkg in packages.split(): + for postfix in postfixes: + if pkg.endswith(postfix): + if not postfix in pkgs: + pkgs[postfix] = {} + pkgs[postfix][pkg] = (pkg[:-len(postfix)], post_getname) + + for prefix in prefixes: + if pkg.startswith(prefix): + if not prefix in pkgs: + pkgs[prefix] = {} + pkgs[prefix][pkg] = (pkg[:-len(prefix)], pre_getname) + + if "-dbg" in pkgs: + pkglibdeps = read_libdep_files(d) + pkglibdeplist = [] + for pkg in pkglibdeps: + for k in pkglibdeps[pkg]: + add_dep(pkglibdeplist, k) + dbgdefaultdeps = ((d.getVar('DEPCHAIN_DBGDEFAULTDEPS') == '1') or (bb.data.inherits_class('packagegroup', d))) + + for suffix in pkgs: + for pkg in pkgs[suffix]: + if d.getVarFlag('RRECOMMENDS:' + pkg, 'nodeprrecs'): + continue + (base, func) = pkgs[suffix][pkg] + if suffix == "-dev": + pkg_adddeprrecs(pkg, base, suffix, func, depends, d) + elif suffix == "-dbg": + if not dbgdefaultdeps: + pkg_addrrecs(pkg, base, suffix, func, pkglibdeplist, d) + continue + if len(pkgs[suffix]) == 1: + pkg_addrrecs(pkg, base, suffix, func, rdepends, d) + else: + rdeps = [] + for dep in bb.utils.explode_deps(d.getVar('RDEPENDS:' + base) or ""): + add_dep(rdeps, dep) + pkg_addrrecs(pkg, base, suffix, func, rdeps, d) + From patchwork Wed Jan 4 15:47:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A666C54EBF for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) by mx.groups.io with SMTP id smtpd.web10.15380.1672847241462200168 for ; Wed, 04 Jan 2023 07:47:21 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=c35tZLsD; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f45.google.com with SMTP id d17so13674119wrs.2 for ; Wed, 04 Jan 2023 07:47:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=d9q8zIWaAlp0keP+ZwIcTQeIi//cIHzwznow4ks6o/E=; b=c35tZLsDD7vaONieKl/+dAGWwad4HvywY4Qlh8PCGSIBcCW/doPo7VPr2iGiOFAcIT OAovVTzEjz+vFeBlBCMnfzBwV4sUjVD6RjOCLZTdoCzmZ47LUeFWSEWk2H0kVg1QHG/+ 4LbRcertHdbYKND++QYdpC9angIpGmRDG33bc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d9q8zIWaAlp0keP+ZwIcTQeIi//cIHzwznow4ks6o/E=; b=aeKI8VIHqQxmT9i63yCxAvOFCGCo48UDDGhZoYxVs6lTI5477DdkyGHCV80DVGM7UB uDwYkb72kPLPRf6wuq7jfAY8+SJhoVXo62RYjvr8NwN//zAzlVyn/Y1K6RcfOc3g4J1O M1CbZLnkb/xsQYOwP9YKRKZNmlGTgpMLfEokmM5nEhlOo+H2DOtBNGNalXtthnMRy55+ UMZUZRRY9V6OT4nu0biDQuLSZIDv94tho+VeEAJyRJ62Tqm7Ee0KG9u8CVivq6m77tAG XX5PkdEySMMUnHKvJlpMzQmahCnT7fGeEApG0x8fmiahvoFfmVwg9031jUJEhxbktTxu ieMg== X-Gm-Message-State: AFqh2kpnUXfQQjhhSN3D2CIFn6RFcIVnGUknLvdmJgDcr4GnsW6KCxb+ Z11UThlqat+yoE6D8l6VcMTAdXX3X549nE2g X-Google-Smtp-Source: AMrXdXvS20VFQJhaqghbPDS+QoB9fMEh/UsZbN+eYw/aQuNI2v3jw6TUrY3k1HiXLWjxMyTWp3ylvg== X-Received: by 2002:a5d:5908:0:b0:279:4938:eb98 with SMTP id v8-20020a5d5908000000b002794938eb98mr23847545wrd.11.1672847239698; Wed, 04 Jan 2023 07:47:19 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:19 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 06/10] package: Drop unused function and obsolete comment Date: Wed, 4 Jan 2023 15:47:10 +0000 Message-Id: <20230104154714.1168535-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175493 The function appears unused and the comment obsolete, drop them. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index 21a50bbb45e..5c250776f8d 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -243,18 +243,6 @@ python () { d.appendVarFlag('do_package', 'deptask', " do_packagedata") } -def checkbuildpath(file, d): - tmpdir = d.getVar('TMPDIR') - with open(file) as f: - file_content = f.read() - if tmpdir in file_content: - return True - - return False - -# -# Used by do_packagedata (and possibly other routines post do_package) -# PRSERV_ACTIVE = "${@bool(d.getVar("PRSERV_HOST"))}" PRSERV_ACTIVE[vardepvalue] = "${PRSERV_ACTIVE}" From patchwork Wed Jan 4 15:47:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ADAFC4708D for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web11.15530.1672847242002948597 for ; Wed, 04 Jan 2023 07:47:22 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=KeoCVRxk; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f44.google.com with SMTP id c65-20020a1c3544000000b003cfffd00fc0so29845881wma.1 for ; Wed, 04 Jan 2023 07:47:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Hbtd9hUsrEmnaMofo+tU8eDm/LIRVOKHL4+jnI5juUU=; b=KeoCVRxk0nvLolUUfSVjhuC1Lxo+rftrVTQVz6TY1p2y6HJqABEid1+Zw9SMa8oOsX MKqfWBVc9bpCKP/JkCoAoDZj/JaL/el1sIaM2pgbz2N9tE6hQHBOzS05+BnFgFAlDhY6 LamqNwWP0PhSkqo8AA+oEXdQjaJPiXiVjhGQk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hbtd9hUsrEmnaMofo+tU8eDm/LIRVOKHL4+jnI5juUU=; b=dl9jKnl2kjYlCQX6UC/Td7AsYTdrOWpfe+7GdXzjNt8tdtvRXQVIqMp+Lw0dWOJ7TB TNEHBGd2zzDM9cSpN68TKWtpan2De4Ho0YUzd5BW3dflCbEJrzURuDZDwXAsRe1hTYPV xs0IZHvY1X6e1FKXyXQFeW3qGBNmlBKSdhZcZigFZUwGFm/PX+sLxzo9L5ZAwpcCh6tf nwuA5fHrHLANMePi2UEpTy7CLslBG0Un2SYtGZnvWVZ/sW5YIL7xt/t0kxEYfp6vP9Lm D5WxN7Vun8qZZp2YDT/ocbGPDSJ/ayS9lr9C/4ctnEZYH5dvjp2knfGfGasi9eCpZYen r6nw== X-Gm-Message-State: AFqh2krvnk/9Bc3BmeIJ+tOdlI5XzflGoGwqT1Z5IffzGhlhaCKC73c7 9k1JeggufCSShvxdWEGT8E3/wmlUrIkz5Dqs X-Google-Smtp-Source: AMrXdXvL6EpZY0QqbWOQIxwqrOzPbH64CchHn7fR0MiIxOsCE/iEDXqLJj6fBnn2LXIfMTiS/seusA== X-Received: by 2002:a1c:4c12:0:b0:3c6:e63e:89a6 with SMTP id z18-20020a1c4c12000000b003c6e63e89a6mr34168212wmf.2.1672847240236; Wed, 04 Jan 2023 07:47:20 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:19 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 07/10] package: Move mapping_rename_hook to packagedata function library Date: Wed, 4 Jan 2023 15:47:11 +0000 Message-Id: <20230104154714.1168535-7-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175494 This function is used by the packaging tasks/classes and makes much more sense in the packagedata function library. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 13 ------------- meta/classes-global/package_deb.bbclass | 2 +- meta/classes-global/package_ipk.bbclass | 2 +- meta/classes-global/package_rpm.bbclass | 2 +- meta/lib/oe/packagedata.py | 10 +++++++++- 5 files changed, 12 insertions(+), 17 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index 5c250776f8d..d334da6fb3c 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -607,16 +607,3 @@ python do_packagedata_setscene () { } addtask do_packagedata_setscene -# -# Helper functions for the package writing classes -# - -def mapping_rename_hook(d): - """ - Rewrite variables to account for package renaming in things - like debian.bbclass or manual PKG variable name changes - """ - pkg = d.getVar("PKG") - oe.packagedata.runtime_mapping_rename("RDEPENDS", pkg, d) - oe.packagedata.runtime_mapping_rename("RRECOMMENDS", pkg, d) - oe.packagedata.runtime_mapping_rename("RSUGGESTS", pkg, d) diff --git a/meta/classes-global/package_deb.bbclass b/meta/classes-global/package_deb.bbclass index 99e377b4551..5aba3837bde 100644 --- a/meta/classes-global/package_deb.bbclass +++ b/meta/classes-global/package_deb.bbclass @@ -174,7 +174,7 @@ def deb_write_pkg(pkg, d): ctrlfile.write(custom_fields_chunk) ctrlfile.write("\n") - mapping_rename_hook(localdata) + oe.packagedata.mapping_rename_hook(localdata) def debian_cmp_remap(var): # dpkg does not allow for '(', ')' or ':' in a dependency name diff --git a/meta/classes-global/package_ipk.bbclass b/meta/classes-global/package_ipk.bbclass index 9b75f5cf1c3..b4b7bc9ac27 100644 --- a/meta/classes-global/package_ipk.bbclass +++ b/meta/classes-global/package_ipk.bbclass @@ -167,7 +167,7 @@ def ipk_write_pkg(pkg, d): ctrlfile.write(custom_fields_chunk) ctrlfile.write("\n") - mapping_rename_hook(localdata) + oe.packagedata.mapping_rename_hook(localdata) def debian_cmp_remap(var): # In debian '>' and '<' do not mean what it appears they mean diff --git a/meta/classes-global/package_rpm.bbclass b/meta/classes-global/package_rpm.bbclass index 16631688b10..85d0bd7fce9 100644 --- a/meta/classes-global/package_rpm.bbclass +++ b/meta/classes-global/package_rpm.bbclass @@ -365,7 +365,7 @@ python write_specfile () { translate_vers('RCONFLICTS', localdata) # Map the dependencies into their final form - mapping_rename_hook(localdata) + oe.packagedata.mapping_rename_hook(localdata) splitrdepends = localdata.getVar('RDEPENDS') or "" splitrrecommends = localdata.getVar('RRECOMMENDS') or "" diff --git a/meta/lib/oe/packagedata.py b/meta/lib/oe/packagedata.py index efa600ddcd3..162ff60d72f 100644 --- a/meta/lib/oe/packagedata.py +++ b/meta/lib/oe/packagedata.py @@ -340,4 +340,12 @@ fi and not bb.data.inherits_class('packagegroup', d): write_extra_runtime_pkgs(global_variants, packages, pkgdatadir) - +def mapping_rename_hook(d): + """ + Rewrite variables to account for package renaming in things + like debian.bbclass or manual PKG variable name changes + """ + pkg = d.getVar("PKG") + oe.packagedata.runtime_mapping_rename("RDEPENDS", pkg, d) + oe.packagedata.runtime_mapping_rename("RRECOMMENDS", pkg, d) + oe.packagedata.runtime_mapping_rename("RSUGGESTS", pkg, d) From patchwork Wed Jan 4 15:47:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08AEBC4332F for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web11.15532.1672847242538559780 for ; Wed, 04 Jan 2023 07:47:22 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=cT2WIspQ; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f46.google.com with SMTP id t15so24551591wro.9 for ; Wed, 04 Jan 2023 07:47:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Vgrd8iDV0v+d1l6C0+GMdBNGx1e5cY7gNjIKrOovpCc=; b=cT2WIspQr8GL/mxdd91oONOsscJzxoXuyXVpZ1VUMsvHfkNV59S/jWEvU/3g8ba3aO eT2sBBp7L+BWF770FaokYb/mUygzjHsXpqbUJxJzEfdNeWPhQI2o+i7nDdpP87Sj+VIq 8Q5EGLz/0NlEUWG6wcn716JuO0UoQ/EdtMKs0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vgrd8iDV0v+d1l6C0+GMdBNGx1e5cY7gNjIKrOovpCc=; b=gc8vO56q3qA/FhUlGPQKDJpklcg77lSRDM3+7D4xHoDsSoE1MFnbE63csMZegAo3Cb b8ie6khhZnOLU3bBMv4Ctb1tLM4Wnaoac+2lZpLolsiYk+D9CujcvPGdMODtj+E32HEh 6KAGGPsgx3ajJJqManPOiMI+O8cWbW0+LjfYYpCFL7jHxGKf1m76aVu6wZNSqzTYJMWt GwOXaovhk0wkYaO8/G8beQMJm7lwH9CLDvYONetLkzofY14EC0AC7N85VjzoBr6pZdGo AKUbADanMNfNEsYP+wRC9G2XN69pP5mUpi2fDySqNnM8RVuYXdl5h9ONZu+BSaVEKrZd lomw== X-Gm-Message-State: AFqh2krIrd4haBgG/AEcR/0AEVPUXGo5M7HOkK4w5m7NwMDry8oscRKG 2FWAOkFnsvw7iNh/vZdAOTXa88PS8NHC8e1W X-Google-Smtp-Source: AMrXdXvLBTghgnpUwYYDp2tAeDE4S2EgLCioAm4wc2c4Qgj6YzIdiv7l5nroAHKo1L532Y2F3fBgCw== X-Received: by 2002:adf:e48d:0:b0:27b:86ef:be7d with SMTP id i13-20020adfe48d000000b0027b86efbe7dmr20599072wrm.42.1672847240834; Wed, 04 Jan 2023 07:47:20 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:20 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 08/10] python3-cython: Use PACKAGE_PREPROCESS_FUNCS instead of PACKAGEBUILDPKGD Date: Wed, 4 Jan 2023 15:47:12 +0000 Message-Id: <20230104154714.1168535-8-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175495 The recipe can use the more commonly used PACKAGE_PREPROCESS_FUNCS instead of PACKAGEBUILDPKGD which is likely to be removed soon, there should be no change in behaviour. Signed-off-by: Richard Purdie --- meta/recipes-devtools/python/python3-cython_0.29.32.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-devtools/python/python3-cython_0.29.32.bb b/meta/recipes-devtools/python/python3-cython_0.29.32.bb index 8fed1cf94d0..1c4853e14af 100644 --- a/meta/recipes-devtools/python/python3-cython_0.29.32.bb +++ b/meta/recipes-devtools/python/python3-cython_0.29.32.bb @@ -17,7 +17,7 @@ do_install:append() { mv ${D}${bindir}/cygdb ${D}${bindir}/cygdb3 } -PACKAGEBUILDPKGD += "cython_fix_sources" +PACKAGE_PREPROCESS_FUNCS += "cython_fix_sources" cython_fix_sources () { for f in ${PKGD}/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}/Cython/Compiler/FlowControl.c \ From patchwork Wed Jan 4 15:47:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E6E1C54EBD for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web11.15534.1672847243177034808 for ; Wed, 04 Jan 2023 07:47:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=WhP/ErTM; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id fm16-20020a05600c0c1000b003d96fb976efso24545751wmb.3 for ; Wed, 04 Jan 2023 07:47:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=lRVhKK9yhGoV23Ow6VTNbA/LVo5ZgOUa07g3lPnE3vE=; b=WhP/ErTM7cTjI6KlEBU7kg8g208VUQwHzjV+S8vuhHUmdMVVDEYrxbFobaQx6x6vgm pBDncgM3eh5xQB5YMp9Xn1Z0XhrgG0J21ouT+T12ZkuV6XX6z8Cl3NtSTS4+s11LAC/3 zaMMV0lmKpRHQfH73kkfqzWzfp1MxrrNICLEQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lRVhKK9yhGoV23Ow6VTNbA/LVo5ZgOUa07g3lPnE3vE=; b=t1C+RskPhgdimuLBn0b94mWr93/UAy5OgYokNQGyl/b09EJ9iL22pdFY2aFv9QU0tY gfnyc7TH+lfa9VP6PgZFoJpMAO4E8d2qNeaK0ZdmPxhd1/MOVUO8MijjTWYd4MaZOJGc Zm9tASK3bf+wdrLdnRtHnIeRdaRpCqBdx81zoO08HimUGiTaU2X/8LC8aGArcaWR0C7O +Mrs8Qv/OVwEvcONRbJ7JgiVCbLhYm2jgfJc1FrP2/DIZYNPbxbcKHfEJjGA+p0GUmjd yvq6s0Z15kSBI3fWj2vc9DfnnwUiYyWCAYg/TQxMpmlcJcg1HkMuO2pqPLjkKmzt/vRZ Omgw== X-Gm-Message-State: AFqh2kq5EF9aB/27VIvGTIdlVDFq0ljgzC/P2KKIBs0Omb/mcZ5Sbw17 jl1xyOL/iCFcqCI6Ifh+cB9d7dk1X06FzF67 X-Google-Smtp-Source: AMrXdXvLcYeyFUzb5VBpq7BOGvR7HD8UCPNuZPLJ4Pfvxgwwz6jJQ5NxTd8x2yXSTTyPwdnUinoXGQ== X-Received: by 2002:a05:600c:1d98:b0:3d3:48f4:7a69 with SMTP id p24-20020a05600c1d9800b003d348f47a69mr42045339wms.17.1672847241446; Wed, 04 Jan 2023 07:47:21 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:21 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 09/10] package: Drop support for PACKAGEBUILDPKGD function customisation Date: Wed, 4 Jan 2023 15:47:13 +0000 Message-Id: <20230104154714.1168535-9-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175498 We don't really need the customisation capability of PACKAGEBUILDPKGD since we also have PACKAGE_PREPROCESS_FUNCS which is generally used by recipes. Drop PACKAGEBUILDPKGD to have some simpler/clearer code. This allows some function wrappers to be removed too. Signed-off-by: Richard Purdie --- meta/classes-global/package.bbclass | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-) diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass index d334da6fb3c..7e96601cd95 100644 --- a/meta/classes-global/package.bbclass +++ b/meta/classes-global/package.bbclass @@ -353,14 +353,6 @@ python perform_packagecopy () { perform_packagecopy[cleandirs] = "${PKGD}" perform_packagecopy[dirs] = "${PKGD}" -python fixup_perms () { - oe.package.fixup_perms(d) -} - -python split_and_strip_files () { - oe.package.process_split_and_strip_files(d) -} - python populate_packages () { oe.package.populate_packages(d) } @@ -453,15 +445,9 @@ def gen_packagevar(d, pkgvars="PACKAGEVARS"): ret.append('_exclude_incompatible-%s' % p) return " ".join(ret) -PACKAGE_PREPROCESS_FUNCS ?= "" + # Functions for setting up PKGD -PACKAGEBUILDPKGD ?= " \ - package_prepare_pkgdata \ - perform_packagecopy \ - ${PACKAGE_PREPROCESS_FUNCS} \ - split_and_strip_files \ - fixup_perms \ - " +PACKAGE_PREPROCESS_FUNCS ?= "" # Functions which split PKGD up into separate packages PACKAGESPLITFUNCS ?= " \ package_do_split_locales \ @@ -528,8 +514,12 @@ python do_package () { # Setup PKGD (from D) ########################################################################### - for f in (d.getVar('PACKAGEBUILDPKGD') or '').split(): + bb.build.exec_func("package_prepare_pkgdata", d) + bb.build.exec_func("perform_packagecopy", d) + for f in (d.getVar('PACKAGE_PREPROCESS_FUNCS') or '').split(): bb.build.exec_func(f, d) + oe.package.process_split_and_strip_files(d) + oe.package.fixup_perms(d) ########################################################################### # Split up PKGD into PKGDEST @@ -562,7 +552,7 @@ python do_package () { } do_package[dirs] = "${SHLIBSWORKDIR} ${D}" -do_package[vardeps] += "${PACKAGEBUILDPKGD} ${PACKAGESPLITFUNCS} ${PACKAGEFUNCS} ${@gen_packagevar(d)}" +do_package[vardeps] += "${PACKAGE_PREPROCESS_FUNCS} ${PACKAGESPLITFUNCS} ${PACKAGEFUNCS} ${@gen_packagevar(d)}" addtask package after do_install SSTATETASKS += "do_package" From patchwork Wed Jan 4 15:47:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DB32C4708E for ; Wed, 4 Jan 2023 15:47:24 +0000 (UTC) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mx.groups.io with SMTP id smtpd.web11.15527.1672847238552390024 for ; Wed, 04 Jan 2023 07:47:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=VbwE8yYe; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f46.google.com with SMTP id bk16so20209696wrb.11 for ; Wed, 04 Jan 2023 07:47:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=gWnD3gyDeX37uhk82yfzy2No04rwaIzddPLJ8d18DK0=; b=VbwE8yYeCNBkCSXOI9tJuiwe90JwQY+h1729msSN1RyA5yfdWlLdK521O8jUDKwAyC 7N/N46H62DCYXlx/2fI+ppj+vXAxLSLIC4+GN9PO2bQQX8GpkaVwcU5A7i7V52nSwso9 q1iTy63UJZGxyc8WBI6CNF0aMhwqxxcyX0OfU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gWnD3gyDeX37uhk82yfzy2No04rwaIzddPLJ8d18DK0=; b=PT+Zi3sX4eTYSJwrVMWwQ7bGDTfDB5ONPBy58jeZh4ejGkrG/jCvOaH3BDJGD521C5 VWDD0AKp35aMNe73WwsJA9lMJxq5Sb7Ao2c2E/vcB4tKgil40T8ZJRSP68URpuRAsGeR GouZZbR7lxbg8K5YuwG04NDaShuv5GvtrLv0XbXfbut8ajjpZgXWoozmswaHuI/QiRtI skfiTODBnLV22KpchocHM4NdFoWtxAoplI5Dkuu5YoGcsLDCoVBZisa6EwY3aFVZd78H ZKwvywA+UHQMF9Y9oUYYFIIhem/I/w1+x6HYG1GK7Q7wxE6c/gkkxE5Kz8l5g/979tWr f2/w== X-Gm-Message-State: AFqh2kru+R6V7eDDUb2ZuE0nOroTC5lySmK7yLJelkOgjc22sJ0tcNWG U4/BBqaafJXe89yBnNNkyvK7RD8trY9ZFS2g X-Google-Smtp-Source: AMrXdXv6uWZIM6FAr9okIIZx2aYGFg+sdZKljJ7xCHZtEkas2gFzwmI4DgHkK1CexnlYMayc+V3Y3Q== X-Received: by 2002:a5d:5405:0:b0:284:8a24:59e6 with SMTP id g5-20020a5d5405000000b002848a2459e6mr17247143wrv.3.1672847242120; Wed, 04 Jan 2023 07:47:22 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:fcc2:4e85:26d2:6527]) by smtp.gmail.com with ESMTPSA id u3-20020adff883000000b002423edd7e50sm34812560wrp.32.2023.01.04.07.47.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 07:47:21 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 10/10] recipes/classes: Drop prepend/append usage with PACKAGESPLITFUNCS Date: Wed, 4 Jan 2023 15:47:14 +0000 Message-Id: <20230104154714.1168535-10-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> References: <20230104154714.1168535-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jan 2023 15:47:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/175497 As far as I can tell, none of these uses of PACKAGESPLITFUNCS need append/prepend operators, the standard += and =+ can work just fine. Since OE-Core is copied a lot, use the preferred syntax which is also simpler to parse and change. Signed-off-by: Richard Purdie --- meta/classes-recipe/kernel-module-split.bbclass | 2 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/systemd.bbclass | 2 +- meta/classes-recipe/update-alternatives.bbclass | 2 +- meta/classes-recipe/update-rc.d.bbclass | 2 +- .../wpa-supplicant/wpa-supplicant_2.10.bb | 2 +- meta/recipes-devtools/perl/perl_5.36.0.bb | 2 +- meta/recipes-graphics/mesa/mesa.inc | 2 +- .../gstreamer/gstreamer1.0-plugins-license.inc | 2 +- .../gstreamer/gstreamer1.0-plugins-packaging.inc | 4 ++-- meta/recipes-support/libevent/libevent_2.1.12.bb | 2 +- 11 files changed, 12 insertions(+), 12 deletions(-) diff --git a/meta/classes-recipe/kernel-module-split.bbclass b/meta/classes-recipe/kernel-module-split.bbclass index 08c2e54e860..49437174878 100644 --- a/meta/classes-recipe/kernel-module-split.bbclass +++ b/meta/classes-recipe/kernel-module-split.bbclass @@ -35,7 +35,7 @@ do_install:append() { } KERNEL_SPLIT_MODULES ?= "1" -PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages " +PACKAGESPLITFUNCS =+ "split_kernel_module_packages " KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index 553e17bb4ea..c36632b0a7e 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -705,7 +705,7 @@ pkg_postinst:${KERNEL_PACKAGE_NAME}-base () { fi } -PACKAGESPLITFUNCS:prepend = "split_kernel_packages " +PACKAGESPLITFUNCS =+ "split_kernel_packages " python split_kernel_packages () { do_split_packages(d, root='${nonarch_base_libdir}/firmware', file_regex=r'^(.*)\.(bin|fw|cis|csp|dsp)$', output_pattern='${KERNEL_PACKAGE_NAME}-firmware-%s', description='Firmware for %s', recursive=True, extra_depends='') diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-recipe/systemd.bbclass index f6564c2b310..ae98043d299 100644 --- a/meta/classes-recipe/systemd.bbclass +++ b/meta/classes-recipe/systemd.bbclass @@ -205,7 +205,7 @@ python systemd_populate_packages() { systemd_check_services() } -PACKAGESPLITFUNCS:prepend = "systemd_populate_packages " +PACKAGESPLITFUNCS =+ "systemd_populate_packages " python rm_systemd_unitdir (){ import shutil diff --git a/meta/classes-recipe/update-alternatives.bbclass b/meta/classes-recipe/update-alternatives.bbclass index 970d9bcd450..ec81763c897 100644 --- a/meta/classes-recipe/update-alternatives.bbclass +++ b/meta/classes-recipe/update-alternatives.bbclass @@ -265,7 +265,7 @@ def update_alternatives_alt_targets(d, pkg): return updates -PACKAGESPLITFUNCS:prepend = "populate_packages_updatealternatives " +PACKAGESPLITFUNCS =+ "populate_packages_updatealternatives " python populate_packages_updatealternatives () { if not update_alternatives_enabled(d): diff --git a/meta/classes-recipe/update-rc.d.bbclass b/meta/classes-recipe/update-rc.d.bbclass index cb2aaba57c7..a61d2c62ab4 100644 --- a/meta/classes-recipe/update-rc.d.bbclass +++ b/meta/classes-recipe/update-rc.d.bbclass @@ -68,7 +68,7 @@ python __anonymous() { update_rc_after_parse(d) } -PACKAGESPLITFUNCS:prepend = "${@bb.utils.contains('DISTRO_FEATURES', 'sysvinit', 'populate_packages_updatercd ', '', d)}" +PACKAGESPLITFUNCS =+ "${@bb.utils.contains('DISTRO_FEATURES', 'sysvinit', 'populate_packages_updatercd ', '', d)}" PACKAGESPLITFUNCS:remove:class-nativesdk = "populate_packages_updatercd " populate_packages_updatercd[vardeps] += "updatercd_prerm updatercd_postrm updatercd_postinst" diff --git a/meta/recipes-connectivity/wpa-supplicant/wpa-supplicant_2.10.bb b/meta/recipes-connectivity/wpa-supplicant/wpa-supplicant_2.10.bb index 4e27b92bc69..254cfd3f0c3 100644 --- a/meta/recipes-connectivity/wpa-supplicant/wpa-supplicant_2.10.bb +++ b/meta/recipes-connectivity/wpa-supplicant/wpa-supplicant_2.10.bb @@ -134,4 +134,4 @@ python split_wpa_supplicant_libs () { d.setVar('RRECOMMENDS:' + pn + '-plugins', ' '.join(split_packages)) d.appendVar('RRECOMMENDS:' + pn + '-dbg', ' ' + ' '.join(split_dbg_packages)) } -PACKAGESPLITFUNCS:prepend = "split_wpa_supplicant_libs " +PACKAGESPLITFUNCS += "split_wpa_supplicant_libs " diff --git a/meta/recipes-devtools/perl/perl_5.36.0.bb b/meta/recipes-devtools/perl/perl_5.36.0.bb index 2dc558aaa5f..3458e6a5b05 100644 --- a/meta/recipes-devtools/perl/perl_5.36.0.bb +++ b/meta/recipes-devtools/perl/perl_5.36.0.bb @@ -302,7 +302,7 @@ ALTERNATIVE_LINK_NAME[Thread.3] = "${mandir}/man3/Thread.3" ALLOW_EMPTY:${PN}-modules = "1" PACKAGES += "${PN}-modules " -PACKAGESPLITFUNCS:prepend = "split_perl_packages " +PACKAGESPLITFUNCS =+ "split_perl_packages " python split_perl_packages () { libdir = d.expand('${libdir}/perl5/${PV}') diff --git a/meta/recipes-graphics/mesa/mesa.inc b/meta/recipes-graphics/mesa/mesa.inc index 93b3b59a914..2535dd403da 100644 --- a/meta/recipes-graphics/mesa/mesa.inc +++ b/meta/recipes-graphics/mesa/mesa.inc @@ -291,7 +291,7 @@ python mesa_populate_packages() { do_split_packages(d, pipe_drivers_root, r'^pipe_(.*)\.so$', 'mesa-driver-pipe-%s', 'Mesa %s pipe driver', extra_depends='') } -PACKAGESPLITFUNCS:prepend = "mesa_populate_packages " +PACKAGESPLITFUNCS =+ "mesa_populate_packages " PACKAGES_DYNAMIC += "^mesa-driver-.*" PACKAGES_DYNAMIC:class-native = "^mesa-driver-.*-native" diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-license.inc b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-license.inc index 6638f135404..03595e7d32a 100644 --- a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-license.inc +++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-license.inc @@ -1,7 +1,7 @@ # This .inc file contains functionality for automatically setting # the the license of all plugins according to the GSTREAMER_GPL. -PACKAGESPLITFUNCS:append = " set_gstreamer_license " +PACKAGESPLITFUNCS += " set_gstreamer_license " python set_gstreamer_license () { import oe.utils diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc index afc7be40c8d..837d5a1a627 100644 --- a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc +++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc @@ -9,8 +9,8 @@ # Dynamically generate packages for all enabled plugins PACKAGES_DYNAMIC = "^${PN}-.* ^libgst.*" -PACKAGESPLITFUNCS:prepend = " split_gstreamer10_packages " -PACKAGESPLITFUNCS:append = " set_gstreamer10_metapkg_rdepends " +PACKAGESPLITFUNCS =+ " split_gstreamer10_packages " +PACKAGESPLITFUNCS += " set_gstreamer10_metapkg_rdepends " python split_gstreamer10_packages () { gst_libdir = d.expand('${libdir}/gstreamer-1.0') diff --git a/meta/recipes-support/libevent/libevent_2.1.12.bb b/meta/recipes-support/libevent/libevent_2.1.12.bb index 55a65e6c081..88b81fd56e0 100644 --- a/meta/recipes-support/libevent/libevent_2.1.12.bb +++ b/meta/recipes-support/libevent/libevent_2.1.12.bb @@ -41,7 +41,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*$" python split_libevent_libs () { do_split_packages(d, '${libdir}', r'^libevent_([a-z]*)-.*\.so\..*', '${PN}-%s', '${SUMMARY} (%s)', prepend=True, allow_links=True) } -PACKAGESPLITFUNCS:prepend = "split_libevent_libs " +PACKAGESPLITFUNCS =+ "split_libevent_libs " BBCLASSEXTEND = "native nativesdk"