| Message ID | 20231107151243.10853-1-andrey.popov@yadro.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs | expand |
On 7 Nov 2023, at 15:12, Andrey Popov via lists.openembedded.org <andrey.popov=yadro.com@lists.openembedded.org> wrote: > > Since syslinux is only compatible with platforms that use x86-based > CPUs, this change allows creation of bootable ISO images for other > EFI-compatible platforms by replacing invocation of the isohybrid > tool for those platforms with a python script that creates MBR > partition table with a single entry that points to a bootable > EFI image placed inside the ISO image. I’ll reply with comments on the code shortly, but my initial thoughts are mainly does anyone actually use ISO images still, especially on non-x86 systems? I’m curious what problem you’re trying to solve here. Ross
On Tue, Nov 28, 2023 at 9:55 AM Ross Burton <ross.burton@arm.com> wrote: > > I’ll reply with comments on the code shortly, but my initial thoughts are > mainly does anyone actually use ISO images still, especially on non-x86 > systems? I’m curious what problem you’re trying to solve here. > I can only speak for the projects I am involved in - yes, ISO usage is still *very much* alive and well. Mostly x86 type systems for the time being, but non-x86 is on the horizon. ..Ch:W..
Hello, Ross! Currently, we are using a Yocto-based network equipment software distribution called TanoWRT (https://tano-systems.gitlab.io/meta-tanowrt/). This distribution provides a way to create images that can be used to facilitate network boot process using PXE. Those images are actually ISO images, and they work well for x86-based platforms, since they use already available syslinux stubs that were used for that purpose for a very long time. Unfortunately, syslinux does not support platforms that use CPU of different architectures, which, however, still support network boot process as per the UEFI spec. For those platforms, the image that is used for network boot is still the very same ISO image, the only difference is that it has the 2nd stage bootloader in a standard EFI PE format prepended to it. I hope that my explanation answers you question. Best regards, Andrey.
On 28 Nov 2023, at 19:00, Chuck Wolber <chuckwolber@gmail.com> wrote: > > On Tue, Nov 28, 2023 at 9:55 AM Ross Burton <ross.burton@arm.com> wrote: > > I’ll reply with comments on the code shortly, but my initial thoughts are mainly does anyone actually use ISO images still, especially on non-x86 systems? I’m curious what problem you’re trying to solve here. > > I can only speak for the projects I am involved in - yes, ISO usage is still *very much* alive and well. Mostly x86 type systems for the time being, but non-x86 is on the horizon. And by ISO we’re talking about proper burnt CD or DVDs? Ross
On Tue, Nov 28, 2023 at 11:50 AM Ross Burton <Ross.Burton@arm.com> wrote: > On 28 Nov 2023, at 19:00, Chuck Wolber <chuckwolber@gmail.com> wrote: > > > > On Tue, Nov 28, 2023 at 9:55 AM Ross Burton <ross.burton@arm.com> wrote: > > > > I’ll reply with comments on the code shortly, but my initial thoughts > are mainly does anyone actually use ISO images still, especially on non-x86 > systems? I’m curious what problem you’re trying to solve here. > > > > I can only speak for the projects I am involved in - yes, ISO usage is > still *very much* alive and well. Mostly x86 type systems for the time > being, but non-x86 is on the horizon. > > And by ISO we’re talking about proper burnt CD or DVDs? > Currently yes, but probably moving to dd'ing the ISO to a USB stick in the next few years. ..Ch:W..
On 7 Nov 2023, at 15:12, Andrey Popov via lists.openembedded.org <andrey.popov=yadro.com@lists.openembedded.org> wrote: > @@ -29,8 +29,10 @@ do_bootimg[depends] += "dosfstools-native:do_populate_sysroot \ > mtools-native:do_populate_sysroot \ > cdrtools-native:do_populate_sysroot \ > virtual/kernel:do_deploy \ > - ${MLPREFIX}syslinux:do_populate_sysroot \ > - syslinux-native:do_populate_sysroot \ > + ${@d.getVar('MLPREFIX') + 'syslinux:do_populate_sysroot \ > + syslinux-native:do_populate_sysroot' if __import__('re').match('i.86', d.getVar('TARGET_ARCH')) or \ > + __import__('re').match('x86_64', d.getVar('TARGET_ARCH')) else \ > + 'xorriso-native:do_populate_sysroot'} \ > ${@'%s:do_image_%s' % (d.getVar('PN'), d.getVar('LIVE_ROOTFS_TYPE').replace('-', '_')) if d.getVar('ROOTFS') else ''} \ > " Instead of looking at the TARGET_ARCH, please use the perfectly good machine features that already exist: pcbios and efi. If the target is EFI-only there’s no point in supporting syslinux, and it would be nice to be able to drop it entirely one day in the future. This class shouldn’t be checking the architecture at all, simply check for the efi and pcbios features. This also pulls in a hard-dependency on xorriso which isn’t in oe-core, so that will break the users of the class that just use core. xorriso also doesn’t exist in any layer anymore as I removed it from meta-oe in nanbield, so I know that you’ve not tested this code with master or nanbield. > @@ -71,14 +73,92 @@ MKISOFS_OPTIONS = "-no-emul-boot -boot-load-size 4 -boot-info-table" > BOOTIMG_VOLUME_ID ?= "boot" > BOOTIMG_EXTRA_SPACE ?= "512" > > +def compute_chs(sector_z): > + C = int(sector_z / (63 * 255)) > + H = int((sector_z % (63 * 255)) / 63) > + # convert zero-based sector to CHS format > + S = int(sector_z % 63) + 1 > + # munge accord to partition table format > + S = (S & 0x3f) | (((C >> 8) & 0x3) << 6) > + C = (C & 0xFF) > + return (C, H, S) > + > +def mk_efi_part_table(iso, start, length): > + from struct import pack > + > + # Compute starting and ending CHS addresses for the partition entry. > + (s_C, s_H, s_S) = compute_chs(start) > + (e_C, e_H, e_S) = compute_chs(start + length - 1) > + > + # Write the 66 byte partition table to bytes 0x1BE through 0x1FF in > + # sector 0 of the .ISO. > + # > + # See the partition table format here: > + # http://en.wikipedia.org/wiki/Master_boot_record#Sector_layout > + f = open(iso, 'r+b') > + f.seek(0x1BE) > + f.write(pack("<8BLL48xH", 0x80, s_H, s_S, s_C, > + 0xEF, e_H, e_S, e_C, start, length, 0xAA55)) > + f.close() Why not continue to use syslinux-native to provide isohybrid? My understanding is that isohybrid simply messes with partition tables and doesn’t depend on a target syslinux. Of course if I’m wrong then this needs to be explained in the class: we’ve a problem with complicated classes being written without any explanation of what is happening so lets try to improve the situation instead of making it worse. > +def runtool(cmdln_or_args): > + import subprocess > + > + if isinstance(cmdln_or_args, list): > + cmd = cmdln_or_args[0] > + shell = False > + else: > + import shlex > + cmd = shlex.split(cmdln_or_args)[0] > + shell = True > + > + sout = subprocess.PIPE > + serr = subprocess.STDOUT > + > + try: > + process = subprocess.Popen(cmdln_or_args, stdout=sout, > + stderr=serr, shell=shell) > + sout, serr = process.communicate() > + # combine stdout and stderr, filter None out and decode > + out = ''.join([out.decode('utf-8') for out in [sout, serr] if out]) > + except OSError as err: > + bb.fatal("Cannot run command %s: %s" % (cmd, err)) > + > + return process.returncode, out > + > +def exec_cmd(cmd_and_args): > + args = cmd_and_args.split() > + ret, out = runtool(args) > + out = out.strip() > + if ret != 0: > + bb.fatal("exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \ > + (cmd_and_args, ret, out)) > + return ret, out These two functions are basically a bad reimplementation of subproccess.run(). Don’t copy-paste, just use modern API. Ross
Hello, Ross! Since it's no longer possible to use xorriso in recent versions, I would like to ask if it's possible to apply an updated version of this patch to the kirkstone branch. If so, I will send the new version matching that branch. Best regards, Andrey.
On Wed, Nov 29, 2023 at 7:26 AM Andrey Popov <andrey.popov@yadro.com> wrote: > > Hello, Ross! > > Since it's no longer possible to use xorriso in recent versions, I would like to ask if it's possible to apply an updated version of this patch to the kirkstone branch. > If so, I will send the new version matching that branch. Since this adds a new feature it isn't something that I can take for kirkstone. Steve > > Best regards, Andrey. > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#191459): https://lists.openembedded.org/g/openembedded-core/message/191459 > Mute This Topic: https://lists.openembedded.org/mt/102444305/3620601 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 29 Nov 2023, at 17:26, Andrey Popov via lists.openembedded.org <andrey.popov=yadro.com@lists.openembedded.org> wrote:
> Since it's no longer possible to use xorriso in recent versions, I would like to ask if it's possible to apply an updated version of this patch to the kirkstone branch.
Xorriso has been removed but it was replaced with the separate components, so you don’t need to depend on the entire suite just for one part.
The point that these are in meta-oe still stands, though.
It’s not possible to submit new features to a stable branch without it landing in master first. Otherwise someone (including you) would use the feature in kirkstone, upgrade, and wonder why this feature doesn’t exist.
Ross
Hello, Ross, Steve! Would it be possible to apply backported patch to kirkstone branch if updated patch actually lands for master? Best regards, Andrey.
On Thu, Nov 30, 2023 at 6:20 AM Andrey Popov <andrey.popov@yadro.com> wrote: > > Hello, Ross, Steve! > > Would it be possible to apply backported patch to kirkstone branch if updated patch actually lands for master? Since it is a new feature, no it couldn't be taken for a stable branch. Steve > Best regards, Andrey. > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#191514): https://lists.openembedded.org/g/openembedded-core/message/191514 > Mute This Topic: https://lists.openembedded.org/mt/102444305/3620601 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/meta/classes-recipe/image-live.bbclass b/meta/classes-recipe/image-live.bbclass index 95dd44a..ac7fc1b 100644 --- a/meta/classes-recipe/image-live.bbclass +++ b/meta/classes-recipe/image-live.bbclass @@ -2,15 +2,15 @@ # # SPDX-License-Identifier: MIT -# Creates a bootable image using syslinux, your kernel and an optional +# Creates a bootable image using syslinux (for x86), your kernel and an optional # initrd # # End result is two things: # -# 1. A .hddimg file which is an msdos filesystem containing syslinux, a kernel, -# an initrd and a rootfs image. These can be written to harddisks directly and -# also booted on USB flash disks (write them there with dd). +# 1. A .hddimg file which is an msdos filesystem containing syslinux (for x86), +# a kernel, an initrd and a rootfs image. These can be written to harddisks +# directly and also booted on USB flash disks (write them there with dd). # # 2. A CD .iso image @@ -29,8 +29,10 @@ do_bootimg[depends] += "dosfstools-native:do_populate_sysroot \ mtools-native:do_populate_sysroot \ cdrtools-native:do_populate_sysroot \ virtual/kernel:do_deploy \ - ${MLPREFIX}syslinux:do_populate_sysroot \ - syslinux-native:do_populate_sysroot \ + ${@d.getVar('MLPREFIX') + 'syslinux:do_populate_sysroot \ + syslinux-native:do_populate_sysroot' if __import__('re').match('i.86', d.getVar('TARGET_ARCH')) or \ + __import__('re').match('x86_64', d.getVar('TARGET_ARCH')) else \ + 'xorriso-native:do_populate_sysroot'} \ ${@'%s:do_image_%s' % (d.getVar('PN'), d.getVar('LIVE_ROOTFS_TYPE').replace('-', '_')) if d.getVar('ROOTFS') else ''} \ " @@ -71,14 +73,92 @@ MKISOFS_OPTIONS = "-no-emul-boot -boot-load-size 4 -boot-info-table" BOOTIMG_VOLUME_ID ?= "boot" BOOTIMG_EXTRA_SPACE ?= "512" +def compute_chs(sector_z): + C = int(sector_z / (63 * 255)) + H = int((sector_z % (63 * 255)) / 63) + # convert zero-based sector to CHS format + S = int(sector_z % 63) + 1 + # munge accord to partition table format + S = (S & 0x3f) | (((C >> 8) & 0x3) << 6) + C = (C & 0xFF) + return (C, H, S) + +def mk_efi_part_table(iso, start, length): + from struct import pack + + # Compute starting and ending CHS addresses for the partition entry. + (s_C, s_H, s_S) = compute_chs(start) + (e_C, e_H, e_S) = compute_chs(start + length - 1) + + # Write the 66 byte partition table to bytes 0x1BE through 0x1FF in + # sector 0 of the .ISO. + # + # See the partition table format here: + # http://en.wikipedia.org/wiki/Master_boot_record#Sector_layout + f = open(iso, 'r+b') + f.seek(0x1BE) + f.write(pack("<8BLL48xH", 0x80, s_H, s_S, s_C, + 0xEF, e_H, e_S, e_C, start, length, 0xAA55)) + f.close() + +def runtool(cmdln_or_args): + import subprocess + + if isinstance(cmdln_or_args, list): + cmd = cmdln_or_args[0] + shell = False + else: + import shlex + cmd = shlex.split(cmdln_or_args)[0] + shell = True + + sout = subprocess.PIPE + serr = subprocess.STDOUT + + try: + process = subprocess.Popen(cmdln_or_args, stdout=sout, + stderr=serr, shell=shell) + sout, serr = process.communicate() + # combine stdout and stderr, filter None out and decode + out = ''.join([out.decode('utf-8') for out in [sout, serr] if out]) + except OSError as err: + bb.fatal("Cannot run command %s: %s" % (cmd, err)) + + return process.returncode, out + +def exec_cmd(cmd_and_args): + args = cmd_and_args.split() + ret, out = runtool(args) + out = out.strip() + if ret != 0: + bb.fatal("exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \ + (cmd_and_args, ret, out)) + return ret, out + +def install_efi_part_table(iso_img): + find_efi_img_cmd = "xorriso -indev %s -find /efi.img \ + -name efi.img -exec report_lba --" % iso_img + ret, out = exec_cmd(find_efi_img_cmd) + efi_img_start = -1 + efi_img_length = -1 + for line in out.split("\n"): + if "File data lba:" in line and "/efi.img" in line: + file_stat = line[14:].split(',') + efi_img_start = int(file_stat[1].strip()) * 4 + efi_img_length = int(int(file_stat[3].strip()) / 512) + break + if (efi_img_start < 0) or (efi_img_length < 0): + bb.fatal("Failed to determine /efi.img attributes") + mk_efi_part_table(iso_img, efi_img_start, efi_img_length) + populate_live() { - populate_kernel $1 + populate_kernel $1 if [ -s "${ROOTFS}" ]; then install -m 0644 ${ROOTFS} $1/rootfs.img fi } -build_iso() { +build_iso_base() { # Only create an ISO if we have an INITRD and the live or iso image type was selected if [ -z "${INITRD}" ] || [ "${@bb.utils.contains_any('IMAGE_FSTYPES', 'live iso', '1', '0', d)}" != "1" ]; then bbnote "ISO image will not be created." @@ -104,11 +184,13 @@ build_iso() { fi # EFI only - if [ "${PCBIOS}" != "1" ] && [ "${EFI}" = "1" ] ; then - # Work around bug in isohybrid where it requires isolinux.bin - # In the boot catalog, even though it is not used - mkdir -p ${ISODIR}/${ISOLINUXDIR} - install -m 0644 ${STAGING_DATADIR}/syslinux/isolinux.bin ${ISODIR}${ISOLINUXDIR} + if [ -n "$(echo ${TARGET_ARCH} | grep -e i.86)" ] || [ "${TARGET_ARCH}" = "x86_64" ]; then + if [ "${PCBIOS}" != "1" ] && [ "${EFI}" = "1" ]; then + # Work around bug in isohybrid where it requires isolinux.bin + # In the boot catalog, even though it is not used + mkdir -p ${ISODIR}/${ISOLINUXDIR} + install -m 0644 ${STAGING_DATADIR}/syslinux/isolinux.bin ${ISODIR}${ISOLINUXDIR} + fi fi # We used to have support for zisofs; this is a relic of that @@ -128,7 +210,15 @@ build_iso() { fi fi - if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ] ; then + if [ -z "$(echo ${TARGET_ARCH} | grep -e i.86)" ] && [ "${TARGET_ARCH}" != "x86_64" ]; then + mkisofs -A ${BOOTIMG_VOLUME_ID} -V ${BOOTIMG_VOLUME_ID} \ + -o ${IMGDEPLOYDIR}/${IMAGE_NAME}.iso \ + $mkisofs_compress_opts $mkisofs_iso_level \ + ${ISODIR} + return + fi + + if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ]; then # PCBIOS only media mkisofs -V ${BOOTIMG_VOLUME_ID} \ -o ${IMGDEPLOYDIR}/${IMAGE_NAME}.iso \ @@ -150,6 +240,17 @@ build_iso() { isohybrid $isohybrid_args ${IMGDEPLOYDIR}/${IMAGE_NAME}.iso } +python build_iso() { + import re + + bb.build.exec_func("build_iso_base", d) + target_arch = d.getVar("TARGET_ARCH") + if not re.match("i.86", target_arch) and not re.match("x86_64", target_arch) \ + and d.getVar("EFI") == "1": + install_efi_part_table(d.getVar("IMGDEPLOYDIR") + "/" + \ + d.getVar("IMAGE_NAME") + ".iso") +} + build_fat_img() { FATSOURCEDIR=$1 FATIMG=$2
Since syslinux is only compatible with platforms that use x86-based CPUs, this change allows creation of bootable ISO images for other EFI-compatible platforms by replacing invocation of the isohybrid tool for those platforms with a python script that creates MBR partition table with a single entry that points to a bootable EFI image placed inside the ISO image. Signed-off-by: Andrey Popov <andrey.popov@yadro.com> --- meta/classes-recipe/image-live.bbclass | 129 ++++++++++++++++++++++--- 1 file changed, 115 insertions(+), 14 deletions(-)