diff mbox series

[v2] wic: Configurable addressing of GPT main table

Message ID 20240208115017.23047-1-davide.cardillo@seco.com
State New
Headers show
Series [v2] wic: Configurable addressing of GPT main table | expand

Commit Message

Davide Cardillo Feb. 8, 2024, 11:50 a.m. UTC
Issue:
the usage of a gpt table with i.MX processor is not compatible by default.
This is due to a memory conflict: the main partition table is placed into
the same memory space used to flash the header of the U-Boot. This last
can not be moved since the ROM code of the i.MX6 starts to read to that
fixed address (0x400).
This problem can be solved using a feature of the GPT partition table for
witch the main table can be moved to a desired memory area.
The WIC library present in poky manages the GPT table through some
standard Linux commands (gdisk and sgdisk) which ones have already the
opportunity to perform this task.
This patch introduces this feature with the addition of some wic's options,
used also in wks file.

New partition placement behavior:
Since now all partitions are placed after the gpt table presents into the
first 1KB of memory (including both the gpt header at the first byte of
memory and the main partition table staring at 0x400).
Now the main gpt table can be placed in other address spaces and a partition
can be placed also before the gpt table. Recap:
- by default, all partitions are placed sequentially, one after the other.
The first partition implicitly starts after the GPT partition table.
- adding a flag to the item of a partition into the wks file, the partition
can be placed, starting from a fixed memory offset (neglecting the memory
space occupied by the GPT table).
Combining the moving of the main partition table and the absolute (not yet
relative) positioning of a partition, now it is possible to place one or more
   partitions before the GPT table.

New flags:
- main-ptable-offset (size in KB): to add as property of the bootloader item.
Specifies the main partition table location in KB.
- fixed-align (boolean): to add as property of a partition. Specifies that the
partition must to placed with absolute offset, used in conjunction with --align
property

Example:
setting used for a wic for an i.MX6 processor, where the main parttion table is
placed with absolute offset of 4MB

bootloader --ptable gpt --main-ptable-offset 4096K
part / --source rawcopy --sourceparams="file=u-boot.imx" --ondisk mmcblk --align 1 --size 1M --no-table --fixed-align

Test environment:
This patch is a part of Edgehog OS (https://git.seco.com/edgehog) a
custom Yocto distribution by SECO SpA (www.seco.com), used on multiple
architectures (both ARM and x86).
This solution is widely used by all SECO customers who use a hardware
solution based on NXP i.MX6 processors.

Signed-off-by: Davide Cardillo <davide.cardillo@seco.com>
Reviewed-by: Tobias Poganiuch <tobias.poganiuch@seco.com>
Tested-by: Andrea Da Col <andrea.dacol@seco.com>
Tested-by: Daniel Rinaldi <daniel.rinaldi@seco.com>
---
 scripts/lib/wic/ksparser.py              |   2 +
 scripts/lib/wic/partition.py             |   1 +
 scripts/lib/wic/plugins/imager/direct.py | 101 +++++++++++++++++++++--
 3 files changed, 96 insertions(+), 8 deletions(-)

Comments

Richard Purdie Feb. 8, 2024, 11:56 a.m. UTC | #1
On Thu, 2024-02-08 at 12:50 +0100, Davide Cardillo via
lists.openembedded.org wrote:
> Issue:
> the usage of a gpt table with i.MX processor is not compatible by default.
> This is due to a memory conflict: the main partition table is placed into
> the same memory space used to flash the header of the U-Boot. This last
> can not be moved since the ROM code of the i.MX6 starts to read to that
> fixed address (0x400).
> This problem can be solved using a feature of the GPT partition table for
> witch the main table can be moved to a desired memory area.
> The WIC library present in poky manages the GPT table through some
> standard Linux commands (gdisk and sgdisk) which ones have already the
> opportunity to perform this task.
> This patch introduces this feature with the addition of some wic's options,
> used also in wks file.
> 
> New partition placement behavior:
> Since now all partitions are placed after the gpt table presents into the
> first 1KB of memory (including both the gpt header at the first byte of
> memory and the main partition table staring at 0x400).
> Now the main gpt table can be placed in other address spaces and a partition
> can be placed also before the gpt table. Recap:
> - by default, all partitions are placed sequentially, one after the other.
> The first partition implicitly starts after the GPT partition table.
> - adding a flag to the item of a partition into the wks file, the partition
> can be placed, starting from a fixed memory offset (neglecting the memory
> space occupied by the GPT table).
> Combining the moving of the main partition table and the absolute (not yet
> relative) positioning of a partition, now it is possible to place one or more
>    partitions before the GPT table.
> 
> New flags:
> - main-ptable-offset (size in KB): to add as property of the bootloader item.
> Specifies the main partition table location in KB.
> - fixed-align (boolean): to add as property of a partition. Specifies that the
> partition must to placed with absolute offset, used in conjunction with --align
> property
> 
> Example:
> setting used for a wic for an i.MX6 processor, where the main parttion table is
> placed with absolute offset of 4MB
> 
> bootloader --ptable gpt --main-ptable-offset 4096K
> part / --source rawcopy --sourceparams="file=u-boot.imx" --ondisk mmcblk --align 1 --size 1M --no-table --fixed-align
> 
> Test environment:
> This patch is a part of Edgehog OS (https://git.seco.com/edgehog) a
> custom Yocto distribution by SECO SpA (www.seco.com), used on multiple
> architectures (both ARM and x86).
> This solution is widely used by all SECO customers who use a hardware
> solution based on NXP i.MX6 processors.

This looks like something we should add some kind of test case for to
lib/oeqa/selftest/cases/wic* ?

Cheers,

Richard
diff mbox series

Patch

diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 7ef3dc83dd..e9c83cf1a3 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -148,6 +148,7 @@  class KickStart():
         part = subparsers.add_parser('part')
         part.add_argument('mountpoint', nargs='?')
         part.add_argument('--active', action='store_true')
+        part.add_argument('--fixed-align', action='store_true')
         part.add_argument('--align', type=int)
         part.add_argument('--offset', type=sizetype("K", True))
         part.add_argument('--exclude-path', nargs='+')
@@ -197,6 +198,7 @@  class KickStart():
                                 default='msdos')
         bootloader.add_argument('--timeout', type=int)
         bootloader.add_argument('--source')
+        bootloader.add_argument('--main-ptable-offset', type=sizetype("K"), default=0)
 
         include = subparsers.add_parser('include')
         include.add_argument('path', type=cannedpathtype)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 795707ec5d..953ff1c9c6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -26,6 +26,7 @@  class Partition():
         self.args = args
         self.active = args.active
         self.align = args.align
+        self.fixed_align = args.fixed_align
         self.disk = args.disk
         self.device = None
         self.extra_space = args.extra_space
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 9b619e41c1..5b0beac1eb 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -16,6 +16,7 @@  import random
 import shutil
 import tempfile
 import uuid
+import operator
 
 from time import strftime
 
@@ -66,6 +67,7 @@  class DirectPlugin(ImagerPlugin):
         self.workdir = self.setup_workdir(options.workdir)
         self._image = None
         self.ptable_format = self.ks.bootloader.ptable
+        self.main_ptable_offset = self.ks.bootloader.main_ptable_offset
         self.parts = self.ks.partitions
 
         # as a convenience, set source to the boot partition source
@@ -77,6 +79,7 @@  class DirectPlugin(ImagerPlugin):
 
         image_path = self._full_path(self.workdir, self.parts[0].disk, "direct")
         self._image = PartitionedImage(image_path, self.ptable_format,
+                                       self.main_ptable_offset,
                                        self.parts, self.native_sysroot,
                                        options.extra_space)
 
@@ -295,15 +298,27 @@  GPT_OVERHEAD = 34
 # Size of a sector in bytes
 SECTOR_SIZE = 512
 
+class memoryRegion():
+    def __init__(self):
+        self.start_sector = 1
+        self.end_sector = 1
+        self.part_num = 0
+
+    def __init__(self, part_num, start_sec, end_sec):
+        self.part_num = part_num
+        self.start_sector = start_sec
+        self.end_sector = end_sec
+
 class PartitionedImage():
     """
     Partitioned image in a file.
     """
 
-    def __init__(self, path, ptable_format, partitions, native_sysroot=None, extra_space=0):
+    def __init__(self, path, ptable_format, main_ptable_offset, partitions, native_sysroot=None, extra_space=0):
         self.path = path  # Path to the image file
         self.numpart = 0  # Number of allocated partitions
         self.realpart = 0 # Number of partitions in the partition table
+        self.fixedalign_part = 0 # Number of partitions with absolute alignment
         self.primary_part_num = 0  # Number of primary partitions (msdos)
         self.extendedpart = 0      # Create extended partition before this logical partition (msdos)
         self.extended_size_sec = 0 # Size of exteded partition (msdos)
@@ -312,6 +327,7 @@  class PartitionedImage():
         self.min_size = 0 # Minimum required disk size to fit
                           # all partitions (in bytes)
         self.ptable_format = ptable_format  # Partition table format
+        self.main_ptable_offset = main_ptable_offset # for GPT table, offset
         # Disk system identifier
         if os.getenv('SOURCE_DATE_EPOCH'):
             self.identifier = random.Random(int(os.getenv('SOURCE_DATE_EPOCH'))).randint(1, 0xffffffff)
@@ -320,6 +336,7 @@  class PartitionedImage():
 
         self.partitions = partitions
         self.partimages = []
+        self.used_memory_region = []
         # Size of a sector used in calculations
         self.sector_size = SECTOR_SIZE
         self.native_sysroot = native_sysroot
@@ -372,6 +389,30 @@  class PartitionedImage():
             # Converting kB to sectors for parted
             part.size_sec = part.disk_size * 1024 // self.sector_size
 
+    def _add_busy_region(self, part_num, start_sec, end_sec):
+        self.used_memory_region.append(memoryRegion(part_num, start_sec, end_sec))
+        self.used_memory_region = sorted(self.used_memory_region, key=operator.attrgetter('start_sector'))
+
+    def _print_busy_region_list(self):
+        list = "Busy memory region (in sector)\n"
+        list += "\tpart#\tstart\tEnd\n"
+        for num in range(len(self.used_memory_region)):
+            list +="\t%s\t%s\t%s\n" % (self.used_memory_region[num].part_num, \
+                self.used_memory_region[num].start_sector, self.used_memory_region[num].end_sector)
+        logger.debug(list)
+
+    def _check_memory_region(self, start_sec, end_sec ):
+        """ Check if the specified region is already flagges as used. Returns
+        zero if not used, 1 otherwise"""
+        for num in range(len(self.used_memory_region)):
+            region = self.used_memory_region[num]
+            if ( ( region.start_sector < start_sec and start_sec < region.end_sector) or \
+                    ( region.start_sector < end_sec and end_sec < region.end_sector) ):
+                return 1
+            if ( start_sec < region.start_sector and end_sec > region.end_sector ):
+                return 1
+        return 0
+
     def layout_partitions(self):
         """ Layout the partitions, meaning calculate the position of every
         partition on the disk. The 'ptable_format' parameter defines the
@@ -383,6 +424,35 @@  class PartitionedImage():
         # partitions not listed in the table are not included.
         num_real_partitions = len([p for p in self.partitions if not p.no_table])
 
+        # The partitions flagged as fixed alignment partition are placed into image
+        # before the others. Instead, relative-aligned partitions are placed in a
+        # serialized fashion, with control of the intersection of memory areas.
+        # The base address of the relative-aligned partitions is, as before,
+        # the maximum address accupied by the partition table
+
+        # Flag as used the momoery required for partition table
+        self._add_busy_region(-1, 0, MBR_OVERHEAD)
+        if self.ptable_format == "gpt":
+            offset = self.main_ptable_offset * 1024 // self.sector_size
+            self._add_busy_region(-1, offset, offset + GPT_OVERHEAD)
+
+        # Search all partition with fixed position into the image
+        for num in range(len(self.partitions)):
+            part = self.partitions[num]
+            if not part.fixed_align:
+                continue
+            if not part.no_table:
+                raise WicError("A partition with fixed alignment must have no_table flag set")
+
+            start_sector = (part.align * 1024 // self.sector_size)
+            end_sector = start_sector + part.size_sec - 1
+            if self._check_memory_region(start_sector, end_sector) == 1:
+                self._print_busy_region_list()
+                raise WicError("A partition wants to use an already used memory region (sectors %d - %d)" \
+                    % (start_sector, end_sector))
+            self._add_busy_region(num, start_sector, end_sector)
+            part.start = start_sector
+
         # Go through partitions in the order they are added in .ks file
         for num in range(len(self.partitions)):
             part = self.partitions[num]
@@ -411,7 +481,7 @@  class PartitionedImage():
                 if self.ptable_format == "msdos":
                     overhead = MBR_OVERHEAD
                 elif self.ptable_format in ("gpt", "gpt-hybrid"):
-                    overhead = GPT_OVERHEAD
+                    overhead = (self.main_ptable_offset * 1024 // self.sector_size) + GPT_OVERHEAD
 
                 # Skip one sector required for the partitioning scheme overhead
                 self.offset += overhead
@@ -426,7 +496,7 @@  class PartitionedImage():
                     self.offset += 2
 
             align_sectors = 0
-            if part.align:
+            if not part.fixed_align and part.align:
                 # If not first partition and we do have alignment set we need
                 # to align the partition.
                 # FIXME: This leaves a empty spaces to the disk. To fill the
@@ -448,7 +518,7 @@  class PartitionedImage():
                     # increase the offset so we actually start the partition on right alignment
                     self.offset += align_sectors
 
-            if part.offset is not None:
+            if not part.fixed_align and part.offset is not None:
                 offset = part.offset // self.sector_size
 
                 if offset * self.sector_size != part.offset:
@@ -463,14 +533,20 @@  class PartitionedImage():
 
                 self.offset = offset
 
-            part.start = self.offset
-            self.offset += part.size_sec
-
             if not part.no_table:
                 part.num = self.realpart
             else:
                 part.num = 0
 
+            if not part.fixed_align:
+                part.start = self.offset
+                self.offset += part.size_sec
+                if self._check_memory_region(part.start, self.offset) == 1:
+                    self._print_busy_region_list()
+                    raise WicError("A partition wants to use an already used memory region (sectors %d - %d)" \
+                        % (part.start, self.offset))
+                self._add_busy_region(part.num, part.start, self.offset)
+
             if self.ptable_format == "msdos" and not part.no_table:
                 if part.type == 'logical':
                     self.logical_part_cnt += 1
@@ -491,11 +567,13 @@  class PartitionedImage():
                          part.num, part.start, self.offset - 1, part.size_sec,
                          part.size_sec * self.sector_size)
 
+        self._print_busy_region_list()
+
         # Once all the partitions have been layed out, we can calculate the
         # minumim disk size
         self.min_size = self.offset
         if self.ptable_format in ("gpt", "gpt-hybrid"):
-            self.min_size += GPT_OVERHEAD
+            self.min_size += (self.main_ptable_offset * 1024 // self.sector_size) + GPT_OVERHEAD
 
         self.min_size *= self.sector_size
         self.min_size += self.extra_space
@@ -662,6 +740,13 @@  class PartitionedImage():
                     mbr = mbr_file.read(512)
                     image_file.write(mbr)
 
+        if self.ptable_format == "gpt" and self.main_ptable_offset > 0:
+            main_ptable_sectors = self.main_ptable_offset * 1024 // self.sector_size
+            logger.debug("Move the main GPT partition table forward by %s sector(s)", main_ptable_sectors)
+            cmd = "sgdisk -j %d %s" % (main_ptable_sectors, self.path)
+            #cmd = "(echo -e 'x'; echo -e 'j' ; echo -e '%d'; echo -e 'w'; echo -e 'Y') | gdisk %s" % (main_ptable_sectors, self.path)
+            exec_native_cmd(cmd, self.native_sysroot)
+
     def cleanup(self):
         pass