[v3] classes: Add a new bbclass that abstracts the generation of FIT blobs

Submitted by Nandor Han on March 27, 2020, 8:29 a.m. | Patch ID: 171365

Details

Message ID 20200327082918.1213345-1-nandor.han@vaisala.com
State Superseded
Headers show

Commit Message

Nandor Han March 27, 2020, 8:29 a.m.
FIT format is very versatile allowing various combination of booting
sequences. In the same time different U-Boot boot stages can use FIT
blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
FIT blob). Because of the allowed level of customization, the generation
of a FIT blob using a fixed image tree source, becomes challenging and
increase the level of complexity where different configurations and
combinations are needed.

This bbclass will know how to generate a FIT blob, leaving the mechanics
of the process (dependencies, task order...) to be handled by the users
of the bbclass. In the same time will allow to separate the knowledge of
the FIT format leaving the user code cleaner and more readable.

Signed-off-by: Nandor Han <nandor.han@vaisala.com>
---

Notes:
    Testing
    -------

    1. linux-yocto_5.4.bbappend was modified to have the following configuration:

    ```
    inherit fit-image

    KERNEL_IMAGE_NODE[name] = "kernel"
    KERNEL_IMAGE_NODE[description] = "${PF}"
    KERNEL_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/zImage")'
    KERNEL_IMAGE_NODE[type] = "kernel"
    KERNEL_IMAGE_NODE[arch] = "${ARCH}"
    KERNEL_IMAGE_NODE[os] = "linux"
    KERNEL_IMAGE_NODE[compression] = "none"
    KERNEL_IMAGE_NODE[load] = "${UBOOT_LOADADDRESS}"
    KERNEL_IMAGE_NODE[entry] = "${UBOOT_ENTRYPOINT}"
    KERNEL_IMAGE_NODE[hash] = "sha256"

    FDT_IMAGE_NODE[name] = "fdt"
    FDT_IMAGE_NODE[description] = "FDT blob"
    FDT_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/dts/am335x-bone.dtb")'
    FDT_IMAGE_NODE[type] = "flat_dt"
    FDT_IMAGE_NODE[arch] = "${ARCH}"
    FDT_IMAGE_NODE[compression] = "none"
    FDT_IMAGE_NODE[hash] = "sha256"

    CONF1_CONF_NODE[name] = "conf"
    CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
    CONF1_CONF_NODE[kernel] = "kernel"
    CONF1_CONF_NODE[fdt] = "fdt"

    FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
    FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
    FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_NODE', 'name') or ""}"
    ```
    2. Build the kernel: `bitbake virtual/kernel`
    3. Verify that `image-fit.itb` is present in the build directory: PASS
    4. Disassemble the image using the command: `dtc -I dtb -O dts image-fit.itb`
    5. Verify that the FIT source contains the expected configuration: PASS

    Changes since v1:
    ----------------
    - Change the format of short-log to "<target>: <summary>"

    Changes since v2:
    ----------------
    - rename the file from `fit-image` to `fit_image` to
    successfully export the class functions.
    - adding new sanity checks.
    - add missing dependency.
    - fix a variable reference in a debug log.

 meta/classes/fit_image.bbclass | 382 +++++++++++++++++++++++++++++++++
 1 file changed, 382 insertions(+)
 create mode 100644 meta/classes/fit_image.bbclass

Patch hide | download patch | download mbox

diff --git a/meta/classes/fit_image.bbclass b/meta/classes/fit_image.bbclass
new file mode 100644
index 0000000000..87d92db122
--- /dev/null
+++ b/meta/classes/fit_image.bbclass
@@ -0,0 +1,382 @@ 
+#
+# The class will facilitate the generation of FIT blobs.
+#
+# Glossary
+#    FIT - Flattened uImage Tree
+#
+# Requirements:
+#
+#    * The user need to specify the image content using the format specified in the "FIT Image API" section.
+#
+# FIT Image API
+#
+# The bbclass is using variable and variable flags to declare the FIT image content.
+#
+#    * Sub-Images and Configuration Nodes
+#
+#       ** A sub-image node content is declared using the format `VAR_NODE[<property-name>] = <value>`.
+#         * VAR_NODE - freely selected name of the variable representing a node.
+#         * <property-name> - a sub-image property (e.g. description, type...).
+#         * <value> - the property value.
+#             Depending of the property the value can support different formats.
+#           ** Property Values Formats
+#
+#            string property
+#            ---------------
+#            format: "<text>" - in case the property expects a text.
+#                (e.g. IMAGE_NODE_KERNEL[type] = "kernel")
+#
+#            address property
+#            ----------------
+#            format: "<address>" - in case the property expects an address.
+#                (e.g. IMAGE_NODE_KERNEL[entry] = "0xABCDABCD")
+#
+#            hash property
+#            -------------
+#            format: "<hash type>" - for hash property the hash type needs to be specified.
+#                (e.g. IMAGE_NODE_KERNEL[hash] = "sha256")
+#
+#            sub-image signature property
+#            ----------------------------
+#            format: "<algo>;<key-name-hint>;" - for image signature node.
+#                Both algorithm and key name needs to be provided.
+#                (e.g. IMAGE_NODE_KERNEL[signature] = "sha256,rsa2048;kernel;"
+#
+#            configuration signature property
+#            --------------------------------
+#            format: "<algo>;<key-name-hint>;<sub-image list>" - for configuration signature properties algorithm,
+#                key name and sub-image nodes needs to be provided.
+#                (e.g. CONF_NODE_CONF1[signature] = "sha256,rsa2048;kernel;"kernel","fdt";")
+#
+#       ** Sub-Image and Configuration Nodes Flags
+#              See the code for supported flags.
+#
+#    * FIT_IMAGES_NODE - contains a list of variables used to declare the sub-images nodes, separated by space.
+#                    (e.g. FIT_IMAGES_NODE = "IMAGE_NODE_KERNEL IMAGE_NODE_FDT")
+#
+#    * FIT_CONFIGURATIONS_NODE - contains a list of variables used to declare the configuration nodes,
+#          separated by space. (e.g. FIT_CONFIGURATIONS_NODE = "CONF_NODE_CONF1")
+#        ** Flags
+#           - "default": used to configure the default configuration node.
+#                 (e.g. FIT_CONFIGURATIONS_NODE[default] = "conf@0")
+#
+# Example:
+# This is part of a linux_%.bbappend recipe.
+#
+# KERNEL_IMAGE_NODE[name] = "kernel"
+# KERNEL_IMAGE_NODE[description] = "${PF}"
+# KERNEL_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/zImage")'
+# KERNEL_IMAGE_NODE[type] = "kernel"
+# KERNEL_IMAGE_NODE[arch] = "${ARCH}"
+# KERNEL_IMAGE_NODE[os] = "linux"
+# KERNEL_IMAGE_NODE[compression] = "none"
+# KERNEL_IMAGE_NODE[load] = "${UBOOT_LOADADDRESS}"
+# KERNEL_IMAGE_NODE[entry] = "${UBOOT_ENTRYPOINT}"
+# KERNEL_IMAGE_NODE[hash] = "sha256"
+#
+# FDT_IMAGE_NODE[name] = "fdt"
+# FDT_IMAGE_NODE[description] = "FDT blob"
+# FDT_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/dts/am335x-bone.dtb")'
+# FDT_IMAGE_NODE[type] = "flat_dt"
+# FDT_IMAGE_NODE[arch] = "${ARCH}"
+# FDT_IMAGE_NODE[compression] = "none"
+# FDT_IMAGE_NODE[hash] = "sha256"
+#
+# CONF1_CONF_NODE[name] = "conf"
+# CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
+# CONF1_CONF_NODE[kernel] = "kernel"
+# CONF1_CONF_NODE[fdt] = "fdt"
+#
+# FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
+# FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
+# FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_NODE', 'name') or ""}"
+#
+
+DEPENDS += "\
+    dtc-native \
+    u-boot-mkimage-native \
+"
+
+FIT_IMAGE_DESCRIPTION ??= "Generic FIT image"
+FIT_IMAGE_FILENAME ??= "image-fit"
+FIT_IMAGE_UBOOT_MKIMAGE_OPTS ??= ""
+
+def get_subimage_node_rules():
+    """
+    Defines the properties format and validation for sub-image nodes.
+
+    :return: Return a dictionary with the format rules.
+    """
+
+    #
+    # Rules Format: [Mandatory, Template String, Dictionary keys]
+    # Mandatory: True, False - used to verify if this property is mandatory for generating the image.
+    #            Note: Doesn't take in consideration the conditionally mandatory property.
+    #                  (see: U-Boot/doc/uImage.FIT/source_file_format.txt)
+    # Template String: Property content format. Used to generate the parameter content.
+    # Dictionary Keys: Keys used to be replaced in the Template String.
+    #
+    from collections import OrderedDict
+
+    rules = OrderedDict()
+
+    rules['description'] = [True, '= "$value"', ["value"]]
+    rules['data'] = [True, '= $value', ["value"]]
+    rules['type'] = [True, '= "$value"', ["value"]]
+    rules['arch'] = [False, '= "$value"', ["value"]]
+    rules['os'] = [False, '= "$value"', ["value"]]
+    rules['compression'] = [True, '= "$value"', ["value"]]
+    rules['load'] = [False, '= <$value>', ["value"]]
+    rules['entry'] = [False, '= <$value>', ["value"]]
+    rules['hash'] = [False, '{ algo = "$algo"; }', ['algo']]
+    rules['signature'] = [False, '{ algo = "$algo"; key-name-hint = "$key"; }', ['algo', 'key']]
+
+    return rules
+
+
+def get_conf_node_rules():
+    """
+    Defines the properties format and validation for configuration nodes.
+
+    :return: Return a dictionary with the format rules.
+    """
+    #
+    # Rules Format: [Mandatory, Template String, Dictionary keys]
+    # Mandatory: True, False - used to verify if this property is mandatory for generating the image.
+    #            Note: Doesn't take in consideration the conditionally mandatory property.
+    #                  (see: U-Boot/doc/uImage.FIT/source_file_format.txt)
+    # Template String: Property content format. Used to generate the parameter content.
+    # Dictionary Keys: Keys used to be replaced in the Template String.
+    #
+    from collections import OrderedDict
+
+    rules = OrderedDict()
+
+    rules['description'] = [True, '= "$value"', ["value"]]
+    rules['kernel'] = [True, '= "$value"', ["value"]]
+    rules['ramdisk'] = [False, '= "$value"', ["value"]]
+    rules['fdt'] = [False, '= "$value"', ["value"]]
+    rules['loadables'] = [False, '= "$value"', ["value"]]
+    rules['signature'] = [False, '{ algo = "$algo"; key-name-hint = "$key"; sign-images = $images; }',
+                          ['algo', 'key', 'images']]
+
+    return rules
+
+
+def generate_node(name, params, rules):
+    """
+    Generates a node.
+
+    :param name: Node name.
+    :param params: A dictionary containing the properties values.
+    :param rules: A dictionary containing the properties values validation and format.
+
+    :return: A string containing the node, including the new line characters.
+    """
+    from string import Template
+
+    content = []
+
+    for rule in rules.keys():
+        if rule in params.keys():
+            content.append('{param} {value}; '.format(
+                param=rule,
+                value=Template(rules[rule][1]).substitute(
+                    dict(zip(rules[rule][2], params[rule].split(';'))))))
+        elif rules[rule][0]:
+            bb.fatal('Missing mandatory parameter "{param}" from "{section}" section'.format(param=rule, section=name))
+
+    content = """
+                     """.join(content)
+    node = """   {name} {{
+                     {content}
+              }};
+           """.format(name=name, content=content)
+
+    return node
+
+
+def get_section_configuration(var, d):
+    """
+    Generates a string build from variable's flags.
+
+    :param var: variable to extract the flags.
+    :param d: bitbake environment.
+
+    :return: A string with the format '<flag1> = <value1>; <flag2> = <value2>;...'.
+    """
+    flags = d.getVarFlags(var)
+    if flags is not None:
+        flags = dict((flag, d.expand(value))
+                     for flag, value in list(flags.items()))
+    else:
+        flags = {}
+
+    configuration = ''.join((
+              """{name} = "{value}";
+              """.format(name=name, value=value) for name, value in flags.items()))
+
+    return configuration
+
+
+def get_section_properties(var, d):
+    """
+    Extract the nodes and parameters for a section.
+
+    :param var: variable containing the variable names of the nodes that are part of this section.
+    :param d: bitbake environment.
+
+    :return: a list containing dictionaries with section nodes parameters.
+    """
+    nodes = []
+    parameters = {}
+
+    for node in d.getVar(var).split():
+        parameters = d.getVarFlags(node)
+        if parameters is not None:
+            parameters = dict((parameter, d.expand(value))
+                              for parameter, value in list(parameters.items()))
+            nodes.append(parameters)
+
+    return nodes
+
+
+def generate_section(var, rules, d):
+    """
+    Generates a section node (configuration or sub-image).
+
+    :param var: Variable to extract the node names.
+    :param rules: Rules to use for generating this section.
+    :param d: bitbake environment.
+
+    :return: A string containing the section, including the new line characters.
+    """
+
+    section = get_section_configuration(var, d)
+
+    nodes_parameters = get_section_properties(var, d)
+    for parameters in nodes_parameters:
+        name = parameters.pop('name')
+        node = generate_node(name, parameters, rules)
+        section += node
+
+    return section
+
+
+def get_fit_image_template():
+    """
+    Get the FIT format.
+
+    :return: A Template string containing the FIT image format.
+    """
+    from string import Template
+
+    template = Template("""/dts-v1/;
+        /{
+             description = "$description";
+             #address-cells = <1>;
+             images {
+                 $images_section
+             };
+             configurations {
+                 $configurations_section
+             };
+        };""")
+    return template
+
+
+def generate_image_tree_source(d):
+    """
+    Generates a string containing the image tree source.
+
+    :return: A string representing the image tree.
+    """
+    from string import Template
+
+    values = {}
+    values['description'] = d.getVar('FIT_IMAGE_DESCRIPTION')
+
+    image_rules = get_subimage_node_rules()
+    if d.getVar('FIT_IMAGES_NODE', False) is None:
+       bb.fatal("Please add the FIT image nodes to FIT_IMAGES_NODE variable.")
+    values['images_section'] = generate_section('FIT_IMAGES_NODE', image_rules, d)
+
+    conf_rules = get_conf_node_rules()
+    if d.getVar('FIT_CONFIGURATIONS_NODE', False) is None:
+       bb.fatal("Please add the FIT configuration nodes to FIT_CONFIGURATIONS_NODE variable.")
+    values['configurations_section'] = generate_section('FIT_CONFIGURATIONS_NODE', conf_rules, d)
+
+    image_tree_source = get_fit_image_template().substitute(values)
+
+    return image_tree_source
+
+
+def generate_image_blob(file_name, image, d):
+    """
+    Generates a FIT blob.
+
+    :param file_name: FIT blob file name.
+    :param image: String containing the image tree source.
+    :param d: Bitbake environment.
+    """
+    import tempfile
+    import subprocess
+
+    bb.debug(1, "Generated FIT source is:\n {image}".format(image=image))
+
+    builddir = d.getVar('B')
+    blob_file_name = file_name + '.itb'
+
+    try:
+        fd, faux = tempfile.mkstemp(dir=builddir, prefix=file_name, suffix=".its")
+        with os.fdopen(fd, "w") as f:
+            f.write(image)
+
+        mkimage_opts = d.getVar('FIT_IMAGE_UBOOT_MKIMAGE_OPTS').split() or ""
+        cmd = ['mkimage', '-f', faux]
+        cmd.extend(mkimage_opts)
+        cmd.append('{output_name}'.format(output_name=blob_file_name))
+
+        ret = subprocess.run(
+            cmd,
+            check=True,
+            universal_newlines=True,
+            cwd=builddir,
+            stderr=subprocess.STDOUT,
+            stdout=subprocess.PIPE)
+
+        bb.debug(1, "Command for generating the FIT blob is:\n {cmd}".format(cmd=" ".join(ret.args)))
+
+    except subprocess.CalledProcessError as e:
+        bb.fatal('Failed to generate the FIT blob: {message}: {output}'.format(
+            message=str(e), output=e.stdout))
+    finally:
+        os.remove(faux)
+
+
+def generate_fit_image(file_name, d):
+    """
+    Create and generate FIT blob.
+
+    :param file_name: FIT blob file name.
+    :param d: Bitbake environment.
+    """
+    image = generate_image_tree_source(d)
+    generate_image_blob(file_name, image, d)
+
+
+python fit_image_do_generate_fit_image() {
+    generate_fit_image(d.getVar('FIT_IMAGE_FILENAME'), d)
+}
+
+do_generate_fit_image[vardeps] += " \
+    ${FIT_CONFIGURATIONS_NODE} \
+    ${FIT_IMAGES_NODE} \
+    FIT_CONFIGURATIONS_NODE \
+    FIT_IMAGES_NODE \
+    FIT_IMAGE_FILENAME \
+"
+
+addtask do_generate_fit_image after do_compile before do_deploy
+
+EXPORT_FUNCTIONS do_generate_fit_image

Comments

Zach Booth March 27, 2020, 3:11 p.m.
On Fri, Mar 27, 2020 at 3:29 AM Nandor Han <nandor.han@vaisala.com> wrote:
>
> FIT format is very versatile allowing various combination of booting
> sequences. In the same time different U-Boot boot stages can use FIT
> blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
> FIT blob). Because of the allowed level of customization, the generation
> of a FIT blob using a fixed image tree source, becomes challenging and
> increase the level of complexity where different configurations and
> combinations are needed.
>
> This bbclass will know how to generate a FIT blob, leaving the mechanics
> of the process (dependencies, task order...) to be handled by the users
> of the bbclass. In the same time will allow to separate the knowledge of
> the FIT format leaving the user code cleaner and more readable.
>
> Signed-off-by: Nandor Han <nandor.han@vaisala.com>

This class looks very useful, but I did have a question. How would you
account for creating nodes dynamically? One use case of this would be
adding a DT node for each reference in KERNEL_DEVICETREE. Would that
functionality be expected to go in the recipe including this class or
the class itself?

Thanks,
Zach

> ---
>
> Notes:
>     Testing
>     -------
>
>     1. linux-yocto_5.4.bbappend was modified to have the following configuration:
>
>     ```
>     inherit fit-image
>
>     KERNEL_IMAGE_NODE[name] = "kernel"
>     KERNEL_IMAGE_NODE[description] = "${PF}"
>     KERNEL_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/zImage")'
>     KERNEL_IMAGE_NODE[type] = "kernel"
>     KERNEL_IMAGE_NODE[arch] = "${ARCH}"
>     KERNEL_IMAGE_NODE[os] = "linux"
>     KERNEL_IMAGE_NODE[compression] = "none"
>     KERNEL_IMAGE_NODE[load] = "${UBOOT_LOADADDRESS}"
>     KERNEL_IMAGE_NODE[entry] = "${UBOOT_ENTRYPOINT}"
>     KERNEL_IMAGE_NODE[hash] = "sha256"
>
>     FDT_IMAGE_NODE[name] = "fdt"
>     FDT_IMAGE_NODE[description] = "FDT blob"
>     FDT_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/dts/am335x-bone.dtb")'
>     FDT_IMAGE_NODE[type] = "flat_dt"
>     FDT_IMAGE_NODE[arch] = "${ARCH}"
>     FDT_IMAGE_NODE[compression] = "none"
>     FDT_IMAGE_NODE[hash] = "sha256"
>
>     CONF1_CONF_NODE[name] = "conf"
>     CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
>     CONF1_CONF_NODE[kernel] = "kernel"
>     CONF1_CONF_NODE[fdt] = "fdt"
>
>     FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
>     FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
>     FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_NODE', 'name') or ""}"
>     ```
>     2. Build the kernel: `bitbake virtual/kernel`
>     3. Verify that `image-fit.itb` is present in the build directory: PASS
>     4. Disassemble the image using the command: `dtc -I dtb -O dts image-fit.itb`
>     5. Verify that the FIT source contains the expected configuration: PASS
>
>     Changes since v1:
>     ----------------
>     - Change the format of short-log to "<target>: <summary>"
>
>     Changes since v2:
>     ----------------
>     - rename the file from `fit-image` to `fit_image` to
>     successfully export the class functions.
>     - adding new sanity checks.
>     - add missing dependency.
>     - fix a variable reference in a debug log.
>
>  meta/classes/fit_image.bbclass | 382 +++++++++++++++++++++++++++++++++
>  1 file changed, 382 insertions(+)
>  create mode 100644 meta/classes/fit_image.bbclass
>
> diff --git a/meta/classes/fit_image.bbclass b/meta/classes/fit_image.bbclass
> new file mode 100644
> index 0000000000..87d92db122
> --- /dev/null
> +++ b/meta/classes/fit_image.bbclass
> @@ -0,0 +1,382 @@
> +#
> +# The class will facilitate the generation of FIT blobs.
> +#
> +# Glossary
> +#    FIT - Flattened uImage Tree
> +#
> +# Requirements:
> +#
> +#    * The user need to specify the image content using the format specified in the "FIT Image API" section.
> +#
> +# FIT Image API
> +#
> +# The bbclass is using variable and variable flags to declare the FIT image content.
> +#
> +#    * Sub-Images and Configuration Nodes
> +#
> +#       ** A sub-image node content is declared using the format `VAR_NODE[<property-name>] = <value>`.
> +#         * VAR_NODE - freely selected name of the variable representing a node.
> +#         * <property-name> - a sub-image property (e.g. description, type...).
> +#         * <value> - the property value.
> +#             Depending of the property the value can support different formats.
> +#           ** Property Values Formats
> +#
> +#            string property
> +#            ---------------
> +#            format: "<text>" - in case the property expects a text.
> +#                (e.g. IMAGE_NODE_KERNEL[type] = "kernel")
> +#
> +#            address property
> +#            ----------------
> +#            format: "<address>" - in case the property expects an address.
> +#                (e.g. IMAGE_NODE_KERNEL[entry] = "0xABCDABCD")
> +#
> +#            hash property
> +#            -------------
> +#            format: "<hash type>" - for hash property the hash type needs to be specified.
> +#                (e.g. IMAGE_NODE_KERNEL[hash] = "sha256")
> +#
> +#            sub-image signature property
> +#            ----------------------------
> +#            format: "<algo>;<key-name-hint>;" - for image signature node.
> +#                Both algorithm and key name needs to be provided.
> +#                (e.g. IMAGE_NODE_KERNEL[signature] = "sha256,rsa2048;kernel;"
> +#
> +#            configuration signature property
> +#            --------------------------------
> +#            format: "<algo>;<key-name-hint>;<sub-image list>" - for configuration signature properties algorithm,
> +#                key name and sub-image nodes needs to be provided.
> +#                (e.g. CONF_NODE_CONF1[signature] = "sha256,rsa2048;kernel;"kernel","fdt";")
> +#
> +#       ** Sub-Image and Configuration Nodes Flags
> +#              See the code for supported flags.
> +#
> +#    * FIT_IMAGES_NODE - contains a list of variables used to declare the sub-images nodes, separated by space.
> +#                    (e.g. FIT_IMAGES_NODE = "IMAGE_NODE_KERNEL IMAGE_NODE_FDT")
> +#
> +#    * FIT_CONFIGURATIONS_NODE - contains a list of variables used to declare the configuration nodes,
> +#          separated by space. (e.g. FIT_CONFIGURATIONS_NODE = "CONF_NODE_CONF1")
> +#        ** Flags
> +#           - "default": used to configure the default configuration node.
> +#                 (e.g. FIT_CONFIGURATIONS_NODE[default] = "conf@0")
> +#
> +# Example:
> +# This is part of a linux_%.bbappend recipe.
> +#
> +# KERNEL_IMAGE_NODE[name] = "kernel"
> +# KERNEL_IMAGE_NODE[description] = "${PF}"
> +# KERNEL_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/zImage")'
> +# KERNEL_IMAGE_NODE[type] = "kernel"
> +# KERNEL_IMAGE_NODE[arch] = "${ARCH}"
> +# KERNEL_IMAGE_NODE[os] = "linux"
> +# KERNEL_IMAGE_NODE[compression] = "none"
> +# KERNEL_IMAGE_NODE[load] = "${UBOOT_LOADADDRESS}"
> +# KERNEL_IMAGE_NODE[entry] = "${UBOOT_ENTRYPOINT}"
> +# KERNEL_IMAGE_NODE[hash] = "sha256"
> +#
> +# FDT_IMAGE_NODE[name] = "fdt"
> +# FDT_IMAGE_NODE[description] = "FDT blob"
> +# FDT_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/dts/am335x-bone.dtb")'
> +# FDT_IMAGE_NODE[type] = "flat_dt"
> +# FDT_IMAGE_NODE[arch] = "${ARCH}"
> +# FDT_IMAGE_NODE[compression] = "none"
> +# FDT_IMAGE_NODE[hash] = "sha256"
> +#
> +# CONF1_CONF_NODE[name] = "conf"
> +# CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
> +# CONF1_CONF_NODE[kernel] = "kernel"
> +# CONF1_CONF_NODE[fdt] = "fdt"
> +#
> +# FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
> +# FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
> +# FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_NODE', 'name') or ""}"
> +#
> +
> +DEPENDS += "\
> +    dtc-native \
> +    u-boot-mkimage-native \
> +"
> +
> +FIT_IMAGE_DESCRIPTION ??= "Generic FIT image"
> +FIT_IMAGE_FILENAME ??= "image-fit"
> +FIT_IMAGE_UBOOT_MKIMAGE_OPTS ??= ""
> +
> +def get_subimage_node_rules():
> +    """
> +    Defines the properties format and validation for sub-image nodes.
> +
> +    :return: Return a dictionary with the format rules.
> +    """
> +
> +    #
> +    # Rules Format: [Mandatory, Template String, Dictionary keys]
> +    # Mandatory: True, False - used to verify if this property is mandatory for generating the image.
> +    #            Note: Doesn't take in consideration the conditionally mandatory property.
> +    #                  (see: U-Boot/doc/uImage.FIT/source_file_format.txt)
> +    # Template String: Property content format. Used to generate the parameter content.
> +    # Dictionary Keys: Keys used to be replaced in the Template String.
> +    #
> +    from collections import OrderedDict
> +
> +    rules = OrderedDict()
> +
> +    rules['description'] = [True, '= "$value"', ["value"]]
> +    rules['data'] = [True, '= $value', ["value"]]
> +    rules['type'] = [True, '= "$value"', ["value"]]
> +    rules['arch'] = [False, '= "$value"', ["value"]]
> +    rules['os'] = [False, '= "$value"', ["value"]]
> +    rules['compression'] = [True, '= "$value"', ["value"]]
> +    rules['load'] = [False, '= <$value>', ["value"]]
> +    rules['entry'] = [False, '= <$value>', ["value"]]
> +    rules['hash'] = [False, '{ algo = "$algo"; }', ['algo']]
> +    rules['signature'] = [False, '{ algo = "$algo"; key-name-hint = "$key"; }', ['algo', 'key']]
> +
> +    return rules
> +
> +
> +def get_conf_node_rules():
> +    """
> +    Defines the properties format and validation for configuration nodes.
> +
> +    :return: Return a dictionary with the format rules.
> +    """
> +    #
> +    # Rules Format: [Mandatory, Template String, Dictionary keys]
> +    # Mandatory: True, False - used to verify if this property is mandatory for generating the image.
> +    #            Note: Doesn't take in consideration the conditionally mandatory property.
> +    #                  (see: U-Boot/doc/uImage.FIT/source_file_format.txt)
> +    # Template String: Property content format. Used to generate the parameter content.
> +    # Dictionary Keys: Keys used to be replaced in the Template String.
> +    #
> +    from collections import OrderedDict
> +
> +    rules = OrderedDict()
> +
> +    rules['description'] = [True, '= "$value"', ["value"]]
> +    rules['kernel'] = [True, '= "$value"', ["value"]]
> +    rules['ramdisk'] = [False, '= "$value"', ["value"]]
> +    rules['fdt'] = [False, '= "$value"', ["value"]]
> +    rules['loadables'] = [False, '= "$value"', ["value"]]
> +    rules['signature'] = [False, '{ algo = "$algo"; key-name-hint = "$key"; sign-images = $images; }',
> +                          ['algo', 'key', 'images']]
> +
> +    return rules
> +
> +
> +def generate_node(name, params, rules):
> +    """
> +    Generates a node.
> +
> +    :param name: Node name.
> +    :param params: A dictionary containing the properties values.
> +    :param rules: A dictionary containing the properties values validation and format.
> +
> +    :return: A string containing the node, including the new line characters.
> +    """
> +    from string import Template
> +
> +    content = []
> +
> +    for rule in rules.keys():
> +        if rule in params.keys():
> +            content.append('{param} {value}; '.format(
> +                param=rule,
> +                value=Template(rules[rule][1]).substitute(
> +                    dict(zip(rules[rule][2], params[rule].split(';'))))))
> +        elif rules[rule][0]:
> +            bb.fatal('Missing mandatory parameter "{param}" from "{section}" section'.format(param=rule, section=name))
> +
> +    content = """
> +                     """.join(content)
> +    node = """   {name} {{
> +                     {content}
> +              }};
> +           """.format(name=name, content=content)
> +
> +    return node
> +
> +
> +def get_section_configuration(var, d):
> +    """
> +    Generates a string build from variable's flags.
> +
> +    :param var: variable to extract the flags.
> +    :param d: bitbake environment.
> +
> +    :return: A string with the format '<flag1> = <value1>; <flag2> = <value2>;...'.
> +    """
> +    flags = d.getVarFlags(var)
> +    if flags is not None:
> +        flags = dict((flag, d.expand(value))
> +                     for flag, value in list(flags.items()))
> +    else:
> +        flags = {}
> +
> +    configuration = ''.join((
> +              """{name} = "{value}";
> +              """.format(name=name, value=value) for name, value in flags.items()))
> +
> +    return configuration
> +
> +
> +def get_section_properties(var, d):
> +    """
> +    Extract the nodes and parameters for a section.
> +
> +    :param var: variable containing the variable names of the nodes that are part of this section.
> +    :param d: bitbake environment.
> +
> +    :return: a list containing dictionaries with section nodes parameters.
> +    """
> +    nodes = []
> +    parameters = {}
> +
> +    for node in d.getVar(var).split():
> +        parameters = d.getVarFlags(node)
> +        if parameters is not None:
> +            parameters = dict((parameter, d.expand(value))
> +                              for parameter, value in list(parameters.items()))
> +            nodes.append(parameters)
> +
> +    return nodes
> +
> +
> +def generate_section(var, rules, d):
> +    """
> +    Generates a section node (configuration or sub-image).
> +
> +    :param var: Variable to extract the node names.
> +    :param rules: Rules to use for generating this section.
> +    :param d: bitbake environment.
> +
> +    :return: A string containing the section, including the new line characters.
> +    """
> +
> +    section = get_section_configuration(var, d)
> +
> +    nodes_parameters = get_section_properties(var, d)
> +    for parameters in nodes_parameters:
> +        name = parameters.pop('name')
> +        node = generate_node(name, parameters, rules)
> +        section += node
> +
> +    return section
> +
> +
> +def get_fit_image_template():
> +    """
> +    Get the FIT format.
> +
> +    :return: A Template string containing the FIT image format.
> +    """
> +    from string import Template
> +
> +    template = Template("""/dts-v1/;
> +        /{
> +             description = "$description";
> +             #address-cells = <1>;
> +             images {
> +                 $images_section
> +             };
> +             configurations {
> +                 $configurations_section
> +             };
> +        };""")
> +    return template
> +
> +
> +def generate_image_tree_source(d):
> +    """
> +    Generates a string containing the image tree source.
> +
> +    :return: A string representing the image tree.
> +    """
> +    from string import Template
> +
> +    values = {}
> +    values['description'] = d.getVar('FIT_IMAGE_DESCRIPTION')
> +
> +    image_rules = get_subimage_node_rules()
> +    if d.getVar('FIT_IMAGES_NODE', False) is None:
> +       bb.fatal("Please add the FIT image nodes to FIT_IMAGES_NODE variable.")
> +    values['images_section'] = generate_section('FIT_IMAGES_NODE', image_rules, d)
> +
> +    conf_rules = get_conf_node_rules()
> +    if d.getVar('FIT_CONFIGURATIONS_NODE', False) is None:
> +       bb.fatal("Please add the FIT configuration nodes to FIT_CONFIGURATIONS_NODE variable.")
> +    values['configurations_section'] = generate_section('FIT_CONFIGURATIONS_NODE', conf_rules, d)
> +
> +    image_tree_source = get_fit_image_template().substitute(values)
> +
> +    return image_tree_source
> +
> +
> +def generate_image_blob(file_name, image, d):
> +    """
> +    Generates a FIT blob.
> +
> +    :param file_name: FIT blob file name.
> +    :param image: String containing the image tree source.
> +    :param d: Bitbake environment.
> +    """
> +    import tempfile
> +    import subprocess
> +
> +    bb.debug(1, "Generated FIT source is:\n {image}".format(image=image))
> +
> +    builddir = d.getVar('B')
> +    blob_file_name = file_name + '.itb'
> +
> +    try:
> +        fd, faux = tempfile.mkstemp(dir=builddir, prefix=file_name, suffix=".its")
> +        with os.fdopen(fd, "w") as f:
> +            f.write(image)
> +
> +        mkimage_opts = d.getVar('FIT_IMAGE_UBOOT_MKIMAGE_OPTS').split() or ""
> +        cmd = ['mkimage', '-f', faux]
> +        cmd.extend(mkimage_opts)
> +        cmd.append('{output_name}'.format(output_name=blob_file_name))
> +
> +        ret = subprocess.run(
> +            cmd,
> +            check=True,
> +            universal_newlines=True,
> +            cwd=builddir,
> +            stderr=subprocess.STDOUT,
> +            stdout=subprocess.PIPE)
> +
> +        bb.debug(1, "Command for generating the FIT blob is:\n {cmd}".format(cmd=" ".join(ret.args)))
> +
> +    except subprocess.CalledProcessError as e:
> +        bb.fatal('Failed to generate the FIT blob: {message}: {output}'.format(
> +            message=str(e), output=e.stdout))
> +    finally:
> +        os.remove(faux)
> +
> +
> +def generate_fit_image(file_name, d):
> +    """
> +    Create and generate FIT blob.
> +
> +    :param file_name: FIT blob file name.
> +    :param d: Bitbake environment.
> +    """
> +    image = generate_image_tree_source(d)
> +    generate_image_blob(file_name, image, d)
> +
> +
> +python fit_image_do_generate_fit_image() {
> +    generate_fit_image(d.getVar('FIT_IMAGE_FILENAME'), d)
> +}
> +
> +do_generate_fit_image[vardeps] += " \
> +    ${FIT_CONFIGURATIONS_NODE} \
> +    ${FIT_IMAGES_NODE} \
> +    FIT_CONFIGURATIONS_NODE \
> +    FIT_IMAGES_NODE \
> +    FIT_IMAGE_FILENAME \
> +"
> +
> +addtask do_generate_fit_image after do_compile before do_deploy
> +
> +EXPORT_FUNCTIONS do_generate_fit_image
> --
> 2.24.1
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136798): https://lists.openembedded.org/g/openembedded-core/message/136798
Mute This Topic: https://lists.openembedded.org/mt/72583032/3617530
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Richard Purdie March 27, 2020, 4:35 p.m.
On Fri, 2020-03-27 at 10:29 +0200, Nandor Han wrote:
> FIT format is very versatile allowing various combination of booting
> sequences. In the same time different U-Boot boot stages can use FIT
> blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
> FIT blob). Because of the allowed level of customization, the generation
> of a FIT blob using a fixed image tree source, becomes challenging and
> increase the level of complexity where different configurations and
> combinations are needed.
> 
> This bbclass will know how to generate a FIT blob, leaving the mechanics
> of the process (dependencies, task order...) to be handled by the users
> of the bbclass. In the same time will allow to separate the knowledge of
> the FIT format leaving the user code cleaner and more readable.
> 
> Signed-off-by: Nandor Han <nandor.han@vaisala.com>

Have you considered adding automated tests for this?

Does adding this class remove or obsolete any of the existing
code/support?

Cheers,

Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136799): https://lists.openembedded.org/g/openembedded-core/message/136799
Mute This Topic: https://lists.openembedded.org/mt/72583032/3617530
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Nandor Han March 27, 2020, 5:18 p.m.
On 2020-03-27 18:35, Richard Purdie wrote:
> On Fri, 2020-03-27 at 10:29 +0200, Nandor Han wrote:
>> FIT format is very versatile allowing various combination of booting
>> sequences. In the same time different U-Boot boot stages can use FIT
>> blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
>> FIT blob). Because of the allowed level of customization, the generation
>> of a FIT blob using a fixed image tree source, becomes challenging and
>> increase the level of complexity where different configurations and
>> combinations are needed.
>>
>> This bbclass will know how to generate a FIT blob, leaving the mechanics
>> of the process (dependencies, task order...) to be handled by the users
>> of the bbclass. In the same time will allow to separate the knowledge of
>> the FIT format leaving the user code cleaner and more readable.
>>
>> Signed-off-by: Nandor Han <nandor.han@vaisala.com>
> 
> Have you considered adding automated tests for this?
> 

Thanks for feedback Rickard. I haven't considered adding any extra
tests (automate), but I think is a great idea. I'll try to have a look.

> Does adding this class remove or obsolete any of the existing
> code/support?
> 

Currently no, AFAIK. But this is only the first step and my plan is to 
go forward and update the `kernel-fitimage.bbclass` to use this class. 
This way we can very easily extend the FIT source dynamically and in the 
same time reduce the complexity of the `kernel-fitimage.bbclass`. More 
over I'm planning to create also a U-Boot related class that will be 
able to generate a FIT blobs that can contain U-Boot... and is used by SPL.

> Cheers,
> 
> Richard
> 

Nandor
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136802): https://lists.openembedded.org/g/openembedded-core/message/136802
Mute This Topic: https://lists.openembedded.org/mt/72583032/3617530
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Nandor Han March 27, 2020, 5:33 p.m.
On 2020-03-27 17:11, Zach Booth wrote:
> On Fri, Mar 27, 2020 at 3:29 AM Nandor Han <nandor.han@vaisala.com> wrote:
>>
>> FIT format is very versatile allowing various combination of booting
>> sequences. In the same time different U-Boot boot stages can use FIT
>> blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
>> FIT blob). Because of the allowed level of customization, the generation
>> of a FIT blob using a fixed image tree source, becomes challenging and
>> increase the level of complexity where different configurations and
>> combinations are needed.
>>
>> This bbclass will know how to generate a FIT blob, leaving the mechanics
>> of the process (dependencies, task order...) to be handled by the users
>> of the bbclass. In the same time will allow to separate the knowledge of
>> the FIT format leaving the user code cleaner and more readable.
>>
>> Signed-off-by: Nandor Han <nandor.han@vaisala.com>
> 
> This class looks very useful, but I did have a question. How would you
> account for creating nodes dynamically? One use case of this would be
> adding a DT node for each reference in KERNEL_DEVICETREE. Would that
> functionality be expected to go in the recipe including this class or
> the class itself?
> 

Thanks Zack for feedback. Like I mentioned in one of Rickard's answer, 
I'm planning to update the `kernel-fitimage.bbclass` to use this class.

In my local repo I have already a class that does what you're saying.

e.g
```
...
24 python do_generate_fit_image() {
  25     import os.path
  26
  27     device_trees = (d.getVar("KERNEL_DEVICETREE").split())
  28     kernel_key_path = 
'{path}'.format(path=d.getVar("SECURITY_DIR_KEYS_RD") or "")
  29     conf_signature = (d.getVar("CONF_NODE_CONF1") or "")
  30     image_name = ""
  31     mkimage_opts = ""
  32
  33     for dtb in device_trees:
  34         d.setVarFlag("IMAGE_NODE_FDT", "data", 
'/incbin/("./arch/{arch}/boot/dts/{dtb}")'.format(
  35          arch=d.getVar("ARCH"), dtb=dtb))
  36         d.setVarFlag("IMAGE_NODE_FDT", "description", dtb)
  37
  38         if os.path.exists(kernel_key_path):
  39             image_name = 
"kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0], 
suffix=".rdkeys")
  40             mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_SIGNED")
  41             d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS", ("-k {key} -r 
{extra}".format(
  42              key=kernel_key_path, extra=mkimage_opts)))
  43             generate_fit_image(image_name, d)
  44
  45         if not conf_signature:
  46             d.delVarFlag("CONF_NODE_CONF1", "signature")
  47
  48         mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_UNSIGNED")
  49         d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS", 
"{extra}".format(extra=mkimage_opts))
  50         image_name = 
"kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0], 
suffix=".unsigned")
  51         generate_fit_image(image_name, d)
  52
  53         if not conf_signature:
  54             d.setVarFlag("CONF_NODE_CONF1", "signature", 
conf_signature)
  55 }
...
```

The code above will generate a separate FIT blob for every device tree 
declared in KERNEL_DEVICETREE. However this functionality overlaps with 
`kernel-fitimage` and my target is to keep this class as simple as 
possible and refactor `kernel-fitimage` to use the `fit_image` class. 
The code above will go in `kernel-fitimage`.

Later on I'm planning to add a different class that can be use to 
generate U-Boot FIT blobs, which can be used by SPL.

So this is only the first step :)

> Thanks,
> Zach
> 

<snip>

Nandor
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136818): https://lists.openembedded.org/g/openembedded-core/message/136818
Mute This Topic: https://lists.openembedded.org/mt/72583032/3617530
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Denys Dmytriyenko March 27, 2020, 6:16 p.m.
On Fri, Mar 27, 2020 at 07:33:24PM +0200, Nandor Han wrote:
> On 2020-03-27 17:11, Zach Booth wrote:
> >On Fri, Mar 27, 2020 at 3:29 AM Nandor Han <nandor.han@vaisala.com> wrote:
> >>
> >>FIT format is very versatile allowing various combination of booting
> >>sequences. In the same time different U-Boot boot stages can use FIT
> >>blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
> >>FIT blob). Because of the allowed level of customization, the generation
> >>of a FIT blob using a fixed image tree source, becomes challenging and
> >>increase the level of complexity where different configurations and
> >>combinations are needed.
> >>
> >>This bbclass will know how to generate a FIT blob, leaving the mechanics
> >>of the process (dependencies, task order...) to be handled by the users
> >>of the bbclass. In the same time will allow to separate the knowledge of
> >>the FIT format leaving the user code cleaner and more readable.
> >>
> >>Signed-off-by: Nandor Han <nandor.han@vaisala.com>
> >
> >This class looks very useful, but I did have a question. How would you
> >account for creating nodes dynamically? One use case of this would be
> >adding a DT node for each reference in KERNEL_DEVICETREE. Would that
> >functionality be expected to go in the recipe including this class or
> >the class itself?
> >
> 
> Thanks Zack for feedback. Like I mentioned in one of Rickard's
> answer, I'm planning to update the `kernel-fitimage.bbclass` to use
> this class.
> 
> In my local repo I have already a class that does what you're saying.
> 
> e.g
> ```
> ...
> 24 python do_generate_fit_image() {
>  25     import os.path
>  26
>  27     device_trees = (d.getVar("KERNEL_DEVICETREE").split())
>  28     kernel_key_path =
> '{path}'.format(path=d.getVar("SECURITY_DIR_KEYS_RD") or "")
>  29     conf_signature = (d.getVar("CONF_NODE_CONF1") or "")
>  30     image_name = ""
>  31     mkimage_opts = ""
>  32
>  33     for dtb in device_trees:
>  34         d.setVarFlag("IMAGE_NODE_FDT", "data",
> '/incbin/("./arch/{arch}/boot/dts/{dtb}")'.format(
>  35          arch=d.getVar("ARCH"), dtb=dtb))
>  36         d.setVarFlag("IMAGE_NODE_FDT", "description", dtb)
>  37
>  38         if os.path.exists(kernel_key_path):
>  39             image_name =
> "kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
> suffix=".rdkeys")
>  40             mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_SIGNED")
>  41             d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS", ("-k {key}
> -r {extra}".format(
>  42              key=kernel_key_path, extra=mkimage_opts)))
>  43             generate_fit_image(image_name, d)
>  44
>  45         if not conf_signature:
>  46             d.delVarFlag("CONF_NODE_CONF1", "signature")
>  47
>  48         mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_UNSIGNED")
>  49         d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS",
> "{extra}".format(extra=mkimage_opts))
>  50         image_name =
> "kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
> suffix=".unsigned")
>  51         generate_fit_image(image_name, d)
>  52
>  53         if not conf_signature:
>  54             d.setVarFlag("CONF_NODE_CONF1", "signature",
> conf_signature)
>  55 }
> ...
> ```
> 
> The code above will generate a separate FIT blob for every device
> tree declared in KERNEL_DEVICETREE. However this functionality

So, no multiple device trees in a single FIT image? No multiple 
configurations, ramdisks, etc?


> overlaps with `kernel-fitimage` and my target is to keep this class
> as simple as possible and refactor `kernel-fitimage` to use the
> `fit_image` class. The code above will go in `kernel-fitimage`.
> 
> Later on I'm planning to add a different class that can be use to
> generate U-Boot FIT blobs, which can be used by SPL.
> 
> So this is only the first step :)
> 
> >Thanks,
> >Zach
> >
> 
> <snip>
> 
> Nandor

>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136820): https://lists.openembedded.org/g/openembedded-core/message/136820
Mute This Topic: https://lists.openembedded.org/mt/72583032/3617530
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Nandor Han March 27, 2020, 6:24 p.m.
On 2020-03-27 20:16, Denys Dmytriyenko wrote:
> On Fri, Mar 27, 2020 at 07:33:24PM +0200, Nandor Han wrote:
>> On 2020-03-27 17:11, Zach Booth wrote:
>>> On Fri, Mar 27, 2020 at 3:29 AM Nandor Han <nandor.han@vaisala.com> wrote:
>>>>
>>>> FIT format is very versatile allowing various combination of booting
>>>> sequences. In the same time different U-Boot boot stages can use FIT
>>>> blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
>>>> FIT blob). Because of the allowed level of customization, the generation
>>>> of a FIT blob using a fixed image tree source, becomes challenging and
>>>> increase the level of complexity where different configurations and
>>>> combinations are needed.
>>>>
>>>> This bbclass will know how to generate a FIT blob, leaving the mechanics
>>>> of the process (dependencies, task order...) to be handled by the users
>>>> of the bbclass. In the same time will allow to separate the knowledge of
>>>> the FIT format leaving the user code cleaner and more readable.
>>>>
>>>> Signed-off-by: Nandor Han <nandor.han@vaisala.com>
>>>
>>> This class looks very useful, but I did have a question. How would you
>>> account for creating nodes dynamically? One use case of this would be
>>> adding a DT node for each reference in KERNEL_DEVICETREE. Would that
>>> functionality be expected to go in the recipe including this class or
>>> the class itself?
>>>
>>
>> Thanks Zack for feedback. Like I mentioned in one of Rickard's
>> answer, I'm planning to update the `kernel-fitimage.bbclass` to use
>> this class.
>>
>> In my local repo I have already a class that does what you're saying.
>>
>> e.g
>> ```
>> ...
>> 24 python do_generate_fit_image() {
>>   25     import os.path
>>   26
>>   27     device_trees = (d.getVar("KERNEL_DEVICETREE").split())
>>   28     kernel_key_path =
>> '{path}'.format(path=d.getVar("SECURITY_DIR_KEYS_RD") or "")
>>   29     conf_signature = (d.getVar("CONF_NODE_CONF1") or "")
>>   30     image_name = ""
>>   31     mkimage_opts = ""
>>   32
>>   33     for dtb in device_trees:
>>   34         d.setVarFlag("IMAGE_NODE_FDT", "data",
>> '/incbin/("./arch/{arch}/boot/dts/{dtb}")'.format(
>>   35          arch=d.getVar("ARCH"), dtb=dtb))
>>   36         d.setVarFlag("IMAGE_NODE_FDT", "description", dtb)
>>   37
>>   38         if os.path.exists(kernel_key_path):
>>   39             image_name =
>> "kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
>> suffix=".rdkeys")
>>   40             mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_SIGNED")
>>   41             d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS", ("-k {key}
>> -r {extra}".format(
>>   42              key=kernel_key_path, extra=mkimage_opts)))
>>   43             generate_fit_image(image_name, d)
>>   44
>>   45         if not conf_signature:
>>   46             d.delVarFlag("CONF_NODE_CONF1", "signature")
>>   47
>>   48         mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_UNSIGNED")
>>   49         d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS",
>> "{extra}".format(extra=mkimage_opts))
>>   50         image_name =
>> "kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
>> suffix=".unsigned")
>>   51         generate_fit_image(image_name, d)
>>   52
>>   53         if not conf_signature:
>>   54             d.setVarFlag("CONF_NODE_CONF1", "signature",
>> conf_signature)
>>   55 }
>> ...
>> ```
>>
>> The code above will generate a separate FIT blob for every device
>> tree declared in KERNEL_DEVICETREE. However this functionality
> 
> So, no multiple device trees in a single FIT image? No multiple
> configurations, ramdisks, etc?
> 
> 

Given the class API is possible to configure your FIT source as you 
want. However, my plan is to refactor `kernel-fitimage` class to have 
this kind of features. The above code was only an example, something 
that I'm working on. I want to get this in and then it's easier to add 
new functionalities.

>> overlaps with `kernel-fitimage` and my target is to keep this class
>> as simple as possible and refactor `kernel-fitimage` to use the
>> `fit_image` class. The code above will go in `kernel-fitimage`.
>>
>> Later on I'm planning to add a different class that can be use to
>> generate U-Boot FIT blobs, which can be used by SPL.
>>
>> So this is only the first step :)
>>

Nandor
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136821): https://lists.openembedded.org/g/openembedded-core/message/136821
Mute This Topic: https://lists.openembedded.org/mt/72583032/3617530
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Denys Dmytriyenko March 30, 2020, 4:07 p.m.
On Fri, Mar 27, 2020 at 08:24:49PM +0200, Nandor Han wrote:
> On 2020-03-27 20:16, Denys Dmytriyenko wrote:
> >On Fri, Mar 27, 2020 at 07:33:24PM +0200, Nandor Han wrote:
> >>On 2020-03-27 17:11, Zach Booth wrote:
> >>>On Fri, Mar 27, 2020 at 3:29 AM Nandor Han <nandor.han@vaisala.com> wrote:
> >>>>
> >>>>FIT format is very versatile allowing various combination of booting
> >>>>sequences. In the same time different U-Boot boot stages can use FIT
> >>>>blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
> >>>>FIT blob). Because of the allowed level of customization, the generation
> >>>>of a FIT blob using a fixed image tree source, becomes challenging and
> >>>>increase the level of complexity where different configurations and
> >>>>combinations are needed.
> >>>>
> >>>>This bbclass will know how to generate a FIT blob, leaving the mechanics
> >>>>of the process (dependencies, task order...) to be handled by the users
> >>>>of the bbclass. In the same time will allow to separate the knowledge of
> >>>>the FIT format leaving the user code cleaner and more readable.
> >>>>
> >>>>Signed-off-by: Nandor Han <nandor.han@vaisala.com>
> >>>
> >>>This class looks very useful, but I did have a question. How would you
> >>>account for creating nodes dynamically? One use case of this would be
> >>>adding a DT node for each reference in KERNEL_DEVICETREE. Would that
> >>>functionality be expected to go in the recipe including this class or
> >>>the class itself?
> >>>
> >>
> >>Thanks Zack for feedback. Like I mentioned in one of Rickard's
> >>answer, I'm planning to update the `kernel-fitimage.bbclass` to use
> >>this class.
> >>
> >>In my local repo I have already a class that does what you're saying.
> >>
> >>e.g
> >>```
> >>...
> >>24 python do_generate_fit_image() {
> >>  25     import os.path
> >>  26
> >>  27     device_trees = (d.getVar("KERNEL_DEVICETREE").split())
> >>  28     kernel_key_path =
> >>'{path}'.format(path=d.getVar("SECURITY_DIR_KEYS_RD") or "")
> >>  29     conf_signature = (d.getVar("CONF_NODE_CONF1") or "")
> >>  30     image_name = ""
> >>  31     mkimage_opts = ""
> >>  32
> >>  33     for dtb in device_trees:
> >>  34         d.setVarFlag("IMAGE_NODE_FDT", "data",
> >>'/incbin/("./arch/{arch}/boot/dts/{dtb}")'.format(
> >>  35          arch=d.getVar("ARCH"), dtb=dtb))
> >>  36         d.setVarFlag("IMAGE_NODE_FDT", "description", dtb)
> >>  37
> >>  38         if os.path.exists(kernel_key_path):
> >>  39             image_name =
> >>"kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
> >>suffix=".rdkeys")
> >>  40             mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_SIGNED")
> >>  41             d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS", ("-k {key}
> >>-r {extra}".format(
> >>  42              key=kernel_key_path, extra=mkimage_opts)))
> >>  43             generate_fit_image(image_name, d)
> >>  44
> >>  45         if not conf_signature:
> >>  46             d.delVarFlag("CONF_NODE_CONF1", "signature")
> >>  47
> >>  48         mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_UNSIGNED")
> >>  49         d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS",
> >>"{extra}".format(extra=mkimage_opts))
> >>  50         image_name =
> >>"kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
> >>suffix=".unsigned")
> >>  51         generate_fit_image(image_name, d)
> >>  52
> >>  53         if not conf_signature:
> >>  54             d.setVarFlag("CONF_NODE_CONF1", "signature",
> >>conf_signature)
> >>  55 }
> >>...
> >>```
> >>
> >>The code above will generate a separate FIT blob for every device
> >>tree declared in KERNEL_DEVICETREE. However this functionality
> >
> >So, no multiple device trees in a single FIT image? No multiple
> >configurations, ramdisks, etc?
> 
> Given the class API is possible to configure your FIT source as you
> want. However, my plan is to refactor `kernel-fitimage` class to
> have this kind of features. The above code was only an example,
> something that I'm working on. I want to get this in and then it's
> easier to add new functionalities.

Considering that fitimage is a critical functionality used by many BSPs 
(even though it's not used in OE-Core itself), breaking it would upset 
so many people... Not sure about this gradual approach.
Nandor Han March 30, 2020, 4:46 p.m.
On 2020-03-30 19:07, Denys Dmytriyenko wrote:
> On Fri, Mar 27, 2020 at 08:24:49PM +0200, Nandor Han wrote:
>> On 2020-03-27 20:16, Denys Dmytriyenko wrote:
>>> On Fri, Mar 27, 2020 at 07:33:24PM +0200, Nandor Han wrote:
>>>> On 2020-03-27 17:11, Zach Booth wrote:
>>>>> On Fri, Mar 27, 2020 at 3:29 AM Nandor Han <nandor.han@vaisala.com> wrote:
>>>>>>
>>>>>> FIT format is very versatile allowing various combination of booting
>>>>>> sequences. In the same time different U-Boot boot stages can use FIT
>>>>>> blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
>>>>>> FIT blob). Because of the allowed level of customization, the generation
>>>>>> of a FIT blob using a fixed image tree source, becomes challenging and
>>>>>> increase the level of complexity where different configurations and
>>>>>> combinations are needed.
>>>>>>
>>>>>> This bbclass will know how to generate a FIT blob, leaving the mechanics
>>>>>> of the process (dependencies, task order...) to be handled by the users
>>>>>> of the bbclass. In the same time will allow to separate the knowledge of
>>>>>> the FIT format leaving the user code cleaner and more readable.
>>>>>>
>>>>>> Signed-off-by: Nandor Han <nandor.han@vaisala.com>
>>>>>
>>>>> This class looks very useful, but I did have a question. How would you
>>>>> account for creating nodes dynamically? One use case of this would be
>>>>> adding a DT node for each reference in KERNEL_DEVICETREE. Would that
>>>>> functionality be expected to go in the recipe including this class or
>>>>> the class itself?
>>>>>
>>>>
>>>> Thanks Zack for feedback. Like I mentioned in one of Rickard's
>>>> answer, I'm planning to update the `kernel-fitimage.bbclass` to use
>>>> this class.
>>>>
>>>> In my local repo I have already a class that does what you're saying.
>>>>
>>>> e.g
>>>> ```
>>>> ...
>>>> 24 python do_generate_fit_image() {
>>>>   25     import os.path
>>>>   26
>>>>   27     device_trees = (d.getVar("KERNEL_DEVICETREE").split())
>>>>   28     kernel_key_path =
>>>> '{path}'.format(path=d.getVar("SECURITY_DIR_KEYS_RD") or "")
>>>>   29     conf_signature = (d.getVar("CONF_NODE_CONF1") or "")
>>>>   30     image_name = ""
>>>>   31     mkimage_opts = ""
>>>>   32
>>>>   33     for dtb in device_trees:
>>>>   34         d.setVarFlag("IMAGE_NODE_FDT", "data",
>>>> '/incbin/("./arch/{arch}/boot/dts/{dtb}")'.format(
>>>>   35          arch=d.getVar("ARCH"), dtb=dtb))
>>>>   36         d.setVarFlag("IMAGE_NODE_FDT", "description", dtb)
>>>>   37
>>>>   38         if os.path.exists(kernel_key_path):
>>>>   39             image_name =
>>>> "kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
>>>> suffix=".rdkeys")
>>>>   40             mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_SIGNED")
>>>>   41             d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS", ("-k {key}
>>>> -r {extra}".format(
>>>>   42              key=kernel_key_path, extra=mkimage_opts)))
>>>>   43             generate_fit_image(image_name, d)
>>>>   44
>>>>   45         if not conf_signature:
>>>>   46             d.delVarFlag("CONF_NODE_CONF1", "signature")
>>>>   47
>>>>   48         mkimage_opts = d.getVar("FIT_IMAGE_MKIMAGE_OPTS_UNSIGNED")
>>>>   49         d.setVar("FIT_IMAGE_UBOOT_MKIMAGE_OPTS",
>>>> "{extra}".format(extra=mkimage_opts))
>>>>   50         image_name =
>>>> "kernel-{dtb_name}{suffix}".format(dtb_name=os.path.splitext(dtb)[0],
>>>> suffix=".unsigned")
>>>>   51         generate_fit_image(image_name, d)
>>>>   52
>>>>   53         if not conf_signature:
>>>>   54             d.setVarFlag("CONF_NODE_CONF1", "signature",
>>>> conf_signature)
>>>>   55 }
>>>> ...
>>>> ```
>>>>
>>>> The code above will generate a separate FIT blob for every device
>>>> tree declared in KERNEL_DEVICETREE. However this functionality
>>>
>>> So, no multiple device trees in a single FIT image? No multiple
>>> configurations, ramdisks, etc?
>>
>> Given the class API is possible to configure your FIT source as you
>> want. However, my plan is to refactor `kernel-fitimage` class to
>> have this kind of features. The above code was only an example,
>> something that I'm working on. I want to get this in and then it's
>> easier to add new functionalities.
> 
> Considering that fitimage is a critical functionality used by many BSPs
> (even though it's not used in OE-Core itself), breaking it would upset
> so many people... Not sure about this gradual approach.
> 

Totally agree :). That's the beauty of this class...nothing is impacted. 
My plan is to build gradually on top of this class different classes and 
impact nobody about this. The users will be able to seamlessly take in 
use the final solution without any big changes.

Nandor
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136862): https://lists.openembedded.org/g/openembedded-core/message/136862
Mute This Topic: https://lists.openembedded.org/mt/72583032/3617530
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Nandor Han May 27, 2020, 3:58 a.m.
This is a system generated Comment: Patch 171365 was automatically marked as superseded by patch 172860.