[RFC,1/3] meta/files: add layer setup JSON schema and example

From: Joshua Watt <JPEWhacker@gmail.com>

Defines a common schema for layer setup that can be consumed by tools to
know how to fetch and assemble layers for end users. Also includes an
example of the layer setup that constructs poky/meta-intel/imaginary product layer
for reference.

The schema can be used to validate a layer setup file with the commands:

 $ python3 -m pip install jsonschema
 $ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>

Alex: I made the following modifications to Joshua's original commit:

- moved the files from meta/lib to meta/files

- the example json showcases a multi-repo, multi-layer setup with additional
configurations and machines, instead of just poky - closer to a typical product

- the schema is modified so that:

-- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items.
This allows using them as dicts inside Python, and makes the json more compact and readable.

-- added 'contains_this_file' property to source object

-- added 'buildconfigs', 'machines' and 'distros' properties to layer objects.

-- replaced 'remote' property with a 'oneOf' definition for git with a specific
'git-remote' property. 'oneOf' is problematic when schema validation fails:
the diagnostic is only that none of oneOf variants matched, which is too non-specific.

-- added 'describe' property to 'git-remote' object.

FIXME:
- oe-selftest that validates the example json against the schema using python3-jsonschema-native

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/files/layers.example.json | 115 ++++++++++++++++++++++++++++++
 meta/files/layers.schema.json  | 125 +++++++++++++++++++++++++++++++++
 2 files changed, 240 insertions(+)
 create mode 100644 meta/files/layers.example.json
 create mode 100644 meta/files/layers.schema.json

On Fri, Jul 8, 2022 at 2:17 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> From: Joshua Watt <JPEWhacker@gmail.com>
>
> Defines a common schema for layer setup that can be consumed by tools to
> know how to fetch and assemble layers for end users. Also includes an
> example of the layer setup that constructs poky/meta-intel/imaginary product layer
> for reference.
>
> The schema can be used to validate a layer setup file with the commands:
>
>  $ python3 -m pip install jsonschema
>  $ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
>
> Alex: I made the following modifications to Joshua's original commit:
>
> - moved the files from meta/lib to meta/files
>
> - the example json showcases a multi-repo, multi-layer setup with additional
> configurations and machines, instead of just poky - closer to a typical product
>
> - the schema is modified so that:
>
> -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items.
> This allows using them as dicts inside Python, and makes the json more compact and readable.

As long as the order is truly irrelevant because different
implementations (and even different versions of Python!) will result
in a different processing order.

>
> -- added 'contains_this_file' property to source object
>
> -- added 'buildconfigs', 'machines' and 'distros' properties to layer objects.
>
> -- replaced 'remote' property with a 'oneOf' definition for git with a specific
> 'git-remote' property. 'oneOf' is problematic when schema validation fails:
> the diagnostic is only that none of oneOf variants matched, which is too non-specific.
>
> -- added 'describe' property to 'git-remote' object.
>
> FIXME:
> - oe-selftest that validates the example json against the schema using python3-jsonschema-native
>
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  meta/files/layers.example.json | 115 ++++++++++++++++++++++++++++++
>  meta/files/layers.schema.json  | 125 +++++++++++++++++++++++++++++++++
>  2 files changed, 240 insertions(+)
>  create mode 100644 meta/files/layers.example.json
>  create mode 100644 meta/files/layers.schema.json
>
> diff --git a/meta/files/layers.example.json b/meta/files/layers.example.json
> new file mode 100644
> index 0000000000..f606567b9d
> --- /dev/null
> +++ b/meta/files/layers.example.json
> @@ -0,0 +1,115 @@
> +{

Missing the version? Although I see it's in the schema.

> +    "sources": {
> +        "meta-alex": {
> +            "contains_this_file": true,
> +            "git-remote": {

The problem with this is that there has been some desire for other
fetching mechanisms; Perhaps we can say git is only supported, but
that was the reason I used `oneOf`: it allows for other fetching types
in the future.

> +                "branch": "master",
> +                "describe": "",
> +                "remotes": {
> +                    "remote-alex": {
> +                        "uri": "https://github.com/kanavin/meta-alex"
> +                    }

Multiple remotes are nice.... but not strictly necessary, since you
can only have one rev checked out. I don't care too much one way or
another but it's something to think about since it makes the fetching
more complicated.

> +                },
> +                "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5"
> +            },
> +            "layers": {
> +                "meta-alex": {
> +                    "buildconfigs": {
> +                        "conf/templates/configuration-gadget": {},
> +                        "conf/templates/configuration-gizmo": {}

I think these fall under the same category as the machines below, but
I don't use  TEMPLATECONF so I'm not sure.

> +                    },
> +                    "subpath": ""
> +                }
> +            },
> +            "path": "meta-alex"
> +        },
> +        "meta-intel": {
> +            "git-remote": {
> +                "branch": "master",
> +                "describe": "15.0-hardknott-3.3-310-g0a96edae",
> +                "remotes": {
> +                    "origin": {
> +                        "uri": "git://git.yoctoproject.org/meta-intel"
> +                    }
> +                },
> +                "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a"
> +            },
> +            "layers": {
> +                "meta-intel": {
> +                    "machines": {
> +                        "intel-core2-32": {},
> +                        "intel-corei7-64": {},
> +                        "intel-skylake-64": {}

Why does this need to be listed here? Isn't this just the contents of
the conf/machine directory in the layer? I really don't want to be
duplicating information; IHMO it's better to let the layer itself be
canonical about what machines/distros/etc. it supports

> +                    },
> +                    "subpath": ""
> +                }
> +            },
> +            "path": "meta-intel"
> +        },
> +        "poky": {
> +            "git-remote": {
> +                "branch": "akanavin/setup-layers",
> +                "describe": "4.1_M1-374-g9dda719b2a",
> +                "remotes": {
> +                    "origin": {
> +                        "uri": "git://git.yoctoproject.org/poky"
> +                    },
> +                    "poky-contrib": {
> +                        "uri": "ssh://git@push.yoctoproject.org/poky-contrib"
> +                    }
> +                },
> +                "rev": "9dda719b2a4727a4d43a6ab8d9e23f8ca68790ec"
> +            },
> +            "layers": {
> +                "meta": {
> +                    "distros": {
> +                        "defaultsetup": {}

Same as machines above

> +                    },
> +                    "machines": {
> +                        "qemuarm": {},
> +                        "qemuarm64": {},
> +                        "qemuarmv5": {},
> +                        "qemumips": {},
> +                        "qemumips64": {},
> +                        "qemuppc": {},
> +                        "qemuppc64": {},
> +                        "qemuriscv32": {},
> +                        "qemuriscv64": {},
> +                        "qemux86": {},
> +                        "qemux86-64": {}
> +                    },
> +                    "subpath": "meta"
> +                },
> +                "meta-poky": {
> +                    "buildconfigs": {
> +                        "conf": {}
> +                    },
> +                    "distros": {
> +                        "poky": {},
> +                        "poky-altcfg": {},
> +                        "poky-bleeding": {},
> +                        "poky-tiny": {}
> +                    },
> +                    "subpath": "meta-poky"
> +                },
> +                "meta-selftest": {
> +                    "machines": {
> +                        "qemux86copy": {}
> +                    },
> +                    "subpath": "meta-selftest"
> +                },
> +                "meta-yocto-bsp": {
> +                    "machines": {
> +                        "beaglebone-yocto": {},
> +                        "edgerouter": {},
> +                        "genericx86": {},
> +                        "genericx86-64": {}
> +                    },
> +                    "subpath": "meta-yocto-bsp"
> +                }
> +            },
> +            "path": "poky"
> +        }
> +    },
> +    "version": "1.0"
> +}
> \ No newline at end of file
> diff --git a/meta/files/layers.schema.json b/meta/files/layers.schema.json
> new file mode 100644
> index 0000000000..ba46e00b52
> --- /dev/null
> +++ b/meta/files/layers.schema.json
> @@ -0,0 +1,125 @@
> +{
> +    "description": "OpenEmbedder Layer Setup Manifest",
> +    "type": "object",
> +    "additionalProperties": false,
> +    "required": [
> +        "version"
> +    ],
> +    "properties": {
> +        "version": {
> +            "description": "The version of this document; currently '1.0'",
> +            "enum": ["1.0"]
> +        },
> +        "sources": {
> +            "description": "The dict of layer sources",
> +            "type": "object",
> +            "patternProperties": { ".*" : {
> +                "type": "object",
> +                "description": "The upstream source from which a set of layers may be fetched",
> +                "additionalProperties": false,
> +                "required": [
> +                    "path"
> +                ],
> +                "properties": {
> +                    "path": {
> +                        "description": "The path where this layer source will be placed when fetching",
> +                        "type": "string"
> +                    },
> +                    "description": {
> +                        "description": "A description of this layer source",
> +                        "type": "string"
> +                    },
> +                    "contains_this_file": {
> +                        "description": "Whether the directory with the layer source also contains this json description. Tools may want to skip the checkout of the source then.",
> +                        "type": "boolean"
> +                    },
> +                    "layers": {
> +                        "description": "The dict of layers to be used from this upstream source",
> +                        "type": "object",
> +                        "patternProperties": { ".*" : {
> +                            "description": "A layer from the upstream source",
> +                            "type": "object",
> +                            "additionalProperties": false,
> +                            "properties": {
> +                                "subpath": {
> +                                    "description": "The subpath (relative to the source root) for this layer. Omit if the source root is the layer path",
> +                                    "type": "string"
> +                                },
> +                                "buildconfigs": {
> +                                    "description": "A dict of template build configurations in the layer, keyed by relative path to the layer root",
> +                                    "type": "object",
> +                                    "patternProperties": { ".*" : {
> +                                        "description": "Properties of a build confuguration template",
> +                                        "type": "object",
> +                                        "additionalProperties": false,
> +                                        "properties":{}
> +                                    }}
> +                                },
> +                                "machines": {
> +                                    "description": "A dict of machine definitions in the layer",
> +                                    "type": "object",
> +                                    "patternProperties": { ".*" : {
> +                                        "description": "Properties of a machine definition",
> +                                        "type": "object",
> +                                        "additionalProperties": false,
> +                                        "properties":{}
> +                                    }}
> +                                },
> +                                "distros": {
> +                                    "description": "A dict of distro definitions in the layer",
> +                                    "type": "object",
> +                                    "patternProperties": { ".*" : {
> +                                        "description": "Properties of a distro definition",
> +                                        "type": "object",
> +                                        "additionalProperties": false,
> +                                        "properties":{}
> +                                    }}
> +                                }
> +                            }
> +                        }}
> +                    },
> +                    "git-remote": {
> +                                "description": "A remote git source from which to fetch",
> +                                "type": "object",
> +                                "additionalProperties": false,
> +                                "required": [
> +                                    "rev"
> +                                ],
> +                                "properties": {
> +                                    "branch": {
> +                                        "description": "The git branch to fetch (optional)",
> +                                        "type": "string"
> +                                    },
> +                                    "rev": {
> +                                        "description": "The git revision to checkout",
> +                                        "type": "string"
> +                                    },
> +                                    "describe": {
> +                                        "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).",
> +                                        "type": "string"
> +                                    },
> +                                    "remotes": {
> +                                        "description": "The dict of git remotes to add to this repository",
> +                                        "type": "object",
> +                                        "patternProperties": { ".*" : {
> +                                            "description": "A git remote",
> +                                            "type": "object",
> +                                            "addtionalProperties": false,
> +                                            "required": [
> +                                                "uri"
> +                                            ],
> +                                            "properties": {
> +                                                "uri": {
> +                                                    "description": "The URI for the remote",
> +                                                    "type": "string"
> +                                                }
> +                                            }
> +                                        }}
> +                                    }
> +                                }
> +                    }
> +                }
> +            }
> +        }}
> +    }
> +}
> --
> 2.30.2
>

On Fri, 8 Jul 2022 at 21:33, Joshua Watt <jpewhacker@gmail.com> wrote:
> > -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items.
> > This allows using them as dicts inside Python, and makes the json more compact and readable.
>
> As long as the order is truly irrelevant because different
> implementations (and even different versions of Python!) will result
> in a different processing order.

Yes, for all of those things it's irrelevant.

> > +++ b/meta/files/layers.example.json
> > @@ -0,0 +1,115 @@
> > +{
>
> Missing the version? Although I see it's in the schema.

It's at the end of the file. File is written via python's json module
which decides where it goes.

> > +    "sources": {
> > +        "meta-alex": {
> > +            "contains_this_file": true,
> > +            "git-remote": {
>
> The problem with this is that there has been some desire for other
> fetching mechanisms; Perhaps we can say git is only supported, but
> that was the reason I used `oneOf`: it allows for other fetching types
> in the future.

'git-remote' is not a mandatory property in the json, it can be
absent, and instead there can be 'svn-remote', 'tarball' or
'delivered-by-doves'. Anything you want. Just propose an object
definition into the schema.

>
> > +                "branch": "master",
> > +                "describe": "",
> > +                "remotes": {
> > +                    "remote-alex": {
> > +                        "uri": "https://github.com/kanavin/meta-alex"
> > +                    }
>
> Multiple remotes are nice.... but not strictly necessary, since you
> can only have one rev checked out. I don't care too much one way or
> another but it's something to think about since it makes the fetching
> more complicated.

It isn't at all more complicated to handle multiple remotes in the
fetch script which I suggest you look at and try out. I don't want to
have to guess which of the remotes is the right one either when
writing the json, or when using it: I'm writing them all, and fetching
from all as well, ignoring the failures (the failure only happens if
needed revision is absent and can't be checked out). You can write and
check out single remotes if you so wish, but then you have to decide
how to choose out of several.

>
> > +                },
> > +                "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5"
> > +            },
> > +            "layers": {
> > +                "meta-alex": {
> > +                    "buildconfigs": {
> > +                        "conf/templates/configuration-gadget": {},
> > +                        "conf/templates/configuration-gizmo": {}
>
> I think these fall under the same category as the machines below, but
> I don't use  TEMPLATECONF so I'm not sure.

They're not quite same as machines or distros, as they can be anywhere
in the layer. This saves you the trouble of walking over the layer
tree, especially when look at the layer repo through the browser.

> > +                "meta-intel": {
> > +                    "machines": {
> > +                        "intel-core2-32": {},
> > +                        "intel-corei7-64": {},
> > +                        "intel-skylake-64": {}
>
> Why does this need to be listed here? Isn't this just the contents of
> the conf/machine directory in the layer? I really don't want to be
> duplicating information; IHMO it's better to let the layer itself be
> canonical about what machines/distros/etc. it supports

Then you would have to check out all the layers first and look at them
one by one. The idea is that this information is available upfront,
and for all of the layers in a build. It can be used, for example to
be used by Richard's proposed oe-setup tool, and as the json is never
written or edited manually, it won't get out of sync wrt git revision
it lists for the layer.

The plain list of machines isn't that interesting, but if it comes
with additional information such as descriptions, architectures or
kernel versions etc., then it's quite useful to capture this in a
standard format. We could maybe look at how buildroot handles this.

Alex

On Fri, Jul 8, 2022 at 3:03 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> On Fri, 8 Jul 2022 at 21:33, Joshua Watt <jpewhacker@gmail.com> wrote:
> > > -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items.
> > > This allows using them as dicts inside Python, and makes the json more compact and readable.
> >
> > As long as the order is truly irrelevant because different
> > implementations (and even different versions of Python!) will result
> > in a different processing order.
>
> Yes, for all of those things it's irrelevant.

I did think of one way in which the order matters: If you are planning
on populating BBLAYERS from this list, the order would matter. If you
are doing that (or we want to allow it at all) it would need to be a
list instead of a dictionary.

>
> > > +++ b/meta/files/layers.example.json
> > > @@ -0,0 +1,115 @@
> > > +{
> >
> > Missing the version? Although I see it's in the schema.
>
> It's at the end of the file. File is written via python's json module
> which decides where it goes.

Yep sorry.

>
> > > +    "sources": {
> > > +        "meta-alex": {
> > > +            "contains_this_file": true,
> > > +            "git-remote": {
> >
> > The problem with this is that there has been some desire for other
> > fetching mechanisms; Perhaps we can say git is only supported, but
> > that was the reason I used `oneOf`: it allows for other fetching types
> > in the future.
>
> 'git-remote' is not a mandatory property in the json, it can be
> absent, and instead there can be 'svn-remote', 'tarball' or
> 'delivered-by-doves'. Anything you want. Just propose an object
> definition into the schema.
>
> >
> > > +                "branch": "master",
> > > +                "describe": "",
> > > +                "remotes": {
> > > +                    "remote-alex": {
> > > +                        "uri": "https://github.com/kanavin/meta-alex"
> > > +                    }
> >
> > Multiple remotes are nice.... but not strictly necessary, since you
> > can only have one rev checked out. I don't care too much one way or
> > another but it's something to think about since it makes the fetching
> > more complicated.
>
> It isn't at all more complicated to handle multiple remotes in the
> fetch script which I suggest you look at and try out. I don't want to
> have to guess which of the remotes is the right one either when
> writing the json, or when using it: I'm writing them all, and fetching
> from all as well, ignoring the failures (the failure only happens if
> needed revision is absent and can't be checked out). You can write and
> check out single remotes if you so wish, but then you have to decide
> how to choose out of several.
>
> >
> > > +                },
> > > +                "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5"
> > > +            },
> > > +            "layers": {
> > > +                "meta-alex": {
> > > +                    "buildconfigs": {
> > > +                        "conf/templates/configuration-gadget": {},
> > > +                        "conf/templates/configuration-gizmo": {}
> >
> > I think these fall under the same category as the machines below, but
> > I don't use  TEMPLATECONF so I'm not sure.
>
> They're not quite same as machines or distros, as they can be anywhere
> in the layer. This saves you the trouble of walking over the layer
> tree, especially when look at the layer repo through the browser.

Configuring _what_ you want to build is probably even more contentious
that describing how to fetch your layers, so for starters I'd really
like to just focus on the layer fetching; TEMPLATECONF is far from the
only method for doing build configuration and I don't think we should
bake it into the "layer setup" piece at this point (same for machine &
distro, although they aren't generally contentious, see below).

>
> > > +                "meta-intel": {
> > > +                    "machines": {
> > > +                        "intel-core2-32": {},
> > > +                        "intel-corei7-64": {},
> > > +                        "intel-skylake-64": {}
> >
> > Why does this need to be listed here? Isn't this just the contents of
> > the conf/machine directory in the layer? I really don't want to be
> > duplicating information; IHMO it's better to let the layer itself be
> > canonical about what machines/distros/etc. it supports
>
> Then you would have to check out all the layers first and look at them
> one by one. The idea is that this information is available upfront,
> and for all of the layers in a build. It can be used, for example to
> be used by Richard's proposed oe-setup tool, and as the json is never
> written or edited manually, it won't get out of sync wrt git revision
> it lists for the layer.
>
> The plain list of machines isn't that interesting, but if it comes
> with additional information such as descriptions, architectures or
> kernel versions etc., then it's quite useful to capture this in a
> standard format. We could maybe look at how buildroot handles this.

I suspect that not including the machine/distro/template stuff for now
might give us better adoption as standard way to describe layers,
which is another reason I'm a little hesitant to include them (in
addition to duplicating information available in the layer itself)

>
> Alex

On Mon, 11 Jul 2022 at 17:46, Joshua Watt <jpewhacker@gmail.com> wrote:
> I did think of one way in which the order matters: If you are planning
> on populating BBLAYERS from this list, the order would matter. If you
> are doing that (or we want to allow it at all) it would need to be a
> list instead of a dictionary.

This would not be allowed at all. This dict is strictly for checkout,
actual BBLAYERS is set in bblayers.conf.sample coming from a
configuration template pointed to by the json or from some other
description in the json or elsewhere.

> > They're not quite same as machines or distros, as they can be anywhere
> > in the layer. This saves you the trouble of walking over the layer
> > tree, especially when look at the layer repo through the browser.
>
> Configuring _what_ you want to build is probably even more contentious
> that describing how to fetch your layers, so for starters I'd really
> like to just focus on the layer fetching; TEMPLATECONF is far from the
> only method for doing build configuration and I don't think we should
> bake it into the "layer setup" piece at this point (same for machine &
> distro, although they aren't generally contentious, see below).

What are those other methods? TEMPLATECONF is contentious if you have
ugly legacy scripts (understood by no one) that write into local.conf,
and don't want to replace that with a clean set of templates. For new
projects or teaching beginners I really would want to encourage only
static, prefabricated templates that restrict local.conf to just two
lines: distro and machine. Describing where templateconfs are is not
mandatory in the scheme; if you don't want them, no one forces you to
have the respective items in the json or use them if they're there.
But they are standard, simple, and part of the core, and they work. I
don't see why we can't have them now, and then argue about those other
ways, what use cases they serve (that static templates cannot), and
how to integrate them into the schema and tools.

> I suspect that not including the machine/distro/template stuff for now
> might give us better adoption as standard way to describe layers,
> which is another reason I'm a little hesitant to include them (in
> addition to duplicating information available in the layer itself)

None of these are mandatory properties. You do not have to write them
into actual jsons, and if they are present, you are free to ignore
them. But I do want them, as I want something that can be used to
actually set up a build all the way to bitbake, instead of just going
halfway, and not being as functional as kas. We can make disclaimers
in the schema description fields.

Alex