bitbake: doc: Add section for variable context

From: Simone Weiß <simone.p.weiss@posteo.com>

This is inspired by the same section in the yocto-docs. It aims to provide
information in what contexts(recipes, .conf, bbclass,...) a variable is usually
used. For that I tried to group similar variables, so that a short overview is
given. This was inspired by [YOCTO #14072], but of course does not implement a
warning if a variable is used in an unintended context.

Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
---
 ...bake-user-manual-ref-variables-context.rst | 77 +++++++++++++++++++
 bitbake/doc/index.rst                         |  1 +
 2 files changed, 78 insertions(+)
 create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst

Hi Simone,

Thanks a lot for the new document. It looks super useful!

On 2/11/24 at 11:40, Simone Weiß wrote:
> From: Simone Weiß <simone.p.weiss@posteo.com>
>
> This is inspired by the same section in the yocto-docs. It aims to provide
> information in what contexts(recipes, .conf, bbclass,...) a variable is usually
> used. For that I tried to group similar variables, so that a short overview is
> given. This was inspired by [YOCTO #14072], but of course does not implement a
> warning if a variable is used in an unintended context.
>
> Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
> ---
>   ...bake-user-manual-ref-variables-context.rst | 77 +++++++++++++++++++
>   bitbake/doc/index.rst                         |  1 +
>   2 files changed, 78 insertions(+)
>   create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst
>
> diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst

Oops, you should generate this patch from the bitbake source repository, 
not from Poky. I managed to tweak your patch so that it applies, but you 
don't want to add extra work on RP's shoulders.

> new file mode 100644
> index 0000000000..9b94e09299
> --- /dev/null
> +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst
> @@ -0,0 +1,77 @@
> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
> +
> +****************
> +Variable Context
> +****************
> +
> +Variables might only have an impact or can be used in certain contexts. Some
> +should only be used in global files like ``.conf``, while others are intended only
> +for local files like ``.bb``. This chapter aims to describe some important variable
> +contexts.
> +
> +.. _ref-varlcontext-configuration:
> +
> +#.  **BitBakes own configuration**: Variables starting with `BB_` usually


For the prefix of variable names, I'd use double quotes through out the 
document (here ``BB_``), as you did for "PREFERRED_" at the end by the 
way ;)
This looks more natural to me.

> +    configure the behaviour of BitBake itself. For example, one could configure:
> +
> +    - system resources, like disk space to be used (:term:`BB_DISKMON_DIRS`),
> +      or number of tasks to be run in parallel by BitBake (:term:`BB_NUMBER_THREADS`).

s/number of tasks/the number of tasks/


Another thing is that we usually capitalize the list items, so here it 
would be "- System resources" instead. Could you change this everywhere 
in your patch?

> +
> +    - how the fetchers shall behave, e.g., :term:`BB_FETCH_PREMIRRORONLY` is used
> +      by BitBake to determine if BitBake's fetcher shall search only
> +      :term:`PREMIRRORS` for files.
> +
> +    Those variables are usually configured globally.
> +
> +#.  **BitBake configuration**: There are variables:
> +
> +    - like :term:`B` or :term:`T`, that are used to specify directories used by
> +      BitBake during the build of a particular recipe. Those variables are
> +      specified in ``bitbake.conf``. Some, like :term:`B`, are quite often
> +      overwritten in recipes.
> +
> +    - starting with `FAKEROOT` configure how the fakeroot command is handled.


I would improve this line by writing:

- starting with ``FAKEROOT``, to configure how the ``fakeroot`` command 
is handled.

> +      Those are usually set by ``bitbake.conf`` and might get adapted in a
> +      ``bbclass``.
> +
> +    - detailing where BitBake will store and fetch information from for


s/from for/from, for/

> +      preservation between build runs like :term:`CACHE`, :term:`DL_DIR` or

s/preservation/data reuse/?
This may be more explicit.

> +      :term:`PERSISTENT_DIR`. Those are usually global.
> +
> +
> +#.  **Layers and files**: Variables starting with `LAYER` configure how
> +    BitBake handles layers. Additionally, variables starting with `BB` configure
> +    how layers and files are handled. For example:
> +
> +    - :term:`LAYERDEPENDS` is used to configure on which layers a given layer
> +      depends.
> +
> +    - the configured layers are contained in :term:`BBLAYERS` and files in
> +      :term:`BBFILES`.
> +
> +    Those variables are often used in the files ``layer.conf`` and ``bblayers.conf``.
> +
> +#.  **Recipes and packages**: Variables handling recipes and packages can be split into:
> +
> +    - :term:`PN`, :term:`PV` or :term:`PF` for example, contain information about the
> +      name or revision of a recipe or package. Usually, the default set in
> +      ``bitbake.conf`` is used, but those are from time to time overwritten in recipes.
> +
> +    - :term:`SUMMARY`, :term:`DESCRIPTION`, :term:`LICENSE` or :term:`HOMEPAGE`
> +      contain the expected information and should be set specifically for every
> +      recipe.
> +
> +    - In recipes, variables are also used to control build and runtime
> +      dependencies between recipes/packages with other recipes/packages. The
> +      most common should be: :term:`PROVIDES`, :term:`RPROVIDES`, :term:`DEPENDS`,
> +      and :term:`RDEPENDS`.
> +
> +    - There are further variables starting with `SRC` that specify the sources in
> +      a recipe like :term:`SRC_URI` or :term:`SRCDATE`. Those are also usually set
> +      in recipes.
> +
> +    - Which version or  provider of a recipe should be given preference when
> +      multiple recipes would provide the same item, is controlled by variables
> +      starting with ``PREFERRED_``. Those are normally set in the configuration
> +      files of a ``MACHINE`` or ``DISTRO``.
> +
> diff --git a/bitbake/doc/index.rst b/bitbake/doc/index.rst
> index 3ff8b1580f..30d2a5e3c7 100644
> --- a/bitbake/doc/index.rst
> +++ b/bitbake/doc/index.rst
> @@ -15,6 +15,7 @@ BitBake User Manual
>      bitbake-user-manual/bitbake-user-manual-metadata
>      bitbake-user-manual/bitbake-user-manual-fetching
>      bitbake-user-manual/bitbake-user-manual-ref-variables
> +   bitbake-user-manual/bitbake-user-manual-ref-variables-context
>      bitbake-user-manual/bitbake-user-manual-hello


Weird, I don't see the new document in the left menu bar after the 
variable index.

IMHO, it would make more sense to have this right after the 
"bitbake-user-manual-metadata" document, because it would be too late 
after the variable index.
You'll also have to change the header settings (title with "===" instead 
of "***").

Thanks in advance for the V2. There's not so much to change :}
Cheers
Michael.

Hi Simone,

On 2/11/24 11:40, Simone Weiß via lists.yoctoproject.org wrote:
> [You don't often get email from simone.p.weiss=posteo.com@lists.yoctoproject.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> From: Simone Weiß <simone.p.weiss@posteo.com>
> 
> This is inspired by the same section in the yocto-docs. It aims to provide

Should we make yocto-docs's point to that new page once merged? I don't 
like duplicated documentation :)

> information in what contexts(recipes, .conf, bbclass,...) a variable is usually
> used. For that I tried to group similar variables, so that a short overview is
> given. This was inspired by [YOCTO #14072], but of course does not implement a
> warning if a variable is used in an unintended context.
> 
> Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
> ---
>   ...bake-user-manual-ref-variables-context.rst | 77 +++++++++++++++++++
>   bitbake/doc/index.rst                         |  1 +
>   2 files changed, 78 insertions(+)
>   create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst
> 
> diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst
> new file mode 100644
> index 0000000000..9b94e09299
> --- /dev/null
> +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst
> @@ -0,0 +1,77 @@
> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
> +
> +****************
> +Variable Context
> +****************
> +
> +Variables might only have an impact or can be used in certain contexts. Some
> +should only be used in global files like ``.conf``, while others are intended only
> +for local files like ``.bb``. This chapter aims to describe some important variable
> +contexts.
> +

I think we have to be careful with the variables set in classes, 
especially because if I'm not mistaken the ones in meta/classes-global/ 
are to be defined in the conf context? and the ones in meta/classes/ 
could be either defined in the conf or in the recipe context?

> +.. _ref-varlcontext-configuration:
> +

Trying to figure out if the l is a typo?

> +#.  **BitBakes own configuration**: Variables starting with `BB_` usually

s/BitBakes/BitBake's/ I think?

> +    configure the behaviour of BitBake itself. For example, one could configure:
> +
> +    - system resources, like disk space to be used (:term:`BB_DISKMON_DIRS`),
> +      or number of tasks to be run in parallel by BitBake (:term:`BB_NUMBER_THREADS`).
> +
> +    - how the fetchers shall behave, e.g., :term:`BB_FETCH_PREMIRRORONLY` is used
> +      by BitBake to determine if BitBake's fetcher shall search only
> +      :term:`PREMIRRORS` for files.
> +
> +    Those variables are usually configured globally.
> +
> +#.  **BitBake configuration**: There are variables:
> +
> +    - like :term:`B` or :term:`T`, that are used to specify directories used by
> +      BitBake during the build of a particular recipe. Those variables are
> +      specified in ``bitbake.conf``. Some, like :term:`B`, are quite often
> +      overwritten in recipes.
> +
> +    - starting with `FAKEROOT` configure how the fakeroot command is handled.
> +      Those are usually set by ``bitbake.conf`` and might get adapted in a
> +      ``bbclass``.
> +
> +    - detailing where BitBake will store and fetch information from for
> +      preservation between build runs like :term:`CACHE`, :term:`DL_DIR` or
> +      :term:`PERSISTENT_DIR`. Those are usually global.
> +
> +
> +#.  **Layers and files**: Variables starting with `LAYER` configure how
> +    BitBake handles layers. Additionally, variables starting with `BB` configure
> +    how layers and files are handled. For example:
> +
> +    - :term:`LAYERDEPENDS` is used to configure on which layers a given layer
> +      depends.
> +
> +    - the configured layers are contained in :term:`BBLAYERS` and files in
> +      :term:`BBFILES`.
> +
> +    Those variables are often used in the files ``layer.conf`` and ``bblayers.conf``.
> +
> +#.  **Recipes and packages**: Variables handling recipes and packages can be split into:
> +
> +    - :term:`PN`, :term:`PV` or :term:`PF` for example, contain information about the
> +      name or revision of a recipe or package. Usually, the default set in
> +      ``bitbake.conf`` is used, but those are from time to time overwritten in recipes.
> +
> +    - :term:`SUMMARY`, :term:`DESCRIPTION`, :term:`LICENSE` or :term:`HOMEPAGE`
> +      contain the expected information and should be set specifically for every
> +      recipe.
> +
> +    - In recipes, variables are also used to control build and runtime
> +      dependencies between recipes/packages with other recipes/packages. The
> +      most common should be: :term:`PROVIDES`, :term:`RPROVIDES`, :term:`DEPENDS`,
> +      and :term:`RDEPENDS`.
> +
> +    - There are further variables starting with `SRC` that specify the sources in
> +      a recipe like :term:`SRC_URI` or :term:`SRCDATE`. Those are also usually set
> +      in recipes.
> +
> +    - Which version or  provider of a recipe should be given preference when

Spurious white space between or and provider.

> +      multiple recipes would provide the same item, is controlled by variables
> +      starting with ``PREFERRED_``. Those are normally set in the configuration
> +      files of a ``MACHINE`` or ``DISTRO``.
> +

There's another context which is the task context. It's also important 
to document it as a variable that is added, modified or deleted in a 
task will NOT be present, NOT be modified or still be present in another 
task.

I think a nice visualization could be concentric circles.

The outer one is the conf context, it applies to everything but can only 
be modified from conf files (distro, local conf, layer conf, machine 
conf, etc...). Then the one inside would be the recipe context, it 
applies to everything within THAT recipe and its bbappend, it can only 
be changed from the recipe itself or bbappends. Then the one inside 
would be the task context, it applies to the task only, it can be 
changed from within the task itself or via an append/prepend of the task 
itself (maybe prefuncs/postfuncs too?).

The outer circle would be one big circle (for one "kind", e.g. native, 
target, nativesdk,...), then inside you would have a plethora of smaller 
middle circle, each for a recipe, and inside each smaller middle circle 
you would have a plethora of even smaller inner circles, each for a 
task. This would make it obvious that you cannot modify the context of a 
middle circle from another middle circle, they are completely different.

Cheers,
Quentin