[7/7] ref-manual: add migration section for 3.2

Submitted by Paul Eggleton on Oct. 28, 2020, 1:18 a.m. | Patch ID: 177596

Details

Message ID 18956e99fafa0d898c37a785e44015af2eb532d8.1603839557.git.paul.eggleton@linux.microsoft.com
State New
Headers show

Commit Message

Paul Eggleton Oct. 28, 2020, 1:18 a.m.
From: Paul Eggleton <paul.eggleton@microsoft.com>

This covers most of the changes that would require action on the part of
the user that I was able to see by scouring the commits. Some of the text
was borrowed from commit messages and edited.

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
---
 documentation/ref-manual/migration-3.2.rst | 261 +++++++++++++++++++++++++++++
 documentation/ref-manual/migration.rst     |   1 +
 2 files changed, 262 insertions(+)
 create mode 100644 documentation/ref-manual/migration-3.2.rst

Patch hide | download patch | download mbox

diff --git a/documentation/ref-manual/migration-3.2.rst b/documentation/ref-manual/migration-3.2.rst
new file mode 100644
index 0000000..a56c5d4
--- /dev/null
+++ b/documentation/ref-manual/migration-3.2.rst
@@ -0,0 +1,261 @@ 
+Moving to the Yocto Project 3.2 Release
+=======================================
+
+This section provides migration information for moving to the Yocto
+Project 3.2 Release from the prior release.
+
+.. _migration-3.2-minimum-system-requirements:
+
+Minimum system requirements
+---------------------------
+
+``gcc`` version 6.0 is now required at minimum on the build host. For older
+host distributions where this is not available, you can use the
+``buildtools-extended-tarball`` (easily installable using
+``scripts/install-buildtools``).
+
+
+.. _migration-3.2-removed-recipes:
+
+Removed recipes
+---------------
+
+The following recipes have been removed:
+
+- ``bjam-native``: replaced by ``boost-build-native``
+- ``avahi-ui``: folded into the main ``avahi`` recipe - the GTK UI can be disabled using :term:`PACKAGECONFIG` for ``avahi``.
+- ``build-compare``: no longer needed with the removal of the ``packagefeed-stability`` class
+- ``dhcp``: obsolete, functionally replaced by ``dhcpcd`` and ``kea``
+- ``libmodulemd-v1``: replaced by ``libmodulemd``
+- ``packagegroup-core-device-devel``: obsolete
+
+
+.. _migration-3.2-removed-classes:
+
+Removed classes
+---------------
+
+The following classes (.bbclass files) have been removed:
+
+-  ``spdx``: obsolete - the Yocto Project is a strong supporter of SPDX, but this class was old code using a dated approach and had the potential to be misleading. The ``meta-sdpxscanner`` layer is a much more modern and active approach to handling this and is recommended as a replacement.
+
+- ``packagefeed-stability``: this class had become obsolete with the advent of hash equivalence and reproducible builds.
+
+
+pseudo path filtering and mismatch behaviour
+--------------------------------------------
+
+pseudo now operates on a filtered subset of files. This is a significant change
+to the way pseudo operates within OpenEmbedded - by default, pseudo monitors and
+logs (adds to its database) any file created or modified whilst in a ``fakeroot``
+environment. However, there are large numbers of files that we simply don't care
+about the permissions of whilst in that ``fakeroot`` context, for example ${:term:`S`}, ${:term:`B`}, ${:term:`T`},
+${:term:`SSTATE_DIR`}, the central sstate control directories, and others.
+
+As of this release, new functionality in pseudo is enabled to ignore these
+directory trees (controlled using a new :term:`PSEUDO_IGNORE_PATHS` variable)
+resulting in a cleaner database with less chance of "stray" mismatches if files
+are modified outside pseudo context. It also should reduce some overhead from
+pseudo as the interprocess round trip to the server is avoided.
+
+There is a possible complication where some existing recipe may break, for
+example, a recipe was found to be writing to ``${B}/install`` for
+``make install`` in ``do_install`` and since ``${B}`` is listed as not to be tracked,
+there were errors trying to ``chown root`` for files in this location. Another
+example was the ``tcl`` recipe where the source directory ``S`` is set to a
+subdirectory of the source tree but files were written out to the directory
+structure above that subdirectory. For these types of cases in your own recipes,
+extend ``PSEUDO_IGNORE_PATHS`` to cover additional paths that pseudo should not
+be monitoring.
+
+In addition, pseudo's behaviour on mismatches has now been changed - rather
+than doing what turns out to be a rather dangerous "fixup" if it sees a file
+with a different path but the same inode as another file it has previously seen,
+pseudo will throw an ``abort()`` and direct you to a `wiki page <https://wiki.yoctoproject.org/wiki/Pseudo_Abort>`
+that explains how to deal with this.
+
+
+.. _migration-3.2-packagegroup-core-device-devel:
+
+packagegroup-core-device-devel no longer included in images built for qemu* machines
+------------------------------------------------------------------------------------
+
+``packagegroup-core-device-devel`` was previously added automatically to
+images build for ``qemu*`` machines, however the purpose of the group and what
+it should contain is no longer clear, and in general, adding userspace
+development items to images is best done at the image/class level; thus this
+packagegroup was removed.
+
+This packagegroup previously pulled in the following:
+
+- ``distcc-config``
+- ``nfs-export-root``
+- ``bash``
+- ``binutils-symlinks``
+
+If you still need any of these in your image build for a ``qemu*`` machine
+then you will add them explicitly to :term:`IMAGE_INSTALL` or another
+appropriate place in the dependency chain for your image (if you have not
+already done so).
+
+
+.. _migration-3.2-dhcp:
+
+DHCP server/client replaced
+---------------------------
+
+The ``dhcp`` software package has become unmaintained and thus has been
+functionally replaced by ``dhcpcd`` (client) and ``kea`` (server). You will
+need to replace references to the recipe/package names as appropriate - most
+commonly, at the package level ``dhcp-client`` should be replaced by
+``dhcpcd`` and ``dhcp-server`` should be replaced by ``kea``. If you have any
+custom configuration files for these they will need to be adapted - refer to
+the upstream documentation for ``dhcpcd`` and ``kea`` for further details.
+
+
+.. _migration-3.2-packaging-changes:
+
+Packaging changes
+-----------------
+
+- ``python3``: the ``urllib`` python package has now moved into the core package, as it is used more commonly than just netclient (e.g. email, xml, mimetypes, pydoc). In addition, the ``pathlib`` module is now also part of the core package.
+
+- ``iptables``: ``iptables-apply`` and ``ip6tables-apply`` have been split out to their own package to avoid a bash dependency in the main ``iptables`` package
+
+
+.. _migration-3.2-package-qa-checks:
+
+Package QA check changes
+------------------------
+
+Previously, the following package QA checks triggered warnings, however they can
+be indicators of genuine underlying problems and are therefore now treated as
+errors:
+
+- :ref:`already-stripped <qa-check-already-stripped>`
+- :ref:`compile-host-path <qa-check-compile-host-path>`
+- :ref:`installed-vs-shipped <qa-check-installed-vs-shipped>`
+- :ref:`ldflags <qa-check-ldflags>`
+- :ref:`pn-overrides <qa-check-pn-overrides>`
+- :ref:`rpaths <qa-check-rpaths>`
+- :ref:`staticdev <qa-check-staticdev>`
+- :ref:`unknown-configure-option <qa-check-unknown-configure-option>`
+- :ref:`useless-rpaths <qa-check-useless-rpaths>`
+
+In addition, the following new checks were added and default to triggering an error:
+
+- :ref:`shebang-size <qa-check-shebang-size>`: Check for shebang (#!) lines longer than 128 characters, which can give an error at runtime depending on the operating system.
+
+- :ref:`unhandled-features-check <qa-check-unhandled-features-check>`: Check if any of the variables supported by the :ref:`features_check <ref-classes-features_check>` class is set while not inheriting the class itself.
+
+- :ref:`missing-update-alternatives <qa-check-missing-update-alternatives>`: Check if the recipe sets the :term:`ALTERNATIVE` variable for any of its packages, and does not inherit the :ref:`update-alternatives <ref-classes-update-alternatives>` class.
+
+- A trailing slash or duplicated slashes in the value of :term:`S` or :term:`B` will now trigger a warning so that they can be removed and path comparisons can be more reliable - remove any instances of these in your recipes if the warning is displayed.
+
+
+.. _migration-3.2-src-uri-file-globbing:
+
+Globbing no longer supported in ``file://`` entries in ``SRC_URI``
+------------------------------------------------------------------
+
+Globbing (``*`` and ``?`` wildcards) in ``file://`` URLs within :term:`SRC_URI`
+did not properly support file checksums, thus changes to the source files
+would not always change the do_fetch task checksum, and consequently would
+not ensure that the changed files would be incorporated in subsequent builds.
+
+Unfortunately it is not practical to make globbing work generically here, so
+the decision was taken to remove support for globs in ``file://`` URLs.
+If you have any usage of these in your recipes, then you will now need to
+either add each of the files that you expect to match explicitly, or
+alternatively if you still need files to be pulled in dynamically, put the
+files into a subdirectory and reference that instead.
+
+
+.. _migration-3.2-deploydir-clean:
+
+deploy class now cleans ``DEPLOYDIR`` before ``do_deploy``
+----------------------------------------------------------
+
+``do_deploy`` as implemented in the :ref:`deploy <ref-classes-deploy>` class now cleans up ${:term:`DEPLOYDIR`} before running, just as ``do_install`` cleans up ${:term:`D`} before running. This reduces the risk of DEPLOYDIR being accidentally contaminated by files from previous runs, possibly even with different config, in case of incremental builds.
+
+Most recipes and classes that inherit the ``deploy`` class or interact with ``do_deploy`` are unlikely to be affected by this unless they add ``prefuncs`` to ``do_deploy`` *which also* put files into ``${DEPLOYDIR}`` - these should be refactored to use ``do_deploy_prepend`` instead.
+
+
+.. _migration-3.2-nativesdk-sdk-provides-dummy:
+
+Custom SDK / SDK-style recipes need to include ``nativesdk-sdk-provides-dummy``
+-------------------------------------------------------------------------------
+
+All ``nativesdk`` packages require ``/bin/sh`` due to their postinstall scriptlets, thus this package has to be dummy-provided within the SDK and ``nativesdk-sdk-provides-dummy`` now does this. If you have a custom SDK recipe (or your own SDK-style recipe similar to e.g. ``buildtools-tarball``), you will need to ensure ``nativesdk-sdk-provides-dummy`` or an equivalent is included in :term:`TOOLCHAIN_HOST_TASK`.
+
+
+``ld.so.conf`` now moved back to main ``glibc`` package
+-------------------------------------------------------
+
+There are cases where one doesn't want ldconfig on target (e.g. for
+read-only root filesystems, it's rather pointless), yet one still
+needs ``/etc/ld.so.conf`` to be present at image build time:
+
+When some recipe installs libraries to a non-standard location, and
+therefore installs in a file in ``/etc/ld.so.conf.d/foo.conf``, we
+need ``/etc/ld.so.conf`` containing: ::
+
+  include /etc/ld.so.conf.d/*.conf
+
+in order to get those other locations picked up.
+
+Thus ``/etc/ld.so.conf`` is now in the main ``glibc`` package so that
+there's always an ``ld.so.conf`` present when the build-time ``ldconfig``
+runs towards the end of image construction.
+
+The ``ld.so.conf`` and ``ld.so.conf.d/*.conf`` files do not take up
+significant space (at least not compared to the ~700kB ``ldconfig`` binary), and they
+might be needed in case ``ldconfig`` is installable, so they are left
+in place after the image is built. Technically it would be possible to
+remove them if desired, though it would not be trivial if you still
+wanted the build-time ldconfig to function (:term:`ROOTFS_POSTPROCESS_COMMAND`
+will not work as ``ldconfig`` is run after the functions referred to
+by that variable).
+
+
+.. _migration-3.2-virgl:
+
+Host DRI drivers now used for GL support within ``runqemu``
+-----------------------------------------------------------
+
+``runqemu`` now uses the mesa-native libraries everywhere virgl is used
+(i.e. when ``gl``, ``gl-es`` or ``egl-headless`` options are specified),
+but instructs them to load DRI drivers from the host. Unfortunately this
+may not work well with proprietary graphics drivers such as those from
+Nvidia; if you are using such drivers then you may need to switch to an
+alternative (such as Nouveau in the case of Nvidia hardware) or avoid
+using the GL options.
+
+
+.. _migration-3.2-initramfs-suffix:
+
+initramfs images now use a blank suffix
+---------------------------------------
+
+The reference initramfs images (``core-image-minimal-initramfs``,
+``core-image-tiny-initramfs`` and ``core-image-testmaster-initramfs``) now
+set an empty string for :term:`IMAGE_NAME_SUFFIX`, which otherwise defaults
+to ``".rootfs"``. These images aren't root filesystems and thus the rootfs
+label didn't make sense. If you are looking for the output files generated
+by these image recipes directly then you will need to adapt to the new
+naming without the ``.rootfs`` part.
+
+
+.. _migration-3.2-misc:
+
+Miscellaneous changes
+---------------------
+
+- Support for the long-deprecated ``PACKAGE_GROUP`` variable has now been removed - replace any remaining instances with :term:`FEATURE_PACKAGES`.
+- The ``FILESPATHPKG`` variable, having been previously deprecated, has now been removed. Replace any remaining references with appropriate use of :term:`FILESEXTRAPATHS`.
+- Erroneous use of ``inherit +=`` (instead of ``INHERIT +=``) in a configuration file now triggers an error instead of silently being ignored.
+- ptest support has been removed from the ``kbd`` recipe, as upstream has moved to autotest which is difficult to work with in a cross-compilation environment.
+- ``oe.utils.is_machine_specific()`` and ``oe.utils.machine_paths()`` have been removed as their utility was questionable. In the unlikely event that you have references to these in your own code, then the code will need to be reworked.
+- The ``i2ctransfer`` module is now disabled by default when building ``busybox`` in order to be consistent with disabling the other i2c tools there. If you do wish the i2ctransfer module to be built in busybox then add ``CONFIG_I2CTRANSFER=y`` to your custom busybox configuration.
+- In the ``Upstream-Status`` header convention for patches, ``Accepted`` has been replaced with ``Backport`` as these almost always mean the same thing i.e. the patch is already upstream and may need to be removed in a future recipe upgrade. If you are adding these headers to your own patches then use ``Backport`` to indicate that the patch has been sent upstream.
+- The ``tune-supersparc.inc`` tune file has been removed as it does not appear to be widely used and no longer works.
diff --git a/documentation/ref-manual/migration.rst b/documentation/ref-manual/migration.rst
index 20288b0..8d64a7d 100644
--- a/documentation/ref-manual/migration.rst
+++ b/documentation/ref-manual/migration.rst
@@ -27,4 +27,5 @@  notes for a given release.
    migration-2.7
    migration-3.0
    migration-3.1
+   migration-3.2
 

Comments

Quentin Schulz Oct. 28, 2020, 1:12 p.m.
Hi Paul,

On October 28, 2020 2:18:35 AM GMT+01:00, Paul Eggleton <paul.eggleton@linux.microsoft.com> wrote:
>+In addition, pseudo's behaviour on mismatches has now been changed -
>rather
>+than doing what turns out to be a rather dangerous "fixup" if it sees
>a file
>+with a different path but the same inode as another file it has
>previously seen,
>+pseudo will throw an ``abort()`` and direct you to a `wiki page
><https://wiki.yoctoproject.org/wiki/Pseudo_Abort>`

:yocto_wiki:`wiki page </wiki/Pseudo_Abort>`

>+that explains how to deal with this.
>+
>+
>+.. _migration-3.2-packagegroup-core-device-devel:
>+
>+packagegroup-core-device-devel no longer included in images built for
>qemu* machines
>+------------------------------------------------------------------------------------
>+
>+``packagegroup-core-device-devel`` was previously added automatically
>to
>+images build for ``qemu*`` machines, however the purpose of the group

s/build/built/?

>and what
>+it should contain is no longer clear, and in general, adding userspace
>+development items to images is best done at the image/class level;
>thus this
>+packagegroup was removed.
>+
>+This packagegroup previously pulled in the following:
>+
>+- ``distcc-config``
>+- ``nfs-export-root``
>+- ``bash``
>+- ``binutils-symlinks``
>+
>+If you still need any of these in your image build for a ``qemu*``

s/build/built/?

>machine
>+then you will add them explicitly to :term:`IMAGE_INSTALL` or another
>+appropriate place in the dependency chain for your image (if you have
>not
>+already done so).
>+
>+
>+.. _migration-3.2-dhcp:
>+
>+DHCP server/client replaced
>+---------------------------
>+
>+The ``dhcp`` software package has become unmaintained and thus has
>been
>+functionally replaced by ``dhcpcd`` (client) and ``kea`` (server). You
>will
>+need to replace references to the recipe/package names as appropriate
>- most
>+commonly, at the package level ``dhcp-client`` should be replaced by
>+``dhcpcd`` and ``dhcp-server`` should be replaced by ``kea``. If you
>have any
>+custom configuration files for these they will need to be adapted -
>refer to
>+the upstream documentation for ``dhcpcd`` and ``kea`` for further
>details.
>+
>+
>+.. _migration-3.2-packaging-changes:
>+
>+Packaging changes
>+-----------------
>+
>+- ``python3``: the ``urllib`` python package has now moved into the
>core package, as it is used more commonly than just netclient (e.g.
>email, xml, mimetypes, pydoc). In addition, the ``pathlib`` module is
>now also part of the core package.
>+
>+- ``iptables``: ``iptables-apply`` and ``ip6tables-apply`` have been
>split out to their own package to avoid a bash dependency in the main
>``iptables`` package
>+
>+
>+.. _migration-3.2-package-qa-checks:
>+
>+Package QA check changes
>+------------------------
>+
>+Previously, the following package QA checks triggered warnings,
>however they can
>+be indicators of genuine underlying problems and are therefore now
>treated as
>+errors:
>+
>+- :ref:`already-stripped <qa-check-already-stripped>`
>+- :ref:`compile-host-path <qa-check-compile-host-path>`
>+- :ref:`installed-vs-shipped <qa-check-installed-vs-shipped>`
>+- :ref:`ldflags <qa-check-ldflags>`
>+- :ref:`pn-overrides <qa-check-pn-overrides>`
>+- :ref:`rpaths <qa-check-rpaths>`
>+- :ref:`staticdev <qa-check-staticdev>`
>+- :ref:`unknown-configure-option <qa-check-unknown-configure-option>`
>+- :ref:`useless-rpaths <qa-check-useless-rpaths>`
>+
>+In addition, the following new checks were added and default to
>triggering an error:
>+
>+- :ref:`shebang-size <qa-check-shebang-size>`: Check for shebang (#!)
>lines longer than 128 characters, which can give an error at runtime
>depending on the operating system.
>+
>+- :ref:`unhandled-features-check <qa-check-unhandled-features-check>`:
>Check if any of the variables supported by the :ref:`features_check
><ref-classes-features_check>` class is set while not inheriting the
>class itself.
>+
>+- :ref:`missing-update-alternatives
><qa-check-missing-update-alternatives>`: Check if the recipe sets the
>:term:`ALTERNATIVE` variable for any of its packages, and does not
>inherit the :ref:`update-alternatives
><ref-classes-update-alternatives>` class.
>+

If you rename the references to ref-qa-checks-<qa-check> don't forget to change them here too.

>+- A trailing slash or duplicated slashes in the value of :term:`S` or
>:term:`B` will now trigger a warning so that they can be removed and
>path comparisons can be more reliable - remove any instances of these
>in your recipes if the warning is displayed.
>+
>+
>+.. _migration-3.2-src-uri-file-globbing:
>+
>+Globbing no longer supported in ``file://`` entries in ``SRC_URI``
>+------------------------------------------------------------------
>+
>+Globbing (``*`` and ``?`` wildcards) in ``file://`` URLs within
>:term:`SRC_URI`
>+did not properly support file checksums, thus changes to the source
>files
>+would not always change the do_fetch task checksum, and consequently
>would
>+not ensure that the changed files would be incorporated in subsequent
>builds.
>+
>+Unfortunately it is not practical to make globbing work generically
>here, so
>+the decision was taken to remove support for globs in ``file://``
>URLs.
>+If you have any usage of these in your recipes, then you will now need
>to
>+either add each of the files that you expect to match explicitly, or
>+alternatively if you still need files to be pulled in dynamically, put
>the
>+files into a subdirectory and reference that instead.
>+
>+
>+.. _migration-3.2-deploydir-clean:
>+
>+deploy class now cleans ``DEPLOYDIR`` before ``do_deploy``
>+----------------------------------------------------------
>+
>+``do_deploy`` as implemented in the :ref:`deploy <ref-classes-deploy>`
>class now cleans up ${:term:`DEPLOYDIR`} before running, just as
>``do_install`` cleans up ${:term:`D`} before running. This reduces the
>risk of DEPLOYDIR being accidentally contaminated by files from

``DEPLOYDIR``?

>previous runs, possibly even with different config, in case of
>incremental builds.
>+
>+Most recipes and classes that inherit the ``deploy`` class or interact
>with ``do_deploy`` are unlikely to be affected by this unless they add
>``prefuncs`` to ``do_deploy`` *which also* put files into
>``${DEPLOYDIR}`` - these should be refactored to use
>``do_deploy_prepend`` instead.
>+
>+
>+.. _migration-3.2-nativesdk-sdk-provides-dummy:
>+
>+Custom SDK / SDK-style recipes need to include
>``nativesdk-sdk-provides-dummy``
>+-------------------------------------------------------------------------------
>+
>+All ``nativesdk`` packages require ``/bin/sh`` due to their
>postinstall scriptlets, thus this package has to be dummy-provided
>within the SDK and ``nativesdk-sdk-provides-dummy`` now does this. If
>you have a custom SDK recipe (or your own SDK-style recipe similar to
>e.g. ``buildtools-tarball``), you will need to ensure
>``nativesdk-sdk-provides-dummy`` or an equivalent is included in
>:term:`TOOLCHAIN_HOST_TASK`.
>+
>+
>+``ld.so.conf`` now moved back to main ``glibc`` package
>+-------------------------------------------------------
>+
>+There are cases where one doesn't want ldconfig on target (e.g. for

``ldconfig``?

Quentin
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#518): https://lists.yoctoproject.org/g/docs/message/518
Mute This Topic: https://lists.yoctoproject.org/mt/77853548/3617530
Group Owner: docs+owner@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Richard Purdie Oct. 28, 2020, 2:06 p.m.
Hi All,

On Wed, 2020-10-28 at 14:12 +0100, Quentin Schulz wrote:
> Hi Paul,
> 
> On October 28, 2020 2:18:35 AM GMT+01:00, Paul Eggleton <
> paul.eggleton@linux.microsoft.com> wrote:
> > +In addition, pseudo's behaviour on mismatches has now been changed
> > -
> > rather
> > +than doing what turns out to be a rather dangerous "fixup" if it
> > sees
> > a file
> > +with a different path but the same inode as another file it has
> > previously seen,
> > +pseudo will throw an ``abort()`` and direct you to a `wiki page
> > <https://wiki.yoctoproject.org/wiki/Pseudo_Abort>`
> 
> :yocto_wiki:`wiki page </wiki/Pseudo_Abort>`

Thanks for the review! Since time is short and I suspect Paul will be
fine with the tweaks, I've squashed the tweaks into the patches in
master-next. I've not make the qa-check changes, those may be easiest
in a follow up patch if someone wants to send one?

Cheers,

Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#521): https://lists.yoctoproject.org/g/docs/message/521
Mute This Topic: https://lists.yoctoproject.org/mt/77853548/3617530
Group Owner: docs+owner@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-