wic: honor the SOURCE_DATE_EPOCH in case of updated fstab

From: Sergei Zhmylev <s.zhmylev@yadro.com>

In case user requested to build a binary repeatable package,
it's required to honor the SOURCE_DATE_EPOCH environment
variable. So forcefully set mtime inside all the routines
which modify fstab in case it is updated.

Signed-off-by: Sergei Zhmylev <s.zhmylev@yadro.com>
---
 scripts/lib/wic/partition.py             | 5 ++++-
 scripts/lib/wic/plugins/imager/direct.py | 3 +++
 scripts/lib/wic/plugins/source/rootfs.py | 2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)

Hello,

This seems to have caused the following failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/6301/steps/13/logs/stdio

| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-bsp/grub/grub-efi_2.06.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-core/glibc/cross-localedef-native_2.36.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-core/os-release/os-release.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-core/systemd/systemd-boot_251.4.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/binutils/binutils-cross_2.39.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/cdrtools/cdrtools-native_3.01.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/python/python3_3.10.6.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/qemu/qemu-system-native_7.1.0.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/qemu/qemuwrapper-cross_1.0.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb:do_populate_sysroot', '/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-kernel/kmod/depmodwrapper-cross_1.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/btrfs-tools/btrfs-tools_5.19.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/createrepo-c/createrepo-c_0.20.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/dnf/dnf_4.14.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/dosfstools/dosfstools_4.2.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.46.5.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/erofs-utils/erofs-utils_1.5.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/fdisk/gptfdisk_1.0.9.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/makedevs/makedevs_1.0.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/mtools/mtools_4.0.41.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/opkg-utils/opkg-utils_0.5.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/opkg/opkg_0.6.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/python/python3-installer_0.5.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/python/python3-picobuild_0.2.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/python/python3-pip_22.2.2.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/python/python3-setuptools_65.0.2.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/python/python3-wheel_0.37.1.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/python/python3_3.10.6.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/rpm/rpm_4.18.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/squashfs-tools/squashfs-tools_git.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-extended/parted/parted_3.5.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-extended/pigz/pigz_2.7.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-extended/zip/zip_3.0.bb:do_populate_sysroot', 'virtual:native:/home/pokybuild/yocto-worker/build-appliance/build/meta/recipes-support/bmap-tools/bmap-tools_git.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: ['dosfstools-native', 'gptfdisk-native', 'parted-native']
| NOTE: Skipping as already exists in sysroot: ['grub-efi', 'cross-localedef-native', 'ldconfig-native', 'os-release', 'systemd-boot', 'binutils-cross-x86_64', 'cdrtools-native', 'python3', 'qemu-system-native', 'qemuwrapper-cross', 'syslinux', 'depmodwrapper-cross', 'update-rc.d-native', 'btrfs-tools-native', 'createrepo-c-native', 'dnf-native', 'e2fsprogs-native', 'erofs-utils-native', 'makedevs-native', 'mtools-native', 'opkg-utils-native', 'opkg-native', 'pseudo-native', 'python3-installer-native', 'python3-picobuild-native', 'python3-pip-native', 'python3-setuptools-native', 'python3-wheel-native', 'python3-native', 'rpm-native', 'squashfs-tools-native', 'syslinux-native', 'pigz-native', 'zip-native', 'bmap-tools-native', 'shadow-native', 'mkfontscale-native', 'python3-flit-core-native', 'texinfo-dummy-native', 'libtool-native', 'gtk+3-native', 'gdk-pixbuf-native', 'util-linux-libuuid-native', 'lz4-native', 'python3-six-native', 'coreutils-native', 'libarchive-native', 'libsolv-native', 'e2fsprogs', 'glibc', 'util-linux', 'gcc-runtime', 'gdbm-native', 'openssl-native', 'zlib-native', 'sqlite3-native', 'libtirpc-native', 'libnsl2-native', 'bzip2-native', 'readline-native', 'xz-native', 'ncurses-native', 'libffi-native', 'desktop-file-utils-native', 'shared-mime-info-native', 'debianutils-native', 'qemu-native', 'perl-native', 'gettext-minimal-native', 'file-native', 'libgcrypt-native', 'elfutils-native', 'zstd-native', 'popt-native', 'lua-native', 'util-linux-native', 'base-passwd', 'glib-2.0-native', 'virglrenderer-native', 'dtc-native', 'libepoxy-native', 'alsa-lib-native', 'libsdl2-native', 'pixman-native', 'curl-native', 'cmake-native', 'expat-native', 'libxml2-native', 'lzo-native', 'attr-native', 'gnu-efi', 'libcap', 'kmod-native', 'nasm-native', 'gconf-native', 'openssl', 'readline', 'sqlite3', 'ncurses', 'zlib', 'gdbm', 'util-linux-libuuid', 'bzip2', 'libtirpc', 'libffi', 'libnsl2', 'libxcrypt', 'opkg-utils', 'xz', 'gnu-config-native', 'flex-native', 'librepo-native', 'python3-iniparse-native', 'libdnf-native', 'libcomps-native', 'libfontenc-native', 'freetype-native', 'util-macros-native', 'xorgproto-native', 'unzip-native', 'libxdamage-native', 'libxi-native', 'atk-native', 'pango-native', 'libxext-native', 'at-spi2-atk-native', 'libjpeg-turbo-native', 'libxfixes-native', 'fontconfig-native', 'libxcursor-native', 'gobject-introspection-native', 'libxcomposite-native', 'libxrender-native', 'libpng-native', 'libx11-native', 'libxrandr-native', 'cairo-native', 'attr', 'autoconf-archive', 'linux-libc-headers', 'bash-completion', 'libcap-ng', 'libgcc', 'itstool-native', 'make-native', 'libcap-native', 'libgpg-error-native', 'libmicrohttpd-native', 'libpcre2-native', 'libcap-ng-native', 'gettext-native', 'libpcre-native', 'libdrm-native', 'mesa-native', 'groff-native', 'dbus-native', 'intltool-native', 'dbus-glib-native', 'm4-native', 'gpgme-native', 'libcheck-native', 'libmodulemd-native', 'swig-native', 'json-c-native', 'font-util-native', 'fribidi-native', 'harfbuzz-native', 'libxft-native', 'at-spi2-core-native', 'xtrans-native', 'libxcb-native', 'libsm-native', 'gnutls-native', 'libpciaccess-native', 'libpthread-stubs-native', 'xrandr-native', 'wayland-native', 'wayland-protocols-native', 'python3-mako-native', 'chrpath-native', 'libxshmfence-native', 'makedepend-native', 'libxxf86vm-native', 'libxml-parser-perl-native', 'libassuan-native', 'libyaml-native', 'icu-native', 'libxtst-native', 'libxau-native', 'libxdmcp-native', 'xcb-proto-native', 'libice-native', 'libunistring-native', 'nettle-native', 'libidn2-native', 'gmp-native', 'python3-pygments-native', 'python3-markupsafe-native']
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing python function set_image_size
| DEBUG: 1308850.400000 = 1006808 * 1.300000
| DEBUG: 43354290.400000 = max(1308850.400000, 65536)[1308850.400000] + 42045440
| DEBUG: 43354291.000000 = int(43354290.400000)
| DEBUG: 43354291 = aligned(43354291)
| DEBUG: returning 43354291
| DEBUG: Python function set_image_size finished
| DEBUG: Executing shell function do_image_wic
| INFO: Creating image(s)...
| 
| ERROR: _exec_cmd: export PATH=/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/hosttools:$PATH;export PSEUDO_PREFIX=/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr;export PSEUDO_LOCALSTATEDIR=/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/rootfs/../pseudo;export PSEUDO_PASSWD=/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/rootfs;export PSEUDO_NOSYMLINKEXP=1;export PSEUDO_IGNORE_PATHS=/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/tmp-wic/rootfs_platform.2.ext4,/usr/,/etc/,/usr/lib,/dev/,/run/,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/temp,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/recipe-sysroot,/srv/autobuilder/autobuilder.yocto.io/pub/sstate,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/stamps,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/sstate-control,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/buildstats,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/sysroots-components,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/pkgdata,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/deploy-,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/sstate-build-package_,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/sstate-install-package_,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/pkgdata-sysroot,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/deploy,/home/pokybuild/yocto-worker/build-appliance/build/build/cache,/home/pokybuild/yocto-worker/build-appliance/build/scripts,/home/pokybuild/yocto-worker/build-appliance/build/meta/files/common-licenses,/home/pokybuild/yocto-worker/build-appliance/build/meta/COPYING,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/i686-deploy-build-appliance-image-populate-sdk,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/oe-sdk-repo,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/sstate-build-populate_sdk,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/build-wic,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/intercept_scripts,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/oe-rootfs-repo,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/sstate-build-image_complete,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/build-appliance-image-15.0.0,/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/build-appliance-image-15.0.0;/home/pokybuild/yocto-worker/build-appliance/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/bin/pseudo fsck.ext4 -pvfD /home/pokybuild/yocto-worker/build-appliance/build/build/tmp/work/qemux86_64-poky-linux/build-appliance-image/15.0.0-r0/tmp-wic/rootfs_platform.2.ext4 returned '1' instead of 0
| output: platform: Superblock last write time (Sun Sep 18 23:05:48 2072,
| 	now = Thu Oct 13 08:33:43 2022) is in the future.
| FIXED.
| 
|        57969 inodes used (0.82%, out of 7045120)
|            9 non-contiguous files (0.0%)
|           33 non-contiguous directories (0.1%)
|              # of inodes with ind/dind/tind blocks: 0/0/0
|              Extent depth histogram: 56728/10
|       770336 blocks used (5.47%, out of 14090144)
|            0 bad blocks
|            1 large file
| 
|        47374 regular files
|         9238 directories
|            0 character device files
|            0 block device files
|            1 fifo
|         1203 links
|         1347 symbolic links (1222 fast symbolic links)
|            0 sockets
| ------------
|        59163 files
| 
| WARNING: exit code 1 from a shell command.
NOTE: recipe build-appliance-image-15.0.0-r0: task do_image_wic: Failed

On 12/10/2022 11:53:09+0300, Sergey Zhmylev wrote:
> From: Sergei Zhmylev <s.zhmylev@yadro.com>
> 
> In case user requested to build a binary repeatable package,
> it's required to honor the SOURCE_DATE_EPOCH environment
> variable. So forcefully set mtime inside all the routines
> which modify fstab in case it is updated.
> 
> Signed-off-by: Sergei Zhmylev <s.zhmylev@yadro.com>
> ---
>  scripts/lib/wic/partition.py             | 5 ++++-
>  scripts/lib/wic/plugins/imager/direct.py | 3 +++
>  scripts/lib/wic/plugins/source/rootfs.py | 2 +-
>  3 files changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index e50871b8d7..cac1bab4dd 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -291,6 +291,9 @@ class Partition():
>          if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update:
>              debugfs_script_path = os.path.join(cr_workdir, "debugfs_script")
>              with open(debugfs_script_path, "w") as f:
> +                if os.getenv('SOURCE_DATE_EPOCH'):
> +                    fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> +                    f.write("set_current_time %s\n" % (fstab_time))
>                  f.write("cd etc\n")
>                  f.write("rm fstab\n")
>                  f.write("write %s fstab\n" % (self.updated_fstab_path))
> @@ -353,7 +356,7 @@ class Partition():
>          exec_native_cmd(mcopy_cmd, native_sysroot)
>  
>          if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update:
> -            mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
> +            mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
>              exec_native_cmd(mcopy_cmd, native_sysroot)
>  
>          chmod_cmd = "chmod 644 %s" % rootfs
> diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
> index da483daed5..3e766972b4 100644
> --- a/scripts/lib/wic/plugins/imager/direct.py
> +++ b/scripts/lib/wic/plugins/imager/direct.py
> @@ -149,6 +149,9 @@ class DirectPlugin(ImagerPlugin):
>              self.updated_fstab_path = os.path.join(self.workdir, "fstab")
>              with open(self.updated_fstab_path, "w") as f:
>                  f.writelines(fstab_lines)
> +            if os.getenv('SOURCE_DATE_EPOCH'):
> +                fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> +                os.utime(self.updated_fstab_path, (fstab_time, fstab_time))
>  
>      def _full_path(self, path, name, extention):
>          """ Construct full file path to a file we generate. """
> diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> index fc06312ee4..e29f3a4c2f 100644
> --- a/scripts/lib/wic/plugins/source/rootfs.py
> +++ b/scripts/lib/wic/plugins/source/rootfs.py
> @@ -224,7 +224,7 @@ class RootfsPlugin(SourcePlugin):
>              if part.update_fstab_in_rootfs and part.has_fstab and not part.no_fstab_update:
>                  fstab_path = os.path.join(new_rootfs, "etc/fstab")
>                  # Assume that fstab should always be owned by root with fixed permissions
> -                install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
> +                install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path)
>                  if new_pseudo:
>                      pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
>                  else:
> -- 
> 2.37.2
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#171660): https://lists.openembedded.org/g/openembedded-core/message/171660
> Mute This Topic: https://lists.openembedded.org/mt/94276997/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>