Message ID | 20240404132902.68631-2-mikko.rapeli@linaro.org |
---|---|
State | New |
Headers | show |
Series | [1/2] linux-yocto-dev: remove duplicate DEPENDS | expand |
Hi Mikko, Did you test with the below change already merged on master? Looks like it was to fix the same issue. git: git-replacement-native: depend on ca-certificate https://git.yoctoproject.org/poky/commit/meta/recipes-devtools/git?id=421083c46c97bf758496b8c58402aea5d74aa097 Jose Mikko Rapeli <mikko.rapeli@linaro.org> escreveu (quinta, 4/04/2024 à(s) 14:29): > If linux-yocto-dev is compiled without specific SRCREV, it uses > AUTOREV which tries to update to latest available commit. This is > currently failing with these steps: > > $ rm -rf tmp*/work/*/linux-yocto-dev && \ > bitbake -c do_configure mc:machine:linux-yocto-dev ; \ > bitbake -c do_clean mc:machine:linux-yocto-dev > ... > Parsing recipes...WARNING: > /home/builder/src/base/build/../poky/meta/recipes-kernel/linux/ > linux-yocto-dev.bb: Exception during build_dependencies for > fetcher_hashes_dummyfunc > WARNING: /home/builder/src/base/build/../poky/meta/recipes-kernel/linux/ > linux-yocto-dev.bb: Error during finalise of > /home/builder/src/base/build/../poky/meta/recipes-kernel/linux/ > linux-yocto-dev.bb > WARNING: /home/builder/src/base/build/../poky/meta/recipes-kernel/linux/ > linux-yocto-dev.bb: Exception during build_dependencies for > fetcher_hashes_dummyfunc > WARNING: /home/builder/src/base/build/../poky/meta/recipes-kernel/linux/ > linux-yocto-dev.bb: Error during finalise of > mc:poky:/home/builder/src/base/build/../poky/meta/recipes-kernel/linux/ > linux-yocto-dev.bb > ERROR: ExpansionError during parsing > /home/builder/src/base/build/../poky/meta/recipes-kernel/linux/ > linux-yocto-dev.bb > Traceback (most recent call last): > File "/home/builder/src/base/poky/bitbake/lib/bb/fetch2/__init__.py", > line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at > 0x7f34e5f61330>, d=<bb.data_smart.DataSmart object at 0x7f34e5ff22f0>, > name='machine'): > d.setVar("__BBAUTOREV_ACTED_UPON", True) > > srcrev = ud.method.latest_revision(ud, d, name) > > File "/home/builder/src/base/poky/bitbake/lib/bb/fetch2/__init__.py", > line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at > 0x7f34e5f61330>, d=<bb.data_smart.DataSmart object at 0x7f34e5ff22f0>, > name='machine'): > except KeyError: > > revs[key] = rev = self._latest_revision(ud, d, name) > return rev > File "/home/builder/src/base/poky/bitbake/lib/bb/fetch2/git.py", line > 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at > 0x7f34e5f61330>, d=<bb.data_smart.DataSmart object at 0x7f34e5ff22f0>, > name='machine'): > > > output = self._lsremote(ud, d, "") > # Tags of the form ^{} may not work, need to fallback to > other form > File "/home/builder/src/base/poky/bitbake/lib/bb/fetch2/git.py", line > 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f34e5f61330>, > d=<bb.data_smart.DataSmart object at 0x7f34e5ff22f0>, search=''): > bb.fetch2.check_network_access(d, cmd, repourl) > > output = runfetchcmd(cmd, d, True) > if not output: > File "/home/builder/src/base/poky/bitbake/lib/bb/fetch2/__init__.py", > line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/15818/bus"; export > PATH="/home/builder/src/base/build/tmp_poky/sysroots-uninative/x86_64-linux/usr/bin:/home/builder/src/base/poky/scripts:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/usr/bin/aarch64-poky-linux:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot/usr/bin/crossscripts:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/usr/sbin:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/usr/bin:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/sbin:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/bin:/home/builder/src/base/poky/bitbake/bin:/home/builder/src/base/build/tmp_poky/hosttools"; > export HOME="/home/builder"; git -c gc.autoDetach=false -c core.pager=cat > -c safe.bareRepository=all ls-remote > https://git.yoctoproject.org/linux-yocto-dev.git ', > d=<bb.data_smart.DataSmart object at 0x7f34e5ff22f0>, quiet=True, > cleanup=[], log=None, workdir=None): > > > raise FetchError(error_message) > > bb.data_smart.ExpansionError: Failure expanding variable > fetcher_hashes_dummyfunc[vardepvalue], expression was > ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher > failure: Fetch command export PSEUDO_DISABLED=1; export > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/15818/bus"; export > PATH="/home/builder/src/base/build/tmp_poky/sysroots-uninative/x86_64-linux/usr/bin:/home/builder/src/base/poky/scripts:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/usr/bin/aarch64-poky-linux:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot/usr/bin/crossscripts:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/usr/sbin:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/usr/bin:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/sbin:/home/builder/src/base/build/tmp_poky/work/poky-poky-linux/linux-yocto-dev/6.8+git/recipe-sysroot-native/bin:/home/builder/src/base/poky/bitbake/bin:/home/builder/src/base/build/tmp_poky/hosttools"; > export HOME="/home/builder"; git -c gc.autoDetach=false -c core.pager=cat > -c safe.bareRepository=all ls-remote > https://git.yoctoproject.org/linux-yocto-dev.git failed with exit code > 128, output: > fatal: unable to access 'https://git.yoctoproject.org/linux-yocto-dev.git/': > error setting certificate file: > /home/builder/src/base/build/tmp_poky/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt > > The variable dependency chain for the failure is: > fetcher_hashes_dummyfunc[vardepvalue] > > ERROR: Parsing halted due to errors, see error messages above > > Summary: There were 6 WARNING messages. > Summary: There were 2 ERROR messages, returning a non-zero exit code. > > This state is not recoverable with bitbake calls. All of them fail from > now on. > "rm -rf tmp/work/*/linux-yocto-dev" recovers the situation > and bitbake commands work again. > > Root cause is curl-native, dependency of git-native, which > has --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt > which for native build target is wrong and points to curl-native build > directory path > > /home/builder/src/base/build/tmp_poky/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt > > Since git is a build time host package dependency listed in > https://docs.yoctoproject.org/dev/singleindex.html#build-host-packages > then its dependencies like curl and ca-certificates are too, it should > be safe for curl-native to use the default host ca-certificates path > instead of the one in recipe specific sysroots which would need to be set > with complicated > environment variables. Set non-default ca-certificates path only for > target and nativesdk builds. > > Reported-by: Mathieu Poirier <mathieu.poirier@linaro.org> > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> > --- > meta/recipes-support/curl/curl_8.6.0.bb | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/meta/recipes-support/curl/curl_8.6.0.bb > b/meta/recipes-support/curl/curl_8.6.0.bb > index 49ba0cb4a7..da5571ca14 100644 > --- a/meta/recipes-support/curl/curl_8.6.0.bb > +++ b/meta/recipes-support/curl/curl_8.6.0.bb > @@ -73,11 +73,16 @@ PACKAGECONFIG[zstd] = "--with-zstd,--without-zstd,zstd" > EXTRA_OECONF = " \ > --disable-libcurl-option \ > --disable-ntlm-wb \ > - --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > --without-libpsl \ > --enable-optimize \ > ${@'--without-ssl' if (bb.utils.filter('PACKAGECONFIG', 'gnutls > mbedtls openssl', d) == '') else ''} \ > " > +EXTRA_OECONF:class-target = " \ > + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > +" > +EXTRA_OECONF:class-nativesdk = " \ > + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > +" > > fix_absolute_paths () { > # cleanup buildpaths from curl-config > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#197964): > https://lists.openembedded.org/g/openembedded-core/message/197964 > Mute This Topic: https://lists.openembedded.org/mt/105328588/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Thu, 2024-04-04 at 16:29 +0300, Mikko Rapeli wrote: > If linux-yocto-dev is compiled without specific SRCREV, it uses > AUTOREV which tries to update to latest available commit. This is > currently failing with these steps: > > $ rm -rf tmp*/work/*/linux-yocto-dev && \ > bitbake -c do_configure mc:machine:linux-yocto-dev ; \ > bitbake -c do_clean mc:machine:linux-yocto-dev > [...] > The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue] > > ERROR: Parsing halted due to errors, see error messages above > > Summary: There were 6 WARNING messages. > Summary: There were 2 ERROR messages, returning a non-zero exit code. > > This state is not recoverable with bitbake calls. All of them fail from now on. > "rm -rf tmp/work/*/linux-yocto-dev" recovers the situation > and bitbake commands work again. > > Root cause is curl-native, dependency of git-native, which > has --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt > which for native build target is wrong and points to curl-native build > directory path > /home/builder/src/base/build/tmp_poky/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt > > Since git is a build time host package dependency listed in > https://docs.yoctoproject.org/dev/singleindex.html#build-host-packages > then its dependencies like curl and ca-certificates are too, it should > be safe for curl-native to use the default host ca-certificates path > instead of the one in recipe specific sysroots which would need to be set with complicated > environment variables. Set non-default ca-certificates path only for > target and nativesdk builds. > > Reported-by: Mathieu Poirier <mathieu.poirier@linaro.org> > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> > --- > meta/recipes-support/curl/curl_8.6.0.bb | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/meta/recipes-support/curl/curl_8.6.0.bb b/meta/recipes-support/curl/curl_8.6.0.bb > index 49ba0cb4a7..da5571ca14 100644 > --- a/meta/recipes-support/curl/curl_8.6.0.bb > +++ b/meta/recipes-support/curl/curl_8.6.0.bb > @@ -73,11 +73,16 @@ PACKAGECONFIG[zstd] = "--with-zstd,--without-zstd,zstd" > EXTRA_OECONF = " \ > --disable-libcurl-option \ > --disable-ntlm-wb \ > - --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > --without-libpsl \ > --enable-optimize \ > ${@'--without-ssl' if (bb.utils.filter('PACKAGECONFIG', 'gnutls mbedtls openssl', d) == '') else ''} \ > " > +EXTRA_OECONF:class-target = " \ > + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > +" > +EXTRA_OECONF:class-nativesdk = " \ > + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > +" > > fix_absolute_paths () { > # cleanup buildpaths from curl-config This change is fraught with danger :(. I have a feeling we've gone around in circles as in some cases you don't have the ca-certs on the host, or they're in unusual paths so the previous conclusion was we should always have them present in the sysroot if curl-native is being used. Yes, that does mean we have to set the environment correctly to relocate curl's paths appropriately. Certainly at this point in the release cycle I'm very nervous about changing this around. Cheers, Richard
Hi, On Thu, Apr 04, 2024 at 02:57:20PM +0100, Jose Quaresma wrote: > Hi Mikko, > > Did you test with the below change already merged on master? > Looks like it was to fix the same issue. > > git: git-replacement-native: depend on ca-certificate > https://git.yoctoproject.org/poky/commit/meta/recipes-devtools/git?id=421083c46c97bf758496b8c58402aea5d74aa097 No, I was on slighlty older commit 47c201da56155f80e18ee1269096ffec1864bce0. 421083c46c97bf758496b8c58402aea5d74aa097 fixes the issue too, thanks for the hint! This patch can be ignored. Cheers, -Mikko
Hi, On Thu, Apr 04, 2024 at 03:13:08PM +0100, Richard Purdie wrote: > On Thu, 2024-04-04 at 16:29 +0300, Mikko Rapeli wrote: > > If linux-yocto-dev is compiled without specific SRCREV, it uses > > AUTOREV which tries to update to latest available commit. This is > > currently failing with these steps: > > > > $ rm -rf tmp*/work/*/linux-yocto-dev && \ > > bitbake -c do_configure mc:machine:linux-yocto-dev ; \ > > bitbake -c do_clean mc:machine:linux-yocto-dev > > [...] > > > The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue] > > > > ERROR: Parsing halted due to errors, see error messages above > > > > Summary: There were 6 WARNING messages. > > Summary: There were 2 ERROR messages, returning a non-zero exit code. > > > > This state is not recoverable with bitbake calls. All of them fail from now on. > > "rm -rf tmp/work/*/linux-yocto-dev" recovers the situation > > and bitbake commands work again. > > > > Root cause is curl-native, dependency of git-native, which > > has --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt > > which for native build target is wrong and points to curl-native build > > directory path > > /home/builder/src/base/build/tmp_poky/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt > > > > Since git is a build time host package dependency listed in > > https://docs.yoctoproject.org/dev/singleindex.html#build-host-packages > > then its dependencies like curl and ca-certificates are too, it should > > be safe for curl-native to use the default host ca-certificates path > > instead of the one in recipe specific sysroots which would need to be set with complicated > > environment variables. Set non-default ca-certificates path only for > > target and nativesdk builds. > > > > Reported-by: Mathieu Poirier <mathieu.poirier@linaro.org> > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> > > --- > > �meta/recipes-support/curl/curl_8.6.0.bb | 7 ++++++- > > �1 file changed, 6 insertions(+), 1 deletion(-) > > > > diff --git a/meta/recipes-support/curl/curl_8.6.0.bb b/meta/recipes-support/curl/curl_8.6.0.bb > > index 49ba0cb4a7..da5571ca14 100644 > > --- a/meta/recipes-support/curl/curl_8.6.0.bb > > +++ b/meta/recipes-support/curl/curl_8.6.0.bb > > @@ -73,11 +73,16 @@ PACKAGECONFIG[zstd] = "--with-zstd,--without-zstd,zstd" > > �EXTRA_OECONF = " \ > > ���� --disable-libcurl-option \ > > ���� --disable-ntlm-wb \ > > -��� --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > > ���� --without-libpsl \ > > ���� --enable-optimize \ > > ���� ${@'--without-ssl' if (bb.utils.filter('PACKAGECONFIG', 'gnutls mbedtls openssl', d) == '') else ''} \ > > �" > > +EXTRA_OECONF:class-target = " \ > > +��� --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > > +" > > +EXTRA_OECONF:class-nativesdk = " \ > > +��� --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ > > +" > > � > > �fix_absolute_paths () { > > � # cleanup buildpaths from curl-config > > This change is fraught with danger :(. > > I have a feeling we've gone around in circles as in some cases you > don't have the ca-certs on the host, or they're in unusual paths so the > previous conclusion was we should always have them present in the > sysroot if curl-native is being used. Yes, that does mean we have to > set the environment correctly to relocate curl's paths appropriately. > > Certainly at this point in the release cycle I'm very nervous about > changing this around. No worries, 421083c46c97bf758496b8c58402aea5d74aa097 already on master fixes the issue. We're lagging few weeks behind master branch but trying to catch up. Cheers, -Mikko
diff --git a/meta/recipes-support/curl/curl_8.6.0.bb b/meta/recipes-support/curl/curl_8.6.0.bb index 49ba0cb4a7..da5571ca14 100644 --- a/meta/recipes-support/curl/curl_8.6.0.bb +++ b/meta/recipes-support/curl/curl_8.6.0.bb @@ -73,11 +73,16 @@ PACKAGECONFIG[zstd] = "--with-zstd,--without-zstd,zstd" EXTRA_OECONF = " \ --disable-libcurl-option \ --disable-ntlm-wb \ - --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ --without-libpsl \ --enable-optimize \ ${@'--without-ssl' if (bb.utils.filter('PACKAGECONFIG', 'gnutls mbedtls openssl', d) == '') else ''} \ " +EXTRA_OECONF:class-target = " \ + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ +" +EXTRA_OECONF:class-nativesdk = " \ + --with-ca-bundle=${sysconfdir}/ssl/certs/ca-certificates.crt \ +" fix_absolute_paths () { # cleanup buildpaths from curl-config