Revert "protobuf: stage protoc binary to sysroot"

This reverts commit a0557fe5433620717eeb00d3b16801711337b1a4.

As said by Ross[Ø]:
"Putting the _target_ protoc into the sysroot for executation at _build_
time isn't useful because even if it has the right architecture, the
tune might be incompatible.  Recipes which want protoc should just depend
on protobuf-native."

This has been reverted recently by Samuli[1]:
"If protoc is enabled for the build, recipes using protobuf will
fail when protoc is not available in the recipe sysroot"

Be the revert is incorret as This is an issue coming from qtgrpc
other recipes that use protobuf or gRPC compiler, proplery looks for
the binary in the correct sysroot folder.

Qtgrpc recipe should fix this issue at the recipe level, for example this
is what I've done for "etcd-cpp-apiv3" recipe[2] that doesn't need this
patch to properly compile.

So keeping this hack doesn't seems to be a correct fix.

Note that qtgrpc recipe isn't available on meta-oe nor any other public
layers.

0: https://patchwork.yoctoproject.org/project/oe/patch/20230904161230.377450-1-ross.burton@arm.com/
1: https://patchwork.yoctoproject.org/project/oe/patch/20230927051101.3088498-1-samuli.piippo@qt.io/
2: https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3/commit/47f0d9e0326f3cc31c801a0ecf7312d1049ece3e

CC: Samuli Piippo <samuli.piippo@gmail.com>
CC: Ross Burton <ross.burton@arm.com>
Signed-off-by: Clément Péron <peron.clem@gmail.com>
---
 meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb | 3 ---
 1 file changed, 3 deletions(-)

I agree, protoc should _not_ be in sysroot, because target binary can't 
run on host system.

Vyacheslav

On 30.09.2023 19:30, Clément Péron wrote:
> This reverts commit a0557fe5433620717eeb00d3b16801711337b1a4.
>
> As said by Ross[Ø]:
> "Putting the _target_ protoc into the sysroot for executation at _build_
> time isn't useful because even if it has the right architecture, the
> tune might be incompatible.  Recipes which want protoc should just depend
> on protobuf-native."
>
> This has been reverted recently by Samuli[1]:
> "If protoc is enabled for the build, recipes using protobuf will
> fail when protoc is not available in the recipe sysroot"
>
> Be the revert is incorret as This is an issue coming from qtgrpc
> other recipes that use protobuf or gRPC compiler, proplery looks for
> the binary in the correct sysroot folder.
>
> Qtgrpc recipe should fix this issue at the recipe level, for example this
> is what I've done for "etcd-cpp-apiv3" recipe[2] that doesn't need this
> patch to properly compile.
>
> So keeping this hack doesn't seems to be a correct fix.
>
> Note that qtgrpc recipe isn't available on meta-oe nor any other public
> layers.
>
> 0: https://patchwork.yoctoproject.org/project/oe/patch/20230904161230.377450-1-ross.burton@arm.com/
> 1: https://patchwork.yoctoproject.org/project/oe/patch/20230927051101.3088498-1-samuli.piippo@qt.io/
> 2: https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3/commit/47f0d9e0326f3cc31c801a0ecf7312d1049ece3e
>
> CC: Samuli Piippo <samuli.piippo@gmail.com>
> CC: Ross Burton <ross.burton@arm.com>
> Signed-off-by: Clément Péron <peron.clem@gmail.com>
> ---
>   meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb | 3 ---
>   1 file changed, 3 deletions(-)
>
> diff --git a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
> index 06d73d648..1edc21cdf 100644
> --- a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
> +++ b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
> @@ -101,9 +101,6 @@ PACKAGE_BEFORE_PN = "${PN}-compiler ${PN}-lite"
>   FILES:${PN}-compiler = "${bindir} ${libdir}/libprotoc${SOLIBS}"
>   FILES:${PN}-lite = "${libdir}/libprotobuf-lite${SOLIBS}"
>   
> -# CMake requires binaries to exist in sysroot, even if they have wrong architecture.
> -SYSROOT_DIRS += "${bindir}"
> -
>   RDEPENDS:${PN}-compiler = "${PN}"
>   RDEPENDS:${PN}-dev += "${PN}-compiler"
>   RDEPENDS:${PN}-ptest = "bash ${@bb.utils.contains('PACKAGECONFIG', 'python', 'python3-protobuf', '', d)}"
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#105284): https://lists.openembedded.org/g/openembedded-devel/message/105284
> Mute This Topic: https://lists.openembedded.org/mt/101679410/4455192
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [uvv.mail@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Sat, Sep 30, 2023 at 5:48 PM Vyacheslav Yurkov <uvv.mail@gmail.com>
wrote:

> I agree, protoc should _not_ be in sysroot, because target binary can't
> run on host system.
>

Yeah this seems correct approach to me

>
> Vyacheslav
>
> On 30.09.2023 19:30, Clément Péron wrote:
> > This reverts commit a0557fe5433620717eeb00d3b16801711337b1a4.
> >
> > As said by Ross[Ø]:
> > "Putting the _target_ protoc into the sysroot for executation at _build_
> > time isn't useful because even if it has the right architecture, the
> > tune might be incompatible.  Recipes which want protoc should just depend
> > on protobuf-native."
> >
> > This has been reverted recently by Samuli[1]:
> > "If protoc is enabled for the build, recipes using protobuf will
> > fail when protoc is not available in the recipe sysroot"
> >
> > Be the revert is incorret as This is an issue coming from qtgrpc
> > other recipes that use protobuf or gRPC compiler, proplery looks for
> > the binary in the correct sysroot folder.
> >
> > Qtgrpc recipe should fix this issue at the recipe level, for example this
> > is what I've done for "etcd-cpp-apiv3" recipe[2] that doesn't need this
> > patch to properly compile.
> >
> > So keeping this hack doesn't seems to be a correct fix.
> >
> > Note that qtgrpc recipe isn't available on meta-oe nor any other public
> > layers.
> >
> > 0:
> https://patchwork.yoctoproject.org/project/oe/patch/20230904161230.377450-1-ross.burton@arm.com/
> > 1:
> https://patchwork.yoctoproject.org/project/oe/patch/20230927051101.3088498-1-samuli.piippo@qt.io/
> > 2:
> https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3/commit/47f0d9e0326f3cc31c801a0ecf7312d1049ece3e
> >
> > CC: Samuli Piippo <samuli.piippo@gmail.com>
> > CC: Ross Burton <ross.burton@arm.com>
> > Signed-off-by: Clément Péron <peron.clem@gmail.com>
> > ---
> >   meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb | 3 ---
> >   1 file changed, 3 deletions(-)
> >
> > diff --git a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
> b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
> > index 06d73d648..1edc21cdf 100644
> > --- a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
> > +++ b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
> > @@ -101,9 +101,6 @@ PACKAGE_BEFORE_PN = "${PN}-compiler ${PN}-lite"
> >   FILES:${PN}-compiler = "${bindir} ${libdir}/libprotoc${SOLIBS}"
> >   FILES:${PN}-lite = "${libdir}/libprotobuf-lite${SOLIBS}"
> >
> > -# CMake requires binaries to exist in sysroot, even if they have wrong
> architecture.
> > -SYSROOT_DIRS += "${bindir}"
> > -
> >   RDEPENDS:${PN}-compiler = "${PN}"
> >   RDEPENDS:${PN}-dev += "${PN}-compiler"
> >   RDEPENDS:${PN}-ptest = "bash ${@bb.utils.contains('PACKAGECONFIG',
> 'python', 'python3-protobuf', '', d)}"
> >
> >
> >
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#105290):
> https://lists.openembedded.org/g/openembedded-devel/message/105290
> Mute This Topic: https://lists.openembedded.org/mt/101679410/1997914
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
> raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

well, target bindir isn't in default PATH so it wont get executed by default

I have used this work around in some meta-ros recipes where the .cmake file
was checking the binary existence even when this .cmake was used only to
get right list of libraries to link with (and the binary was not called at
all - which imho can happen with protobuf as well)

On Sun, 1 Oct 2023, 05:03 Khem Raj, <raj.khem@gmail.com> wrote:

>
>
> On Sat, Sep 30, 2023 at 5:48 PM Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
>
>> I agree, protoc should _not_ be in sysroot, because target binary can't
>> run on host system.
>>
>
> Yeah this seems correct approach to me
>
>>
>> Vyacheslav
>>
>> On 30.09.2023 19:30, Clément Péron wrote:
>> > This reverts commit a0557fe5433620717eeb00d3b16801711337b1a4.
>> >
>> > As said by Ross[Ø]:
>> > "Putting the _target_ protoc into the sysroot for executation at _build_
>> > time isn't useful because even if it has the right architecture, the
>> > tune might be incompatible.  Recipes which want protoc should just
>> depend
>> > on protobuf-native."
>> >
>> > This has been reverted recently by Samuli[1]:
>> > "If protoc is enabled for the build, recipes using protobuf will
>> > fail when protoc is not available in the recipe sysroot"
>> >
>> > Be the revert is incorret as This is an issue coming from qtgrpc
>> > other recipes that use protobuf or gRPC compiler, proplery looks for
>> > the binary in the correct sysroot folder.
>> >
>> > Qtgrpc recipe should fix this issue at the recipe level, for example
>> this
>> > is what I've done for "etcd-cpp-apiv3" recipe[2] that doesn't need this
>> > patch to properly compile.
>> >
>> > So keeping this hack doesn't seems to be a correct fix.
>> >
>> > Note that qtgrpc recipe isn't available on meta-oe nor any other public
>> > layers.
>> >
>> > 0:
>> https://patchwork.yoctoproject.org/project/oe/patch/20230904161230.377450-1-ross.burton@arm.com/
>> > 1:
>> https://patchwork.yoctoproject.org/project/oe/patch/20230927051101.3088498-1-samuli.piippo@qt.io/
>> > 2:
>> https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3/commit/47f0d9e0326f3cc31c801a0ecf7312d1049ece3e
>> >
>> > CC: Samuli Piippo <samuli.piippo@gmail.com>
>> > CC: Ross Burton <ross.burton@arm.com>
>> > Signed-off-by: Clément Péron <peron.clem@gmail.com>
>> > ---
>> >   meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb | 3 ---
>> >   1 file changed, 3 deletions(-)
>> >
>> > diff --git a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
>> b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
>> > index 06d73d648..1edc21cdf 100644
>> > --- a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
>> > +++ b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
>> > @@ -101,9 +101,6 @@ PACKAGE_BEFORE_PN = "${PN}-compiler ${PN}-lite"
>> >   FILES:${PN}-compiler = "${bindir} ${libdir}/libprotoc${SOLIBS}"
>> >   FILES:${PN}-lite = "${libdir}/libprotobuf-lite${SOLIBS}"
>> >
>> > -# CMake requires binaries to exist in sysroot, even if they have wrong
>> architecture.
>> > -SYSROOT_DIRS += "${bindir}"
>> > -
>> >   RDEPENDS:${PN}-compiler = "${PN}"
>> >   RDEPENDS:${PN}-dev += "${PN}-compiler"
>> >   RDEPENDS:${PN}-ptest = "bash ${@bb.utils.contains('PACKAGECONFIG',
>> 'python', 'python3-protobuf', '', d)}"
>> >
>> >
>> >
>>
>>
>>
>>
>>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#105291):
> https://lists.openembedded.org/g/openembedded-devel/message/105291
> Mute This Topic: https://lists.openembedded.org/mt/101679410/3617156
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
> martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

Hi Martin,

On Sun, 1 Oct 2023 at 12:11, Martin Jansa <martin.jansa@gmail.com> wrote:
>
> well, target bindir isn't in default PATH so it wont get executed by
default

Is it a reason to allow mixing executable arch/tune?

How do you handle it when you want to install protoc/grpc-cpp-plugin for
the target too?

>
> I have used this work around in some meta-ros recipes where the .cmake
file was checking the binary existence even when this .cmake was used only
to get right list of libraries to link with (and the binary was not called
at all - which imho can happen with protobuf as well)

As you say it's a workaround, and it's an issue with the CMake recipe.

If we can properly fix it at the recipe level, then why should we keep this
workaround/hack?

Also if you really need it, you can have a .bbappend for protoc/grpc in the
downstream layer. But I would be in favor to avoid this by default no?

Clement

>
> On Sun, 1 Oct 2023, 05:03 Khem Raj, <raj.khem@gmail.com> wrote:
>>
>>
>>
>> On Sat, Sep 30, 2023 at 5:48 PM Vyacheslav Yurkov <uvv.mail@gmail.com>
wrote:
>>>
>>> I agree, protoc should _not_ be in sysroot, because target binary can't
>>> run on host system.
>>
>>
>> Yeah this seems correct approach to me
>>>
>>>
>>> Vyacheslav
>>>
>>> On 30.09.2023 19:30, Clément Péron wrote:
>>> > This reverts commit a0557fe5433620717eeb00d3b16801711337b1a4.
>>> >
>>> > As said by Ross[Ø]:
>>> > "Putting the _target_ protoc into the sysroot for executation at
_build_
>>> > time isn't useful because even if it has the right architecture, the
>>> > tune might be incompatible.  Recipes which want protoc should just
depend
>>> > on protobuf-native."
>>> >
>>> > This has been reverted recently by Samuli[1]:
>>> > "If protoc is enabled for the build, recipes using protobuf will
>>> > fail when protoc is not available in the recipe sysroot"
>>> >
>>> > Be the revert is incorret as This is an issue coming from qtgrpc
>>> > other recipes that use protobuf or gRPC compiler, proplery looks for
>>> > the binary in the correct sysroot folder.
>>> >
>>> > Qtgrpc recipe should fix this issue at the recipe level, for example
this
>>> > is what I've done for "etcd-cpp-apiv3" recipe[2] that doesn't need
this
>>> > patch to properly compile.
>>> >
>>> > So keeping this hack doesn't seems to be a correct fix.
>>> >
>>> > Note that qtgrpc recipe isn't available on meta-oe nor any other
public
>>> > layers.
>>> >
>>> > 0:
https://patchwork.yoctoproject.org/project/oe/patch/20230904161230.377450-1-ross.burton@arm.com/
>>> > 1:
https://patchwork.yoctoproject.org/project/oe/patch/20230927051101.3088498-1-samuli.piippo@qt.io/
>>> > 2:
https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3/commit/47f0d9e0326f3cc31c801a0ecf7312d1049ece3e
>>> >
>>> > CC: Samuli Piippo <samuli.piippo@gmail.com>
>>> > CC: Ross Burton <ross.burton@arm.com>
>>> > Signed-off-by: Clément Péron <peron.clem@gmail.com>
>>> > ---
>>> >   meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb | 3 ---
>>> >   1 file changed, 3 deletions(-)
>>> >
>>> > diff --git a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
>>> > index 06d73d648..1edc21cdf 100644
>>> > --- a/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
>>> > +++ b/meta-oe/recipes-devtools/protobuf/protobuf_4.23.4.bb
>>> > @@ -101,9 +101,6 @@ PACKAGE_BEFORE_PN = "${PN}-compiler ${PN}-lite"
>>> >   FILES:${PN}-compiler = "${bindir} ${libdir}/libprotoc${SOLIBS}"
>>> >   FILES:${PN}-lite = "${libdir}/libprotobuf-lite${SOLIBS}"
>>> >
>>> > -# CMake requires binaries to exist in sysroot, even if they have
wrong architecture.
>>> > -SYSROOT_DIRS += "${bindir}"
>>> > -
>>> >   RDEPENDS:${PN}-compiler = "${PN}"
>>> >   RDEPENDS:${PN}-dev += "${PN}-compiler"
>>> >   RDEPENDS:${PN}-ptest = "bash ${@bb.utils.contains('PACKAGECONFIG',
'python', 'python3-protobuf', '', d)}"
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#105291):
https://lists.openembedded.org/g/openembedded-devel/message/105291
>> Mute This Topic: https://lists.openembedded.org/mt/101679410/3617156
>> Group Owner: openembedded-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
martin.jansa@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

On Sun, Oct 1, 2023 at 12:31 PM Clément Péron <peron.clem@gmail.com> wrote:

> Hi Martin,
>
> On Sun, 1 Oct 2023 at 12:11, Martin Jansa <martin.jansa@gmail.com> wrote:
> >
> > well, target bindir isn't in default PATH so it wont get executed by
> default
>
> Is it a reason to allow mixing executable arch/tune?
>

What is getting mixed? target sysroot is still for target and native
sysroot for native.

How do you handle it when you want to install protoc/grpc-cpp-plugin for
> the target too?
>

SYSROOT_DIRS variable doesn't affect how it's packaged for target, does it?
It just allows the binary from bindir to be available in RSS next to
binaries from libdir, both still separate from native RSS as it should.

> I have used this work around in some meta-ros recipes where the .cmake
> file was checking the binary existence even when this .cmake was used only
> to get right list of libraries to link with (and the binary was not called
> at all - which imho can happen with protobuf as well)
>
> As you say it's a workaround, and it's an issue with the CMake recipe.
>

Issue with .cmake file from protobuf, right?


> If we can properly fix it at the recipe level, then why should we keep
> this workaround/hack?
>
> Also if you really need it, you can have a .bbappend for protoc/grpc in
> the downstream layer. But I would be in favor to avoid this by default no?
>

I haven't seen any proper fix for this exact issue. Yes you can patch
.cmake generation not to include bindir when building for target (while it
needs it for native build), but then that change will break possible use of
protobuf on target, so the SYSROOT_DIRS still seems like lesser evil here.

Here are some examples from meta-ros:
https://github.com/ros/meta-ros/commit/875ceed4b1c693c2eb0f2b7ea85385be576a62cc
https://github.com/ros/meta-ros/commit/67c03755543655ad0634e078bc30d8ab9969a16c
https://github.com/ros/meta-ros/commit/5e61eb0520d2841c0da7e14eecc833b3c4104533
https://github.com/ros/meta-ros/commit/a53bb7cc8b45a72b704e75311d68b37a6db50cdf

I don't care about protobuf, but the explanation from Samuli looks like the
same issue as what I remember from above and I don't see how SYSROOT_DIRS
causes mixing executables or any issues for installing them for target.

Cheers,

>

On 01.10.2023 20:22, Martin Jansa wrote:
>
> Issue with .cmake file from protobuf, right?

No, it's the issue with the cmake of the project that uses protobuf. But 
that depends on a user case, perhaps I didn't get it properly.

>     If we can properly fix it at the recipe level, then why should we
>     keep this workaround/hack?
>
>     Also if you really need it, you can have a .bbappend for
>     protoc/grpc in the downstream layer. But I would be in favor to
>     avoid this by default no?
>
>
> I haven't seen any proper fix for this exact issue. Yes you can patch 
> .cmake generation not to include bindir when building for target 
> (while it needs it for native build), but then that change will break 
> possible use of protobuf on target, so the SYSROOT_DIRS still seems 
> like lesser evil here.
>
> Here are some examples from meta-ros:
> https://github.com/ros/meta-ros/commit/875ceed4b1c693c2eb0f2b7ea85385be576a62cc
> https://github.com/ros/meta-ros/commit/67c03755543655ad0634e078bc30d8ab9969a16c
> https://github.com/ros/meta-ros/commit/5e61eb0520d2841c0da7e14eecc833b3c4104533
> https://github.com/ros/meta-ros/commit/a53bb7cc8b45a72b704e75311d68b37a6db50cdf
>
> I don't care about protobuf, but the explanation from Samuli looks 
> like the same issue as what I remember from above and I don't see how 
> SYSROOT_DIRS causes mixing executables or any issues for installing 
> them for target.

I didn't look at the original issue of Samuli, but your issue is also 
fixed by a workaround. IMHO, the proper fix would be not to make CMake 
"happy" by providing a binary it can't use, but a proper export of 
cross-compiled tools for 
https://github.com/geographiclib/geographiclib/blob/main/CMakeLists.txt 
library. I'd say the workflow should be similar to protobuf/grpc 
recipes, i.e. a native recipe builds the libraries only, and the target 
recipe builds tools and libraries.

Regards,
Slava

I'd like to follow-up on this and say that I see this issue now with 
nativesdk build, in particular nativesdk-grpc recipe fails in master 
with the same error. I hope that's partially related to the issue 
everybody is confused about, but I'd like understand how to properly fix it.

|   The imported target "protobuf::protoc" references the file
|
| 
"/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
|
|   but this file does not exist.  Possible reasons include:
|
|   * The file was deleted, renamed, or moved to another location.

The protoc-23.4.0 file already exists in the recipe-sysroot-native.

And we do want protoc to be present in the SDK, but nativesdk build is a 
cross-compilation, because SDKMACHINE and HOST might be different.
In this particular grpc case there's an option to say where the actual 
protoc executable is ( 
-D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc 
is passed to grpc).

So there shouldn't be any issues here, but nativesdk build still fails.

Looking into it further I see that protoc is actually packaged by 
protobuf-compiler package, but when I try to add a package dependency to 
grpc recipe I get this:

nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler

Moreover, if I go into devshell and forcefully install 
nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy 
and compiles!

So something wrong with the dependencies in protobuf recipe, but I can't 
figure out yet what exactly. Any ideas?

Regards,
Slava

On 02.10.2023 18:25, Vyacheslav Yurkov via lists.openembedded.org wrote:
> On 01.10.2023 20:22, Martin Jansa wrote:
>>
>> Issue with .cmake file from protobuf, right?
>
> No, it's the issue with the cmake of the project that uses protobuf. 
> But that depends on a user case, perhaps I didn't get it properly.
>
>>     If we can properly fix it at the recipe level, then why should we
>>     keep this workaround/hack?
>>
>>     Also if you really need it, you can have a .bbappend for
>>     protoc/grpc in the downstream layer. But I would be in favor to
>>     avoid this by default no?
>>
>>
>> I haven't seen any proper fix for this exact issue. Yes you can patch 
>> .cmake generation not to include bindir when building for target 
>> (while it needs it for native build), but then that change will break 
>> possible use of protobuf on target, so the SYSROOT_DIRS still seems 
>> like lesser evil here.
>>
>> Here are some examples from meta-ros:
>> https://github.com/ros/meta-ros/commit/875ceed4b1c693c2eb0f2b7ea85385be576a62cc
>> https://github.com/ros/meta-ros/commit/67c03755543655ad0634e078bc30d8ab9969a16c
>> https://github.com/ros/meta-ros/commit/5e61eb0520d2841c0da7e14eecc833b3c4104533
>> https://github.com/ros/meta-ros/commit/a53bb7cc8b45a72b704e75311d68b37a6db50cdf
>>
>> I don't care about protobuf, but the explanation from Samuli looks 
>> like the same issue as what I remember from above and I don't see how 
>> SYSROOT_DIRS causes mixing executables or any issues for installing 
>> them for target.
>
> I didn't look at the original issue of Samuli, but your issue is also 
> fixed by a workaround. IMHO, the proper fix would be not to make CMake 
> "happy" by providing a binary it can't use, but a proper export of 
> cross-compiled tools for 
> https://github.com/geographiclib/geographiclib/blob/main/CMakeLists.txt 
> library. I'd say the workflow should be similar to protobuf/grpc 
> recipes, i.e. a native recipe builds the libraries only, and the 
> target recipe builds tools and libraries.
>
> Regards,
> Slava

Following simple test recipe will fail now when trying to use Protobuf with
CMake.

inherit cmake
DEPENDS += "protobuf"

do_configure:prepend() {
    echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
}

BBCLASSEXTEND = "nativesdk"

---
CMake Error at
tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
(message):
  The imported target "protobuf::protoc" references the file


 "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"

  but this file does not exist.
---

I don't quite see why every project/recipe should fix this independently
when the simple workaround for this yocto-specific issue is available
(namely the use of SYSROOT_DIRS).

-samuli

ps. qtgrpc recipe and other Qt6 recipes are now visible in the layerindex
https://layers.openembedded.org/layerindex/recipe/348210/


On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:

> I'd like to follow-up on this and say that I see this issue now with
> nativesdk build, in particular nativesdk-grpc recipe fails in master with
> the same error. I hope that's partially related to the issue everybody is
> confused about, but I'd like understand how to properly fix it.
>
> |   The imported target "protobuf::protoc" references the file
> |
> |
> "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
> |
> |   but this file does not exist.  Possible reasons include:
> |
> |   * The file was deleted, renamed, or moved to another location.
>
> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>
> And we do want protoc to be present in the SDK, but nativesdk build is a
> cross-compilation, because SDKMACHINE and HOST might be different.
> In this particular grpc case there's an option to say where the actual
> protoc executable is (
> -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
> is passed to grpc).
>
> So there shouldn't be any issues here, but nativesdk build still fails.
>
> Looking into it further I see that protoc is actually packaged by
> protobuf-compiler package, but when I try to add a package dependency to
> grpc recipe I get this:
>
> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>
> Moreover, if I go into devshell and forcefully install
> nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and
> compiles!
>
> So something wrong with the dependencies in protobuf recipe, but I can't
> figure out yet what exactly. Any ideas?
>
> Regards,
> Slava
>
> On 02.10.2023 18:25, Vyacheslav Yurkov via lists.openembedded.org wrote:
>
> On 01.10.2023 20:22, Martin Jansa wrote:
>
>
> Issue with .cmake file from protobuf, right?
>
>
> No, it's the issue with the cmake of the project that uses protobuf. But
> that depends on a user case, perhaps I didn't get it properly.
>
>
>
>> If we can properly fix it at the recipe level, then why should we keep
>> this workaround/hack?
>>
>> Also if you really need it, you can have a .bbappend for protoc/grpc in
>> the downstream layer. But I would be in favor to avoid this by default no?
>>
>
> I haven't seen any proper fix for this exact issue. Yes you can patch
> .cmake generation not to include bindir when building for target (while it
> needs it for native build), but then that change will break possible use of
> protobuf on target, so the SYSROOT_DIRS still seems like lesser evil here.
>
> Here are some examples from meta-ros:
>
> https://github.com/ros/meta-ros/commit/875ceed4b1c693c2eb0f2b7ea85385be576a62cc
>
> https://github.com/ros/meta-ros/commit/67c03755543655ad0634e078bc30d8ab9969a16c
>
> https://github.com/ros/meta-ros/commit/5e61eb0520d2841c0da7e14eecc833b3c4104533
>
> https://github.com/ros/meta-ros/commit/a53bb7cc8b45a72b704e75311d68b37a6db50cdf
>
> I don't care about protobuf, but the explanation from Samuli looks like
> the same issue as what I remember from above and I don't see how
> SYSROOT_DIRS causes mixing executables or any issues for installing them
> for target.
>
>
> I didn't look at the original issue of Samuli, but your issue is also
> fixed by a workaround. IMHO, the proper fix would be not to make CMake
> "happy" by providing a binary it can't use, but a proper export of
> cross-compiled tools for
> https://github.com/geographiclib/geographiclib/blob/main/CMakeLists.txt
> library. I'd say the workflow should be similar to protobuf/grpc recipes,
> i.e. a native recipe builds the libraries only, and the target recipe
> builds tools and libraries.
>
> Regards,
> Slava
>
>
>

On 18 Oct 2023, at 13:28, Samuli Piippo <samuli.piippo@gmail.com> wrote:
> I don't quite see why every project/recipe should fix this independently when the simple workaround for this yocto-specific issue is available (namely the use of SYSROOT_DIRS).

I’d say the one fix should be done in the protobuf cmake fragments.  It’s trying to find a binary _it can’t run_ which is wrong.

Ross

On Wed, Oct 18, 2023 at 5:27 PM Ross Burton <ross.burton@arm.com> wrote:

> On 18 Oct 2023, at 13:28, Samuli Piippo <samuli.piippo@gmail.com> wrote:
> > I don't quite see why every project/recipe should fix this independently
> when the simple workaround for this yocto-specific issue is available
> (namely the use of SYSROOT_DIRS).
>
> I’d say the one fix should be done in the protobuf cmake fragments.  It’s
> trying to find a binary _it can’t run_ which is wrong.
>

It's not that bad as long as it doesn't really plan to run it (only check
that it exists and that its fragment doesn't reference missing files).

The same fragment will be used from PN-dev if someone tries to use protobuf
on target and in that case the binary should exist and run. Good luck
trying to provide different cmake fragments to be used from target protobuf
in RSS and from target protobuf-dev on target.

Cheers,

I think the problem goes down to how dependencies are populated in the 
sysroot. I raised this question in oe-core mailing list 
https://lists.openembedded.org/g/openembedded-core/message/189324 , but 
didn't get an answer yet.

DEPENDS += "protobuf" will only install the main package of protobuf to 
the sysroot, because it assumes that nativesdk-protobuf-compiler is also 
part of the main package (which is not). I guess the part of the system 
populating sysroot deals with recipes only, and not with packages. I'm 
still trying to understand how that should be addressed properly.
My understanding is that either all packages should be installed to 
sysroot (which recipe claims it provides), or we need a way to indicate 
a package level dependency (i.e. nativesdk-protobuf-compiler in this case)

Any ideas?

Slava

On 18.10.2023 14:28, Samuli Piippo wrote:
> Following simple test recipe will fail now when trying to use Protobuf 
> with CMake.
>
> inherit cmake
> DEPENDS += "protobuf"
>
> do_configure:prepend() {
>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
> }
>
> BBCLASSEXTEND = "nativesdk"
>
> ---
> CMake Error at 
> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107 
> (message):
>   The imported target "protobuf::protoc" references the file
>
>  "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>
>   but this file does not exist.
> ---
>
> I don't quite see why every project/recipe should fix this 
> independently when the simple workaround for this yocto-specific issue 
> is available (namely the use of SYSROOT_DIRS).
>
> -samuli
>
> ps. qtgrpc recipe and other Qt6 recipes are now visible in the 
> layerindex https://layers.openembedded.org/layerindex/recipe/348210/
>
>
> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com> 
> wrote:
>
>     I'd like to follow-up on this and say that I see this issue now
>     with nativesdk build, in particular nativesdk-grpc recipe fails in
>     master with the same error. I hope that's partially related to the
>     issue everybody is confused about, but I'd like understand how to
>     properly fix it.
>
>     |   The imported target "protobuf::protoc" references the file
>     |
>     |
>     "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>     |
>     |   but this file does not exist.  Possible reasons include:
>     |
>     |   * The file was deleted, renamed, or moved to another location.
>
>     The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>
>     And we do want protoc to be present in the SDK, but nativesdk
>     build is a cross-compilation, because SDKMACHINE and HOST might be
>     different.
>     In this particular grpc case there's an option to say where the
>     actual protoc executable is (
>     -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
>     is passed to grpc).
>
>     So there shouldn't be any issues here, but nativesdk build still
>     fails.
>
>     Looking into it further I see that protoc is actually packaged by
>     protobuf-compiler package, but when I try to add a package
>     dependency to grpc recipe I get this:
>
>     nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>
>     Moreover, if I go into devshell and forcefully install
>     nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is
>     happy and compiles!
>
>     So something wrong with the dependencies in protobuf recipe, but I
>     can't figure out yet what exactly. Any ideas?
>
>     Regards,
>     Slava
>

No, it installs whatever recipe stages in do_populate_sysroot which is
controlled by SYSROOT_DIRS, so the solution still makes sense to me as
explained before and IMHO is better than the alternatives (changing the
protobuf .cmake files to be different in sysroot and in PN-dev).

On Fri, Oct 20, 2023 at 8:17 AM Vyacheslav Yurkov <uvv.mail@gmail.com>
wrote:

> I think the problem goes down to how dependencies are populated in the
> sysroot. I raised this question in oe-core mailing list
> https://lists.openembedded.org/g/openembedded-core/message/189324 , but
> didn't get an answer yet.
>
> DEPENDS += "protobuf" will only install the main package of protobuf to
> the sysroot, because it assumes that nativesdk-protobuf-compiler is also
> part of the main package (which is not). I guess the part of the system
> populating sysroot deals with recipes only, and not with packages. I'm
> still trying to understand how that should be addressed properly.
> My understanding is that either all packages should be installed to
> sysroot (which recipe claims it provides), or we need a way to indicate a
> package level dependency (i.e. nativesdk-protobuf-compiler in this case)
>
> Any ideas?
>
> Slava
>
> On 18.10.2023 14:28, Samuli Piippo wrote:
>
> Following simple test recipe will fail now when trying to use Protobuf
> with CMake.
>
> inherit cmake
> DEPENDS += "protobuf"
>
> do_configure:prepend() {
>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
> }
>
> BBCLASSEXTEND = "nativesdk"
>
> ---
> CMake Error at
> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
> (message):
>   The imported target "protobuf::protoc" references the file
>
>
>  "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>
>   but this file does not exist.
> ---
>
> I don't quite see why every project/recipe should fix this independently
> when the simple workaround for this yocto-specific issue is available
> (namely the use of SYSROOT_DIRS).
>
> -samuli
>
> ps. qtgrpc recipe and other Qt6 recipes are now visible in the layerindex
> https://layers.openembedded.org/layerindex/recipe/348210/
>
>
> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
>
>> I'd like to follow-up on this and say that I see this issue now with
>> nativesdk build, in particular nativesdk-grpc recipe fails in master with
>> the same error. I hope that's partially related to the issue everybody is
>> confused about, but I'd like understand how to properly fix it.
>>
>> |   The imported target "protobuf::protoc" references the file
>> |
>> |
>> "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>> |
>> |   but this file does not exist.  Possible reasons include:
>> |
>> |   * The file was deleted, renamed, or moved to another location.
>>
>> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>>
>> And we do want protoc to be present in the SDK, but nativesdk build is a
>> cross-compilation, because SDKMACHINE and HOST might be different.
>> In this particular grpc case there's an option to say where the actual
>> protoc executable is (
>> -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
>> is passed to grpc).
>>
>> So there shouldn't be any issues here, but nativesdk build still fails.
>>
>> Looking into it further I see that protoc is actually packaged by
>> protobuf-compiler package, but when I try to add a package dependency to
>> grpc recipe I get this:
>>
>> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>>
>> Moreover, if I go into devshell and forcefully install
>> nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and
>> compiles!
>>
>> So something wrong with the dependencies in protobuf recipe, but I can't
>> figure out yet what exactly. Any ideas?
>>
>> Regards,
>> Slava
>>
>>

Dependency to the recipe name will always install all packages to RSS.
You can verify this by checking that the libprotoc.so from the
nativesdk-protobuf-compile package is in the sysroot.
The problem you are facing is the fact that binaries are not installed into
the sysroot (for nativesdk builds), so you are only missing the
bin/protoc-23.4.0 file.
The proposed and twice reverted fix for this is the aforementioned
SYSROOT_DIRS.

-samuli

On Fri, 20 Oct 2023 at 09:17, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:

> I think the problem goes down to how dependencies are populated in the
> sysroot. I raised this question in oe-core mailing list
> https://lists.openembedded.org/g/openembedded-core/message/189324 , but
> didn't get an answer yet.
>
> DEPENDS += "protobuf" will only install the main package of protobuf to
> the sysroot, because it assumes that nativesdk-protobuf-compiler is also
> part of the main package (which is not). I guess the part of the system
> populating sysroot deals with recipes only, and not with packages. I'm
> still trying to understand how that should be addressed properly.
> My understanding is that either all packages should be installed to
> sysroot (which recipe claims it provides), or we need a way to indicate a
> package level dependency (i.e. nativesdk-protobuf-compiler in this case)
>
> Any ideas?
>
> Slava
>
> On 18.10.2023 14:28, Samuli Piippo wrote:
>
> Following simple test recipe will fail now when trying to use Protobuf
> with CMake.
>
> inherit cmake
> DEPENDS += "protobuf"
>
> do_configure:prepend() {
>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
> }
>
> BBCLASSEXTEND = "nativesdk"
>
> ---
> CMake Error at
> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
> (message):
>   The imported target "protobuf::protoc" references the file
>
>
>  "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>
>   but this file does not exist.
> ---
>
> I don't quite see why every project/recipe should fix this independently
> when the simple workaround for this yocto-specific issue is available
> (namely the use of SYSROOT_DIRS).
>
> -samuli
>
> ps. qtgrpc recipe and other Qt6 recipes are now visible in the layerindex
> https://layers.openembedded.org/layerindex/recipe/348210/
>
>
> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
>
>> I'd like to follow-up on this and say that I see this issue now with
>> nativesdk build, in particular nativesdk-grpc recipe fails in master with
>> the same error. I hope that's partially related to the issue everybody is
>> confused about, but I'd like understand how to properly fix it.
>>
>> |   The imported target "protobuf::protoc" references the file
>> |
>> |
>> "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>> |
>> |   but this file does not exist.  Possible reasons include:
>> |
>> |   * The file was deleted, renamed, or moved to another location.
>>
>> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>>
>> And we do want protoc to be present in the SDK, but nativesdk build is a
>> cross-compilation, because SDKMACHINE and HOST might be different.
>> In this particular grpc case there's an option to say where the actual
>> protoc executable is (
>> -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
>> is passed to grpc).
>>
>> So there shouldn't be any issues here, but nativesdk build still fails.
>>
>> Looking into it further I see that protoc is actually packaged by
>> protobuf-compiler package, but when I try to add a package dependency to
>> grpc recipe I get this:
>>
>> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>>
>> Moreover, if I go into devshell and forcefully install
>> nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and
>> compiles!
>>
>> So something wrong with the dependencies in protobuf recipe, but I can't
>> figure out yet what exactly. Any ideas?
>>
>> Regards,
>> Slava
>>
>>

do_prepare_recipe_sysroot doesn't install any packages to RSS. Only package
manager on target or in do_rootfs uses packages.

It installs the files from dependency populate_sysroot task.

On Fri, Oct 20, 2023 at 9:11 AM Samuli Piippo <samuli.piippo@gmail.com>
wrote:

> Dependency to the recipe name will always install all packages to RSS.
> You can verify this by checking that the libprotoc.so from the
> nativesdk-protobuf-compile package is in the sysroot.
> The problem you are facing is the fact that binaries are not installed
> into the sysroot (for nativesdk builds), so you are only missing the
> bin/protoc-23.4.0 file.
> The proposed and twice reverted fix for this is the aforementioned
> SYSROOT_DIRS.
>
> -samuli
>
> On Fri, 20 Oct 2023 at 09:17, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
>
>> I think the problem goes down to how dependencies are populated in the
>> sysroot. I raised this question in oe-core mailing list
>> https://lists.openembedded.org/g/openembedded-core/message/189324 , but
>> didn't get an answer yet.
>>
>> DEPENDS += "protobuf" will only install the main package of protobuf to
>> the sysroot, because it assumes that nativesdk-protobuf-compiler is also
>> part of the main package (which is not). I guess the part of the system
>> populating sysroot deals with recipes only, and not with packages. I'm
>> still trying to understand how that should be addressed properly.
>> My understanding is that either all packages should be installed to
>> sysroot (which recipe claims it provides), or we need a way to indicate a
>> package level dependency (i.e. nativesdk-protobuf-compiler in this case)
>>
>> Any ideas?
>>
>> Slava
>>
>> On 18.10.2023 14:28, Samuli Piippo wrote:
>>
>> Following simple test recipe will fail now when trying to use Protobuf
>> with CMake.
>>
>> inherit cmake
>> DEPENDS += "protobuf"
>>
>> do_configure:prepend() {
>>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
>> }
>>
>> BBCLASSEXTEND = "nativesdk"
>>
>> ---
>> CMake Error at
>> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
>> (message):
>>   The imported target "protobuf::protoc" references the file
>>
>>
>>  "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>
>>   but this file does not exist.
>> ---
>>
>> I don't quite see why every project/recipe should fix this independently
>> when the simple workaround for this yocto-specific issue is available
>> (namely the use of SYSROOT_DIRS).
>>
>> -samuli
>>
>> ps. qtgrpc recipe and other Qt6 recipes are now visible in the layerindex
>> https://layers.openembedded.org/layerindex/recipe/348210/
>>
>>
>> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com>
>> wrote:
>>
>>> I'd like to follow-up on this and say that I see this issue now with
>>> nativesdk build, in particular nativesdk-grpc recipe fails in master with
>>> the same error. I hope that's partially related to the issue everybody is
>>> confused about, but I'd like understand how to properly fix it.
>>>
>>> |   The imported target "protobuf::protoc" references the file
>>> |
>>> |
>>> "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>> |
>>> |   but this file does not exist.  Possible reasons include:
>>> |
>>> |   * The file was deleted, renamed, or moved to another location.
>>>
>>> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>>>
>>> And we do want protoc to be present in the SDK, but nativesdk build is a
>>> cross-compilation, because SDKMACHINE and HOST might be different.
>>> In this particular grpc case there's an option to say where the actual
>>> protoc executable is (
>>> -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
>>> is passed to grpc).
>>>
>>> So there shouldn't be any issues here, but nativesdk build still fails.
>>>
>>> Looking into it further I see that protoc is actually packaged by
>>> protobuf-compiler package, but when I try to add a package dependency to
>>> grpc recipe I get this:
>>>
>>> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>>>
>>> Moreover, if I go into devshell and forcefully install
>>> nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and
>>> compiles!
>>>
>>> So something wrong with the dependencies in protobuf recipe, but I can't
>>> figure out yet what exactly. Any ideas?
>>>
>>> Regards,
>>> Slava
>>>
>>>

I'm a bit confused here.

I just did 'find 
tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/ -name 
"protoc*"' and I see protoc binary only in recipe-sysroot-native. That 
tells me that protoc, which is a part of nativesdk-protobuf-compiler, is 
not installed to recipe-sysroot.

What am I missing here?

Slava

On 20.10.2023 09:35, Martin Jansa wrote:
> do_prepare_recipe_sysroot doesn't install any packages to RSS. Only 
> package manager on target or in do_rootfs uses packages.
>
> It installs the files from dependency populate_sysroot task.
>
> On Fri, Oct 20, 2023 at 9:11 AM Samuli Piippo 
> <samuli.piippo@gmail.com> wrote:
>
>     Dependency to the recipe name will always install all packages to RSS.
>     You can verify this by checking that the libprotoc.so from the
>     nativesdk-protobuf-compile package is in the sysroot.
>     The problem you are facing is the fact that binaries are not
>     installed into the sysroot (for nativesdk builds), so you are only
>     missing the bin/protoc-23.4.0 file.
>     The proposed and twice reverted fix for this is the aforementioned
>     SYSROOT_DIRS.
>
>     -samuli
>
>     On Fri, 20 Oct 2023 at 09:17, Vyacheslav Yurkov
>     <uvv.mail@gmail.com> wrote:
>
>         I think the problem goes down to how dependencies are
>         populated in the sysroot. I raised this question in oe-core
>         mailing list
>         https://lists.openembedded.org/g/openembedded-core/message/189324
>         , but didn't get an answer yet.
>
>         DEPENDS += "protobuf" will only install the main package of
>         protobuf to the sysroot, because it assumes that
>         nativesdk-protobuf-compiler is also part of the main package
>         (which is not). I guess the part of the system populating
>         sysroot deals with recipes only, and not with packages. I'm
>         still trying to understand how that should be addressed properly.
>         My understanding is that either all packages should be
>         installed to sysroot (which recipe claims it provides), or we
>         need a way to indicate a package level dependency (i.e.
>         nativesdk-protobuf-compiler in this case)
>
>         Any ideas?
>
>         Slava
>
>         On 18.10.2023 14:28, Samuli Piippo wrote:
>>         Following simple test recipe will fail now when trying to use
>>         Protobuf with CMake.
>>
>>         inherit cmake
>>         DEPENDS += "protobuf"
>>
>>         do_configure:prepend() {
>>             echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
>>         }
>>
>>         BBCLASSEXTEND = "nativesdk"
>>
>>         ---
>>         CMake Error at
>>         tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
>>         (message):
>>           The imported target "protobuf::protoc" references the file
>>
>>          "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>
>>           but this file does not exist.
>>         ---
>>
>>         I don't quite see why every project/recipe should fix this
>>         independently when the simple workaround for this
>>         yocto-specific issue is available (namely the use
>>         of SYSROOT_DIRS).
>>
>>         -samuli
>>
>>         ps. qtgrpc recipe and other Qt6 recipes are now visible in
>>         the layerindex
>>         https://layers.openembedded.org/layerindex/recipe/348210/
>>
>>
>>         On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov
>>         <uvv.mail@gmail.com> wrote:
>>
>>             I'd like to follow-up on this and say that I see this
>>             issue now with nativesdk build, in particular
>>             nativesdk-grpc recipe fails in master with the same
>>             error. I hope that's partially related to the issue
>>             everybody is confused about, but I'd like understand how
>>             to properly fix it.
>>
>>             |   The imported target "protobuf::protoc" references the
>>             file
>>             |
>>             |
>>             "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>             |
>>             |   but this file does not exist.  Possible reasons include:
>>             |
>>             |   * The file was deleted, renamed, or moved to another
>>             location.
>>
>>             The protoc-23.4.0 file already exists in the
>>             recipe-sysroot-native.
>>
>>             And we do want protoc to be present in the SDK, but
>>             nativesdk build is a cross-compilation, because
>>             SDKMACHINE and HOST might be different.
>>             In this particular grpc case there's an option to say
>>             where the actual protoc executable is (
>>             -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
>>             is passed to grpc).
>>
>>             So there shouldn't be any issues here, but nativesdk
>>             build still fails.
>>
>>             Looking into it further I see that protoc is actually
>>             packaged by protobuf-compiler package, but when I try to
>>             add a package dependency to grpc recipe I get this:
>>
>>             nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>>
>>             Moreover, if I go into devshell and forcefully install
>>             nativesdk-libprotobuf-compiler, then `bitbake
>>             nativesdk-grpc` is happy and compiles!
>>
>>             So something wrong with the dependencies in protobuf
>>             recipe, but I can't figure out yet what exactly. Any ideas?
>>
>>             Regards,
>>             Slava
>>

Hi,

I have pushed a branch on my repo:
https://github.com/clementperon/meta-openembedded/tree/test_protobuf

and tried to reproduce the issue on my setup with the following kas yaml file.

But I don't see any issue at the configure stage.

cat XXXX/cortexa57-poky-linux-musl/test-protobuf/0.3/temp/log.do_configure

Show that -- Version:23.4.0 of Protobuf is found without any issue

Cmake host version is
$> cmake --version
cmake version 3.27.7

I'm missing something ?

Regards,
Clement

kas.yml
====================
header:
  version: 8

machine: qemuarm64
distro: poky
target:
  - test-protobuf

repos:
  poky:
    url: https://git.yoctoproject.org/git/poky
    path: layers/poky
    refspec: master
    layers:
      meta:
      meta-poky:
      meta-yocto-bsp:

  meta-openembedded:
    url: https://github.com/clementperon/meta-openembedded
    path: layers/meta-openembedded
    refspec: test_protobuf
    layers:
      meta-oe:
      meta-networking:
      meta-python:

bblayers_conf_header:
  standard: |
    POKY_BBLAYERS_CONF_VERSION = "2"
    BBPATH = "${TOPDIR}"
    BBFILES ?= ""

local_conf_header:
  libc: |
    TCLIBC = "musl"
  standard: |
    CONF_VERSION = "2"
    PACKAGE_CLASSES = "package_ipk"
  debug-tweaks: |
    EXTRA_IMAGE_FEATURES ?= "debug-tweaks"

  diskmon: |
    BB_DISKMON_DIRS = "\
        STOPTASKS,${TMPDIR},1G,100K \
        STOPTASKS,${DL_DIR},1G,100K \
        STOPTASKS,${SSTATE_DIR},1G,100K \
        STOPTASKS,/tmp,100M,100K \
        ABORT,${TMPDIR},100M,1K \
        ABORT,${DL_DIR},100M,1K \
        ABORT,${SSTATE_DIR},100M,1K \
        ABORT,/tmp,10M,1K"
======================

On Sun, 22 Oct 2023 at 20:45, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>
> I'm a bit confused here.
>
> I just did 'find tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/ -name "protoc*"' and I see protoc binary only in recipe-sysroot-native. That tells me that protoc, which is a part of nativesdk-protobuf-compiler, is not installed to recipe-sysroot.
>
> What am I missing here?
>
> Slava
>
> On 20.10.2023 09:35, Martin Jansa wrote:
>
> do_prepare_recipe_sysroot doesn't install any packages to RSS. Only package manager on target or in do_rootfs uses packages.
>
> It installs the files from dependency populate_sysroot task.
>
> On Fri, Oct 20, 2023 at 9:11 AM Samuli Piippo <samuli.piippo@gmail.com> wrote:
>>
>> Dependency to the recipe name will always install all packages to RSS.
>> You can verify this by checking that the libprotoc.so from the nativesdk-protobuf-compile package is in the sysroot.
>> The problem you are facing is the fact that binaries are not installed into the sysroot (for nativesdk builds), so you are only missing the bin/protoc-23.4.0 file.
>> The proposed and twice reverted fix for this is the aforementioned SYSROOT_DIRS.
>>
>> -samuli
>>
>> On Fri, 20 Oct 2023 at 09:17, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>>>
>>> I think the problem goes down to how dependencies are populated in the sysroot. I raised this question in oe-core mailing list https://lists.openembedded.org/g/openembedded-core/message/189324 , but didn't get an answer yet.
>>>
>>> DEPENDS += "protobuf" will only install the main package of protobuf to the sysroot, because it assumes that nativesdk-protobuf-compiler is also part of the main package (which is not). I guess the part of the system populating sysroot deals with recipes only, and not with packages. I'm still trying to understand how that should be addressed properly.
>>> My understanding is that either all packages should be installed to sysroot (which recipe claims it provides), or we need a way to indicate a package level dependency (i.e. nativesdk-protobuf-compiler in this case)
>>>
>>> Any ideas?
>>>
>>> Slava
>>>
>>> On 18.10.2023 14:28, Samuli Piippo wrote:
>>>
>>> Following simple test recipe will fail now when trying to use Protobuf with CMake.
>>>
>>> inherit cmake
>>> DEPENDS += "protobuf"
>>>
>>> do_configure:prepend() {
>>>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
>>> }
>>>
>>> BBCLASSEXTEND = "nativesdk"
>>>
>>> ---
>>> CMake Error at tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107 (message):
>>>   The imported target "protobuf::protoc" references the file
>>>
>>>      "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>>
>>>   but this file does not exist.
>>> ---
>>>
>>> I don't quite see why every project/recipe should fix this independently when the simple workaround for this yocto-specific issue is available (namely the use of SYSROOT_DIRS).
>>>
>>> -samuli
>>>
>>> ps. qtgrpc recipe and other Qt6 recipes are now visible in the layerindex https://layers.openembedded.org/layerindex/recipe/348210/
>>>
>>>
>>> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>>>>
>>>> I'd like to follow-up on this and say that I see this issue now with nativesdk build, in particular nativesdk-grpc recipe fails in master with the same error. I hope that's partially related to the issue everybody is confused about, but I'd like understand how to properly fix it.
>>>>
>>>> |   The imported target "protobuf::protoc" references the file
>>>> |
>>>> |      "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>>> |
>>>> |   but this file does not exist.  Possible reasons include:
>>>> |
>>>> |   * The file was deleted, renamed, or moved to another location.
>>>>
>>>> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>>>>
>>>> And we do want protoc to be present in the SDK, but nativesdk build is a cross-compilation, because SDKMACHINE and HOST might be different.
>>>> In this particular grpc case there's an option to say where the actual protoc executable is ( -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc is passed to grpc).
>>>>
>>>> So there shouldn't be any issues here, but nativesdk build still fails.
>>>>
>>>> Looking into it further I see that protoc is actually packaged by protobuf-compiler package, but when I try to add a package dependency to grpc recipe I get this:
>>>>
>>>> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>>>>
>>>> Moreover, if I go into devshell and forcefully install nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and compiles!
>>>>
>>>> So something wrong with the dependencies in protobuf recipe, but I can't figure out yet what exactly. Any ideas?
>>>>
>>>> Regards,
>>>> Slava
>>>>
>
>

The protoc binary is not enabled for target builds. Try nativesdk build and
it should then fail to find it.

On Tue, 24 Oct 2023 at 14:53, Clément Péron <peron.clem@gmail.com> wrote:

> Hi,
>
> I have pushed a branch on my repo:
> https://github.com/clementperon/meta-openembedded/tree/test_protobuf
>
> and tried to reproduce the issue on my setup with the following kas yaml
> file.
>
> But I don't see any issue at the configure stage.
>
> cat XXXX/cortexa57-poky-linux-musl/test-protobuf/0.3/temp/log.do_configure
>
> Show that -- Version:23.4.0 of Protobuf is found without any issue
>
> Cmake host version is
> $> cmake --version
> cmake version 3.27.7
>
> I'm missing something ?
>
> Regards,
> Clement
>
> kas.yml
> ====================
> header:
>   version: 8
>
> machine: qemuarm64
> distro: poky
> target:
>   - test-protobuf
>
> repos:
>   poky:
>     url: https://git.yoctoproject.org/git/poky
>     path: layers/poky
>     refspec: master
>     layers:
>       meta:
>       meta-poky:
>       meta-yocto-bsp:
>
>   meta-openembedded:
>     url: https://github.com/clementperon/meta-openembedded
>     path: layers/meta-openembedded
>     refspec: test_protobuf
>     layers:
>       meta-oe:
>       meta-networking:
>       meta-python:
>
> bblayers_conf_header:
>   standard: |
>     POKY_BBLAYERS_CONF_VERSION = "2"
>     BBPATH = "${TOPDIR}"
>     BBFILES ?= ""
>
> local_conf_header:
>   libc: |
>     TCLIBC = "musl"
>   standard: |
>     CONF_VERSION = "2"
>     PACKAGE_CLASSES = "package_ipk"
>   debug-tweaks: |
>     EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
>
>   diskmon: |
>     BB_DISKMON_DIRS = "\
>         STOPTASKS,${TMPDIR},1G,100K \
>         STOPTASKS,${DL_DIR},1G,100K \
>         STOPTASKS,${SSTATE_DIR},1G,100K \
>         STOPTASKS,/tmp,100M,100K \
>         ABORT,${TMPDIR},100M,1K \
>         ABORT,${DL_DIR},100M,1K \
>         ABORT,${SSTATE_DIR},100M,1K \
>         ABORT,/tmp,10M,1K"
> ======================
>
> On Sun, 22 Oct 2023 at 20:45, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
> >
> > I'm a bit confused here.
> >
> > I just did 'find
> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/ -name
> "protoc*"' and I see protoc binary only in recipe-sysroot-native. That
> tells me that protoc, which is a part of nativesdk-protobuf-compiler, is
> not installed to recipe-sysroot.
> >
> > What am I missing here?
> >
> > Slava
> >
> > On 20.10.2023 09:35, Martin Jansa wrote:
> >
> > do_prepare_recipe_sysroot doesn't install any packages to RSS. Only
> package manager on target or in do_rootfs uses packages.
> >
> > It installs the files from dependency populate_sysroot task.
> >
> > On Fri, Oct 20, 2023 at 9:11 AM Samuli Piippo <samuli.piippo@gmail.com>
> wrote:
> >>
> >> Dependency to the recipe name will always install all packages to RSS.
> >> You can verify this by checking that the libprotoc.so from the
> nativesdk-protobuf-compile package is in the sysroot.
> >> The problem you are facing is the fact that binaries are not installed
> into the sysroot (for nativesdk builds), so you are only missing the
> bin/protoc-23.4.0 file.
> >> The proposed and twice reverted fix for this is the aforementioned
> SYSROOT_DIRS.
> >>
> >> -samuli
> >>
> >> On Fri, 20 Oct 2023 at 09:17, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
> >>>
> >>> I think the problem goes down to how dependencies are populated in the
> sysroot. I raised this question in oe-core mailing list
> https://lists.openembedded.org/g/openembedded-core/message/189324 , but
> didn't get an answer yet.
> >>>
> >>> DEPENDS += "protobuf" will only install the main package of protobuf
> to the sysroot, because it assumes that nativesdk-protobuf-compiler is also
> part of the main package (which is not). I guess the part of the system
> populating sysroot deals with recipes only, and not with packages. I'm
> still trying to understand how that should be addressed properly.
> >>> My understanding is that either all packages should be installed to
> sysroot (which recipe claims it provides), or we need a way to indicate a
> package level dependency (i.e. nativesdk-protobuf-compiler in this case)
> >>>
> >>> Any ideas?
> >>>
> >>> Slava
> >>>
> >>> On 18.10.2023 14:28, Samuli Piippo wrote:
> >>>
> >>> Following simple test recipe will fail now when trying to use Protobuf
> with CMake.
> >>>
> >>> inherit cmake
> >>> DEPENDS += "protobuf"
> >>>
> >>> do_configure:prepend() {
> >>>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
> >>> }
> >>>
> >>> BBCLASSEXTEND = "nativesdk"
> >>>
> >>> ---
> >>> CMake Error at
> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
> (message):
> >>>   The imported target "protobuf::protoc" references the file
> >>>
> >>>
> "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
> >>>
> >>>   but this file does not exist.
> >>> ---
> >>>
> >>> I don't quite see why every project/recipe should fix this
> independently when the simple workaround for this yocto-specific issue is
> available (namely the use of SYSROOT_DIRS).
> >>>
> >>> -samuli
> >>>
> >>> ps. qtgrpc recipe and other Qt6 recipes are now visible in the
> layerindex https://layers.openembedded.org/layerindex/recipe/348210/
> >>>
> >>>
> >>> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
> >>>>
> >>>> I'd like to follow-up on this and say that I see this issue now with
> nativesdk build, in particular nativesdk-grpc recipe fails in master with
> the same error. I hope that's partially related to the issue everybody is
> confused about, but I'd like understand how to properly fix it.
> >>>>
> >>>> |   The imported target "protobuf::protoc" references the file
> >>>> |
> >>>> |
> "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
> >>>> |
> >>>> |   but this file does not exist.  Possible reasons include:
> >>>> |
> >>>> |   * The file was deleted, renamed, or moved to another location.
> >>>>
> >>>> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
> >>>>
> >>>> And we do want protoc to be present in the SDK, but nativesdk build
> is a cross-compilation, because SDKMACHINE and HOST might be different.
> >>>> In this particular grpc case there's an option to say where the
> actual protoc executable is (
> -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
> is passed to grpc).
> >>>>
> >>>> So there shouldn't be any issues here, but nativesdk build still
> fails.
> >>>>
> >>>> Looking into it further I see that protoc is actually packaged by
> protobuf-compiler package, but when I try to add a package dependency to
> grpc recipe I get this:
> >>>>
> >>>> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
> >>>>
> >>>> Moreover, if I go into devshell and forcefully install
> nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and
> compiles!
> >>>>
> >>>> So something wrong with the dependencies in protobuf recipe, but I
> can't figure out yet what exactly. Any ideas?
> >>>>
> >>>> Regards,
> >>>> Slava
> >>>>
> >
> >
>

Hi Clément,

Were you able to reproduce the issue?

On Tue, 24 Oct 2023 at 16:33, Samuli Piippo via lists.openembedded.org
<samuli.piippo=gmail.com@lists.openembedded.org> wrote:

> The protoc binary is not enabled for target builds. Try nativesdk build
> and it should then fail to find it.
>
> On Tue, 24 Oct 2023 at 14:53, Clément Péron <peron.clem@gmail.com> wrote:
>
>> Hi,
>>
>> I have pushed a branch on my repo:
>> https://github.com/clementperon/meta-openembedded/tree/test_protobuf
>>
>> and tried to reproduce the issue on my setup with the following kas yaml
>> file.
>>
>> But I don't see any issue at the configure stage.
>>
>> cat XXXX/cortexa57-poky-linux-musl/test-protobuf/0.3/temp/log.do_configure
>>
>> Show that -- Version:23.4.0 of Protobuf is found without any issue
>>
>> Cmake host version is
>> $> cmake --version
>> cmake version 3.27.7
>>
>> I'm missing something ?
>>
>> Regards,
>> Clement
>>
>> kas.yml
>> ====================
>> header:
>>   version: 8
>>
>> machine: qemuarm64
>> distro: poky
>> target:
>>   - test-protobuf
>>
>> repos:
>>   poky:
>>     url: https://git.yoctoproject.org/git/poky
>>     path: layers/poky
>>     refspec: master
>>     layers:
>>       meta:
>>       meta-poky:
>>       meta-yocto-bsp:
>>
>>   meta-openembedded:
>>     url: https://github.com/clementperon/meta-openembedded
>>     path: layers/meta-openembedded
>>     refspec: test_protobuf
>>     layers:
>>       meta-oe:
>>       meta-networking:
>>       meta-python:
>>
>> bblayers_conf_header:
>>   standard: |
>>     POKY_BBLAYERS_CONF_VERSION = "2"
>>     BBPATH = "${TOPDIR}"
>>     BBFILES ?= ""
>>
>> local_conf_header:
>>   libc: |
>>     TCLIBC = "musl"
>>   standard: |
>>     CONF_VERSION = "2"
>>     PACKAGE_CLASSES = "package_ipk"
>>   debug-tweaks: |
>>     EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
>>
>>   diskmon: |
>>     BB_DISKMON_DIRS = "\
>>         STOPTASKS,${TMPDIR},1G,100K \
>>         STOPTASKS,${DL_DIR},1G,100K \
>>         STOPTASKS,${SSTATE_DIR},1G,100K \
>>         STOPTASKS,/tmp,100M,100K \
>>         ABORT,${TMPDIR},100M,1K \
>>         ABORT,${DL_DIR},100M,1K \
>>         ABORT,${SSTATE_DIR},100M,1K \
>>         ABORT,/tmp,10M,1K"
>> ======================
>>
>> On Sun, 22 Oct 2023 at 20:45, Vyacheslav Yurkov <uvv.mail@gmail.com>
>> wrote:
>> >
>> > I'm a bit confused here.
>> >
>> > I just did 'find
>> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/ -name
>> "protoc*"' and I see protoc binary only in recipe-sysroot-native. That
>> tells me that protoc, which is a part of nativesdk-protobuf-compiler, is
>> not installed to recipe-sysroot.
>> >
>> > What am I missing here?
>> >
>> > Slava
>> >
>> > On 20.10.2023 09:35, Martin Jansa wrote:
>> >
>> > do_prepare_recipe_sysroot doesn't install any packages to RSS. Only
>> package manager on target or in do_rootfs uses packages.
>> >
>> > It installs the files from dependency populate_sysroot task.
>> >
>> > On Fri, Oct 20, 2023 at 9:11 AM Samuli Piippo <samuli.piippo@gmail.com>
>> wrote:
>> >>
>> >> Dependency to the recipe name will always install all packages to RSS.
>> >> You can verify this by checking that the libprotoc.so from the
>> nativesdk-protobuf-compile package is in the sysroot.
>> >> The problem you are facing is the fact that binaries are not installed
>> into the sysroot (for nativesdk builds), so you are only missing the
>> bin/protoc-23.4.0 file.
>> >> The proposed and twice reverted fix for this is the aforementioned
>> SYSROOT_DIRS.
>> >>
>> >> -samuli
>> >>
>> >> On Fri, 20 Oct 2023 at 09:17, Vyacheslav Yurkov <uvv.mail@gmail.com>
>> wrote:
>> >>>
>> >>> I think the problem goes down to how dependencies are populated in
>> the sysroot. I raised this question in oe-core mailing list
>> https://lists.openembedded.org/g/openembedded-core/message/189324 , but
>> didn't get an answer yet.
>> >>>
>> >>> DEPENDS += "protobuf" will only install the main package of protobuf
>> to the sysroot, because it assumes that nativesdk-protobuf-compiler is also
>> part of the main package (which is not). I guess the part of the system
>> populating sysroot deals with recipes only, and not with packages. I'm
>> still trying to understand how that should be addressed properly.
>> >>> My understanding is that either all packages should be installed to
>> sysroot (which recipe claims it provides), or we need a way to indicate a
>> package level dependency (i.e. nativesdk-protobuf-compiler in this case)
>> >>>
>> >>> Any ideas?
>> >>>
>> >>> Slava
>> >>>
>> >>> On 18.10.2023 14:28, Samuli Piippo wrote:
>> >>>
>> >>> Following simple test recipe will fail now when trying to use
>> Protobuf with CMake.
>> >>>
>> >>> inherit cmake
>> >>> DEPENDS += "protobuf"
>> >>>
>> >>> do_configure:prepend() {
>> >>>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
>> >>> }
>> >>>
>> >>> BBCLASSEXTEND = "nativesdk"
>> >>>
>> >>> ---
>> >>> CMake Error at
>> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
>> (message):
>> >>>   The imported target "protobuf::protoc" references the file
>> >>>
>> >>>
>> "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>> >>>
>> >>>   but this file does not exist.
>> >>> ---
>> >>>
>> >>> I don't quite see why every project/recipe should fix this
>> independently when the simple workaround for this yocto-specific issue is
>> available (namely the use of SYSROOT_DIRS).
>> >>>
>> >>> -samuli
>> >>>
>> >>> ps. qtgrpc recipe and other Qt6 recipes are now visible in the
>> layerindex https://layers.openembedded.org/layerindex/recipe/348210/
>> >>>
>> >>>
>> >>> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com>
>> wrote:
>> >>>>
>> >>>> I'd like to follow-up on this and say that I see this issue now with
>> nativesdk build, in particular nativesdk-grpc recipe fails in master with
>> the same error. I hope that's partially related to the issue everybody is
>> confused about, but I'd like understand how to properly fix it.
>> >>>>
>> >>>> |   The imported target "protobuf::protoc" references the file
>> >>>> |
>> >>>> |
>> "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>> >>>> |
>> >>>> |   but this file does not exist.  Possible reasons include:
>> >>>> |
>> >>>> |   * The file was deleted, renamed, or moved to another location.
>> >>>>
>> >>>> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>> >>>>
>> >>>> And we do want protoc to be present in the SDK, but nativesdk build
>> is a cross-compilation, because SDKMACHINE and HOST might be different.
>> >>>> In this particular grpc case there's an option to say where the
>> actual protoc executable is (
>> -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
>> is passed to grpc).
>> >>>>
>> >>>> So there shouldn't be any issues here, but nativesdk build still
>> fails.
>> >>>>
>> >>>> Looking into it further I see that protoc is actually packaged by
>> protobuf-compiler package, but when I try to add a package dependency to
>> grpc recipe I get this:
>> >>>>
>> >>>> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>> >>>>
>> >>>> Moreover, if I go into devshell and forcefully install
>> nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and
>> compiles!
>> >>>>
>> >>>> So something wrong with the dependencies in protobuf recipe, but I
>> can't figure out yet what exactly. Any ideas?
>> >>>>
>> >>>> Regards,
>> >>>> Slava
>> >>>>
>> >
>> >
>>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#105722):
> https://lists.openembedded.org/g/openembedded-devel/message/105722
> Mute This Topic: https://lists.openembedded.org/mt/101679410/3617605
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
> samuli.piippo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

Hi Samuli,

Le mer. 8 nov. 2023, 22:30, Samuli Piippo <samuli.piippo@gmail.com> a
écrit :

> Hi Clément,
>
> Were you able to reproduce the issue?
>

Sorry I'm on vacation without access to a laptop at the moment.

But if I understand the issue is not related to a target or native recipes
issue, right?

It's only when building the SDK and then try to build an application with
it right?

Maybe there is a missing dependency when building the SDK.

Regards,
Clement



> On Tue, 24 Oct 2023 at 16:33, Samuli Piippo via lists.openembedded.org
> <samuli.piippo=gmail.com@lists.openembedded.org> wrote:
>
>> The protoc binary is not enabled for target builds. Try nativesdk build
>> and it should then fail to find it.
>>
>> On Tue, 24 Oct 2023 at 14:53, Clément Péron <peron.clem@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have pushed a branch on my repo:
>>> https://github.com/clementperon/meta-openembedded/tree/test_protobuf
>>>
>>> and tried to reproduce the issue on my setup with the following kas yaml
>>> file.
>>>
>>> But I don't see any issue at the configure stage.
>>>
>>> cat
>>> XXXX/cortexa57-poky-linux-musl/test-protobuf/0.3/temp/log.do_configure
>>>
>>> Show that -- Version:23.4.0 of Protobuf is found without any issue
>>>
>>> Cmake host version is
>>> $> cmake --version
>>> cmake version 3.27.7
>>>
>>> I'm missing something ?
>>>
>>> Regards,
>>> Clement
>>>
>>> kas.yml
>>> ====================
>>> header:
>>>   version: 8
>>>
>>> machine: qemuarm64
>>> distro: poky
>>> target:
>>>   - test-protobuf
>>>
>>> repos:
>>>   poky:
>>>     url: https://git.yoctoproject.org/git/poky
>>>     path: layers/poky
>>>     refspec: master
>>>     layers:
>>>       meta:
>>>       meta-poky:
>>>       meta-yocto-bsp:
>>>
>>>   meta-openembedded:
>>>     url: https://github.com/clementperon/meta-openembedded
>>>     path: layers/meta-openembedded
>>>     refspec: test_protobuf
>>>     layers:
>>>       meta-oe:
>>>       meta-networking:
>>>       meta-python:
>>>
>>> bblayers_conf_header:
>>>   standard: |
>>>     POKY_BBLAYERS_CONF_VERSION = "2"
>>>     BBPATH = "${TOPDIR}"
>>>     BBFILES ?= ""
>>>
>>> local_conf_header:
>>>   libc: |
>>>     TCLIBC = "musl"
>>>   standard: |
>>>     CONF_VERSION = "2"
>>>     PACKAGE_CLASSES = "package_ipk"
>>>   debug-tweaks: |
>>>     EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
>>>
>>>   diskmon: |
>>>     BB_DISKMON_DIRS = "\
>>>         STOPTASKS,${TMPDIR},1G,100K \
>>>         STOPTASKS,${DL_DIR},1G,100K \
>>>         STOPTASKS,${SSTATE_DIR},1G,100K \
>>>         STOPTASKS,/tmp,100M,100K \
>>>         ABORT,${TMPDIR},100M,1K \
>>>         ABORT,${DL_DIR},100M,1K \
>>>         ABORT,${SSTATE_DIR},100M,1K \
>>>         ABORT,/tmp,10M,1K"
>>> ======================
>>>
>>> On Sun, 22 Oct 2023 at 20:45, Vyacheslav Yurkov <uvv.mail@gmail.com>
>>> wrote:
>>> >
>>> > I'm a bit confused here.
>>> >
>>> > I just did 'find
>>> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/ -name
>>> "protoc*"' and I see protoc binary only in recipe-sysroot-native. That
>>> tells me that protoc, which is a part of nativesdk-protobuf-compiler, is
>>> not installed to recipe-sysroot.
>>> >
>>> > What am I missing here?
>>> >
>>> > Slava
>>> >
>>> > On 20.10.2023 09:35, Martin Jansa wrote:
>>> >
>>> > do_prepare_recipe_sysroot doesn't install any packages to RSS. Only
>>> package manager on target or in do_rootfs uses packages.
>>> >
>>> > It installs the files from dependency populate_sysroot task.
>>> >
>>> > On Fri, Oct 20, 2023 at 9:11 AM Samuli Piippo <samuli.piippo@gmail.com>
>>> wrote:
>>> >>
>>> >> Dependency to the recipe name will always install all packages to RSS.
>>> >> You can verify this by checking that the libprotoc.so from the
>>> nativesdk-protobuf-compile package is in the sysroot.
>>> >> The problem you are facing is the fact that binaries are not
>>> installed into the sysroot (for nativesdk builds), so you are only missing
>>> the bin/protoc-23.4.0 file.
>>> >> The proposed and twice reverted fix for this is the aforementioned
>>> SYSROOT_DIRS.
>>> >>
>>> >> -samuli
>>> >>
>>> >> On Fri, 20 Oct 2023 at 09:17, Vyacheslav Yurkov <uvv.mail@gmail.com>
>>> wrote:
>>> >>>
>>> >>> I think the problem goes down to how dependencies are populated in
>>> the sysroot. I raised this question in oe-core mailing list
>>> https://lists.openembedded.org/g/openembedded-core/message/189324 , but
>>> didn't get an answer yet.
>>> >>>
>>> >>> DEPENDS += "protobuf" will only install the main package of protobuf
>>> to the sysroot, because it assumes that nativesdk-protobuf-compiler is also
>>> part of the main package (which is not). I guess the part of the system
>>> populating sysroot deals with recipes only, and not with packages. I'm
>>> still trying to understand how that should be addressed properly.
>>> >>> My understanding is that either all packages should be installed to
>>> sysroot (which recipe claims it provides), or we need a way to indicate a
>>> package level dependency (i.e. nativesdk-protobuf-compiler in this case)
>>> >>>
>>> >>> Any ideas?
>>> >>>
>>> >>> Slava
>>> >>>
>>> >>> On 18.10.2023 14:28, Samuli Piippo wrote:
>>> >>>
>>> >>> Following simple test recipe will fail now when trying to use
>>> Protobuf with CMake.
>>> >>>
>>> >>> inherit cmake
>>> >>> DEPENDS += "protobuf"
>>> >>>
>>> >>> do_configure:prepend() {
>>> >>>     echo "find_package(Protobuf CONFIG)" > ${S}/CMakeLists.txt
>>> >>> }
>>> >>>
>>> >>> BBCLASSEXTEND = "nativesdk"
>>> >>>
>>> >>> ---
>>> >>> CMake Error at
>>> tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/cmake/protobuf/protobuf-targets.cmake:107
>>> (message):
>>> >>>   The imported target "protobuf::protoc" references the file
>>> >>>
>>> >>>
>>> "tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-test/1.0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>> >>>
>>> >>>   but this file does not exist.
>>> >>> ---
>>> >>>
>>> >>> I don't quite see why every project/recipe should fix this
>>> independently when the simple workaround for this yocto-specific issue is
>>> available (namely the use of SYSROOT_DIRS).
>>> >>>
>>> >>> -samuli
>>> >>>
>>> >>> ps. qtgrpc recipe and other Qt6 recipes are now visible in the
>>> layerindex https://layers.openembedded.org/layerindex/recipe/348210/
>>> >>>
>>> >>>
>>> >>> On Thu, 12 Oct 2023 at 15:06, Vyacheslav Yurkov <uvv.mail@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>> I'd like to follow-up on this and say that I see this issue now
>>> with nativesdk build, in particular nativesdk-grpc recipe fails in master
>>> with the same error. I hope that's partially related to the issue everybody
>>> is confused about, but I'd like understand how to properly fix it.
>>> >>>>
>>> >>>> |   The imported target "protobuf::protoc" references the file
>>> >>>> |
>>> >>>> |
>>> "/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/bin/protoc-23.4.0"
>>> >>>> |
>>> >>>> |   but this file does not exist.  Possible reasons include:
>>> >>>> |
>>> >>>> |   * The file was deleted, renamed, or moved to another location.
>>> >>>>
>>> >>>> The protoc-23.4.0 file already exists in the recipe-sysroot-native.
>>> >>>>
>>> >>>> And we do want protoc to be present in the SDK, but nativesdk build
>>> is a cross-compilation, because SDKMACHINE and HOST might be different.
>>> >>>> In this particular grpc case there's an option to say where the
>>> actual protoc executable is (
>>> -D_gRPC_PROTOBUF_PROTOC_EXECUTABLE=/home/uvv/projects/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-grpc/1.56.2/recipe-sysroot-native/usr/bin/protoc
>>> is passed to grpc).
>>> >>>>
>>> >>>> So there shouldn't be any issues here, but nativesdk build still
>>> fails.
>>> >>>>
>>> >>>> Looking into it further I see that protoc is actually packaged by
>>> protobuf-compiler package, but when I try to add a package dependency to
>>> grpc recipe I get this:
>>> >>>>
>>> >>>> nativesdk-protobuf RPROVIDES nativesdk-protobuf-compiler
>>> >>>>
>>> >>>> Moreover, if I go into devshell and forcefully install
>>> nativesdk-libprotobuf-compiler, then `bitbake nativesdk-grpc` is happy and
>>> compiles!
>>> >>>>
>>> >>>> So something wrong with the dependencies in protobuf recipe, but I
>>> can't figure out yet what exactly. Any ideas?
>>> >>>>
>>> >>>> Regards,
>>> >>>> Slava
>>> >>>>
>>> >
>>> >
>>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#105722):
>> https://lists.openembedded.org/g/openembedded-devel/message/105722
>> Mute This Topic: https://lists.openembedded.org/mt/101679410/3617605
>> Group Owner: openembedded-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
>> samuli.piippo@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>>

Hi Clément,
No, the issue is when you just build nativesdk recipe (we are not at the 
stage to use the SDK yet).

And that was exactly my question how to propagate the dependency 
correctly for nativesdk build.

Regards,
Slava

On 09.11.2023 01:13, Clément Péron wrote:
> Hi Samuli,
>
> Le mer. 8 nov. 2023, 22:30, Samuli Piippo <samuli.piippo@gmail.com> a 
> écrit :
>
>     Hi Clément,
>
>     Were you able to reproduce the issue?
>
>
> Sorry I'm on vacation without access to a laptop at the moment.
>
> But if I understand the issue is not related to a target or native 
> recipes issue, right?
>
> It's only when building the SDK and then try to build an application 
> with it right?
>
> Maybe there is a missing dependency when building the SDK.
>
> Regards,
> Clement
>

DEPENDS += "protobuf" is enough to have dependencies resolved.

Problem is that bitbake's do_populate_sysroot does not take anything from
bindir.
This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to not get the
protoc binary into the recipe_sysroot as expected by CMake.

This can be fixed with SYSROOT_DIRS.

On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:

> Hi Clément,
> No, the issue is when you just build nativesdk recipe (we are not at the
> stage to use the SDK yet).
>
> And that was exactly my question how to propagate the dependency correctly
> for nativesdk build.
>
> Regards,
> Slava
>
> On 09.11.2023 01:13, Clément Péron wrote:
>
> Hi Samuli,
>
> Le mer. 8 nov. 2023, 22:30, Samuli Piippo <samuli.piippo@gmail.com> a
> écrit :
>
>> Hi Clément,
>>
>> Were you able to reproduce the issue?
>>
>
> Sorry I'm on vacation without access to a laptop at the moment.
>
> But if I understand the issue is not related to a target or native recipes
> issue, right?
>
> It's only when building the SDK and then try to build an application with
> it right?
>
> Maybe there is a missing dependency when building the SDK.
>
> Regards,
> Clement
>
>
>

Thanks for the explanation, that was the missing piece for me to get an 
understanding of the problem :)

I think it's not only me, but also others, were confused by the fact 
that we are talking about native(sdk), but try to fix something with a 
non-native variable.

I've just gave it a try and have two suggestions how this can be addressed.

1. Recipe based solution:
SYSROOT_DIRS:append:class-nativesdk = " $bindir}"

2. oe-core based solution:
--- a/meta/classes-global/staging.bbclass
+++ b/meta/classes-global/staging.bbclass
@@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
      ${localstatedir} \
  "
  SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
+SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
  SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
  SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"

The second one will invalidate nativesdk sstate, but might cover not 
only grpc issue (I would assume gdbus-codegen or any other generator has 
the same issue when used for SDK build).

What are your thoughs?

Thanks,
Slava

On 09.11.2023 14:35, Samuli Piippo wrote:
> DEPENDS += "protobuf" is enough to have dependencies resolved.
>
> Problem is that bitbake's do_populate_sysroot does not take anything 
> from bindir.
> This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to not 
> get the protoc binary into the recipe_sysroot as expected by CMake.
>
> This can be fixed with SYSROOT_DIRS.
>
> On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>
>     Hi Clément,
>     No, the issue is when you just build nativesdk recipe (we are not
>     at the stage to use the SDK yet).
>
>     And that was exactly my question how to propagate the dependency
>     correctly for nativesdk build.
>
>     Regards,
>     Slava
>

Hi Samuli,

Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov <uvv.mail@gmail.com> a
écrit :

> Thanks for the explanation, that was the missing piece for me to get an
> understanding of the problem :)
>
> I think it's not only me, but also others, were confused by the fact that
> we are talking about native(sdk), but try to fix something with a
> non-native variable.
>
> I've just gave it a try and have two suggestions how this can be addressed.
>
> 1. Recipe based solution:
> SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>

Is it not just missing something like

DEPENDS:append:class-nativesdk = " protobuf-native "


Regards

Clement




> 2. oe-core based solution:
> --- a/meta/classes-global/staging.bbclass
> +++ b/meta/classes-global/staging.bbclass
> @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>      ${localstatedir} \
>  "
>  SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
> +SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
>  SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
>  SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
>
> The second one will invalidate nativesdk sstate, but might cover not only
> grpc issue (I would assume gdbus-codegen or any other generator has the
> same issue when used for SDK build).
>
> What are your thoughs?
>
> Thanks,
> Slava
>
> On 09.11.2023 14:35, Samuli Piippo wrote:
>
> DEPENDS += "protobuf" is enough to have dependencies resolved.
>
> Problem is that bitbake's do_populate_sysroot does not take anything from
> bindir.
> This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to not get
> the protoc binary into the recipe_sysroot as expected by CMake.
>
> This can be fixed with SYSROOT_DIRS.
>
> On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>
>> Hi Clément,
>> No, the issue is when you just build nativesdk recipe (we are not at the
>> stage to use the SDK yet).
>>
>> And that was exactly my question how to propagate the dependency
>> correctly for nativesdk build.
>>
>> Regards,
>> Slava
>>
>
>

On 10.11.2023 02:23, Clément Péron wrote:
> Hi Samuli,
>
> Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov <uvv.mail@gmail.com> a 
> écrit :
>
>     Thanks for the explanation, that was the missing piece for me to
>     get an understanding of the problem :)
>
>     I think it's not only me, but also others, were confused by the
>     fact that we are talking about native(sdk), but try to fix
>     something with a non-native variable.
>
>     I've just gave it a try and have two suggestions how this can be
>     addressed.
>
>     1. Recipe based solution:
>     SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>
>
> Is it not just missing something like
>
> DEPENDS:append:class-nativesdk= " protobuf-native "
> Regards
> Clement

This is already implicit by DEPENDS += "protobuf". Take a look at my 
second suggestion. You can see that ${bindir} is staged for native 
recipes, but not for nativesdk.

>
>     2. oe-core based solution:
>     --- a/meta/classes-global/staging.bbclass
>     +++ b/meta/classes-global/staging.bbclass
>     @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>          ${localstatedir} \
>      "
>      SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
>     +SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
>      SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
>      SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
>
>     The second one will invalidate nativesdk sstate, but might cover
>     not only grpc issue (I would assume gdbus-codegen or any other
>     generator has the same issue when used for SDK build).
>
>     What are your thoughs?
>
>     Thanks,
>     Slava
>
>     On 09.11.2023 14:35, Samuli Piippo wrote:
>>     DEPENDS += "protobuf" is enough to have dependencies resolved.
>>
>>     Problem is that bitbake's do_populate_sysroot does not take
>>     anything from bindir.
>>     This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to
>>     not get the protoc binary into the recipe_sysroot as expected by
>>     CMake.
>>
>>     This can be fixed with SYSROOT_DIRS.
>>
>>     On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov
>>     <uvv.mail@gmail.com> wrote:
>>
>>         Hi Clément,
>>         No, the issue is when you just build nativesdk recipe (we are
>>         not at the stage to use the SDK yet).
>>
>>         And that was exactly my question how to propagate the
>>         dependency correctly for nativesdk build.
>>
>>         Regards,
>>         Slava
>>
>

FTR, I suggested oe-core solution in the corresponding mailing list.

Slava

On 10.11.2023 07:19, Vyacheslav Yurkov via lists.openembedded.org wrote:
> On 10.11.2023 02:23, Clément Péron wrote:
>> Hi Samuli,
>>
>> Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov <uvv.mail@gmail.com> a 
>> écrit :
>>
>>     Thanks for the explanation, that was the missing piece for me to
>>     get an understanding of the problem :)
>>
>>     I think it's not only me, but also others, were confused by the
>>     fact that we are talking about native(sdk), but try to fix
>>     something with a non-native variable.
>>
>>     I've just gave it a try and have two suggestions how this can be
>>     addressed.
>>
>>     1. Recipe based solution:
>>     SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>>
>>
>> Is it not just missing something like
>>
>> DEPENDS:append:class-nativesdk= " protobuf-native "
>> Regards
>> Clement
>
> This is already implicit by DEPENDS += "protobuf". Take a look at my 
> second suggestion. You can see that ${bindir} is staged for native 
> recipes, but not for nativesdk.
>
>>
>>     2. oe-core based solution:
>>     --- a/meta/classes-global/staging.bbclass
>>     +++ b/meta/classes-global/staging.bbclass
>>     @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>>          ${localstatedir} \
>>      "
>>      SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
>>     +SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
>>      SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
>>      SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
>>
>>     The second one will invalidate nativesdk sstate, but might cover
>>     not only grpc issue (I would assume gdbus-codegen or any other
>>     generator has the same issue when used for SDK build).
>>
>>     What are your thoughs?
>>
>>     Thanks,
>>     Slava
>>
>>     On 09.11.2023 14:35, Samuli Piippo wrote:
>>>     DEPENDS += "protobuf" is enough to have dependencies resolved.
>>>
>>>     Problem is that bitbake's do_populate_sysroot does not take
>>>     anything from bindir.
>>>     This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to
>>>     not get the protoc binary into the recipe_sysroot as expected by
>>>     CMake.
>>>
>>>     This can be fixed with SYSROOT_DIRS.
>>>
>>>     On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov
>>>     <uvv.mail@gmail.com> wrote:
>>>
>>>         Hi Clément,
>>>         No, the issue is when you just build nativesdk recipe (we
>>>         are not at the stage to use the SDK yet).
>>>
>>>         And that was exactly my question how to propagate the
>>>         dependency correctly for nativesdk build.
>>>
>>>         Regards,
>>>         Slava
>>>
>>

This is not strictly a nativesdk issue.
If you enable PACKAGECONFIG = "compiler" for protobuf, it will then build
protoc for the target.
This will then cause grpc target build to fail in the same way as nativesdk
build.

On Fri, 10 Nov 2023 at 08:40, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:

> FTR, I suggested oe-core solution in the corresponding mailing list.
>
> Slava
>
> On 10.11.2023 07:19, Vyacheslav Yurkov via lists.openembedded.org wrote:
>
> On 10.11.2023 02:23, Clément Péron wrote:
>
> Hi Samuli,
>
> Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov <uvv.mail@gmail.com> a
> écrit :
>
>> Thanks for the explanation, that was the missing piece for me to get an
>> understanding of the problem :)
>>
>> I think it's not only me, but also others, were confused by the fact that
>> we are talking about native(sdk), but try to fix something with a
>> non-native variable.
>>
>> I've just gave it a try and have two suggestions how this can be
>> addressed.
>>
>> 1. Recipe based solution:
>> SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>>
>
> Is it not just missing something like
>
> DEPENDS:append:class-nativesdk = " protobuf-native "
>
>  Regards
>
> Clement
>
>
> This is already implicit by DEPENDS += "protobuf". Take a look at my
> second suggestion. You can see that ${bindir} is staged for native recipes,
> but not for nativesdk.
>
>
>> 2. oe-core based solution:
>> --- a/meta/classes-global/staging.bbclass
>> +++ b/meta/classes-global/staging.bbclass
>> @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>>      ${localstatedir} \
>>  "
>>  SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
>> +SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
>>  SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
>>  SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
>>
>> The second one will invalidate nativesdk sstate, but might cover not only
>> grpc issue (I would assume gdbus-codegen or any other generator has the
>> same issue when used for SDK build).
>>
>> What are your thoughs?
>>
>> Thanks,
>> Slava
>>
>> On 09.11.2023 14:35, Samuli Piippo wrote:
>>
>> DEPENDS += "protobuf" is enough to have dependencies resolved.
>>
>> Problem is that bitbake's do_populate_sysroot does not take anything from
>> bindir.
>> This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to not get
>> the protoc binary into the recipe_sysroot as expected by CMake.
>>
>> This can be fixed with SYSROOT_DIRS.
>>
>> On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov <uvv.mail@gmail.com>
>> wrote:
>>
>>> Hi Clément,
>>> No, the issue is when you just build nativesdk recipe (we are not at the
>>> stage to use the SDK yet).
>>>
>>> And that was exactly my question how to propagate the dependency
>>> correctly for nativesdk build.
>>>
>>> Regards,
>>> Slava
>>>
>>
>>
>

You sure? Because I remember the test in grpc to check for 
CROSS_COMPILING in cmake to avoid that.

Slava

On 10.11.2023 08:12, Samuli Piippo wrote:
> This is not strictly a nativesdk issue.
> If you enable PACKAGECONFIG = "compiler" for protobuf, it will then 
> build protoc for the target.
> This will then cause grpc target build to fail in the same way as 
> nativesdk build.
>
> On Fri, 10 Nov 2023 at 08:40, Vyacheslav Yurkov <uvv.mail@gmail.com> 
> wrote:
>
>     FTR, I suggested oe-core solution in the corresponding mailing list.
>
>     Slava
>
>     On 10.11.2023 07:19, Vyacheslav Yurkov via lists.openembedded.org
>     <http://lists.openembedded.org> wrote:
>>     On 10.11.2023 02:23, Clément Péron wrote:
>>>     Hi Samuli,
>>>
>>>     Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov
>>>     <uvv.mail@gmail.com> a écrit :
>>>
>>>         Thanks for the explanation, that was the missing piece for
>>>         me to get an understanding of the problem :)
>>>
>>>         I think it's not only me, but also others, were confused by
>>>         the fact that we are talking about native(sdk), but try to
>>>         fix something with a non-native variable.
>>>
>>>         I've just gave it a try and have two suggestions how this
>>>         can be addressed.
>>>
>>>         1. Recipe based solution:
>>>         SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>>>
>>>
>>>     Is it not just missing something like
>>>
>>>     DEPENDS:append:class-nativesdk= " protobuf-native "
>>>     Regards
>>>     Clement
>>
>>     This is already implicit by DEPENDS += "protobuf". Take a look at
>>     my second suggestion. You can see that ${bindir} is staged for
>>     native recipes, but not for nativesdk.
>>
>>>
>>>         2. oe-core based solution:
>>>         --- a/meta/classes-global/staging.bbclass
>>>         +++ b/meta/classes-global/staging.bbclass
>>>         @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>>>              ${localstatedir} \
>>>          "
>>>          SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
>>>         +SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
>>>          SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
>>>          SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
>>>
>>>         The second one will invalidate nativesdk sstate, but might
>>>         cover not only grpc issue (I would assume gdbus-codegen or
>>>         any other generator has the same issue when used for SDK build).
>>>
>>>         What are your thoughs?
>>>
>>>         Thanks,
>>>         Slava
>>>
>>>         On 09.11.2023 14:35, Samuli Piippo wrote:
>>>>         DEPENDS += "protobuf" is enough to have dependencies resolved.
>>>>
>>>>         Problem is that bitbake's do_populate_sysroot does not take
>>>>         anything from bindir.
>>>>         This leads to nativesdk-(qt)grpc's
>>>>         do_prepare_recipe_sysroot to not get the protoc binary into
>>>>         the recipe_sysroot as expected by CMake.
>>>>
>>>>         This can be fixed with SYSROOT_DIRS.
>>>>
>>>>         On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov
>>>>         <uvv.mail@gmail.com> wrote:
>>>>
>>>>             Hi Clément,
>>>>             No, the issue is when you just build nativesdk recipe
>>>>             (we are not at the stage to use the SDK yet).
>>>>
>>>>             And that was exactly my question how to propagate the
>>>>             dependency correctly for nativesdk build.
>>>>
>>>>             Regards,
>>>>             Slava
>>>>

| CMake Error at
/home/sapiippo/qt6/build/tmp/work/cortexa57-poky-linux/grpc/1.59.2/recipe-sysroot/usr/lib/cmake/protobuf/protobuf-targets.cmake:116
(message):
|   The imported target "protobuf::protoc" references the file
|
|
     "/home/sapiippo/qt6/build/tmp/work/cortexa57-poky-linux/grpc/1.59.2/recipe-sysroot/usr/bin/protoc-23.4.0"
|
|   but this file does not exist.  Possible reasons include:

On Fri, 10 Nov 2023 at 09:38, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:

> You sure? Because I remember the test in grpc to check for CROSS_COMPILING
> in cmake to avoid that.
>
> Slava
>
> On 10.11.2023 08:12, Samuli Piippo wrote:
>
> This is not strictly a nativesdk issue.
> If you enable PACKAGECONFIG = "compiler" for protobuf, it will then build
> protoc for the target.
> This will then cause grpc target build to fail in the same way as
> nativesdk build.
>
> On Fri, 10 Nov 2023 at 08:40, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
>
>> FTR, I suggested oe-core solution in the corresponding mailing list.
>>
>> Slava
>>
>> On 10.11.2023 07:19, Vyacheslav Yurkov via lists.openembedded.org wrote:
>>
>> On 10.11.2023 02:23, Clément Péron wrote:
>>
>> Hi Samuli,
>>
>> Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov <uvv.mail@gmail.com> a
>> écrit :
>>
>>> Thanks for the explanation, that was the missing piece for me to get an
>>> understanding of the problem :)
>>>
>>> I think it's not only me, but also others, were confused by the fact
>>> that we are talking about native(sdk), but try to fix something with a
>>> non-native variable.
>>>
>>> I've just gave it a try and have two suggestions how this can be
>>> addressed.
>>>
>>> 1. Recipe based solution:
>>> SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>>>
>>
>> Is it not just missing something like
>>
>> DEPENDS:append:class-nativesdk = " protobuf-native "
>>
>>  Regards
>>
>> Clement
>>
>>
>> This is already implicit by DEPENDS += "protobuf". Take a look at my
>> second suggestion. You can see that ${bindir} is staged for native recipes,
>> but not for nativesdk.
>>
>>
>>> 2. oe-core based solution:
>>> --- a/meta/classes-global/staging.bbclass
>>> +++ b/meta/classes-global/staging.bbclass
>>> @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>>>      ${localstatedir} \
>>>  "
>>>  SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
>>> +SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
>>>  SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
>>>  SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
>>>
>>> The second one will invalidate nativesdk sstate, but might cover not
>>> only grpc issue (I would assume gdbus-codegen or any other generator has
>>> the same issue when used for SDK build).
>>>
>>> What are your thoughs?
>>>
>>> Thanks,
>>> Slava
>>>
>>> On 09.11.2023 14:35, Samuli Piippo wrote:
>>>
>>> DEPENDS += "protobuf" is enough to have dependencies resolved.
>>>
>>> Problem is that bitbake's do_populate_sysroot does not take anything
>>> from bindir.
>>> This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to not get
>>> the protoc binary into the recipe_sysroot as expected by CMake.
>>>
>>> This can be fixed with SYSROOT_DIRS.
>>>
>>> On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov <uvv.mail@gmail.com>
>>> wrote:
>>>
>>>> Hi Clément,
>>>> No, the issue is when you just build nativesdk recipe (we are not at
>>>> the stage to use the SDK yet).
>>>>
>>>> And that was exactly my question how to propagate the dependency
>>>> correctly for nativesdk build.
>>>>
>>>> Regards,
>>>> Slava
>>>>
>>>
>

OK, but that seems to be similar issue, although a different one :)
I recall I disabled compiler by default for target, because it's rare 
use case.

So compiler can be enabled (if needed), but then perhaps another tweak 
with sysroot will be necessary in the recipe? I suggest to keep it as a 
comment in the recipe. What do you think?

Slava

On 10.11.2023 08:50, Samuli Piippo wrote:
>
> | CMake Error at 
> /home/sapiippo/qt6/build/tmp/work/cortexa57-poky-linux/grpc/1.59.2/recipe-sysroot/usr/lib/cmake/protobuf/protobuf-targets.cmake:116 
> (message):
> |   The imported target "protobuf::protoc" references the file
> |
> | 
>      "/home/sapiippo/qt6/build/tmp/work/cortexa57-poky-linux/grpc/1.59.2/recipe-sysroot/usr/bin/protoc-23.4.0"
> |
> |   but this file does not exist.  Possible reasons include:
>
> On Fri, 10 Nov 2023 at 09:38, Vyacheslav Yurkov <uvv.mail@gmail.com> 
> wrote:
>
>     You sure? Because I remember the test in grpc to check for
>     CROSS_COMPILING in cmake to avoid that.
>
>     Slava
>
>     On 10.11.2023 08:12, Samuli Piippo wrote:
>>     This is not strictly a nativesdk issue.
>>     If you enable PACKAGECONFIG = "compiler" for protobuf, it will
>>     then build protoc for the target.
>>     This will then cause grpc target build to fail in the same way as
>>     nativesdk build.
>>
>>     On Fri, 10 Nov 2023 at 08:40, Vyacheslav Yurkov
>>     <uvv.mail@gmail.com> wrote:
>>
>>         FTR, I suggested oe-core solution in the corresponding
>>         mailing list.
>>
>>         Slava
>>
>>         On 10.11.2023 07:19, Vyacheslav Yurkov via
>>         lists.openembedded.org <http://lists.openembedded.org> wrote:
>>>         On 10.11.2023 02:23, Clément Péron wrote:
>>>>         Hi Samuli,
>>>>
>>>>         Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov
>>>>         <uvv.mail@gmail.com> a écrit :
>>>>
>>>>             Thanks for the explanation, that was the missing piece
>>>>             for me to get an understanding of the problem :)
>>>>
>>>>             I think it's not only me, but also others, were
>>>>             confused by the fact that we are talking about
>>>>             native(sdk), but try to fix something with a non-native
>>>>             variable.
>>>>
>>>>             I've just gave it a try and have two suggestions how
>>>>             this can be addressed.
>>>>
>>>>             1. Recipe based solution:
>>>>             SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>>>>
>>>>
>>>>         Is it not just missing something like
>>>>
>>>>         DEPENDS:append:class-nativesdk= " protobuf-native "
>>>>         Regards
>>>>         Clement
>>>
>>>         This is already implicit by DEPENDS += "protobuf". Take a
>>>         look at my second suggestion. You can see that ${bindir} is
>>>         staged for native recipes, but not for nativesdk.
>>>
>>>>
>>>>             2. oe-core based solution:
>>>>             --- a/meta/classes-global/staging.bbclass
>>>>             +++ b/meta/classes-global/staging.bbclass
>>>>             @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>>>>                  ${localstatedir} \
>>>>              "
>>>>              SYSROOT_DIRS:append:class-native = "
>>>>             ${SYSROOT_DIRS_NATIVE}"
>>>>             +SYSROOT_DIRS:append:class-nativesdk = "
>>>>             ${SYSROOT_DIRS_NATIVE}"
>>>>              SYSROOT_DIRS:append:class-cross = "
>>>>             ${SYSROOT_DIRS_NATIVE}"
>>>>              SYSROOT_DIRS:append:class-crosssdk = "
>>>>             ${SYSROOT_DIRS_NATIVE}"
>>>>
>>>>             The second one will invalidate nativesdk sstate, but
>>>>             might cover not only grpc issue (I would assume
>>>>             gdbus-codegen or any other generator has the same issue
>>>>             when used for SDK build).
>>>>
>>>>             What are your thoughs?
>>>>
>>>>             Thanks,
>>>>             Slava
>>>>
>>>>             On 09.11.2023 14:35, Samuli Piippo wrote:
>>>>>             DEPENDS += "protobuf" is enough to have dependencies
>>>>>             resolved.
>>>>>
>>>>>             Problem is that bitbake's do_populate_sysroot does not
>>>>>             take anything from bindir.
>>>>>             This leads to nativesdk-(qt)grpc's
>>>>>             do_prepare_recipe_sysroot to not get the protoc binary
>>>>>             into the recipe_sysroot as expected by CMake.
>>>>>
>>>>>             This can be fixed with SYSROOT_DIRS.
>>>>>
>>>>>             On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov
>>>>>             <uvv.mail@gmail.com> wrote:
>>>>>
>>>>>                 Hi Clément,
>>>>>                 No, the issue is when you just build nativesdk
>>>>>                 recipe (we are not at the stage to use the SDK yet).
>>>>>
>>>>>                 And that was exactly my question how to propagate
>>>>>                 the dependency correctly for nativesdk build.
>>>>>
>>>>>                 Regards,
>>>>>                 Slava
>>>>>
>

Agreed that enabling compiler in target would be rarely used.

Having
SYSROOT_DIRS += "${bindir}"
would, however, cover both builds without additional tweaks.
Since target build doesn't have any other binaries, this wouldn't do
anything extra with default PACKAGECONFIG.

On Fri, 10 Nov 2023 at 10:06, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:

> OK, but that seems to be similar issue, although a different one :)
> I recall I disabled compiler by default for target, because it's rare use
> case.
>
> So compiler can be enabled (if needed), but then perhaps another tweak
> with sysroot will be necessary in the recipe? I suggest to keep it as a
> comment in the recipe. What do you think?
>
> Slava
>
> On 10.11.2023 08:50, Samuli Piippo wrote:
>
>
> | CMake Error at
> /home/sapiippo/qt6/build/tmp/work/cortexa57-poky-linux/grpc/1.59.2/recipe-sysroot/usr/lib/cmake/protobuf/protobuf-targets.cmake:116
> (message):
> |   The imported target "protobuf::protoc" references the file
> |
> |
>      "/home/sapiippo/qt6/build/tmp/work/cortexa57-poky-linux/grpc/1.59.2/recipe-sysroot/usr/bin/protoc-23.4.0"
> |
> |   but this file does not exist.  Possible reasons include:
>
> On Fri, 10 Nov 2023 at 09:38, Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
>
>> You sure? Because I remember the test in grpc to check for
>> CROSS_COMPILING in cmake to avoid that.
>>
>> Slava
>>
>> On 10.11.2023 08:12, Samuli Piippo wrote:
>>
>> This is not strictly a nativesdk issue.
>> If you enable PACKAGECONFIG = "compiler" for protobuf, it will then build
>> protoc for the target.
>> This will then cause grpc target build to fail in the same way as
>> nativesdk build.
>>
>> On Fri, 10 Nov 2023 at 08:40, Vyacheslav Yurkov <uvv.mail@gmail.com>
>> wrote:
>>
>>> FTR, I suggested oe-core solution in the corresponding mailing list.
>>>
>>> Slava
>>>
>>> On 10.11.2023 07:19, Vyacheslav Yurkov via lists.openembedded.org wrote:
>>>
>>> On 10.11.2023 02:23, Clément Péron wrote:
>>>
>>> Hi Samuli,
>>>
>>> Le ven. 10 nov. 2023, 00:05, Vyacheslav Yurkov <uvv.mail@gmail.com> a
>>> écrit :
>>>
>>>> Thanks for the explanation, that was the missing piece for me to get an
>>>> understanding of the problem :)
>>>>
>>>> I think it's not only me, but also others, were confused by the fact
>>>> that we are talking about native(sdk), but try to fix something with a
>>>> non-native variable.
>>>>
>>>> I've just gave it a try and have two suggestions how this can be
>>>> addressed.
>>>>
>>>> 1. Recipe based solution:
>>>> SYSROOT_DIRS:append:class-nativesdk = " $bindir}"
>>>>
>>>
>>> Is it not just missing something like
>>>
>>> DEPENDS:append:class-nativesdk = " protobuf-native "
>>>
>>>  Regards
>>>
>>> Clement
>>>
>>>
>>> This is already implicit by DEPENDS += "protobuf". Take a look at my
>>> second suggestion. You can see that ${bindir} is staged for native recipes,
>>> but not for nativesdk.
>>>
>>>
>>>> 2. oe-core based solution:
>>>> --- a/meta/classes-global/staging.bbclass
>>>> +++ b/meta/classes-global/staging.bbclass
>>>> @@ -26,6 +26,7 @@ SYSROOT_DIRS_NATIVE = " \
>>>>      ${localstatedir} \
>>>>  "
>>>>  SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
>>>> +SYSROOT_DIRS:append:class-nativesdk = " ${SYSROOT_DIRS_NATIVE}"
>>>>  SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
>>>>  SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
>>>>
>>>> The second one will invalidate nativesdk sstate, but might cover not
>>>> only grpc issue (I would assume gdbus-codegen or any other generator has
>>>> the same issue when used for SDK build).
>>>>
>>>> What are your thoughs?
>>>>
>>>> Thanks,
>>>> Slava
>>>>
>>>> On 09.11.2023 14:35, Samuli Piippo wrote:
>>>>
>>>> DEPENDS += "protobuf" is enough to have dependencies resolved.
>>>>
>>>> Problem is that bitbake's do_populate_sysroot does not take anything
>>>> from bindir.
>>>> This leads to nativesdk-(qt)grpc's do_prepare_recipe_sysroot to not get
>>>> the protoc binary into the recipe_sysroot as expected by CMake.
>>>>
>>>> This can be fixed with SYSROOT_DIRS.
>>>>
>>>> On Thu, 9 Nov 2023 at 12:19, Vyacheslav Yurkov <uvv.mail@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Clément,
>>>>> No, the issue is when you just build nativesdk recipe (we are not at
>>>>> the stage to use the SDK yet).
>>>>>
>>>>> And that was exactly my question how to propagate the dependency
>>>>> correctly for nativesdk build.
>>>>>
>>>>> Regards,
>>>>> Slava
>>>>>
>>>>
>>
>

Shall we go for round 3 of the patch then? :)

I'd wait for Clément feedback and then all 3 of us can sign it off.

I would also like to have a short comment/explanation before the 
SYSROOT_DIRS as to why it's required in this particular recipe.

FTR, as expected Richard didn't like the other suggestion, because the 
size of sstate is affected by that.

Slava

On 10.11.2023 09:44, Samuli Piippo wrote:
> Agreed that enabling compiler in target would be rarely used.
>
> Having
> SYSROOT_DIRS += "${bindir}"
> would, however, cover both builds without additional tweaks.
> Since target build doesn't have any other binaries, this wouldn't do 
> anything extra with default PACKAGECONFIG.
>
> On Fri, 10 Nov 2023 at 10:06, Vyacheslav Yurkov <uvv.mail@gmail.com> 
> wrote:
>
>     OK, but that seems to be similar issue, although a different one :)
>     I recall I disabled compiler by default for target, because it's
>     rare use case.
>
>     So compiler can be enabled (if needed), but then perhaps another
>     tweak with sysroot will be necessary in the recipe? I suggest to
>     keep it as a comment in the recipe. What do you think?
>
>     Slava
>