| Message ID | 20231120144628.52664-3-adrian.freihofer@siemens.com |
|---|---|
| State | New |
| Headers | show |
| Series | cmake-qemu | expand |
Hi Adrian, Adrian Freihofer <adrian.freihofer@gmail.com> escreveu no dia segunda, 20/11/2023 à(s) 14:46: > Provide an easy way to enable cmake-qemu e.g. on distro level for all > recipes inheriting the cmake.bbclass. > > Therefore a new variable is introduced: OECMAKE_EXEWRAPPER_QEMU. > > If set to "1" all recipes built with CMake can use qemu to execute > cross-compiled binaries on the build machine. > One use case is the execution of cross-compiled unit tests or ptests, > e.g. in an SDK context. Therefore, a variable that activates Qemu for > all CMake recipes is much more useful than just providing the optional > cmake-qemu.bbclass. > Further information can be found in the CMake documentation in the > CMAKE_CROSSCOMPILING_EMULATOR section. > > There are some use cases for qemu-user, but users should not be > encouraged to use it as an alternative for clean cross-compilation. > Since the core does not need it either it is optional. > Are you sure this works in the majority of scenarios? Last time I worked on this qemu user mode failed to run multithreading applications. Jose > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> > --- > meta/classes-recipe/cmake.bbclass | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/meta/classes-recipe/cmake.bbclass > b/meta/classes-recipe/cmake.bbclass > index d978b88944..4d8e2a206f 100644 > --- a/meta/classes-recipe/cmake.bbclass > +++ b/meta/classes-recipe/cmake.bbclass > @@ -14,6 +14,10 @@ B = "${WORKDIR}/build" > # The supported options are "Unix Makefiles" or "Ninja". > OECMAKE_GENERATOR ?= "Ninja" > > +# qemu-user can be used for special use cases, but not by default > +OECMAKE_EXEWRAPPER_QEMU ??= "0" > +inherit ${@'cmake-qemu' if d.getVar('OECMAKE_EXEWRAPPER_QEMU') else ''} > + > python() { > generator = d.getVar("OECMAKE_GENERATOR") > if "Unix Makefiles" in generator: > -- > 2.41.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#190880): > https://lists.openembedded.org/g/openembedded-core/message/190880 > Mute This Topic: https://lists.openembedded.org/mt/102708283/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Jose Quaresma via lists.openembedded.org <quaresma.jose= gmail.com@lists.openembedded.org> escreveu no dia segunda, 20/11/2023 à(s) 17:17: > Hi Adrian, > > Adrian Freihofer <adrian.freihofer@gmail.com> escreveu no dia segunda, > 20/11/2023 à(s) 14:46: > >> Provide an easy way to enable cmake-qemu e.g. on distro level for all >> recipes inheriting the cmake.bbclass. >> >> Therefore a new variable is introduced: OECMAKE_EXEWRAPPER_QEMU. >> >> If set to "1" all recipes built with CMake can use qemu to execute >> cross-compiled binaries on the build machine. >> One use case is the execution of cross-compiled unit tests or ptests, >> e.g. in an SDK context. Therefore, a variable that activates Qemu for >> all CMake recipes is much more useful than just providing the optional >> cmake-qemu.bbclass. >> Further information can be found in the CMake documentation in the >> CMAKE_CROSSCOMPILING_EMULATOR section. >> >> There are some use cases for qemu-user, but users should not be >> encouraged to use it as an alternative for clean cross-compilation. >> Since the core does not need it either it is optional. >> > > Are you sure this works in the majority of scenarios? > Last time I worked on this qemu user mode failed to run > multithreading applications. > Sorry, I was wrong above. The issue was not multithreading but with the fork system call. > > Jose > > >> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> >> --- >> meta/classes-recipe/cmake.bbclass | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/meta/classes-recipe/cmake.bbclass >> b/meta/classes-recipe/cmake.bbclass >> index d978b88944..4d8e2a206f 100644 >> --- a/meta/classes-recipe/cmake.bbclass >> +++ b/meta/classes-recipe/cmake.bbclass >> @@ -14,6 +14,10 @@ B = "${WORKDIR}/build" >> # The supported options are "Unix Makefiles" or "Ninja". >> OECMAKE_GENERATOR ?= "Ninja" >> >> +# qemu-user can be used for special use cases, but not by default >> +OECMAKE_EXEWRAPPER_QEMU ??= "0" >> +inherit ${@'cmake-qemu' if d.getVar('OECMAKE_EXEWRAPPER_QEMU') else ''} >> + >> python() { >> generator = d.getVar("OECMAKE_GENERATOR") >> if "Unix Makefiles" in generator: >> -- >> 2.41.0 >> >> >> >> >> > > -- > Best regards, > > José Quaresma > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#190925): > https://lists.openembedded.org/g/openembedded-core/message/190925 > Mute This Topic: https://lists.openembedded.org/mt/102708283/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
> > > > Are you sure this works in the majority of scenarios? > > Last time I worked on this qemu user mode failed to run > > multithreading applications. > > > > > Sorry, I was wrong above. > The issue was not multithreading but with the fork system call. > Hi Jose I was using it for running Unit Tests on the host before I deployed them to the target device as ptest. This worked well. It is well integrated with cmake and therefore also with IDEs. So I thought that there are certain use cases where it would be nice to have it opt-in. meson offers qemu-user as well (because of gobject). So if qemu-user is not usable there will be other challenges as well. There is also a MACHINE_FEATURE which allows to support it for ARCHs where it works well but not for ARCHs where qemu-user is known to be broken. I think it would be a nice optional feature. Regards, Adrian
<adrian.freihofer@gmail.com> escreveu no dia segunda, 20/11/2023 à(s) 18:41: > > > > > > Are you sure this works in the majority of scenarios? > > > Last time I worked on this qemu user mode failed to run > > > multithreading applications. > > > > > > > > > Sorry, I was wrong above. > > The issue was not multithreading but with the fork system call. > > > > Hi Jose > > I was using it for running Unit Tests on the host before I deployed > them to the target device as ptest. This worked well. It is well > integrated with cmake and therefore also with IDEs. So I thought that > there are certain use cases where it would be nice to have it opt-in. > > meson offers qemu-user as well (because of gobject). So if qemu-user is > not usable there will be other challenges as well. > > There is also a MACHINE_FEATURE which allows to support it for ARCHs > where it works well but not for ARCHs where qemu-user is known to be > broken. > > I think it would be a nice optional feature. > I agree without any doubt on that part. But I believe this doesn't work for applications that are using the fork system call. So consider this will also break any ptest using fork, giving the wrong impression the test is failing. Here is my attempt to add support in meson to run the ptest at build time. https://lists.openembedded.org/g/openembedded-core/topic/89289917 Jose > > Regards, > Adrian >
On Tue, 21 Nov 2023 at 12:12, Jose Quaresma <quaresma.jose@gmail.com> wrote: >> I was using it for running Unit Tests on the host before I deployed >> them to the target device as ptest. This worked well. It is well >> integrated with cmake and therefore also with IDEs. So I thought that >> there are certain use cases where it would be nice to have it opt-in. >> >> meson offers qemu-user as well (because of gobject). So if qemu-user is >> not usable there will be other challenges as well. >> >> There is also a MACHINE_FEATURE which allows to support it for ARCHs >> where it works well but not for ARCHs where qemu-user is known to be >> broken. >> >> I think it would be a nice optional feature. > > > I agree without any doubt on that part. > > But I believe this doesn't work for applications that are using the fork system call. > So consider this will also break any ptest using fork, giving the wrong impression the test is failing. I think this needs to be double checked. I find it odd that qemu usermode would fail on such a basic thing, and it would not show up anywhere in places where it's used in core (not just g-i - all the various postinst utilities for generating indexes etc.). Alex
Alexander Kanavin <alex.kanavin@gmail.com> escreveu no dia terça, 21/11/2023 à(s) 11:21: > On Tue, 21 Nov 2023 at 12:12, Jose Quaresma <quaresma.jose@gmail.com> > wrote: > >> I was using it for running Unit Tests on the host before I deployed > >> them to the target device as ptest. This worked well. It is well > >> integrated with cmake and therefore also with IDEs. So I thought that > >> there are certain use cases where it would be nice to have it opt-in. > >> > >> meson offers qemu-user as well (because of gobject). So if qemu-user is > >> not usable there will be other challenges as well. > >> > >> There is also a MACHINE_FEATURE which allows to support it for ARCHs > >> where it works well but not for ARCHs where qemu-user is known to be > >> broken. > >> > >> I think it would be a nice optional feature. > > > > > > I agree without any doubt on that part. > > > > But I believe this doesn't work for applications that are using the fork > system call. > > So consider this will also break any ptest using fork, giving the wrong > impression the test is failing. > > I think this needs to be double checked. I find it odd that qemu > usermode would fail on such a basic thing, and it would not show up > anywhere in places where it's used in core (not just g-i - all the > various postinst utilities for generating indexes etc.). > > Alex > There is no recent information on the internet about the fork issue on qemu user mode so this may no longer be a problem. I can back again, not now, testing some of the oe-core packages who uses the meson build system to get some results about failing/passing. The approach is the same used in this patchset. It will be good also to have some numbers and example recipes in oe-core where we can use this patchset. In conclusion, this is really a great future imo and improves a lot the integrations of new tests in the ptest infrastructure. Jose
On Mon, 2023-11-20 at 15:46 +0100, Adrian Freihofer wrote: > Provide an easy way to enable cmake-qemu e.g. on distro level for all > recipes inheriting the cmake.bbclass. > > Therefore a new variable is introduced: OECMAKE_EXEWRAPPER_QEMU. > > If set to "1" all recipes built with CMake can use qemu to execute > cross-compiled binaries on the build machine. > One use case is the execution of cross-compiled unit tests or ptests, > e.g. in an SDK context. Therefore, a variable that activates Qemu for > all CMake recipes is much more useful than just providing the optional > cmake-qemu.bbclass. > Further information can be found in the CMake documentation in the > CMAKE_CROSSCOMPILING_EMULATOR section. > > There are some use cases for qemu-user, but users should not be > encouraged to use it as an alternative for clean cross-compilation. > Since the core does not need it either it is optional. > > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> > --- > meta/classes-recipe/cmake.bbclass | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/meta/classes-recipe/cmake.bbclass b/meta/classes-recipe/cmake.bbclass > index d978b88944..4d8e2a206f 100644 > --- a/meta/classes-recipe/cmake.bbclass > +++ b/meta/classes-recipe/cmake.bbclass > @@ -14,6 +14,10 @@ B = "${WORKDIR}/build" > # The supported options are "Unix Makefiles" or "Ninja". > OECMAKE_GENERATOR ?= "Ninja" > > +# qemu-user can be used for special use cases, but not by default > +OECMAKE_EXEWRAPPER_QEMU ??= "0" > +inherit ${@'cmake-qemu' if d.getVar('OECMAKE_EXEWRAPPER_QEMU') else ''} > + I don't really see why this is needed. If a recipe is using things which need this, wouldn't they just inherit cmake-qemu? You could make cmake-qemu inherit cmake? Cheers, Richard
On Tue, 2023-11-21 at 12:21 +0100, Alexander Kanavin wrote: > On Tue, 21 Nov 2023 at 12:12, Jose Quaresma <quaresma.jose@gmail.com> wrote: > > > I was using it for running Unit Tests on the host before I deployed > > > them to the target device as ptest. This worked well. It is well > > > integrated with cmake and therefore also with IDEs. So I thought that > > > there are certain use cases where it would be nice to have it opt-in. > > > > > > meson offers qemu-user as well (because of gobject). So if qemu-user is > > > not usable there will be other challenges as well. > > > > > > There is also a MACHINE_FEATURE which allows to support it for ARCHs > > > where it works well but not for ARCHs where qemu-user is known to be > > > broken. > > > > > > I think it would be a nice optional feature. > > > > > > I agree without any doubt on that part. > > > > But I believe this doesn't work for applications that are using the fork system call. > > So consider this will also break any ptest using fork, giving the wrong impression the test is failing. > > I think this needs to be double checked. I find it odd that qemu > usermode would fail on such a basic thing, and it would not show up > anywhere in places where it's used in core (not just g-i - all the > various postinst utilities for generating indexes etc.). It is more that the exec() calls are problematic so it depends what the code is doing after the fork(). exec*() can be made to work under usermode emulation if you setup binfmt_misc and enable it properly. This can be setup and gets into the territory of what scratchbox tried to do for example but it does usually need more permissions that our build user would typically have. We have no desire to need that level of system access. We do also want to maintain the ability to bring up/build platforms that do not have qemu user mode support. Cheers, Richard
diff --git a/meta/classes-recipe/cmake.bbclass b/meta/classes-recipe/cmake.bbclass index d978b88944..4d8e2a206f 100644 --- a/meta/classes-recipe/cmake.bbclass +++ b/meta/classes-recipe/cmake.bbclass @@ -14,6 +14,10 @@ B = "${WORKDIR}/build" # The supported options are "Unix Makefiles" or "Ninja". OECMAKE_GENERATOR ?= "Ninja" +# qemu-user can be used for special use cases, but not by default +OECMAKE_EXEWRAPPER_QEMU ??= "0" +inherit ${@'cmake-qemu' if d.getVar('OECMAKE_EXEWRAPPER_QEMU') else ''} + python() { generator = d.getVar("OECMAKE_GENERATOR") if "Unix Makefiles" in generator:
Provide an easy way to enable cmake-qemu e.g. on distro level for all recipes inheriting the cmake.bbclass. Therefore a new variable is introduced: OECMAKE_EXEWRAPPER_QEMU. If set to "1" all recipes built with CMake can use qemu to execute cross-compiled binaries on the build machine. One use case is the execution of cross-compiled unit tests or ptests, e.g. in an SDK context. Therefore, a variable that activates Qemu for all CMake recipes is much more useful than just providing the optional cmake-qemu.bbclass. Further information can be found in the CMake documentation in the CMAKE_CROSSCOMPILING_EMULATOR section. There are some use cases for qemu-user, but users should not be encouraged to use it as an alternative for clean cross-compilation. Since the core does not need it either it is optional. Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> --- meta/classes-recipe/cmake.bbclass | 4 ++++ 1 file changed, 4 insertions(+)