| Message ID | 20220304115525.2240283-1-ross.burton@arm.com |
|---|---|
| State | Accepted, archived |
| Commit | 341d2b35986e48e4954c591be8bc037a5557452a |
| Headers | show |
| Series | classes: add setuptools3_legacy | expand |
For me that would be a candidate to be added to meta-python, or are there any core recipes that currently break because of the new behavior? Furthermore the variables names are the same, so if people accidentally (or due to complex injection trees/classes/distro settings/whatever have new setuptools3 and this class here in the same recipe, things become rather unpredictable - as of now there are the same but surely they will start to diverge at one point. While we are at it pypi class needs a fallback too, because for me this is the standard way of packaging python recipes and new class points to the new setuptools implementation - so if we want to have this here, we might need a fallback for pypi class as well On 04.03.22 12:55, Ross Burton wrote: > Following a good discussion with PyPA upstream[1] the migration of the > setuptools3.bbclass to use bdist_wheel+pip turns out to be more complex > than thought. > > Essentially, we're midway through a lot of changes: the future of Python > packaging is wheels and pip, but those by design are not as flexible as > traditional distutils and setup.py. > > Specifically, with traditional distutils the package can implement its > own install task and write arbitrary files (such as init scripts). With > wheels this is explicity impossible, so packages that do this cannot use > the new setuptools class and must continue to use the build/install tasks > as before. > > This class is the old setuptools behaviour, bought back. However, as > distutils and the setuptools install task are both deprecated and will > soon be removed entirely, any users of this class should be moving to an > alternative build tool, be it a modern Python tool which works with > wheels, or a non-Pythonic tool such as Meson. > > [1] https://github.com/pypa/packaging-problems/issues/576 > > Signed-off-by: Ross Burton <ross.burton@arm.com> > --- > meta/classes/setuptools3_legacy.bbclass | 78 +++++++++++++++++++++++++ > 1 file changed, 78 insertions(+) > create mode 100644 meta/classes/setuptools3_legacy.bbclass > > diff --git a/meta/classes/setuptools3_legacy.bbclass b/meta/classes/setuptools3_legacy.bbclass > new file mode 100644 > index 0000000000..5a99daadb5 > --- /dev/null > +++ b/meta/classes/setuptools3_legacy.bbclass > @@ -0,0 +1,78 @@ > +# This class is for packages which use the deprecated setuptools behaviour, > +# specifically custom install tasks which don't work correctly with bdist_wheel. > +# This behaviour is deprecated in setuptools[1] and won't work in the future, so > +# all users of this should consider their options: pure Python modules can use a > +# modern Python tool such as build[2], or packages which are doing more (such as > +# installing init scripts) should use a fully-featured build system such as Meson. > +# > +# [1] https://setuptools.pypa.io/en/latest/history.html#id142 > +# [2] https://pypi.org/project/build/ > + > +inherit setuptools3-base > + > +B = "${WORKDIR}/build" > + > +SETUPTOOLS_BUILD_ARGS ?= "" > +SETUPTOOLS_INSTALL_ARGS ?= "--root=${D} \ > + --prefix=${prefix} \ > + --install-lib=${PYTHON_SITEPACKAGES_DIR} \ > + --install-data=${datadir}" > + > +SETUPTOOLS_PYTHON = "python3" > +SETUPTOOLS_PYTHON:class-native = "nativepython3" > + > +SETUPTOOLS_SETUP_PATH ?= "${S}" > + > +setuptools3_legacy_do_configure() { > + : > +} > + > +setuptools3_legacy_do_compile() { > + cd ${SETUPTOOLS_SETUP_PATH} > + NO_FETCH_BUILD=1 \ > + STAGING_INCDIR=${STAGING_INCDIR} \ > + STAGING_LIBDIR=${STAGING_LIBDIR} \ > + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ > + build --build-base=${B} ${SETUPTOOLS_BUILD_ARGS} || \ > + bbfatal_log "'${PYTHON_PN} setup.py build ${SETUPTOOLS_BUILD_ARGS}' execution failed." > +} > +setuptools3_legacy_do_compile[vardepsexclude] = "MACHINE" > + > +setuptools3_legacy_do_install() { > + cd ${SETUPTOOLS_SETUP_PATH} > + install -d ${D}${PYTHON_SITEPACKAGES_DIR} > + STAGING_INCDIR=${STAGING_INCDIR} \ > + STAGING_LIBDIR=${STAGING_LIBDIR} \ > + PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \ > + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ > + build --build-base=${B} install --skip-build ${SETUPTOOLS_INSTALL_ARGS} || \ > + bbfatal_log "'${PYTHON_PN} setup.py install ${SETUPTOOLS_INSTALL_ARGS}' execution failed." > + > + # support filenames with *spaces* > + find ${D} -name "*.py" -exec grep -q ${D} {} \; \ > + -exec sed -i -e s:${D}::g {} \; > + > + for i in ${D}${bindir}/* ${D}${sbindir}/*; do > + if [ -f "$i" ]; then > + sed -i -e s:${PYTHON}:${USRBINPATH}/env\ ${SETUPTOOLS_PYTHON}:g $i > + sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i > + fi > + done > + > + rm -f ${D}${PYTHON_SITEPACKAGES_DIR}/easy-install.pth > + > + # > + # FIXME: Bandaid against wrong datadir computation > + # > + if [ -e ${D}${datadir}/share ]; then > + mv -f ${D}${datadir}/share/* ${D}${datadir}/ > + rmdir ${D}${datadir}/share > + fi > +} > +setuptools3_legacy_do_install[vardepsexclude] = "MACHINE" > + > +EXPORT_FUNCTIONS do_configure do_compile do_install > + > +export LDSHARED="${CCLD} -shared" > +DEPENDS += "python3-setuptools-native" > + > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#162715): https://lists.openembedded.org/g/openembedded-core/message/162715 > Mute This Topic: https://lists.openembedded.org/mt/89547022/3647476 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Fri, 4 Mar 2022 at 12:44, Konrad Weihmann <kweihmann@outlook.com> wrote: > For me that would be a candidate to be added to meta-python, or are > there any core recipes that currently break because of the new behavior? This is restoring existing behaviour, so I believe it should be in core. There's going to be enough recipes using this that it is justified in core. I expect we'll have deleted this by the time the next LTS comes around too. > Furthermore the variables names are the same, so if people accidentally > (or due to complex injection trees/classes/distro settings/whatever have > new setuptools3 and this class here in the same recipe, things become > rather unpredictable - as of now there are the same but surely they will > start to diverge at one point. That's intentional: if you have a recipe which needs the legacy behaviour, just change the import from setuptool3 to setuptools3_legacy. > While we are at it pypi class needs a fallback too, because for me this > is the standard way of packaging python recipes and new class points to > the new setuptools implementation - so if we want to have this here, we > might need a fallback for pypi class as well My understanding is that almost anything on pypi is already a pure Python module and is mostly unaffected by the changes. Ross
On 04.03.22 14:13, Ross Burton wrote: > On Fri, 4 Mar 2022 at 12:44, Konrad Weihmann <kweihmann@outlook.com> wrote: >> For me that would be a candidate to be added to meta-python, or are >> there any core recipes that currently break because of the new behavior? > > This is restoring existing behaviour, so I believe it should be in > core. There's going to be enough recipes using this that it is > justified in core. I expect we'll have deleted this by the time the > next LTS comes around too. > >> Furthermore the variables names are the same, so if people accidentally >> (or due to complex injection trees/classes/distro settings/whatever have >> new setuptools3 and this class here in the same recipe, things become >> rather unpredictable - as of now there are the same but surely they will >> start to diverge at one point. > > That's intentional: if you have a recipe which needs the legacy > behaviour, just change the import from setuptool3 to > setuptools3_legacy. Can you name any publicly available recipe that requires that? For me this is the same situation as with the python2 > python3 transition - there wasn't any fallback provided in core, but outside of core it was - so I would propose to do the same here. If that legacy class will be part of core it's likely that it will remain till EOL of kirkstone which is somewhat in ~2025 (by current planning) - and meta-python2 showed that even then some recipes still haven't migrated (yes, chromium looking at you). All in all I think core should just provide *one* way to do it while the legacy cases can easily live outside in a separate layer (meta-python for instance as the "old" distutils have been put there as well). In additional everyone was worried so much about reputation lately that doing this long announced transition, while keeping the "old" way enabled (and creating more maintenance effort in the end), would definitely send the wrong signal to the community. > >> While we are at it pypi class needs a fallback too, because for me this >> is the standard way of packaging python recipes and new class points to >> the new setuptools implementation - so if we want to have this here, we >> might need a fallback for pypi class as well > > My understanding is that almost anything on pypi is already a pure > Python module and is mostly unaffected by the changes. > > Ross
On Fri, Mar 4, 2022 at 6:06 AM Konrad Weihmann <kweihmann@outlook.com> wrote: > > > > On 04.03.22 14:13, Ross Burton wrote: > > On Fri, 4 Mar 2022 at 12:44, Konrad Weihmann <kweihmann@outlook.com> wrote: > >> For me that would be a candidate to be added to meta-python, or are > >> there any core recipes that currently break because of the new behavior? > > > > This is restoring existing behaviour, so I believe it should be in > > core. There's going to be enough recipes using this that it is > > justified in core. I expect we'll have deleted this by the time the > > next LTS comes around too. > > > >> Furthermore the variables names are the same, so if people accidentally > >> (or due to complex injection trees/classes/distro settings/whatever have > >> new setuptools3 and this class here in the same recipe, things become > >> rather unpredictable - as of now there are the same but surely they will > >> start to diverge at one point. > > > > That's intentional: if you have a recipe which needs the legacy > > behaviour, just change the import from setuptool3 to > > setuptools3_legacy. > > Can you name any publicly available recipe that requires that? > > For me this is the same situation as with the python2 > python3 > transition - there wasn't any fallback provided in core, but outside of > core it was - so I would propose to do the same here. > > If that legacy class will be part of core it's likely that it will > remain till EOL of kirkstone which is somewhat in ~2025 (by current > planning) - and meta-python2 showed that even then some recipes still > haven't migrated (yes, chromium looking at you). latest chromium does not need py2 in OE context anymore for a year or so now. > All in all I think core should just provide *one* way to do it while the > legacy cases can easily live outside in a separate layer (meta-python > for instance as the "old" distutils have been put there as well). > There are recipes in meta-oe and meta-python which would need this and both layer only list core as sole dependency, if we put it in either of these layers, then one will depend on other which is not nice, so I rather would prefer it in core. , regarding distutils classes we did move them to meta-python but there are recipes in meta-oe needing them e.g. unattended-upgrades which now are forced to be moved to meta-python which is less than ideal but can be lived with. seuptool-legacy is similar to autotools-brokensep in spirit. python2 issue was of a different nature > In additional everyone was worried so much about reputation lately that > doing this long announced transition, while keeping the "old" way > enabled (and creating more maintenance effort in the end), would > definitely send the wrong signal to the community. > > > > >> While we are at it pypi class needs a fallback too, because for me this > >> is the standard way of packaging python recipes and new class points to > >> the new setuptools implementation - so if we want to have this here, we > >> might need a fallback for pypi class as well > > > > My understanding is that almost anything on pypi is already a pure > > Python module and is mostly unaffected by the changes. > > > > Ross > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#162718): https://lists.openembedded.org/g/openembedded-core/message/162718 > Mute This Topic: https://lists.openembedded.org/mt/89547022/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Em sex., 4 de mar. de 2022 às 13:56, Khem Raj <raj.khem@gmail.com> escreveu: > There are recipes in meta-oe and meta-python which would need this > and both layer only list core as sole dependency, if we put it in either of > these layers, then one will depend on other which is not nice, so I rather > would prefer it in core. Couldn't those recipes (depending on Python) go to meta-python?
On Fri, Mar 4, 2022 at 10:01 AM Otavio Salvador <otavio.salvador@ossystems.com.br> wrote: > > > > Em sex., 4 de mar. de 2022 às 13:56, Khem Raj <raj.khem@gmail.com> escreveu: >> >> There are recipes in meta-oe and meta-python which would need this >> and both layer only list core as sole dependency, if we put it in either of >> these layers, then one will depend on other which is not nice, so I rather >> would prefer it in core. > > > Couldn't those recipes (depending on Python) go to meta-python? not without making meta-python as core dependency for meta-oe, its better to keep them independent. > > -- > Otavio Salvador O.S. Systems > http://www.ossystems.com.br http://code.ossystems.com.br > Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750
On Fri, Mar 4, 2022 at 11:20 AM Konrad Weihmann <kweihmann@outlook.com> wrote: > > If it's about meta-python why can't be placed there? I don't follow the argument that some edge case stuff outside of core requires the project to have additional (as of now not time limited) maintenance effort - if there would be a clear plan to remove that before kirkstone release I might agree, but with the release looming this all feels like abandoning the commitment for the upcoming release. it is about many layers e.g. meta-oe, meta-xfce, meta-python and perhaps many more that I have not come across yet. We want to enable a wider ecosystem and we do carry features in OE-core to do so from time to time. Core in itself is a reference distribution, its not meant to carry just what it needs to support a minimum distro it serves as a baseline for many distros which may include other layers. At times we do put specific features in specific layers, this is not one of those. > > And just to emphasize my claim, if there's nothing in core that would require this patch it won't get tested, so it's another dead end. > > In the end it's the decision of the project leads, but I would be personally be disappointed to take the easy way out here. > > On 4 Mar 2022 19:01, Otavio Salvador <otavio.salvador@ossystems.com.br> wrote: > > > > Em sex., 4 de mar. de 2022 às 13:56, Khem Raj <raj.khem@gmail.com> escreveu: > > There are recipes in meta-oe and meta-python which would need this > and both layer only list core as sole dependency, if we put it in either of > these layers, then one will depend on other which is not nice, so I rather > would prefer it in core. > > > Couldn't those recipes (depending on Python) go to meta-python? > > -- > Otavio Salvador O.S. Systems > http://www.ossystems.com.br http://code.ossystems.com.br > Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750 > >
On Fri, Mar 4, 2022 at 3:14 PM Khem Raj <raj.khem@gmail.com> wrote: > > On Fri, Mar 4, 2022 at 11:20 AM Konrad Weihmann <kweihmann@outlook.com> wrote: > > > > If it's about meta-python why can't be placed there? I don't follow the argument that some edge case stuff outside of core requires the project to have additional (as of now not time limited) maintenance effort - if there would be a clear plan to remove that before kirkstone release I might agree, but with the release looming this all feels like abandoning the commitment for the upcoming release. > > it is about many layers e.g. meta-oe, meta-xfce, meta-python and > perhaps many more that I have not come across yet. We want to enable a > wider ecosystem and we > do carry features in OE-core to do so from time to time. Core in > itself is a reference distribution, its not meant to carry just what > it needs to support a minimum distro > it serves as a baseline for many distros which may include other > layers. At times we do put specific features in specific layers, this > is not one of those. Agreed. I'd prefer that this be in core, versus meta-python. Bruce > > > > > And just to emphasize my claim, if there's nothing in core that would require this patch it won't get tested, so it's another dead end. > > > > In the end it's the decision of the project leads, but I would be personally be disappointed to take the easy way out here. > > > > On 4 Mar 2022 19:01, Otavio Salvador <otavio.salvador@ossystems.com.br> wrote: > > > > > > > > Em sex., 4 de mar. de 2022 às 13:56, Khem Raj <raj.khem@gmail.com> escreveu: > > > > There are recipes in meta-oe and meta-python which would need this > > and both layer only list core as sole dependency, if we put it in either of > > these layers, then one will depend on other which is not nice, so I rather > > would prefer it in core. > > > > > > Couldn't those recipes (depending on Python) go to meta-python? > > > > -- > > Otavio Salvador O.S. Systems > > http://www.ossystems.com.br http://code.ossystems.com.br > > Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750 > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#162756): https://lists.openembedded.org/g/openembedded-core/message/162756 > Mute This Topic: https://lists.openembedded.org/mt/89547022/1050810 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/meta/classes/setuptools3_legacy.bbclass b/meta/classes/setuptools3_legacy.bbclass new file mode 100644 index 0000000000..5a99daadb5 --- /dev/null +++ b/meta/classes/setuptools3_legacy.bbclass @@ -0,0 +1,78 @@ +# This class is for packages which use the deprecated setuptools behaviour, +# specifically custom install tasks which don't work correctly with bdist_wheel. +# This behaviour is deprecated in setuptools[1] and won't work in the future, so +# all users of this should consider their options: pure Python modules can use a +# modern Python tool such as build[2], or packages which are doing more (such as +# installing init scripts) should use a fully-featured build system such as Meson. +# +# [1] https://setuptools.pypa.io/en/latest/history.html#id142 +# [2] https://pypi.org/project/build/ + +inherit setuptools3-base + +B = "${WORKDIR}/build" + +SETUPTOOLS_BUILD_ARGS ?= "" +SETUPTOOLS_INSTALL_ARGS ?= "--root=${D} \ + --prefix=${prefix} \ + --install-lib=${PYTHON_SITEPACKAGES_DIR} \ + --install-data=${datadir}" + +SETUPTOOLS_PYTHON = "python3" +SETUPTOOLS_PYTHON:class-native = "nativepython3" + +SETUPTOOLS_SETUP_PATH ?= "${S}" + +setuptools3_legacy_do_configure() { + : +} + +setuptools3_legacy_do_compile() { + cd ${SETUPTOOLS_SETUP_PATH} + NO_FETCH_BUILD=1 \ + STAGING_INCDIR=${STAGING_INCDIR} \ + STAGING_LIBDIR=${STAGING_LIBDIR} \ + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ + build --build-base=${B} ${SETUPTOOLS_BUILD_ARGS} || \ + bbfatal_log "'${PYTHON_PN} setup.py build ${SETUPTOOLS_BUILD_ARGS}' execution failed." +} +setuptools3_legacy_do_compile[vardepsexclude] = "MACHINE" + +setuptools3_legacy_do_install() { + cd ${SETUPTOOLS_SETUP_PATH} + install -d ${D}${PYTHON_SITEPACKAGES_DIR} + STAGING_INCDIR=${STAGING_INCDIR} \ + STAGING_LIBDIR=${STAGING_LIBDIR} \ + PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \ + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ + build --build-base=${B} install --skip-build ${SETUPTOOLS_INSTALL_ARGS} || \ + bbfatal_log "'${PYTHON_PN} setup.py install ${SETUPTOOLS_INSTALL_ARGS}' execution failed." + + # support filenames with *spaces* + find ${D} -name "*.py" -exec grep -q ${D} {} \; \ + -exec sed -i -e s:${D}::g {} \; + + for i in ${D}${bindir}/* ${D}${sbindir}/*; do + if [ -f "$i" ]; then + sed -i -e s:${PYTHON}:${USRBINPATH}/env\ ${SETUPTOOLS_PYTHON}:g $i + sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i + fi + done + + rm -f ${D}${PYTHON_SITEPACKAGES_DIR}/easy-install.pth + + # + # FIXME: Bandaid against wrong datadir computation + # + if [ -e ${D}${datadir}/share ]; then + mv -f ${D}${datadir}/share/* ${D}${datadir}/ + rmdir ${D}${datadir}/share + fi +} +setuptools3_legacy_do_install[vardepsexclude] = "MACHINE" + +EXPORT_FUNCTIONS do_configure do_compile do_install + +export LDSHARED="${CCLD} -shared" +DEPENDS += "python3-setuptools-native" +
Following a good discussion with PyPA upstream[1] the migration of the setuptools3.bbclass to use bdist_wheel+pip turns out to be more complex than thought. Essentially, we're midway through a lot of changes: the future of Python packaging is wheels and pip, but those by design are not as flexible as traditional distutils and setup.py. Specifically, with traditional distutils the package can implement its own install task and write arbitrary files (such as init scripts). With wheels this is explicity impossible, so packages that do this cannot use the new setuptools class and must continue to use the build/install tasks as before. This class is the old setuptools behaviour, bought back. However, as distutils and the setuptools install task are both deprecated and will soon be removed entirely, any users of this class should be moving to an alternative build tool, be it a modern Python tool which works with wheels, or a non-Pythonic tool such as Meson. [1] https://github.com/pypa/packaging-problems/issues/576 Signed-off-by: Ross Burton <ross.burton@arm.com> --- meta/classes/setuptools3_legacy.bbclass | 78 +++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 meta/classes/setuptools3_legacy.bbclass