mbox series

[0/7] Devtool/Recipetool: adding pypi support

Message ID 20231117111218.3344066-1-jstephan@baylibre.com
Headers show
Series Devtool/Recipetool: adding pypi support | expand

Message

Julien Stephan Nov. 17, 2023, 11:12 a.m. UTC
Hello all,

This series adds basic support for the pypi class in devtool/recipetool.

The idea is to be able to detect that a package is available on pypi and
in that case inherits from pypi class. This helps to produce cleaner
recipes for pypi packages.

To do this, I am adding a new optional callback "process_url" that
plugin can register to implement some logic on matching url.

By implementing this callback in create_buildsys_python we can match on
pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".

To create a recipe taking advantage of the pypi class we can use one of the
following new syntax:

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:

* recipetool create https://files.pythonhosted.org/packages/<...>

If the URL points to a github URL or a release tarball not coming from
"files.pythonhosted.org", the created recipe is the same as before.
One can also use the newly added "--no-pypi" switch to NOT inherit 
from pypi class on matching URL, to keep legacy behaviour.

This series also contains some bug fixes I found during my testing.

Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support 

Cheers
Julien

Julien Stephan (7):
  bitbake: utils: remove spaces on empty lines
  recipetool: create_buildsys_python.py: initialize metadata
  recipetool: create: add trailing newlines
  recipetool: create: add new optional process_url callback for plugins
  recipetool: create_buildsys_python: add pypi support
  oeqa/selftest/recipetool: remove spaces on empty lines
  oeqa/selftest/recipetool/devtool: add test for pypi class

 bitbake/lib/bb/utils.py                       |  16 +--
 meta/lib/oeqa/selftest/cases/devtool.py       |   4 +-
 meta/lib/oeqa/selftest/cases/recipetool.py    | 114 +++++++++++++-----
 scripts/lib/devtool/standard.py               |   3 +
 scripts/lib/recipetool/create.py              |  63 ++++++----
 .../lib/recipetool/create_buildsys_python.py  |  73 +++++++++++
 6 files changed, 208 insertions(+), 65 deletions(-)

Comments

Alexandre Belloni Nov. 18, 2023, 10:10 a.m. UTC | #1
Hello,

The tests seem to be failing on the AB:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6065/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6062/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6013/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6078/steps/14/logs/stdio

On 17/11/2023 12:12:11+0100, Julien Stephan wrote:
> Hello all,
> 
> This series adds basic support for the pypi class in devtool/recipetool.
> 
> The idea is to be able to detect that a package is available on pypi and
> in that case inherits from pypi class. This helps to produce cleaner
> recipes for pypi packages.
> 
> To do this, I am adding a new optional callback "process_url" that
> plugin can register to implement some logic on matching url.
> 
> By implementing this callback in create_buildsys_python we can match on
> pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".
> 
> To create a recipe taking advantage of the pypi class we can use one of the
> following new syntax:
> 
> * recipetool create https://pypi.org/project/<package>
> * recipetool create https://pypi.org/project/<package>/<version>
> * recipetool create https://pypi.org/project/<package> --version <version>
> 
> or the old syntax:
> 
> * recipetool create https://files.pythonhosted.org/packages/<...>
> 
> If the URL points to a github URL or a release tarball not coming from
> "files.pythonhosted.org", the created recipe is the same as before.
> One can also use the newly added "--no-pypi" switch to NOT inherit 
> from pypi class on matching URL, to keep legacy behaviour.
> 
> This series also contains some bug fixes I found during my testing.
> 
> Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support 
> 
> Cheers
> Julien
> 
> Julien Stephan (7):
>   bitbake: utils: remove spaces on empty lines
>   recipetool: create_buildsys_python.py: initialize metadata
>   recipetool: create: add trailing newlines
>   recipetool: create: add new optional process_url callback for plugins
>   recipetool: create_buildsys_python: add pypi support
>   oeqa/selftest/recipetool: remove spaces on empty lines
>   oeqa/selftest/recipetool/devtool: add test for pypi class
> 
>  bitbake/lib/bb/utils.py                       |  16 +--
>  meta/lib/oeqa/selftest/cases/devtool.py       |   4 +-
>  meta/lib/oeqa/selftest/cases/recipetool.py    | 114 +++++++++++++-----
>  scripts/lib/devtool/standard.py               |   3 +
>  scripts/lib/recipetool/create.py              |  63 ++++++----
>  .../lib/recipetool/create_buildsys_python.py  |  73 +++++++++++
>  6 files changed, 208 insertions(+), 65 deletions(-)
> 
> -- 
> 2.42.0
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#190836): https://lists.openembedded.org/g/openembedded-core/message/190836
> Mute This Topic: https://lists.openembedded.org/mt/102645283/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Alexandre Belloni Nov. 18, 2023, 1:48 p.m. UTC | #2
On 18/11/2023 11:10:58+0100, Alexandre Belloni via lists.openembedded.org wrote:
> Hello,
> 
> The tests seem to be failing on the AB:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6065/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6062/steps/15/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6013/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6078/steps/14/logs/stdio

Actually this seems to have been caused by something else.

> 
> On 17/11/2023 12:12:11+0100, Julien Stephan wrote:
> > Hello all,
> > 
> > This series adds basic support for the pypi class in devtool/recipetool.
> > 
> > The idea is to be able to detect that a package is available on pypi and
> > in that case inherits from pypi class. This helps to produce cleaner
> > recipes for pypi packages.
> > 
> > To do this, I am adding a new optional callback "process_url" that
> > plugin can register to implement some logic on matching url.
> > 
> > By implementing this callback in create_buildsys_python we can match on
> > pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".
> > 
> > To create a recipe taking advantage of the pypi class we can use one of the
> > following new syntax:
> > 
> > * recipetool create https://pypi.org/project/<package>
> > * recipetool create https://pypi.org/project/<package>/<version>
> > * recipetool create https://pypi.org/project/<package> --version <version>
> > 
> > or the old syntax:
> > 
> > * recipetool create https://files.pythonhosted.org/packages/<...>
> > 
> > If the URL points to a github URL or a release tarball not coming from
> > "files.pythonhosted.org", the created recipe is the same as before.
> > One can also use the newly added "--no-pypi" switch to NOT inherit 
> > from pypi class on matching URL, to keep legacy behaviour.
> > 
> > This series also contains some bug fixes I found during my testing.
> > 
> > Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support 
> > 
> > Cheers
> > Julien
> > 
> > Julien Stephan (7):
> >   bitbake: utils: remove spaces on empty lines
> >   recipetool: create_buildsys_python.py: initialize metadata
> >   recipetool: create: add trailing newlines
> >   recipetool: create: add new optional process_url callback for plugins
> >   recipetool: create_buildsys_python: add pypi support
> >   oeqa/selftest/recipetool: remove spaces on empty lines
> >   oeqa/selftest/recipetool/devtool: add test for pypi class
> > 
> >  bitbake/lib/bb/utils.py                       |  16 +--
> >  meta/lib/oeqa/selftest/cases/devtool.py       |   4 +-
> >  meta/lib/oeqa/selftest/cases/recipetool.py    | 114 +++++++++++++-----
> >  scripts/lib/devtool/standard.py               |   3 +
> >  scripts/lib/recipetool/create.py              |  63 ++++++----
> >  .../lib/recipetool/create_buildsys_python.py  |  73 +++++++++++
> >  6 files changed, 208 insertions(+), 65 deletions(-)
> > 
> > -- 
> > 2.42.0
> > 
> 
> > 
> > 
> > 
> 
> 
> -- 
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#190862): https://lists.openembedded.org/g/openembedded-core/message/190862
> Mute This Topic: https://lists.openembedded.org/mt/102645283/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>