| Message ID | 20230405101609.3309660-1-enrico.scholz@sigma-chemnitz.de |
|---|---|
| State | New |
| Headers | show |
| Series | fetch:crate: create versioned 'name' entries | expand |
Hello,
There are multiple issues with this patch.
First, this is not compatible with recipes which will override the name
parameter in their crate:// entry like zvariant in meta-selftest layer.
Second, the cargo-update-recipe-crates class already did that modification
for you (it set name parameter to be pkg-vers and it expect bitbake fetcher
to respect that),
by the way, the zavariant I talk about was built with usage of this class
Last, this modification makes the bitbake unit test fail.
To bootstrap a cargo base recipe which have not this checksum list, below
are
the steps to make:
- make the recipe inherit cargo-update-recipe-crates
- remove all crate:// from your recipe
- create an empty ${BPN}-crates.inc file and make your recipe requires it
- execute "bitbake -c update_crates your_recipe"
- copy paste the output of bitbake about the missing checksums in your
${BPN}-crates.inc
And you should have a buildable cargo based recipe with these.
On Wed, 5 Apr 2023 at 12:16, Enrico Scholz via lists.openembedded.org
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote:
> From: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
>
> It is common for rust packages to depend on different versions of the
> same crate. E.g.
>
> | crate://crates.io/windows_x86_64_msvc/0.42.2 \
> | crate://crates.io/windows_x86_64_msvc/0.48.0 \
>
> Identification only by the plain crate name makes the sha256sum
> ambiguous
>
> | SRC_URI[windows_x86_64_msvc.sha256sum] =
> "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
> | SRC_URI[windows_x86_64_msvc.sha256sum] =
> "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"
>
> and requires lot of manual work to fix the SRC_URI list.
>
> Making the 'name' property unique by appending the version allows
> direct copy & paste of reported sha256sum errors to complete the
> crates list.
>
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
> lib/bb/fetch2/crate.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/crate.py b/lib/bb/fetch2/crate.py
> index 590dc9c12681..a7021e5b367c 100644
> --- a/lib/bb/fetch2/crate.py
> +++ b/lib/bb/fetch2/crate.py
> @@ -72,7 +72,7 @@ class Crate(Wget):
> ud.url = "https://%s/%s/%s/download" % (host, name, version)
> ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
> if 'name' not in ud.parm:
> - ud.parm['name'] = name
> + ud.parm['name'] = '%s-%s' % (name, version)
>
> logger.debug2("Fetching %s to %s" % (ud.url,
> ud.parm['downloadfilename']))
>
> --
> 2.39.2
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14645):
> https://lists.openembedded.org/g/bitbake-devel/message/14645
> Mute This Topic: https://lists.openembedded.org/mt/98079257/6213388
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> frederic.martinsons@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > First, this is not compatible with recipes which will override the name > parameter in their crate:// entry like zvariant in meta-selftest > layer. When this is really a problem (I do not see why), it existed already before. > Second, the cargo-update-recipe-crates class already did that > modification for you (it set name parameter to be pkg-vers and it > expect bitbake fetcher to respect that), cargo-update-recipe-crates is not very useful and has huge bootstrapping problems (see below). > Last, this modification makes the bitbake unit test fail. ok; my fault. I will send a v2 > To bootstrap a cargo base recipe which have not this checksum list, below > are > the steps to make: > - make the recipe inherit cargo-update-recipe-crates > - remove all crate:// from your recipe > - create an empty ${BPN}-crates.inc file and make your recipe requires it > - execute "bitbake -c update_crates your_recipe" This does not work. This class requires Cargo.lock, but to have Cargo.lock, do_compile() must be executed. This fails when required crates are not available yet. Enrico
On Wed, 5 Apr 2023 at 14:09, Enrico Scholz <enrico.scholz@sigma-chemnitz.de> wrote: > > To bootstrap a cargo base recipe which have not this checksum list, below > > are > > the steps to make: > > - make the recipe inherit cargo-update-recipe-crates > > - remove all crate:// from your recipe > > - create an empty ${BPN}-crates.inc file and make your recipe requires > it > > - execute "bitbake -c update_crates your_recipe" > > This does not work. This class requires Cargo.lock, but to have > Cargo.lock, do_compile() must be executed. This fails when required > crates are not available yet. > > > Enrico > A rust project must have a Cargo.lock in its source ( https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries ) unless it is a pure library but I don't see the use case of building a rust library that wouldn't be a simple crate on crates.io.
Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > A rust project must have a Cargo.lock in its source ( > https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries It is recommended (due to documentation issues), but not required. Enrico
It remains the problem with zvariant recipe in meta-selftest layer but I'll let others answer since they may have a bigger picture on that Le mer. 5 avr. 2023, 14:24, Enrico Scholz <enrico.scholz@sigma-chemnitz.de> a écrit : > Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > > > A rust project must have a Cargo.lock in its source ( > > > https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries > > It is recommended (due to documentation issues), but not required. > > > Enrico >
No, I take that back, it should not be an issue since you set the name only if it is not already there. But, to be sure, did you try to build some existing cargo based recipe (zvariant but also python3-bcrypt for example). Anyway, I tend to think we should keep using cargo-update-update-recipe-crates as the only entry point to update those recipe so maybe we want to made modifications there to handle the most possible use case (missing Cargo.lock, bad Cargo.lock, some other corner cases... Etc) Le mer. 5 avr. 2023, 14:50, Frederic Martinsons via lists.openembedded.org <frederic.martinsons=gmail.com@lists.openembedded.org> a écrit : > It remains the problem with zvariant recipe in meta-selftest layer but > I'll let others answer since they may have a bigger picture on that > > Le mer. 5 avr. 2023, 14:24, Enrico Scholz <enrico.scholz@sigma-chemnitz.de> > a écrit : > >> Frédéric Martinsons <frederic.martinsons@gmail.com> writes: >> >> > A rust project must have a Cargo.lock in its source ( >> > >> https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries >> >> It is recommended (due to documentation issues), but not required. >> >> >> Enrico >> > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14656): > https://lists.openembedded.org/g/bitbake-devel/message/14656 > Mute This Topic: https://lists.openembedded.org/mt/98079257/6213388 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ > frederic.martinsons@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > No, I take that back, it should not be an issue since you set the name > only if it is not already there. But, to be sure, did you try to build > some existing cargo based recipe (zvariant but also python3-bcrypt for > example). It will break cargo related recipes which were created between 2023-03-16 and 2023-03-31. Packages before this date do not contain sha256sum and after this date, they have an explicit 'name' (which matches the one generated by my patch). Enrico
Le mer. 5 avr. 2023, 18:45, Enrico Scholz <enrico.scholz@sigma-chemnitz.de> a écrit : > Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > > > No, I take that back, it should not be an issue since you set the name > > only if it is not already there. But, to be sure, did you try to build > > some existing cargo based recipe (zvariant but also python3-bcrypt for > > example). > > It will break cargo related recipes which were created between 2023-03-16 > and 2023-03-31. Packages before this date do not contain sha256sum and > after this date, they have an explicit 'name' (which matches the one > generated by my patch). > > > Enrico > Ok, I guess these dates match my patches series, if I do things rights, I handle all the recipes I know of (in meta-openembedded and openembedded-core at least) >
On Wed, 2023-04-05 at 19:05 +0200, Frederic Martinsons wrote: > > > Le mer. 5 avr. 2023, 18:45, Enrico Scholz > <enrico.scholz@sigma-chemnitz.de> a écrit : > > Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > > > > > No, I take that back, it should not be an issue since you set the > > > name > > > only if it is not already there. But, to be sure, did you try to > > > build > > > some existing cargo based recipe (zvariant but also python3- > > > bcrypt for > > > example). > > > > It will break cargo related recipes which were created between > > 2023-03-16 > > and 2023-03-31. Packages before this date do not contain sha256sum > > and > > after this date, they have an explicit 'name' (which matches the > > one > > generated by my patch). > > > > > > Enrico > > Ok, I guess these dates match my patches series, if I do things > rights, I handle all the recipes I know of (in meta-openembedded and > openembedded-core at least) This now gives me a problem as rc1 for 4.2 built yesterday. I'm guessing we don't want a stable release series with one form for urls for six months and a different one in master? Does this mean I need to scrap rc1, merge this, wait for the recipe updates and then rebuild the release? Cheers, Richard
Le jeu. 6 avr. 2023, 10:45, Richard Purdie < richard.purdie@linuxfoundation.org> a écrit : > On Wed, 2023-04-05 at 19:05 +0200, Frederic Martinsons wrote: > > > > > > Le mer. 5 avr. 2023, 18:45, Enrico Scholz > > <enrico.scholz@sigma-chemnitz.de> a écrit : > > > Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > > > > > > > No, I take that back, it should not be an issue since you set the > > > > name > > > > only if it is not already there. But, to be sure, did you try to > > > > build > > > > some existing cargo based recipe (zvariant but also python3- > > > > bcrypt for > > > > example). > > > > > > It will break cargo related recipes which were created between > > > 2023-03-16 > > > and 2023-03-31. Packages before this date do not contain sha256sum > > > and > > > after this date, they have an explicit 'name' (which matches the > > > one > > > generated by my patch). > > > > > > > > > Enrico > > > > Ok, I guess these dates match my patches series, if I do things > > rights, I handle all the recipes I know of (in meta-openembedded and > > openembedded-core at least) > > This now gives me a problem as rc1 for 4.2 built yesterday. I'm > guessing we don't want a stable release series with one form for urls > for six months and a different one in master? > > Does this mean I need to scrap rc1, merge this, wait for the recipe > updates and then rebuild the release? > > Cheers, > > Richard > I'm not sure I understand, is the patch we talk already merged? In the 4.2? This patch seems to me an improvement and without it, all the recipes I know of was correctly built (even those in meta-security from what I read from Armin in https://lists.openembedded.org/g/bitbake-devel/message/14658)
On Thu, 2023-04-06 at 12:05 +0200, Frédéric Martinsons wrote: > > > Le jeu. 6 avr. 2023, 10:45, Richard Purdie > <richard.purdie@linuxfoundation.org> a écrit : > > On Wed, 2023-04-05 at 19:05 +0200, Frederic Martinsons wrote: > > > > > > > > > Le mer. 5 avr. 2023, 18:45, Enrico Scholz > > > <enrico.scholz@sigma-chemnitz.de> a écrit : > > > > Frédéric Martinsons <frederic.martinsons@gmail.com> writes: > > > > > > > > > No, I take that back, it should not be an issue since you set > > > > > the > > > > > name > > > > > only if it is not already there. But, to be sure, did you try > > > > > to > > > > > build > > > > > some existing cargo based recipe (zvariant but also python3- > > > > > bcrypt for > > > > > example). > > > > > > > > It will break cargo related recipes which were created between > > > > 2023-03-16 > > > > and 2023-03-31. Packages before this date do not contain > > > > sha256sum > > > > and > > > > after this date, they have an explicit 'name' (which matches > > > > the > > > > one > > > > generated by my patch). > > > > > > > > > > > > Enrico > > > > > > Ok, I guess these dates match my patches series, if I do things > > > rights, I handle all the recipes I know of (in meta-openembedded > > > and > > > openembedded-core at least) > > > > This now gives me a problem as rc1 for 4.2 built yesterday. I'm > > guessing we don't want a stable release series with one form for > > urls > > for six months and a different one in master? > > > > Does this mean I need to scrap rc1, merge this, wait for the recipe > > updates and then rebuild the release? > > > > Cheers, > > > > Richard > > I'm not sure I understand, is the patch we talk already merged? In > the 4.2? 4.2 is what is in the mickledore and bitbake 2.4 branches, i.e. the recent changes but without the patch in this thread. > This patch seems to me an improvement and without it, all the recipes > I know of was correctly built (even those in meta-security from what > I read from Armin in > https://lists.openembedded.org/g/bitbake-devel/message/14658) I'm worried that people seem to be saying we need the patch in this thread but it creates an incompatibility with what was already updated previously. Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> writes: >> > It will break cargo related recipes which were created between >> > 2023-03-16 and 2023-03-31. Packages before this date do not >> > contain sha256sum and after this date, they have an explicit >> > 'name' (which matches the one generated by my patch). >> >> Ok, I guess these dates match my patches series, if I do things >> rights, I handle all the recipes I know of (in meta-openembedded and >> openembedded-core at least) > > This now gives me a problem as rc1 for 4.2 built yesterday. I'm > guessing we don't want a stable release series with one form for urls > for six months and a different one in master? > > Does this mean I need to scrap rc1, merge this, wait for the recipe > updates and then rebuild the release? To avoid breaking builds: 1. OE recipes without explicit ';name=' need to be updated; this can be done by 'bitbake -c update_crates'. Afais, this affects only | meta/recipes-devtools/python/python3-bcrypt_4.0.1.bb | meta/recipes-devtools/python/python3-cryptography_39.0.2.bb and in oe-meta | meta-oe/recipes-core/uutils-coreutils/uutils-coreutils_0.0.17.bb | meta-python/recipes-devtools/python/python3-pyruvate_1.1.2.bb 2. the patch can be applied to bitbake 3. [optionally] change cargo-update-recipe-crates.bbclass to remove the explicit (and now redundant) ';name=' from generated SRC_URI list. This can be done post-mickledore Enrico
On Thu, 2023-04-06 at 12:20 +0200, Enrico Scholz wrote: > Richard Purdie <richard.purdie@linuxfoundation.org> writes: > > > > > It will break cargo related recipes which were created between > > > > 2023-03-16 and 2023-03-31. Packages before this date do not > > > > contain sha256sum and after this date, they have an explicit > > > > 'name' (which matches the one generated by my patch). > > > > > > Ok, I guess these dates match my patches series, if I do things > > > rights, I handle all the recipes I know of (in meta-openembedded and > > > openembedded-core at least) > > > > This now gives me a problem as rc1 for 4.2 built yesterday. I'm > > guessing we don't want a stable release series with one form for urls > > for six months and a different one in master? > > > > Does this mean I need to scrap rc1, merge this, wait for the recipe > > updates and then rebuild the release? > > To avoid breaking builds: > > 1. OE recipes without explicit ';name=' need to be updated; this can be > done by 'bitbake -c update_crates'. Afais, this affects only > > | meta/recipes-devtools/python/python3-bcrypt_4.0.1.bb > | meta/recipes-devtools/python/python3-cryptography_39.0.2.bb > > and in oe-meta > > | meta-oe/recipes-core/uutils-coreutils/uutils-coreutils_0.0.17.bb > | meta-python/recipes-devtools/python/python3-pyruvate_1.1.2.bb > > 2. the patch can be applied to bitbake > > 3. [optionally] change cargo-update-recipe-crates.bbclass to remove the > explicit (and now redundant) ';name=' from generated SRC_URI list. > This can be done post-mickledore I've gone ahead and stopped rc1 since I know in the long run this issue is going to cause problems. Short term I'll merge this patch and the fixes needed to the oe-core recipes. Now I've been looking more closely at this, I did start to wonder if we shouldn't just put ;sha256sum=XXX parameters onto the crate urls we generate though? If we do that, we don't need to name them at all. That is something we could move to at a later date if it make sense so it is less important from a release standpoint. Cheers, Richard
Oh my, I see a lot of patches going around at the moment, sorry for the trouble, I didn't expect the changes I made would be so soon in a release Le jeu. 6 avr. 2023, 14:14, Richard Purdie < richard.purdie@linuxfoundation.org> a écrit : > On Thu, 2023-04-06 at 12:20 +0200, Enrico Scholz wrote: > > Richard Purdie <richard.purdie@linuxfoundation.org> writes: > > > > > > > It will break cargo related recipes which were created between > > > > > 2023-03-16 and 2023-03-31. Packages before this date do not > > > > > contain sha256sum and after this date, they have an explicit > > > > > 'name' (which matches the one generated by my patch). > > > > > > > > Ok, I guess these dates match my patches series, if I do things > > > > rights, I handle all the recipes I know of (in meta-openembedded and > > > > openembedded-core at least) > > > > > > This now gives me a problem as rc1 for 4.2 built yesterday. I'm > > > guessing we don't want a stable release series with one form for urls > > > for six months and a different one in master? > > > > > > Does this mean I need to scrap rc1, merge this, wait for the recipe > > > updates and then rebuild the release? > > > > To avoid breaking builds: > > > > 1. OE recipes without explicit ';name=' need to be updated; this can be > > done by 'bitbake -c update_crates'. Afais, this affects only > > > > | meta/recipes-devtools/python/python3-bcrypt_4.0.1.bb > > | meta/recipes-devtools/python/python3-cryptography_39.0.2.bb > > > > and in oe-meta > > > > | meta-oe/recipes-core/uutils-coreutils/uutils-coreutils_0.0.17.bb > > | meta-python/recipes-devtools/python/python3-pyruvate_1.1.2.bb > > > > 2. the patch can be applied to bitbake > > > > 3. [optionally] change cargo-update-recipe-crates.bbclass to remove the > > explicit (and now redundant) ';name=' from generated SRC_URI list. > > This can be done post-mickledore > > I've gone ahead and stopped rc1 since I know in the long run this issue > is going to cause problems. > > Short term I'll merge this patch and the fixes needed to the oe-core > recipes. > > Now I've been looking more closely at this, I did start to wonder if we > shouldn't just put ;sha256sum=XXX parameters onto the crate urls we > generate though? If we do that, we don't need to name them at all. > > That is something we could move to at a later date if it make sense so > it is less important from a release standpoint. > > Cheers, > > Richard >
On Thu, 2023-04-06 at 15:00 +0200, Frédéric Martinsons wrote: > Oh my, I see a lot of patches going around at the moment, sorry for > the trouble, I didn't expect the changes I made would be so soon in a > release In hindsight I probably shouldn't have taken things so close to the release but there are pros and cons to taking or not taking. I thought things were sorted out and it turned out they weren't quite done. I suspect we'd not have had some of the feedback had I not merged them. I've merged a few changes so hopefully this resolves things for bitbake and OE-Core. The other layers should be able to work things out from there. Cheers, Richard
Le jeu. 6 avr. 2023, 15:44, Richard Purdie < richard.purdie@linuxfoundation.org> a écrit : > On Thu, 2023-04-06 at 15:00 +0200, Frédéric Martinsons wrote: > > Oh my, I see a lot of patches going around at the moment, sorry for > > the trouble, I didn't expect the changes I made would be so soon in a > > release > > In hindsight I probably shouldn't have taken things so close to the > release but there are pros and cons to taking or not taking. I thought > things were sorted out and it turned out they weren't quite done. > > I suspect we'd not have had some of the feedback had I not merged them. > > I've merged a few changes so hopefully this resolves things for bitbake > and OE-Core. The other layers should be able to work things out from > there. > > Cheers, > > Richard > Hope so too. >
Richard Purdie <richard.purdie@linuxfoundation.org> writes: > Now I've been looking more closely at this, I did start to wonder if we > shouldn't just put ;sha256sum=XXX parameters onto the crate urls we > generate though? If we do that, we don't need to name them at all. yes; I think, it is a good idea for autogenerated code. But for manually written one, I somehow still prefer the dedicated SRC_URI[<name>.sha256sum] syntax because it easier to read and fix. Enrico
Richard Purdie <richard.purdie@linuxfoundation.org> writes: > Now I've been looking more closely at this, I did start to wonder if we > shouldn't just put ;sha256sum=XXX parameters onto the crate urls we > generate though? If we do that, we don't need to name them at all. I just tested this with SRC_URI += " \ crate://crates.io/addr2line/0.19.0;sha256sum=a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97 \ crate://crates.io/adler/1.0.2;adler-1.0.2.sha256sum=f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe \ " and got | do_fetch: Missing SRC_URI checksum, please add those to the recipe: | SRC_URI[addr2line-0.19.0.sha256sum] = "a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97" It seems that ';sha256sum' does not suffice but that you have to dup the 'name'. Enrico
Maybe it missed something in crate fetcher but, providing this fetcher inherits wget one, and if wget supports this attribute, so should crate fetcher do. I quickly look into differents recipe of oe-core, meta-oe, meta-security etc and I don't see a SRC_URI which uses sha256sum inline attribute, so it may not be correctly parsed. On Thu, 6 Apr 2023 at 16:18, Enrico Scholz <enrico.scholz@sigma-chemnitz.de> wrote: > Richard Purdie <richard.purdie@linuxfoundation.org> writes: > > > Now I've been looking more closely at this, I did start to wonder if we > > shouldn't just put ;sha256sum=XXX parameters onto the crate urls we > > generate though? If we do that, we don't need to name them at all. > > I just tested this with > > SRC_URI += " \ > crate:// > crates.io/addr2line/0.19.0;sha256sum=a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97 > \ > crate:// > crates.io/adler/1.0.2;adler-1.0.2.sha256sum=f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe > \ > " > > and got > > | do_fetch: Missing SRC_URI checksum, please add those to the recipe: > | SRC_URI[addr2line-0.19.0.sha256sum] = > "a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97" > > > It seems that ';sha256sum' does not suffice but that you have to dup the > 'name'. > > > > Enrico >
diff --git a/lib/bb/fetch2/crate.py b/lib/bb/fetch2/crate.py index 590dc9c12681..a7021e5b367c 100644 --- a/lib/bb/fetch2/crate.py +++ b/lib/bb/fetch2/crate.py @@ -72,7 +72,7 @@ class Crate(Wget): ud.url = "https://%s/%s/%s/download" % (host, name, version) ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version) if 'name' not in ud.parm: - ud.parm['name'] = name + ud.parm['name'] = '%s-%s' % (name, version) logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename']))