| Message ID | 20221030173815.10212-2-alex.kiernan@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] cargo_common.bbclass: Fix typos | expand |
There doesn’t seem to be a test case or an example for this. How can we ensure the code is correct? Alex On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com> wrote: > Since disable network was added cargo configurations which reference git > repos fail as they attempt to fetch across the network as part of > do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths` > is used, as this is documented as only working for packages which exist > in crates.io. > > Add parsing of the SRC_URIs for git repos and include `[patch]` sections > to redirect to the checked out source repos which the bitbake fetcher > has already populated. > > There are still cases which don't work - if you have multiple copies of > the same repo with different revisions, there's currently no way to > represent that and anything using a repo which has a virtual manifest > will fail to build (see https://github.com/rust-lang/cargo/issues/4934). > > Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> > --- > meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++ > 1 file changed, 30 insertions(+) > > diff --git a/meta/classes-recipe/cargo_common.bbclass > b/meta/classes-recipe/cargo_common.bbclass > index f503a001dd8e..63b13829084b 100644 > --- a/meta/classes-recipe/cargo_common.bbclass > +++ b/meta/classes-recipe/cargo_common.bbclass > @@ -116,6 +116,36 @@ cargo_common_do_configure () { > EOF > } > > +python cargo_common_do_patch_paths() { > + cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config") > + if not os.path.exists(cargo_config): > + return > + > + src_uri = (d.getVar('SRC_URI') or "").split() > + if len(src_uri) == 0: > + return > + > + patches = dict() > + workdir = d.getVar('WORKDIR') > + fetcher = bb.fetch2.Fetch(src_uri, d) > + for url in fetcher.urls: > + ud = fetcher.ud[url] > + if ud.type == 'git': > + name = ud.parm.get('name') > + destsuffix = ud.parm.get('destsuffix') > + if name is not None and destsuffix is not None: > + repo = '%s://%s%s' % (ud.proto, ud.host, ud.path) > + path = '%s = { path = "%s" }' % (name, > os.path.join(workdir, destsuffix)) > + patches.setdefault(repo, []).append(path) > + > + with open(cargo_config, "a+") as config: > + for k, v in patches.items(): > + print('\n[patch."%s"]' % k, file=config) > + for name in v: > + print(name, file=config) > +} > +do_configure[postfuncs] += "cargo_common_do_patch_paths" > + > oe_cargo_fix_env () { > export CC="${RUST_TARGET_CC}" > export CXX="${RUST_TARGET_CXX}" > -- > 2.35.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#172284): > https://lists.openembedded.org/g/openembedded-core/message/172284 > Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
I was wondering about how to do that - my test case was https://github.com/akiernan/uuid-test, but that's clearly not adequate; the recipe there comes from `cargo bitbake` as `bitbake -c update_crates` doesn't deal with git dependencies, or dig down into transitive deps, which also needs solving. I'm not clear if something like this could be added as part of the stuff that's in master-next for rust tests? Or if we need to build out something new. On Sun, Oct 30, 2022 at 5:58 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > There doesn’t seem to be a test case or an example for this. How can we ensure the code is correct? > > Alex > > On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com> wrote: >> >> Since disable network was added cargo configurations which reference git >> repos fail as they attempt to fetch across the network as part of >> do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths` >> is used, as this is documented as only working for packages which exist >> in crates.io. >> >> Add parsing of the SRC_URIs for git repos and include `[patch]` sections >> to redirect to the checked out source repos which the bitbake fetcher >> has already populated. >> >> There are still cases which don't work - if you have multiple copies of >> the same repo with different revisions, there's currently no way to >> represent that and anything using a repo which has a virtual manifest >> will fail to build (see https://github.com/rust-lang/cargo/issues/4934). >> >> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> >> --- >> meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++ >> 1 file changed, 30 insertions(+) >> >> diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass >> index f503a001dd8e..63b13829084b 100644 >> --- a/meta/classes-recipe/cargo_common.bbclass >> +++ b/meta/classes-recipe/cargo_common.bbclass >> @@ -116,6 +116,36 @@ cargo_common_do_configure () { >> EOF >> } >> >> +python cargo_common_do_patch_paths() { >> + cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config") >> + if not os.path.exists(cargo_config): >> + return >> + >> + src_uri = (d.getVar('SRC_URI') or "").split() >> + if len(src_uri) == 0: >> + return >> + >> + patches = dict() >> + workdir = d.getVar('WORKDIR') >> + fetcher = bb.fetch2.Fetch(src_uri, d) >> + for url in fetcher.urls: >> + ud = fetcher.ud[url] >> + if ud.type == 'git': >> + name = ud.parm.get('name') >> + destsuffix = ud.parm.get('destsuffix') >> + if name is not None and destsuffix is not None: >> + repo = '%s://%s%s' % (ud.proto, ud.host, ud.path) >> + path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix)) >> + patches.setdefault(repo, []).append(path) >> + >> + with open(cargo_config, "a+") as config: >> + for k, v in patches.items(): >> + print('\n[patch."%s"]' % k, file=config) >> + for name in v: >> + print(name, file=config) >> +} >> +do_configure[postfuncs] += "cargo_common_do_patch_paths" >> + >> oe_cargo_fix_env () { >> export CC="${RUST_TARGET_CC}" >> export CXX="${RUST_TARGET_CXX}" >> -- >> 2.35.1 >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#172284): https://lists.openembedded.org/g/openembedded-core/message/172284 >> Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >>
It would also help if there’s an actual recipe and component somewhere where the problem exists now. The uuid-test does look like a made up example. Let’s merge the crate updater to core first, then we can think of handling other ways to specify dependencies. ‘cargo bitbake’ is pretty broken and dead. Alex On Sun 30. Oct 2022 at 19.12, Alex Kiernan <alex.kiernan@gmail.com> wrote: > I was wondering about how to do that - my test case was > https://github.com/akiernan/uuid-test, but that's clearly not > adequate; the recipe there comes from `cargo bitbake` as `bitbake -c > update_crates` doesn't deal with git dependencies, or dig down into > transitive deps, which also needs solving. > > I'm not clear if something like this could be added as part of the > stuff that's in master-next for rust tests? Or if we need to build out > something new. > > On Sun, Oct 30, 2022 at 5:58 PM Alexander Kanavin > <alex.kanavin@gmail.com> wrote: > > > > There doesn’t seem to be a test case or an example for this. How can we > ensure the code is correct? > > > > Alex > > > > On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com> > wrote: > >> > >> Since disable network was added cargo configurations which reference git > >> repos fail as they attempt to fetch across the network as part of > >> do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths` > >> is used, as this is documented as only working for packages which exist > >> in crates.io. > >> > >> Add parsing of the SRC_URIs for git repos and include `[patch]` sections > >> to redirect to the checked out source repos which the bitbake fetcher > >> has already populated. > >> > >> There are still cases which don't work - if you have multiple copies of > >> the same repo with different revisions, there's currently no way to > >> represent that and anything using a repo which has a virtual manifest > >> will fail to build (see https://github.com/rust-lang/cargo/issues/4934 > ). > >> > >> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> > >> --- > >> meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++ > >> 1 file changed, 30 insertions(+) > >> > >> diff --git a/meta/classes-recipe/cargo_common.bbclass > b/meta/classes-recipe/cargo_common.bbclass > >> index f503a001dd8e..63b13829084b 100644 > >> --- a/meta/classes-recipe/cargo_common.bbclass > >> +++ b/meta/classes-recipe/cargo_common.bbclass > >> @@ -116,6 +116,36 @@ cargo_common_do_configure () { > >> EOF > >> } > >> > >> +python cargo_common_do_patch_paths() { > >> + cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config") > >> + if not os.path.exists(cargo_config): > >> + return > >> + > >> + src_uri = (d.getVar('SRC_URI') or "").split() > >> + if len(src_uri) == 0: > >> + return > >> + > >> + patches = dict() > >> + workdir = d.getVar('WORKDIR') > >> + fetcher = bb.fetch2.Fetch(src_uri, d) > >> + for url in fetcher.urls: > >> + ud = fetcher.ud[url] > >> + if ud.type == 'git': > >> + name = ud.parm.get('name') > >> + destsuffix = ud.parm.get('destsuffix') > >> + if name is not None and destsuffix is not None: > >> + repo = '%s://%s%s' % (ud.proto, ud.host, ud.path) > >> + path = '%s = { path = "%s" }' % (name, > os.path.join(workdir, destsuffix)) > >> + patches.setdefault(repo, []).append(path) > >> + > >> + with open(cargo_config, "a+") as config: > >> + for k, v in patches.items(): > >> + print('\n[patch."%s"]' % k, file=config) > >> + for name in v: > >> + print(name, file=config) > >> +} > >> +do_configure[postfuncs] += "cargo_common_do_patch_paths" > >> + > >> oe_cargo_fix_env () { > >> export CC="${RUST_TARGET_CC}" > >> export CXX="${RUST_TARGET_CXX}" > >> -- > >> 2.35.1 > >> > >> > >> -=-=-=-=-=-=-=-=-=-=-=- > >> Links: You receive all messages sent to this group. > >> View/Reply Online (#172284): > https://lists.openembedded.org/g/openembedded-core/message/172284 > >> Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489 > >> Group Owner: openembedded-core+owner@lists.openembedded.org > >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > >> -=-=-=-=-=-=-=-=-=-=-=- > >> > > > -- > Alex Kiernan >
On Sun, Oct 30, 2022 at 6:35 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > It would also help if there’s an actual recipe and component somewhere where the problem exists now. The uuid-test does look like a made up example. > That's fair... basically it was demonstrating the problem I have with a bunch of private repos, which I expect will crop up a lot in the embedded rust world - if you're doing public dev, everything exists as crates, in the private world you're consuming crates + internal stuff. Not sure where to try and find a real world example as I'd expect pretty much everyone will push to crates.io if their code is open - searching around for a bit, I've failed to find any (though a couple which have some dev only dependencies which come through git). > Let’s merge the crate updater to core first, then we can think of handling other ways to specify dependencies. ‘cargo bitbake’ is pretty broken and dead. > The crate updater feels like it's going to be directionally better - testing it locally, it works for things which only consume things from crates.io, but that's only one out of thirty or so components I have to deal with. > Alex > > On Sun 30. Oct 2022 at 19.12, Alex Kiernan <alex.kiernan@gmail.com> wrote: >> >> I was wondering about how to do that - my test case was >> https://github.com/akiernan/uuid-test, but that's clearly not >> adequate; the recipe there comes from `cargo bitbake` as `bitbake -c >> update_crates` doesn't deal with git dependencies, or dig down into >> transitive deps, which also needs solving. >> >> I'm not clear if something like this could be added as part of the >> stuff that's in master-next for rust tests? Or if we need to build out >> something new. >> >> On Sun, Oct 30, 2022 at 5:58 PM Alexander Kanavin >> <alex.kanavin@gmail.com> wrote: >> > >> > There doesn’t seem to be a test case or an example for this. How can we ensure the code is correct? >> > >> > Alex >> > >> > On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com> wrote: >> >> >> >> Since disable network was added cargo configurations which reference git >> >> repos fail as they attempt to fetch across the network as part of >> >> do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths` >> >> is used, as this is documented as only working for packages which exist >> >> in crates.io. >> >> >> >> Add parsing of the SRC_URIs for git repos and include `[patch]` sections >> >> to redirect to the checked out source repos which the bitbake fetcher >> >> has already populated. >> >> >> >> There are still cases which don't work - if you have multiple copies of >> >> the same repo with different revisions, there's currently no way to >> >> represent that and anything using a repo which has a virtual manifest >> >> will fail to build (see https://github.com/rust-lang/cargo/issues/4934). >> >> >> >> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> >> >> --- >> >> meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++ >> >> 1 file changed, 30 insertions(+) >> >> >> >> diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass >> >> index f503a001dd8e..63b13829084b 100644 >> >> --- a/meta/classes-recipe/cargo_common.bbclass >> >> +++ b/meta/classes-recipe/cargo_common.bbclass >> >> @@ -116,6 +116,36 @@ cargo_common_do_configure () { >> >> EOF >> >> } >> >> >> >> +python cargo_common_do_patch_paths() { >> >> + cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config") >> >> + if not os.path.exists(cargo_config): >> >> + return >> >> + >> >> + src_uri = (d.getVar('SRC_URI') or "").split() >> >> + if len(src_uri) == 0: >> >> + return >> >> + >> >> + patches = dict() >> >> + workdir = d.getVar('WORKDIR') >> >> + fetcher = bb.fetch2.Fetch(src_uri, d) >> >> + for url in fetcher.urls: >> >> + ud = fetcher.ud[url] >> >> + if ud.type == 'git': >> >> + name = ud.parm.get('name') >> >> + destsuffix = ud.parm.get('destsuffix') >> >> + if name is not None and destsuffix is not None: >> >> + repo = '%s://%s%s' % (ud.proto, ud.host, ud.path) >> >> + path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix)) >> >> + patches.setdefault(repo, []).append(path) >> >> + >> >> + with open(cargo_config, "a+") as config: >> >> + for k, v in patches.items(): >> >> + print('\n[patch."%s"]' % k, file=config) >> >> + for name in v: >> >> + print(name, file=config) >> >> +} >> >> +do_configure[postfuncs] += "cargo_common_do_patch_paths" >> >> + >> >> oe_cargo_fix_env () { >> >> export CC="${RUST_TARGET_CC}" >> >> export CXX="${RUST_TARGET_CXX}" >> >> -- >> >> 2.35.1 >> >> >> >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#172284): https://lists.openembedded.org/g/openembedded-core/message/172284 >> >> Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489 >> >> Group Owner: openembedded-core+owner@lists.openembedded.org >> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] >> >> -=-=-=-=-=-=-=-=-=-=-=- >> >> >> >> >> -- >> Alex Kiernan -- Alex Kiernan
diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass index f503a001dd8e..63b13829084b 100644 --- a/meta/classes-recipe/cargo_common.bbclass +++ b/meta/classes-recipe/cargo_common.bbclass @@ -116,6 +116,36 @@ cargo_common_do_configure () { EOF } +python cargo_common_do_patch_paths() { + cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config") + if not os.path.exists(cargo_config): + return + + src_uri = (d.getVar('SRC_URI') or "").split() + if len(src_uri) == 0: + return + + patches = dict() + workdir = d.getVar('WORKDIR') + fetcher = bb.fetch2.Fetch(src_uri, d) + for url in fetcher.urls: + ud = fetcher.ud[url] + if ud.type == 'git': + name = ud.parm.get('name') + destsuffix = ud.parm.get('destsuffix') + if name is not None and destsuffix is not None: + repo = '%s://%s%s' % (ud.proto, ud.host, ud.path) + path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix)) + patches.setdefault(repo, []).append(path) + + with open(cargo_config, "a+") as config: + for k, v in patches.items(): + print('\n[patch."%s"]' % k, file=config) + for name in v: + print(name, file=config) +} +do_configure[postfuncs] += "cargo_common_do_patch_paths" + oe_cargo_fix_env () { export CC="${RUST_TARGET_CC}" export CXX="${RUST_TARGET_CXX}"
Since disable network was added cargo configurations which reference git repos fail as they attempt to fetch across the network as part of do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths` is used, as this is documented as only working for packages which exist in crates.io. Add parsing of the SRC_URIs for git repos and include `[patch]` sections to redirect to the checked out source repos which the bitbake fetcher has already populated. There are still cases which don't work - if you have multiple copies of the same repo with different revisions, there's currently no way to represent that and anything using a repo which has a virtual manifest will fail to build (see https://github.com/rust-lang/cargo/issues/4934). Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> --- meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++ 1 file changed, 30 insertions(+)