| Message ID | 20240110115335.92914-1-uvv.mail@gmail.com |
|---|---|
| State | Accepted, archived |
| Commit | 9f220f61e3e44a650a46ee997b47f1d87b7c4ef0 |
| Headers | show |
| Series | [1/3] recipetool: Don't fail on local go modules | expand |
On Wed, 2024-01-10 at 12:53 +0100, Vyacheslav Yurkov wrote: > Local modules are usually referenced with a 'replace' directive in > go.mod file. If that's the case, remove them from populating SRC_URI. > > Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> > --- > scripts/lib/recipetool/create_go.py | 32 +++++++++++++++++------------ > 1 file changed, 19 insertions(+), 13 deletions(-) Do these changes mean we need to improve the test coverage in oe- selftest for recipetool? Cheers, Richard
On 10.01.2024 14:59, Richard Purdie wrote: > On Wed, 2024-01-10 at 12:53 +0100, Vyacheslav Yurkov wrote: >> Local modules are usually referenced with a 'replace' directive in >> go.mod file. If that's the case, remove them from populating SRC_URI. >> >> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> >> --- >> scripts/lib/recipetool/create_go.py | 32 +++++++++++++++++------------ >> 1 file changed, 19 insertions(+), 13 deletions(-) > > Do these changes mean we need to improve the test coverage in oe- > selftest for recipetool? Test converage needs improvement, yes. I haven't considered the case for local modules since this seems to be rare(?). Are there public projects which use this mechanism? Best regards Lukas > > Cheers, > > Richard
On 10.01.2024 16:45, Lukas Funke wrote: > On 10.01.2024 14:59, Richard Purdie wrote: >> On Wed, 2024-01-10 at 12:53 +0100, Vyacheslav Yurkov wrote: >>> Local modules are usually referenced with a 'replace' directive in >>> go.mod file. If that's the case, remove them from populating SRC_URI. >>> >>> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> >>> --- >>> scripts/lib/recipetool/create_go.py | 32 >>> +++++++++++++++++------------ >>> 1 file changed, 19 insertions(+), 13 deletions(-) >> >> Do these changes mean we need to improve the test coverage in oe- >> selftest for recipetool? > > Test converage needs improvement, yes. I haven't considered the case > for local modules since this seems to be rare(?). Are there public > projects which use this mechanism? > > Best regards > Lukas > I added coverage for local modules in my v2. I'm not a golang expert, but it seems that local modules in golang is a means to "place a patched version into the project space until upstream fixes it" more or less like Yocto does with Upstream patches. It's used a lot by in-house projects, but you could also get a glimpse on github with this query https://github.com/search?q=path%3A%2Fgo.mod+NOT+is%3Afork+content%3A%2F%5Ereplace+.*%2F&type=code&query=content%3Ago.mod+replace&p=3 . Note, Github is able to display only first 100 search results (Github API restriction). One thing I don't like in my v2 is that I need to look up a public project to add required coverage. I would rather improve tests further by having everything in one repository and just one test, i.e. it has to be a small test project that could be used to test all go-related functionality. The question is though where it should be placed? I could create it on my Github account, but would that be a good option? Richard, any suggestions on where these projects for selftest coverage should be hosted? Slava
On Tue, 2024-01-16 at 09:33 +0100, Vyacheslav Yurkov wrote: > On 10.01.2024 16:45, Lukas Funke wrote: > > On 10.01.2024 14:59, Richard Purdie wrote: > > > On Wed, 2024-01-10 at 12:53 +0100, Vyacheslav Yurkov wrote: > > > > Local modules are usually referenced with a 'replace' directive in > > > > go.mod file. If that's the case, remove them from populating SRC_URI. > > > > > > > > Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> > > > > --- > > > > scripts/lib/recipetool/create_go.py | 32 > > > > +++++++++++++++++------------ > > > > 1 file changed, 19 insertions(+), 13 deletions(-) > > > > > > Do these changes mean we need to improve the test coverage in oe- > > > selftest for recipetool? > > > > Test converage needs improvement, yes. I haven't considered the case > > for local modules since this seems to be rare(?). Are there public > > projects which use this mechanism? > > > > Best regards > > Lukas > > > > I added coverage for local modules in my v2. > > I'm not a golang expert, but it seems that local modules in golang is a > means to "place a patched version into the project space until upstream > fixes it" more or less like Yocto does with Upstream patches. It's used > a lot by in-house projects, but you could also get a glimpse on github > with this query > https://github.com/search?q=path%3A%2Fgo.mod+NOT+is%3Afork+content%3A%2F%5Ereplace+.*%2F&type=code&query=content%3Ago.mod+replace&p=3 > . Note, Github is able to display only first 100 search results (Github > API restriction). > > One thing I don't like in my v2 is that I need to look up a public > project to add required coverage. I would rather improve tests further > by having everything in one repository and just one test, i.e. it has to > be a small test project that could be used to test all go-related > functionality. The question is though where it should be placed? I could > create it on my Github account, but would that be a good option? > > Richard, any suggestions on where these projects for selftest coverage > should be hosted? Thanks for the tests, they are massively helpful in maintaining this code. We could always add something to git.yoctoproject.org, we do have a few test repositories there already for various purposes like this. If you want to do that, please send an email to helpdesk@yoctoproject.org with an ssh key (or I think a pointer to your github account), requesting setup for the repo and a brief description of what it is for. Cheers, Richard
diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py index 21dcb41271..bd20b07f91 100644 --- a/scripts/lib/recipetool/create_go.py +++ b/scripts/lib/recipetool/create_go.py @@ -504,7 +504,7 @@ class GoRecipeHandler(RecipeHandler): return inline_fcn, commit - def __go_handle_dependencies(self, go_mod, localfilesdir, extravalues, d): + def __go_handle_dependencies(self, go_mod, srctree, localfilesdir, extravalues, d): src_uris = [] src_revs = [] @@ -525,6 +525,23 @@ class GoRecipeHandler(RecipeHandler): return src_rev + # we first go over replacement list, because we are essentialy + # interested only in the replaced path + if go_mod['Replace']: + for replacement in go_mod['Replace']: + oldpath = replacement['Old']['Path'] + path = replacement['New']['Path'] + if 'Version' in replacement['New']: + version = replacement['New']['Version'] + + if os.path.exists(os.path.join(srctree, path)): + # the module refers to the local path, remove it from requirement list + # because it's a local module + go_mod['Require'][:] = [v for v in go_mod['Require'] if v.get('Path') != oldpath] + else: + # Replace the path and the version, so we don't iterate replacement list anymore + go_mod['Require']['Path'] = path + for require in go_mod['Require']: path = require['Path'] version = require['Version'] @@ -534,17 +551,6 @@ class GoRecipeHandler(RecipeHandler): src_uris.append(inline_fcn) src_revs.append(generate_src_rev(path, version, commithash)) - if go_mod['Replace']: - for replacement in go_mod['Replace']: - oldpath = replacement['Old']['Path'] - path = replacement['New']['Path'] - version = replacement['New']['Version'] - - inline_fcn, commithash = self.__generate_srcuri_inline_fcn( - path, version, oldpath) - src_uris.append(inline_fcn) - src_revs.append(generate_src_rev(path, version, commithash)) - pn, _ = determine_from_url(go_mod['Module']['Path']) go_mods_basename = "%s-modules.inc" % pn @@ -693,7 +699,7 @@ class GoRecipeHandler(RecipeHandler): self.__rewrite_src_uri(lines_before, ["file://modules.txt"]) - self.__go_handle_dependencies(go_mod, localfilesdir, extravalues, d) + self.__go_handle_dependencies(go_mod, srctree, localfilesdir, extravalues, d) lines_before.append("require ${BPN}-modules.inc") # Do generic license handling
Local modules are usually referenced with a 'replace' directive in go.mod file. If that's the case, remove them from populating SRC_URI. Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> --- scripts/lib/recipetool/create_go.py | 32 +++++++++++++++++------------ 1 file changed, 19 insertions(+), 13 deletions(-)