| Message ID | 20231017132647.352938-1-lukas.funke-oss@weidmueller.com |
|---|---|
| Headers | show |
| Series | recipetool: Add handler to create go recipes | expand |
Hey Lukas, Thanks a lot for the patch. A few questions/comments from my initial test below. - I tried it with a go-based backend I have by providing ssh URL to github. It seems like the GO_IMPORT is set to a module name from go.mod of my project, which of course fails to fetch it like that, because it's not a valid URL. How is it supposed to be used? - I've got about 20 lines like: INFO: Please add the following line for 'vendor/google.golang.org/protobuf/LICENSE' to a 'lib/recipetool/licenses.csv' and replace `Unknown` with the license: 02d4002e9171d41a8fad93aa7faf3956,Unknown INFO: Please add the following line for 'vendor/gopkg.in/yaml.v3/LICENSE' to a 'lib/recipetool/licenses.csv' and replace `Unknown` with the license: 3c91c17266710e16afdbb2b6d15c761c,Unknown I believe I have to go through all of them manually and set a proper license in the recipe. Still the questions arises, what should be a better way to reduce amount of this work? Adding more hashes/licenses to lib/recipetool/licenses.csv would be a way to go? I'm just afraid that file might explode if we use it like that... - Could please clarify where does the version from go.mod hide? Is it taken directly from go.mod? I'm trying to understand what should be the workflow when a module version should be bumped up in the go.mod. Will that be reflected in the recipe in any way? Thanks, Slava On 17.10.2023 15:26, lukas.funke-oss@weidmueller.com wrote: > From: Lukas Funke <lukas.funke@weidmueller.com> > > This patch series adds a recipetool handler in order to create 'go' recipes. > Each recipe contains a list of dependencies in their SRC_URI > variable which are derived from the projects `go.mod` file. For each > dependency the corresponding license file uri/hash is added. > > The recipe may not work ad-hoc, but is a good starting point to create > a working recipe and have a working offline-build. > > Lukas Funke (4): > classes: go-vendor: Add go-vendor class > selftest: recipetool: Add test for go recipe handler > recipetool: Ignore *.go files while scanning for licenses > recipetool: Add handler to create go recipes > > meta/classes/go-vendor.bbclass | 135 ++++ > meta/lib/oeqa/selftest/cases/recipetool.py | 163 +++++ > scripts/lib/recipetool/create.py | 2 +- > scripts/lib/recipetool/create_go.py | 730 +++++++++++++++++++++ > 4 files changed, 1029 insertions(+), 1 deletion(-) > create mode 100644 meta/classes/go-vendor.bbclass > create mode 100644 scripts/lib/recipetool/create_go.py >
Hi Slava, On 22.10.2023 20:34, Vyacheslav Yurkov wrote: > Hey Lukas, > Thanks a lot for the patch. A few questions/comments from my initial > test below. > > - I tried it with a go-based backend I have by providing ssh URL to > github. It seems like the GO_IMPORT is set to a module name from go.mod > of my project, which of course fails to fetch it like that, because it's > not a valid URL. How is it supposed to be used? Your assumption is correct: the GO_IMPORT is taken from the go.mod file. I've not considered the case where the repo URL is unequal to the module path. This may require manual rework of the recipe. Another solution would be to take the source URL from the recipetool. I think there is no correct solution to this problem, because probably most people might want to have the original solution, since they are creating recipes from oss components (i.e. from github). > - I've got about 20 lines like: > INFO: Please add the following line for > 'vendor/google.golang.org/protobuf/LICENSE' to a > 'lib/recipetool/licenses.csv' and replace `Unknown` with the license: > 02d4002e9171d41a8fad93aa7faf3956,Unknown > INFO: Please add the following line for > 'vendor/gopkg.in/yaml.v3/LICENSE' to a 'lib/recipetool/licenses.csv' and > replace `Unknown` with the license: > 3c91c17266710e16afdbb2b6d15c761c,Unknown > > I believe I have to go through all of them manually and set a proper > license in the recipe. Still the questions arises, what should be a > better way to reduce amount of this work? Adding more hashes/licenses to > lib/recipetool/licenses.csv would be a way to go? I'm just afraid that > file might explode if we use it like that... The file will explode anyway. Thanks 'go'. Adding more intelligence to the underlying license detection could be an idea: in your first example, the license file contains actually two licenses: MIT and Apache. Yocto has no way to detect those cases. > > - Could please clarify where does the version from go.mod hide? Is it > taken directly from go.mod? I'm trying to understand what should be the > workflow when a module version should be bumped up in the go.mod. Will > that be reflected in the recipe in any way? No, currently the go-version doesn't play a role ATM. Except one case when you have a go.mod file with go < 1.17. These go.mod files don't include indirect dependencies. I hope you might find this helpful and thanks for testing! Cheers, Lukas > > Thanks, > Slava > > On 17.10.2023 15:26, lukas.funke-oss@weidmueller.com wrote: >> From: Lukas Funke <lukas.funke@weidmueller.com> >> >> This patch series adds a recipetool handler in order to create 'go' >> recipes. >> Each recipe contains a list of dependencies in their SRC_URI >> variable which are derived from the projects `go.mod` file. For each >> dependency the corresponding license file uri/hash is added. >> >> The recipe may not work ad-hoc, but is a good starting point to create >> a working recipe and have a working offline-build. >> >> Lukas Funke (4): >> classes: go-vendor: Add go-vendor class >> selftest: recipetool: Add test for go recipe handler >> recipetool: Ignore *.go files while scanning for licenses >> recipetool: Add handler to create go recipes >> >> meta/classes/go-vendor.bbclass | 135 ++++ >> meta/lib/oeqa/selftest/cases/recipetool.py | 163 +++++ >> scripts/lib/recipetool/create.py | 2 +- >> scripts/lib/recipetool/create_go.py | 730 +++++++++++++++++++++ >> 4 files changed, 1029 insertions(+), 1 deletion(-) >> create mode 100644 meta/classes/go-vendor.bbclass >> create mode 100644 scripts/lib/recipetool/create_go.py >> >
On 23.10.2023 14:18, Lukas Funke wrote: > Hi Slava, > > On 22.10.2023 20:34, Vyacheslav Yurkov wrote: >> Hey Lukas, >> Thanks a lot for the patch. A few questions/comments from my initial >> test below. >> >> - I tried it with a go-based backend I have by providing ssh URL to >> github. It seems like the GO_IMPORT is set to a module name from >> go.mod of my project, which of course fails to fetch it like that, >> because it's not a valid URL. How is it supposed to be used? > > Your assumption is correct: the GO_IMPORT is taken from the go.mod file. > > I've not considered the case where the repo URL is unequal to the > module path. This may require manual rework of the recipe. Another > solution would be to take the source URL from the recipetool. I think > there is no correct solution to this problem, because probably most > people might want to have the original solution, since they are > creating recipes from oss components (i.e. from github). Ah, it could be that module name is not used correctly in our backend. I'll try to put a proper URL there, thanks. > > > The file will explode anyway. Thanks 'go'. > > Adding more intelligence to the underlying license detection could be > an idea: in your first example, the license file contains actually two > licenses: MIT and Apache. Yocto has no way to detect those cases. Yeah, I only mentioned 2 out of 20 in my email to not make it huge ;) There are indeed around 20. I'll try to prepare the list and send the patch to include them. > >> >> - Could please clarify where does the version from go.mod hide? Is it >> taken directly from go.mod? I'm trying to understand what should be >> the workflow when a module version should be bumped up in the go.mod. >> Will that be reflected in the recipe in any way? > > No, currently the go-version doesn't play a role ATM. Except one case > when you have a go.mod file with go < 1.17. These go.mod files don't > include indirect dependencies. > Still trying to wrap my head around... When there's no version at parsing stage, how this will affect reproducibility? If it's not known, then whenever the version is bumped up in go.mod, a manual 'clean all' will be required? (It's probably the same as now though). Thanks, Slava
On 23.10.2023 14:18, Lukas Funke wrote: > Hi Slava, > > On 22.10.2023 20:34, Vyacheslav Yurkov wrote: >> Hey Lukas, >> Thanks a lot for the patch. A few questions/comments from my initial >> test below. >> >> - I tried it with a go-based backend I have by providing ssh URL to >> github. It seems like the GO_IMPORT is set to a module name from >> go.mod of my project, which of course fails to fetch it like that, >> because it's not a valid URL. How is it supposed to be used? > > Your assumption is correct: the GO_IMPORT is taken from the go.mod file. > > I've not considered the case where the repo URL is unequal to the > module path. This may require manual rework of the recipe. Another > solution would be to take the source URL from the recipetool. I think > there is no correct solution to this problem, because probably most > people might want to have the original solution, since they are > creating recipes from oss components (i.e. from github). Some more results from my tests. - I refactored module name to contain a valid URL... It seems though that current version of go_src_uri does not handle ssh URLs, and all required info from URL was lost (git@ component, ssh protocol, .git suffix). - I placed the correct URL into SRC_URI, but do_go_vendor still failed with following stacktrace: File: '/home/uvv/projects/yocto-lorch-mapro/openembedded-core/meta/classes/go-vendor.bbclass', lineno: 86, function: do_go_vendor 0082: # path = github.com/foo/bar 0083: # version = v1.2.3 0084: 0085: p = destsuffix[len(default_destsuffix)+1:] *** 0086: path, version = p.split('@') 0087: 0088: subdir = fetcher.ud[url].parm.get('go_subdir') 0089: subdir = "" if not subdir else subdir 0090: Exception: ValueError: not enough values to unpack (expected 2, got 1) The reason is that my go.mod name does not have a version component. If I understood the convention https://go.dev/ref/mod#introduction, it's not a required component, so this should be taken into account. Thanks, Slava
Hi Slava, On 23.10.2023 19:05, Vyacheslav Yurkov wrote: > On 23.10.2023 14:18, Lukas Funke wrote: >> Hi Slava, >> >> On 22.10.2023 20:34, Vyacheslav Yurkov wrote: >>> Hey Lukas, >>> Thanks a lot for the patch. A few questions/comments from my initial >>> test below. >>> >>> - I tried it with a go-based backend I have by providing ssh URL to >>> github. It seems like the GO_IMPORT is set to a module name from >>> go.mod of my project, which of course fails to fetch it like that, >>> because it's not a valid URL. How is it supposed to be used? >> >> Your assumption is correct: the GO_IMPORT is taken from the go.mod file. >> >> I've not considered the case where the repo URL is unequal to the >> module path. This may require manual rework of the recipe. Another >> solution would be to take the source URL from the recipetool. I think >> there is no correct solution to this problem, because probably most >> people might want to have the original solution, since they are >> creating recipes from oss components (i.e. from github). > > Ah, it could be that module name is not used correctly in our backend. > I'll try to put a proper URL there, thanks. > >> >> >> The file will explode anyway. Thanks 'go'. >> >> Adding more intelligence to the underlying license detection could be >> an idea: in your first example, the license file contains actually two >> licenses: MIT and Apache. Yocto has no way to detect those cases. > > Yeah, I only mentioned 2 out of 20 in my email to not make it huge ;) > There are indeed around 20. I'll try to prepare the list and send the > patch to include them. > >> >>> >>> - Could please clarify where does the version from go.mod hide? Is it >>> taken directly from go.mod? I'm trying to understand what should be >>> the workflow when a module version should be bumped up in the go.mod. >>> Will that be reflected in the recipe in any way? >> >> No, currently the go-version doesn't play a role ATM. Except one case >> when you have a go.mod file with go < 1.17. These go.mod files don't >> include indirect dependencies. >> > > Still trying to wrap my head around... When there's no version at > parsing stage, how this will affect reproducibility? If it's not known, > then whenever the version is bumped up in go.mod, a manual 'clean all' > will be required? (It's probably the same as now though). Maybe I don't understand the problem: Is it required for the go module to have the *same* version as the golang package in yocto? In my understanding, when the golang version is greater-equal to the go.mod version we're good? > > Thanks, > Slava
On 23.10.2023 20:06, Vyacheslav Yurkov wrote: > On 23.10.2023 14:18, Lukas Funke wrote: >> Hi Slava, >> >> On 22.10.2023 20:34, Vyacheslav Yurkov wrote: >>> Hey Lukas, >>> Thanks a lot for the patch. A few questions/comments from my initial >>> test below. >>> >>> - I tried it with a go-based backend I have by providing ssh URL to >>> github. It seems like the GO_IMPORT is set to a module name from >>> go.mod of my project, which of course fails to fetch it like that, >>> because it's not a valid URL. How is it supposed to be used? >> >> Your assumption is correct: the GO_IMPORT is taken from the go.mod file. >> >> I've not considered the case where the repo URL is unequal to the >> module path. This may require manual rework of the recipe. Another >> solution would be to take the source URL from the recipetool. I think >> there is no correct solution to this problem, because probably most >> people might want to have the original solution, since they are >> creating recipes from oss components (i.e. from github). > > Some more results from my tests. > > - I refactored module name to contain a valid URL... It seems though > that current version of go_src_uri does not handle ssh URLs, and all > required info from URL was lost (git@ component, ssh protocol, .git > suffix). Currently only https is handled. > > - I placed the correct URL into SRC_URI, but do_go_vendor still failed > with following stacktrace: > > File: > '/home/uvv/projects/yocto-lorch-mapro/openembedded-core/meta/classes/go-vendor.bbclass', lineno: 86, function: do_go_vendor > 0082: # path = github.com/foo/bar > 0083: # version = v1.2.3 > 0084: > 0085: p = destsuffix[len(default_destsuffix)+1:] > *** 0086: path, version = p.split('@') > 0087: > 0088: subdir = fetcher.ud[url].parm.get('go_subdir') > 0089: subdir = "" if not subdir else subdir > 0090: > Exception: ValueError: not enough values to unpack (expected 2, got 1) > > The reason is that my go.mod name does not have a version component. If > I understood the convention https://go.dev/ref/mod#introduction, it's > not a required component, so this should be taken into account. This error could happen if your dependencies don't have a version. I've never seen this in my experiments. Maybe check your go.mod file for the missing version info. > > Thanks, > Slava
Hey Lukas On 24.10.2023 08:33, Lukas Funke wrote: >> >> - I placed the correct URL into SRC_URI, but do_go_vendor still >> failed with following stacktrace: >> >> File: >> '/home/uvv/projects/yocto-lorch-mapro/openembedded-core/meta/classes/go-vendor.bbclass', >> lineno: 86, function: do_go_vendor >> 0082: # path = github.com/foo/bar >> 0083: # version = v1.2.3 >> 0084: >> 0085: p = destsuffix[len(default_destsuffix)+1:] >> *** 0086: path, version = p.split('@') >> 0087: >> 0088: subdir = fetcher.ud[url].parm.get('go_subdir') >> 0089: subdir = "" if not subdir else subdir >> 0090: >> Exception: ValueError: not enough values to unpack (expected 2, got 1) >> >> The reason is that my go.mod name does not have a version component. >> If I understood the convention https://go.dev/ref/mod#introduction, >> it's not a required component, so this should be taken into account. > > This error could happen if your dependencies don't have a version. > I've never seen this in my experiments. Maybe check your go.mod file > for the missing version info. I debugged it a bit and see that the error is actually caused by my URL modification. The URL that works for me looks like SRC_URI = git://git@${GO_IMPORT}.git;... The parsing expects the version after "@", which is not right anymore. Slava
On 24.10.2023 08:19, Lukas Funke wrote: >>> >>>> >>>> - Could please clarify where does the version from go.mod hide? Is >>>> it taken directly from go.mod? I'm trying to understand what should >>>> be the workflow when a module version should be bumped up in the >>>> go.mod. Will that be reflected in the recipe in any way? >>> >>> No, currently the go-version doesn't play a role ATM. Except one >>> case when you have a go.mod file with go < 1.17. These go.mod files >>> don't include indirect dependencies. >>> >> >> Still trying to wrap my head around... When there's no version at >> parsing stage, how this will affect reproducibility? If it's not >> known, then whenever the version is bumped up in go.mod, a manual >> 'clean all' will be required? (It's probably the same as now though). > > Maybe I don't understand the problem: Is it required for the go module > to have the *same* version as the golang package in yocto? In my > understanding, when the golang version is greater-equal to the go.mod > version we're good? I think I mixed up with revisions here a bit. What I meant is how the bitbake would know if versions of dependent components in go.mod have been updated. The easy answer I guess is that the revision of the main recipe (that contains go.mod) needs to be updated for that, and I hope that bitbake would refetch new versions from go.mod, but I didn't check it yet. The more complicated scenario, what if I use a devtool workflow? Will the fetcher be able to reparse go.mod in this case? Slava
On 24.10.2023 09:12, Vyacheslav Yurkov wrote: > Hey Lukas > > On 24.10.2023 08:33, Lukas Funke wrote: >>> >>> - I placed the correct URL into SRC_URI, but do_go_vendor still >>> failed with following stacktrace: >>> >>> File: >>> '/home/uvv/projects/yocto-lorch-mapro/openembedded-core/meta/classes/go-vendor.bbclass', lineno: 86, function: do_go_vendor >>> 0082: # path = github.com/foo/bar >>> 0083: # version = v1.2.3 >>> 0084: >>> 0085: p = destsuffix[len(default_destsuffix)+1:] >>> *** 0086: path, version = p.split('@') >>> 0087: >>> 0088: subdir = fetcher.ud[url].parm.get('go_subdir') >>> 0089: subdir = "" if not subdir else subdir >>> 0090: >>> Exception: ValueError: not enough values to unpack (expected 2, got 1) >>> >>> The reason is that my go.mod name does not have a version component. >>> If I understood the convention https://go.dev/ref/mod#introduction, >>> it's not a required component, so this should be taken into account. >> >> This error could happen if your dependencies don't have a version. >> I've never seen this in my experiments. Maybe check your go.mod file >> for the missing version info. > > I debugged it a bit and see that the error is actually caused by my URL > modification. The URL that works for me looks like > SRC_URI = git://git@${GO_IMPORT}.git;... > > The parsing expects the version after "@", which is not right anymore. The problem here is to distiguish between the actual project SRC_URI and it's dependencies. This is currently done by comparing the SRC_URI entry to the GO_IMPORT variable. If they match then it's not a dependency. But you are correct: this can be solved in a more general manner. Good catch. I'll try to fix it in the next version. > > Slava
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded- > core@lists.openembedded.org> On Behalf Of Lukas Funke > Sent: den 17 oktober 2023 15:27 > To: openembedded-core@lists.openembedded.org > Cc: Bruce Ashfield <bruce.ashfield@gmail.com>; Vyacheslav Yurkov > <uvv.mail@gmail.com>; Martin Jansa <martin.jansa@gmail.com>; Lukas Funke > <lukas.funke@weidmueller.com> > Subject: [OE-Core][PATCH v2 0/4] recipetool: Add handler to create go > recipes > > From: Lukas Funke <lukas.funke@weidmueller.com> > > This patch series adds a recipetool handler in order to create 'go' recipes. > Each recipe contains a list of dependencies in their SRC_URI > variable which are derived from the projects `go.mod` file. For each > dependency the corresponding license file uri/hash is added. > > The recipe may not work ad-hoc, but is a good starting point to create > a working recipe and have a working offline-build. > > Lukas Funke (4): > classes: go-vendor: Add go-vendor class > selftest: recipetool: Add test for go recipe handler > recipetool: Ignore *.go files while scanning for licenses > recipetool: Add handler to create go recipes > > meta/classes/go-vendor.bbclass | 135 ++++ > meta/lib/oeqa/selftest/cases/recipetool.py | 163 +++++ > scripts/lib/recipetool/create.py | 2 +- > scripts/lib/recipetool/create_go.py | 730 +++++++++++++++++++++ > 4 files changed, 1029 insertions(+), 1 deletion(-) > create mode 100644 meta/classes/go-vendor.bbclass > create mode 100644 scripts/lib/recipetool/create_go.py > > -- > 2.30.2 I do not know much (anything) about Go modules, but we have a couple of recipes that use Go modules, and I would like to offer some suggestions. I would recommend to separate the recipe and the main Go module from the dependencies similar to how it is done for Rust by the cargo-update-recipe-crates bbclass, which puts the dependencies in a separate include file. In our recipes we use the following pattern: LICENSE = "... & ${GO_MOD_LICENSES}" LIC_FILES_CHKSUM = "file://src/${GO_IMPORT}/LICENSE;md5=..." require ${BPN}-licenses.inc ... SRC_URI = "..." require ${BPN}-go-mods.inc and then in the ${BPN}-licenses.inc file we have something like: GO_MOD_LICENSES = "Apache-2.0 & BSD-2-Clause & BSD-3-Clause & MIT" LIC_FILES_CHKSUM += "\ ... " and in the ${BPN}-go-mods.inc file: SRC_URI += "\ ... " Separating the three files like the above keeps the automatically generated parts out of the main recipe, which I believe is a good thing. And yes, we have some bbclass and some tool to help with the above, similar to what you are adding in this series. //Peter
From: Lukas Funke <lukas.funke@weidmueller.com> This patch series adds a recipetool handler in order to create 'go' recipes. Each recipe contains a list of dependencies in their SRC_URI variable which are derived from the projects `go.mod` file. For each dependency the corresponding license file uri/hash is added. The recipe may not work ad-hoc, but is a good starting point to create a working recipe and have a working offline-build. Lukas Funke (4): classes: go-vendor: Add go-vendor class selftest: recipetool: Add test for go recipe handler recipetool: Ignore *.go files while scanning for licenses recipetool: Add handler to create go recipes meta/classes/go-vendor.bbclass | 135 ++++ meta/lib/oeqa/selftest/cases/recipetool.py | 163 +++++ scripts/lib/recipetool/create.py | 2 +- scripts/lib/recipetool/create_go.py | 730 +++++++++++++++++++++ 4 files changed, 1029 insertions(+), 1 deletion(-) create mode 100644 meta/classes/go-vendor.bbclass create mode 100644 scripts/lib/recipetool/create_go.py