| Message ID | 20231102155313.309534-1-lukas.funke-oss@weidmueller.com |
|---|---|
| Headers | show |
| Series | recipetool: Add handler to create go recipes | expand |
On Thu, 2023-11-02 at 16:53 +0100, Lukas Funke wrote: > From: Lukas Funke <lukas.funke@weidmueller.com> > > This patch series adds a recipetool handler in order to create 'go' recipes. > Each recipe contains a list of dependencies in their SRC_URI. > Dependencies are derived from the projects `go.mod` file. For each > dependency the corresponding license file uri/hash is added. > > The recipe may not work ad-hoc, but is a good starting point to create > a working recipe and have a working offline-build. > > In addition to the main recipe three additional files will be generated: > - $pn-modules.inc > - $pn-license.inc > - modules.txt > > Changes from v2: > - Generate separate *.inc for go dependencies and licenses > - Adapted oe-selftest according to change above > - Incorparate community suggestions > > Lukas Funke (4): > classes: go-vendor: Add go-vendor class > selftest: recipetool: Add test for go recipe handler > recipetool: Ignore *.go files while scanning for licenses > recipetool: Add handler to create go recipes This looks to be getting there but the code failed in testing: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2371/steps/14/logs/stdio since removeprefix is python 3.9 syntax and 3.8 is our minimum version at present. Cheers, Richard
On Sun, 2023-11-05 at 08:30 +0000, Richard Purdie via lists.openembedded.org wrote: > On Thu, 2023-11-02 at 16:53 +0100, Lukas Funke wrote: > > From: Lukas Funke <lukas.funke@weidmueller.com> > > > > This patch series adds a recipetool handler in order to create 'go' recipes. > > Each recipe contains a list of dependencies in their SRC_URI. > > Dependencies are derived from the projects `go.mod` file. For each > > dependency the corresponding license file uri/hash is added. > > > > The recipe may not work ad-hoc, but is a good starting point to create > > a working recipe and have a working offline-build. > > > > In addition to the main recipe three additional files will be generated: > > - $pn-modules.inc > > - $pn-license.inc > > - modules.txt > > > > Changes from v2: > > - Generate separate *.inc for go dependencies and licenses > > - Adapted oe-selftest according to change above > > - Incorparate community suggestions > > > > Lukas Funke (4): > > classes: go-vendor: Add go-vendor class > > selftest: recipetool: Add test for go recipe handler > > recipetool: Ignore *.go files while scanning for licenses > > recipetool: Add handler to create go recipes > > This looks to be getting there but the code failed in testing: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2371/steps/14/logs/stdio > > since removeprefix is python 3.9 syntax and 3.8 is our minimum version > at present. Since these were through testing, there was no other negative review feedback, the patches appeared to do what I thought we'd all agreed was needed and I had the minor tweak needed for the python version issue I did merge them. I squashed the tweak in with a comment in the commit message. Cheers, Richard
Hi Lukas,
Thanks for the v3. I know I'm late to the party, because it's already
merged, but my testing results are below anyway.
The series seems to have handled my ssh URL correctly.
I've got an error in do_go_vendor though.
File: '<poky>'/meta/classes/go-vendor.bbclass', lineno: 166, function:
do_go_vendor
0162:
0163: # Copy vendor manifest
0164: modules_txt_src = os.path.join(d.getVar('WORKDIR'),
"modules.txt")
0165: bb.debug(1, "cp %s --> %s" % (modules_txt_src, vendor_dir))
*** 0166: shutil.copy2(modules_txt_src, vendor_dir)
0167:
0168: # Clean up vendor dir
0169: # We only require the modules in the modules_txt file
0170: fetched_paths = set([os.path.relpath(x[0], vendor_dir)
for x in os.walk(vendor_dir)])
File: '/usr/lib64/python3.9/shutil.py', lineno: 444, function: copy2
0440: resembles GNU's "cp -P src dst".
0441: """
0442: if os.path.isdir(dst):
0443: dst = os.path.join(dst, os.path.basename(src))
*** 0444: copyfile(src, dst, follow_symlinks=follow_symlinks)
0445: copystat(src, dst, follow_symlinks=follow_symlinks)
0446: return dst
0447:
0448:def ignore_patterns(*patterns):
File: '/usr/lib64/python3.9/shutil.py', lineno: 264, function: copyfile
0260:
0261: if not follow_symlinks and _islink(src):
0262: os.symlink(os.readlink(src), dst)
0263: else:
*** 0264: with open(src, 'rb') as fsrc:
0265: try:
0266: with open(dst, 'wb') as fdst:
0267: # macOS
0268: if _HAS_FCOPYFILE:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'<WORKDIR>/modules.txt'
I've located "modules.txt" in my $WORKDIR/$GO_IMPORT/vendor directory
though, which seems to be consistent with
https://go.dev/ref/mod#go-mod-file-go . Am I missing something or I can
send a fixup?
Slava
On 02.11.2023 16:53, lukas.funke-oss@weidmueller.com wrote:
> From: Lukas Funke <lukas.funke@weidmueller.com>
>
> This patch series adds a recipetool handler in order to create 'go' recipes.
> Each recipe contains a list of dependencies in their SRC_URI.
> Dependencies are derived from the projects `go.mod` file. For each
> dependency the corresponding license file uri/hash is added.
>
> The recipe may not work ad-hoc, but is a good starting point to create
> a working recipe and have a working offline-build.
>
> In addition to the main recipe three additional files will be generated:
> - $pn-modules.inc
> - $pn-license.inc
> - modules.txt
>
> Changes from v2:
> - Generate separate *.inc for go dependencies and licenses
> - Adapted oe-selftest according to change above
> - Incorparate community suggestions
>
> Lukas Funke (4):
> classes: go-vendor: Add go-vendor class
> selftest: recipetool: Add test for go recipe handler
> recipetool: Ignore *.go files while scanning for licenses
> recipetool: Add handler to create go recipes
>
> meta/classes/go-vendor.bbclass | 135 ++++
> meta/lib/oeqa/selftest/cases/recipetool.py | 163 +++++
> scripts/lib/recipetool/create.py | 2 +-
> scripts/lib/recipetool/create_go.py | 730 +++++++++++++++++++++
> 4 files changed, 1029 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes/go-vendor.bbclass
> create mode 100644 scripts/lib/recipetool/create_go.py
>
Hi Slava, On 15.11.2023 06:59, Vyacheslav Yurkov wrote: > Hi Lukas, > Thanks for the v3. I know I'm late to the party, because it's already > merged, but my testing results are below anyway. No problem. Party is still going on. The 'modules.txt' is intendet to be in your ${WORKDIR} as it is part of the SRC_URI, generated by the recipetool. My guess would be, that there is some missconfiguration in your SRC_URI? > > The series seems to have handled my ssh URL correctly. > > I've got an error in do_go_vendor though. > File: '<poky>'/meta/classes/go-vendor.bbclass', lineno: 166, function: > do_go_vendor > 0162: > 0163: # Copy vendor manifest > 0164: modules_txt_src = os.path.join(d.getVar('WORKDIR'), > "modules.txt") > 0165: bb.debug(1, "cp %s --> %s" % (modules_txt_src, vendor_dir)) > *** 0166: shutil.copy2(modules_txt_src, vendor_dir) > 0167: > 0168: # Clean up vendor dir > 0169: # We only require the modules in the modules_txt file > 0170: fetched_paths = set([os.path.relpath(x[0], vendor_dir) > for x in os.walk(vendor_dir)]) > File: '/usr/lib64/python3.9/shutil.py', lineno: 444, function: copy2 > 0440: resembles GNU's "cp -P src dst". > 0441: """ > 0442: if os.path.isdir(dst): > 0443: dst = os.path.join(dst, os.path.basename(src)) > *** 0444: copyfile(src, dst, follow_symlinks=follow_symlinks) > 0445: copystat(src, dst, follow_symlinks=follow_symlinks) > 0446: return dst > 0447: > 0448:def ignore_patterns(*patterns): > File: '/usr/lib64/python3.9/shutil.py', lineno: 264, function: copyfile > 0260: > 0261: if not follow_symlinks and _islink(src): > 0262: os.symlink(os.readlink(src), dst) > 0263: else: > *** 0264: with open(src, 'rb') as fsrc: > 0265: try: > 0266: with open(dst, 'wb') as fdst: > 0267: # macOS > 0268: if _HAS_FCOPYFILE: > Exception: FileNotFoundError: [Errno 2] No such file or directory: > '<WORKDIR>/modules.txt' > > I've located "modules.txt" in my $WORKDIR/$GO_IMPORT/vendor directory > though, which seems to be consistent with > https://go.dev/ref/mod#go-mod-file-go . Am I missing something or I can > send a fixup? > > Slava > > On 02.11.2023 16:53, lukas.funke-oss@weidmueller.com wrote: >> From: Lukas Funke <lukas.funke@weidmueller.com> >> >> This patch series adds a recipetool handler in order to create 'go' >> recipes. >> Each recipe contains a list of dependencies in their SRC_URI. >> Dependencies are derived from the projects `go.mod` file. For each >> dependency the corresponding license file uri/hash is added. >> >> The recipe may not work ad-hoc, but is a good starting point to create >> a working recipe and have a working offline-build. >> >> In addition to the main recipe three additional files will be generated: >> - $pn-modules.inc >> - $pn-license.inc >> - modules.txt >> >> Changes from v2: >> - Generate separate *.inc for go dependencies and licenses >> - Adapted oe-selftest according to change above >> - Incorparate community suggestions >> >> Lukas Funke (4): >> classes: go-vendor: Add go-vendor class >> selftest: recipetool: Add test for go recipe handler >> recipetool: Ignore *.go files while scanning for licenses >> recipetool: Add handler to create go recipes >> >> meta/classes/go-vendor.bbclass | 135 ++++ >> meta/lib/oeqa/selftest/cases/recipetool.py | 163 +++++ >> scripts/lib/recipetool/create.py | 2 +- >> scripts/lib/recipetool/create_go.py | 730 +++++++++++++++++++++ >> 4 files changed, 1029 insertions(+), 1 deletion(-) >> create mode 100644 meta/classes/go-vendor.bbclass >> create mode 100644 scripts/lib/recipetool/create_go.py >> > Best regards Lukas
On 15.11.2023 08:30, Lukas Funke wrote: > Hi Slava, > > No problem. Party is still going on. The 'modules.txt' is intendet to > be in your ${WORKDIR} as it is part of the SRC_URI, generated by > the recipetool. My guess would be, that there is some > missconfiguration in your SRC_URI? Yes, I noticed that it should be there in recipetool tests. But generated recipe didn't have It in SRC_URI. I tried to use generated recipe as is. I'll try to double check later. Slava
Hi Lukas, Sorry for the late reply. I think I see what's going on. The module I'm working on is already "vendored", which means there's no modules.txt created for my project by recipetool, since it's already provided in the repository itself. I guess this has pros and cons, and I'm currently not sure which approach is better. On one hand, offline vendoring (no by the recipetool) has an advantage for SDK builds, when people don't use the BSP directly. On the other hand, Yocto style vendoring (when modules.txt is stored in the BSP) is more transparent for dependency management. I.e. I would need to update the recipe when my dependencies change. Slava On 15.11.2023 08:34, Vyacheslav Yurkov via lists.openembedded.org wrote: > On 15.11.2023 08:30, Lukas Funke wrote: >> Hi Slava, >> >> No problem. Party is still going on. The 'modules.txt' is intendet to >> be in your ${WORKDIR} as it is part of the SRC_URI, generated by >> the recipetool. My guess would be, that there is some >> missconfiguration in your SRC_URI? > > Yes, I noticed that it should be there in recipetool tests. But > generated recipe didn't have It in SRC_URI. I tried to use generated > recipe as is. I'll try to double check later. > > Slava
From: Lukas Funke <lukas.funke@weidmueller.com> This patch series adds a recipetool handler in order to create 'go' recipes. Each recipe contains a list of dependencies in their SRC_URI. Dependencies are derived from the projects `go.mod` file. For each dependency the corresponding license file uri/hash is added. The recipe may not work ad-hoc, but is a good starting point to create a working recipe and have a working offline-build. In addition to the main recipe three additional files will be generated: - $pn-modules.inc - $pn-license.inc - modules.txt Changes from v2: - Generate separate *.inc for go dependencies and licenses - Adapted oe-selftest according to change above - Incorparate community suggestions Lukas Funke (4): classes: go-vendor: Add go-vendor class selftest: recipetool: Add test for go recipe handler recipetool: Ignore *.go files while scanning for licenses recipetool: Add handler to create go recipes meta/classes/go-vendor.bbclass | 135 ++++ meta/lib/oeqa/selftest/cases/recipetool.py | 163 +++++ scripts/lib/recipetool/create.py | 2 +- scripts/lib/recipetool/create_go.py | 730 +++++++++++++++++++++ 4 files changed, 1029 insertions(+), 1 deletion(-) create mode 100644 meta/classes/go-vendor.bbclass create mode 100644 scripts/lib/recipetool/create_go.py