Message ID | 20240116165602.2053194-1-William.Lyu@windriver.com |
---|---|
State | New |
Headers | show |
Series | [1/2] perl: Add option to prevent COMPLEMENTARY_GLOB from ignoring perl-modules | expand |
If this is only for fixing OpenSSL ptest, then it’s not needed perhaps, once OpenSSL recipe is fixed? Alex On Tue 16. Jan 2024 at 17.56, Lyu, William via lists.openembedded.org <william.lyu=windriver.com@lists.openembedded.org> wrote: > From: William Lyu <William.Lyu@windriver.com> > > When perl-modules is added via COMPLEMENTARY_GLOB, all perl-module-* > packages recommended by perl-modules are ignored due to the defined > behavior of COMPLEMENTARY_GLOB. > > This patch creates a new package perl-modules-force-all that RDEPENDS on > the same set of packages perl-modules recommends. If another recipe > always needs the packages recommended by perl-modules, it can add > perl-modules-force-all to RDEPENDS to make sure that the recommended > packages by perl-modules are not ignored by COMPLEMENTARY_GLOB. > > Signed-off-by: William Lyu <William.Lyu@windriver.com> > --- > meta/recipes-devtools/perl/perl_5.38.2.bb | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > > diff --git a/meta/recipes-devtools/perl/perl_5.38.2.bb > b/meta/recipes-devtools/perl/perl_5.38.2.bb > index a9d684cfc5..f75c5aa971 100644 > --- a/meta/recipes-devtools/perl/perl_5.38.2.bb > +++ b/meta/recipes-devtools/perl/perl_5.38.2.bb > @@ -311,6 +311,14 @@ ALTERNATIVE_LINK_NAME[Thread.3] = > "${mandir}/man3/Thread.3" > ALLOW_EMPTY:${PN}-modules = "1" > PACKAGES += "${PN}-modules " > > +# When perl-modules is added through COMPLEMENTARY_GLOB, as it is > recommending > +# packages, all its recommendations will be ignored. So, we create a > +# perl-modules-force-all package depending on the same set of packages as > listed > +# in perl-modules to provide a way to guarantee COMPLEMENTARY_GLOB not > removing > +# these packages. > +ALLOW_EMPTY:${PN}-modules-force-all = "1" > +PACKAGES += "${PN}-modules-force-all " > + > PACKAGESPLITFUNCS =+ "split_perl_packages" > > python split_perl_packages () { > @@ -328,6 +336,20 @@ python split_perl_packages () { > packages = filter(lambda p: 'perl-module-' in p, > d.getVar('PACKAGES').split()) > d.setVar(d.expand("RRECOMMENDS:${PN}-modules"), ' '.join(packages)) > > + # perl-modules-force-all provides a way to prevent COMPLEMENTARY_GLOB > + # from removing the packages specified in perl-modules. > + # perl-modules-force-all contains the same set of packages as > perl-modules. > + d.setVar( > + d.expand("RDEPENDS:${PN}-modules-force-all"), > + d.getVar(d.expand("RRECOMMENDS:${PN}-modules")) > + ) > + > + # As perl-modules-force-all depends on all other, potentially > non-existent, > + # perl packages, we allow these packages to be empty to prevent build > + # errors. > + for pkg in > d.getVar(d.expand("RDEPENDS:${PN}-modules-force-all")).split(): > + d.setVar(d.expand("ALLOW_EMPTY:" + pkg), "1") > + > # Read the pre-generated dependency file, and use it to set module > dependecies > for line in open(d.expand("${WORKDIR}") + > '/perl-rdepends.txt').readlines(): > splitline = line.split() > -- > 2.43.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#193867): > https://lists.openembedded.org/g/openembedded-core/message/193867 > Mute This Topic: https://lists.openembedded.org/mt/103766391/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
I have iteratively added the following 45 modules, and I still not at the point of covering all perl-module-* dependencies for openssl-test. Should I continue on with this effort? Using "perl-module-force-all" is a lot more convenient without conflicting with any existing usage of "perl-modules" - even including the ones that depends on "perl-modules" having the property of being a soft dependency ("perl-modules" RRECOMMENDS all of its packages). William List of perl-module-* I have iteratively added up to this point: perl-module-base perl-module-carp perl-module-config perl-module-constant perl-module-cwd perl-module-data-dumper perl-module-digest-sha perl-module-encode perl-module-errno perl-module-exporter perl-module-fcntl perl-module-feature perl-module-file-basename perl-module-file-compare perl-module-file-copy perl-module-file-glob perl-module-filehandle perl-module-file-path perl-module-file-spec perl-module-file-spec-functions perl-module-file-temp perl-module-findbin perl-module-getopt-long perl-module-getopt-std perl-module-io-select perl-module-io-socket perl-module-ipc-cmd perl-module-ipc-open2 perl-module-lib perl-module-list-util perl-module-math-bigint perl-module-mime-base64 perl-module-parent perl-module-pod-html perl-module-pod-usage perl-module-posix perl-module-scalar-util perl-module-storable perl-module-strict perl-module-test-harness perl-module-test-more perl-module-time-local perl-module-utf8 perl-module-vars perl-module-warnings
How are you making the list? I think this might be somewhat sped up, if you first grep ${WORKDIR}/packages-split/openssl-ptest/usr/lib/openssl/ptest/test/ (and util/perl/OpenSSL/ as well) for "^use " and transform the output lines into 'perl-module-something'. Then try it on the target in qemu to see whether something is missing. Alex On Wed, 17 Jan 2024 at 23:44, Lyu, William via lists.openembedded.org <william.lyu=windriver.com@lists.openembedded.org> wrote: > > I have iteratively added the following 45 modules, and I still not at the point of covering all perl-module-* dependencies for openssl-test. Should I continue on with this effort? Using "perl-module-force-all" is a lot more convenient without conflicting with any existing usage of "perl-modules" - even including the ones that depends on "perl-modules" having the property of being a soft dependency ("perl-modules" RRECOMMENDS all of its packages). > > William > > List of perl-module-* I have iteratively added up to this point: > > perl-module-base > perl-module-carp > perl-module-config > perl-module-constant > perl-module-cwd > perl-module-data-dumper > perl-module-digest-sha > perl-module-encode > perl-module-errno > perl-module-exporter > perl-module-fcntl > perl-module-feature > perl-module-file-basename > perl-module-file-compare > perl-module-file-copy > perl-module-file-glob > perl-module-filehandle > perl-module-file-path > perl-module-file-spec > perl-module-file-spec-functions > perl-module-file-temp > perl-module-findbin > perl-module-getopt-long > perl-module-getopt-std > perl-module-io-select > perl-module-io-socket > perl-module-ipc-cmd > perl-module-ipc-open2 > perl-module-lib > perl-module-list-util > perl-module-math-bigint > perl-module-mime-base64 > perl-module-parent > perl-module-pod-html > perl-module-pod-usage > perl-module-posix > perl-module-scalar-util > perl-module-storable > perl-module-strict > perl-module-test-harness > perl-module-test-more > perl-module-time-local > perl-module-utf8 > perl-module-vars > perl-module-warnings > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#193949): https://lists.openembedded.org/g/openembedded-core/message/193949 > Mute This Topic: https://lists.openembedded.org/mt/103766391/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Following your suggestions and iteratively adding packages based on error message, I am finally able to come up with quite a long list of perl-module-* packages that fix the dependencies of openssl-ptest. However, I am worried about if the list I came up with is truly comprehensive in the sense that this list covers all dependencies for other architectures and other OSes. I see that a lot of openssl-ptest subtests are marked as "skipped". I am not sure if these dependencies will be sufficient if these subtests are not skipped. Also, I am not sure if the packages I added are arch or OS specific. The following is the list of perl-module-* I found. This first portion is identified by following your suggestion of using grep "^\s*use". perl-module-extutils-makemaker \ perl-module-file-find \ perl-module-if \ perl-module-pod-checker \ perl-module-tap-harness \ perl-module-tap-parser-aggregator \ perl-module-tap-parser-source \ perl-module-tap-parser-sourcehandler-perl \ perl-module-text-parsewords \ perl-module-carp \ perl-module-cwd \ perl-module-data-dumper \ perl-module-digest-sha \ perl-module-encode \ perl-module-errno \ perl-module-exporter \ perl-module-fcntl \ perl-module-feature \ perl-module-file-basename \ perl-module-file-compare \ perl-module-file-copy \ perl-module-file-glob \ perl-module-filehandle \ perl-module-file-path \ perl-module-file-spec \ perl-module-file-spec-functions \ perl-module-file-temp \ perl-module-findbin \ perl-module-getopt-long \ perl-module-getopt-std \ perl-module-io-select \ perl-module-io-socket \ perl-module-ipc-cmd \ perl-module-ipc-open2 \ perl-module-lib \ perl-module-list-util \ perl-module-math-bigint \ perl-module-mime-base64 \ perl-module-pod-html \ perl-module-posix \ perl-module-scalar-util \ perl-module-storable \ perl-module-test-harness \ perl-module-test-more \ perl-module-time-local \ perl-module-utf8 \ perl-module-vars \ \ The following are added iteratively based on error reports from actually running the test. perl-module-pod-text \ perl-module-encode-encoding \ perl-module-metadata \ perl-module-locale \ \ The following are added due to internal dependencies between different TAP modules. I added a few iteratively, and there are still some errors. So, I just grepped and added all tap modules. perl-module-tap-base \ perl-module-tap-formatter-base \ perl-module-tap-formatter-color \ perl-module-tap-formatter-console \ perl-module-tap-formatter-console-parallelsession \ perl-module-tap-formatter-console-session \ perl-module-tap-formatter-file \ perl-module-tap-formatter-file-session \ perl-module-tap-formatter-session \ perl-module-tap-harness \ perl-module-tap-harness-env \ perl-module-tap-object \ perl-module-tap-parser \ perl-module-tap-parser-aggregator \ perl-module-tap-parser-grammar \ perl-module-tap-parser-iterator \ perl-module-tap-parser-iterator-array \ perl-module-tap-parser-iteratorfactory \ perl-module-tap-parser-iterator-process \ perl-module-tap-parser-iterator-stream \ perl-module-tap-parser-multiplexer \ perl-module-tap-parser-result \ perl-module-tap-parser-result-bailout \ perl-module-tap-parser-result-comment \ perl-module-tap-parser-resultfactory \ perl-module-tap-parser-result-plan \ perl-module-tap-parser-result-pragma \ perl-module-tap-parser-result-test \ perl-module-tap-parser-result-unknown \ perl-module-tap-parser-result-version \ perl-module-tap-parser-result-yaml \ perl-module-tap-parser-scheduler \ perl-module-tap-parser-scheduler-job \ perl-module-tap-parser-scheduler-spinner \ perl-module-tap-parser-source \ perl-module-tap-parser-sourcehandler \ perl-module-tap-parser-sourcehandler-executable \ perl-module-tap-parser-sourcehandler-file \ perl-module-tap-parser-sourcehandler-handle \ perl-module-tap-parser-sourcehandler-perl \ perl-module-tap-parser-sourcehandler-rawtap \ perl-module-tap-parser-yamlish-reader \ perl-module-tap-parser-yamlish-writer \ \ Only perl-module-extutils-mm-unix was requested. However, I noticed that the suffix has "unix", so I assumed that this module is OS dependant. In case openssl-ptest needs to be built for other OSes, I added all perl-module-extutils-mm*. perl-module-extutils-mm \ perl-module-extutils-mm-aix \ perl-module-extutils-mm-any \ perl-module-extutils-mm-beos \ perl-module-extutils-mm-cygwin \ perl-module-extutils-mm-darwin \ perl-module-extutils-mm-dos \ perl-module-extutils-mm-macos \ perl-module-extutils-mm-nw5 \ perl-module-extutils-mm-os2 \ perl-module-extutils-mm-os390 \ perl-module-extutils-mm-qnx \ perl-module-extutils-mm-unix \ perl-module-extutils-mm-uwin \ perl-module-extutils-mm-vms \ perl-module-extutils-mm-vos \ perl-module-extutils-mm-win32 \ perl-module-extutils-mm-win95 \ ===== APPENDIX: Commands I Used to Identify the Potential List of Packages Needed ===== These are details and can be ignored. Under "{build directory}/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/rootfs/usr/lib/openssl/ptest", I executed the following command to convert all imported modules using "use" statements into "perl-module-*": $ grep -r -n -I "^\s*use " | sed -E 's/.*:[0-9]+://' | sed -E 's/;/\n/g' | sed -E 's/^ *use *//' | sed -E 's/ /\n/g' | awk '!/^\s*$/' | tr '[:upper:]' '[:lower:]' | sed -E 's/::/-/g' | sed -E 's/^/perl-module-/' | sort | uniq I saved the above list into a file "PARSED_USE". Then, I listed all directories under "tmp/work/core2-64-poky-linux/perl/5.38.2/packages-split" and saved this list as a list of valid perl-module-* into a file "VALID_MODULE". I used the following command to determine which line in file "PARSED_USE" is a valid perl-module-* based on "VALID_MODULE" using the following command: $ cat PARSED_USE VALID_MODULE | sort | uniq -c | sort -nk1,1 | grep -E '^ *2'
Thank you for doing this deep level investigation. I think this pretty much shows that listing all modules in a way that can be maintained isn't feasible (the idea was that maybe there's only a few of them needed for ptest). Which brings us back to the question: why is perl-modules using RRECOMMENDS and not RDEPENDS? I went through git history, and perl recipe had RRECOMMENDS from the very first mass-import commit in 2005, so the answer is likely no one knows or remembers. I also went and checked python: in there python3-modules is using RDEPENDS. So I think the easiest way out is to simply replace RRECOMMENDS with RDEPENDS in perl as well. Can you make a patch? Alex On Fri, 19 Jan 2024 at 19:11, Lyu, William via lists.openembedded.org <william.lyu=windriver.com@lists.openembedded.org> wrote: > > Following your suggestions and iteratively adding packages based on error message, I am finally able to come up with quite a long list of perl-module-* packages that fix the dependencies of openssl-ptest. > > > However, I am worried about if the list I came up with is truly comprehensive in the sense that this list covers all dependencies for other architectures and other OSes. I see that a lot of openssl-ptest subtests are marked as "skipped". I am not sure if these dependencies will be sufficient if these subtests are not skipped. Also, I am not sure if the packages I added are arch or OS specific. > > The following is the list of perl-module-* I found. > > This first portion is identified by following your suggestion of using grep "^\s*use". > perl-module-extutils-makemaker \ > perl-module-file-find \ > perl-module-if \ > perl-module-pod-checker \ > perl-module-tap-harness \ > perl-module-tap-parser-aggregator \ > perl-module-tap-parser-source \ > perl-module-tap-parser-sourcehandler-perl \ > perl-module-text-parsewords \ > perl-module-carp \ > perl-module-cwd \ > perl-module-data-dumper \ > perl-module-digest-sha \ > perl-module-encode \ > perl-module-errno \ > perl-module-exporter \ > perl-module-fcntl \ > perl-module-feature \ > perl-module-file-basename \ > perl-module-file-compare \ > perl-module-file-copy \ > perl-module-file-glob \ > perl-module-filehandle \ > perl-module-file-path \ > perl-module-file-spec \ > perl-module-file-spec-functions \ > perl-module-file-temp \ > perl-module-findbin \ > perl-module-getopt-long \ > perl-module-getopt-std \ > perl-module-io-select \ > perl-module-io-socket \ > perl-module-ipc-cmd \ > perl-module-ipc-open2 \ > perl-module-lib \ > perl-module-list-util \ > perl-module-math-bigint \ > perl-module-mime-base64 \ > perl-module-pod-html \ > perl-module-posix \ > perl-module-scalar-util \ > perl-module-storable \ > perl-module-test-harness \ > perl-module-test-more \ > perl-module-time-local \ > perl-module-utf8 \ > perl-module-vars \ > \ > > The following are added iteratively based on error reports from actually running the test. > perl-module-pod-text \ > perl-module-encode-encoding \ > perl-module-metadata \ > perl-module-locale \ > \ > > The following are added due to internal dependencies between different TAP modules. I added a few iteratively, and there are still some errors. So, I just grepped and added all tap modules. > perl-module-tap-base \ > perl-module-tap-formatter-base \ > perl-module-tap-formatter-color \ > perl-module-tap-formatter-console \ > perl-module-tap-formatter-console-parallelsession \ > perl-module-tap-formatter-console-session \ > perl-module-tap-formatter-file \ > perl-module-tap-formatter-file-session \ > perl-module-tap-formatter-session \ > perl-module-tap-harness \ > perl-module-tap-harness-env \ > perl-module-tap-object \ > perl-module-tap-parser \ > perl-module-tap-parser-aggregator \ > perl-module-tap-parser-grammar \ > perl-module-tap-parser-iterator \ > perl-module-tap-parser-iterator-array \ > perl-module-tap-parser-iteratorfactory \ > perl-module-tap-parser-iterator-process \ > perl-module-tap-parser-iterator-stream \ > perl-module-tap-parser-multiplexer \ > perl-module-tap-parser-result \ > perl-module-tap-parser-result-bailout \ > perl-module-tap-parser-result-comment \ > perl-module-tap-parser-resultfactory \ > perl-module-tap-parser-result-plan \ > perl-module-tap-parser-result-pragma \ > perl-module-tap-parser-result-test \ > perl-module-tap-parser-result-unknown \ > perl-module-tap-parser-result-version \ > perl-module-tap-parser-result-yaml \ > perl-module-tap-parser-scheduler \ > perl-module-tap-parser-scheduler-job \ > perl-module-tap-parser-scheduler-spinner \ > perl-module-tap-parser-source \ > perl-module-tap-parser-sourcehandler \ > perl-module-tap-parser-sourcehandler-executable \ > perl-module-tap-parser-sourcehandler-file \ > perl-module-tap-parser-sourcehandler-handle \ > perl-module-tap-parser-sourcehandler-perl \ > perl-module-tap-parser-sourcehandler-rawtap \ > perl-module-tap-parser-yamlish-reader \ > perl-module-tap-parser-yamlish-writer \ > \ > > Only perl-module-extutils-mm-unix was requested. However, I noticed that the suffix has "unix", so I assumed that this module is OS dependant. In case openssl-ptest needs to be built for other OSes, I added all perl-module-extutils-mm*. > perl-module-extutils-mm \ > perl-module-extutils-mm-aix \ > perl-module-extutils-mm-any \ > perl-module-extutils-mm-beos \ > perl-module-extutils-mm-cygwin \ > perl-module-extutils-mm-darwin \ > perl-module-extutils-mm-dos \ > perl-module-extutils-mm-macos \ > perl-module-extutils-mm-nw5 \ > perl-module-extutils-mm-os2 \ > perl-module-extutils-mm-os390 \ > perl-module-extutils-mm-qnx \ > perl-module-extutils-mm-unix \ > perl-module-extutils-mm-uwin \ > perl-module-extutils-mm-vms \ > perl-module-extutils-mm-vos \ > perl-module-extutils-mm-win32 \ > perl-module-extutils-mm-win95 \ > > ===== APPENDIX: Commands I Used to Identify the Potential List of Packages Needed ===== > These are details and can be ignored. > > Under "{build directory}/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/rootfs/usr/lib/openssl/ptest", I executed the following command to convert all imported modules using "use" statements into "perl-module-*": > $ grep -r -n -I "^\s*use " | sed -E 's/.*:[0-9]+://' | sed -E 's/;/\n/g' | sed -E 's/^ *use *//' | sed -E 's/ /\n/g' | awk '!/^\s*$/' | tr '[:upper:]' '[:lower:]' | sed -E 's/::/-/g' | sed -E 's/^/perl-module-/' | sort | uniq > > I saved the above list into a file "PARSED_USE". Then, I listed all directories under "tmp/work/core2-64-poky-linux/perl/5.38.2/packages-split" and saved this list as a list of valid perl-module-* into a file "VALID_MODULE". > > I used the following command to determine which line in file "PARSED_USE" is a valid perl-module-* based on "VALID_MODULE" using the following command: > $ cat PARSED_USE VALID_MODULE | sort | uniq -c | sort -nk1,1 | grep -E '^ *2' > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#194062): https://lists.openembedded.org/g/openembedded-core/message/194062 > Mute This Topic: https://lists.openembedded.org/mt/103766391/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/meta/recipes-devtools/perl/perl_5.38.2.bb b/meta/recipes-devtools/perl/perl_5.38.2.bb index a9d684cfc5..f75c5aa971 100644 --- a/meta/recipes-devtools/perl/perl_5.38.2.bb +++ b/meta/recipes-devtools/perl/perl_5.38.2.bb @@ -311,6 +311,14 @@ ALTERNATIVE_LINK_NAME[Thread.3] = "${mandir}/man3/Thread.3" ALLOW_EMPTY:${PN}-modules = "1" PACKAGES += "${PN}-modules " +# When perl-modules is added through COMPLEMENTARY_GLOB, as it is recommending +# packages, all its recommendations will be ignored. So, we create a +# perl-modules-force-all package depending on the same set of packages as listed +# in perl-modules to provide a way to guarantee COMPLEMENTARY_GLOB not removing +# these packages. +ALLOW_EMPTY:${PN}-modules-force-all = "1" +PACKAGES += "${PN}-modules-force-all " + PACKAGESPLITFUNCS =+ "split_perl_packages" python split_perl_packages () { @@ -328,6 +336,20 @@ python split_perl_packages () { packages = filter(lambda p: 'perl-module-' in p, d.getVar('PACKAGES').split()) d.setVar(d.expand("RRECOMMENDS:${PN}-modules"), ' '.join(packages)) + # perl-modules-force-all provides a way to prevent COMPLEMENTARY_GLOB + # from removing the packages specified in perl-modules. + # perl-modules-force-all contains the same set of packages as perl-modules. + d.setVar( + d.expand("RDEPENDS:${PN}-modules-force-all"), + d.getVar(d.expand("RRECOMMENDS:${PN}-modules")) + ) + + # As perl-modules-force-all depends on all other, potentially non-existent, + # perl packages, we allow these packages to be empty to prevent build + # errors. + for pkg in d.getVar(d.expand("RDEPENDS:${PN}-modules-force-all")).split(): + d.setVar(d.expand("ALLOW_EMPTY:" + pkg), "1") + # Read the pre-generated dependency file, and use it to set module dependecies for line in open(d.expand("${WORKDIR}") + '/perl-rdepends.txt').readlines(): splitline = line.split()