| Message ID | 20240116165602.2053194-1-William.Lyu@windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] perl: Add option to prevent COMPLEMENTARY_GLOB from ignoring perl-modules | expand |
If this is only for fixing OpenSSL ptest, then it’s not needed perhaps, once OpenSSL recipe is fixed? Alex On Tue 16. Jan 2024 at 17.56, Lyu, William via lists.openembedded.org <william.lyu=windriver.com@lists.openembedded.org> wrote: > From: William Lyu <William.Lyu@windriver.com> > > When perl-modules is added via COMPLEMENTARY_GLOB, all perl-module-* > packages recommended by perl-modules are ignored due to the defined > behavior of COMPLEMENTARY_GLOB. > > This patch creates a new package perl-modules-force-all that RDEPENDS on > the same set of packages perl-modules recommends. If another recipe > always needs the packages recommended by perl-modules, it can add > perl-modules-force-all to RDEPENDS to make sure that the recommended > packages by perl-modules are not ignored by COMPLEMENTARY_GLOB. > > Signed-off-by: William Lyu <William.Lyu@windriver.com> > --- > meta/recipes-devtools/perl/perl_5.38.2.bb | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > > diff --git a/meta/recipes-devtools/perl/perl_5.38.2.bb > b/meta/recipes-devtools/perl/perl_5.38.2.bb > index a9d684cfc5..f75c5aa971 100644 > --- a/meta/recipes-devtools/perl/perl_5.38.2.bb > +++ b/meta/recipes-devtools/perl/perl_5.38.2.bb > @@ -311,6 +311,14 @@ ALTERNATIVE_LINK_NAME[Thread.3] = > "${mandir}/man3/Thread.3" > ALLOW_EMPTY:${PN}-modules = "1" > PACKAGES += "${PN}-modules " > > +# When perl-modules is added through COMPLEMENTARY_GLOB, as it is > recommending > +# packages, all its recommendations will be ignored. So, we create a > +# perl-modules-force-all package depending on the same set of packages as > listed > +# in perl-modules to provide a way to guarantee COMPLEMENTARY_GLOB not > removing > +# these packages. > +ALLOW_EMPTY:${PN}-modules-force-all = "1" > +PACKAGES += "${PN}-modules-force-all " > + > PACKAGESPLITFUNCS =+ "split_perl_packages" > > python split_perl_packages () { > @@ -328,6 +336,20 @@ python split_perl_packages () { > packages = filter(lambda p: 'perl-module-' in p, > d.getVar('PACKAGES').split()) > d.setVar(d.expand("RRECOMMENDS:${PN}-modules"), ' '.join(packages)) > > + # perl-modules-force-all provides a way to prevent COMPLEMENTARY_GLOB > + # from removing the packages specified in perl-modules. > + # perl-modules-force-all contains the same set of packages as > perl-modules. > + d.setVar( > + d.expand("RDEPENDS:${PN}-modules-force-all"), > + d.getVar(d.expand("RRECOMMENDS:${PN}-modules")) > + ) > + > + # As perl-modules-force-all depends on all other, potentially > non-existent, > + # perl packages, we allow these packages to be empty to prevent build > + # errors. > + for pkg in > d.getVar(d.expand("RDEPENDS:${PN}-modules-force-all")).split(): > + d.setVar(d.expand("ALLOW_EMPTY:" + pkg), "1") > + > # Read the pre-generated dependency file, and use it to set module > dependecies > for line in open(d.expand("${WORKDIR}") + > '/perl-rdepends.txt').readlines(): > splitline = line.split() > -- > 2.43.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#193867): > https://lists.openembedded.org/g/openembedded-core/message/193867 > Mute This Topic: https://lists.openembedded.org/mt/103766391/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
I have iteratively added the following 45 modules, and I still not at the point of covering all perl-module-* dependencies for openssl-test. Should I continue on with this effort? Using "perl-module-force-all" is a lot more convenient without conflicting with any existing usage of "perl-modules" - even including the ones that depends on "perl-modules" having the property of being a soft dependency ("perl-modules" RRECOMMENDS all of its packages).
William
List of perl-module-* I have iteratively added up to this point:
perl-module-base
perl-module-carp
perl-module-config
perl-module-constant
perl-module-cwd
perl-module-data-dumper
perl-module-digest-sha
perl-module-encode
perl-module-errno
perl-module-exporter
perl-module-fcntl
perl-module-feature
perl-module-file-basename
perl-module-file-compare
perl-module-file-copy
perl-module-file-glob
perl-module-filehandle
perl-module-file-path
perl-module-file-spec
perl-module-file-spec-functions
perl-module-file-temp
perl-module-findbin
perl-module-getopt-long
perl-module-getopt-std
perl-module-io-select
perl-module-io-socket
perl-module-ipc-cmd
perl-module-ipc-open2
perl-module-lib
perl-module-list-util
perl-module-math-bigint
perl-module-mime-base64
perl-module-parent
perl-module-pod-html
perl-module-pod-usage
perl-module-posix
perl-module-scalar-util
perl-module-storable
perl-module-strict
perl-module-test-harness
perl-module-test-more
perl-module-time-local
perl-module-utf8
perl-module-vars
perl-module-warnings
How are you making the list? I think this might be somewhat sped up,
if you first grep
${WORKDIR}/packages-split/openssl-ptest/usr/lib/openssl/ptest/test/
(and util/perl/OpenSSL/ as well) for "^use " and transform the output
lines into 'perl-module-something'. Then try it on the target in qemu
to see whether something is missing.
Alex
On Wed, 17 Jan 2024 at 23:44, Lyu, William via lists.openembedded.org
<william.lyu=windriver.com@lists.openembedded.org> wrote:
>
> I have iteratively added the following 45 modules, and I still not at the point of covering all perl-module-* dependencies for openssl-test. Should I continue on with this effort? Using "perl-module-force-all" is a lot more convenient without conflicting with any existing usage of "perl-modules" - even including the ones that depends on "perl-modules" having the property of being a soft dependency ("perl-modules" RRECOMMENDS all of its packages).
>
> William
>
> List of perl-module-* I have iteratively added up to this point:
>
> perl-module-base
> perl-module-carp
> perl-module-config
> perl-module-constant
> perl-module-cwd
> perl-module-data-dumper
> perl-module-digest-sha
> perl-module-encode
> perl-module-errno
> perl-module-exporter
> perl-module-fcntl
> perl-module-feature
> perl-module-file-basename
> perl-module-file-compare
> perl-module-file-copy
> perl-module-file-glob
> perl-module-filehandle
> perl-module-file-path
> perl-module-file-spec
> perl-module-file-spec-functions
> perl-module-file-temp
> perl-module-findbin
> perl-module-getopt-long
> perl-module-getopt-std
> perl-module-io-select
> perl-module-io-socket
> perl-module-ipc-cmd
> perl-module-ipc-open2
> perl-module-lib
> perl-module-list-util
> perl-module-math-bigint
> perl-module-mime-base64
> perl-module-parent
> perl-module-pod-html
> perl-module-pod-usage
> perl-module-posix
> perl-module-scalar-util
> perl-module-storable
> perl-module-strict
> perl-module-test-harness
> perl-module-test-more
> perl-module-time-local
> perl-module-utf8
> perl-module-vars
> perl-module-warnings
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#193949): https://lists.openembedded.org/g/openembedded-core/message/193949
> Mute This Topic: https://lists.openembedded.org/mt/103766391/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Following your suggestions and iteratively adding packages based on error message, I am finally able to come up with quite a long list of perl-module-* packages that fix the dependencies of openssl-ptest.
However, I am worried about if the list I came up with is truly comprehensive in the sense that this list covers all dependencies for other architectures and other OSes. I see that a lot of openssl-ptest subtests are marked as "skipped". I am not sure if these dependencies will be sufficient if these subtests are not skipped. Also, I am not sure if the packages I added are arch or OS specific.
The following is the list of perl-module-* I found.
This first portion is identified by following your suggestion of using grep "^\s*use".
perl-module-extutils-makemaker \
perl-module-file-find \
perl-module-if \
perl-module-pod-checker \
perl-module-tap-harness \
perl-module-tap-parser-aggregator \
perl-module-tap-parser-source \
perl-module-tap-parser-sourcehandler-perl \
perl-module-text-parsewords \
perl-module-carp \
perl-module-cwd \
perl-module-data-dumper \
perl-module-digest-sha \
perl-module-encode \
perl-module-errno \
perl-module-exporter \
perl-module-fcntl \
perl-module-feature \
perl-module-file-basename \
perl-module-file-compare \
perl-module-file-copy \
perl-module-file-glob \
perl-module-filehandle \
perl-module-file-path \
perl-module-file-spec \
perl-module-file-spec-functions \
perl-module-file-temp \
perl-module-findbin \
perl-module-getopt-long \
perl-module-getopt-std \
perl-module-io-select \
perl-module-io-socket \
perl-module-ipc-cmd \
perl-module-ipc-open2 \
perl-module-lib \
perl-module-list-util \
perl-module-math-bigint \
perl-module-mime-base64 \
perl-module-pod-html \
perl-module-posix \
perl-module-scalar-util \
perl-module-storable \
perl-module-test-harness \
perl-module-test-more \
perl-module-time-local \
perl-module-utf8 \
perl-module-vars \
\
The following are added iteratively based on error reports from actually running the test.
perl-module-pod-text \
perl-module-encode-encoding \
perl-module-metadata \
perl-module-locale \
\
The following are added due to internal dependencies between different TAP modules. I added a few iteratively, and there are still some errors. So, I just grepped and added all tap modules.
perl-module-tap-base \
perl-module-tap-formatter-base \
perl-module-tap-formatter-color \
perl-module-tap-formatter-console \
perl-module-tap-formatter-console-parallelsession \
perl-module-tap-formatter-console-session \
perl-module-tap-formatter-file \
perl-module-tap-formatter-file-session \
perl-module-tap-formatter-session \
perl-module-tap-harness \
perl-module-tap-harness-env \
perl-module-tap-object \
perl-module-tap-parser \
perl-module-tap-parser-aggregator \
perl-module-tap-parser-grammar \
perl-module-tap-parser-iterator \
perl-module-tap-parser-iterator-array \
perl-module-tap-parser-iteratorfactory \
perl-module-tap-parser-iterator-process \
perl-module-tap-parser-iterator-stream \
perl-module-tap-parser-multiplexer \
perl-module-tap-parser-result \
perl-module-tap-parser-result-bailout \
perl-module-tap-parser-result-comment \
perl-module-tap-parser-resultfactory \
perl-module-tap-parser-result-plan \
perl-module-tap-parser-result-pragma \
perl-module-tap-parser-result-test \
perl-module-tap-parser-result-unknown \
perl-module-tap-parser-result-version \
perl-module-tap-parser-result-yaml \
perl-module-tap-parser-scheduler \
perl-module-tap-parser-scheduler-job \
perl-module-tap-parser-scheduler-spinner \
perl-module-tap-parser-source \
perl-module-tap-parser-sourcehandler \
perl-module-tap-parser-sourcehandler-executable \
perl-module-tap-parser-sourcehandler-file \
perl-module-tap-parser-sourcehandler-handle \
perl-module-tap-parser-sourcehandler-perl \
perl-module-tap-parser-sourcehandler-rawtap \
perl-module-tap-parser-yamlish-reader \
perl-module-tap-parser-yamlish-writer \
\
Only perl-module-extutils-mm-unix was requested. However, I noticed that the suffix has "unix", so I assumed that this module is OS dependant. In case openssl-ptest needs to be built for other OSes, I added all perl-module-extutils-mm*.
perl-module-extutils-mm \
perl-module-extutils-mm-aix \
perl-module-extutils-mm-any \
perl-module-extutils-mm-beos \
perl-module-extutils-mm-cygwin \
perl-module-extutils-mm-darwin \
perl-module-extutils-mm-dos \
perl-module-extutils-mm-macos \
perl-module-extutils-mm-nw5 \
perl-module-extutils-mm-os2 \
perl-module-extutils-mm-os390 \
perl-module-extutils-mm-qnx \
perl-module-extutils-mm-unix \
perl-module-extutils-mm-uwin \
perl-module-extutils-mm-vms \
perl-module-extutils-mm-vos \
perl-module-extutils-mm-win32 \
perl-module-extutils-mm-win95 \
===== APPENDIX: Commands I Used to Identify the Potential List of Packages Needed =====
These are details and can be ignored.
Under "{build directory}/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/rootfs/usr/lib/openssl/ptest", I executed the following command to convert all imported modules using "use" statements into "perl-module-*":
$ grep -r -n -I "^\s*use " | sed -E 's/.*:[0-9]+://' | sed -E 's/;/\n/g' | sed -E 's/^ *use *//' | sed -E 's/ /\n/g' | awk '!/^\s*$/' | tr '[:upper:]' '[:lower:]' | sed -E 's/::/-/g' | sed -E 's/^/perl-module-/' | sort | uniq
I saved the above list into a file "PARSED_USE". Then, I listed all directories under "tmp/work/core2-64-poky-linux/perl/5.38.2/packages-split" and saved this list as a list of valid perl-module-* into a file "VALID_MODULE".
I used the following command to determine which line in file "PARSED_USE" is a valid perl-module-* based on "VALID_MODULE" using the following command:
$ cat PARSED_USE VALID_MODULE | sort | uniq -c | sort -nk1,1 | grep -E '^ *2'
Thank you for doing this deep level investigation. I think this pretty much shows that listing all modules in a way that can be maintained isn't feasible (the idea was that maybe there's only a few of them needed for ptest). Which brings us back to the question: why is perl-modules using RRECOMMENDS and not RDEPENDS? I went through git history, and perl recipe had RRECOMMENDS from the very first mass-import commit in 2005, so the answer is likely no one knows or remembers. I also went and checked python: in there python3-modules is using RDEPENDS. So I think the easiest way out is to simply replace RRECOMMENDS with RDEPENDS in perl as well. Can you make a patch? Alex On Fri, 19 Jan 2024 at 19:11, Lyu, William via lists.openembedded.org <william.lyu=windriver.com@lists.openembedded.org> wrote: > > Following your suggestions and iteratively adding packages based on error message, I am finally able to come up with quite a long list of perl-module-* packages that fix the dependencies of openssl-ptest. > > > However, I am worried about if the list I came up with is truly comprehensive in the sense that this list covers all dependencies for other architectures and other OSes. I see that a lot of openssl-ptest subtests are marked as "skipped". I am not sure if these dependencies will be sufficient if these subtests are not skipped. Also, I am not sure if the packages I added are arch or OS specific. > > The following is the list of perl-module-* I found. > > This first portion is identified by following your suggestion of using grep "^\s*use". > perl-module-extutils-makemaker \ > perl-module-file-find \ > perl-module-if \ > perl-module-pod-checker \ > perl-module-tap-harness \ > perl-module-tap-parser-aggregator \ > perl-module-tap-parser-source \ > perl-module-tap-parser-sourcehandler-perl \ > perl-module-text-parsewords \ > perl-module-carp \ > perl-module-cwd \ > perl-module-data-dumper \ > perl-module-digest-sha \ > perl-module-encode \ > perl-module-errno \ > perl-module-exporter \ > perl-module-fcntl \ > perl-module-feature \ > perl-module-file-basename \ > perl-module-file-compare \ > perl-module-file-copy \ > perl-module-file-glob \ > perl-module-filehandle \ > perl-module-file-path \ > perl-module-file-spec \ > perl-module-file-spec-functions \ > perl-module-file-temp \ > perl-module-findbin \ > perl-module-getopt-long \ > perl-module-getopt-std \ > perl-module-io-select \ > perl-module-io-socket \ > perl-module-ipc-cmd \ > perl-module-ipc-open2 \ > perl-module-lib \ > perl-module-list-util \ > perl-module-math-bigint \ > perl-module-mime-base64 \ > perl-module-pod-html \ > perl-module-posix \ > perl-module-scalar-util \ > perl-module-storable \ > perl-module-test-harness \ > perl-module-test-more \ > perl-module-time-local \ > perl-module-utf8 \ > perl-module-vars \ > \ > > The following are added iteratively based on error reports from actually running the test. > perl-module-pod-text \ > perl-module-encode-encoding \ > perl-module-metadata \ > perl-module-locale \ > \ > > The following are added due to internal dependencies between different TAP modules. I added a few iteratively, and there are still some errors. So, I just grepped and added all tap modules. > perl-module-tap-base \ > perl-module-tap-formatter-base \ > perl-module-tap-formatter-color \ > perl-module-tap-formatter-console \ > perl-module-tap-formatter-console-parallelsession \ > perl-module-tap-formatter-console-session \ > perl-module-tap-formatter-file \ > perl-module-tap-formatter-file-session \ > perl-module-tap-formatter-session \ > perl-module-tap-harness \ > perl-module-tap-harness-env \ > perl-module-tap-object \ > perl-module-tap-parser \ > perl-module-tap-parser-aggregator \ > perl-module-tap-parser-grammar \ > perl-module-tap-parser-iterator \ > perl-module-tap-parser-iterator-array \ > perl-module-tap-parser-iteratorfactory \ > perl-module-tap-parser-iterator-process \ > perl-module-tap-parser-iterator-stream \ > perl-module-tap-parser-multiplexer \ > perl-module-tap-parser-result \ > perl-module-tap-parser-result-bailout \ > perl-module-tap-parser-result-comment \ > perl-module-tap-parser-resultfactory \ > perl-module-tap-parser-result-plan \ > perl-module-tap-parser-result-pragma \ > perl-module-tap-parser-result-test \ > perl-module-tap-parser-result-unknown \ > perl-module-tap-parser-result-version \ > perl-module-tap-parser-result-yaml \ > perl-module-tap-parser-scheduler \ > perl-module-tap-parser-scheduler-job \ > perl-module-tap-parser-scheduler-spinner \ > perl-module-tap-parser-source \ > perl-module-tap-parser-sourcehandler \ > perl-module-tap-parser-sourcehandler-executable \ > perl-module-tap-parser-sourcehandler-file \ > perl-module-tap-parser-sourcehandler-handle \ > perl-module-tap-parser-sourcehandler-perl \ > perl-module-tap-parser-sourcehandler-rawtap \ > perl-module-tap-parser-yamlish-reader \ > perl-module-tap-parser-yamlish-writer \ > \ > > Only perl-module-extutils-mm-unix was requested. However, I noticed that the suffix has "unix", so I assumed that this module is OS dependant. In case openssl-ptest needs to be built for other OSes, I added all perl-module-extutils-mm*. > perl-module-extutils-mm \ > perl-module-extutils-mm-aix \ > perl-module-extutils-mm-any \ > perl-module-extutils-mm-beos \ > perl-module-extutils-mm-cygwin \ > perl-module-extutils-mm-darwin \ > perl-module-extutils-mm-dos \ > perl-module-extutils-mm-macos \ > perl-module-extutils-mm-nw5 \ > perl-module-extutils-mm-os2 \ > perl-module-extutils-mm-os390 \ > perl-module-extutils-mm-qnx \ > perl-module-extutils-mm-unix \ > perl-module-extutils-mm-uwin \ > perl-module-extutils-mm-vms \ > perl-module-extutils-mm-vos \ > perl-module-extutils-mm-win32 \ > perl-module-extutils-mm-win95 \ > > ===== APPENDIX: Commands I Used to Identify the Potential List of Packages Needed ===== > These are details and can be ignored. > > Under "{build directory}/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/rootfs/usr/lib/openssl/ptest", I executed the following command to convert all imported modules using "use" statements into "perl-module-*": > $ grep -r -n -I "^\s*use " | sed -E 's/.*:[0-9]+://' | sed -E 's/;/\n/g' | sed -E 's/^ *use *//' | sed -E 's/ /\n/g' | awk '!/^\s*$/' | tr '[:upper:]' '[:lower:]' | sed -E 's/::/-/g' | sed -E 's/^/perl-module-/' | sort | uniq > > I saved the above list into a file "PARSED_USE". Then, I listed all directories under "tmp/work/core2-64-poky-linux/perl/5.38.2/packages-split" and saved this list as a list of valid perl-module-* into a file "VALID_MODULE". > > I used the following command to determine which line in file "PARSED_USE" is a valid perl-module-* based on "VALID_MODULE" using the following command: > $ cat PARSED_USE VALID_MODULE | sort | uniq -c | sort -nk1,1 | grep -E '^ *2' > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#194062): https://lists.openembedded.org/g/openembedded-core/message/194062 > Mute This Topic: https://lists.openembedded.org/mt/103766391/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/meta/recipes-devtools/perl/perl_5.38.2.bb b/meta/recipes-devtools/perl/perl_5.38.2.bb index a9d684cfc5..f75c5aa971 100644 --- a/meta/recipes-devtools/perl/perl_5.38.2.bb +++ b/meta/recipes-devtools/perl/perl_5.38.2.bb @@ -311,6 +311,14 @@ ALTERNATIVE_LINK_NAME[Thread.3] = "${mandir}/man3/Thread.3" ALLOW_EMPTY:${PN}-modules = "1" PACKAGES += "${PN}-modules " +# When perl-modules is added through COMPLEMENTARY_GLOB, as it is recommending +# packages, all its recommendations will be ignored. So, we create a +# perl-modules-force-all package depending on the same set of packages as listed +# in perl-modules to provide a way to guarantee COMPLEMENTARY_GLOB not removing +# these packages. +ALLOW_EMPTY:${PN}-modules-force-all = "1" +PACKAGES += "${PN}-modules-force-all " + PACKAGESPLITFUNCS =+ "split_perl_packages" python split_perl_packages () { @@ -328,6 +336,20 @@ python split_perl_packages () { packages = filter(lambda p: 'perl-module-' in p, d.getVar('PACKAGES').split()) d.setVar(d.expand("RRECOMMENDS:${PN}-modules"), ' '.join(packages)) + # perl-modules-force-all provides a way to prevent COMPLEMENTARY_GLOB + # from removing the packages specified in perl-modules. + # perl-modules-force-all contains the same set of packages as perl-modules. + d.setVar( + d.expand("RDEPENDS:${PN}-modules-force-all"), + d.getVar(d.expand("RRECOMMENDS:${PN}-modules")) + ) + + # As perl-modules-force-all depends on all other, potentially non-existent, + # perl packages, we allow these packages to be empty to prevent build + # errors. + for pkg in d.getVar(d.expand("RDEPENDS:${PN}-modules-force-all")).split(): + d.setVar(d.expand("ALLOW_EMPTY:" + pkg), "1") + # Read the pre-generated dependency file, and use it to set module dependecies for line in open(d.expand("${WORKDIR}") + '/perl-rdepends.txt').readlines(): splitline = line.split()