| Message ID | 20240227085722.2025994-1-xiangyu.chen@eng.windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] bash: add locale alias for ja_JP shiftjis in run-ptest | expand |
Wait. Why is the correct locale not generated? How do other systems do it? There's an issue elsewhere, possibly in glibc-locale, and this works around that, instead of fixing the original problem. Alex On Tue, 27 Feb 2024 at 09:57, Xiangyu Chen <xiangyu.chen@eng.windriver.com> wrote: > > From: Xiangyu Chen <xiangyu.chen@windriver.com> > > Bash has a test case that needs ja_JP shift jis locale. > > It will check current system has ja_JP.SJIS locale or not, but in yocto, > this locale named ja_JP.shiftjis, they are the same. > So create a locale alias, making ja_JP.SJIS link to ja_JP.shiftjis. > > Also add warning message if current system doesn't contain shift jis locale. > > Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com> > --- > meta/recipes-extended/bash/bash/run-ptest | 21 +++++++++++++++++++++ > 1 file changed, 21 insertions(+) > > diff --git a/meta/recipes-extended/bash/bash/run-ptest b/meta/recipes-extended/bash/bash/run-ptest > index 738ad3c42c..3cafb5e2ea 100644 > --- a/meta/recipes-extended/bash/bash/run-ptest > +++ b/meta/recipes-extended/bash/bash/run-ptest > @@ -3,6 +3,7 @@ > en_US=`locale -a | grep en_US*` > fr_FR=`locale -a | grep fr_FR*` > de_DE=`locale -a | grep de_DE*` > +ja_JP_SJIS=`locale -a | grep ja_JP.shiftjis` > > if [ -z "$en_US" ] > then > @@ -19,8 +20,28 @@ then > echo "Warning: The de_DE* locales is needed to run the intl.tests, please add it." > fi > > +if [ -z "$ja_JP_SJIS" ] > +then > + echo "Warning: The ja_JP.shiftjis locales is needed to run the intl.tests, please add it." > +else > +# Yocto generated shiftjis locale named ja_JP.shiftjis, but bash's test script using ja_JP.SJIS > +# They are the same one, so add an alias in locale.alias > + if [ -f /usr/share/locale/locale.alias ] > + then > + cp /usr/share/locale/locale.alias /usr/share/locale/locale.alias.bak > + fi > + > + echo "ja_JP.SJIS ja_JP.shiftjis" >> /usr/share/locale/locale.alias > +fi > + > + > useradd bashtest > chown -R bashtest:bashtest tests > setpriv --reuid bashtest --rgid bashtest --clear-groups --reset-env $(dirname "$0")/run-bash-ptests > chown -R root:root tests > userdel -r bashtest > + > +if [ -f /usr/share/locale/locale.alias.bak ] > +then > + mv /usr/share/locale/locale.alias.bak /usr/share/locale/locale.alias > +fi > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#196244): https://lists.openembedded.org/g/openembedded-core/message/196244 > Mute This Topic: https://lists.openembedded.org/mt/104599698/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 2/27/24 17:41, Alexander Kanavin wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > Wait. Why is the correct locale not generated? How do other systems do > it? There's an issue elsewhere, possibly in glibc-locale, and this > works around that, instead of fixing the original problem. Hi Alex, Since SHIFTJIS is not a ISO2022/posix compliant and not in the list of glibc locale data support, it was generated by manual with localedef and don't have a unified name. The following name has been already found on different linux: ja_JP.SJIS ja_JP.SHIFT_JIS ja_JP.shiftjis japanese.sjis Br, Xiangyu > > Alex > > On Tue, 27 Feb 2024 at 09:57, Xiangyu Chen > <xiangyu.chen@eng.windriver.com> wrote: >> From: Xiangyu Chen <xiangyu.chen@windriver.com> >> >> Bash has a test case that needs ja_JP shift jis locale. >> >> It will check current system has ja_JP.SJIS locale or not, but in yocto, >> this locale named ja_JP.shiftjis, they are the same. >> So create a locale alias, making ja_JP.SJIS link to ja_JP.shiftjis. >> >> Also add warning message if current system doesn't contain shift jis locale. >> >> Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com> >> --- >> meta/recipes-extended/bash/bash/run-ptest | 21 +++++++++++++++++++++ >> 1 file changed, 21 insertions(+) >> >> diff --git a/meta/recipes-extended/bash/bash/run-ptest b/meta/recipes-extended/bash/bash/run-ptest >> index 738ad3c42c..3cafb5e2ea 100644 >> --- a/meta/recipes-extended/bash/bash/run-ptest >> +++ b/meta/recipes-extended/bash/bash/run-ptest >> @@ -3,6 +3,7 @@ >> en_US=`locale -a | grep en_US*` >> fr_FR=`locale -a | grep fr_FR*` >> de_DE=`locale -a | grep de_DE*` >> +ja_JP_SJIS=`locale -a | grep ja_JP.shiftjis` >> >> if [ -z "$en_US" ] >> then >> @@ -19,8 +20,28 @@ then >> echo "Warning: The de_DE* locales is needed to run the intl.tests, please add it." >> fi >> >> +if [ -z "$ja_JP_SJIS" ] >> +then >> + echo "Warning: The ja_JP.shiftjis locales is needed to run the intl.tests, please add it." >> +else >> +# Yocto generated shiftjis locale named ja_JP.shiftjis, but bash's test script using ja_JP.SJIS >> +# They are the same one, so add an alias in locale.alias >> + if [ -f /usr/share/locale/locale.alias ] >> + then >> + cp /usr/share/locale/locale.alias /usr/share/locale/locale.alias.bak >> + fi >> + >> + echo "ja_JP.SJIS ja_JP.shiftjis" >> /usr/share/locale/locale.alias >> +fi >> + >> + >> useradd bashtest >> chown -R bashtest:bashtest tests >> setpriv --reuid bashtest --rgid bashtest --clear-groups --reset-env $(dirname "$0")/run-bash-ptests >> chown -R root:root tests >> userdel -r bashtest >> + >> +if [ -f /usr/share/locale/locale.alias.bak ] >> +then >> + mv /usr/share/locale/locale.alias.bak /usr/share/locale/locale.alias >> +fi >> -- >> 2.34.1 >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#196244): https://lists.openembedded.org/g/openembedded-core/message/196244 >> Mute This Topic: https://lists.openembedded.org/mt/104599698/1686489 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >>
On Wed, 28 Feb 2024 at 03:45, Xiangyu Chen <xiangyu.chen@eng.windriver.com> wrote: > Since SHIFTJIS is not a ISO2022/posix compliant and not in the list of > glibc locale data support, it was generated by manual with localedef and > don't have a unified name. > > > The following name has been already found on different linux: > > ja_JP.SJIS ja_JP.SHIFT_JIS ja_JP.shiftjis japanese.sjis Hello Chen, I need to note that we specifically exclude bash from ptests, and this is the reason: https://git.yoctoproject.org/poky/commit/?id=2758f6d0337ec84c9143546bc65a80559e8d7808 As a result, bash ptest would keep regressing in various ways, simply because it's not executed, and I would suggest that you either first fix these failures, so we can restore bash in meta/conf/distro/include/ptests-packagelists.inc, or drop bash from your own testing. Is there a particular reason you need to run those tests? Alex
On 2/28/24 17:38, Alexander Kanavin wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Wed, 28 Feb 2024 at 03:45, Xiangyu Chen > <xiangyu.chen@eng.windriver.com> wrote: >> Since SHIFTJIS is not a ISO2022/posix compliant and not in the list of >> glibc locale data support, it was generated by manual with localedef and >> don't have a unified name. >> >> >> The following name has been already found on different linux: >> >> ja_JP.SJIS ja_JP.SHIFT_JIS ja_JP.shiftjis japanese.sjis > Hello Chen, > > I need to note that we specifically exclude bash from ptests, and this > is the reason: > https://git.yoctoproject.org/poky/commit/?id=2758f6d0337ec84c9143546bc65a80559e8d7808 Hi Alex, > > As a result, bash ptest would keep regressing in various ways, simply > because it's not executed, and I would suggest that you either first > fix these failures, so we can restore bash in > meta/conf/distro/include/ptests-packagelists.inc, or drop bash from > your own testing. Is there a particular reason you need to run those > tests? Thanks for your information, recently I am trying to fix the bash ptest failures. Currently, in my local setup with master oe, the test failure is 5(total 83). The failed cases are run-intl run-type run-array run-execscript run-read. After checking the failure log, the following is a simple of failure cases analysis: 1) run-array: egrep output warning messages cause the final log is different from pre-log. A patch has been sent to bash bug-report list to use grep -v -e instead of egrep[1]. 2) run-type: there is a patch need to rebase, fix has been merged to oe-core master[2] 3) run-execscript: it reports the failure but not really failed, it just differing only in line numbers is produced. 4) run-intl: due to no shiftjis locale in system, some of unicode tests were skipped. That cause actually run 1251 cases mismatch with total 1378 cases. 5) run-read failure cannot 100% reproduce, when it failed, a character was missing in test log, still in debugging.. And also i met a problem, most of cases are comparing test log with pre-test log from maintainer as the test result that line number/space/path/new line is produced causes diff failed, but in fact they are not a real test failure, the result still need to verify by manual, seems cannot do the automation test, any suggestion on that? Thanks, Ref: [1] https://lists.gnu.org/archive/html/bug-bash/2024-02/msg00154.html [2] https://git.openembedded.org/openembedded-core/commit/?id=ecb698ca5b4f76937dd3bee927465b17866d50be Br, Xiangyu > > Alex
On Thu, 29 Feb 2024 at 11:00, Xiangyu Chen <xiangyu.chen@eng.windriver.com> wrote: > And also i met a problem, most of cases are comparing test log with > pre-test log from maintainer as the test result that line > number/space/path/new line is produced causes diff failed, > > but in fact they are not a real test failure, the result still need to > verify by manual, seems cannot do the automation test, any suggestion on > that? All of this seems to indicate that the bash test suite is not meant for independent execution, and is only intended for bash maintainers. Perhaps you should clarify this with them first, and then also point out specific problems you are facing. Basically, engage with bash upstream, and see if they're interested in helping you to improve tests to the point where we can re-enable them. If there's no such interest, we shouldn't have to continuously add fixes and workarounds. Whether the whole effort is worth your time is a question I leave for you to answer :) Alex
On 2/29/24 18:09, Alexander Kanavin wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Thu, 29 Feb 2024 at 11:00, Xiangyu Chen > <xiangyu.chen@eng.windriver.com> wrote: > >> And also i met a problem, most of cases are comparing test log with >> pre-test log from maintainer as the test result that line >> number/space/path/new line is produced causes diff failed, >> >> but in fact they are not a real test failure, the result still need to >> verify by manual, seems cannot do the automation test, any suggestion on >> that? > All of this seems to indicate that the bash test suite is not meant > for independent execution, and is only intended for bash maintainers. > Perhaps you should clarify this with them first, and then also point > out specific problems you are facing. Basically, engage with bash > upstream, and see if they're interested in helping you to improve > tests to the point where we can re-enable them. If there's no such > interest, we shouldn't have to continuously add fixes and workarounds. > > Whether the whole effort is worth your time is a question I leave for > you to answer :) Aha, thanks for your information :) Yes, from the result, it seems only intended for bash maintainers and not fit for other automation yet. I think firstly i need to check the history of bash maillist to see whether others sent this topic and the result of discussion on that. Thanks :D Br, Xiangyu > > Alex
Hi Chen, It does not seem to be the main issue, but here is an other strange thing I noticed in the locale.alias file. Indeed, the file (provided by the glibc-locale-alias package) in my environment already contains this line: japanese.sjis ja_JP.SJIS I understand that ja_JP.SJIS is expected to be the "real" locale name. However, your patch sets it as an alias for ja_JP.shiftjis. Regards, Jonathan Le jeu. 29 févr. 2024 à 11:27, Xiangyu Chen <xiangyu.chen@eng.windriver.com> a écrit : > > On 2/29/24 18:09, Alexander Kanavin wrote: > > CAUTION: This email comes from a non Wind River email account! > > Do not click links or open attachments unless you recognize the sender > and know the content is safe. > > > > On Thu, 29 Feb 2024 at 11:00, Xiangyu Chen > > <xiangyu.chen@eng.windriver.com> wrote: > > > >> And also i met a problem, most of cases are comparing test log with > >> pre-test log from maintainer as the test result that line > >> number/space/path/new line is produced causes diff failed, > >> > >> but in fact they are not a real test failure, the result still need to > >> verify by manual, seems cannot do the automation test, any suggestion on > >> that? > > All of this seems to indicate that the bash test suite is not meant > > for independent execution, and is only intended for bash maintainers. > > Perhaps you should clarify this with them first, and then also point > > out specific problems you are facing. Basically, engage with bash > > upstream, and see if they're interested in helping you to improve > > tests to the point where we can re-enable them. If there's no such > > interest, we shouldn't have to continuously add fixes and workarounds. > > > > Whether the whole effort is worth your time is a question I leave for > > you to answer :) > > Aha, thanks for your information :) > > Yes, from the result, it seems only intended for bash maintainers and > not fit for other automation yet. > > I think firstly i need to check the history of bash maillist to see > whether others sent this topic and the result of discussion on that. > > > Thanks :D > > > Br, > > Xiangyu > > > > > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#196427): > https://lists.openembedded.org/g/openembedded-core/message/196427 > Mute This Topic: https://lists.openembedded.org/mt/104599698/8060005 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > jonathan@joggee.fr] > -=-=-=-=-=-=-=-=-=-=-=- > >
diff --git a/meta/recipes-extended/bash/bash/run-ptest b/meta/recipes-extended/bash/bash/run-ptest index 738ad3c42c..3cafb5e2ea 100644 --- a/meta/recipes-extended/bash/bash/run-ptest +++ b/meta/recipes-extended/bash/bash/run-ptest @@ -3,6 +3,7 @@ en_US=`locale -a | grep en_US*` fr_FR=`locale -a | grep fr_FR*` de_DE=`locale -a | grep de_DE*` +ja_JP_SJIS=`locale -a | grep ja_JP.shiftjis` if [ -z "$en_US" ] then @@ -19,8 +20,28 @@ then echo "Warning: The de_DE* locales is needed to run the intl.tests, please add it." fi +if [ -z "$ja_JP_SJIS" ] +then + echo "Warning: The ja_JP.shiftjis locales is needed to run the intl.tests, please add it." +else +# Yocto generated shiftjis locale named ja_JP.shiftjis, but bash's test script using ja_JP.SJIS +# They are the same one, so add an alias in locale.alias + if [ -f /usr/share/locale/locale.alias ] + then + cp /usr/share/locale/locale.alias /usr/share/locale/locale.alias.bak + fi + + echo "ja_JP.SJIS ja_JP.shiftjis" >> /usr/share/locale/locale.alias +fi + + useradd bashtest chown -R bashtest:bashtest tests setpriv --reuid bashtest --rgid bashtest --clear-groups --reset-env $(dirname "$0")/run-bash-ptests chown -R root:root tests userdel -r bashtest + +if [ -f /usr/share/locale/locale.alias.bak ] +then + mv /usr/share/locale/locale.alias.bak /usr/share/locale/locale.alias +fi