libc-locale: fix ASCII compatible warning cause build failure.

From: Xiangyu Chen <xiangyu.chen@windriver.com>

Some locales are not listed in glibc locales support list, but can be generated,
here using ja_JP.SHIFT_JIS as an example. We can add following line into local.conf
to enable and generate it:

GLIBC_GENERATE_LOCALES += "en_GB.UTF-8 en_US.UTF-8 ja_JP.UTF-8 ja_JP.SHIFT_JIS"
IMAGE_LINGUAS += "ja-jp en-us ja-jp.shift-jis"

The localedef tool would report a warning and exit with 1, that cause build failure,
error message as below:

[warning] character map `SHIFT_JIS' is not ASCII compatible, locale not ISO C compliant [--no-warnings=ascii]

So add a --no-warnings=ascii in libc-package.bbclass to fix build failure if someone needs those locale
in yocto.

Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com>
---
 meta/classes-recipe/libc-package.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

On 2024-02-26 4:59 a.m., Xiangyu Chen via lists.openembedded.org wrote:
> From: Xiangyu Chen<xiangyu.chen@windriver.com>
>
> Some locales are not listed in glibc locales support list, but can be generated,
> here using ja_JP.SHIFT_JIS as an example. We can add following line into local.conf
> to enable and generate it:
>
> GLIBC_GENERATE_LOCALES += "en_GB.UTF-8 en_US.UTF-8 ja_JP.UTF-8 ja_JP.SHIFT_JIS"
> IMAGE_LINGUAS += "ja-jp en-us ja-jp.shift-jis"
>
> The localedef tool would report a warning and exit with 1, that cause build failure,
> error message as below:
>
> [warning] character map `SHIFT_JIS' is not ASCII compatible, locale not ISO C compliant [--no-warnings=ascii]
>
> So add a --no-warnings=ascii in libc-package.bbclass to fix build failure if someone needs those locale
> in yocto.

For context, Xiangyu is changing the libc-package.bbclass so that we can 
generate a ja_JP.SJIS  / ja_JP.SHIFT_JIS locale
to make another of the bash ptests pass.

I don't know if that locale is commonly used these days but apparently 
it's not POSIX complaint:

https://discourse.nixos.org/t/building-unsupported-locales-ja-jp-sjis/3612


Richard, Mark, Sundeep, Jonathan, Khem,

Any idea if this is a change that we should be merging to master
rather than skipping the bash ptest?

Xiangyu,

Did you test musl? ( I don't know if musl even supports locales, 
off-hand! ;-)  )

../Randy


>
> Signed-off-by: Xiangyu Chen<xiangyu.chen@windriver.com>
> ---
>   meta/classes-recipe/libc-package.bbclass | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/libc-package.bbclass b/meta/classes-recipe/libc-package.bbclass
> index bf160b115d..c06a2ce90a 100644
> --- a/meta/classes-recipe/libc-package.bbclass
> +++ b/meta/classes-recipe/libc-package.bbclass
> @@ -278,7 +278,7 @@ python package_do_split_gconvs () {
>                   bb.fatal("unknown arch:" + target_arch + " for locale_arch_options")
>   
>               localedef_opts += " --force --no-hard-links --no-archive --prefix=%s \
> -                --inputfile=%s/%s/i18n/locales/%s --charmap=%s %s/%s" \
> +                --inputfile=%s/%s/i18n/locales/%s --charmap=%s %s/%s --no-warnings=ascii" \
>                   % (treedir, treedir, datadir, locale, encoding, outputpath, name)
>   
>               cmd = "PATH=\"%s\" I18NPATH=\"%s\" GCONV_PATH=\"%s\" cross-localedef %s" % \
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#196197):https://lists.openembedded.org/g/openembedded-core/message/196197
> Mute This Topic:https://lists.openembedded.org/mt/104579284/3616765
> Group Owner:openembedded-core+owner@lists.openembedded.org
> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [randy.macleod@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Mon, Feb 26, 2024 at 5:16 PM Randy MacLeod
<randy.macleod@windriver.com> wrote:
>
> On 2024-02-26 4:59 a.m., Xiangyu Chen via lists.openembedded.org wrote:
>
> From: Xiangyu Chen <xiangyu.chen@windriver.com>
>
> Some locales are not listed in glibc locales support list, but can be generated,
> here using ja_JP.SHIFT_JIS as an example. We can add following line into local.conf
> to enable and generate it:
>
> GLIBC_GENERATE_LOCALES += "en_GB.UTF-8 en_US.UTF-8 ja_JP.UTF-8 ja_JP.SHIFT_JIS"
> IMAGE_LINGUAS += "ja-jp en-us ja-jp.shift-jis"
>
> The localedef tool would report a warning and exit with 1, that cause build failure,
> error message as below:
>
> [warning] character map `SHIFT_JIS' is not ASCII compatible, locale not ISO C compliant [--no-warnings=ascii]
>
> So add a --no-warnings=ascii in libc-package.bbclass to fix build failure if someone needs those locale
> in yocto.
>
> For context, Xiangyu is changing the libc-package.bbclass so that we can generate a ja_JP.SJIS  / ja_JP.SHIFT_JIS locale
> to make another of the bash ptests pass.
>
> I don't know if that locale is commonly used these days but apparently it's not POSIX complaint:
>
> https://discourse.nixos.org/t/building-unsupported-locales-ja-jp-sjis/3612
>
>
> Richard, Mark, Sundeep, Jonathan, Khem,
>
> Any idea if this is a change that we should be merging to master
> rather than skipping the bash ptest?

yes, it should be merged into master, this is default behavior in
glibc, but since we do generate the locales using cross localedef
we do not get that, so this change will make it so.

>
> Xiangyu,
>
> Did you test musl? ( I don't know if musl even supports locales, off-hand! ;-)  )

We do not have to worry about musl for glibc locales, it does not use them.

>
> ../Randy
>
>
>
> Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com>
> ---
>  meta/classes-recipe/libc-package.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/libc-package.bbclass b/meta/classes-recipe/libc-package.bbclass
> index bf160b115d..c06a2ce90a 100644
> --- a/meta/classes-recipe/libc-package.bbclass
> +++ b/meta/classes-recipe/libc-package.bbclass
> @@ -278,7 +278,7 @@ python package_do_split_gconvs () {
>                  bb.fatal("unknown arch:" + target_arch + " for locale_arch_options")
>
>              localedef_opts += " --force --no-hard-links --no-archive --prefix=%s \
> -                --inputfile=%s/%s/i18n/locales/%s --charmap=%s %s/%s" \
> +                --inputfile=%s/%s/i18n/locales/%s --charmap=%s %s/%s --no-warnings=ascii" \
>                  % (treedir, treedir, datadir, locale, encoding, outputpath, name)
>
>              cmd = "PATH=\"%s\" I18NPATH=\"%s\" GCONV_PATH=\"%s\" cross-localedef %s" % \
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#196197): https://lists.openembedded.org/g/openembedded-core/message/196197
> Mute This Topic: https://lists.openembedded.org/mt/104579284/3616765
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
> --
> # Randy MacLeod
> # Wind River Linux

On 2/27/24 09:16, Randy MacLeod wrote:
> On 2024-02-26 4:59 a.m., Xiangyu Chen via lists.openembedded.org wrote:
>> From: Xiangyu Chen<xiangyu.chen@windriver.com>
>>
>> Some locales are not listed in glibc locales support list, but can be generated,
>> here using ja_JP.SHIFT_JIS as an example. We can add following line into local.conf
>> to enable and generate it:
>>
>> GLIBC_GENERATE_LOCALES += "en_GB.UTF-8 en_US.UTF-8 ja_JP.UTF-8 ja_JP.SHIFT_JIS"
>> IMAGE_LINGUAS += "ja-jp en-us ja-jp.shift-jis"
>>
>> The localedef tool would report a warning and exit with 1, that cause build failure,
>> error message as below:
>>
>> [warning] character map `SHIFT_JIS' is not ASCII compatible, locale not ISO C compliant [--no-warnings=ascii]
>>
>> So add a --no-warnings=ascii in libc-package.bbclass to fix build failure if someone needs those locale
>> in yocto.
>
> For context, Xiangyu is changing the libc-package.bbclass so that we 
> can generate a ja_JP.SJIS  / ja_JP.SHIFT_JIS locale
> to make another of the bash ptests pass.
>
> I don't know if that locale is commonly used these days but apparently 
> it's not POSIX complaint:
>
> https://discourse.nixos.org/t/building-unsupported-locales-ja-jp-sjis/3612
>
>
> Richard, Mark, Sundeep, Jonathan, Khem,
>
> Any idea if this is a change that we should be merging to master
> rather than skipping the bash ptest?
>
> Xiangyu,
>
> Did you test musl? ( I don't know if musl even supports locales, 
> off-hand! ;-)  )
>
Hi Randy,


Yes, but I meet a build failure that seems musl cannot provide ja_jp 
locale (or needs to add more configurations in local.conf? )

Error info:

ERROR: Nothing RPROVIDES 'locale-base-ja-jp'


Br,

Xiangyu


> ../Randy
>
>
>> Signed-off-by: Xiangyu Chen<xiangyu.chen@windriver.com>
>> ---
>>   meta/classes-recipe/libc-package.bbclass | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes-recipe/libc-package.bbclass b/meta/classes-recipe/libc-package.bbclass
>> index bf160b115d..c06a2ce90a 100644
>> --- a/meta/classes-recipe/libc-package.bbclass
>> +++ b/meta/classes-recipe/libc-package.bbclass
>> @@ -278,7 +278,7 @@ python package_do_split_gconvs () {
>>                   bb.fatal("unknown arch:" + target_arch + " for locale_arch_options")
>>   
>>               localedef_opts += " --force --no-hard-links --no-archive --prefix=%s \
>> -                --inputfile=%s/%s/i18n/locales/%s --charmap=%s %s/%s" \
>> +                --inputfile=%s/%s/i18n/locales/%s --charmap=%s %s/%s --no-warnings=ascii" \
>>                   % (treedir, treedir, datadir, locale, encoding, outputpath, name)
>>   
>>               cmd = "PATH=\"%s\" I18NPATH=\"%s\" GCONV_PATH=\"%s\" cross-localedef %s" % \
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#196197):https://lists.openembedded.org/g/openembedded-core/message/196197
>> Mute This Topic:https://lists.openembedded.org/mt/104579284/3616765
>> Group Owner:openembedded-core+owner@lists.openembedded.org
>> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [randy.macleod@windriver.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>
> -- 
> # Randy MacLeod
> # Wind River Linux

On Mon, 2024-02-26 at 20:16 -0500, Randy MacLeod via lists.openembedded.org wrote:
>  On 2024-02-26 4:59 a.m., Xiangyu Chen via lists.openembedded.org wrote:
>  From: Xiangyu Chen <xiangyu.chen@windriver.com>
> > 
> > Some locales are not listed in glibc locales support list, but can be generated,
> > here using ja_JP.SHIFT_JIS as an example. We can add following line into local.conf
> > to enable and generate it:
> > 
> > GLIBC_GENERATE_LOCALES += "en_GB.UTF-8 en_US.UTF-8 ja_JP.UTF-8 ja_JP.SHIFT_JIS"
> > IMAGE_LINGUAS += "ja-jp en-us ja-jp.shift-jis"
> > 
> > The localedef tool would report a warning and exit with 1, that cause build failure,
> > error message as below:
> > 
> > [warning] character map `SHIFT_JIS' is not ASCII compatible, locale not ISO C compliant [--no-warnings=ascii]
> > 
> > So add a --no-warnings=ascii in libc-package.bbclass to fix build failure if someone needs those locale
> > in yocto.
> 
> For context, Xiangyu is changing the libc-package.bbclass so that we can generate a ja_JP.SJIS  / ja_JP.SHIFT_JIS locale
>  to make another of the bash ptests pass.
> 
> I don't know if that locale is commonly used these days but apparently it's not POSIX complaint: 
> 
> https://discourse.nixos.org/t/building-unsupported-locales-ja-jp-sjis/3612
>  
> 
> 
> 
> Richard, Mark, Sundeep, Jonathan, Khem,
>  
> 
> 
> 
>  
> 
> 
> 
> Any idea if this is a change that we should be merging to master
>  rather than skipping the bash ptest?
>  

We don't run the bash ptests as they in PTEST_PROBLEMS as they're
apparently non-deterministic according to the file.

I think I'd therefore just prefer to not do this.

Cheers,

Richard