Bug 11153

Summary:	Generated locales does not support UTF-8 encoding
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Yocto User <yoctouser>
Component:	core	Assignee:	Ross Burton <ross.burton>
Status:	RESOLVED NOTABUG	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	meta.mr.watcher, meta.watcher
Version:	2.2.1
Target Milestone:	2.3 M4
Hardware:	Other
OS:	arm
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Yocto User 2017-03-14 16:22:51 UTC

Preconditions/Environment
-------------------------
Host system: Ubuntu 14.04.5 LTS (Trusty Tahr)
Yocto version: YP Core - Morty 2.2.1
Target: SABRE Board for Smart Devices based on i.MX 6Quad processor
Additional meta: meta-freescale, meta-freescale-3rdparty, meta-freescale-distro

Triggering Action/Cause
-----------------------
Modified the local.conf to specify the desired locales with the add of the following variable:
IMAGE_LINGUAS = "de-de fr-fr en-gb en-gb.iso-8859-1 en-us en-us.iso-8859-1 zh-cn"

Modified the local.conf to exclude BusyBox and include some standard utilities:
IMAGE_INSTALL_append = " glibc-utils util-linux coreutils"

Built a base image with the command:
bitbake core-image-base

Booted the image into the SABRE board

Edited the /etc/profile file as follow:
export LC_ALL=en_US
export LANG=en_US
export LANGUAGE=en_US

Rebooted the board

Expectation
-----------
The en_US locale is set, the system fully supports UTF-8 encoding. I can manage and display file names that include UTF-8 characters.

Actual Result
-------------
The locale is set to "en_US", below the output of the "locale" command:
LANG=en_US
LC_CTYPE="en_US"
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_COLLATE="en_US"
LC_MONETARY="en_US"
LC_MESSAGES="en_US"
LC_PAPER="en_US"
LC_NAME="en_US"
LC_ADDRESS="en_US"
LC_TELEPHONE="en_US"
LC_MEASUREMENT="en_US"
LC_IDENTIFICATION="en_US"
LC_ALL=en_US

but the system does not support correctly the UTF-8 encoding.

Please consider the following scenarios:

CASE 1) From target device to outsite
- Connect to the target with a terminal
- Check that the terminal encoding is set to UTF-8
- Touch a file into a pendrive including in the file name accented characters, e.g. à,è,ì,ò,ù
- Connect the pendrive to a Windows system, browse the pen drive
- The file name appears corrupted
- Connect the pendrive to a Linux system (desktop distro, e.g. Ubuntu), browse the pen drive
- The file name appears corrupted (the file name is displayed corrupted exactly as in Windows)

CASE 2) From outside to target device
- Connect a pendrive to a Windows system
- Create a file into the pendrive including in the file name accented characters, e.g. à,è,ì,ò,ù
- Connect the pendrive to a Linux system (desktop distro, e.g. Ubuntu), browse the pen drive
- The file name appears correctly
- Connect to the target with a terminal
- Check that the terminal encoding is set to UTF-8
- Connect the pendrive to the target, browser the pendrive
- The file name appears corrupted

Why en_US locale is not working well with UTF-8? 
Why en_US locale does not have the .UTF-8 suffix? 
How can I check that the en_US locale supports the UTF-8 encoding?

Reproducibility
---------------
Always

Thanks in advance for your time

Best regards,
Fabio Emiliani

Comment 1 Yocto User 2017-03-15 10:04:48 UTC

As suggested here: http://stackoverflow.com/questions/42784435/how-to-check-if-a-locale-is-utf-8/42797421#42797421, I've used the nl_langinfo API to detected the encoding of a locale. This is the output:

./LangInfoInspector $(locale -a)
C ->ANSI_X3.4-1968
de_DE ->UTF-8
en_GB ->UTF-8
en_GB.ISO-8859-1 ->ISO-8859-1
en_US ->UTF-8
en_US.ISO-8859-1 ->ISO-8859-1
fr_FR ->UTF-8
POSIX ->ANSI_X3.4-1968
zh_CN ->UTF-8

so I confirm that en_US, en_GB and similar are UTF-8 encoded (even if there is no .utf8 suffix).

Comment 2 Yocto User 2017-03-17 13:50:02 UTC

I've made further tests and noticed that ISO-8859-1 locales works fine.

If using en_US.ISO-8859-1 I'm able to see accented characters such as: à,è,ì,ò,ù. 

See below:
root@imx6qsabresd:/media/win/italiano# ls -la
total 12
drwxr-xr-x 3 root root 4096 Mar 13 15:11 .
drwxr-xr-x 8 root root 4096 Mar 13 14:57 ..
drwxr-xr-x 2 root root 4096 Mar 13 14:58 mà_però
-rwxr-xr-x 1 root root    0 Mar 13 14:57 mà_però.txt

The directory and file mà_però has been created under Windows. Accented à and ò are correctly displayed.

Tested under PuTTY using the following settings (Change settings -> Window -> Translation):
ISO-8859-1:1998 (Latin-1, West Europe)

Comment 3 Yocto User 2017-03-20 15:38:42 UTC

The problem is caused by the mount of the pendrive. The default mount options include: iocharset=iso8859-1, with this option the pendrive file system is interpreted as ISO 8859-1 and not UTF-8. Solved using iocharset=utf8. See: http://unix.stackexchange.com/q/352144/191458