11153 2017-03-14 16:22:51 +0000 Generated locales does not support UTF-8 encoding 2017-03-20 15:38:42 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.2.1 Other arm RESOLVED NOTABUG Medium+ normal 2.3 M4 1 yoctouser ross.burton meta.mr.watcher meta.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 71256 0 yoctouser 2017-03-14 16:22:51 +0000 Preconditions/Environment ------------------------- Host system: Ubuntu 14.04.5 LTS (Trusty Tahr) Yocto version: YP Core - Morty 2.2.1 Target: SABRE Board for Smart Devices based on i.MX 6Quad processor Additional meta: meta-freescale, meta-freescale-3rdparty, meta-freescale-distro Triggering Action/Cause ----------------------- Modified the local.conf to specify the desired locales with the add of the following variable: IMAGE_LINGUAS = "de-de fr-fr en-gb en-gb.iso-8859-1 en-us en-us.iso-8859-1 zh-cn" Modified the local.conf to exclude BusyBox and include some standard utilities: IMAGE_INSTALL_append = " glibc-utils util-linux coreutils" Built a base image with the command: bitbake core-image-base Booted the image into the SABRE board Edited the /etc/profile file as follow: export LC_ALL=en_US export LANG=en_US export LANGUAGE=en_US Rebooted the board Expectation ----------- The en_US locale is set, the system fully supports UTF-8 encoding. I can manage and display file names that include UTF-8 characters. Actual Result ------------- The locale is set to "en_US", below the output of the "locale" command: LANG=en_US LC_CTYPE="en_US" LC_NUMERIC="en_US" LC_TIME="en_US" LC_COLLATE="en_US" LC_MONETARY="en_US" LC_MESSAGES="en_US" LC_PAPER="en_US" LC_NAME="en_US" LC_ADDRESS="en_US" LC_TELEPHONE="en_US" LC_MEASUREMENT="en_US" LC_IDENTIFICATION="en_US" LC_ALL=en_US but the system does not support correctly the UTF-8 encoding. Please consider the following scenarios: CASE 1) From target device to outsite - Connect to the target with a terminal - Check that the terminal encoding is set to UTF-8 - Touch a file into a pendrive including in the file name accented characters, e.g. à,è,ì,ò,ù - Connect the pendrive to a Windows system, browse the pen drive - The file name appears corrupted - Connect the pendrive to a Linux system (desktop distro, e.g. Ubuntu), browse the pen drive - The file name appears corrupted (the file name is displayed corrupted exactly as in Windows) CASE 2) From outside to target device - Connect a pendrive to a Windows system - Create a file into the pendrive including in the file name accented characters, e.g. à,è,ì,ò,ù - Connect the pendrive to a Linux system (desktop distro, e.g. Ubuntu), browse the pen drive - The file name appears correctly - Connect to the target with a terminal - Check that the terminal encoding is set to UTF-8 - Connect the pendrive to the target, browser the pendrive - The file name appears corrupted Why en_US locale is not working well with UTF-8? Why en_US locale does not have the .UTF-8 suffix? How can I check that the en_US locale supports the UTF-8 encoding? Reproducibility --------------- Always Thanks in advance for your time Best regards, Fabio Emiliani 71299 1 yoctouser 2017-03-15 10:04:48 +0000 As suggested here: http://stackoverflow.com/questions/42784435/how-to-check-if-a-locale-is-utf-8/42797421#42797421, I've used the nl_langinfo API to detected the encoding of a locale. This is the output: ./LangInfoInspector $(locale -a) C ->ANSI_X3.4-1968 de_DE ->UTF-8 en_GB ->UTF-8 en_GB.ISO-8859-1 ->ISO-8859-1 en_US ->UTF-8 en_US.ISO-8859-1 ->ISO-8859-1 fr_FR ->UTF-8 POSIX ->ANSI_X3.4-1968 zh_CN ->UTF-8 so I confirm that en_US, en_GB and similar are UTF-8 encoded (even if there is no .utf8 suffix). 71388 2 yoctouser 2017-03-17 13:50:02 +0000 I've made further tests and noticed that ISO-8859-1 locales works fine. If using en_US.ISO-8859-1 I'm able to see accented characters such as: à,è,ì,ò,ù. See below: root@imx6qsabresd:/media/win/italiano# ls -la total 12 drwxr-xr-x 3 root root 4096 Mar 13 15:11 . drwxr-xr-x 8 root root 4096 Mar 13 14:57 .. drwxr-xr-x 2 root root 4096 Mar 13 14:58 mà_però -rwxr-xr-x 1 root root 0 Mar 13 14:57 mà_però.txt The directory and file mà_però has been created under Windows. Accented à and ò are correctly displayed. Tested under PuTTY using the following settings (Change settings -> Window -> Translation): ISO-8859-1:1998 (Latin-1, West Europe) 71450 3 yoctouser 2017-03-20 15:38:42 +0000 The problem is caused by the mount of the pendrive. The default mount options include: iocharset=iso8859-1, with this option the pendrive file system is interpreted as ISO 8859-1 and not UTF-8. Solved using iocharset=utf8. See: http://unix.stackexchange.com/q/352144/191458