Bug 10165

Summary:	Check that the locale en_US.UTF-8 is available
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	David Frey <dfrey>
Component:	bitbake	Assignee:	Frank de Brabander <debrabander>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	bluelightning, debrabander, Falital, poky.bs.watcher, poky.watcher, ross.burton, sgw, vincent.libre
Version:	unspecified
Target Milestone:	4.99
Hardware:	x86
OS:	Multiple
Whiteboard:	NEWCOMER
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description David Frey 2016-08-19 16:08:28 UTC

I was trying to follow the 'hello world' example of the Bitbake documentation from appendix a.

This is the output from my Bitbake run:
$ bitbake printHello
Parsing recipes: 100% |#########################################################################################################################################################################################################################################################################################| Time: 0:00:00
Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |######################################################################################################################################################################################################################################################################################| Time: 0:00:00
NOTE: Executing RunQueue Tasks
Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a utf-8 when python starts or things won't work.
NOTE: Tasks Summary: Attempted 1 tasks of which 1 didn't need to be rerun and all succeeded.

$ locale
LANG=en_CA.UTF-8
LANGUAGE=en_CA:en
LC_CTYPE="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME="en_CA.UTF-8"
LC_COLLATE="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_PAPER="en_CA.UTF-8"
LC_NAME="en_CA.UTF-8"
LC_ADDRESS="en_CA.UTF-8"
LC_TELEPHONE="en_CA.UTF-8"
LC_MEASUREMENT="en_CA.UTF-8"
LC_IDENTIFICATION="en_CA.UTF-8"
LC_ALL=

I am running from the master branch of bitbake which is currently version 3ff1c66.

I did a bit of investigation and bitbake-worker's call to sys.getfilesystemencoding() is returning "ascii".  Interestingly, if I just run "/usr/bin/env python3" and then call sys.getfilesystemencoding(), I get "utf-8".

I asked about this problem in #oe on freenode and Richard Purdie asked if I have the en_US locale installed.  I did not.  I ran dpkg-reconfigure locales and selected en_US.utf-8 to be generated and the problem went away.  I still had en_CA.utf-8 as my default locale though.

Suggestion: If bitbake requires en_US.utf-8, then the error message should indicate that.  If bitbake only requires any utf-8 locale then it should work with en_CA.utf-8 (and any other utf-8 locale).

Comment 1 Ross Burton 2016-08-24 21:15:22 UTC

bitbake itself doesn't care what locale as long as UTF-8 is supported, so this is odd.  Are you using buildtools?  What release/git SHA are you using of OE?

Comment 2 David Frey 2016-08-25 16:20:04 UTC

This is the version of bitbake that I am using:

$ git remote -v
origin  git://git.openembedded.org/bitbake (fetch)
origin  git://git.openembedded.org/bitbake (push)

$ git log -n 1 HEAD
commit 3ff1c66e6f336e5de7dcbc983a97fcd19ddc6b81
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Thu Aug 18 10:05:26 2016 +0100

    bitbake: Update version to 1.31.1
    
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Comment 3 David Frey 2016-08-25 16:22:31 UTC

And no, I am not using buildtools.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.5 (jessie)
Release:        8.5
Codename:       jessie

Comment 4 Ross Burton 2016-08-25 16:50:04 UTC

I've a slightly newer version of bitbake locally and the same release of Debian.

Are you using oe-core, or your own metadata layer?

Comment 5 Ross Burton 2016-08-25 16:51:08 UTC

Whilst the check shouldn't be failing, we should extend the check in bitbake to be "is current locale utf-8 and is en_US.UTF-8 available?".

Comment 6 David Frey 2016-08-26 15:36:30 UTC

(In reply to comment #4)
> I've a slightly newer version of bitbake locally and the same release of
> Debian.
> 
> Are you using oe-core, or your own metadata layer?

I am pretty new to bitbake, so I don't even really understand the question.  What I can tell you is that I just cloned the bitbake repository and then wrote a few .bb and .conf files as directed by the "hello world" example from appendix A of the bitbake documentation.

Relevant files in my directory are:
conf/bitbake.conf
conf/bblayers.conf
classes/base.bbclass
mainLayer/printHello.bb
mainLayer/conf/layer.conf

Comment 7 Paul Eggleton 2017-03-21 03:27:17 UTC

We are actually setting LC_ALL in a couple of places to en_US.UTF-8. I think that's unfortunate, but that probably does mean we need to state that that's what we need installed (but not selected).

Comment 8 Vincent Rubiolo 2017-10-10 05:52:13 UTC

I encountered the same problem today when building Pyro from the git tree within a Fedora 23 Docker container:
[yocto@abf49f8bb774 build]$ bitbake -h
Please use a locale setting which supports utf-8.
Python can't change the filesystem locale after loading so we need a utf-8 when python starts or things won't work.

The locale is set to POSIX by default:
$ locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=


My HEAD is at:
commit 717303e6fbcbbe181ad9645d762eb5a85d934523 (HEAD -> pyro, origin/pyro)
Author: Gerson Fernando Budke <nandojve@gmail.com>
Date:   Fri Sep 22 07:29:08 2017 +0200

    image.bbclass: Sorted ctypes to avoid basehash error
[...]

The workaround for me was to export LANG=en.US_utf-8 via Docker ENV, as suggested by https://github.com/fedora-cloud/docker-brew-fedora/issues/14:


It would be nice if the bitbake error message was more explicit indeed.

Comment 9 Ross Burton 2017-10-10 09:10:09 UTC

Vincent, your situation isn't the same.  In your case you started with a non-UTF8 locale, and were told by bitbake to use a UTF-8 locale.  I don't think we want to start telling people to use en_US.UTF-8 as that is only relevant for Americans.

(This bug is about the obscure error you get if en_US.UTF-8 isn't available at all)

Comment 10 Sawas Etairidis 2022-12-07 11:04:19 UTC

Hi Frank, 

I also wanted to start with this one but you beat me to it :). I would like to share what I found in my investigation. The reason for this happens is that the LC_ALL enviroment variable is set to en_US.UTF-8 if the system does not support this encoding all processes started by bitbake revert to a filesystem encoding ascii. The line which does this is in ./bitbake/lib/bb/utils.py::622 of source rev a0dcf93bc919757b23a52597eacb1777df4adf24

Regards
Savvas

Comment 11 Frank de Brabander 2022-12-08 14:58:05 UTC

Check added, fix merged in master/master-next as a4ce040a.