Bug 9395

Summary: Cleanly support unicode throughout Toaster
Product: [Build System, Metadata & Runtime] Toaster Reporter: Joshua Lock <joshuagloe>
Component: toasterAssignee: David Reyna <david.reyna>
Status: RESOLVED WORKSFORME QA Contact: Libertad <libertad.gonzalez.de.la.cruz>
Severity: critical    
Priority: Medium CC: aveselinov, belen.barros.pena, brian.avery, david.reyna, jessica.zhang, stanciux.mihail
Version: 2.2   
Target Milestone: Future   
Hardware: x86   
OS: x86   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Joshua Lock 2016-04-05 12:25:52 UTC
In https://bugzilla.yoctoproject.org/show_bug.cgi?id=9142 we observed exceptions being raised when an ImagePkgList MetadataEvent contains non-ASCII characters. Worse still, these exceptions were invisible to the Toaster UI.

A workaround patch has been proposed for 2.1 to workaround the specific issue of handling file paths with non-ASCII characters, however we should make an effort in the 2.2 timeframe to ensure that we have unicode support throughout Toaster.

This is likely to be made much easier should the plan to switch BitBake to Python3 go ahead, see:

https://docs.python.org/2/howto/unicode.html
https://docs.python.org/3/howto/unicode.html
Comment 1 aveselinov 2016-10-19 07:47:00 UTC
I'm getting this error whenever I try to build:

'ascii' codec can't decode byte 0xc2 in position 10: ordinal not in range(128)

On Ubuntu 15.04 machine.
Comment 2 Joshua Lock 2016-10-19 15:08:32 UTC
(In reply to comment #1)
> I'm getting this error whenever I try to build:
> 
> 'ascii' codec can't decode byte 0xc2 in position 10: ordinal not in
> range(128)
> 
> On Ubuntu 15.04 machine.

Can you provide more details on your setup so that we can try and reproduce? Is there a more detailed backtrace available?
Comment 3 brian avery 2016-10-19 16:02:54 UTC
In particular, for additional information we need: 
toaster version (jethro,krogoth,morty,master...?)
build target : (core-image-minimal,bash,...?)
Additional Layers if any?
Variables changed if any?
Comment 4 aveselinov 2016-10-20 06:55:37 UTC
I have downloaded the vendor BSP for my board (which have a collection of meta layers + poky)
I have installed Toaster krogoth following the manual from yocto website.

compulab-bsp/
├── build
│   ├── buildhistory
│   ├── cache
│   ├── conf
│   └── tmp
├── downloads
│   ├── git2
│   └── uninative
├── sources
│   ├── base
│   ├── Documentation
│   ├── meta-browser      //not used
│   ├── meta-compulab      
│   ├── meta-fsl-arm
│   ├── meta-fsl-arm-extra
│   ├── meta-fsl-demos    //not used
│   ├── meta-openembedded
│   ├── meta-qt5          //not used
│   └── poky
├── _toaster_clones
└── venv
    ├── bin
    ├── include
    ├── lib
    └── local

I init Virtual env:
$ source venv/bin/activate

And start toaster:
$ source sources/poky/bitbake/bin/toaster

The webserver address that I get it isn't 127.0.0.1:8000 but 0.0.0.0:8000

target image: core-image-base
no additional variables

The traceback error msg that I get when I press the build button:

2016-10-20 08:45:34,486 ERROR runbuilds: Error launching build 'ascii' codec can't decode byte 0xc2 in position 10: ordinal not in range(128)
Traceback (most recent call last):
  File "/home/anatoli/compulab-bsp/sources/poky/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py", line 67, in schedule
    "%d:%d" % (br.pk, bec.be.pk))
  File "/home/anatoli/compulab-bsp/sources/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 255, in triggerBuild
    layers = self.setLayers(bitbake, layers, targets)
  File "/home/anatoli/compulab-bsp/sources/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 155, in setLayers
    self._shellcmd('git clone "%s" "%s"' % (giturl, localdirname))
  File "/home/anatoli/compulab-bsp/sources/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 68, in _shellcmd
    err = "command: %s \n%s" % (command, err)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 10: ordinal not in range(128)
2016-10-20 08:45:35,533 DEBUG Saving error 'ascii' codec can't decode byte 0xc2 in position 10: ordinal not in range(128) (cm-fx6-toaster (Release krogoth (krogoth), BBV krogoth (Branch: krogoth)) failed)
Comment 5 brian avery 2016-11-09 22:16:24 UTC
Hmm, we are having trouble reproducing this.
Thanks for the additional information!

A couple of things,
0) in krogoth you get a server bound to 0.0.0.0, which will allow you to access the server from a different machine. in morty, it defaults to 127.0.0.1:8000, so it wil only be accessible locally unless you add webport=0.0.0.0:8000 to the toaster start command.
1) Could you try the new release? it is called morty.  We did a switchover to Python 3.4 in the Morty release and cleaned up some (but i fear not all) of the unicode issues.
2) it looks like the command that is failing is     self._shellcmd('git clone "%s" "%s"' % (giturl, localdirname))
and it looks like a non ascii character is in one of these.  Any idea where it's creeping in?

3) I couldn't replicate your error on krogoth. Where doed meta-compulab come from?  did you pull all the other layers in usig toaster and layers.openembedded.org?
Comment 6 brian avery 2016-11-14 21:02:01 UTC
waiting for more info so I can reproduce
Comment 7 brian avery 2016-11-28 19:12:48 UTC
Not enough information to reproduce the specific occurrence of the bug.  The general work of insuring unicode support throughout toaster is still a good idea.
Comment 8 brian avery 2017-06-15 15:10:47 UTC
failed to reproduce :(