Bug 34

Summary: String collation failing
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Matt Hoosier <matt.hoosier>
Component: meta-yoctoAssignee: Dongxiao Xu <dongxiao.xu>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Low CC: poky.bs.watcher, poky.watcher, sgw
Version: unspecified   
Target Milestone: ---   
Hardware: x86   
OS: x86   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Matt Hoosier 2007-10-24 14:26:59 UTC
The glibc built-in mechanism for doing locale-dependent string sorting seems to
be malfunctioning with glibc 2.5 as present in Blinky.

The following test program should report -1 as the result of calling wcscoll(),
but it rather thinks that they're equal.

#include <wchar.h>
#include <locale.h>
#include <stdio.h>
#include <stdlib.h>

#define A "A"
#define B "B"
#define WCS_A L"A"
#define WCS_B L"B"

int
main ()
{
    const wchar_t * a = WCS_A;
    const wchar_t * b = WCS_B;

    setenv ("LC_ALL", "en_GB.utf8", 1);
    setlocale (LC_ALL, "");

    printf ("wcscoll(\"%s\", \"%s\"): %d\n", A, B, wcscoll (a, b));
    return 0;
}

I think this might be something particular to glibc-2.5, because I've used this
test successfully on glibc-2.3 from an external CSL toolchain, and also
successfully on my Gutsy desktop running glibc-2.6.1
Comment 1 Richard Purdie 2007-10-24 14:53:19 UTC
Some quick tests here show two devices with current poky images give -1 as the
result so I can't seem to reproduce the problem. I don't think glibc has changed
since blinky. Were you using the standard toolchain or an external one?
Comment 2 Matt Hoosier 2007-10-24 15:11:13 UTC
Standard poky toolchain. I don't think it makes any difference, because I unset
all my LANG-related environment variables on my desktop to rule it out, but just
in case: do you have anything like $LANG or $LC_ALL set on your device's
environments?
Comment 3 Richard Purdie 2007-10-26 12:44:21 UTC
I just checked and nothing language related is set...
Comment 4 Richard Purdie 2007-10-26 13:23:34 UTC
Something I noticed which may or may not be related is that on my Ubuntu system,
 ls /usr/lib/locale/en_GB gives "en_GB/ en_GB.utf8/" in poky based images we
have "en_GB/ en_GB.iso8859/" (assuming the en_GB.iso8859 package is installed)
with I think en_GB being the utf8 version.

Just to try and make sure we're comparing the same systems, you have binary
locale generation enabled? Which locales are installed?

Comment 5 Richard Purdie 2007-12-18 15:26:00 UTC
Matt, did you resolve this issue?
Comment 6 Matt Hoosier 2007-12-18 20:11:56 UTC
No, in fact I just had this crop up as the cause for another symptom lately.

Regarding your question about my setup and set of installed locales:

* I've tried both with and without binary locale generation; same result.

* I install with a whole gaggle of locales; en_GB is among them.

* When using binary locales, I have the following choices in /usr/lib/locale for UK English:

  /usr/lib/locale/en_GB

  I know this looks like an easy culprit since the source snippet uses "en_GB.utf8", but other locale-dependent things seem to work okay. And if I change the failing example to use "en_GB", things still fail.

* When using nonbinary locales, it's harder to tell for sure what locales survived on the device since they're all stuff away in /usr/lib/locale/locale-archive. I assume they're the same?
Comment 7 Richard Purdie 2007-12-18 20:17:59 UTC
Are you using prelink?
Comment 8 Matt Hoosier 2007-12-18 20:32:39 UTC
I currently use prelink, though I've tested and got the failure on images explicitly built to leave that out.
Comment 9 Matt Hoosier 2007-12-18 20:33:47 UTC
I'm going to test this on a plain old poky-image-sato and see what happens. Perhaps there's some unintended clobbering in my company's proprietary software stack.
Comment 10 Richard Purdie 2007-12-18 20:37:32 UTC
I was just thinking that poky trunk has some bugfixes to the prelink arm_eabi patch and that the old version could in theory see problems like that bad test result. If you can test a standard poky-image-sato that would be good, I have tested and couldn't reproduce in that...
Comment 11 Matt Hoosier 2007-12-18 20:58:14 UTC
poky-image-sato still gives the me same result. I don't think we really have anything too invasive redefined for our machine setup; only usual stuff like selecting the serial console, naming the rootfs device, etc.

I wonder whether you ever happened to try this with the include/tune-xscale.conf machine type. It's a longshot, but perhaps buggy code is being emitted for the locale module?
Comment 12 Richard Purdie 2007-12-18 21:31:33 UTC
richard@max:/tmp$ /usr/local/poky/eabi-glibc/arm/bin/arm-poky-linux-gnueabi-gcc test.c -o test
richard@max:/tmp$ scp test root@192.168.1.37:/
richard@max:/tmp$ ssh root@192.168.1.37
root@spitz:~# /test
wcscoll("A", "B"): -1

spitz uses include/tune-xscale.conf

The only difference I can see is that this is with poky trunk rather than blinky but I can't think of any changes that would affect this...
Comment 13 Matt Hoosier 2007-12-18 22:53:45 UTC
Hmm, I just completely fresh build of poky-image-sato with the inclusion of tune-xscale.conf commented out.

The collation routines work in this configuration. I'm going to try to see whether this is related to the code in libc-2.5.so or the data payload of the locales, and post back.
Comment 14 Matt Hoosier 2007-12-19 03:54:58 UTC
It turns out that I didn't control all the variables in my test for building with(out) xscale tuning. DEBUG_BUILD is the real culprit for me.

Seems that localedef, when built with debugging info and fewer optimizations, emits locale data that's broken somehow. I can take a system built with DEBUG_BUILD=1 and overwrite /usr/lib/locale/locale-archive with a copy generated on a system using DEBUG_BUILD=0, and the string collation is repaired.
Comment 15 Matt Hoosier 2007-12-20 00:03:06 UTC
The bug's (still don't know precisely what) in ld-collate.c.

If you recompile just that one file using ${FULL_OPTIMIZATION} rather than ${DEBUG_OPTIMIZATION}, the collation tables generated are usable again.
Comment 16 Dongxiao Xu 2010-08-25 22:27:14 UTC
Test the program on netbook with glibc-2.10.1, the result returns -1 correctly. I think we can close this bug, Saul?
Comment 17 Dongxiao Xu 2010-09-01 19:04:18 UTC
Marked it as WORKSFORME since it could not be reproduced on my side.