Bug 335 - Bitbake of ncurses fails with undefined symbol: _PyUnicodeUCS4_AsDefaultEncodedString
Summary: Bitbake of ncurses fails with undefined symbol: _PyUnicodeUCS4_AsDefaultEncod...
Status: VERIFIED FIXED
Alias: None
Product: Meta-yocto
Classification: Build System, Metadata & Runtime
Component: meta-yocto (show other bugs)
Version: 0.9
Hardware: x86 Multiple
: Medium normal
Target Milestone: 0.9 M4
Assignee: Saul Wold
QA Contact:
URL:
Whiteboard:
: 370 (view as bug list)
Depends on:
Blocks:
 
Reported: 2010-09-16 13:28 UTC by Daniel Díaz
Modified: 2010-12-21 06:07 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified: Program Management
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Díaz 2010-09-16 13:28:52 UTC
Trying to compile (or clean) ncurses or ncurses-native fails due to an undefined symbol in Python's operator.so:

-----8<----------8<----------8<-----
$ time bitbake -c clean ncurses-native
NOTE: Handling BitBake files: | (0936/0936) [100 %]
Parsing of 936 .bb files complete (853 cached, 83 parsed). 1069 targets, 53 skipped, 0 masked, 0 errors.

OE Build Configuration:
BB_VERSION        = "1.11.0"
METADATA_BRANCH   = "master"
METADATA_REVISION = "c01e8760ba5552e91f3af764c1233c2462494213"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "pandaboard"
DISTRO            = "poky"
DISTRO_VERSION    = "3.3+snapshot-20100916"
TARGET_FPU        = "soft"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing runqueue
NOTE: Full skip list set([])
NOTE: Running task 1 of 1 (ID: 0, virtual:native:/home/ddiazlocal/poky/meta/recipes-core/ncurses/ncurses_5.4.bb, do_clean)
Traceback (most recent call last):
  File "/home/ddiazlocal/poky/scripts/..//bitbake//bin/bitbake-runtask", line 32, in <module>
    import bb.event
  File "/home/ddiazlocal/poky/scripts/..//bitbake/lib/bb/__init__.py", line 31, in <module>
    import bb.msg
  File "/home/ddiazlocal/poky/scripts/..//bitbake/lib/bb/msg.py", line 26, in <module>
    import collections
  File "/usr/lib/python2.6/collections.py", line 9, in <module>
    from operator import itemgetter as _itemgetter
ImportError: /usr/lib/python2.6/lib-dynload/operator.so: undefined symbol: _PyUnicodeUCS4_AsDefaultEncodedString
ERROR: Task 0 (virtual:native:/home/ddiazlocal/poky/meta/recipes-core/ncurses/ncurses_5.4.bb, do_clean) failed with 1
ERROR: 'virtual:native:/home/ddiazlocal/poky/meta/recipes-core/ncurses/ncurses_5.4.bb' failed

real    0m19.596s
user    0m16.080s
sys     0m0.360s
----->8---------->8---------->8-----

This is on Gentoo with python-2.6.5-r3 (latest of the Python 2.6.x series). For what it's worth, no previous version of operator.so presented that symbol:

-----8<----------8<----------8<-----
$ for i in 2.4 2.5 2.6; do echo "Python ${i}: `nm -D /usr/lib/python${i}/lib-dynload/operator.so | grep UCS`"; done
Python 2.4: 
Python 2.5: 
Python 2.6:          U _PyUnicodeUCS4_AsDefaultEncodedString
----->8---------->8---------->8-----

All other packages seem to build correctly.
Comment 1 Saul Wold 2010-09-20 09:10:13 UTC
David,

Can you try this again with the latest master, I was not able to reproduce this issue.  We do not have a Gentoo System, but I have tried it on Fedora 64 as well as OpenSuse 64.

Thanks
Comment 2 Daniel Díaz 2010-09-25 01:09:01 UTC
(In reply to comment #1)
> David,
> 
> Can you try this again with the latest master, I was not able to reproduce this
> issue.  We do not have a Gentoo System, but I have tried it on Fedora 64 as
> well as OpenSuse 64.
> 
> Thanks

Paul,

It failed just the same with the newer master:
-----8<----------8<----------8<-----
$ time bitbake ncurses-native 
NOTE: Handling BitBake files: \ (0839/0839) [100 %]
Parsing of 839 .bb files complete (793 cached, 46 parsed). 967 targets, 57 skipped, 0 masked, 0 errors.

OE Build Configuration:
BB_VERSION        = "1.11.0"
METADATA_BRANCH   = "master"
METADATA_REVISION = "09f1f6939e5ec6a64c602bfb609e247082736ce8"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "pandaboard"
DISTRO            = "poky"
DISTRO_VERSION    = "3.3+snapshot-20100925"
TARGET_FPU        = "soft"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing runqueue
NOTE: Full skip list set([])
Traceback (most recent call last):
  File "/home/ddiazlocal/poky/scripts/..//bitbake//bin/bitbake-runtask", line 32, in <module>
    import bb.event
  File "/home/ddiazlocal/poky/scripts/..//bitbake/lib/bb/__init__.py", line 31, in <module>
    import bb.msg
  File "/home/ddiazlocal/poky/scripts/..//bitbake/lib/bb/msg.py", line 26, in <module>
    import collections
  File "/usr/lib/python2.6/collections.py", line 9, in <module>
    from operator import itemgetter as _itemgetter
ImportError: /usr/lib/python2.6/lib-dynload/operator.so: undefined symbol: _PyUnicodeUCS4_AsDefaultEncodedString
NOTE: Running task 335 of 457 (ID: 12, virtual:native:/home/ddiazlocal/poky/meta/recipes-core/ncurses/ncurses_5.4.bb, do_package)
ERROR: Task 12 (virtual:native:/home/ddiazlocal/poky/meta/recipes-core/ncurses/ncurses_5.4.bb, do_package) failed with 1
ERROR: 'virtual:native:/home/ddiazlocal/poky/meta/recipes-core/ncurses/ncurses_5.4.bb' failed

real    0m13.365s
user    0m12.053s
sys     0m0.364s
----->8---------->8---------->8-----

It's important to note that my host's Python is indeed built with UCS4, whereas Poky's isn't:
-----8<----------8<----------8<-----
$ find . -name operator.so | while read n; do echo "$n: `nm -D $n | grep UCS`"; done
tmp/work/i686-linux/python-native-2.6.5-ml8.3/Python-2.6.5/build/lib.linux-i686-2.6/operator.so:          U _PyUnicodeUCS2_AsDefaultEncodedString
tmp/work/i686-linux/python-native-2.6.5-ml8.3/image/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/python2.6/lib-dynload/operator.so:          U _PyUnicodeUCS2_AsDefaultEncodedString
tmp/work/i686-linux/python-native-2.6.5-ml8.3/sysroot-destdir/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/python2.6/lib-dynload/operator.so:          U _PyUnicodeUCS2_AsDefaultEncodedString
tmp/work/i686-linux/python-native-2.6.5-ml8.3/pstage-build-populate-sysroot/sysroots/i686-linux/usr/lib/python2.6/lib-dynload/operator.so:          U _PyUnicodeUCS2_AsDefaultEncodedString
----->8---------->8---------->8-----
Comment 3 Daniel Díaz 2010-09-25 01:11:20 UTC
(In reply to comment #2)
[...]
> It's important to note that my host's Python is indeed built with UCS4, whereas
> Poky's isn't:
> -----8<----------8<----------8<-----
> $ find . -name operator.so | while read n; do echo "$n: `nm -D $n | grep UCS`";
> done
> tmp/work/i686-linux/python-native-2.6.5-ml8.3/Python-2.6.5/build/lib.linux-i686-2.6/operator.so:
>          U _PyUnicodeUCS2_AsDefaultEncodedString
> tmp/work/i686-linux/python-native-2.6.5-ml8.3/image/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/python2.6/lib-dynload/operator.so:
>          U _PyUnicodeUCS2_AsDefaultEncodedString
> tmp/work/i686-linux/python-native-2.6.5-ml8.3/sysroot-destdir/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/python2.6/lib-dynload/operator.so:
>          U _PyUnicodeUCS2_AsDefaultEncodedString
> tmp/work/i686-linux/python-native-2.6.5-ml8.3/pstage-build-populate-sysroot/sysroots/i686-linux/usr/lib/python2.6/lib-dynload/operator.so:
>          U _PyUnicodeUCS2_AsDefaultEncodedString
> ----->8---------->8---------->8-----

Missed one more:
tmp/sysroots/i686-linux/usr/lib/python2.6/lib-dynload/operator.so:          U _PyUnicodeUCS2_AsDefaultEncodedString
Comment 4 Saul Wold 2010-09-30 17:19:41 UTC
Daniel,

Could you do me a favor and run bitbake under strace (strace -s 8096 -f -o strace.out bitbake ...) and then attach the strace.out file to the bug. This will help me determine what might be happening.

Thanks for your help.
Sau!
Comment 5 Daniel Díaz 2010-10-01 00:19:29 UTC
(In reply to comment #4)
> Daniel,
> Could you do me a favor and run bitbake under strace (strace -s 8096 -f -o
> strace.out bitbake ...) and then attach the strace.out file to the bug. This
> will help me determine what might be happening.

Good idea! It's a bit big, so I'm putting it here instead:
  http://afuera.cortijodelrio.net/~ddiaz/bitbake-strace.out.gz
Comment 6 Richard Purdie 2010-10-01 04:01:16 UTC
The key part of the log is:
10079 stat64("/usr/bin/python", {st_mode=S_IFREG|0755, st_size=9532, ...}) = 0
10079 execve("/usr/bin/python2.6", ["/usr/bin/python2.6", "/home/ddiazlocal/poky/scripts/..//bitbake//bin/bitbake-runtask", "virtual:native:/home/ddiazlocal/poky/meta/recipes-core/ncurses/ncurses_5.4.bb", "do_package", "False"], [/* 83 vars */]) = 0
10079 brk(0)                            = 0x804b000
10079 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7725000
10079 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls/i686/sse2/libpython2.6.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
10079 stat64("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls/i686/sse2", 0xbfcccee0) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls/i686/libpython2.6.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
10079 stat64("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls/i686", 0xbfcccee0) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls/sse2/libpython2.6.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
10079 stat64("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls/sse2", 0xbfcccee0) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls/libpython2.6.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
10079 stat64("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/tls", 0xbfcccee0) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/i686/sse2/libpython2.6.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
10079 stat64("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/i686/sse2", 0xbfcccee0) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/i686/libpython2.6.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
10079 stat64("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/i686", 0xbfcccee0) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/sse2/libpython2.6.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
10079 stat64("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/sse2", 0xbfcccee0) = -1 ENOENT (No such file or directory)
10079 open("/home/ddiazlocal/poky/build/tmp/sysroots/i686-linux/usr/lib/libpython2.6.so.1.0", O_RDONLY) = 3

which shows it using the system python with the poky python lib. That would cause exactly the error you see. Now the question becomes why?
Comment 7 Richard Purdie 2010-10-01 04:02:56 UTC
If you comment out the export LD_LIBRARY_PATH in the ncurses.inc file, what happens?
Comment 8 Daniel Díaz 2010-10-02 01:40:05 UTC
> If you comment out the export LD_LIBRARY_PATH in the ncurses.inc file, what
> happens?

That is pure, black, black magic!:

> NOTE: Tasks Summary: Attempted 712 tasks of which 711 didn't need to be rerun and 0 failed.

Commenting out LD_LIBRARY_PATH did the trick. Thanks a bunch, Richard!
Comment 9 Jessica 2010-10-04 13:10:53 UTC
Commenting out LD_LIBRARY_PATH fix the problem
Comment 10 Jessica 2010-10-04 13:12:27 UTC
Daniel's last message verify the fix
Comment 11 Jessica 2010-10-04 13:14:02 UTC
*** Bug 370 has been marked as a duplicate of this bug. ***
Comment 12 Dexuan Cui 2010-10-07 20:16:58 UTC
(In reply to comment #10)
> Daniel's last message verify the fix

Yes, I can also verify the fix ("Commenting out LD_LIBRARY_PATH fix the problem").