Bug 1374 - Kernel panic on multilib sato image: lib64-core-image-sato
Summary: Kernel panic on multilib sato image: lib64-core-image-sato
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.1
Assignee: Mark Hatle
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2011-08-16 21:32 UTC by Dongxiao Xu
Modified: 2011-09-13 17:35 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments
Multilib kernel panic (116.62 KB, image/jpeg)
2011-08-16 21:32 UTC, Dongxiao Xu
no flags Details
prelink log (125.49 KB, text/plain)
2011-08-18 22:07 UTC, Dongxiao Xu
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dongxiao Xu 2011-08-16 21:32:01 UTC
Created attachment 204 [details]
Multilib kernel panic

After successfully built out lib64-core-image-sato image and try to launch it in QEmu, we met kernel panic. See attached picture for detailed info.
Comment 1 Dongxiao Xu 2011-08-16 22:36:42 UTC
CC Ke and Lianhao
Comment 2 Dongxiao Xu 2011-08-18 17:59:08 UTC
This issue should be related with prelink. I tried to remove the line in image-prelink.bbclass:

IMAGE_PREPROCESS_COMMAND += "prelink_image; "

and then redo the rootfs, image could boot up successfully.

Mark, do you have any idea on that?

Thanks,
Dongxiao
Comment 3 Mark Hatle 2011-08-18 21:50:50 UTC
in your local.conf search for:

USER_CLASSES ?= "image-mklibs image-prelink"


Remove image prelink.  Does the system boot?

If it doesn, what is your machine configuration, I need to know what the target is to understand what might be going wrong.

I am currently working on a prelink problem that affects PPC64 userspace, but it's possible it affects more then just PPC64.
Comment 4 Dongxiao Xu 2011-08-18 22:07:10 UTC
Created attachment 208 [details]
prelink log
Comment 5 Dongxiao Xu 2011-08-18 22:07:31 UTC
Yes, target system could boot successfully if remove the "image-prelink" in USER_CLASSES.

My target machine is qemux86-64.

Some more findings on my side:
I added some debug info and enabled the verbose output of prelink, there are some error information reporting "undefined symbol", see attached file. 

These undefined symbol (memset, strcpy, etc) belong to libc.so.6, and I checked it with objdump. There do exist such functions.
Comment 6 Dongxiao Xu 2011-08-21 20:51:41 UTC
Hi Mark,

I just built a normal core-image-sato (non-multilib) image of qemux86-64 with baselib set to "lib64", it also met kernel panic when boot.

You said you were trying to solve a ppc64 prelink issue. I saw in bitbake.conf, only ppc64's baselib is set to "lib64". Therefore this bug may also related with that.

Thanks,
Dongxiao
Comment 7 Dongxiao Xu 2011-08-23 01:01:34 UTC
In prelink package, there is one tool named prelink-rtld which is somewhat similar with ldd. 

Running prelink-rtld with format like:

RTLD_TRACE_PRELINKING=/usr/lib64/libfreetype.so.6 PRELINK_SYSROOT=/distro/dongxiao/build-lib/tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs /distro/dongxiao/build-lib/tmp/work/x86_64-linux/prelink-native-1.0+git1+ac461e73b17253a4da25c5aafeac7193b553156c-r5/git/trunk/src//prelink-rtld --target-paths "/usr/lib64/libfreetype.so.6"

The above command will report errors like:

...
lookup 0x00000000dead0000 0x0000000000002a78 -> 0x00000000dead0000 0x0000000000014250 /1 FT_GlyphLoader_CheckSubGlyphs
lookup 0x00000000dead0000 0x0000000000003108 -> 0x00000000dead0000 0x0000000000012df0 /1 FT_Stream_GetUShortLE
lookup 0x00000000dead0000 0x00000000000026e8 -> 0x00000000dead0000 0x000000000001e4a0 /1 FT_Stroker_Export
lookup 0x00000000dead0000 0x00000000000020b8 -> 0x00000000dead0000 0x0000000000010220 /1 ft_corner_orientation
undefined symbol: strcmp        (/usr/lib64/libfreetype.so.6)
lookup 0x00000000dead0000 0x00000000000021d8 -> 0x00000000dead0000 0x0000000000011c10 /1 FT_Get_Module_Interface
lookup 0x00000000dead0000 0x0000000000002a60 -> 0x00000000dead0000 0x000000000001b040 /1 FT_Glyph_Copy
lookup 0x00000000dead0000 0x0000000000003540 -> 0x00000000dead0000 0x0000000000014f60 /1 FT_Outline_New_Internal
lookup 0x00000000dead0000 0x0000000000002748 -> 0x00000000dead0000 0x000000000001ac90 /1 FT_Done_Glyph
lookup 0x00000000dead0000 0x0000000000002fe8 -> 0x00000000dead0000 0x00000000000145e0 /1 FT_GlyphLoader_CreateExtra
lookup 0x00000000dead0000 0x0000000000001db8 -> 0x00000000dead0000 0x00000000000150b0 /1 FT_New_Library
lookup 0x00000000dead0000 0x0000000000003048 -> 0x00000000dead0000 0x0000000000015740 /1 ft_glyphslot_alloc_bitmap
lookup 0x00000000dead0000 0x00000000000029a0 -> 0x00000000dead0000 0x000000000000fb80 /1 FT_MulDiv_No_Round
lookup 0x00000000dead0000 0x00000000000014d0 -> 0x00000000dead1000 0x0000003e13234640 /1 longjmp
lookup 0x00000000dead0000 0x0000000000002bc8 -> 0x00000000dead0000 0x0000000000014e80 /1 FT_Outline_Done_Internal
lookup 0x00000000dead0000 0x00000000000031e0 -> 0x00000000dead0000 0x000000000001e010 /1 FT_Stroker_BeginSubPath
lookup 0x00000000dead0000 0x0000000000001b90 -> 0x00000000dead0000 0x000000000001ab80 /1 FT_Glyph_Transform
lookup 0x00000000dead0000 0x00000000000035b8 -> 0x00000000dead0000 0x00000000000127a0 /1 FT_Outline_Get_Orientation
undefined symbol: memcmp        (/usr/lib64/libfreetype.so.6)
lookup 0x00000000dead0000 0x0000000000001500 -> 0x00000000dead1000 0x0000003e132d02a0 /1 munmap
lookup 0x00000000dead0000 0x0000000000001c50 -> 0x00000000dead0000 0x000000000005e240 /1 FTC_ImageCache_New
lookup 0x00000000dead0000 0x0000000000002c70 -> 0x00000000dead0000 0x0000000000062970 /1 ft_lzwstate_done
lookup 0x00000000dead0000 0x0000000000002e80 -> 0x00000000dead0000 0x0000000000011210 /1 FT_Get_Char_Index
lookup 0x00000000dead0000 0x0000000000001a88 -> 0x00000000dead0000 0x0000000000010530 /1 ft_validator_error
lookup 0x00000000dead0000 0x00000000000032b8 -> 0x00000000dead0000 0x0000000000016150 /1 ft_mem_strcpyn
undefined symbol: strncpy       (/usr/lib64/libfreetype.so.6)
lookup 0x00000000dead0000 0x0000000000003420 -> 0x00000000dead0000 0x0000000000013580 /1 FT_Stream_ReadULongLE
lookup 0x00000000dead0000 0x00000000000034e0 -> 0x00000000dead0000 0x0000000000012520 /1 FT_Outline_Get_CBox
lookup 0x00000000dead0000 0x00000000000017d0 -> 0x00000000dead0000 0x0000000000012da0 /1 FT_Stream_GetChar
...


GDB into the code will find the error is happened in the FCT() function, which is a inner part of the lookup functions. This code is derived from old glibc. I wonder if updating it to latest glibc/eglibc will help.

bug 1331 might be similar with this one.

Thanks,
Dongxiao
Comment 8 Dongxiao Xu 2011-08-23 18:52:52 UTC
Fantastic thing is that, using Mark's prelink-cross WIP tree (http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=mhatle/cross_prelink) which synched prelink element with latest eglibc, this issue gone away.

So the root cause should be related with the old "symbol search" code in prelink.

Thanks Mark!
Comment 9 Dongxiao Xu 2011-09-13 04:40:26 UTC
Hi Mark, 

Could you help to mark the bug as fixed then I can verify it.

Thanks,
Dongxiao
Comment 10 Mark Hatle 2011-09-13 08:16:59 UTC
New version was checked in.  This particular issue is resolved.

Note, BUG 1473 has been discovered recently and may impact testing.
Comment 11 Dongxiao Xu 2011-09-13 17:35:33 UTC
Verified against latest master.