Bug 431

Summary: start x86 unfs qemu fails after source /opt/poky/envrionment-i586-XXX files
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: liping Ke <liping.ke>
Component: meta-yoctoAssignee: Scott Garman <scott.a.garman>
Status: VERIFIED FIXED QA Contact:
Severity: major    
Priority: High CC: jessica.zhang, kevin.tian, lianhao.lu, liping.ke, poky.bs.watcher, poky.watcher, richard.purdie, sgw
Version: 0.9   
Target Milestone: 0.9 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Program Management Documentation change: ---

Description liping Ke 2010-10-11 19:37:16 UTC
1) source /opt/poky/environment-setup-i586XXX
2) start unfs qemu

It will report:

KERNEL: [/home/lke/images/bzImage-qemuX86.bin]
ROOTFS: [/home/lke/unfs]
FSTYPE: [nfs]
Setting up tap interface under sudo
[sudo] password for lke:
tunctl failed:
Comment 1 liping Ke 2010-10-11 19:40:43 UTC
Arm has the same problem.
Comment 2 liping Ke 2010-10-11 19:41:50 UTC
change component
Comment 3 Lianhao Lu 2010-10-11 19:42:42 UTC
PPC had the same problem.
Comment 5 Lianhao Lu 2010-10-11 19:49:21 UTC
If we source the environment file from poky build tree(meta-ide-support), i.e. tmp/environment-setup-XXX, seems the problem gone.
Comment 6 Scott Garman 2010-10-12 13:14:05 UTC
This is happening because tunctl is segfaulting. tunctl is segfaulting because it was compiled with the eglibc that is included in the native sysroot, but the runtime linker is not configured to load the libraries in the native sysroot.

If one sets LD_LIBRARY_PATH to the sysroot's lib/ directory, the problem goes away. But if LD_LIBRARY_PATH is then exported, system commands such as 'ls' will fail. I'm currently looking into possible solutions.
Comment 7 Kevin Tian 2010-10-12 20:03:30 UTC
(In reply to comment #6)
> This is happening because tunctl is segfaulting. tunctl is segfaulting because
> it was compiled with the eglibc that is included in the native sysroot, but the
> runtime linker is not configured to load the libraries in the native sysroot.
> If one sets LD_LIBRARY_PATH to the sysroot's lib/ directory, the problem goes
> away. But if LD_LIBRARY_PATH is then exported, system commands such as 'ls'
> will fail. I'm currently looking into possible solutions.

I'm a bit confusing here. Ideally all -native packages are linked to eglibc in the native sysroot. What makes tunctl special?
Comment 8 Kevin Tian 2010-10-12 22:52:39 UTC
(In reply to comment #6)
> This is happening because tunctl is segfaulting. tunctl is segfaulting because
> it was compiled with the eglibc that is included in the native sysroot, but the
> runtime linker is not configured to load the libraries in the native sysroot.
> If one sets LD_LIBRARY_PATH to the sysroot's lib/ directory, the problem goes
> away. But if LD_LIBRARY_PATH is then exported, system commands such as 'ls'
> will fail. I'm currently looking into possible solutions.

LD_LIBRARY_PATH has global effects on all commands executed under /opt environment. So it's not a good thing to play with. Since libc version is incompatible between /opt and build system, you always break one side.

Ideally if rpath is set correctly for nativesdk packages, they should search /opt first before the build system path. There may have some problem on how those nativesdk packages are linked.

Lianhao just did an experiment. Many /opt commands trigger same segmentation fault, such as genent, genconf, etc. But there're several success with opkg-cl, curl, ncurse, etc. However we didn't find any direct clues from recipes. It's possible that they work just because they don't trigger incompatible interfaces. Perhaps it's a general issue with how a nativesdk package is generated.
Comment 9 Kevin Tian 2010-10-13 00:19:16 UTC
Lianhao and I spent some time on this, and now we think we've got the reason but may require help to make the patch.

First, there's an issue that some nativesdk binaries are linked to libc.so.6 without desired rpath, such as tunctl, pseudo, etc. That's a bug but generally OK. The order of library search is LD_LIBRARY_PATH->rpath->ld.so.cache->default path. binutils-crosssdk has set default path to /opt, and there's no LD_LIBRARY_PATH and ld.so.cache. So /opt/.../libc.so.6 is found which works

then due to below commit:

commit ad042da70348e9f301a25b855248e8fbce495ab8
Author: Richard Purdie <rpurdie@linux.intel.com>
Date:   Fri Oct 8 15:20:42 2010 +0100

    meta-toolchain: Fix libGL.so not found issues

    Fix libGL.so not found issues by making sure ld.so.cache from the host syste
    is used to locate libraries.

    Also, fix a /usr/ reference that should be /var to keep the installation
    tarball clean.

    [BUGID #228]

    Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>

a ld.so.cache is created which links to host /etc/ld.so.cache to help search libGL.so, which all contains host library paths. This then breaks binaries which don't have rpath encoded.

There do have some /opt binaries with rpath encoded, such as ncurse, curl, etc.

The temporary workaround for SDK team is to remove /opt/.../etc/ld.so.cache since their build systems don't have libGL.so issue. The right fix is perhaps to enforce strict rpath for all nativesdk binaries, but we don't know why it's not there yet.
Comment 10 Richard Purdie 2010-10-14 12:58:41 UTC
Should be fixed in http://git.pokylinux.org/cgit.cgi/poky/commit/?id=462aef73fad5f01afa89fe76da57032fc5abceee but need someone to test.
Comment 11 Saul Wold 2010-10-14 14:37:30 UTC
Checked into 431
Comment 12 Jessica 2010-10-14 16:11:40 UTC
Verified against latest master by trying tunctl, extract-poky-sdk