Bug 649 - QEMU Segfaults with nographic option
Summary: QEMU Segfaults with nographic option
Status: RESOLVED FIXED
Alias: None
Product: Meta-yocto
Classification: Build System, Metadata & Runtime
Component: meta-yocto (show other bugs)
Version: 0.9
Hardware: x86 Multiple
: High major
Target Milestone: 1.0 M4
Assignee: Edwin Zhai
QA Contact:
URL:
Whiteboard:
: 698 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-01-07 10:26 UTC by Mark Hatle
Modified: 2011-03-31 05:19 UTC (History)
9 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Hatle 2011-01-07 10:26:04 UTC
When the nographic option is selected I receive a segfault.  I believe this is occurring when the system attempts to startup the graphic console:

 (Note this is not mips specific, I get the same error on x86 and ppc)

poky-qemu qemumips nographic


Running qemu-system-mips...
/home/mhatle/git/poky/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-mips -kernel /home/mhatle/git/poky/build/tmp/deploy/images/vmlinux-qemumips.bin -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=no,downscript=no -M malta -hda /home/mhatle/git/poky/build/tmp/deploy/images/poky-image-sato-qemumips.ext3 -no-reboot -vga cirrus -show-cursor -usb -usbdevice wacom-tablet -no-reboot -nographic --append "root=/dev/hda console=ttyS0 console=tty ip=192.168.7.2::192.168.7.1:255.255.255.0 mem=128M "
Linux version 2.6.37-rc8-yocto-standard+ (mhatle@msp-mhatle-lx2) (gcc version 4.5.1 (GCC) ) #1 PREEMPT Fri Jan 7 10:21:15 CST 2011
bootconsole [early0] enabled
CPU revision is: 00019300 (MIPS 24Kc)
FPU revision is: 00000000
Determined physical RAM map:
...

ide1 at 0x170-0x177,0x376 on irq 15
ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
ide-gd driver 1.18
hda: max request size: 512KiB
hda: 1080586 sectors (553 MB) w/256KiB Cache, CHS=1072/255/63
hda: cache flushes supported
/home/mhatle/git/poky/scripts/poky-qemu-internal: line 449: 29535 Segmentation fault      (core dumped) $QEMUBIN -kernel $KERNEL $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT --append "$KERNCMDLINE $SCRIPT_KERNEL_OPT"
Set 'tap0' nonpersistent
Releasing lockfile of preconfigured tap device 'tap0'


In the initial setup the -vga cirrus is still enable, even with the -nographic option could this be part of the problem?
Comment 1 Edwin Zhai 2011-01-11 23:45:58 UTC
Mark,
I can't reproduce this bug with commit 4b055ed in poky master on ubuntu. Generaly speaking, "-vga cirrus" should be removed when specifying "-nographic", but qemu should handle this well and treat "nographic" as higher priority, which is observed on my side.

Can you boot without "nographic"? If this is specific issue with "nographic", and it's hard to reproduce on my side, is it possible to let me use your machine?

Thanks,
edwin
Comment 2 Mark Hatle 2011-01-12 08:04:27 UTC
I removed the GL pass-through patches:


-    file://qemu-git-qemugl-host.patch \
-    file://fix-nogl.patch \
-    file://qemugl-allow-glxcontext-release.patch \
-    file://init-info.patch \
-    file://enable-i386-linux-user.patch \
-    file://vmware-vga-fifo-rewind.patch \

and compiled with --disable-sdl.  After that the crash went away.  My host system has the nvidia graphics drivers installed on it.  So likely it was a combination of the nvidia graphics drivers, and QEMU still using graphics resources even when -nographics was enabled.

I believe this points to a bug in QEMU, that even when nographics is selected it attempts to initialize the OpenGL resources -- which triggers the incompatibility with the Nvidia drivers.  This may or may not be triggered b the -vga cirrus...

We do need to make sure there is a way to run QEMU without graphics being used at all, to avoid problems like this in the future.
Comment 3 Edwin Zhai 2011-02-17 00:10:30 UTC
(In reply to comment #2)
> I removed the GL pass-through patches:
> and compiled with --disable-sdl.  After that the crash went away.  My host
> system has the nvidia graphics drivers installed on it.  So likely it was a
> combination of the nvidia graphics drivers, and QEMU still using graphics
> resources even when -nographics was enabled.

> We do need to make sure there is a way to run QEMU without graphics being used
> at all, to avoid problems like this in the future.

Mark,
As I can't reproduce it on my side, could you pls. retest it after upgrading qemu to 0.13?
If it still exist, could you pls. use gdb to located where the seg fault come from?

Only possible reason is that GL patch is not clean enough and touch display screen. 

In theory, it has nothing to do with nvidia hardware. If qemu runs well in normal, it should also with "nographic". There is no difference regarding hardware resource access. But it's just my guess, need further investigation.
Comment 4 Edwin Zhai 2011-02-17 17:41:46 UTC
Mark,
Could you pls. help to provide more info after upgrading qemu?
Thanks,
Comment 5 Mark Hatle 2011-02-24 10:46:22 UTC
With the latest version of Poky (master as of 2011-02-24), QEMU still segfaults..  

poky-qemu qemuppc nographic

...

 hda: unknown partition table
/home/mhatle/git/poky/scripts/poky-qemu-internal: line 449: 21531 Segmentation fault      (core dumped) $QEMUBIN -kernel $KERNEL $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT --append "$KERNCMDLINE $SCRIPT_KERNEL_OPT"

By applying the patch on the top of the poky-contrib mhatle/qemu branch -- and running with "nographic" as above, it does not crash.

This might be related to the automatic use of "-enable-gl" in the qemux86 and qemux86-64 configurations.
Comment 6 Edwin Zhai 2011-03-09 22:18:49 UTC
Mark,
I have tried latest nightly build, and made sure it works well with "nographic". Can you try it also too see if this issue is machine related or build related?
Sorry for bother you again, but I need reproduce first then fix it:)

    HOWTO:
    1. download disk image
    http://autobuilder.pokylinux.org/nightly/20110305-4/machines/qemu/qemumips/poky-image-sato-qemumips-20110303092454.rootfs.ext3

    2. download mips kernel
    http://autobuilder.pokylinux.org/nightly/20110305-4/machines/qemu/qemumips/vmlinux-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+76f30ca447fd0f06e76d0cb4935b016a1440e23e-r16-qemumips-20110303091203.bin

    3. download native environment from 
http://git.pokylinux.org/cgit.cgi/poky-contrib/plain/poky-lsb.tar.bz2?h=gzhai/for_mark

    4. unpack it under /var/,
       cd poky-lsb/0.9+snapshot/
       source environment-setup-mips-poky-linux
       poky-qemu mips-kernel-name disk-image-name nographic
Comment 7 Mark Hatle 2011-03-10 06:34:02 UTC
To cause the crash you need a computer with an NVIDIA video card, and the proprietary (binary) nvidia X and OpenGL drivers.

If you are unable to get to one, let me know and I'll attempt to get you access to a machine so you can reproduce the problem.

I have not yet tested the exact build you are mentioning below, however I did try the HEAD of bernard on Tuesday an experienced exactly the same problem as before.  Even with "nographic" being set (and also making sure that -enable-gl is not passed to qemu), it still crashes as QEMU starts.

The only way I could work around this was remove the custom QEMU OpenGL patches and disable the SDL library when building qemu.  (This is what the patch does at the top of mhatle/qemu.)

Somehow we need to completely avoid even initializing OpenGL if -enable-gl is not passed to QEMU.
Comment 8 Mark Hatle 2011-03-10 16:21:41 UTC
I have tried the steps you outlined, it still crashes:

hda: 556982 sectors (285 MB) w/256KiB Cache, CHS=552/255/63
hda: cache flushes supported
/opt/poky-lsb/0.9+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/poky-qemu-internal: line 442: 22879 Segmentation fault      (core dumped) $QEMUBIN -kernel $KERNEL $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT --append "$KERNCMDLINE $SCRIPT_KERNEL_OPT"
Set 'tap0' nonpersistent
Releasing lockfile of preconfigured tap device 'tap0'

unfortunately the core that was dumped is bad and all it tells me is that again, it's a segmentation fault.

The host system is a Fedora 13 system, with the binary Nvidia drivers installed: NVIDIA-Linux-x86_64-260.19.21.run

The video card is: 03:00.0 VGA compatible controller: nVidia Corporation G98 [Quadro NVS 295] (rev a1)
Comment 9 Edwin Zhai 2011-03-13 18:50:31 UTC
Mark,
I can reproduce it. debug is on going...
Comment 10 Saul Wold 2011-03-13 22:41:16 UTC
See also 698
Comment 11 Edwin Zhai 2011-03-15 18:44:30 UTC
Finally I found this bug is duplicated bug from 698, caused by nVidia GL
lib. As long as qemu is linked with nVidia GL lib, even upstream qemu,
there must be segfault regardless of "-nographic" setting. 

Possible root cause is: 1. nVidia GL lib provides some symbol override
generic one, and cause segfauls when calling these symbol. 2. The .init
section in nVidia GL lib dirty qemu's data structure and lead the final
segfault.
Kevin and I went through the related symbols, and excluded 1. So we
doubt .init section need some fix. 

Before it got fixed in nVidia GL lib, we can have 3 options:
1. Add some checks in qemu's configure, and remove "-lGL" if detecting
nVidia's GL lib installed. It cause trouble when building and running machine are different.
2. Use dlopen rather than "-lGL", so we can avoid touching bogus GL lib
if specifying "-nographic" or no "-enable-gl". By this way, we can turn of gl when running. But need change source code to access the lib symbol, which is a little bit tricky.
3. Document it as known issue, and ask end user don't use nVidia driver.
Comment 12 Kevin Tian 2011-03-15 19:58:55 UTC
option 1 is not accepted since the build system could be different from the maching running Qemu.

option 2 has some risk. dlopen/dlsyms are not a very stable approach, and we need to wrap all the opengl calls which are not trivial. Some NVidia driver version even overwrites dlsyms with its own version to cause trouble (http://www.nvnews.net/vbulletin/showthread.php?p=2070826). So it also needs careful test to ensure it not blocking other things when making slightly better in this nv case.

option 3 is possibly the choice we can do now for 1.0, given the limited time and the risk from option 2. User could fall back to default xorg-x11 libgl driver to use Qemu, and we may address it in next point release

Edwin will continue post his progress of course to see whether we can have another less risky option
Comment 13 Kevin Tian 2011-03-15 23:04:45 UTC
Edwin and I got another alternative. We could use LD_PRELOAD=/usr/lib/libGL.so to make sure xorg libGL is used instead of nVidia version. dynamic loader (ld.so) binds the symbol in reverse order, while LD_PRELOAD adds specificed library in the head. This ensures xorg libGL taking effect, while user can still use proprietary nVidia driver for his other desktop activity.

We have tested that this approach basically works. Edwin will do more test and then add conditional check in poky-qemu script to make sure LD_PRELOADED prefixed only when nVidia proporietary driver is detected. Then it should have no impact to other build environment
Comment 14 Edwin Zhai 2011-03-15 23:53:29 UTC
Mark,
Can you have a look at where is the nVidia's libGL on your fedora system?
/usr/lib/nvidia-current/libGL.so.xxx, or /usr/lib64/nvidia-current/libGL.so.xxx?

I'll cook up a patch to skip nvidia's libGL if finding it.
Comment 15 Scott Garman 2011-03-16 00:52:27 UTC
Edwin,

In case this is helpful, on my 64-bit Ubuntu 10.10 system, NVIDIA's libGL is in /usr/lib64/nvidia-current/libGL.so
Comment 16 Edwin Zhai 2011-03-16 01:41:18 UTC
Mark & Scott,
I have pushed 2 patches @ the branch http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=gzhai/for_mark

Can you have a try to see if it can fix this issue?

Thanks,
edwin
Comment 17 Scott Garman 2011-03-16 10:49:09 UTC
Edwin,

This appears to fix the problem on my system. Nice work!
Comment 18 Mark Hatle 2011-03-16 17:43:44 UTC
I have verified on Fedora 13 that the nvidia software gets installed into simply /usr/lib64.


lrwxrwxrwx. 1 root root     10 Jan 21 10:54 /usr/lib64/libGL.so -> libGL.so.1*
lrwxrwxrwx. 1 root root     18 Jan 21 10:54 /usr/lib64/libGL.so.1 -> libGL.so.260.19.21*
-rwxr-xr-x. 1 root root 982536 Jan 21 10:54 /usr/lib64/libGL.so.260.19.21*

The system version is removed, it would have been "/usr/lib64/libGL.so.1.2".

You might be able to detect it though by looking for:

libnvidia-cfg.so                 libnvidia-cfg.so.260.19.21       libnvidia-glcore.so.260.19.21
libnvidia-cfg.so.1               libnvidia-compiler.so.260.19.21  libnvidia-tls.so.260.19.21

this libGL.so.260.19.21 has internal dependencies on:

 0x0000000000000001 (NEEDED)             Shared library: [libnvidia-tls.so.260.19.21]
 0x0000000000000001 (NEEDED)             Shared library: [libnvidia-glcore.so.260.19.21]

That is likely a clue as well.

My suggestion is simply looking at the needed values on the libGL.so installed on the system, then looking for "libnvidia" contained within.  If that is found then trigger a warning/error that this is known to conflict with QEMU.
Comment 19 Mark Hatle 2011-03-16 17:57:38 UTC
I attempted to run the poky-contrib gzhai/for_mark branch.  It unfortunately resulted in the same core dump as before.

As mentioned in my previous comment.  Likely a good alternative is to check the libraries actually being loaded.  One method would be:

ldd tmp/sysroots/x86_64-linux/usr/bin/qemu | grep nvidia

if the return code is "0", then qemu was linked against an nvidia driver and may fail to load.  If it's != 0, then the nvidia libraries are not present and things are likely to work.
Comment 20 Kevin Tian 2011-03-16 18:09:36 UTC
yes, this way is more sane though we can expect all nvidia drivers falling into this pattern
Comment 21 Kevin Tian 2011-03-16 18:10:03 UTC
(In reply to comment #20)
> yes, this way is more sane though we can expect all nvidia drivers falling into
> this pattern

'can'->'can't'
Comment 22 Edwin Zhai 2011-03-17 01:56:41 UTC
Mark,
Can you test if following fix works on your fedora system? thanks,
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=gzhai/fix2&id=f596757000465a4b8350e16f21553a23b8bbedfa
Comment 23 Edwin Zhai 2011-03-17 20:33:44 UTC
After discussion with Mark, we agree we just give warnings to end user and document this as known issue, as different distribution need different solution.

New fix is @
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=gzhai/fix3&id=1c56783be10e6d51a801ccbee9e9ec5ae2c5db32
Comment 24 Mark Hatle 2011-03-17 20:39:32 UTC
This patch looks good to me.
Comment 25 Edwin Zhai 2011-03-17 20:49:55 UTC
Mark,
I have created a new bug requesting for the doc: bug 902.
Could you pls. provide fix for fedora?

Thanks,
Comment 26 Edwin Zhai 2011-03-23 19:03:04 UTC
Got fixed in master @ c0298557bad2e431f6dc87b6fda061cab561e14c
Comment 27 Edwin Zhai 2011-03-31 05:19:46 UTC
*** Bug 698 has been marked as a duplicate of this bug. ***