Bug 299

Summary: qemuarm images can not boot up
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: xianchao zhang <xianchao.zhang>
Component: meta-yoctoAssignee: Bruce Ashfield <bruce.ashfield>
Status: VERIFIED FIXED QA Contact:
Severity: critical    
Priority: High CC: jiajun.xu, kevin.tian, kunwei.tian, poky.bs.watcher, poky.watcher, qing.he, richard.purdie, sgw, xianchao.zhang
Version: 0.9   
Target Milestone: 0.9 M3   
Hardware: x86   
OS: arm   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Program Management Documentation change: ---

Description xianchao zhang 2010-09-08 23:34:41 UTC
Testing Env:    
Host: Ubuntu 10.04 LTS
Kernel: 2.6.32-21-generic-pae
Image location: http://autobuilder.pokylinux.org/build-areas/milestone-release/build/build/tmp/deploy/images/

1. download the related image from milestone-release build.
2. boot up qemuarm with minimal rootfs and this kernel:zImage-2.6.34+git0+4294840a21f3c65b4b005dcc606728e684a1e613_0+52f2e29cd24831c424c251c7f06234ef72047fdb-r6-qemuppc-20100909005030.bin
3. during booting up, there will arise kernel call trace like below, and the system was halting.

###################

md: ... autorun DONE.
EXT3-fs (hda): warning: feature flags set on rev 0 fs, running e2fsck is recommended
EXT3-fs (hda): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) readonly on device 3:0.
Freeing unused kernel memory: 184k init
kjournald starting.  Commit interval 5 seconds
request_module: runaway loop modprobe binfmt-4c46
request_module: runaway loop modprobe binfmt-4c46
request_module: runaway loop modprobe binfmt-4c46
request_module: runaway loop modprobe binfmt-4c46
request_module: runaway loop modprobe binfmt-4c46
INFO: task swapper:1 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
swapper       D 00000000     0     1      0 0x00000000
Call Trace:
[c781db30] [22248048] 0x22248048 (unreliable)
[c781dbf0] [c000944c] __switch_to+0x4c/0x6c
[c781dc00] [c0371fc8] schedule+0x41c/0x57c
[c781dc60] [c03729e4] schedule_timeout+0x2a4/0x32c
[c781dcd0] [c03725f8] wait_for_common+0x168/0x1e0
[c781dd10] [c0046c60] call_usermodehelper_exec+0x94/0xb8
[c781dd40] [c0046f7c] __request_module+0x1a0/0x214
[c781dde0] [c00d6f04] search_binary_handler+0x2f0/0x3d0
[c781de40] [c00d82ac] do_execve+0x2b8/0x370
[c781de80] [c00097a8] sys_execve+0x50/0x7c
[c781dea0] [c001264c] ret_from_syscall+0x0/0x38
--- Exception: c01 at kernel_execve+0x8/0x14
    LR = init_post+0x98/0x154
[c781df60] [c0003a7c] init_post+0x3c/0x154 (unreliable)
[c781df80] [c048d2a4] kernel_init+0x184/0x1bc
[c781dff0] [c0012470] original_kernel_thread+0x4c/0x68
INFO: task khelper:28 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
khelper       D 00000000     0    28     27 0x00000000
Call Trace:
[c796fba0] [c796fc48] 0xc796fc48 (unreliable)
[c796fc60] [c000944c] __switch_to+0x4c/0x6c
[c796fc70] [c0371fc8] schedule+0x41c/0x57c
[c796fcd0] [c03729e4] schedule_timeout+0x2a4/0x32c
[c796fd40] [c03725f8] wait_for_common+0x168/0x1e0
[c796fd80] [c0046c60] call_usermodehelper_exec+0x94/0xb8
[c796fdb0] [c0046f7c] __request_module+0x1a0/0x214
[c796fe50] [c00d6f04] search_binary_handler+0x2f0/0x3d0
[c796feb0] [c00d82ac] do_execve+0x2b8/0x370
[c796fef0] [c00097a8] sys_execve+0x50/0x7c
[c796ff10] [c001264c] ret_from_syscall+0x0/0x38
--- Exception: c01 at kernel_execve+0x8/0x14

####################
Comment 1 xianchao zhang 2010-09-09 01:14:17 UTC
(In reply to comment #0)
> Testing Env:    
> Host: Ubuntu 10.04 LTS
> Kernel: 2.6.32-21-generic-pae
> Image location:
> http://autobuilder.pokylinux.org/build-areas/milestone-release/build/build/tmp/deploy/images/
> 
> 1. download the related image from milestone-release build.
> 2. boot up qemuarm with minimal rootfs and this
> kernel:zImage-2.6.34+git0+4294840a21f3c65b4b005dcc606728e684a1e613_0+52f2e29cd24831c424c251c7f06234ef72047fdb-r6-qemuppc-20100909005030.bin
> 3. during booting up, there will arise kernel call trace like below, and the
> system was halting.
Sorry, drop the above comments please, they were not an incorrect configuration, the kernel mismatch the rootfs.

The arm arch kernel zImage-2.6.34+git0+4294840a21f3c65b4b005dcc606728e684a1e613_0+62b5ac5dc508a740fcd4cee432a6c12d0e366389-r6-qemuarm-20100908203316 was also failed to boot up.

During booting up progress, it will pop up a window with black screen, the system seems blocking.
Comment 2 Jiajun Xu 2010-09-09 01:36:39 UTC
If starting qemuarm with option "-nographic", we can see it stops at
"
LTT : ltt-relay init
CLCD: unknown LCD panel ID 0x00001000, using VGA
CLCD: Versatile hardware, VGA display
"

The full log here:
#############
jxu49@vt-nhm4:~/osel/poky-contrib-new/build$ runqemu qemuarm ext3
/home/jxu49/test/M3-20100909/zImage-2.6.34+git0+4294840a21f3c65b4b005dcc606728e684a1e613_0+62b5ac5dc508a740fcd4
cee432a6c12d0e366389-r6-qemuarm-20100908203316.bin
/home/jxu49/test/M3-20100909/poky-image-sdk-qemuarm-20100908220841.rootfs.ext3
ln: creating symbolic link
`/home/jxu49/osel/poky-contrib-new/build/tmp/sysroots/i686-linux/bin/bin/cc':
No such file or directory
ln: creating symbolic link
`/home/jxu49/osel/poky-contrib-new/build/tmp/sysroots/i686-linux/bin/bin/gcc':
No such file or directory
Using preconfigured tap device 'tap0'
Warning: distccd not present, no distcc support loaded.
Running qemu-system-arm...
/home/jxu49/osel/poky-contrib-new/build/tmp/sysroots/i686-linux/usr/bin/qemu-system-arm
-kernel
/home/jxu49/test/M3-20100909/zImage-2.6.34+git0+4294840a21f3c65b4b005dcc606728e684a1e613_0+62b5ac5dc508a740fcd4cee432a6c12d0e366389-r6-qemuarm-20100908203316.bin
-net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=no,downscript=no -M
versatilepb -hda
/home/jxu49/test/M3-20100909/poky-image-sdk-qemuarm-20100908220841.rootfs.ext3
-no-reboot -show-cursor -usb -usbdevice wacom-tablet -nographic --append
"root=/dev/sda console=ttyAMA0,115200 console=tty
ip=192.168.7.2::192.168.7.1:255.255.255.0 mem=64M highres=off"
Uncompressing Linux... done, booting the kernel.
Linux version 2.6.34.6-wr-standard (pokybuild@autobuilder) (gcc version 4.5.0
20100729 (prerelease) (GCC) ) #1 PREEMPT Wed Sep 8 13:28:25 PDT 2010
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00093177
CPU: VIVT data cache, VIVT instruction cache
Machine: ARM-Versatile PB
Memory policy: ECC disabled, Data cache writeback
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16256
Kernel command line: root=/dev/sda console=ttyAMA0,115200 console=tty
ip=192.168.7.2::192.168.7.1:255.255.255.0 mem=64M highres=off
PID hash table entries: 256 (order: -2, 1024 bytes)
Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
Memory: 64MB = 64MB total
Memory: 59232k/59232k available, 6304k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
    vmalloc : 0xc4800000 - 0xd8000000   ( 312 MB)
    lowmem  : 0xc0000000 - 0xc4000000   (  64 MB)
    modules : 0xbf000000 - 0xc0000000   (  16 MB)
      .init : 0xc0008000 - 0xc0029000   ( 132 kB)
      .text : 0xc0029000 - 0xc04c3000   (4712 kB)
      .data : 0xc0514000 - 0xc0544560   ( 194 kB)
SLUB: Genslabs=11, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Hierarchical RCU implementation.
RCU-based detection of stalled CPUs is enabled.
NR_IRQS:192
VIC @f1140000: id 0x00041190, vendor 0x41
Console: colour dummy device 80x30
console [tty0] enabled
Calibrating delay loop... 413.69 BogoMIPS (lpj=2068480)
Security Framework initialized
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
NET: Registered protocol family 16
Serial: AMBA PL011 UART driver
dev:f1: ttyAMA0 at MMIO 0x101f1000 (irq = 12) is a AMBA/PL011
console [ttyAMA0] enabled
dev:f2: ttyAMA1 at MMIO 0x101f2000 (irq = 13) is a AMBA/PL011
dev:f3: ttyAMA2 at MMIO 0x101f3000 (irq = 14) is a AMBA/PL011
fpga:09: ttyAMA3 at MMIO 0x10009000 (irq = 38) is a AMBA/PL011
PCI core found (slot 11)
PCI: bus0: Fast back to back transfers disabled
PCI map irq: slot 0, pin 1, devslot 12, irq: 27
PCI map irq: slot 0, pin 1, devslot 13, irq: 27
pci 0000:00:0d.0: BAR 2: assigned [mem 0x50000000-0x50001fff]
pci 0000:00:0d.0: BAR 2: set to [mem 0x50000000-0x50001fff] (PCI address
[0x50000000-0x50001fff]
pci 0000:00:0d.0: BAR 1: assigned [mem 0x50002000-0x500023ff]
pci 0000:00:0d.0: BAR 1: set to [mem 0x50002000-0x500023ff] (PCI address
[0x50002000-0x500023ff]
pci 0000:00:0c.0: BAR 0: assigned [mem 0x50002400-0x500024ff]
pci 0000:00:0c.0: BAR 0: set to [mem 0x50002400-0x500024ff] (PCI address
[0x50002400-0x500024ff]
pci 0000:00:0d.0: BAR 0: assigned [io  0x44000000-0x440000ff]
pci 0000:00:0d.0: BAR 0: set to [io  0x44000000-0x440000ff] (PCI address
[0x44000000-0x440000ff]
bio: create slab <bio-0> at 0
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource timer3
Switched to NOHz mode on CPU #0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 2048 (order: 2, 16384 bytes)
TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Registering unionfs 2.5.4 (for 2.6.34-rc0)
JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
msgmni has been set to 115
alg: No test for stdrng (krng)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
LTT : ltt-relay init
CLCD: unknown LCD panel ID 0x00001000, using VGA
CLCD: Versatile hardware, VGA display
#############
Comment 3 Bruce Ashfield 2010-09-09 19:38:12 UTC
I've reset all the recent kernel merges. boot still fails. so this probably has been lurking for a while.
Comment 4 Bruce Ashfield 2010-09-09 19:44:19 UTC
Looks to be configuration only. bisecting the options now.
Comment 5 Bruce Ashfield 2010-09-09 20:50:28 UTC
I have the options isolated. 

root@qemuarm:~# uname -a
Linux qemuarm 2.6.34.6-wr-standard #1 PREEMPT Thu Sep 9 23:43:11 EDT 2010 armv5tejl GNU/Linux

a few more tests and this is done.
Comment 6 Bruce Ashfield 2010-09-09 22:11:43 UTC
I just pushed a fix to the kernel repository for this.
I couldn't test the sato image from here, but I've done
multiple text mode boots.

NO_HZ is causing a problem, and on qemu, I'm happy to just
disable it. I've got someone digging into it further now
that I've isolated the option.

The change is on wrs_meta:

http://git.pokylinux.org/?p=linux-2.6-windriver.git;a=summary

And my SRCREV update:

The following changes since commit d52767c4cbc46dc962fe54949ad2672de30ee0ba:
  Bruce Ashfield (1):
        qemu: add qemu-mipsel to the list of simulators

are available in the git repository at:

  ssh://git@git.pokylinux.org/poky-contrib zedd/kernel

Bruce Ashfield (1):
      linux-wrs/meta: fix qemuarm boot issues with 2.6.34.6

 .../conf/distro/include/poky-default-revisions.inc |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Comment 7 Bruce Ashfield 2010-09-09 22:33:18 UTC
I've now been able to boot the sato image and interact with it (mouse/keyboard). So the fix looks complete.
Comment 8 Saul Wold 2010-09-10 14:15:57 UTC
Fixed in both Master and M3 (build3) e41b6843188911fabdea6143b6180e98487ed7d1
Comment 9 Jiajun Xu 2010-09-14 07:48:41 UTC
Xianchao,
Pls. verify if the bug is fixed or not in latest M3 build.