8060 2015-07-28 11:00:35 +0000 Occasional qemuarm kernel panic (linux 4.1, gcc 4.9.3) 2015-10-01 14:29:06 +0000 1 1 7 Build System, Metadata & Runtime OE-Core kernel 1.8 x86 Multiple RESOLVED FIXED Medium normal 2.0 1 richard.purdie anibal.limon anibal.limon bruce.ashfield elizabeth.flanagan jku juro.bystricky tom.zanussi --- --- No (bug/feature does not impact docs) oldest_to_newest 52651 0 richard.purdie 2015-07-28 11:00:35 +0000 https://autobuilder.yoctoproject.org/main/builders/nightly-arm/builds/416/steps/Running%20Sanity%20Tests/logs/stdio 52652 1 richard.purdie 2015-07-28 11:01:06 +0000 ERROR: Qemu log output from /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato-sdk/1.0-r0/testimage/qemu_boot_log.20150728064520: Uncompressing Linux... done, booting the kernel. Booting Linux on physical CPU 0x0 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Initializing cgroup subsys cpuacct Linux version 4.1.2-yocto-standard (pokybuild@opensuse131) (gcc version 4.9.3 (GCC) ) #1 PREEMPT Wed Jul 22 08:52:25 GMT 2015 CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00093177 CPU: VIVT data cache, VIVT instruction cache Machine: ARM-Versatile PB Memory policy: Data cache writeback sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32480 Kernel command line: root=/dev/sda rw console=ttyAMA0,115200 console=tty ip=192.168.7.6::192.168.7.5:255.255.255.0 mem=128M highres=off rootfstype=ext4 console=tty1 console=ttyS0,115200n8 PID hash table entries: 512 (order: -1, 2048 bytes) Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) Memory: 119480K/131072K available (6880K kernel code, 514K rwdata, 1868K rodata, 360K init, 631K bss, 11592K reserved, 0K cma-reserved) Virtual kernel memory layout: vector : 0xffff0000 - 0xffff1000 ( 4 kB) fixmap : 0xffc00000 - 0xfff00000 (3072 kB) vmalloc : 0xc8800000 - 0xff000000 ( 872 MB) lowmem : 0xc0000000 - 0xc8000000 ( 128 MB) modules : 0xbf000000 - 0xc0000000 ( 16 MB) .text : 0xc0008000 - 0xc0893434 (8750 kB) .init : 0xc0894000 - 0xc08ee000 ( 360 kB) .data : 0xc08ee000 - 0xc096ebc4 ( 515 kB) .bss : 0xc096ebc4 - 0xc0a0cb40 ( 632 kB) SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Preemptible hierarchical RCU implementation. Additional per-CPU info printed with stalls. NR_IRQS:224 VIC @f1140000: id 0x00041190, vendor 0x41 FPGA IRQ chip 0 "SIC" @ f1003000, 13 irqs, parent IRQ: 63 clocksource timer3: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns Console: colour dummy device 80x30 console [tty0] enabled Calibrating delay loop... 471.85 BogoMIPS (lpj=2359296) pid_max: default: 32768 minimum: 301 Security Framework initialized Mount-cache hash table entries: 1024 (order: 0, 4096 bytes) Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes) Initializing cgroup subsys blkio Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys debug CPU: Testing write buffer coherency: ok ftrace: allocating 22469 entries in 66 pages Setting up static identity map for 0x8400 - 0x8458 devtmpfs: initialized VFP support v0.3: implementor 41 architecture 1 part 10 variant 9 rev 0 clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns xor: measuring software checksum speed arm4regs : 596.000 MB/sec 8regs : 317.600 MB/sec 32regs : 762.800 MB/sec xor: using function: 32regs (762.800 MB/sec) NET: Registered protocol family 16 DMA: preallocated 256 KiB pool for atomic coherent allocations Serial: AMBA PL011 UART driver dev:f1: ttyAMA0 at MMIO 0x101f1000 (irq = 44, base_baud = 0) is a PL011 rev1 console [ttyAMA0] enabled dev:f2: ttyAMA1 at MMIO 0x101f2000 (irq = 45, base_baud = 0) is a PL011 rev1 dev:f3: ttyAMA2 at MMIO 0x101f3000 (irq = 46, base_baud = 0) is a PL011 rev1 fpga:09: ttyAMA3 at MMIO 0x10009000 (irq = 70, base_baud = 0) is a PL011 rev1 PCI core found (slot 11) PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff pref] pci_bus 0000:00: root bus resource [io 0x1000-0xffff] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff] PCI: bus0: Fast back to back transfers disabled pci 0000:00:0d.0: BAR 2: assigned [mem 0x50000000-0x50001fff] pci 0000:00:0d.0: BAR 1: assigned [mem 0x50002000-0x500023ff] pci 0000:00:0c.0: BAR 0: assigned [mem 0x50002400-0x500024ff] pci 0000:00:0d.0: BAR 0: assigned [io 0x1000-0x10ff] raid6: int32x1 gen() 249 MB/s raid6: int32x1 xor() 153 MB/s raid6: int32x2 gen() 256 MB/s raid6: int32x2 xor() 157 MB/s raid6: int32x4 gen() 110 MB/s raid6: int32x4 xor() 91 MB/s raid6: int32x8 gen() 123 MB/s raid6: int32x8 xor() 78 MB/s raid6: using algorithm int32x2 gen() 256 MB/s raid6: .... xor() 157 MB/s, rmw enabled raid6: using intx1 recovery algorithm vgaarb: loaded SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb Switched to clocksource timer3 NET: Registered protocol family 2 TCP established hash table entries: 1024 (order: 0, 4096 bytes) TCP bind hash table entries: 1024 (order: 0, 4096 bytes) TCP: Hash tables configured (established 1024 bind 1024) UDP hash table entries: 256 (order: 0, 4096 bytes) UDP-Lite hash table entries: 256 (order: 0, 4096 bytes) NET: Registered protocol family 1 RPC: Registered named UNIX socket transport module. RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. pci 0000:00:0c.0: enabling device (0100 -> 0102) futex hash table entries: 256 (order: -1, 3072 bytes) NFS: Registering the id_resolver key type Key type id_resolver registered Key type id_legacy registered jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc. aufs 4.1-20150629 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) io scheduler noop registered io scheduler deadline registered io scheduler cfq registered (default) clcd-pl11x dev:20: PL110 rev0 at 0x10120000 clcd-pl11x dev:20: Versatile hardware, VGA display Console: switching to colour frame buffer device 80x30 brd: module loaded sym53c8xx 0000:00:0d.0: enabling device (0100 -> 0103) sym0: <895a> rev 0x0 at pci 0000:00:0d.0 irq 94 sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking sym0: SCSI BUS has been reset. scsi host0: sym-2.2.3 sym0: unknown interrupt(s) ignored, ISTAT=0x5 DSTAT=0x80 SIST=0x0 scsi 0:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.3. PQ: 0 ANSI: 5 scsi target0:0:0: tagged command queuing enabled, command queue depth 16. scsi target0:0:0: Beginning Domain Validation scsi target0:0:0: Domain Validation skipping write tests scsi target0:0:0: Ending Domain Validation scsi 0:0:2:0: CD-ROM QEMU QEMU CD-ROM 2.3. PQ: 0 ANSI: 5 scsi target0:0:2: tagged command queuing enabled, command queue depth 16. scsi target0:0:2: Beginning Domain Validation scsi target0:0:2: Domain Validation skipping write tests scsi target0:0:2: Ending Domain Validation smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre <nico@fluxnic.net> sd 0:0:0:0: [sda] 4788610 512-byte logical blocks: (2.45 GB/2.28 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA smc91x smc91x.0 eth0: SMC91C11xFD (rev 1) at c8a64000 IRQ 57 [nowait] smc91x smc91x.0 eth0: Ethernet addr: 52:54:00:12:34:56 sd 0:0:0:0: [sda] Attached SCSI disk ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ehci-pci: EHCI PCI platform driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver ohci-pci: OHCI PCI platform driver ohci-pci 0000:00:0c.0: OHCI PCI host controller ohci-pci 0000:00:0c.0: new USB bus registered, assigned bus number 1 ohci-pci 0000:00:0c.0: irq 93, io mem 0x50002400 hub 1-0:1.0: USB hub found hub 1-0:1.0: 3 ports detected uhci_hcd: USB Universal Host Controller Interface driver mousedev: PS/2 mouse device common for all mice rtc-pl031 dev:e8: rtc core: registered pl031 as rtc0 md: linear personality registered for level -1 md: raid0 personality registered for level 0 md: raid1 personality registered for level 1 md: raid10 personality registered for level 10 md: multipath personality registered for level -4 md: faulty personality registered for level -5 device-mapper: ioctl: 4.31.0-ioctl (2015-3-12) initialised: dm-devel@redhat.com mmci-pl18x fpga:05: No vmmc regulator found mmci-pl18x fpga:05: No vqmmc regulator found mmci-pl18x: probe of fpga:05 failed with error -22 mmci-pl18x fpga:0b: No vmmc regulator found mmci-pl18x fpga:0b: No vqmmc regulator found mmci-pl18x: probe of fpga:0b failed with error -22 usbcore: registered new interface driver usbhid usbhid: USB HID core driver oprofile: no performance counters oprofile: using timer interrupt. u32 classifier Actions configured NET: Registered protocol family 10 input: AT Raw Set 2 keyboard as /devices/fpga:06/serio0/input/input0 sit: IPv6 over IPv4 tunneling driver NET: Registered protocol family 17 Key type dns_resolver registered Btrfs loaded console [netcon0] enabled netconsole: network logging started rtc-pl031 dev:e8: setting system clock to 2015-07-28 06:46:35 UTC (1438065995) smc91x smc91x.0 eth0: link up IP-Config: Complete: device=eth0, hwaddr=52:54:00:12:34:56, ipaddr=192.168.7.6, mask=255.255.255.0, gw=192.168.7.5 host=192.168.7.6, domain=, nis-domain=(none) bootserver=255.255.255.255, rootserver=255.255.255.255, rootpath= usb 1-1: new full-speed USB device number 2 using ohci-pci input: Wacom Penpartner Pen as /devices/pci0000:00/0000:00:0c.0/usb1/1-1/1-1:1.0/0003:056A:0000.0001/input/input3 wacom 0003:056A:0000.0001: : USB HID v10.01 Mouse [QEMU Wacom PenPartner] on usb-0000:00:0c.0-1/input0 input: ImExPS/2 Generic Explorer Mouse as /devices/fpga:07/serio1/input/input2 md: Waiting for all devices to be available before autodetect md: If you don't use raid, use raid=noautodetect md: Autodetecting RAID arrays. md: Scanned 0 and added 0 devices. md: autorun ... md: ... autorun DONE. EXT4-fs (sda): mounted filesystem with ordered data mode. Opts: (null) VFS: Mounted root (ext4 filesystem) on device 8:0. devtmpfs: mounted Freeing unused kernel memory: 360K (c0894000 - c08ee000) udevd[70]: starting version 182 random: nonblocking pool is initialized sd 0:0:0:0: [sda] ABORT operation started sd 0:0:0:0: ABORT operation timed-out. Kernel panic - not syncing: assertion "i && sym_get_cam_status(cp->cmd) == DID_SOFT_ERROR" failed: file "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work-shared/qemuarm/kernel-source/drivers/scsi/sym53c8xx_2/sym_hipd.c", line 3399 CPU: 0 PID: 0 Comm: swapper Not tainted 4.1.2-yocto-standard #1 Hardware name: ARM-Versatile PB [<c0016ba4>] (unwind_backtrace) from [<c0013044>] (show_stack+0x20/0x24) [<c0013044>] (show_stack) from [<c0634b20>] (dump_stack+0x20/0x28) [<c0634b20>] (dump_stack) from [<c0630b08>] (panic+0x94/0x1ec) [<c0630b08>] (panic) from [<c042022c>] (sym_interrupt+0x59c/0x17a0) [<c042022c>] (sym_interrupt) from [<c041bb64>] (sym53c8xx_intr+0x54/0xbc) [<c041bb64>] (sym53c8xx_intr) from [<c0059aa8>] (handle_irq_event_percpu+0xb8/0x280) [<c0059aa8>] (handle_irq_event_percpu) from [<c0059ce0>] (handle_irq_event+0x70/0x90) [<c0059ce0>] (handle_irq_event) from [<c005cb3c>] (handle_level_irq+0xe8/0x14c) [<c005cb3c>] (handle_level_irq) from [<c00591ec>] (generic_handle_irq+0x30/0x40) [<c00591ec>] (generic_handle_irq) from [<c03a5588>] (fpga_irq_handle+0x9c/0xa8) [<c03a5588>] (fpga_irq_handle) from [<c00591ec>] (generic_handle_irq+0x30/0x40) [<c00591ec>] (generic_handle_irq) from [<c0059430>] (__handle_domain_irq+0xa0/0xdc) [<c0059430>] (__handle_domain_irq) from [<c0009450>] (vic_handle_irq+0x74/0x98) [<c0009450>] (vic_handle_irq) from [<c063a140>] (__irq_svc+0x40/0x6c) Exception stack(0xc08efe28 to 0xc08efe70) fe20: 00000000 c078b4d3 00000101 20000113 c090ae90 ffff9883 fe40: 00000202 c096f300 00000001 c08eff30 0000000a c08efebc c08efe60 c08efe70 fe60: c0395f50 c0024ea0 20000113 ffffffff [<c063a140>] (__irq_svc) from [<c0024ea0>] (__do_softirq+0x84/0x344) [<c0024ea0>] (__do_softirq) from [<c0025410>] (irq_exit+0x94/0xbc) [<c0025410>] (irq_exit) from [<c0059438>] (__handle_domain_irq+0xa8/0xdc) [<c0059438>] (__handle_domain_irq) from [<c0009450>] (vic_handle_irq+0x74/0x98) [<c0009450>] (vic_handle_irq) from [<c063a140>] (__irq_svc+0x40/0x6c) Exception stack(0xc08eff30 to 0xc08eff78) ff20: 00000000 00093177 00092177 60000013 ff40: c08ee000 00000000 c08f0080 c08f0080 c08cfab8 c7ffc840 008cce44 c08eff84 ff60: 600000d3 c08eff78 c0010268 c0010274 60000013 ffffffff [<c063a140>] (__irq_svc) from [<c0010274>] (arch_cpu_idle+0x3c/0x44) [<c0010274>] (arch_cpu_idle) from [<c004da4c>] (cpu_startup_entry+0x22c/0x280) [<c004da4c>] (cpu_startup_entry) from [<c063022c>] (rest_init+0x80/0x98) [<c063022c>] (rest_init) from [<c0894ca4>] (start_kernel+0x36c/0x3dc) ---[ end Kernel panic - not syncing: assertion "i && sym_get_cam_status(cp->cmd) == DID_SOFT_ERROR" failed: file "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work-shared/qemuarm/kernel-source/drivers/scsi/sym53c8xx_2/sym_hipd.c", line 3399 ERROR: Function failed: core-image-sato-sdk - FAILED to start qemu - check the task log and the boot log 52655 2 richard.purdie 2015-07-28 11:04:41 +0000 Other builds at the same time with the same kernel worked so its an intermittent issue. 52664 3 bruce.ashfield 2015-07-28 13:09:36 +0000 (In reply to comment #2) > Other builds at the same time with the same kernel worked so its an > intermittent issue. Looks like fun. I'll start from the backtrace and see if anything shows up. There is another -stable update to 4.1, so I'll apply it as well. One additional question though: has this ever been seen on subsequent runs with the same qemu ? (it's probably not something that gets done in our QA tests, but I was just wondering, since that will help isolate this to a race, versus compiler, versus something else issue). 52690 4 juro.bystricky 2015-07-29 00:00:58 +0000 seems like others ran into a similar problem: https://www.mail-archive.com/nix-commits@lists.science.uu.nl/msg20945.html 52727 5 juro.bystricky 2015-07-30 15:11:52 +0000 This is not really a segfault but a kernel panic due to an assertion: ---[ end Kernel panic - not syncing: assertion "i && sym_get_cam_status(cp->cmd) == DID_SOFT_ERROR" failed: file "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work-shared/qemuarm/kernel-source/drivers/scsi/sym53c8xx_2/sym_hipd.c", line 3399 52793 6 sjolley.yp.pm 2015-08-03 17:47:46 +0000 It appears Juro has answered the NEEDINFO. If more information is needed please move the bug back to NEEDINFO. 53562 7 jku 2015-08-31 10:51:20 +0000 This seems to continue to be an issue: https://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/458/steps/Running%20Sanity%20Tests/logs/stdio 53935 8 juro.bystricky 2015-09-10 14:32:38 +0000 *** Bug 8269 has been marked as a duplicate of this bug. *** 53966 9 elizabeth.flanagan 2015-09-11 11:23:32 +0000 The common denominator here seems to be that it is failing on opensuse131 workers. 54785 10 anibal.limon 2015-09-29 14:34:10 +0000 Searching i found many discussions that talks about problems on SCSI sym driver [1][2]. We use virtio drivers into qemuarm64 and due to SCSI issues, i'm starting to enable virtio blk device into runqemu also we need to enable kernel config. [1] https://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00093.html [2] https://lists.nongnu.org/archive/html/qemu-devel/2010-07/msg01473.html 54800 11 anibal.limon 2015-09-29 23:00:29 +0000 http://lists.openembedded.org/pipermail/openembedded-core/2015-September/111133.html 54881 12 anibal.limon 2015-10-01 14:29:06 +0000 Now merge, http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ec1146e3e47bc009fef295ffcb1a48a434dd2c7b