Bug 13988

Summary: Intermittent kernel apic boot panic on qemux86-64
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: kernelAssignee: Bruce Ashfield <bruce.ashfield>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium CC: anuj.mittal, randy.macleod, sakib.sajal, tim.orling, tom.zanussi
Version: 3.2   
Target Milestone: 3.3 M1   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Boot log from failed qemu none

Description Richard Purdie 2020-07-20 08:36:19 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/2194/steps/8/logs/step3d

runqemu - INFO - Network configuration: ip=192.168.7.16::192.168.7.15:255.255.255.0
runqemu - INFO - Running /home/pokybuild/yocto-worker/qemux86-64/build/build-st-3389646/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin/qemu-system-x86_64 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:10 -netdev tap,id=net0,ifname=tap7,script=no,downscript=no -drive file=/home/pokybuild/yocto-worker/qemux86-64/build/build-st-3389646/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64-20200720124813.rootfs.ext4,if=virtio,format=raw -usb -device usb-tablet -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0  -nographic  -cpu core2duo -m 256 -serial tcp:127.0.0.1:51487 -serial tcp:127.0.0.1:50261  -pidfile /home/pokybuild/yocto-worker/qemux86-64/build/build-st-3389646/pidfile_3389646 -kernel /home/pokybuild/yocto-worker/qemux86-64/build/build-st-3389646/tmp/deploy/images/qemux86-64/bzImage--5.4.51+git0+caafbdfe38_fed60f1c8e-r0-qemux86-64-20200720110023.bin -append 'root=/dev/vda rw  console=ttyS0 mem=256M ip=192.168.7.16::192.168.7.15:255.255.255.0 oprofile.timer=1 console=tty1 console=ttyS0,115200n8 printk.time=1'
QEMU 5.0.0 monitor - type 'help' for more information
(qemu) 
DEBUG: Waiting at most 1000 seconds for login banner (07/20/20 12:51:54)
DEBUG: Connection from 127.0.0.1:39968
WARNING: Target didn't reach login banner in 1000 seconds (07/20/20 13:08:35)
WARNING: Last 25 lines of text:
[    2.211277] APIC: Switch to symmetric I/O mode setup
[    2.221512] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    2.238273] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[    2.238999] ...trying to set up timer (IRQ0) through the 8259A ...
[    2.239280] ..... (found apic 0 pin 2) ...
[    2.255186] ....... failed.
[    2.255777] ...trying to set up timer as Virtual Wire IRQ...
[    2.272077] ..... failed.
[    2.272410] ...trying to set up timer as ExtINT IRQ...
[    4.252956] ..... failed :(.
[    4.253728] Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.
[    4.254666] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.51-yocto-standard #1
[    4.255351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[    4.256004] Call Trace:
[    4.257707]  dump_stack+0x57/0x7a
[    4.258012]  panic+0x102/0x2b1
[    4.258226]  setup_IO_APIC+0x802/0x841
[    4.258430]  ? clear_IO_APIC+0x39/0x60
[    4.258663]  apic_intr_mode_init+0xf5/0xf7
[    4.258876]  x86_late_time_init+0x24/0x2b
[    4.259080]  start_kernel+0x41f/0x4cf
[    4.259282]  x86_64_start_reservations+0x24/0x26
[    4.259505]  x86_64_start_kernel+0x70/0x74
[    4.259736]  secondary_startup_64+0xa4/0xb0
[    4.261308] ---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option. ]---
WARNING: Check full boot log: /home/pokybuild/yocto-worker/qemux86-64/build/build-st-3389646/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200720125152
WARNING: Qemu ended unexpectedly, dump data from host is in /home/pokybuild/yocto-worker/qemux86-64/build/build-st-3389646/tmp/log/runtime-hostdump/202007201308_qemu
DEBUG: Stopping logging thread
DEBUG: Stop event received
DEBUG: Tearing down logging thread
DEBUG: Sending SIGTERM to runqemu
DEBUG: Sending SIGTERM to runqemu
Comment 1 Richard Purdie 2020-07-20 08:36:53 UTC 
Bruce: FWIW I just merged the recent kernel changes as those had a green build previously
Comment 2 Richard Purdie 2020-07-20 08:43:03 UTC 
Created attachment 4704 [details]
Boot log from failed qemu
Comment 3 Richard Purdie 2020-07-20 08:44:18 UTC 
Build saved as centos8-ty-2:/home/pokybuild/saved-build-20200720 in case we need it for debugging
Comment 4 Bruce Ashfield 2020-07-20 11:41:27 UTC 
Did this only start popping up after the v5.4.51 merge ?

There's only one kernel commit that even looks remotely close:

Author: Marc Zyngier <maz@kernel.org>
Date:   Sun Jun 21 14:43:15 2020 +0100

    irqchip/gic: Atomically update affinity

    commit 005c34ae4b44f085120d7f371121ec7ded677761 upstream.

    The GIC driver uses a RMW sequence to update the affinity, and
    relies on the gic_lock_irqsave/gic_unlock_irqrestore sequences
    to update it atomically.

    But these sequences only expand into anything meaningful if
    the BL_SWITCHER option is selected, which almost never happens.

    It also turns out that using a RMW and locks is just as silly,
    as the GIC distributor supports byte accesses for the GICD_TARGETRn
    registers, which when used make the update atomic by definition.

    Drop the terminally broken code and replace it by a byte write.

    Fixes: 04c8b0f82c7d ("irqchip/gic: Make locking a BL_SWITCHER only feature")
    Cc: stable@vger.kernel.org
    Signed-off-by: Marc Zyngier <maz@kernel.org>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

I'm double checking the config changes as well, but I don't recall anything relevant.
Comment 5 Richard Purdie 2020-07-21 01:44:59 UTC 
(In reply to comment #4)
> Did this only start popping up after the v5.4.51 merge ?

Yes, although whether its coincidence or not, unknown...
Comment 6 Randy MacLeod 2020-07-23 07:39:56 UTC 
Assigned to Bruce, Sakib please try to reproduce when you have time.
Comment 7 Randy MacLeod 2020-11-19 15:47:47 UTC 
Lowered to medium, raise to M+ if seen again.
Comment 8 Randy MacLeod 2020-12-10 16:03:11 UTC 
This bug hasn't been seen for months. Reopen if it's reproduced.
IO priority adjustments may be preventing us from seeing this.