Bug 14037

Summary: qemuarm-alt backtrace during boot
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: kernelAssignee: Bruce Ashfield <bruce.ashfield>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium CC: randy.macleod, tom.zanussi
Version: 3.2   
Target Milestone: 3.3 M1   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2020-09-02 09:06:05 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/1277

[  267.450916] rcu: INFO: rcu_preempt self-detected stall on CPU
[  267.452293] rcu: 	0-...!: (1 ticks this GP) idle=b86/1/0x40000002 softirq=11528/11528 fqs=0 
[  267.452718] 	(t=3238 jiffies g=26517 q=1)
[  267.452935] rcu: rcu_preempt kthread starved for 3238 jiffies! g26517 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[  267.453193] rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[  267.453409] rcu: RCU grace-period kthread stack dump:
[  267.453730] rcu_preempt     I    0    10      2 0x00000000
[  267.454850] [<c0a3612c>] (__schedule) from [<c0a362b0>] (schedule+0xd0/0x128)
[  267.455103] [<c0a362b0>] (schedule) from [<c0a386b4>] (schedule_timeout+0xc8/0xf4)
[  267.455295] [<c0a386b4>] (schedule_timeout) from [<c0288914>] (rcu_gp_kthread+0x2b0/0x92c)
[  267.455512] [<c0288914>] (rcu_gp_kthread) from [<c024fb58>] (kthread+0x158/0x164)
[  267.455706] [<c024fb58>] (kthread) from [<c0200138>] (ret_from_fork+0x14/0x3c)
[  267.455945] Exception stack(0xdb50dfb0 to 0xdb50dff8)
[  267.456174] dfa0:                                     00000000 00000000 00000000 00000000
[  267.456375] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  267.456554] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  267.457139] NMI backtrace for cpu 0
[  267.457493] CPU: 0 PID: 272 Comm: xkbcomp Not tainted 5.8.4-yocto-standard #1
[  267.457688] Hardware name: Generic DT based system
[  267.457880] [<c0210470>] (unwind_backtrace) from [<c020aeac>] (show_stack+0x18/0x1c)
[  267.458081] [<c020aeac>] (show_stack) from [<c06ad264>] (dump_stack+0x88/0xa4)
[  267.458252] [<c06ad264>] (dump_stack) from [<c06b4518>] (nmi_cpu_backtrace+0xa0/0xb0)
[  267.458450] [<c06b4518>] (nmi_cpu_backtrace) from [<c06b45e4>] (nmi_trigger_cpumask_backtrace+0xbc/0x15c)
[  267.458667] [<c06b45e4>] (nmi_trigger_cpumask_backtrace) from [<c028b458>] (rcu_dump_cpu_stacks+0xe8/0x114)
[  267.458947] [<c028b458>] (rcu_dump_cpu_stacks) from [<c028a554>] (rcu_sched_clock_irq+0x360/0x9c4)
[  267.459152] [<c028a554>] (rcu_sched_clock_irq) from [<c02929e4>] (update_process_times+0x34/0x84)
[  267.459352] [<c02929e4>] (update_process_times) from [<c02a3700>] (tick_sched_timer+0x17c/0x1fc)
[  267.459548] [<c02a3700>] (tick_sched_timer) from [<c02937fc>] (__hrtimer_run_queues+0x180/0x2dc)
[  267.459764] [<c02937fc>] (__hrtimer_run_queues) from [<c029400c>] (hrtimer_interrupt+0xc8/0x208)
[  267.459965] [<c029400c>] (hrtimer_interrupt) from [<c088649c>] (arch_timer_handler_virt+0x30/0x38)
[  267.460182] [<c088649c>] (arch_timer_handler_virt) from [<c027d9a4>] (handle_percpu_devid_irq+0x60/0x118)
[  267.460400] [<c027d9a4>] (handle_percpu_devid_irq) from [<c0277d28>] (generic_handle_irq+0x30/0x40)
[  267.460609] [<c0277d28>] (generic_handle_irq) from [<c0278350>] (__handle_domain_irq+0x98/0xa8)
[  267.460822] [<c0278350>] (__handle_domain_irq) from [<c06c3744>] (gic_handle_irq+0x5c/0x94)
[  267.461007] [<c06c3744>] (gic_handle_irq) from [<c0201584>] (__irq_usr+0x44/0x60)
[  267.461165] Exception stack(0xd4719fb0 to 0xd4719ff8)
[  267.461294] 9fa0:                                     00000000 0046743c 0001fbc1 00000029
[  267.461475] 9fc0: 00467420 00000000 00000004 004673b0 00000000 00450c98 00000007 ffffff4d
[  267.461653] 9fe0: 00462ebc bede2548 00443f65 00443f84 20000030 ffffffff
[  269.047354] systemd-xdg-autostart-generator[278]: Not generating service for XDG autostart app-gsettings\x2ddata\x2dconvert-autostart.service, startup phases are not supported.
[  269.050261] systemd-xdg-autostart-generator[278]: Not generating service for XDG autostart app-at\x2dspi\x2ddbus\x2dbus-autostart.service, startup phases are not supported.
[  269.051687] systemd-xdg-autostart-generator[278]: Not generating service for XDG autostart app-pulseaudio-autostart.service, startup phases are not supported.
7
<<< run_serial(): command timed out after 60 seconds without output >>>
Comment 1 Bruce Ashfield 2020-09-03 07:59:52 UTC 
This and all of the AB-int boot failures all seem to point to qemu not getting enough resources and starving in one way or another.

Do we have a way to dump the machine state at the time of failure ? Or do we have a way to pump the priority of the qemu process(es) ?
Comment 2 Randy MacLeod 2020-09-03 15:01:03 UTC 
Sakib and I are working on that now, see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14006
If you have any suggestion about what data to collect, we're all ears.
Comment 3 Randy MacLeod 2020-11-19 15:48:03 UTC 
Lowered to medium, raise to M+ if seen again.
Comment 4 Randy MacLeod 2020-12-10 16:04:48 UTC 
Not seen in a while. Reopen if you see this issue.