https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/1277 [ 267.450916] rcu: INFO: rcu_preempt self-detected stall on CPU [ 267.452293] rcu: 0-...!: (1 ticks this GP) idle=b86/1/0x40000002 softirq=11528/11528 fqs=0 [ 267.452718] (t=3238 jiffies g=26517 q=1) [ 267.452935] rcu: rcu_preempt kthread starved for 3238 jiffies! g26517 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0 [ 267.453193] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [ 267.453409] rcu: RCU grace-period kthread stack dump: [ 267.453730] rcu_preempt I 0 10 2 0x00000000 [ 267.454850] [<c0a3612c>] (__schedule) from [<c0a362b0>] (schedule+0xd0/0x128) [ 267.455103] [<c0a362b0>] (schedule) from [<c0a386b4>] (schedule_timeout+0xc8/0xf4) [ 267.455295] [<c0a386b4>] (schedule_timeout) from [<c0288914>] (rcu_gp_kthread+0x2b0/0x92c) [ 267.455512] [<c0288914>] (rcu_gp_kthread) from [<c024fb58>] (kthread+0x158/0x164) [ 267.455706] [<c024fb58>] (kthread) from [<c0200138>] (ret_from_fork+0x14/0x3c) [ 267.455945] Exception stack(0xdb50dfb0 to 0xdb50dff8) [ 267.456174] dfa0: 00000000 00000000 00000000 00000000 [ 267.456375] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 267.456554] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 267.457139] NMI backtrace for cpu 0 [ 267.457493] CPU: 0 PID: 272 Comm: xkbcomp Not tainted 5.8.4-yocto-standard #1 [ 267.457688] Hardware name: Generic DT based system [ 267.457880] [<c0210470>] (unwind_backtrace) from [<c020aeac>] (show_stack+0x18/0x1c) [ 267.458081] [<c020aeac>] (show_stack) from [<c06ad264>] (dump_stack+0x88/0xa4) [ 267.458252] [<c06ad264>] (dump_stack) from [<c06b4518>] (nmi_cpu_backtrace+0xa0/0xb0) [ 267.458450] [<c06b4518>] (nmi_cpu_backtrace) from [<c06b45e4>] (nmi_trigger_cpumask_backtrace+0xbc/0x15c) [ 267.458667] [<c06b45e4>] (nmi_trigger_cpumask_backtrace) from [<c028b458>] (rcu_dump_cpu_stacks+0xe8/0x114) [ 267.458947] [<c028b458>] (rcu_dump_cpu_stacks) from [<c028a554>] (rcu_sched_clock_irq+0x360/0x9c4) [ 267.459152] [<c028a554>] (rcu_sched_clock_irq) from [<c02929e4>] (update_process_times+0x34/0x84) [ 267.459352] [<c02929e4>] (update_process_times) from [<c02a3700>] (tick_sched_timer+0x17c/0x1fc) [ 267.459548] [<c02a3700>] (tick_sched_timer) from [<c02937fc>] (__hrtimer_run_queues+0x180/0x2dc) [ 267.459764] [<c02937fc>] (__hrtimer_run_queues) from [<c029400c>] (hrtimer_interrupt+0xc8/0x208) [ 267.459965] [<c029400c>] (hrtimer_interrupt) from [<c088649c>] (arch_timer_handler_virt+0x30/0x38) [ 267.460182] [<c088649c>] (arch_timer_handler_virt) from [<c027d9a4>] (handle_percpu_devid_irq+0x60/0x118) [ 267.460400] [<c027d9a4>] (handle_percpu_devid_irq) from [<c0277d28>] (generic_handle_irq+0x30/0x40) [ 267.460609] [<c0277d28>] (generic_handle_irq) from [<c0278350>] (__handle_domain_irq+0x98/0xa8) [ 267.460822] [<c0278350>] (__handle_domain_irq) from [<c06c3744>] (gic_handle_irq+0x5c/0x94) [ 267.461007] [<c06c3744>] (gic_handle_irq) from [<c0201584>] (__irq_usr+0x44/0x60) [ 267.461165] Exception stack(0xd4719fb0 to 0xd4719ff8) [ 267.461294] 9fa0: 00000000 0046743c 0001fbc1 00000029 [ 267.461475] 9fc0: 00467420 00000000 00000004 004673b0 00000000 00450c98 00000007 ffffff4d [ 267.461653] 9fe0: 00462ebc bede2548 00443f65 00443f84 20000030 ffffffff [ 269.047354] systemd-xdg-autostart-generator[278]: Not generating service for XDG autostart app-gsettings\x2ddata\x2dconvert-autostart.service, startup phases are not supported. [ 269.050261] systemd-xdg-autostart-generator[278]: Not generating service for XDG autostart app-at\x2dspi\x2ddbus\x2dbus-autostart.service, startup phases are not supported. [ 269.051687] systemd-xdg-autostart-generator[278]: Not generating service for XDG autostart app-pulseaudio-autostart.service, startup phases are not supported. 7 <<< run_serial(): command timed out after 60 seconds without output >>>
This and all of the AB-int boot failures all seem to point to qemu not getting enough resources and starving in one way or another. Do we have a way to dump the machine state at the time of failure ? Or do we have a way to pump the priority of the qemu process(es) ?
Sakib and I are working on that now, see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14006 If you have any suggestion about what data to collect, we're all ears.
Lowered to medium, raise to M+ if seen again.
Not seen in a while. Reopen if you see this issue.