Bug 16349

Summary: AB-INT: boot hang in qemu (rcu_preempt self-detected stall)
Product: [QA/Testing] Runtime Testing Reporter: Yoann Congal <yoann.congal>
Component: testimageAssignee: Yoann Congal <yoann.congal>
Status: NEW --- QA Contact:
Severity: normal    
Priority: Medium CC: randy.macleod
Version: 6.0.1   
Target Milestone: 6.2   
Hardware: x86   
OS: Multiple   
See Also: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14273
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Failing qemu boot log none

Description Yoann Congal 2026-07-04 22:24:46 UTC
Created attachment 5233 [details]
Failing qemu boot log

qemuarm-oecore debian11-vk-3 wrynose completed at 2026-07-04 15:56:04+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/4048

Log extract:
         Starting RPC Bind...
         Starting Rebuild Journal Catalog...
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  *** ] (2 of 7) A start job is running for…amic Linker Cache (24s / no limit)
[   58.372463] rcu: INFO: rcu_preempt self-detected stall on CPU
[   58.378417] rcu: 	2-...!: (1 ticks this GP) idle=4e64/1/0x40000002 softirq=1037/1037 fqs=1
[   58.382355] rcu: 	(t=2416 jiffies g=1049 q=26 ncpus=4)
[   58.388555] CPU: 2 UID: 997 PID: 160 Comm: rpcbind Not tainted 6.18.35-yocto-standard #1 PREEMPT 
[   58.389204] Hardware name: Generic DT based system
[   58.389675] PC is at 0xb6efd584
[   58.390319] LR is at 0xb6da3f27
[   58.390446] pc : [<b6efd584>]    lr : [<b6da3f27>]    psr: 200f0030
[   58.390722] sp : bef2e9a8  ip : 00000127  fp : b6fa5000
[   58.390932] r10: 00000000  r9 : 0000000a  r8 : 00000001
[   58.391166] r7 : bef2e9b4  r6 : b6f60398  r5 : 01000000  r4 : 00800000
[   58.392113] r3 : 00000000  r2 : 00000001  r1 : 00000004  r0 : 00000000
[   58.392380] Flags: nzCv  IRQs on  FIQs on  Mode USER_32  ISA Thumb  Segment none
[  OK  ] Finished Rebuild Journal Catalog.
[  OK  ] Started RPC Bind.
[  OK  ] Started Rule-based Manager for Device Events and Files.
[  OK  ] Finished Coldplug All udev Devices.
         Starting Wait for udev To Complete Device Initialization...
[ TIME ] Timed out waiting for device /dev/hvc0.
[DEPEND] Dependency failed for Serial Getty on hvc0.
[ TIME ] Timed out waiting for device /dev/ttyAMA0.
[DEPEND] Dependency failed for Serial Getty on ttyAMA0.
[***   ] (2 of 2) A start job is running for…ce Initialization (2min / 4min 3s)
[   58.392983] Call trace: 
[  OK  ] Created slice Slice /system/psplash-start.
         Starting Start psplash boot splash screen...
[  *** ] (3 of 3) A start job is running for…Initialization (2min 9s / 4min 3s)
[  161.552693] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  161.564208] rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-3): P150/1:b..l
[  161.565801] rcu: 	(detected by 0, t=2216 jiffies, g=1885, q=75 ncpus=4)
[  161.571221] task:systemd-udevd   state:R  running task     stack:0     pid:150   tgid:150   ppid:1      task_flags:0x400100 flags:0x00800004
[  161.572418] Call trace: 
[  161.572503]  __schedule from preempt_schedule+0x38/0x54
[  161.573386]  preempt_schedule from do_send_sig_info+0x6c/0x94
[  161.574084]  do_send_sig_info from kill_pid_info_type+0x4c/0x74
[  161.574446]  kill_pid_info_type from do_pidfd_send_signal+0x134/0x14c
[  161.575019]  do_pidfd_send_signal from sys_pidfd_send_signal+0x158/0x164
[  161.575506]  sys_pidfd_send_signal from __sys_trace_return+0x0/0x10
[  161.577855] Exception stack(0xe0f7dfa8 to 0xe0f7dff0)
[  161.584960] dfa0:                   0000000a 00000000 0000000f 0000000a be8718e4 00000000
[  161.585722] dfc0: 0000000a 00000000 005aed78 000001a8 005ceb48 00501448 00000000 00000003
[  161.595713] dfe0: b6fd3c5c be8718d4 b6e3c47b b6c398f8
[  161.598117] rcu: rcu_preempt kthread timer wakeup didn't happen for 2220 jiffies! g1885 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[  161.599851] rcu: 	Possible timer handling issue on cpu=0 timer-softirq=673
[  161.600491] rcu: rcu_preempt kthread starved for 2221 jiffies! g1885 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[  161.601407] rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[  161.602070] rcu: RCU grace-period kthread stack dump:
[  161.602577] task:rcu_preempt     state:I stack:0     pid:16    tgid:16    ppid:2      task_flags:0x208040 flags:0x00000000
[  161.607411] Call trace: 
[  161.607514]  __schedule from schedule+0x3c/0x64
[  161.608293]  schedule from schedule_timeout+0xd8/0x108
[  161.608735]  schedule_timeout from rcu_gp_fqs_loop+0x134/0x3c8
[  161.609362]  rcu_gp_fqs_loop from rcu_gp_kthread+0x118/0x144
[  161.609813]  rcu_gp_kthread from kthread+0x194/0x230
[  161.610201]  kthread from ret_from_fork+0x14/0x38
[  161.610564] Exception stack(0xe087dfb0 to 0xe087dff8)
[  161.610937] dfa0:                                     00000000 00000000 00000000 00000000
[  161.613366] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  161.614178] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  161.615406] rcu: Stack dump where RCU GP kthread last ran:
[  161.616082] CPU: 0 UID: 0 PID: 172 Comm: (udev-worker) Not tainted 6.18.35-yocto-standard #1 PREEMPT 
[  161.617602] Hardware name: Generic DT based system
[  161.617980] PC is at _raw_spin_unlock_irq+0x24/0x54
[  161.618393] LR is at _raw_spin_unlock_irq+0x20/0x54
[  161.619072] pc : [<c0cb0a40>]    lr : [<c0cb0a3c>]    psr: 200f0113
[  161.619535] sp : e0801f80  ip : c1814a5c  fp : c155d500
[  161.619981] r10: be871314  r9 : 00000100  r8 : 00000000
[  161.620332] r7 : c128dc40  r6 : 00000001  r5 : c1403080  r4 : c3b73d80
[  161.621308] r3 : 0000000a  r2 : dfb31e80  r1 : 00000008  r0 : dfb31e60
[  161.626030] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  161.627726] Control: 30c5387d  Table: 45046c40  DAC: fffffffd
[  161.627993] Call trace: 
[  161.628038]  _raw_spin_unlock_irq from tasklet_hi_action+0x14/0x30
[  161.629210]  tasklet_hi_action from handle_softirqs+0x1f4/0x2b4
[  161.629475]  handle_softirqs from __irq_exit_rcu+0x60/0xc4
[  161.630080]  __irq_exit_rcu from irq_exit+0x10/0x18
[  161.630675]  irq_exit from call_with_stack+0x18/0x20
[  161.631027]  call_with_stack from __irq_usr+0x80/0xa0
[  161.631381] Exception stack(0xe1101fb0 to 0xe1101ff8)
[  161.631705] 1fa0:                                     be871390 00000000 00000070 be871390
[  161.632222] 1fc0: 00000000 005cf319 005d0df8 00000000 00000000 be871584 be871314 be87153c
[  OK  ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.


Old related bug:
14273 – qemu CPU starvation triggers rcu_preempt stall detection
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14273
Comment 1 Yoann Congal 2026-07-04 22:38:42 UTC
(In reply to Yoann Congal from comment #0)
> qemuarm-oecore debian11-vk-3 wrynose completed at 2026-07-04 15:56:04+00:00
> hxxps://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/4048

That's a wrynose-nut with the kernel 6.18.35 cherry-picked upgrade.
Comment 2 Randy MacLeod 2026-07-09 14:52:58 UTC
Let's see how often this happens.
It's likely load related:
    https://bugzilla.yoctoproject.org/show_bug.cgi?id=14273