14273 2021-03-02 00:24:04 +0000 qemu CPU starvation triggers rcu_preempt stall detection 2026-07-09 14:53:45 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest 3.3 x86 Multiple RESOLVED FIXED https://bugzilla.yoctoproject.org/show_bug.cgi?id=16349 AB-INT Medium+ normal 3.4 M2 1 alexandre.belloni richard.purdie chee.yang.lee jon.mason kory.maincent randy.macleod richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 89463 0 alexandre.belloni 2021-03-02 00:24:04 +0000 Seen in https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1838/steps/15/logs/stdio Different from the other intermittent qemu failure but probably shares the same root cause. the full stacktrace is (with some userspace traces to show the delay): [ 4.750041] Run /sbin/init as init process [ 5.022537] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ^MINIT: version 2.98 booting^M FBIOPUT_VSCREENINFO failed, double buffering disabledStarting udev [ 5.370751] udevd[122]: starting version 3.2.9 [ 5.383324] udevd[123]: starting eudev-3.2.9 [ 6.490484] EXT4-fs (vda): re-mounted. Opts: (null) [ 6.491969] ext4 filesystem being remounted at / supports timestamps until 2038 (0x7fffffff) [ 28.661921] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 28.663214] (detected by 0, t=21093 jiffies, g=-75, q=10) [ 28.664193] rcu: All QSes seen, last rcu_preempt kthread activity 21093 (4294695063-4294673970), jiffies_till_next_fqs=3, root ->qsmask 0x0 [ 28.666351] rcu: rcu_preempt kthread starved for 21093 jiffies! g-75 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 28.668089] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [ 28.669679] rcu: RCU grace-period kthread stack dump: [ 28.670556] task:rcu_preempt state:R running task stack: 0 pid: 13 ppid: 2 flags:0x00004000 [ 28.672237] Call Trace: [ 28.672730] __schedule+0x1dd/0x570 [ 28.673322] ? __mod_timer+0x277/0x400 [ 28.673957] schedule+0x68/0xe0 [ 28.674470] schedule_timeout+0x8f/0x160 [ 28.675222] ? __next_timer_interrupt+0x110/0x110 [ 28.676030] rcu_gp_kthread+0x55f/0xba0 [ 28.676736] ? note_gp_changes+0x70/0x70 [ 28.677381] kthread+0x145/0x170 [ 28.677931] ? kthread_create_worker_on_cpu+0x70/0x70 [ 28.678791] ret_from_fork+0x22/0x30 [ 28.679524] BUG: scheduling while atomic: rc/172/0x00000002 [ 28.680594] Modules linked in: [ 28.681146] CPU: 0 PID: 172 Comm: rc Not tainted 5.10.16-yocto-standard #1 [ 28.682008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 28.683532] Call Trace: [ 28.683846] dump_stack+0x5e/0x74 [ 28.684291] __schedule_bug.cold+0x47/0x58 [ 28.684812] __schedule+0x3f7/0x570 [ 28.685285] schedule+0x68/0xe0 [ 28.685678] exit_to_user_mode_loop+0x59/0xc0 [ 28.686246] irqentry_exit_to_user_mode+0x59/0x80 [ 28.686833] irqentry_exit+0x3c/0x60 [ 28.687308] exc_page_fault+0x76/0x150 [ 28.687778] ? asm_exc_page_fault+0x8/0x30 [ 28.688318] asm_exc_page_fault+0x1e/0x30 [ 28.688831] RIP: 0033:0x55a26fff0804 [ 28.689310] Code: 4d 63 e2 4f 8d 34 27 41 83 c2 01 4d 63 c1 0f b6 0e 4b 8d 44 05 00 49 63 ea 4c 01 fd 83 3d 3f ee 0c 00 01 48 89 44 24 08 7f 54 <41> 88 0e 4b 8d 74 05 00 44 89 cb 31 d2 0f b6 06 84 c0 0f 85 24 ff [ 28.691675] RSP: 002b:00007ffe47549870 EFLAGS: 00010246 [ 28.692342] RAX: 000055a27010ee94 RBX: 0000000000000013 RCX: 000000000000003b [ 28.693300] RDX: 0000000000000000 RSI: 000055a27010ee93 RDI: 0000000000000000 [ 28.694218] RBP: 000055a27010f001 R08: 0000000000000014 R09: 0000000000000014 [ 28.695702] R10: 0000000000000011 R11: 0000000000000000 R12: 0000000000000010 [ 28.697197] R13: 000055a27010ee80 R14: 000055a27010f000 R15: 000055a27010eff0 89601 1 alexandre.belloni 2021-03-11 19:32:16 +0000 *** Bug 14291 has been marked as a duplicate of this bug. *** 89615 2 kory.maincent 2021-03-12 10:37:56 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3186/steps/13/logs/stdio The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_testimage(d) 0003: File: '/home/pokybuild/yocto-worker/qemuarm64/build/meta/classes/testimage.bbclass', lineno: 131, function: do_testimage 0127: netstat -an 0128:} 0129: 0130:python do_testimage() { *** 0131: testimage_main(d) 0132:} 0133: 0134:addtask testimage 0135:do_testimage[nostamp] = "1" File: '/home/pokybuild/yocto-worker/qemuarm64/build/meta/classes/testimage.bbclass', lineno: 375, function: testimage_main 0371: orig_sigterm_handler = signal.signal(signal.SIGTERM, sigterm_exception) 0372: try: 0373: # We need to check if runqemu ends unexpectedly 0374: # or if the worker send us a SIGTERM *** 0375: tc.target.start(params=d.getVar("TEST_QEMUPARAMS"), runqemuparams=d.getVar("TEST_RUNQEMUPARAMS")) 0376: import threading 0377: try: 0378: threading.Timer(int(d.getVar("TEST_OVERALL_TIMEOUT")), handle_test_timeout, (int(d.getVar("TEST_OVERALL_TIMEOUT")),)).start() 0379: except ValueError: File: '/home/pokybuild/yocto-worker/qemuarm64/build/meta/lib/oeqa/core/target/qemu.py', lineno: 71, function: start 0067: if self.runner.server_ip: 0068: self.server_ip = self.runner.server_ip 0069: else: 0070: self.stop() *** 0071: raise RuntimeError("FAILED to start qemu - check the task log and the boot log") 0072: 0073: def stop(self): 0074: self.runner.stop() Exception: RuntimeError: FAILED to start qemu - check the task log and the boot log 90217 3 alexandre.belloni 2021-05-02 22:24:44 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/3330 qemumips master on debian8-ty-1 90305 4 alexandre.belloni 2021-05-06 16:19:24 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3402/steps/19/logs/stdio qemuarm master on ubuntu1804-ty-3 90332 5 chee.yang.lee 2021-05-10 05:48:28 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3408/steps/13/logs/stdio ubuntu1804-ty-3 qemuarm64 90341 6 chee.yang.lee 2021-05-11 08:41:45 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2123/steps/14/logs/stdio ubuntu1804-ty-1 happen during selftest 90342 7 chee.yang.lee 2021-05-11 09:41:09 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2120/steps/14/logs/stdio ubuntu1804-ty-3 oe-selftest-ubuntu 90356 8 chee.yang.lee 2021-05-13 09:47:52 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3441/steps/13/logs/stdio ubuntu1804-ty-2 qemuarm 90447 9 jon.mason 2021-05-20 20:17:45 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2149/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3 90450 10 jon.mason 2021-05-20 20:45:00 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3454/steps/13/logs/stdio qemuarm ubuntu1804-ty-1 Was locked up prior to being canceled 90459 11 jon.mason 2021-05-21 14:09:18 +0000 More instances: https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3453/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/3463/steps/13/logs/stdio 90488 12 alexandre.belloni 2021-05-27 14:28:25 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2118/steps/12/logs/stdio qemux86-64-ptest ubuntu1804-ty-3 90520 13 alexandre.belloni 2021-05-27 22:49:13 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2160/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3 90528 14 alexandre.belloni 2021-05-28 23:55:23 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3493/steps/13/logs/stdio qemuarm fedora31-ty-1 90641 15 alexandre.belloni 2021-06-16 07:57:04 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1465/steps/14/logs/stdio oe-selftest ubuntu1804-ty-3 90647 16 alexandre.belloni 2021-06-16 08:51:02 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/3471/steps/12/logs/stdio no-x11 ubuntu1804-ty-3 90656 17 alexandre.belloni 2021-06-16 10:56:36 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/3543/steps/18/logs/stdio multilib ubuntu1804-ty-2 90659 18 alexandre.belloni 2021-06-16 14:02:02 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2208/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-2 90661 19 alexandre.belloni 2021-06-16 23:12:01 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/3558/steps/12/logs/stdio qemuarm-oecore ubuntu1804-ty-3 This log has the top output 90685 20 alexandre.belloni 2021-06-17 15:10:01 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3541/steps/13/logs/stdio qemux86-64 ubuntu1804-ty-1 Note that this log would indicate that bug 14399, bug 14352 and bug 14002 are related. 90686 21 alexandre.belloni 2021-06-17 15:15:14 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3540/steps/13/logs/stdio qemux86-64 ubuntu1804-ty-3 90703 22 alexandre.belloni 2021-06-18 09:20:40 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3574/steps/13/logs/stdio qemuarm ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2229/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-2 90714 23 alexandre.belloni 2021-06-20 16:57:33 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3555/steps/19/logs/stdio qemux86 ubuntu1804-ty-3 90732 24 alexandre.belloni 2021-06-24 11:49:33 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/3576/steps/12/logs/stdio musl-qemux86-64 ubuntu1804-ty-1 90737 25 alexandre.belloni 2021-06-24 12:12:25 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2242/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-2 90738 26 alexandre.belloni 2021-06-24 12:19:22 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1488/steps/14/logs/stdio oe-selftest ubuntu1804-ty-1 90770 27 richard.purdie 2021-06-28 09:37:46 +0000 Stalls should no longer cause the hang: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1f802cdf857475eefb9f3ae5032a6f1ce3e01d97