Seen in https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1838/steps/15/logs/stdio Different from the other intermittent qemu failure but probably shares the same root cause. the full stacktrace is (with some userspace traces to show the delay): [ 4.750041] Run /sbin/init as init process [ 5.022537] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ^MINIT: version 2.98 booting^M FBIOPUT_VSCREENINFO failed, double buffering disabledStarting udev [ 5.370751] udevd[122]: starting version 3.2.9 [ 5.383324] udevd[123]: starting eudev-3.2.9 [ 6.490484] EXT4-fs (vda): re-mounted. Opts: (null) [ 6.491969] ext4 filesystem being remounted at / supports timestamps until 2038 (0x7fffffff) [ 28.661921] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 28.663214] (detected by 0, t=21093 jiffies, g=-75, q=10) [ 28.664193] rcu: All QSes seen, last rcu_preempt kthread activity 21093 (4294695063-4294673970), jiffies_till_next_fqs=3, root ->qsmask 0x0 [ 28.666351] rcu: rcu_preempt kthread starved for 21093 jiffies! g-75 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 28.668089] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [ 28.669679] rcu: RCU grace-period kthread stack dump: [ 28.670556] task:rcu_preempt state:R running task stack: 0 pid: 13 ppid: 2 flags:0x00004000 [ 28.672237] Call Trace: [ 28.672730] __schedule+0x1dd/0x570 [ 28.673322] ? __mod_timer+0x277/0x400 [ 28.673957] schedule+0x68/0xe0 [ 28.674470] schedule_timeout+0x8f/0x160 [ 28.675222] ? __next_timer_interrupt+0x110/0x110 [ 28.676030] rcu_gp_kthread+0x55f/0xba0 [ 28.676736] ? note_gp_changes+0x70/0x70 [ 28.677381] kthread+0x145/0x170 [ 28.677931] ? kthread_create_worker_on_cpu+0x70/0x70 [ 28.678791] ret_from_fork+0x22/0x30 [ 28.679524] BUG: scheduling while atomic: rc/172/0x00000002 [ 28.680594] Modules linked in: [ 28.681146] CPU: 0 PID: 172 Comm: rc Not tainted 5.10.16-yocto-standard #1 [ 28.682008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 28.683532] Call Trace: [ 28.683846] dump_stack+0x5e/0x74 [ 28.684291] __schedule_bug.cold+0x47/0x58 [ 28.684812] __schedule+0x3f7/0x570 [ 28.685285] schedule+0x68/0xe0 [ 28.685678] exit_to_user_mode_loop+0x59/0xc0 [ 28.686246] irqentry_exit_to_user_mode+0x59/0x80 [ 28.686833] irqentry_exit+0x3c/0x60 [ 28.687308] exc_page_fault+0x76/0x150 [ 28.687778] ? asm_exc_page_fault+0x8/0x30 [ 28.688318] asm_exc_page_fault+0x1e/0x30 [ 28.688831] RIP: 0033:0x55a26fff0804 [ 28.689310] Code: 4d 63 e2 4f 8d 34 27 41 83 c2 01 4d 63 c1 0f b6 0e 4b 8d 44 05 00 49 63 ea 4c 01 fd 83 3d 3f ee 0c 00 01 48 89 44 24 08 7f 54 <41> 88 0e 4b 8d 74 05 00 44 89 cb 31 d2 0f b6 06 84 c0 0f 85 24 ff [ 28.691675] RSP: 002b:00007ffe47549870 EFLAGS: 00010246 [ 28.692342] RAX: 000055a27010ee94 RBX: 0000000000000013 RCX: 000000000000003b [ 28.693300] RDX: 0000000000000000 RSI: 000055a27010ee93 RDI: 0000000000000000 [ 28.694218] RBP: 000055a27010f001 R08: 0000000000000014 R09: 0000000000000014 [ 28.695702] R10: 0000000000000011 R11: 0000000000000000 R12: 0000000000000010 [ 28.697197] R13: 000055a27010ee80 R14: 000055a27010f000 R15: 000055a27010eff0
*** Bug 14291 has been marked as a duplicate of this bug. ***
https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3186/steps/13/logs/stdio The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_testimage(d) 0003: File: '/home/pokybuild/yocto-worker/qemuarm64/build/meta/classes/testimage.bbclass', lineno: 131, function: do_testimage 0127: netstat -an 0128:} 0129: 0130:python do_testimage() { *** 0131: testimage_main(d) 0132:} 0133: 0134:addtask testimage 0135:do_testimage[nostamp] = "1" File: '/home/pokybuild/yocto-worker/qemuarm64/build/meta/classes/testimage.bbclass', lineno: 375, function: testimage_main 0371: orig_sigterm_handler = signal.signal(signal.SIGTERM, sigterm_exception) 0372: try: 0373: # We need to check if runqemu ends unexpectedly 0374: # or if the worker send us a SIGTERM *** 0375: tc.target.start(params=d.getVar("TEST_QEMUPARAMS"), runqemuparams=d.getVar("TEST_RUNQEMUPARAMS")) 0376: import threading 0377: try: 0378: threading.Timer(int(d.getVar("TEST_OVERALL_TIMEOUT")), handle_test_timeout, (int(d.getVar("TEST_OVERALL_TIMEOUT")),)).start() 0379: except ValueError: File: '/home/pokybuild/yocto-worker/qemuarm64/build/meta/lib/oeqa/core/target/qemu.py', lineno: 71, function: start 0067: if self.runner.server_ip: 0068: self.server_ip = self.runner.server_ip 0069: else: 0070: self.stop() *** 0071: raise RuntimeError("FAILED to start qemu - check the task log and the boot log") 0072: 0073: def stop(self): 0074: self.runner.stop() Exception: RuntimeError: FAILED to start qemu - check the task log and the boot log
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/3330 qemumips master on debian8-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3402/steps/19/logs/stdio qemuarm master on ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3408/steps/13/logs/stdio ubuntu1804-ty-3 qemuarm64
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2123/steps/14/logs/stdio ubuntu1804-ty-1 happen during selftest
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2120/steps/14/logs/stdio ubuntu1804-ty-3 oe-selftest-ubuntu
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3441/steps/13/logs/stdio ubuntu1804-ty-2 qemuarm
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2149/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3454/steps/13/logs/stdio qemuarm ubuntu1804-ty-1 Was locked up prior to being canceled
More instances: https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3453/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/3463/steps/13/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2118/steps/12/logs/stdio qemux86-64-ptest ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2160/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3493/steps/13/logs/stdio qemuarm fedora31-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1465/steps/14/logs/stdio oe-selftest ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/3471/steps/12/logs/stdio no-x11 ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/3543/steps/18/logs/stdio multilib ubuntu1804-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2208/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/3558/steps/12/logs/stdio qemuarm-oecore ubuntu1804-ty-3 This log has the top output
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3541/steps/13/logs/stdio qemux86-64 ubuntu1804-ty-1 Note that this log would indicate that bug 14399, bug 14352 and bug 14002 are related.
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3540/steps/13/logs/stdio qemux86-64 ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3574/steps/13/logs/stdio qemuarm ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2229/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3555/steps/19/logs/stdio qemux86 ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/3576/steps/12/logs/stdio musl-qemux86-64 ubuntu1804-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2242/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1488/steps/14/logs/stdio oe-selftest ubuntu1804-ty-1
Stalls should no longer cause the hang: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1f802cdf857475eefb9f3ae5032a6f1ce3e01d97