AB Collection [1], build [2] ERROR: core-image-sato-1.0-r0 do_testimage: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_testimage(d) 0003: File: '/home/pokybuild/yocto-worker/qemumips-alt/build/meta/classes/testimage.bbclass', lineno: 131, function: do_testimage 0127: netstat -an 0128:} 0129: 0130:python do_testimage() { *** 0131: testimage_main(d) 0132:} 0133: 0134:addtask testimage 0135:do_testimage[nostamp] = "1" File: '/home/pokybuild/yocto-worker/qemumips-alt/build/meta/classes/testimage.bbclass', lineno: 375, function: testimage_main 0371: orig_sigterm_handler = signal.signal(signal.SIGTERM, sigterm_exception) 0372: try: 0373: # We need to check if runqemu ends unexpectedly 0374: # or if the worker send us a SIGTERM *** 0375: tc.target.start(params=d.getVar("TEST_QEMUPARAMS"), runqemuparams=d.getVar("TEST_RUNQEMUPARAMS")) 0376: import threading 0377: try: 0378: threading.Timer(int(d.getVar("TEST_OVERALL_TIMEOUT")), handle_test_timeout, (int(d.getVar("TEST_OVERALL_TIMEOUT")),)).start() 0379: except ValueError: File: '/home/pokybuild/yocto-worker/qemumips-alt/build/meta/lib/oeqa/core/target/qemu.py', lineno: 71, function: start 0067: if self.runner.server_ip: 0068: self.server_ip = self.runner.server_ip 0069: else: 0070: self.stop() *** 0071: raise RuntimeError("FAILED to start qemu - check the task log and the boot log") 0072: 0073: def stop(self): 0074: self.runner.stop() Exception: RuntimeError: FAILED to start qemu - check the task log and the boot log [1] https://swatbot.yoctoproject.org/collection/543/ [2] https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/1410/steps/13/logs/stdio
For previous bugs I remember that qemu speed (without accell) is affected by the I/O load of the host machine, may be related to qemu design.
Happened again here: https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3251/steps/13/logs/stdio
This seems to be the same, but it happened during oe-selftest this time (runtime_test.SystemTap.test_crosstap_pstree): https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1973
Saw it again: https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3278/steps/14/logs/stdio
Again: https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1989
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/3300
(In reply to comment #7) > Happened again: > https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/3300 It is on master branch. Worker: centos7-ty-4 Builder: pkgman-non-rpm
For debugging purpose, adding AB link for old failure: Branch: master-next. Worker: fedora33-ty-1 Builder: qemux86 https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3290
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3362/steps/13/logs/stdio master qemuarm debian10-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3321/steps/13/logs/stdio master-next on debian10-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2035/steps/13/logs/stdio qemux86-64 master on centos7-ty-4
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3367/steps/13/logs/stdio qemux86-64 master on debian10-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/3403/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3415/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3447/steps/14/logs/stdio centos7-ty-4 qemuarm https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3419/steps/13/logs/stdio fedora33-ty-1 qemux86-64
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3433/steps/13/logs/stdio qemux86-64 debian10-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/2108/steps/14/logs/stdio oe-selftest-debian debian10-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3442/steps/13/logs/stdio qemux86 debian10-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/3462 musl-qemux86 ubuntu1604-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3484/steps/13/logs/stdio qemuarm debian10-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1471/steps/15/logs/stdio oe-selftest centos7-ty-4
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3459/steps/13/logs/stdio qemux86-64 debian10-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/1712/steps/13/logs/stdio qemumips-alt debian10-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/3488/steps/14/logs/stdio qemumips64 centos7-ty-4
https://autobuilder.yoctoproject.org/typhoon/#/builders/101/builds/2384/steps/13/logs/stdio qemux86-alt debian10-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/101/builds/2386/steps/12/logs/stdio qemux86-alt debian10-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/3559/steps/13/logs/stdio qemumips64 debian10-ty-2
I forgot to add this one has: runqemu - INFO - Copy done in 302.5292055606842 seconds which would indicate an IO load issue
https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/3581/steps/12/logs/stdio musl-qemux86 ubuntu1804-ty-1
Most likely cause of these is the rcu stall issue, fixed with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1f802cdf857475eefb9f3ae5032a6f1ce3e01d97 We can open new bugs if issues recur. If this does recur, please try and capture the qemu boot log from WORKDIR/testimage/qemu_*.log
bulk change to add Saul's non-WR address.
Bulk change: Remove Saul's old WR address.