14595 2021-10-11 13:20:48 +0000 worker processes exit unexpectedly 2021-10-17 12:38:11 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core unspecified x86 Multiple RESOLVED FIXED AB-INT Medium+ normal 4.0 M1 1 thomas.perrot richard.purdie alexandre.belloni meta.mr.watcher meta.watcher randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 91709 0 thomas.perrot 2021-10-11 13:20:48 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/2167/steps/12/logs/stdio qemux86-world-alt centos7-ty-4 ERROR: Worker process (29925) exited unexpectedly (1), shutting down... ERROR: Worker process (29925) exited unexpectedly (1), shutting down... ERROR: Worker process (29925) exited unexpectedly (1), shutting down... ERROR: Worker process (29925) exited unexpectedly (1), shutting down... NOTE: Sending SIGTERM to remaining 6 tasks NOTE: Sending SIGTERM to remaining 6 tasks NOTE: Sending SIGTERM to remaining 6 tasks NOTE: Sending SIGTERM to remaining 6 tasks NOTE: Tasks Summary: Attempted 14296 tasks of which 2007 didn't need to be rerun and all succeeded. 91747 1 randy.macleod 2021-10-14 14:37:23 +0000 Need to add more logging so that we can debug if it happens. 91748 2 alexandre.belloni 2021-10-14 14:37:59 +0000 From Richard: https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/2167 failed and logged in the grab the cookerdaemon log whilst they were still there. In the log there was the traceback: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 513, in <module> worker.serve() File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 402, in serve self.handle_item(b"newtaskhashes", self.handle_newtaskhashes) File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 420, in handle_item func(self.queue[(len(item) + 2):index]) File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 444, in handle_newtaskhashes self.workerdata["newhashes"] = pickle.loads(data) _pickle.UnpicklingError: invalid load key, '\x00'. invalid load key, '\x00'./home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/lib/bb/runqueue.py:1329: ResourceWarning: unclosed file <_io.BufferedWriter name=131> self.worker = {} Not sure why that could happen but it is helpful to have the data and we could maybe improve that exception handling to dump the invalid data so the next time it happens we get further info. 91771 3 richard.purdie 2021-10-17 12:38:04 +0000 Debug added in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b8be48c718a40bf0abafa9aa15a33dfc6ce9937c which is all we can do until it fails again. 91772 4 richard.purdie 2021-10-17 12:38:11 +0000 See above