Bug 14595

Summary:	worker processes exit unexpectedly
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Thomas Perrot <thomas.perrot>
Component:	core	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	alexandre.belloni, meta.mr.watcher, meta.watcher, randy.macleod
Version:	unspecified
Target Milestone:	4.0 M1
Hardware:	x86
OS:	Multiple
Whiteboard:	AB-INT
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Thomas Perrot 2021-10-11 13:20:48 UTC

https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/2167/steps/12/logs/stdio

qemux86-world-alt centos7-ty-4

ERROR: Worker process (29925) exited unexpectedly (1), shutting down...
ERROR: Worker process (29925) exited unexpectedly (1), shutting down...
ERROR: Worker process (29925) exited unexpectedly (1), shutting down...
ERROR: Worker process (29925) exited unexpectedly (1), shutting down...
NOTE: Sending SIGTERM to remaining 6 tasks
NOTE: Sending SIGTERM to remaining 6 tasks
NOTE: Sending SIGTERM to remaining 6 tasks
NOTE: Sending SIGTERM to remaining 6 tasks
NOTE: Tasks Summary: Attempted 14296 tasks of which 2007 didn't need to be rerun and all succeeded.

Comment 1 Randy MacLeod 2021-10-14 14:37:23 UTC

Need to add more logging so that we can debug if it happens.

Comment 2 Alexandre Belloni 2021-10-14 14:37:59 UTC

From Richard:

https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/2167

failed and logged in the grab the cookerdaemon log whilst they were still there.
In the log there was the traceback:

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 513, in <module>
    worker.serve()
  File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 402, in serve
    self.handle_item(b"newtaskhashes", self.handle_newtaskhashes)
  File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 420, in handle_item
    func(self.queue[(len(item) + 2):index])
  File "/home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/bin/bitbake-worker", line 444, in handle_newtaskhashes
    self.workerdata["newhashes"] = pickle.loads(data)
_pickle.UnpicklingError: invalid load key, '\x00'.
invalid load key, '\x00'./home/pokybuild/yocto-worker/qemux86-world-alt/build/bitbake/lib/bb/runqueue.py:1329: ResourceWarning: unclosed file <_io.BufferedWriter name=131>
  self.worker = {}

Not sure why that could happen but it is helpful to have the data and we could
maybe improve that exception handling to dump the invalid data so the next time
it happens we get further info.

Comment 3 Richard Purdie 2021-10-17 12:38:04 UTC

Debug added in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b8be48c718a40bf0abafa9aa15a33dfc6ce9937c which is all we can do until it fails again.

Comment 4 Richard Purdie 2021-10-17 12:38:11 UTC

See above