| Summary: | bitbake worker stuck using 100% cpu on aborted build | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Jacob Kroon <jacob.kroon> |
| Component: | bitbake | Assignee: | Unassigned <unassigned> |
| Status: | RESOLVED WORKSFORME | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | poky.bs.watcher, poky.watcher, randy.macleod |
| Version: | unspecified | ||
| Target Milestone: | 4.2 M3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
|
Description
Jacob Kroon
2020-03-24 19:20:32 UTC
I got this bug on zeus aswell:
bitbake 6836184ef5220488a1127413c7d2e523fc37e2e9
oe-core c940e4b858d6be28b198770768117ecc098fa0d3
Same results, 100% cpu usage in a Worker process, right after bitbake decides to shutdown tasks due to insufficient disk space (again, I first tried to abort the build by pressing ctrl-c):
NOTE: Task virtual:native:/home/jkroon/Projects/foo-linux/zeus/build/../openembedded-core/meta/recipes-support/shared-mime-info/shared-mime-info_1.10.bb:do_populate_sysroot unihash changed to e3e8d84d1949cf246ee6c76de7631fb27ef4bf746f687ea7386767d423c2013d
NOTE: Setscene tasks completed
Keyboard Interrupt, closing down...
WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.050GB left)
ERROR: Immediately abort since the disk space monitor action is "ABORT"!
NOTE: Sending SIGTERM to remaining 1 tasks
gdb debugging reveals:
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f46a75ce6c0 (LWP 36370) "Worker" 0x00007f46a7b5cf16 in waitpid () from /lib64/libc.so.6
2 Thread 0x7f4699498700 (LWP 36371) "python3" 0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
(gdb) py-bt
Traceback (most recent call first):
<built-in method waitpid of module object at remote 0x7f469a54e230>
File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 455, in process_waitpid
pid, status = os.waitpid(-1, os.WNOHANG)
File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 394, in serve
while self.process_waitpid():
File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 495, in <module>
worker.serve()
(gdb) t 2
[Switching to thread 2 (Thread 0x7f4699498700 (LWP 36371))]
#0 0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib64/python3.7/threading.py", line 300, in wait
gotit = waiter.acquire(True, timeout)
File "/usr/lib64/python3.7/queue.py", line 179, in get
self.not_empty.wait(remaining)
File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 96, in worker_flush
worker_queue_int = worker_queue_int + worker_queue.get(True, 1)
File "/usr/lib64/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
Jacob, For this ramdisk
/tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
the default monitor settings are not appropriate.
Are you aware that you can change the disk space monitoring thresholds?
See:
https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-bb-BB_DISKMON_DIRS
Alternatively just allocate a few more GB of RAM to the ramdisk! ;-)
The 100% CPU usage is still a bug.
(In reply to comment #2) > Jacob, For this ramdisk > /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left) > the default monitor settings are not appropriate. > Are you aware that you can change the disk space monitoring thresholds? > See: > > https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var- > bb-BB_DISKMON_DIRS > Alternatively just allocate a few more GB of RAM to the ramdisk! ;-) > > The 100% CPU usage is still a bug. Oh yeah, I had forgotten about tuning BB_DISKMON_DIRS, thanks for the reminder. The bug isn't *that* annoying to me; I can only afford dedicating 16GB to the ramdisk, and the problem only pops up when qemu-system-native/icedtea7-native/glibc gets scheduled to build in parallel. Jacob, You could try using the PSI /proc/pressure regulation, see: https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-BB_NUMBER_THREADS and https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU There's still some improvements that I need to make to the averaging but it does help on the Yocto AB and Wind River's build systems. Randy, I don't know the details of how the PSI monitoring works in OE, but I doubt it would help the situation I was in; the compile task for icedtea7-native would by itself consume several gigabytes, I doubt there is anything to gain with PSI monitoring here ? What has helped, I think, and which is the reason why I haven't had to abort builds in recent time, is that rm_work class got fixed to cleanup even more leftovers: https://git.openembedded.org/openembedded-core/commit/?id=b25cc45c9b39f79ba0a03c4556cb2e2431677b4e I therefore haven't seen the stuck worker for a long time, so I can't say whether the bug is still relevant in master. Since I haven't seen any activity from other users on this bug, I am fine with closing it if you'd prefer that ? There is a commit to fix the rm_work flow that Jacob linked in a previous comment. Jacob, PSI might help buy ensuring that less work gets started when the system is already under CPU pressure. We also want to work on implementing a per bitbake build job server instance that would also help in levelling out the CPU demand. Anyway, this this is working for you, I've closed the defect. |