Using bitbake master (4780df48d5998d619dc36b699400e344187fc134), while building I first pressed ctrl-C to stop and finish any started tasks, but then I ran out of disk space in my tmpfs TMPDIR, so bitbake sent a SIGTERM to the remaining task, but after that the worker got stuck using 100% CPU. > NOTE: Executing Tasks > > Keyboard Interrupt, closing down... > > WARNING: The free space of /tmp/ramdisk/foo-master-glibc (ramdisk) is running low (0.996GB left) > ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"! > WARNING: The free space of /tmp/ramdisk/foo-master-glibc (ramdisk) is running low (0.063GB left) > ERROR: Immediately abort since the disk space monitor action is "ABORT"! > NOTE: Sending SIGTERM to remaining 1 tasks > Waiting for 1 running tasks to finish: > 0: icedtea7-native-2.1.3-r1.0 do_compile - 1h0m1s (pid 211548) I attached with gdb and got the follow py-bt for the two threads: > (gdb) info threads > Id Target Id Frame > * 1 Thread 0x7f080cebf6c0 (LWP 34784) "Worker" 0x00007f080d478fab in select () from /lib64/libc.so.6 > 2 Thread 0x7f07fedaa700 (LWP 34785) "python3" 0x00007f080d024b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0 > (gdb) py-bt > Traceback (most recent call first): > <built-in method select of module object at remote 0x7f07ffb79d10> > File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 369, in serve > (ready, _, _) = select.select([self.input] + [i.input for i in self.build_pipes.values()], [] , [], 1) > File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 494, in <module> > worker.serve() > (gdb) t 2 > [Switching to thread 2 (Thread 0x7f07fedaa700 (LWP 34785))] > #0 0x00007f080d024b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0 > (gdb) py-bt > Traceback (most recent call first): > File "/usr/lib64/python3.7/threading.py", line 300, in wait > gotit = waiter.acquire(True, timeout) > File "/usr/lib64/python3.7/queue.py", line 179, in get > self.not_empty.wait(remaining) > File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 95, in worker_flush > worker_queue_int = worker_queue_int + worker_queue.get(True, 1) > File "/usr/lib64/python3.7/threading.py", line 870, in run > self._target(*self._args, **self._kwargs) > File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner > self.run() > File "/usr/lib64/python3.7/threading.py", line 890, in _bootstrap > self._bootstrap_inner()
I got this bug on zeus aswell: bitbake 6836184ef5220488a1127413c7d2e523fc37e2e9 oe-core c940e4b858d6be28b198770768117ecc098fa0d3 Same results, 100% cpu usage in a Worker process, right after bitbake decides to shutdown tasks due to insufficient disk space (again, I first tried to abort the build by pressing ctrl-c): NOTE: Task virtual:native:/home/jkroon/Projects/foo-linux/zeus/build/../openembedded-core/meta/recipes-support/shared-mime-info/shared-mime-info_1.10.bb:do_populate_sysroot unihash changed to e3e8d84d1949cf246ee6c76de7631fb27ef4bf746f687ea7386767d423c2013d NOTE: Setscene tasks completed Keyboard Interrupt, closing down... WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left) ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"! WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.050GB left) ERROR: Immediately abort since the disk space monitor action is "ABORT"! NOTE: Sending SIGTERM to remaining 1 tasks gdb debugging reveals: (gdb) info threads Id Target Id Frame * 1 Thread 0x7f46a75ce6c0 (LWP 36370) "Worker" 0x00007f46a7b5cf16 in waitpid () from /lib64/libc.so.6 2 Thread 0x7f4699498700 (LWP 36371) "python3" 0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0 (gdb) py-bt Traceback (most recent call first): <built-in method waitpid of module object at remote 0x7f469a54e230> File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 455, in process_waitpid pid, status = os.waitpid(-1, os.WNOHANG) File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 394, in serve while self.process_waitpid(): File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 495, in <module> worker.serve() (gdb) t 2 [Switching to thread 2 (Thread 0x7f4699498700 (LWP 36371))] #0 0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0 (gdb) py-bt Traceback (most recent call first): File "/usr/lib64/python3.7/threading.py", line 300, in wait gotit = waiter.acquire(True, timeout) File "/usr/lib64/python3.7/queue.py", line 179, in get self.not_empty.wait(remaining) File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 96, in worker_flush worker_queue_int = worker_queue_int + worker_queue.get(True, 1) File "/usr/lib64/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/lib64/python3.7/threading.py", line 890, in _bootstrap self._bootstrap_inner()
Jacob, For this ramdisk /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left) the default monitor settings are not appropriate. Are you aware that you can change the disk space monitoring thresholds? See: https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-bb-BB_DISKMON_DIRS Alternatively just allocate a few more GB of RAM to the ramdisk! ;-) The 100% CPU usage is still a bug.
(In reply to comment #2) > Jacob, For this ramdisk > /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left) > the default monitor settings are not appropriate. > Are you aware that you can change the disk space monitoring thresholds? > See: > > https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var- > bb-BB_DISKMON_DIRS > Alternatively just allocate a few more GB of RAM to the ramdisk! ;-) > > The 100% CPU usage is still a bug. Oh yeah, I had forgotten about tuning BB_DISKMON_DIRS, thanks for the reminder. The bug isn't *that* annoying to me; I can only afford dedicating 16GB to the ramdisk, and the problem only pops up when qemu-system-native/icedtea7-native/glibc gets scheduled to build in parallel.
Jacob, You could try using the PSI /proc/pressure regulation, see: https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-BB_NUMBER_THREADS and https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU There's still some improvements that I need to make to the averaging but it does help on the Yocto AB and Wind River's build systems.
Randy, I don't know the details of how the PSI monitoring works in OE, but I doubt it would help the situation I was in; the compile task for icedtea7-native would by itself consume several gigabytes, I doubt there is anything to gain with PSI monitoring here ? What has helped, I think, and which is the reason why I haven't had to abort builds in recent time, is that rm_work class got fixed to cleanup even more leftovers: https://git.openembedded.org/openembedded-core/commit/?id=b25cc45c9b39f79ba0a03c4556cb2e2431677b4e I therefore haven't seen the stuck worker for a long time, so I can't say whether the bug is still relevant in master. Since I haven't seen any activity from other users on this bug, I am fine with closing it if you'd prefer that ?
There is a commit to fix the rm_work flow that Jacob linked in a previous comment. Jacob, PSI might help buy ensuring that less work gets started when the system is already under CPU pressure. We also want to work on implementing a per bitbake build job server instance that would also help in levelling out the CPU demand. Anyway, this this is working for you, I've closed the defect.