Bug 13843 - bitbake worker stuck using 100% cpu on aborted build
Summary: bitbake worker stuck using 100% cpu on aborted build
Status: RESOLVED WORKSFORME
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 4.2 M3
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-03-24 19:20 UTC by Jacob Kroon
Modified: 2023-03-09 20:36 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jacob Kroon 2020-03-24 19:20:32 UTC
Using bitbake master (4780df48d5998d619dc36b699400e344187fc134), while building I first pressed ctrl-C to stop and finish any started tasks, but then I ran out of disk space in my tmpfs TMPDIR, so bitbake sent a SIGTERM to the remaining task, but after that the worker got stuck using 100% CPU.

> NOTE: Executing Tasks
>
> Keyboard Interrupt, closing down...
>
> WARNING: The free space of /tmp/ramdisk/foo-master-glibc (ramdisk) is running low (0.996GB left)
> ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
> WARNING: The free space of /tmp/ramdisk/foo-master-glibc (ramdisk) is running low (0.063GB left)
> ERROR: Immediately abort since the disk space monitor action is "ABORT"!
> NOTE: Sending SIGTERM to remaining 1 tasks
> Waiting for 1 running tasks to finish:
> 0: icedtea7-native-2.1.3-r1.0 do_compile - 1h0m1s (pid 211548)

I attached with gdb and got the follow py-bt for the two threads:

> (gdb) info threads
>   Id   Target Id                                   Frame
> * 1    Thread 0x7f080cebf6c0 (LWP 34784) "Worker"  0x00007f080d478fab in select () from /lib64/libc.so.6
>   2    Thread 0x7f07fedaa700 (LWP 34785) "python3" 0x00007f080d024b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
> (gdb) py-bt
> Traceback (most recent call first):
>   <built-in method select of module object at remote 0x7f07ffb79d10>
>   File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 369, in serve
>     (ready, _, _) = select.select([self.input] + [i.input for i in self.build_pipes.values()], [] , [], 1)
>   File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 494, in <module>
>     worker.serve()
> (gdb) t 2
> [Switching to thread 2 (Thread 0x7f07fedaa700 (LWP 34785))]
> #0  0x00007f080d024b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
> (gdb) py-bt
> Traceback (most recent call first):
>   File "/usr/lib64/python3.7/threading.py", line 300, in wait
>     gotit = waiter.acquire(True, timeout)
>   File "/usr/lib64/python3.7/queue.py", line 179, in get
>     self.not_empty.wait(remaining)
>   File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 95, in worker_flush
>     worker_queue_int = worker_queue_int + worker_queue.get(True, 1)
>   File "/usr/lib64/python3.7/threading.py", line 870, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
>     self.run()
>   File "/usr/lib64/python3.7/threading.py", line 890, in _bootstrap
>     self._bootstrap_inner()
Comment 1 Jacob Kroon 2020-03-30 07:36:34 UTC
I got this bug on zeus aswell:

bitbake 6836184ef5220488a1127413c7d2e523fc37e2e9
oe-core c940e4b858d6be28b198770768117ecc098fa0d3

Same results, 100% cpu usage in a Worker process, right after bitbake decides to shutdown tasks due to insufficient disk space (again, I first tried to abort the build by pressing ctrl-c):

NOTE: Task virtual:native:/home/jkroon/Projects/foo-linux/zeus/build/../openembedded-core/meta/recipes-support/shared-mime-info/shared-mime-info_1.10.bb:do_populate_sysroot unihash changed to e3e8d84d1949cf246ee6c76de7631fb27ef4bf746f687ea7386767d423c2013d
NOTE: Setscene tasks completed

Keyboard Interrupt, closing down...

WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.050GB left)
ERROR: Immediately abort since the disk space monitor action is "ABORT"!
NOTE: Sending SIGTERM to remaining 1 tasks

gdb debugging reveals:

(gdb) info threads 
  Id   Target Id                                   Frame 
* 1    Thread 0x7f46a75ce6c0 (LWP 36370) "Worker"  0x00007f46a7b5cf16 in waitpid () from /lib64/libc.so.6
  2    Thread 0x7f4699498700 (LWP 36371) "python3" 0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
(gdb) py-bt
Traceback (most recent call first):
  <built-in method waitpid of module object at remote 0x7f469a54e230>
  File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 455, in process_waitpid
    pid, status = os.waitpid(-1, os.WNOHANG)
  File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 394, in serve
    while self.process_waitpid():
  File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 495, in <module>
    worker.serve()
(gdb) t 2
[Switching to thread 2 (Thread 0x7f4699498700 (LWP 36371))]
#0  0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib64/python3.7/threading.py", line 300, in wait
    gotit = waiter.acquire(True, timeout)
  File "/usr/lib64/python3.7/queue.py", line 179, in get
    self.not_empty.wait(remaining)
  File "/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker", line 96, in worker_flush
    worker_queue_int = worker_queue_int + worker_queue.get(True, 1)
  File "/usr/lib64/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
Comment 2 Randy MacLeod 2020-03-30 18:40:44 UTC
Jacob, For this ramdisk
    /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
the default monitor settings are not appropriate. 
Are you aware that you can change the disk space monitoring thresholds? 
See:
  https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-bb-BB_DISKMON_DIRS
Alternatively just allocate a few more GB of RAM to the ramdisk! ;-)

The 100% CPU usage is still a bug.
Comment 3 Jacob Kroon 2020-03-30 20:05:52 UTC
(In reply to comment #2)
> Jacob, For this ramdisk
>     /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
> the default monitor settings are not appropriate. 
> Are you aware that you can change the disk space monitoring thresholds? 
> See:
>  
> https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-
> bb-BB_DISKMON_DIRS
> Alternatively just allocate a few more GB of RAM to the ramdisk! ;-)
> 
> The 100% CPU usage is still a bug.

Oh yeah, I had forgotten about tuning BB_DISKMON_DIRS, thanks for the reminder.

The bug isn't *that* annoying to me; I can only afford dedicating 16GB to the ramdisk, and the problem only pops up when qemu-system-native/icedtea7-native/glibc gets scheduled to build in parallel.
Comment 4 Randy MacLeod 2023-02-02 15:08:17 UTC
Jacob,
You could try using the PSI /proc/pressure regulation, see:
https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-BB_NUMBER_THREADS

and 
https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU

There's still some improvements that I need to make to the averaging but it does help on the Yocto AB and Wind River's build systems.
Comment 5 Jacob Kroon 2023-02-03 03:54:46 UTC
Randy,
I don't know the details of how the PSI monitoring works in OE, but I doubt it would help the situation I was in; the compile task for icedtea7-native would by itself consume several gigabytes, I doubt there is anything to gain with PSI monitoring here ?

What has helped, I think, and which is the reason why I haven't had to abort builds in recent time, is that rm_work class got fixed to cleanup even more leftovers:

https://git.openembedded.org/openembedded-core/commit/?id=b25cc45c9b39f79ba0a03c4556cb2e2431677b4e

I therefore haven't seen the stuck worker for a long time, so I can't say whether the bug is still relevant in master. Since I haven't seen any activity from other users on this bug, I am fine with closing it if you'd prefer that ?
Comment 6 Randy MacLeod 2023-03-09 20:36:53 UTC
There is a commit to fix the rm_work flow that Jacob linked in a previous comment.

Jacob,
PSI might help buy ensuring that less work gets started when the system is already under CPU pressure. We also want to work on implementing a per bitbake build job server instance that would also help in levelling out the CPU demand.

Anyway, this this is working for you, I've closed the defect.