<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>13843</bug_id>
          
          <creation_ts>2020-03-24 19:20:32 +0000</creation_ts>
          <short_desc>bitbake worker stuck using 100% cpu on aborted build</short_desc>
          <delta_ts>2023-03-09 20:36:53 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>BitBake</product>
          <component>bitbake</component>
          <version>unspecified</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>WORKSFORME</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>4.2 M3</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Jacob Kroon">jacob.kroon</reporter>
          <assigned_to name="Unassigned">unassigned</assigned_to>
          <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
    
    <cc>randy.macleod</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>Don&apos;t know</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>86798</commentid>
    <comment_count>0</comment_count>
    <who name="Jacob Kroon">jacob.kroon</who>
    <bug_when>2020-03-24 19:20:32 +0000</bug_when>
    <thetext>Using bitbake master (4780df48d5998d619dc36b699400e344187fc134), while building I first pressed ctrl-C to stop and finish any started tasks, but then I ran out of disk space in my tmpfs TMPDIR, so bitbake sent a SIGTERM to the remaining task, but after that the worker got stuck using 100% CPU.

&gt; NOTE: Executing Tasks
&gt;
&gt; Keyboard Interrupt, closing down...
&gt;
&gt; WARNING: The free space of /tmp/ramdisk/foo-master-glibc (ramdisk) is running low (0.996GB left)
&gt; ERROR: No new tasks can be executed since the disk space monitor action is &quot;STOPTASKS&quot;!
&gt; WARNING: The free space of /tmp/ramdisk/foo-master-glibc (ramdisk) is running low (0.063GB left)
&gt; ERROR: Immediately abort since the disk space monitor action is &quot;ABORT&quot;!
&gt; NOTE: Sending SIGTERM to remaining 1 tasks
&gt; Waiting for 1 running tasks to finish:
&gt; 0: icedtea7-native-2.1.3-r1.0 do_compile - 1h0m1s (pid 211548)

I attached with gdb and got the follow py-bt for the two threads:

&gt; (gdb) info threads
&gt;   Id   Target Id                                   Frame
&gt; * 1    Thread 0x7f080cebf6c0 (LWP 34784) &quot;Worker&quot;  0x00007f080d478fab in select () from /lib64/libc.so.6
&gt;   2    Thread 0x7f07fedaa700 (LWP 34785) &quot;python3&quot; 0x00007f080d024b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
&gt; (gdb) py-bt
&gt; Traceback (most recent call first):
&gt;   &lt;built-in method select of module object at remote 0x7f07ffb79d10&gt;
&gt;   File &quot;/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker&quot;, line 369, in serve
&gt;     (ready, _, _) = select.select([self.input] + [i.input for i in self.build_pipes.values()], [] , [], 1)
&gt;   File &quot;/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker&quot;, line 494, in &lt;module&gt;
&gt;     worker.serve()
&gt; (gdb) t 2
&gt; [Switching to thread 2 (Thread 0x7f07fedaa700 (LWP 34785))]
&gt; #0  0x00007f080d024b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
&gt; (gdb) py-bt
&gt; Traceback (most recent call first):
&gt;   File &quot;/usr/lib64/python3.7/threading.py&quot;, line 300, in wait
&gt;     gotit = waiter.acquire(True, timeout)
&gt;   File &quot;/usr/lib64/python3.7/queue.py&quot;, line 179, in get
&gt;     self.not_empty.wait(remaining)
&gt;   File &quot;/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker&quot;, line 95, in worker_flush
&gt;     worker_queue_int = worker_queue_int + worker_queue.get(True, 1)
&gt;   File &quot;/usr/lib64/python3.7/threading.py&quot;, line 870, in run
&gt;     self._target(*self._args, **self._kwargs)
&gt;   File &quot;/usr/lib64/python3.7/threading.py&quot;, line 926, in _bootstrap_inner
&gt;     self.run()
&gt;   File &quot;/usr/lib64/python3.7/threading.py&quot;, line 890, in _bootstrap
&gt;     self._bootstrap_inner()</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>86861</commentid>
    <comment_count>1</comment_count>
    <who name="Jacob Kroon">jacob.kroon</who>
    <bug_when>2020-03-30 07:36:34 +0000</bug_when>
    <thetext>I got this bug on zeus aswell:

bitbake 6836184ef5220488a1127413c7d2e523fc37e2e9
oe-core c940e4b858d6be28b198770768117ecc098fa0d3

Same results, 100% cpu usage in a Worker process, right after bitbake decides to shutdown tasks due to insufficient disk space (again, I first tried to abort the build by pressing ctrl-c):

NOTE: Task virtual:native:/home/jkroon/Projects/foo-linux/zeus/build/../openembedded-core/meta/recipes-support/shared-mime-info/shared-mime-info_1.10.bb:do_populate_sysroot unihash changed to e3e8d84d1949cf246ee6c76de7631fb27ef4bf746f687ea7386767d423c2013d
NOTE: Setscene tasks completed

Keyboard Interrupt, closing down...

WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
ERROR: No new tasks can be executed since the disk space monitor action is &quot;STOPTASKS&quot;!
WARNING: The free space of /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.050GB left)
ERROR: Immediately abort since the disk space monitor action is &quot;ABORT&quot;!
NOTE: Sending SIGTERM to remaining 1 tasks

gdb debugging reveals:

(gdb) info threads 
  Id   Target Id                                   Frame 
* 1    Thread 0x7f46a75ce6c0 (LWP 36370) &quot;Worker&quot;  0x00007f46a7b5cf16 in waitpid () from /lib64/libc.so.6
  2    Thread 0x7f4699498700 (LWP 36371) &quot;python3&quot; 0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
(gdb) py-bt
Traceback (most recent call first):
  &lt;built-in method waitpid of module object at remote 0x7f469a54e230&gt;
  File &quot;/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker&quot;, line 455, in process_waitpid
    pid, status = os.waitpid(-1, os.WNOHANG)
  File &quot;/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker&quot;, line 394, in serve
    while self.process_waitpid():
  File &quot;/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker&quot;, line 495, in &lt;module&gt;
    worker.serve()
(gdb) t 2
[Switching to thread 2 (Thread 0x7f4699498700 (LWP 36371))]
#0  0x00007f46a7733b18 in do_futex_wait.constprop () from /lib64/libpthread.so.0
(gdb) py-bt
Traceback (most recent call first):
  File &quot;/usr/lib64/python3.7/threading.py&quot;, line 300, in wait
    gotit = waiter.acquire(True, timeout)
  File &quot;/usr/lib64/python3.7/queue.py&quot;, line 179, in get
    self.not_empty.wait(remaining)
  File &quot;/home/jkroon/Projects/foo-linux/bitbake/bin/bitbake-worker&quot;, line 96, in worker_flush
    worker_queue_int = worker_queue_int + worker_queue.get(True, 1)
  File &quot;/usr/lib64/python3.7/threading.py&quot;, line 870, in run
    self._target(*self._args, **self._kwargs)
  File &quot;/usr/lib64/python3.7/threading.py&quot;, line 926, in _bootstrap_inner
    self.run()
  File &quot;/usr/lib64/python3.7/threading.py&quot;, line 890, in _bootstrap
    self._bootstrap_inner()</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>86873</commentid>
    <comment_count>2</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2020-03-30 18:40:44 +0000</bug_when>
    <thetext>Jacob, For this ramdisk
    /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
the default monitor settings are not appropriate. 
Are you aware that you can change the disk space monitoring thresholds? 
See:
  https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-bb-BB_DISKMON_DIRS
Alternatively just allocate a few more GB of RAM to the ramdisk! ;-)

The 100% CPU usage is still a bug.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>86875</commentid>
    <comment_count>3</comment_count>
    <who name="Jacob Kroon">jacob.kroon</who>
    <bug_when>2020-03-30 20:05:52 +0000</bug_when>
    <thetext>(In reply to comment #2)
&gt; Jacob, For this ramdisk
&gt;     /tmp/ramdisk/foo-zeus-glibc (ramdisk) is running low (0.997GB left)
&gt; the default monitor settings are not appropriate. 
&gt; Are you aware that you can change the disk space monitoring thresholds? 
&gt; See:
&gt;  
&gt; https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-
&gt; bb-BB_DISKMON_DIRS
&gt; Alternatively just allocate a few more GB of RAM to the ramdisk! ;-)
&gt; 
&gt; The 100% CPU usage is still a bug.

Oh yeah, I had forgotten about tuning BB_DISKMON_DIRS, thanks for the reminder.

The bug isn&apos;t *that* annoying to me; I can only afford dedicating 16GB to the ramdisk, and the problem only pops up when qemu-system-native/icedtea7-native/glibc gets scheduled to build in parallel.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>94848</commentid>
    <comment_count>4</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2023-02-02 15:08:17 +0000</bug_when>
    <thetext>Jacob,
You could try using the PSI /proc/pressure regulation, see:
https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-BB_NUMBER_THREADS

and 
https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU

There&apos;s still some improvements that I need to make to the averaging but it does help on the Yocto AB and Wind River&apos;s build systems.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>94873</commentid>
    <comment_count>5</comment_count>
    <who name="Jacob Kroon">jacob.kroon</who>
    <bug_when>2023-02-03 03:54:46 +0000</bug_when>
    <thetext>Randy,
I don&apos;t know the details of how the PSI monitoring works in OE, but I doubt it would help the situation I was in; the compile task for icedtea7-native would by itself consume several gigabytes, I doubt there is anything to gain with PSI monitoring here ?

What has helped, I think, and which is the reason why I haven&apos;t had to abort builds in recent time, is that rm_work class got fixed to cleanup even more leftovers:

https://git.openembedded.org/openembedded-core/commit/?id=b25cc45c9b39f79ba0a03c4556cb2e2431677b4e

I therefore haven&apos;t seen the stuck worker for a long time, so I can&apos;t say whether the bug is still relevant in master. Since I haven&apos;t seen any activity from other users on this bug, I am fine with closing it if you&apos;d prefer that ?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>95106</commentid>
    <comment_count>6</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2023-03-09 20:36:53 +0000</bug_when>
    <thetext>There is a commit to fix the rm_work flow that Jacob linked in a previous comment.

Jacob,
PSI might help buy ensuring that less work gets started when the system is already under CPU pressure. We also want to work on implementing a per bitbake build job server instance that would also help in levelling out the CPU demand.

Anyway, this this is working for you, I&apos;ve closed the defect.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>