<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>8900</bug_id>
          
          <creation_ts>2016-01-08 14:52:46 +0000</creation_ts>
          <short_desc>Bitbake never exists when it runs out of inotify watches</short_desc>
          <delta_ts>2016-02-07 18:11:33 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>BitBake</product>
          <component>bitbake</component>
          <version>unspecified</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium</priority>
          <bug_severity>minor</bug_severity>
          <target_milestone>2.1</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Magnus Olsson">magnus</reporter>
          <assigned_to name="Ed Bartosh">eduard.bartosh</assigned_to>
          <cc>eduard.bartosh</cc>
    
    <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>57676</commentid>
    <comment_count>0</comment_count>
    <who name="Magnus Olsson">magnus</who>
    <bug_when>2016-01-08 14:52:46 +0000</bug_when>
    <thetext>Bitbake does not handle running out of inotify watches very well. If there is not enough watches available, it will crash but never shutdown because it&apos;s &quot;delaying shutdown due to active processes which appear to be holding bitbake.lock&quot;. And it cannot be terminated with CTRL+C, you have to kill it forcefully (including the subprocesses).

Expected behaviour: Bitbake should print an error and exit gracefully when it runs out of inotify watches.
Actual behaviour: Bitbake prints an error and gets stuck in an endless loop. You have to kill it manually.

I can reproduce this on my Ubuntu machine where inotify.max_user_watches is 8192 by opening the linux kernel tree using Sublime Text (which will consume most watches) and then try to run bitbake.

The following is a copy of Bitbake output when it runs out inotify max watches. When this happens, it gets stuck in an endless loop that keeps printing messages about &quot;delayed shutdown&quot;, but it never shuts down:

ERROR: No space left on device or exceeds fs.inotify.max_user_watches?                                                    ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches.
ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=&lt;value&gt;.
ERROR: Root privilege is required to modify max_user_watches.
ERROR: Command execution failed: Traceback (most recent call last):
  File &quot;/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/command.py&quot;, line 99, in runAsyncCommand
    self.cooker.updateCache()
  File &quot;/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py&quot;, line 1473, in updateCache
    if not self.parser.parse_next():
  File &quot;/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py&quot;, line 2064, in parse_next
    parsed=parsed, watcher = self.cooker.add_filewatch)
  File &quot;/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cache.py&quot;, line 631, in add_info
    watcher(info_array[0].file_depends)
  File &quot;/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py&quot;, line 211, in add_filewatch
    watcher.add_watch(f, self.watchmask, quiet=False)
  File &quot;/home/mago/dev/slp/meta-oe-core/bitbake/lib/pyinotify.py&quot;, line 1977, in add_watch
    raise WatchManagerError(err, ret_)
WatchManagerError: add_watch: cannot watch /home/mago/dev/slp/slp-build/../meta-openembedded/meta-networking/recipes-connectivity/ufw WD=-1, Errno=No space left on device (ENOSPC)


Summary: There were 5 ERROR messages shown, returning a non-zero exit code.
Delaying shutdown due to active processes which appear to be holding bitbake.lock:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
python  17840 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock
python  17841 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock
python  17842 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock
python  17843 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>58456</commentid>
    <comment_count>1</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2016-02-02 15:23:10 +0000</bug_when>
    <thetext>I can&apos;t reproduce it so far.

If I reduce max_user_watches to 501 with &apos;sysctl -n -w fs.inotify.max_user_watches=501&apos; bitbake fails quite fast, but it&apos;s not stuck:

$ bitbake core-image-minimal
WARNING: Host distribution &quot;openSUSE-13.2&quot; has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |##########################################################################################################| ETA:  00:00:00
Loaded 1315 entries from dependency cache.
ERROR: No space left on device or exceeds fs.inotify.max_user_watches?
ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches.
ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=&lt;value&gt;.
ERROR: Root privilege is required to modify max_user_watches.
ERROR: Command execution failed: Traceback (most recent call last):
  File &quot;/home/ed/git/yocto/poky/bitbake/lib/bb/command.py&quot;, line 101, in runAsyncCommand
    self.cooker.updateCache()
  File &quot;/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py&quot;, line 1541, in updateCache
    if not self.parser.parse_next():
  File &quot;/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py&quot;, line 2163, in parse_next
    parsed=parsed, watcher = self.cooker.add_filewatch)
  File &quot;/home/ed/git/yocto/poky/bitbake/lib/bb/cache.py&quot;, line 642, in add_info
    watcher(info_array[0].file_depends)
  File &quot;/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py&quot;, line 224, in add_filewatch
    watcher.add_watch(f, self.watchmask, quiet=False)
  File &quot;/home/ed/git/yocto/poky/bitbake/lib/pyinotify.py&quot;, line 1977, in add_watch
    raise WatchManagerError(err, ret_)
WatchManagerError: add_watch: cannot watch /home/ed/git/yocto/poky/meta-selftest/recipes-test/error WD=-1, Errno=No space left on device (ENOSPC)


Summary: There was 1 WARNING message shown.
Summary: There were 5 ERROR messages shown, returning a non-zero exit code.



with &apos;sysctl -n -w fs.inotify.max_user_watches=502&apos; it works.

Setting max_user_watches to very small number in another console while bitbake is running doesn&apos;t influence the build. I tried sysctl -n -w fs.inotify.max_user_watches=10&apos;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>58483</commentid>
    <comment_count>2</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2016-02-03 09:40:49 +0000</bug_when>
    <thetext>I was able to reproduce it eventually. However, it&apos;s not always 100% reproducible. Will investigate further.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>58605</commentid>
    <comment_count>3</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2016-02-04 19:34:42 +0000</bug_when>
    <thetext>Fix has been sent for review: http://lists.openembedded.org/pipermail/bitbake-devel/2016-February/007026.html</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>58705</commentid>
    <comment_count>4</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2016-02-07 18:11:33 +0000</bug_when>
    <thetext>The fix has been merged to bitbake master branch: http://cgit.openembedded.org/bitbake/commit/?id=3f67600dc3292bc8208644ce89e8bf7ab95cf2e7</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>