Bug 8900 - Bitbake never exists when it runs out of inotify watches
Summary: Bitbake never exists when it runs out of inotify watches
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium minor
Target Milestone: 2.1
Assignee: Ed Bartosh
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-01-08 14:52 UTC by Magnus Olsson
Modified: 2016-02-07 18:11 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Olsson 2016-01-08 14:52:46 UTC
Bitbake does not handle running out of inotify watches very well. If there is not enough watches available, it will crash but never shutdown because it's "delaying shutdown due to active processes which appear to be holding bitbake.lock". And it cannot be terminated with CTRL+C, you have to kill it forcefully (including the subprocesses).

Expected behaviour: Bitbake should print an error and exit gracefully when it runs out of inotify watches.
Actual behaviour: Bitbake prints an error and gets stuck in an endless loop. You have to kill it manually.

I can reproduce this on my Ubuntu machine where inotify.max_user_watches is 8192 by opening the linux kernel tree using Sublime Text (which will consume most watches) and then try to run bitbake.

The following is a copy of Bitbake output when it runs out inotify max watches. When this happens, it gets stuck in an endless loop that keeps printing messages about "delayed shutdown", but it never shuts down:

ERROR: No space left on device or exceeds fs.inotify.max_user_watches?                                                    ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches.
ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=<value>.
ERROR: Root privilege is required to modify max_user_watches.
ERROR: Command execution failed: Traceback (most recent call last):
  File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/command.py", line 99, in runAsyncCommand
    self.cooker.updateCache()
  File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py", line 1473, in updateCache
    if not self.parser.parse_next():
  File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py", line 2064, in parse_next
    parsed=parsed, watcher = self.cooker.add_filewatch)
  File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cache.py", line 631, in add_info
    watcher(info_array[0].file_depends)
  File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py", line 211, in add_filewatch
    watcher.add_watch(f, self.watchmask, quiet=False)
  File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/pyinotify.py", line 1977, in add_watch
    raise WatchManagerError(err, ret_)
WatchManagerError: add_watch: cannot watch /home/mago/dev/slp/slp-build/../meta-openembedded/meta-networking/recipes-connectivity/ufw WD=-1, Errno=No space left on device (ENOSPC)


Summary: There were 5 ERROR messages shown, returning a non-zero exit code.
Delaying shutdown due to active processes which appear to be holding bitbake.lock:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
python  17840 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock
python  17841 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock
python  17842 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock
python  17843 mago   12u   REG    8,6        0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock
Comment 1 Ed Bartosh 2016-02-02 15:23:10 UTC
I can't reproduce it so far.

If I reduce max_user_watches to 501 with 'sysctl -n -w fs.inotify.max_user_watches=501' bitbake fails quite fast, but it's not stuck:

$ bitbake core-image-minimal
WARNING: Host distribution "openSUSE-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |##########################################################################################################| ETA:  00:00:00
Loaded 1315 entries from dependency cache.
ERROR: No space left on device or exceeds fs.inotify.max_user_watches?
ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches.
ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=<value>.
ERROR: Root privilege is required to modify max_user_watches.
ERROR: Command execution failed: Traceback (most recent call last):
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/command.py", line 101, in runAsyncCommand
    self.cooker.updateCache()
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py", line 1541, in updateCache
    if not self.parser.parse_next():
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py", line 2163, in parse_next
    parsed=parsed, watcher = self.cooker.add_filewatch)
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/cache.py", line 642, in add_info
    watcher(info_array[0].file_depends)
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py", line 224, in add_filewatch
    watcher.add_watch(f, self.watchmask, quiet=False)
  File "/home/ed/git/yocto/poky/bitbake/lib/pyinotify.py", line 1977, in add_watch
    raise WatchManagerError(err, ret_)
WatchManagerError: add_watch: cannot watch /home/ed/git/yocto/poky/meta-selftest/recipes-test/error WD=-1, Errno=No space left on device (ENOSPC)


Summary: There was 1 WARNING message shown.
Summary: There were 5 ERROR messages shown, returning a non-zero exit code.



with 'sysctl -n -w fs.inotify.max_user_watches=502' it works.

Setting max_user_watches to very small number in another console while bitbake is running doesn't influence the build. I tried sysctl -n -w fs.inotify.max_user_watches=10'
Comment 2 Ed Bartosh 2016-02-03 09:40:49 UTC
I was able to reproduce it eventually. However, it's not always 100% reproducible. Will investigate further.
Comment 3 Ed Bartosh 2016-02-04 19:34:42 UTC
Fix has been sent for review: http://lists.openembedded.org/pipermail/bitbake-devel/2016-February/007026.html
Comment 4 Ed Bartosh 2016-02-07 18:11:33 UTC
The fix has been merged to bitbake master branch: http://cgit.openembedded.org/bitbake/commit/?id=3f67600dc3292bc8208644ce89e8bf7ab95cf2e7