| Summary: | Bitbake never exists when it runs out of inotify watches | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Magnus Olsson <magnus> |
| Component: | bitbake | Assignee: | Ed Bartosh <eduard.bartosh> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | minor | ||
| Priority: | Medium | CC: | eduard.bartosh, poky.bs.watcher, poky.watcher |
| Version: | unspecified | ||
| Target Milestone: | 2.1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
I can't reproduce it so far.
If I reduce max_user_watches to 501 with 'sysctl -n -w fs.inotify.max_user_watches=501' bitbake fails quite fast, but it's not stuck:
$ bitbake core-image-minimal
WARNING: Host distribution "openSUSE-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |##########################################################################################################| ETA: 00:00:00
Loaded 1315 entries from dependency cache.
ERROR: No space left on device or exceeds fs.inotify.max_user_watches?
ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches.
ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=<value>.
ERROR: Root privilege is required to modify max_user_watches.
ERROR: Command execution failed: Traceback (most recent call last):
File "/home/ed/git/yocto/poky/bitbake/lib/bb/command.py", line 101, in runAsyncCommand
self.cooker.updateCache()
File "/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py", line 1541, in updateCache
if not self.parser.parse_next():
File "/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py", line 2163, in parse_next
parsed=parsed, watcher = self.cooker.add_filewatch)
File "/home/ed/git/yocto/poky/bitbake/lib/bb/cache.py", line 642, in add_info
watcher(info_array[0].file_depends)
File "/home/ed/git/yocto/poky/bitbake/lib/bb/cooker.py", line 224, in add_filewatch
watcher.add_watch(f, self.watchmask, quiet=False)
File "/home/ed/git/yocto/poky/bitbake/lib/pyinotify.py", line 1977, in add_watch
raise WatchManagerError(err, ret_)
WatchManagerError: add_watch: cannot watch /home/ed/git/yocto/poky/meta-selftest/recipes-test/error WD=-1, Errno=No space left on device (ENOSPC)
Summary: There was 1 WARNING message shown.
Summary: There were 5 ERROR messages shown, returning a non-zero exit code.
with 'sysctl -n -w fs.inotify.max_user_watches=502' it works.
Setting max_user_watches to very small number in another console while bitbake is running doesn't influence the build. I tried sysctl -n -w fs.inotify.max_user_watches=10'
I was able to reproduce it eventually. However, it's not always 100% reproducible. Will investigate further. Fix has been sent for review: http://lists.openembedded.org/pipermail/bitbake-devel/2016-February/007026.html The fix has been merged to bitbake master branch: http://cgit.openembedded.org/bitbake/commit/?id=3f67600dc3292bc8208644ce89e8bf7ab95cf2e7 |
Bitbake does not handle running out of inotify watches very well. If there is not enough watches available, it will crash but never shutdown because it's "delaying shutdown due to active processes which appear to be holding bitbake.lock". And it cannot be terminated with CTRL+C, you have to kill it forcefully (including the subprocesses). Expected behaviour: Bitbake should print an error and exit gracefully when it runs out of inotify watches. Actual behaviour: Bitbake prints an error and gets stuck in an endless loop. You have to kill it manually. I can reproduce this on my Ubuntu machine where inotify.max_user_watches is 8192 by opening the linux kernel tree using Sublime Text (which will consume most watches) and then try to run bitbake. The following is a copy of Bitbake output when it runs out inotify max watches. When this happens, it gets stuck in an endless loop that keeps printing messages about "delayed shutdown", but it never shuts down: ERROR: No space left on device or exceeds fs.inotify.max_user_watches? ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches. ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=<value>. ERROR: Root privilege is required to modify max_user_watches. ERROR: Command execution failed: Traceback (most recent call last): File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/command.py", line 99, in runAsyncCommand self.cooker.updateCache() File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py", line 1473, in updateCache if not self.parser.parse_next(): File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py", line 2064, in parse_next parsed=parsed, watcher = self.cooker.add_filewatch) File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cache.py", line 631, in add_info watcher(info_array[0].file_depends) File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/bb/cooker.py", line 211, in add_filewatch watcher.add_watch(f, self.watchmask, quiet=False) File "/home/mago/dev/slp/meta-oe-core/bitbake/lib/pyinotify.py", line 1977, in add_watch raise WatchManagerError(err, ret_) WatchManagerError: add_watch: cannot watch /home/mago/dev/slp/slp-build/../meta-openembedded/meta-networking/recipes-connectivity/ufw WD=-1, Errno=No space left on device (ENOSPC) Summary: There were 5 ERROR messages shown, returning a non-zero exit code. Delaying shutdown due to active processes which appear to be holding bitbake.lock: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME python 17840 mago 12u REG 8,6 0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock python 17841 mago 12u REG 8,6 0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock python 17842 mago 12u REG 8,6 0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock python 17843 mago 12u REG 8,6 0 5773930 /home/mago/dev/slp/slp-build/bitbake.lock