Bug 14105 - Build fails during parse with pyinotify.WatchManagerError: add_watch: cannot watch /mnt/b/yoe/master/conf WD=-1
Summary: Build fails during parse with pyinotify.WatchManagerError: add_watch: cannot ...
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: oe-core other (show other bugs)
Version: 3.3
Hardware: All Multiple
: Undecided normal
Target Milestone: ---
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-10-22 16:04 UTC by Khem Raj
Modified: 2020-10-29 17:32 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Khem Raj 2020-10-22 16:04:15 UTC
I am seeing below error after a build and changing MACHINE to some other machine and trying another build. This error goes away if I delete tmp/ content then it works.

Traceback (most recent call last):
  File "/mnt/b/yoe/master/sources/bitbake/bin/bitbake", line 35, in <module>
    sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/main.py", line 382, in bitbake_main
    return ui_module.main(server_connection.connection, server_connection.events,
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/ui/knotty.py", line 396, in main
    params.updateToServer(server, os.environ.copy())
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer
    raise Exception("Unable to update the server configuration with local parameters: %s" % error)
Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last):
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/command.py", line 81, in runCommand
    result = command_method(self, commandline)
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/command.py", line 277, in updateConfig
    command.cooker.updateConfigOpts(options, environment, cmdline)
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 488, in updateConfigOpts
    self.reset()
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 1699, in reset
    self.initConfigurationData()
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 373, in initConfigurationData
    self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher)
  File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 289, in add_filewatch
    watcher.add_watch(f, self.watchmask, quiet=False)
  File "/mnt/b/yoe/master/sources/bitbake/lib/pyinotify.py", line 1908, in add_watch
    raise WatchManagerError(err, ret_)
pyinotify.WatchManagerError: add_watch: cannot watch /mnt/b/yoe/master/conf WD=-1, Errno=No space left on device (ENOSPC)
Comment 1 Richard Purdie 2020-10-27 12:46:01 UTC
Sounds like something is using up all the inotify watches. Why an existing build would do that, not sure, sounds like something is misconfigured. We really need a way to reproduce/debug it...

What was the inotify max watches on that machine?
Comment 2 Khem Raj 2020-10-27 13:10:32 UTC
(In reply to comment #1)
> Sounds like something is using up all the inotify watches. Why an existing
> build would do that, not sure, sounds like something is misconfigured. We
> really need a way to reproduce/debug it...
> 
> What was the inotify max watches on that machine?

% cat /proc/sys/fs/inotify/max_user_instances
1024

whenever it happens, its usually after/during when I build eSDK
Comment 3 Khem Raj 2020-10-27 13:11:37 UTC
(In reply to comment #1)
> Sounds like something is using up all the inotify watches. Why an existing
> build would do that, not sure, sounds like something is misconfigured. We
> really need a way to reproduce/debug it...
> 
> What was the inotify max watches on that machine?

and this sdk is big. Around 3GB compressed, with Clang and gcc based toolchains in it and all debug/dev packages added
Comment 4 Richard Purdie 2020-10-27 16:48:29 UTC
Have you tried increasing max_user_instances ?

eSDK involves running bitbake within bitbake which may be the issue, two sets of inotify watches on all the metadata (for memres and cache invalidation) could eat through 1024 quickly. I know we increase this on the autobuilder workers.
Comment 5 Khem Raj 2020-10-27 17:02:50 UTC
(In reply to comment #4)
> Have you tried increasing max_user_instances ?
> 
> eSDK involves running bitbake within bitbake which may be the issue, two
> sets of inotify watches on all the metadata (for memres and cache
> invalidation) could eat through 1024 quickly. I know we increase this on the
> autobuilder workers.

These are distro defaults. What was it changed to on AB. I think these should be documented somewhere for community as well, perhaps How Do I sections in wiki.

OK I will try this out and see if this helps. What is a good value ?
Comment 6 Richard Purdie 2020-10-28 04:51:46 UTC
Michael: Do we have this documented for the autobuilder setup somewhere?
Comment 7 Khem Raj 2020-10-29 12:28:01 UTC
OK I ran into this issue again today. So the situation was

bitbake <imabe> -cpopulate_sdk_ext - Went fine

publish the eSDK on same system - Went fine

but then installing the SDK on same system runs into this error

So I changed 

$ sudo sysctl -w fs.inotify.max_user_instances=4098
$ sudo sysctl -w fs.inotify.max_user_watches=24576

and it did not help.
Comment 8 Khem Raj 2020-10-29 17:32:36 UTC
% sudo sysctl -n -w fs.inotify.max_user_watches=1048576

helped