14105 2020-10-22 16:04:15 +0000 Build fails during parse with pyinotify.WatchManagerError: add_watch: cannot watch /mnt/b/yoe/master/conf WD=-1 2020-10-29 17:32:36 +0000 1 1 7 Build System, Metadata & Runtime OE-Core oe-core other 3.3 All Multiple RESOLVED FIXED Undecided normal --- 1 raj.khem unassigned mhalstead richard.purdie --- --- Don't know oldest_to_newest 88477 0 raj.khem 2020-10-22 16:04:15 +0000 I am seeing below error after a build and changing MACHINE to some other machine and trying another build. This error goes away if I delete tmp/ content then it works. Traceback (most recent call last): File "/mnt/b/yoe/master/sources/bitbake/bin/bitbake", line 35, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "/mnt/b/yoe/master/sources/bitbake/lib/bb/main.py", line 382, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "/mnt/b/yoe/master/sources/bitbake/lib/bb/ui/knotty.py", line 396, in main params.updateToServer(server, os.environ.copy()) File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer raise Exception("Unable to update the server configuration with local parameters: %s" % error) Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/mnt/b/yoe/master/sources/bitbake/lib/bb/command.py", line 81, in runCommand result = command_method(self, commandline) File "/mnt/b/yoe/master/sources/bitbake/lib/bb/command.py", line 277, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 488, in updateConfigOpts self.reset() File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 1699, in reset self.initConfigurationData() File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 373, in initConfigurationData self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher) File "/mnt/b/yoe/master/sources/bitbake/lib/bb/cooker.py", line 289, in add_filewatch watcher.add_watch(f, self.watchmask, quiet=False) File "/mnt/b/yoe/master/sources/bitbake/lib/pyinotify.py", line 1908, in add_watch raise WatchManagerError(err, ret_) pyinotify.WatchManagerError: add_watch: cannot watch /mnt/b/yoe/master/conf WD=-1, Errno=No space left on device (ENOSPC) 88503 1 richard.purdie 2020-10-27 12:46:01 +0000 Sounds like something is using up all the inotify watches. Why an existing build would do that, not sure, sounds like something is misconfigured. We really need a way to reproduce/debug it... What was the inotify max watches on that machine? 88504 2 raj.khem 2020-10-27 13:10:32 +0000 (In reply to comment #1) > Sounds like something is using up all the inotify watches. Why an existing > build would do that, not sure, sounds like something is misconfigured. We > really need a way to reproduce/debug it... > > What was the inotify max watches on that machine? % cat /proc/sys/fs/inotify/max_user_instances 1024 whenever it happens, its usually after/during when I build eSDK 88505 3 raj.khem 2020-10-27 13:11:37 +0000 (In reply to comment #1) > Sounds like something is using up all the inotify watches. Why an existing > build would do that, not sure, sounds like something is misconfigured. We > really need a way to reproduce/debug it... > > What was the inotify max watches on that machine? and this sdk is big. Around 3GB compressed, with Clang and gcc based toolchains in it and all debug/dev packages added 88506 4 richard.purdie 2020-10-27 16:48:29 +0000 Have you tried increasing max_user_instances ? eSDK involves running bitbake within bitbake which may be the issue, two sets of inotify watches on all the metadata (for memres and cache invalidation) could eat through 1024 quickly. I know we increase this on the autobuilder workers. 88507 5 raj.khem 2020-10-27 17:02:50 +0000 (In reply to comment #4) > Have you tried increasing max_user_instances ? > > eSDK involves running bitbake within bitbake which may be the issue, two > sets of inotify watches on all the metadata (for memres and cache > invalidation) could eat through 1024 quickly. I know we increase this on the > autobuilder workers. These are distro defaults. What was it changed to on AB. I think these should be documented somewhere for community as well, perhaps How Do I sections in wiki. OK I will try this out and see if this helps. What is a good value ? 88508 6 richard.purdie 2020-10-28 04:51:46 +0000 Michael: Do we have this documented for the autobuilder setup somewhere? 88514 7 raj.khem 2020-10-29 12:28:01 +0000 OK I ran into this issue again today. So the situation was bitbake <imabe> -cpopulate_sdk_ext - Went fine publish the eSDK on same system - Went fine but then installing the SDK on same system runs into this error So I changed $ sudo sysctl -w fs.inotify.max_user_instances=4098 $ sudo sysctl -w fs.inotify.max_user_watches=24576 and it did not help. 88516 8 raj.khem 2020-10-29 17:32:36 +0000 % sudo sysctl -n -w fs.inotify.max_user_watches=1048576 helped