Created attachment 3580 [details] centos7 failure log -- exceeds fs.inotify.max_user_watches oe-selftest --run-all-tests is unable to finish execution on most systems it is executed. It can fail with errors that indicate either there is no space or that the inode count is exceeded. Both errors are an indication of exceeding the max_user_watches given that no lack of disk space or inode count can be seen when verified. For the following procedure to reproduce the failure, it is recommended to use the fastest machine at hand as the process takes several hours. Procedure: 1. on the poky master branch (dbb247c), init a build environment: poky $ . ./oe-init-build-env 2. in order to execute oe-selftest, add the meta-selftest layer: build $ bitbake-layers add-layer ../meta-selftest 3. run all tests from selftest: build $ oe-selftest --run-all-tests Expected Result: The test session is completed with no python exceptions. Actual Result: The test session shows errors on disk space, inode count and/or max_user_watches. Sample failure logs attached. Reproducibility: 100% on machines that have 8k max_user_watches (default for ubuntu, fedora, centos, debian). Hasn't been reproduced on AB Workers with 64k max_user_watches (all the main AB workers and opensuse workers of the GDC AB).
Created attachment 3581 [details] debian8 fail log -- fs.inotify.max_user_watches adding another sample fail log for debian8
The TC failing is the same one in the 4 different OS where the failure is present: - test case failing is "test_qemux86_directdisk" and is executed after "test_qemu" OS failing (centos7, debian8, fedora23, fedora24) 2017-01-03 01:43:30 - test_qemu (oeqa.selftest.wic.Wic) ... OK (117.807s) 2017-01-03 01:45:27 - test_qemux86_directdisk (oeqa.selftest.wic.Wic) ... FAIL (23.353s)
The workaround for this failure is to set the max_user_watches to 64k on all the workers, the same watch count that opensuse uses as default.
This issue has been double checked on the morty release, it is not present there. Bisecting it now.
the workaround has been proven correct at a debian8 machine that was failing previously.
I'm wondering if it would be quicker to bisect if you reduce max_user_watches to a small enough value that selftest breaks sooner if it starts leaking. Also, why is running out of watches so fatal. Can it be a warning? :)
(In reply to comment #6) > I'm wondering if it would be quicker to bisect if you reduce > max_user_watches to a small enough value that selftest breaks sooner if it > starts leaking. This test can be done manually. I'll keep it in mind. > > Also, why is running out of watches so fatal. Can it be a warning? :) Agree, watches are overestimated.
(In reply to comment #6) > I'm wondering if it would be quicker to bisect if you reduce > max_user_watches to a small enough value that selftest breaks sooner if it > starts leaking. > > Also, why is running out of watches so fatal. Can it be a warning? :) No, it can't. Memory resident bitbake needs this and I don't want multiple different ways for the cache code to work, we've enough different codepaths already. This is a place we need one codepath for everything. We did optmise it such that the overhead is minimal. We do need to figure out why they appear to be leaking though.
I'm pretty sure that ftrace can provide tracing on what processes are creating the inotify watches but I'm struggling to make it do what I want it to do.
Created attachment 3586 [details] pyinotify -h there is this bitbake/lib/pynotify.py that besides being the wrapper that the cooker uses, is also a command line for monitoring them. Usage attached.
Working in a RRS fix due to tinfoil API changes [1], i found an error related to BBCooker don't close all the fd's at ends of bitbake_main, see the stack trace, Traceback (most recent call last): File "./rrs/rrs/tools/rrs_upgrade_history.py", line 263, in <module> upgrade_history(options, logger) File "./rrs/rrs/tools/rrs_upgrade_history.py", line 247, in upgrade_history do_loop(layerbranch, ct, logger) File "./rrs/rrs/tools/rrs_upgrade_history.py", line 185, in do_loop nocheckout=True) File "/srv/rrs/rrs/rrs/tools/common.py", line 103, in load_recipes logger=logger) File "/srv/rrs/rrs/layerindex/recipeparse.py", line 100, in init_parser tinfoil = _setup_tinfoil(bitbakepath, enable_tracking) File "/srv/rrs/rrs/layerindex/recipeparse.py", line 35, in _setup_tinfoil tinfoil.prepare(config_only = True) File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 210, in prepare extrafeatures) File "/srv/rrs/layers/bitbake/lib/bb/main.py", line 513, in setup_bitbake server = start_server(servermodule, configParams, configuration, featureset) File "/srv/rrs/layers/bitbake/lib/bb/main.py", line 354, in start_server cooker = bb.cooker.BBCooker(configuration, features) File "/srv/rrs/layers/bitbake/lib/bb/cooker.py", line 178, in __init__ self.configwatcher = pyinotify.WatchManager() File "/srv/rrs/layers/bitbake/lib/pyinotify.py", line 1764, in __init__ raise OSError(err % self._inotify_wrapper.str_errno()) OSError: Cannot initialize new instance of inotify, Errno=Too many open files (EMFILE) DEBUG: Removed the following variables from the environment: HISTSIZE, VIRTUAL_ENV, MAIL, PS1, HISTCONTROL, DJANGO_SETTINGS_MODULE, TZ, LS_COLORS, SHLVL, LANG, _, HOSTNAME, XDG_SESSION_ID, LESSOPEN, OLDPWD This issue seems similar to this one. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=10863
I sent a patch to the ML, i hope it fix this error. http://lists.openembedded.org/pipermail/bitbake-devel/2017-January/008358.html
Seems like the test that is making the GDC AB fail (ABs with max_user_watches=8k) is "test_postinst_roofs_and_boot" the test case is not failing itself but is showing a lot of warnings and hence making another test cases fail. I did a test with branch http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=JosePerez/watchers_runqemu that takes out this test case and the execution was correct everything PASS. attaching log with both executions.
Created attachment 3590 [details] Execution with test_postinst_roofs_and_boot
Created attachment 3591 [details] Execution without test_postinst_roofs_and_boot
My patch was integrate [1] but i found other errors related to non close pipe's, sockets and inotify inodes, i'm working on it. [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e016eb10b075e280b4e78a04e47b59a173386421
Moving to bitbake component and update title.
I sent other two patches one for pipe leakage [1] and the other that fix this issue about inotify fd's [2]. [1] http://lists.openembedded.org/pipermail/bitbake-devel/2017-January/008370.html [2] http://lists.openembedded.org/pipermail/bitbake-devel/2017-January/008371.html
Fixed in bitbake ae6045b84978940c365c95c33d6996359c3e299d.
Bug not present on 2.3 M3 QA test cycle https://wiki.yoctoproject.org/wiki/WW14_-_2017-03-28_-_Full_Test_Cycle_2.3_M3_rc2#OE-Core