Bug 10873

Created attachment 3581 [details]
debian8 fail log -- fs.inotify.max_user_watches

adding another sample fail log for debian8

The TC failing is the same one in the 4 different OS where the failure is present:

- test case failing is "test_qemux86_directdisk" and is executed after  "test_qemu"

OS failing (centos7, debian8, fedora23, fedora24)

2017-01-03 01:43:30 -   test_qemu (oeqa.selftest.wic.Wic) ... OK (117.807s)
2017-01-03 01:45:27 -   test_qemux86_directdisk (oeqa.selftest.wic.Wic) ... FAIL (23.353s)

The workaround for this failure is to set the max_user_watches to 64k on all the workers, the same watch count that opensuse uses as default.

This issue has been double checked on the morty release, it is not present there. Bisecting it now.

the workaround has been proven correct at a debian8 machine that was failing previously.

I'm wondering if it would be quicker to bisect if you reduce max_user_watches to a small enough value that selftest breaks sooner if it starts leaking.

Also, why is running out of watches so fatal.  Can it be a warning? :)

(In reply to comment #6)
> I'm wondering if it would be quicker to bisect if you reduce
> max_user_watches to a small enough value that selftest breaks sooner if it
> starts leaking.
This test can be done manually. I'll keep it in mind.
> 
> Also, why is running out of watches so fatal.  Can it be a warning? :)
Agree, watches are overestimated.

(In reply to comment #6)
> I'm wondering if it would be quicker to bisect if you reduce
> max_user_watches to a small enough value that selftest breaks sooner if it
> starts leaking.
> 
> Also, why is running out of watches so fatal.  Can it be a warning? :)

No, it can't. Memory resident bitbake needs this and I don't want multiple different ways for the cache code to work, we've enough different codepaths already. This is a place we need one codepath for everything. We did optmise it such that the overhead is minimal. We do need to figure out why they appear to be leaking though.

I'm pretty sure that ftrace can provide tracing on what processes are creating the inotify watches but I'm struggling to make it do what I want it to do.

Created attachment 3586 [details]
pyinotify -h

there is this bitbake/lib/pynotify.py that besides being the wrapper that the cooker uses, is also a command line for monitoring them. Usage attached.

Working in a RRS fix due to tinfoil API changes [1], i found an error related to BBCooker don't close all the fd's at ends of bitbake_main, see the stack trace,

Traceback (most recent call last):
  File "./rrs/rrs/tools/rrs_upgrade_history.py", line 263, in <module>
    upgrade_history(options, logger)
  File "./rrs/rrs/tools/rrs_upgrade_history.py", line 247, in upgrade_history
    do_loop(layerbranch, ct, logger)
  File "./rrs/rrs/tools/rrs_upgrade_history.py", line 185, in do_loop
    nocheckout=True)
  File "/srv/rrs/rrs/rrs/tools/common.py", line 103, in load_recipes
    logger=logger)
  File "/srv/rrs/rrs/layerindex/recipeparse.py", line 100, in init_parser
    tinfoil = _setup_tinfoil(bitbakepath, enable_tracking)
  File "/srv/rrs/rrs/layerindex/recipeparse.py", line 35, in _setup_tinfoil
    tinfoil.prepare(config_only = True)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 210, in prepare
    extrafeatures)
  File "/srv/rrs/layers/bitbake/lib/bb/main.py", line 513, in setup_bitbake
    server = start_server(servermodule, configParams, configuration, featureset)
  File "/srv/rrs/layers/bitbake/lib/bb/main.py", line 354, in start_server
    cooker = bb.cooker.BBCooker(configuration, features)
  File "/srv/rrs/layers/bitbake/lib/bb/cooker.py", line 178, in __init__
    self.configwatcher = pyinotify.WatchManager()
  File "/srv/rrs/layers/bitbake/lib/pyinotify.py", line 1764, in __init__
    raise OSError(err % self._inotify_wrapper.str_errno())
OSError: Cannot initialize new instance of inotify, Errno=Too many open files (EMFILE)
DEBUG: Removed the following variables from the environment: HISTSIZE, VIRTUAL_ENV, MAIL, PS1, HISTCONTROL, DJANGO_SETTINGS_MODULE, TZ, LS_COLORS, SHLVL, LANG, _, HOSTNAME, XDG_SESSION_ID, LESSOPEN, OLDPWD


This issue seems similar to this one.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=10863

I sent a patch to the ML, i hope it fix this error.

http://lists.openembedded.org/pipermail/bitbake-devel/2017-January/008358.html

Seems like the test that is making the GDC AB fail (ABs with max_user_watches=8k) is "test_postinst_roofs_and_boot" the test case is not failing itself but is showing a lot of warnings and hence making another test cases fail. I did a test with branch http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=JosePerez/watchers_runqemu that takes out this test case and the execution was correct everything PASS. attaching log with both executions.

Created attachment 3590 [details]
Execution with test_postinst_roofs_and_boot

Created attachment 3591 [details]
Execution without test_postinst_roofs_and_boot

My patch was integrate [1] but i found other errors related to non close pipe's, sockets and inotify inodes, i'm working on it.

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e016eb10b075e280b4e78a04e47b59a173386421

Moving to bitbake component and update title.

I sent other two patches one for pipe leakage [1] and the other that fix this issue about inotify fd's [2].


[1] http://lists.openembedded.org/pipermail/bitbake-devel/2017-January/008370.html
[2] http://lists.openembedded.org/pipermail/bitbake-devel/2017-January/008371.html

Fixed in bitbake ae6045b84978940c365c95c33d6996359c3e299d.

Bug not present on 2.3 M3 QA test cycle 

https://wiki.yoctoproject.org/wiki/WW14_-_2017-03-28_-_Full_Test_Cycle_2.3_M3_rc2#OE-Core