Bug 7316

Summary: Builds fail to start after a configuration change when Toaster is running in analysis (i.e. interactive) mode
Product: [Build System, Metadata & Runtime] Toaster Reporter: Belen Barros Pena <belen.barros.pena>
Component: toasterAssignee: Alexandru Damian <alexandru.damian>
Status: VERIFIED FIXED QA Contact: Mihail Stanciu <stanciux.mihail>
Severity: major    
Priority: High CC: alexandru.damian, belen.barros.pena, bluelightning, cristiana.voicu, jessica.zhang, richard.purdie
Version: unspecified   
Target Milestone: 1.8 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Belen Barros Pena 2015-02-13 11:03:19 UTC
Found in master:231d4a9d3d89af4b1b5f6ea439c630e4cac82079 

Steps to reproduce:

1. Source the environment script
2. Start Toaster in analysis mode (source toaster start)
3. Build something (e.g. busybox)
4. When the build completes, edit the MACHINE variable in the local.conf file
5. Start a busybox build

The build fails to start with this error:

ERROR: Command execution failed: Traceback (most recent call last):
File "/home/yocto/master/bitbake/lib/bb/command.py", line 99, in
runAsyncCommand
self.cooker.updateCache()
File "/home/yocto/master/bitbake/lib/bb/cooker.py", line 1438, in updateCache
n.process_events()
File "/home/yocto/master/bitbake/lib/pyinotify.py", line 1321, in
process_events
self._default_proc_fun(revent)
File "/home/yocto/master/bitbake/lib/bb/cooker.py", line 174, in
config_notifications
bb.parse.update_cache(event.path)
AttributeError: 'Event' object has no attribute 'path'

This only happens when Toaster is running.
Comment 1 Richard Purdie 2015-02-13 12:13:30 UTC
I reproduced this and traced it. The issue is that there are a ton of writes to the toaster sqlite database and this overflows the inotify queue. Changing cooker.py to do:
 
     def config_notifications(self, event):
+        bb.warn(str(event))
         bb.parse.update_cache(event.path)
         self.baseconfig_valid = False

results in a ton of output ending with:

WARNING: <Event dir=False mask=0x2 maskname=IN_MODIFY name=toaster.sqlite path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite wd=21 >
WARNING: <Event dir=False mask=0x8 maskname=IN_CLOSE_WRITE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x200 maskname=IN_DELETE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x100 maskname=IN_CREATE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x2 maskname=IN_MODIFY name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x2 maskname=IN_MODIFY name=toaster.sqlite path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite wd=21 >
WARNING: <Event dir=False mask=0x8 maskname=IN_CLOSE_WRITE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event mask=0x4000 maskname=IN_Q_OVERFLOW >

so there are two takeaways:

a) We should not be watching build, we clearly are but shouldn't be.
b) We need to trap overflow events and warn the user instead of crashing.
Comment 2 Richard Purdie 2015-02-13 12:17:02 UTC
Now I think about it, conf containing key conf files is in build so we need to watch this. We may need to write the toaster sqlite data somewhere else so we don't overload the inotify queue
Comment 3 Alexandru Damian 2015-02-23 19:38:04 UTC
I reasoned that the overflowing is coming from the fact that the notifications are read very rarely, only during build setup, in updateCache.

But the notification data structures do not influence a currently-running build, so we can gather event data also during the build run, and use the recorded event data for the next build setup.


This patch adds a server idle function that performs inotify process_event during the actual server run.

Testing shows no impact in performance, and that the bug no longer occurs.



Pushed for review on 

poky-contrib: adamian/20150223_bug_7316
Comment 5 Alexandru Damian 2015-03-11 15:50:13 UTC
Merged to master.
Comment 6 Mihail Stanciu 2015-03-13 15:42:08 UTC
Verified on master: 0aff27522d467ee5d02289594207ddef4add7081