| Summary: | Builds fail to start after a configuration change when Toaster is running in analysis (i.e. interactive) mode | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Toaster | Reporter: | Belen Barros Pena <belen.barros.pena> |
| Component: | toaster | Assignee: | Alexandru Damian <alexandru.damian> |
| Status: | VERIFIED FIXED | QA Contact: | Mihail Stanciu <stanciux.mihail> |
| Severity: | major | ||
| Priority: | High | CC: | alexandru.damian, belen.barros.pena, bluelightning, cristiana.voicu, jessica.zhang, richard.purdie |
| Version: | unspecified | ||
| Target Milestone: | 1.8 M4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Belen Barros Pena
2015-02-13 11:03:19 UTC
I reproduced this and traced it. The issue is that there are a ton of writes to the toaster sqlite database and this overflows the inotify queue. Changing cooker.py to do:
def config_notifications(self, event):
+ bb.warn(str(event))
bb.parse.update_cache(event.path)
self.baseconfig_valid = False
results in a ton of output ending with:
WARNING: <Event dir=False mask=0x2 maskname=IN_MODIFY name=toaster.sqlite path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite wd=21 >
WARNING: <Event dir=False mask=0x8 maskname=IN_CLOSE_WRITE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x200 maskname=IN_DELETE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x100 maskname=IN_CREATE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x2 maskname=IN_MODIFY name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event dir=False mask=0x2 maskname=IN_MODIFY name=toaster.sqlite path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite wd=21 >
WARNING: <Event dir=False mask=0x8 maskname=IN_CLOSE_WRITE name=toaster.sqlite-journal path=/media/build1/poky/build pathname=/media/build1/poky/build/toaster.sqlite-journal wd=21 >
WARNING: <Event mask=0x4000 maskname=IN_Q_OVERFLOW >
so there are two takeaways:
a) We should not be watching build, we clearly are but shouldn't be.
b) We need to trap overflow events and warn the user instead of crashing.
Now I think about it, conf containing key conf files is in build so we need to watch this. We may need to write the toaster sqlite data somewhere else so we don't overload the inotify queue I reasoned that the overflowing is coming from the fact that the notifications are read very rarely, only during build setup, in updateCache. But the notification data structures do not influence a currently-running build, so we can gather event data also during the build run, and use the recorded event data for the next build setup. This patch adds a server idle function that performs inotify process_event during the actual server run. Testing shows no impact in performance, and that the bug no longer occurs. Pushed for review on poky-contrib: adamian/20150223_bug_7316 Pushed new version for review on http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=adamian/20150305_review&id=c7b33b28cc014bd6537c77882abb8bb6a24f422c Merged to master. Verified on master: 0aff27522d467ee5d02289594207ddef4add7081 |