(Not sure what the component should be, assigning to Michael on Pauls suggestion) Here is an example of a failed UploadToasterEventlog task: looks like the upload stalls and then timeouts. https://autobuilder.yoctoproject.org/main/builders/nightly-x86-lsb/builds/476/steps/UploadToasterEventlog/logs/stdio My initial thought was that this was a transient network error but decided to look at a successful log as well. This didn't look much better: https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/builds/483/steps/UploadToasterEventlog/logs/stdio The upload seems to stall much the same way but instead of timeing out it spews out a _lot_ of python tracebacks such as the ones below: WARNING: buildhistory is not enabled. Please enable INHERIT += "buildhistory" to see image details. NOTE: ToasterUI waiting for events ERROR: 'module' object has no attribute 'ReachableStamps' Traceback (most recent call last): File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main event = eventHandler.waitEvent(0.25) File "/var/lib/bitbake/bin/toaster-eventreplay", line 139, in waitEvent return self._create_event(nextline) File "/var/lib/bitbake/bin/toaster-eventreplay", line 115, in _create_event event_class = _import_class(event_data['class']) File "/var/lib/bitbake/bin/toaster-eventreplay", line 110, in _import_class return getattr(module, moduleklass) AttributeError: 'module' object has no attribute 'ReachableStamps' WARNING: Error data dump [' File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main\n event = eventHandler.waitEvent(0.25)\n'] {'build_history_enabled': False, 'buildinfohelper': <bb.ui.buildinfohelper.BuildInfoHelper object at 0x7f881d2a8310>, 'console': <logging.StreamHandler object at 0x7f881d2a8210>, 'consolelogfile': u'/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/log/cooker/genericx86-64/20150831174419.log', 'curr': <traceback object at 0x7f881d24b7e8>, 'e': AttributeError("'module' object has no attribute 'ReachableStamps'",), 'error': None, 'errors': 0, 'event': <bb.event.ParseCompleted object at 0x7f881d242b10>, 'eventHandler': <__main__.EventReader instance at 0x7f881d231c20>, 'exc_type': <type 'exceptions.AttributeError'>, 'exc_value': AttributeError("'module' object has no attribute 'ReachableStamps'",), 'exception_data': 'Traceback (most recent call last):\n File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main\n event = eventHandler.waitEvent(0.25)\n File "/var/lib/bitbake/bin/toaster-eventreplay", line 139, in waitEvent\n return self._create_event(nextline)\n File "/var/lib/bitbake/bin/toaster-eventreplay", line 115, in _create_event\n event_class = _import_class(event_data[\'class\'])\n File "/var/lib/bitbake/bin/toaster-eventreplay", line 110, in _import_class\n return getattr(module, moduleklass)\nAttributeError: \'module\' object has no attribute \'ReachableStamps\'\n', 'first': False, 'format': <bb.msg.BBLogFormatter object at 0x7f881d2a8290>, 'format_str': '%(levelname)s: %(message)s', 'helper': <bb.ui.uihelper.BBUIHelper instance at 0x7f881d231cb0>, 'includelogs': u'yes', 'inheritlist': u' toaster report-error poky-sanity package_rpm package_deb package_ipk buildstats image-mklibs image-prelink debian devshell sstate license blacklist sanity', 'interrupted': False, 'loglines': None, 'params': <__main__.MockConfigParameters instance at 0x7f881d231c68>, 'pformat': <function pformat at 0x7f881e15fc80>, 'return_value': 0, 'server': <__main__.MockConnection instance at 0x7f881d405ef0>, 'taskfailures': [], 'tb': <traceback object at 0x7f881d24b7e8>, 'traceback': <module 'traceback' from '/usr/lib/python2.7/traceback.pyc'>, 'warnings': 0}
This keeps happening on all builds, AFAICT
Can you give some details about the environment and how/why Toaster is being used in it (I'm not very familiar with the autobuilder and there's not much about it on the Yocto wiki)? In particular, which version of Toaster is affected (the line numbers in the error don't tally with the version of Toaster I'm looking at on master)? It would also help if we had the file which Toaster is trying to load when the error occurs. Failing that, the line which caused the error (in my version of Toaster, this is printed around the time the error occurs).
For the record: Elizabeth Flanagan turned off the toaster upload for the moment so this is not critical right now. (In reply to comment #2) > Can you give some details about the environment and how/why Toaster is being > used in it (I'm not very familiar with the autobuilder and there's not much > about it on the Yocto wiki)? Belen might be able to comment (he seemed to be of opinion that it's not worth spending time on this right now).
I think this is blocked because the logs are too large to store the way they were. Are there alternate storage options now? Do we still want an autobuilder read-only toaster running?
We have #9884[1] which should result in a patch to make the generation of toaster event logs an option at build trigger time, defaulting to off. This way we can not generate the event logs for every build but allow whomever triggers the build to enable them when desirable. 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=9884
Resolve as duplicate of 9884 where we plan to implement changes to mitigate this issue. *** This bug has been marked as a duplicate of bug 9884 ***