8230 2015-09-01 09:06:05 +0000 toaster upload failures on autobuilder 2016-07-13 21:29:01 +0000 1 1 7 Build System, Metadata & Runtime Toaster toaster unspecified x86 Multiple RESOLVED DUPLICATE 9884 Medium normal Future 1 jku mhalstead belen.barros.pena elliot.smith jessica.zhang joshuagloe stanciux.mihail libertad.gonzalez.de.la.cruz --- --- No (bug/feature does not impact docs) oldest_to_newest 53585 0 jku 2015-09-01 09:06:05 +0000 (Not sure what the component should be, assigning to Michael on Pauls suggestion) Here is an example of a failed UploadToasterEventlog task: looks like the upload stalls and then timeouts. https://autobuilder.yoctoproject.org/main/builders/nightly-x86-lsb/builds/476/steps/UploadToasterEventlog/logs/stdio My initial thought was that this was a transient network error but decided to look at a successful log as well. This didn't look much better: https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/builds/483/steps/UploadToasterEventlog/logs/stdio The upload seems to stall much the same way but instead of timeing out it spews out a _lot_ of python tracebacks such as the ones below: WARNING: buildhistory is not enabled. Please enable INHERIT += "buildhistory" to see image details. NOTE: ToasterUI waiting for events ERROR: 'module' object has no attribute 'ReachableStamps' Traceback (most recent call last): File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main event = eventHandler.waitEvent(0.25) File "/var/lib/bitbake/bin/toaster-eventreplay", line 139, in waitEvent return self._create_event(nextline) File "/var/lib/bitbake/bin/toaster-eventreplay", line 115, in _create_event event_class = _import_class(event_data['class']) File "/var/lib/bitbake/bin/toaster-eventreplay", line 110, in _import_class return getattr(module, moduleklass) AttributeError: 'module' object has no attribute 'ReachableStamps' WARNING: Error data dump [' File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main\n event = eventHandler.waitEvent(0.25)\n'] {'build_history_enabled': False, 'buildinfohelper': <bb.ui.buildinfohelper.BuildInfoHelper object at 0x7f881d2a8310>, 'console': <logging.StreamHandler object at 0x7f881d2a8210>, 'consolelogfile': u'/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/log/cooker/genericx86-64/20150831174419.log', 'curr': <traceback object at 0x7f881d24b7e8>, 'e': AttributeError("'module' object has no attribute 'ReachableStamps'",), 'error': None, 'errors': 0, 'event': <bb.event.ParseCompleted object at 0x7f881d242b10>, 'eventHandler': <__main__.EventReader instance at 0x7f881d231c20>, 'exc_type': <type 'exceptions.AttributeError'>, 'exc_value': AttributeError("'module' object has no attribute 'ReachableStamps'",), 'exception_data': 'Traceback (most recent call last):\n File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main\n event = eventHandler.waitEvent(0.25)\n File "/var/lib/bitbake/bin/toaster-eventreplay", line 139, in waitEvent\n return self._create_event(nextline)\n File "/var/lib/bitbake/bin/toaster-eventreplay", line 115, in _create_event\n event_class = _import_class(event_data[\'class\'])\n File "/var/lib/bitbake/bin/toaster-eventreplay", line 110, in _import_class\n return getattr(module, moduleklass)\nAttributeError: \'module\' object has no attribute \'ReachableStamps\'\n', 'first': False, 'format': <bb.msg.BBLogFormatter object at 0x7f881d2a8290>, 'format_str': '%(levelname)s: %(message)s', 'helper': <bb.ui.uihelper.BBUIHelper instance at 0x7f881d231cb0>, 'includelogs': u'yes', 'inheritlist': u' toaster report-error poky-sanity package_rpm package_deb package_ipk buildstats image-mklibs image-prelink debian devshell sstate license blacklist sanity', 'interrupted': False, 'loglines': None, 'params': <__main__.MockConfigParameters instance at 0x7f881d231c68>, 'pformat': <function pformat at 0x7f881e15fc80>, 'return_value': 0, 'server': <__main__.MockConnection instance at 0x7f881d405ef0>, 'taskfailures': [], 'tb': <traceback object at 0x7f881d24b7e8>, 'traceback': <module 'traceback' from '/usr/lib/python2.7/traceback.pyc'>, 'warnings': 0} 53615 1 jku 2015-09-02 08:15:06 +0000 This keeps happening on all builds, AFAICT 53682 2 elliot.smith 2015-09-03 13:25:29 +0000 Can you give some details about the environment and how/why Toaster is being used in it (I'm not very familiar with the autobuilder and there's not much about it on the Yocto wiki)? In particular, which version of Toaster is affected (the line numbers in the error don't tally with the version of Toaster I'm looking at on master)? It would also help if we had the file which Toaster is trying to load when the error occurs. Failing that, the line which caused the error (in my version of Toaster, this is printed around the time the error occurs). 53685 3 jku 2015-09-03 14:00:12 +0000 For the record: Elizabeth Flanagan turned off the toaster upload for the moment so this is not critical right now. (In reply to comment #2) > Can you give some details about the environment and how/why Toaster is being > used in it (I'm not very familiar with the autobuilder and there's not much > about it on the Yocto wiki)? Belen might be able to comment (he seemed to be of opinion that it's not worth spending time on this right now). 63904 4 mhalstead 2016-07-13 00:59:47 +0000 I think this is blocked because the logs are too large to store the way they were. Are there alternate storage options now? Do we still want an autobuilder read-only toaster running? 63925 5 joshuagloe 2016-07-13 15:54:31 +0000 We have #9884[1] which should result in a patch to make the generation of toaster event logs an option at build trigger time, defaulting to off. This way we can not generate the event logs for every build but allow whomever triggers the build to enable them when desirable. 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=9884 63949 6 joshuagloe 2016-07-13 21:29:01 +0000 Resolve as duplicate of 9884 where we plan to implement changes to mitigate this issue. *** This bug has been marked as a duplicate of bug 9884 ***