Bug 8230

Summary: toaster upload failures on autobuilder
Product: [Build System, Metadata & Runtime] Toaster Reporter: Jussi Kukkonen <jku>
Component: toasterAssignee: Michael Halstead <mhalstead>
Status: RESOLVED DUPLICATE QA Contact: Libertad <libertad.gonzalez.de.la.cruz>
Severity: normal    
Priority: Medium CC: belen.barros.pena, elliot.smith, jessica.zhang, joshuagloe, stanciux.mihail
Version: unspecified   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Jussi Kukkonen 2015-09-01 09:06:05 UTC
(Not sure what the component should be, assigning to Michael on Pauls suggestion)


Here is an example of a failed UploadToasterEventlog task: looks like the upload stalls and then timeouts.

https://autobuilder.yoctoproject.org/main/builders/nightly-x86-lsb/builds/476/steps/UploadToasterEventlog/logs/stdio

My initial thought was that this was a transient network error but decided to look at a successful log as well. This didn't look much better:

https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/builds/483/steps/UploadToasterEventlog/logs/stdio


The upload seems to stall much the same way but instead of timeing out it spews
out a _lot_ of python tracebacks such as the ones below:



WARNING: buildhistory is not enabled. Please enable INHERIT += "buildhistory" to see image details.
NOTE: ToasterUI waiting for events
ERROR: 'module' object has no attribute 'ReachableStamps'
Traceback (most recent call last):
  File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main
    event = eventHandler.waitEvent(0.25)
  File "/var/lib/bitbake/bin/toaster-eventreplay", line 139, in waitEvent
    return self._create_event(nextline)
  File "/var/lib/bitbake/bin/toaster-eventreplay", line 115, in _create_event
    event_class = _import_class(event_data['class'])
  File "/var/lib/bitbake/bin/toaster-eventreplay", line 110, in _import_class
    return getattr(module, moduleklass)
AttributeError: 'module' object has no attribute 'ReachableStamps'

WARNING: Error data dump ['  File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main\n    event = eventHandler.waitEvent(0.25)\n']
{'build_history_enabled': False,
 'buildinfohelper': <bb.ui.buildinfohelper.BuildInfoHelper object at 0x7f881d2a8310>,
 'console': <logging.StreamHandler object at 0x7f881d2a8210>,
 'consolelogfile': u'/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/log/cooker/genericx86-64/20150831174419.log',
 'curr': <traceback object at 0x7f881d24b7e8>,
 'e': AttributeError("'module' object has no attribute 'ReachableStamps'",),
 'error': None,
 'errors': 0,
 'event': <bb.event.ParseCompleted object at 0x7f881d242b10>,
 'eventHandler': <__main__.EventReader instance at 0x7f881d231c20>,
 'exc_type': <type 'exceptions.AttributeError'>,
 'exc_value': AttributeError("'module' object has no attribute 'ReachableStamps'",),
 'exception_data': 'Traceback (most recent call last):\n  File "/var/lib/bitbake/lib/bb/ui/toasterui.py", line 116, in main\n    event = eventHandler.waitEvent(0.25)\n  File "/var/lib/bitbake/bin/toaster-eventreplay", line 139, in waitEvent\n    return self._create_event(nextline)\n  File "/var/lib/bitbake/bin/toaster-eventreplay", line 115, in _create_event\n    event_class = _import_class(event_data[\'class\'])\n  File "/var/lib/bitbake/bin/toaster-eventreplay", line 110, in _import_class\n    return getattr(module, moduleklass)\nAttributeError: \'module\' object has no attribute \'ReachableStamps\'\n',
 'first': False,
 'format': <bb.msg.BBLogFormatter object at 0x7f881d2a8290>,
 'format_str': '%(levelname)s: %(message)s',
 'helper': <bb.ui.uihelper.BBUIHelper instance at 0x7f881d231cb0>,
 'includelogs': u'yes',
 'inheritlist': u' toaster report-error poky-sanity package_rpm package_deb package_ipk buildstats image-mklibs image-prelink debian devshell sstate license blacklist sanity',
 'interrupted': False,
 'loglines': None,
 'params': <__main__.MockConfigParameters instance at 0x7f881d231c68>,
 'pformat': <function pformat at 0x7f881e15fc80>,
 'return_value': 0,
 'server': <__main__.MockConnection instance at 0x7f881d405ef0>,
 'taskfailures': [],
 'tb': <traceback object at 0x7f881d24b7e8>,
 'traceback': <module 'traceback' from '/usr/lib/python2.7/traceback.pyc'>,
 'warnings': 0}
Comment 1 Jussi Kukkonen 2015-09-02 08:15:06 UTC
This keeps happening on all builds, AFAICT
Comment 2 Elliot Smith 2015-09-03 13:25:29 UTC
Can you give some details about the environment and how/why Toaster is being used in it (I'm not very familiar with the autobuilder and there's not much about it on the Yocto wiki)?

In particular, which version of Toaster is affected (the line numbers in the error don't tally with the version of Toaster I'm looking at on master)?

It would also help if we had the file which Toaster is trying to load when the error occurs. Failing that, the line which caused the error (in my version of Toaster, this is printed around the time the error occurs).
Comment 3 Jussi Kukkonen 2015-09-03 14:00:12 UTC
For the record: Elizabeth Flanagan turned off the toaster upload for the moment so this is not critical right now.

(In reply to comment #2)
> Can you give some details about the environment and how/why Toaster is being
> used in it (I'm not very familiar with the autobuilder and there's not much
> about it on the Yocto wiki)?

Belen might be able to comment (he seemed to be of opinion that it's not worth spending time on this right now).
Comment 4 Michael Halstead 2016-07-13 00:59:47 UTC
I think this is blocked because the logs are too large to store the way they were. Are there alternate storage options now? Do we still want an autobuilder read-only toaster running?
Comment 5 Joshua Lock 2016-07-13 15:54:31 UTC
We have #9884[1] which should result in a patch to make the generation of toaster event logs an option at build trigger time, defaulting to off. This way we can not generate the event logs for every build but allow whomever triggers the build to enable them when desirable.


1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=9884
Comment 6 Joshua Lock 2016-07-13 21:29:01 UTC
Resolve as duplicate of 9884 where we plan to implement changes to mitigate this issue.

*** This bug has been marked as a duplicate of bug 9884 ***