Bug 10787 - Resolve autobuilder exceptions due to finishing steps before they've started
Summary: Resolve autobuilder exceptions due to finishing steps before they've started
Status: VERIFIED FIXED
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.3 M2
Assignee: Joshua Lock
QA Contact: Juan Ramos
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-12-12 11:36 UTC by Joshua Lock
Modified: 2017-06-21 18:53 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joshua Lock 2016-12-12 11:36:27 UTC
Several buildsteps in the yocto-autobuilder call self.finished() and/or self.build.stopBuild() *before* the Command.start() has been called.

This results in an exception which doesn't (noticeably) have an effect on the build but does clutter the autobuilders logs, making it harder to find errors which are affecting builds.

Example of the traceback:

2016-12-12 03:09:35+0000 [-] BuildStep.startStep exception in .start
2016-12-12 03:09:35+0000 [-] BuildStep.failed; traceback follows
        Traceback (most recent call last):
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 551, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1187, in unwindGenerator
            return _inlineCallbacks(None, gen, Deferred())
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1045, in _inlineCallbacks
            result = g.send(result)
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/process/buildstep.py", line 605, in _startStep_3
            result = yield defer.maybeDeferred(self.start)
        --- <exception caught here> ---
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 134, in maybeDeferred
            result = f(*args, **kw)
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/autobuilder/buildsteps/ResolveLayerHead.py", line 57, in start
            self.command = cmd
        exceptions.UnboundLocalError: local variable 'cmd' referenced before assignment

2016-12-12 03:09:35+0000 [-] exception during failure processing
        Traceback (most recent call last):
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1187, in unwindGenerator
            return _inlineCallbacks(None, gen, Deferred())
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1043, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/process/buildstep.py", line 610, in _startStep_3
            self.failed(Failure())
        --- <exception caught here> ---
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/process/buildstep.py", line 686, in failed
            self.progress.finish()
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/progress.py", line 85, in finish
            self.buildProgress.stepFinished(self.name)
          File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/progress.py", line 165, in stepFinished
            assert(stepname not in self.finishedSteps)
        exceptions.AssertionError:
Comment 1 Joshua Lock 2016-12-20 13:57:46 UTC
BuildStep.finished()[1] states "The step should not perform any additional processing after calling this method." yet many of our build steps call finished() before they call start()[2]

1. http://docs.buildbot.net/0.8.8/developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.finished
2. http://docs.buildbot.net/0.8.8/developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.start
Comment 2 Joshua Lock 2016-12-20 15:18:00 UTC
I've identified a few patterns to avoid calling start() after finished() has been called without significantly refactoring the affected buildsteps.
Comment 3 Juan Ramos 2017-06-21 18:53:03 UTC
Asked to Joshua, he said it's ok to change this bug to Verified.