Bug 9951

Summary: Toaster builds hang because of a runqueue exception in bitbake
Product: [Build System, Metadata & Runtime] Toaster Reporter: Elliot Smith <elliot.smith>
Component: toasterAssignee: Ed Bartosh <eduard.bartosh>
Status: RESOLVED FIXED QA Contact: Libertad <libertad.gonzalez.de.la.cruz>
Severity: normal    
Priority: Medium+ CC: belen.barros.pena, brian.avery, jessica.zhang, stanciux.mihail
Version: 2.3   
Target Milestone: 2.2 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Elliot Smith 2016-07-15 14:01:47 UTC
In toaster-next, I occasionally see errors which take this form:

--------------------------------------------
NOTE: Logging error 2: {'build': <Build: 2 Project object core-image-minimal>, 'message': 'An uncaught exception occured in runqueue', 'pathname': '/home/ell/dev/toaster/poky/bitbake/lib/bb/runqueue.py', 'level': 2, 'lineno': 1223}
ERROR: An uncaught exception occured in runqueue
Traceback (most recent call last):
  File "/home/ell/dev/toaster/poky/bitbake/lib/bb/runqueue.py", line 1210, in RunQueue.execute_runqueue():
             try:
    >            return self._execute_runqueue()
             except bb.runqueue.TaskFailure:
  File "/home/ell/dev/toaster/poky/bitbake/lib/bb/runqueue.py", line 1144, in RunQueue._execute_runqueue():
                     # needs it
    >                depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
                     self.rqdata.init_progress_reporter.next_stage()
  File "/home/ell/dev/toaster/poky/bitbake/lib/bb/cooker.py", line 741, in BBCooker.buildDependTree(rq=<bb.runqueue.RunQueue object at 0x7f049d83c4a8>, taskdata=<bb.taskdata.TaskData object at 0x7f049f786dd8>):
                     for ei in extra_info:
    >                    depend_tree["pn"][pn][ei] = vars(self.recipecache)[ei][fn]
     
KeyError: '/home/ell/dev/toaster/poky/meta/recipes-devtools/run-postinsts/run-postinsts_1.0.bb'
--------------------------------------------

While this looks like a bitbake error, it also causes the corresponding Toaster build to "hang": the most recent builds area doesn't update.

We may not be able to fix the bitbake error or even reproduce it reliably, but we can detect when exceptions occur at this point in the bitbake run and capture them properly in Toaster, so the build at least fails (and is reported as failed in the UI).
Comment 1 brian avery 2016-07-15 21:07:54 UTC
I believe this bug is a Toaster centric view of Bug #9902.
Just adding this note so that we know they are related.
Comment 2 Ed Bartosh 2016-07-22 15:26:16 UTC
Just wondering could this be fixed by this commit? http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip&id=5e9d53250e94cc0e782ca882bc90b0744d554fd6
Comment 3 brian avery 2016-07-22 16:14:09 UTC
Closing this since RP's no pickle interleave commit (referenced in Ed's commment) *should* fix this.