Bug 3085 - Undefined event handler execution order can lead to unexpected behavior and even failures
Summary: Undefined event handler execution order can lead to unexpected behavior and e...
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.3
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2012-09-10 18:04 UTC by Christopher Larson
Modified: 2012-09-20 12:45 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments
compat, event: use OrderedDict from py2.7 for the event handlers (9.07 KB, patch)
2012-09-10 22:30 UTC, Christopher Larson
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Christopher Larson 2012-09-10 18:04:15 UTC
BitBake runs event handlers in an undefined order (likely determined by the recipe parse order, which is likely determined by the order files are found on disk, which means the order won't change from run to run normally, but the order is still undefined), rather than in either registration order or reverse registration order.

As a result of this, there's no way to ensure that one sanity check runs before or after another one, or one bit of ConfigParsed metadata alteration occurs before or after yours. As one example of a failure this causes, there's no way to ensure your ConfigParsed handler runs after oe_import, which means there's no way to know if oe.data has been imported, for example. As a result, many ConfigParsed handlers have to manually run oe_import(d), to hack around it, causing unnecessary work.

As another example of this sort of failure, we can't always cleanly fail out due to a missing toolchain, depending on the order in which the various toolchain event handlers run. For example, it may attempt to create the toolchain symlinks or extract the sysroot before we've confirmed that the toolchain even exists.

To fix this, bitbake should be altered to run events in registration order. I have a trivial patch implementing this (it'd be even cleaner if we required python 2.7, as we could use OrderedDict, of course), but it'll need hefty testing and buildhistory use to check for problematic event handlers. Of course, any handlers broken by that change could have broken at any time, and would just have gotten lucky regarding event handler execution order.
Comment 1 Richard Purdie 2012-09-10 20:59:40 UTC
In the interests of determinstic builds, we *need* to fix this, I'd say fairly urgently. I'd be tempted to include this in one of sgw's test builds on the autobuilder and call it good if we don't see problems.

Can you attach the patch (or mail it to the bitbake list)?
Comment 2 Christopher Larson 2012-09-10 22:30:52 UTC
Created attachment 782 [details]
compat, event: use OrderedDict from py2.7 for the event handlers

I was playing with a different version, but this is probably cleaner in the long run.
Comment 3 Richard Purdie 2012-09-20 12:45:56 UTC
I've merged this, non-determinism is something we need to avoid. http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=749f4f67444c7be7fe7aac0189776fd540975e09