In the sanity.bbclass, the event: bb.event.SanityCheck is setup to call the 'check_sanity_eventhandler' In this event handler it does a series of activities, and if a reparse is necessary does: reparse = check_sanity(sanity_data) e.data.setVar("BB_INVALIDCONF", reparse) However, if you look at the bitbake code in bitbake/lib/bb/cookerdata.py: bb.parse.init_parser(self.basedata) self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles) ... bb.event.fire(bb.event.ConfigParsed(), self.data) if self.data.getVar("BB_INVALIDCONF", False) is True: self.data.setVar("BB_INVALIDCONF", False) self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles) The event 'SanityCheck' is never used in this context. So the BB_INVALIDCONF setting is never set or acted upon. The following test case shows this. Create a new bbclass and include it in your project. It will should that if SanityCheck is used, the new file is never read. However, if switched to ConfigParsed, it works as you would expect. include conf/wrtemplates.conf WRTEMPLATES = "template/foo template/bar" # Check if we need to reprocess the templates addhandler wrl_template_processing_eventhandler #wrl_template_processing_eventhandler[eventmask] = "bb.event.ConfigParsed" wrl_template_processing_eventhandler[eventmask] = "bb.event.SanityCheck" python wrl_template_processing_eventhandler () { if e.data.getVar("WRTEMPLATES", True) != e.data.getVarFlag("WRTEMPLATES", 'manual', True): f = open(os.path.join(e.data.getVar('TOPDIR', True), 'conf/wrtemplates.conf'), 'w') f.write('WRTEMPLATES[manual] = "%s"\n' % (e.data.getVar("WRTEMPLATES", True))) f.close() e.data.setVar("BB_INVALIDCONF", '1') } addhandler wrl_banner_eventhandler wrl_banner_eventhandler[eventmask] = "bb.event.ParseStarted bb.event.BuildStarted" python wrl_banner_eventhandler () { bb.plain("WRTEMPLATES: %s" % e.data.getVar("WRTEMPLATES", True)) bb.plain("WRTEMPLATES[manual]: %s" % e.data.getVarFlag("WRTEMPLATES", 'manual', True)) }
There is also a likely bitbake bug here as well -- when bitbake handles the BB_INVALIDCONF reparse, it is not sending a new 'ConfigParsed' event. Various components of OE, such as base.bbclass use the ConfigParsed event to manipulate various settings. I believe once reparsed, a new event should be sent. Also in the existing sanity.bbclass, it is unconditionally setting the BB_INVALIDCONF to the value of 'reparse'. This could clear any setting from another sanity event handler. (I have verified this is not the cause of the failure by only setting the value of 'reparse' is != 0.)
Looking at the git log, it seems that re-parsing through sanity got broken by this commit in oe-core: commit 97108a5647f9278280c923ef69d2b0b945a26eef Author: Richard Purdie <richard.purdie@linuxfoundation.org> Date: Wed Mar 26 15:09:06 2014 +0000 sanity.bbclass: Update against bitbake sanity event changes I'm not sure how to fix this, though. One obvious fix is to add bb.event.ConfigParsed to check_sanity_eventhandler[eventmask], as it was. In this casethe same sanity check would be run twice (once for ConfigParsed event and once for SanityCheck event) which isn't very beatiful but that shouldn't be a huge problem. Similar, and uglyish, solution would be to fire SanityCheck at the same time when ConfigParsed is fired. Another solution would be splitting the sanity check in two parts, the other listening to ConfigParsed and the other listening to SanityCheck event. The former would basically only check (or do checks up to) sanity_check_conffiles() which is the only place where reparse is instructed. I don't understand the bitbake code paths well enough to say if doing a full re-parse in BBCooker.updateCache() when BB_INVALIDCONF is set would be a viable option. Any ideas?? Last, as far as I can tell, you're correct about re-sending 'ConfigParsed'.
I believe the way to fix this is to break the sanity event handling into two parts. One part for the actual sanity event handling. (Checking versions, etc.) One part for the configuration file 'rewrite'. If the rewrite happens only in the ConfigParsed step, then it can set the BB_INVALIDCONF when complete. (Also the BB_INVALIDCONF should only be 'set to 1' if needed, but should never be reset to 0 inside of the event handlers... This is because we can have multiple event handlers, all of which may need to trigger a reparse.) I'm also wondering if inside of bitbake it should loop over BB_INVALIDCONF being set. This way if there is a 'series' of upgrade steps required, or something that could cause multiple reloads to be needed it could do this. (An infinite loop check should be added so if something goes wrong, the system can stop after so many reloads and tell the user something is wrong...)
Now, with fresh eyes, I agree with you. Probably the best thing to do (that I can think of, now) is to split it into two parts. I was thinking about this looping yesterday, too, and I think it makes sense. You're also right about unconditionally setting BB_INVALIDCONF to whatever value.
Patches sent for review: https://patchwork.openembedded.org/patch/131649/ https://patchwork.openembedded.org/series/2727/
Fixed in oe-core: http://git.openembedded.org/openembedded-core/commit/?id=8fda70bb74f7c63d393d5424436d034d2cc6c05e and bitbake: http://git.openembedded.org/bitbake/commit/?id=ec1c951a4ee0c33acdde29e578f79ad719a34aca http://git.openembedded.org/bitbake/commit/?id=07a03a1290fd206df2b40ffc28381b5b3c10ba4a