Bug 1782

Summary: Enhance bitbake's parse error handling to indicate where the failure occured
Product: [Build System, Metadata & Runtime] BitBake Reporter: Darren Hart <dvhart>
Component: bitbakeAssignee: Paul Eggleton <bluelightning>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium CC: bluelightning, poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: ---   
Hardware: All   
OS: Multiple   
Whiteboard: Patch merged (12 Mar)
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Darren Hart 2011-11-22 11:37:16 UTC
I recently was refactoring a recipe and moved some logic from one function to another (along with several other changes). Bitbake would fail with a message like:

NOTE: Error expanding variable do_bootimg
ERROR: Unable to parse /home/dvhart/source/poky/meta/recipes-rt/images/core-image-rt.bb

I didn't modify core-image-rt.bb, nor anything it inherited|required|included directly. A few layers down it pulled in bootimg.bbclass, which I had modified. I spent a good deal of time trying to manipulate the do_bootimg assignments I made to no avail. In the end the problem was that I moved a shell fragment into a python function.

So while this was clearly developer error, I suspect it isn't all that uncommon given that we have a healthy mix of shell of python in many recipes. If bitbake could at least point to the file and function where it encountered a problem this would be a huge benefit to debugging.

I imagine this is non-trivial as there is plenty of post processing of variables and other such things that go one. Perhaps the debug flags could force a python syntax validation pass on the individual functions which could catch this sort of thing.
Comment 1 Richard Purdie 2012-01-26 12:38:24 UTC
Darren, I don't suppose you have the actual change that triggered this do you?
Comment 2 Darren Hart 2012-01-26 14:02:53 UTC
(In reply to comment #1)
> Darren, I don't suppose you have the actual change that triggered this do you?

Sorry, I don't. I should have included that. I believe adding any shell-syntax statement into a python function should readily duplicate the result though.
Comment 3 Richard Purdie 2012-02-26 12:15:42 UTC
I think http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ebc0d4252a19e6ab38052473e54138d29b172dfc will have addressed this problem, it does in other variable expansion error cases.
Comment 4 Darren Hart 2012-02-29 19:28:21 UTC
Another case where this happens is here:

$ bitbake linux-yocto -e 
ERROR: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure for URL: 'git:///home/dvhart/source/linux/linux-yocto-3.2.git;protocol=file;nocheckout=1;branch=yocto/standard/fri2,meta;name=machine,meta'. The command git ls-remote file:///home/dvhart/source/linux/linux-yocto-3.2.git yocto/standard/fri2 gave empty output unexpectedly
ERROR: Command execution failed: Exited with 1
NOTE: Error expanding variable do_patch

Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

I'm trying to setup the fri2 to use linux-yocto-3.2 and am using meta-kernel-dev, which replaces the SRC URI in a non-version specific way. I believe the above is failing because it is still trying to use linux-yocto-3.0, but I can't search for PREFERRED_PROVIDER because it fails before getting there. Printing out which recipe it's working on when it fails would help a lot.
Comment 5 Paul Eggleton 2012-03-02 15:42:14 UTC
Reopening as this clearly needs more attention.
Comment 6 Paul Eggleton 2012-03-06 23:57:27 UTC
I've sent a trivial fix to bitbake-devel for this issue:
http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-March/002111.html
Comment 7 Paul Eggleton 2012-03-12 10:16:12 UTC
This is now fixed in master revision d599d90fd56a7ad12a376011453513d874b512de - an additional NOTE is logged upon error containing the recipe file name.