Bug 3177

Summary:	String literal parser too greedy
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Ross Burton <ross.burton>
Component:	bitbake	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium	CC:	jessica.zhang, poky.bs.watcher, poky.watcher
Version:	1.3
Target Milestone:	1.4
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	---

Description Ross Burton 2012-09-26 14:26:31 UTC

When the parser parses a string literal, it appears to do so greedily.

For example, consider the following line in my local.conf:

EXTRA_IMAGE_FEATURES = "debug-tweaks" # dev-pkgs dbg-pkgs tools-sdk"

I'd have thought that would set EXTRA_IMAGE_FEATURES to "debug-tweaks" and the rest of a line is a comment.

(removing the trailing quote results in a parse error, so apparently comments have to be on their own line)

But if you run bitbake -e, it actually parsed:

EXTRA_IMAGE_FEATURES="debug-tweaks\" # dev-pkgs dbg-pkgs tools-sdk"

That is not what I expected!

Comment 1 Richard Purdie 2012-09-26 17:08:35 UTC

Well, yes.

We can make the parser error for this with something like:

__config_regexp__  = re.compile( r"(?P<exp>export\s*)?(?P<var>[a-zA-Z0-9\-_+.${}/]+)(\[(?P<flag>[a-zA-Z0-9\-_+.]+)\])?\s*((?P<colon>:=)|(?P<lazyques>\?\?=)|(?P<ques>\?=)|(?P<append>\+=)|(?P<prepend>=\+)|(?P<predot>=\.)|(?P<postdot>\.=)|=)\s*(?P<apo>['\"])(?!.*(?P=apo).*(?P=apo)$)(?P<value>.*)(?P=apo)$")

in ConfParser.py. (Addition of the negative lookahead assertion (?!.*(?P=apo).*(?P=apo)$) ).

The trouble is this will cause parse failures for things like:

MACHINE_FEATURES_append = "${@oe.utils.features_backfill("MACHINE_FEATURES",d)}"

since there are four " characters in there. We could try and spot unbalanced quotes but my regexp foo is failing for that at the moment.

Comment 2 Richard Purdie 2012-09-26 17:25:45 UTC

We could detect single extra quotes with:

__config_regexp__  = re.compile( r"(?P<exp>export\s*)?(?P<var>[a-zA-Z0-9\-_+.${}/]+)(\[(?P<flag>[a-zA-Z0-9\-_+.]+)\])?\s*((?P<colon>:=)|(?P<lazyques>\?\?=)|(?P<ques>\?=)|(?P<append>\+=)|(?P<prepend>=\+)|(?P<predot>=\.)|(?P<postdot>\.=)|=)\s*(?!'[^']*'[^']*'$)(?!\"[^\"]*\"[^\"]*\"$)(?P<apo>['\"])(?P<value>.*)(?P=apo)$")

which would at least catch the most common error with the addition of two lookahead assertions: (?!'[^']*'[^']*'$)(?!\"[^\"]*\"[^\"]*\"$)

Comment 3 Richard Purdie 2012-10-02 13:09:42 UTC

I merged http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib?id=b85c30bb7d47dd8e9bea0e11029c0b08466603b0 which at least catches the one odd quote problem. Its less than ideal but probably as good as we can fix things right now.

Comment 4 Richard Purdie 2012-10-04 10:27:42 UTC

Marking this as resolved since I think we've done what we can for this.