Bug 3177 - String literal parser too greedy
Summary: String literal parser too greedy
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 1.3
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.4
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2012-09-26 14:26 UTC by Ross Burton
Modified: 2012-10-04 10:27 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ross Burton 2012-09-26 14:26:31 UTC
When the parser parses a string literal, it appears to do so greedily.

For example, consider the following line in my local.conf:

EXTRA_IMAGE_FEATURES = "debug-tweaks" # dev-pkgs dbg-pkgs tools-sdk"

I'd have thought that would set EXTRA_IMAGE_FEATURES to "debug-tweaks" and the rest of a line is a comment.

(removing the trailing quote results in a parse error, so apparently comments have to be on their own line)

But if you run bitbake -e, it actually parsed:

EXTRA_IMAGE_FEATURES="debug-tweaks\" # dev-pkgs dbg-pkgs tools-sdk"

That is not what I expected!
Comment 1 Richard Purdie 2012-09-26 17:08:35 UTC
Well, yes.

We can make the parser error for this with something like:

__config_regexp__  = re.compile( r"(?P<exp>export\s*)?(?P<var>[a-zA-Z0-9\-_+.${}/]+)(\[(?P<flag>[a-zA-Z0-9\-_+.]+)\])?\s*((?P<colon>:=)|(?P<lazyques>\?\?=)|(?P<ques>\?=)|(?P<append>\+=)|(?P<prepend>=\+)|(?P<predot>=\.)|(?P<postdot>\.=)|=)\s*(?P<apo>['\"])(?!.*(?P=apo).*(?P=apo)$)(?P<value>.*)(?P=apo)$")

in ConfParser.py. (Addition of the negative lookahead assertion (?!.*(?P=apo).*(?P=apo)$) ).

The trouble is this will cause parse failures for things like:

MACHINE_FEATURES_append = "${@oe.utils.features_backfill("MACHINE_FEATURES",d)}"

since there are four " characters in there. We could try and spot unbalanced quotes but my regexp foo is failing for that at the moment.
Comment 2 Richard Purdie 2012-09-26 17:25:45 UTC
We could detect single extra quotes with:

__config_regexp__  = re.compile( r"(?P<exp>export\s*)?(?P<var>[a-zA-Z0-9\-_+.${}/]+)(\[(?P<flag>[a-zA-Z0-9\-_+.]+)\])?\s*((?P<colon>:=)|(?P<lazyques>\?\?=)|(?P<ques>\?=)|(?P<append>\+=)|(?P<prepend>=\+)|(?P<predot>=\.)|(?P<postdot>\.=)|=)\s*(?!'[^']*'[^']*'$)(?!\"[^\"]*\"[^\"]*\"$)(?P<apo>['\"])(?P<value>.*)(?P=apo)$")

which would at least catch the most common error with the addition of two lookahead assertions: (?!'[^']*'[^']*'$)(?!\"[^\"]*\"[^\"]*\"$)
Comment 3 Richard Purdie 2012-10-02 13:09:42 UTC
I merged http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib?id=b85c30bb7d47dd8e9bea0e11029c0b08466603b0 which at least catches the one odd quote problem. Its less than ideal but probably as good as we can fix things right now.
Comment 4 Richard Purdie 2012-10-04 10:27:42 UTC
Marking this as resolved since I think we've done what we can for this.