I was trying to do some simple parallel processing in a shell function when I noticed that codeparser.py has a problem with parsing the & To reproduce with poky rev 9b1bd29ea99aa: diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass index 3cff08e..c5545f5 100644 --- a/meta/classes/useradd.bbclass +++ b/meta/classes/useradd.bbclass @@ -53,7 +53,7 @@ if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1 | sed -e 's#[ \t]*$##'` remaining=`echo "$GROUPADD_PARAM" | cut -d ';' -f 2- | sed -e 's#[ \t]*$##'` while test "x$opts" != "x"; do - perform_groupadd "$SYSROOT" "$OPT $opts" + perform_groupadd "$SYSROOT" "$OPT $opts" & wait $! if test "x$opts" = "x$remaining"; then break fi Set up a build dir, run bitbake -g core-image-minimal => WARNING: /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb: Exception during build_dependencies for useradd_preinst WARNING: /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb: Error during finalise of /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb ERROR: Unable to parse /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb Traceback (most recent call last): File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('if_clause', <bb.pysh.pyshyacc.IfCond object at 0x7fb480963390>)]): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480949278>): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('while_clause', <bb.pysh.pyshyacc.WhileLoop object at 0x7fb480949518>)]): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480446668>): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 393, in ShellParser.process_tokens(tokens=[[('pipeline', <bb.pysh.pyshyacc.Pipeline object at 0x7fb480949c88>)]]): for token in tokens: > name, value = token try: ValueError: need more than 1 value to unpack "token" in this case happened to be a list containing a tuple with the two expected values. Here's a crude patch which avoids the problem: diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py index 25938d6..41484ec 100644 --- a/bitbake/lib/bb/codeparser.py +++ b/bitbake/lib/bb/codeparser.py @@ -390,7 +390,14 @@ class ShellParser(): } for token in tokens: - name, value = token + try: + name, value = token + except ValueError: + if isinstance(token, list) and len(token) == 1: + bb.warn('Unexpected token (need list or tuple with two entries): %s' % token) + name, value = token[0] + else: + bb.fatal('Cannot split %s into name and value (need list or tuple with two entries)' % token) try: more_tokens, words = token_handlers[name](value) except KeyError: But to me that looks like a workaround. The root cause is probably in the code which generated the token(s).
Handling a shell command like echo foo | sed -u 's/^/bar: /' & wait $! required a slightly more complex workaround. In that case, the offending token is a list of three tuples. Here's the updated workaround which handles that. Changing indention makes it fairly large. The basic idea is that if a token is a list, process_token_list will be called recursively for the members of that list. diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py index 41484ec..8673b77 100644 --- a/bitbake/lib/bb/codeparser.py +++ b/bitbake/lib/bb/codeparser.py @@ -389,25 +389,24 @@ class ShellParser(): "case_clause": case_clause, } - for token in tokens: - try: + def process_token_list(tokens): + for token in tokens: + if isinstance(token, list): + process_token_list(token) + continue name, value = token - except ValueError: - if isinstance(token, list) and len(token) == 1: - bb.warn('Unexpected token (need list or tuple with two entries): %s' % token) - name, value = token[0] - else: - bb.fatal('Cannot split %s into name and value (need list or tuple with two entries)' % token) - try: - more_tokens, words = token_handlers[name](value) - except KeyError: - raise NotImplementedError("Unsupported token type " + name) + try: + more_tokens, words = token_handlers[name](value) + except KeyError: + raise NotImplementedError("Unsupported token type " + name) + + if more_tokens: + self.process_tokens(more_tokens) - if more_tokens: - self.process_tokens(more_tokens) + if words: + self.process_words(words) - if words: - self.process_words(words) + process_token_list(tokens) def process_words(self, words): """Process a set of 'words' in pyshyacc parlance, which includes
I'll post my preliminary patch on the bitbake mailing list and ask for help there.
Merged in bitbake d18a74de9ac75ba32f84c40620ca9d47c1ef96a3.
to reproduce with poky rev 2.3.1 rc2 6bd890d9e011014cf323e61267f8b256949d44aa the issue this resolved. Set up a build dir, run bitbake -g core-image-minimal => oscar@oscar:~/quemux/build$ bitbake -g core-image-minimal Parsing recipes: 100% |######################################################################################################################| Time: 0:00:25 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: There are 5 recipes to be removed from sysroot x86_64, removing... NOTE: There are 1 recipes to be removed from sysroot allarch, removing... NOTE: There are 5 recipes to be removed from sysroot i586, removing... NOTE: There are 7 recipes to be removed from sysroot qemux86, removing... NOTE: Resolving any missing task queue dependencies NOTE: PN build list saved to 'pn-buildlist' NOTE: Task dependencies saved to 'task-depends.dot' NOTE: Flatened recipe dependencies saved to 'recipe-depends.dot' oscar@oscar:~/quemux/build$