10668 2016-11-15 20:15:34 +0000 shell parser fails when using & for background processes 2017-11-28 17:02:00 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple VERIFIED FIXED https://patchwork.openembedded.org/series/3987/ Medium+ normal 2.3 M1 1 patrick.ohly patrick.ohly jose.perez.carranza oscarx.lopez.arandas poky.bs.watcher poky.watcher ross.burton oscarx.lopez.arandas --- --- No (bug/feature does not impact docs) oldest_to_newest 68335 0 patrick.ohly 2016-11-15 20:15:34 +0000 I was trying to do some simple parallel processing in a shell function when I noticed that codeparser.py has a problem with parsing the & To reproduce with poky rev 9b1bd29ea99aa: diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass index 3cff08e..c5545f5 100644 --- a/meta/classes/useradd.bbclass +++ b/meta/classes/useradd.bbclass @@ -53,7 +53,7 @@ if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1 | sed -e 's#[ \t]*$##'` remaining=`echo "$GROUPADD_PARAM" | cut -d ';' -f 2- | sed -e 's#[ \t]*$##'` while test "x$opts" != "x"; do - perform_groupadd "$SYSROOT" "$OPT $opts" + perform_groupadd "$SYSROOT" "$OPT $opts" & wait $! if test "x$opts" = "x$remaining"; then break fi Set up a build dir, run bitbake -g core-image-minimal => WARNING: /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb: Exception during build_dependencies for useradd_preinst WARNING: /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb: Error during finalise of /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb ERROR: Unable to parse /work/poky/meta/recipes-support/user-creation/xuser-account_0.1.bb Traceback (most recent call last): File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('if_clause', <bb.pysh.pyshyacc.IfCond object at 0x7fb480963390>)]): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480949278>): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('while_clause', <bb.pysh.pyshyacc.WhileLoop object at 0x7fb480949518>)]): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480446668>): if more_tokens: > self.process_tokens(more_tokens) File "/work/poky/bitbake/lib/bb/codeparser.py", line 393, in ShellParser.process_tokens(tokens=[[('pipeline', <bb.pysh.pyshyacc.Pipeline object at 0x7fb480949c88>)]]): for token in tokens: > name, value = token try: ValueError: need more than 1 value to unpack "token" in this case happened to be a list containing a tuple with the two expected values. Here's a crude patch which avoids the problem: diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py index 25938d6..41484ec 100644 --- a/bitbake/lib/bb/codeparser.py +++ b/bitbake/lib/bb/codeparser.py @@ -390,7 +390,14 @@ class ShellParser(): } for token in tokens: - name, value = token + try: + name, value = token + except ValueError: + if isinstance(token, list) and len(token) == 1: + bb.warn('Unexpected token (need list or tuple with two entries): %s' % token) + name, value = token[0] + else: + bb.fatal('Cannot split %s into name and value (need list or tuple with two entries)' % token) try: more_tokens, words = token_handlers[name](value) except KeyError: But to me that looks like a workaround. The root cause is probably in the code which generated the token(s). 68355 1 patrick.ohly 2016-11-16 09:06:39 +0000 Handling a shell command like echo foo | sed -u 's/^/bar: /' & wait $! required a slightly more complex workaround. In that case, the offending token is a list of three tuples. Here's the updated workaround which handles that. Changing indention makes it fairly large. The basic idea is that if a token is a list, process_token_list will be called recursively for the members of that list. diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py index 41484ec..8673b77 100644 --- a/bitbake/lib/bb/codeparser.py +++ b/bitbake/lib/bb/codeparser.py @@ -389,25 +389,24 @@ class ShellParser(): "case_clause": case_clause, } - for token in tokens: - try: + def process_token_list(tokens): + for token in tokens: + if isinstance(token, list): + process_token_list(token) + continue name, value = token - except ValueError: - if isinstance(token, list) and len(token) == 1: - bb.warn('Unexpected token (need list or tuple with two entries): %s' % token) - name, value = token[0] - else: - bb.fatal('Cannot split %s into name and value (need list or tuple with two entries)' % token) - try: - more_tokens, words = token_handlers[name](value) - except KeyError: - raise NotImplementedError("Unsupported token type " + name) + try: + more_tokens, words = token_handlers[name](value) + except KeyError: + raise NotImplementedError("Unsupported token type " + name) + + if more_tokens: + self.process_tokens(more_tokens) - if more_tokens: - self.process_tokens(more_tokens) + if words: + self.process_words(words) - if words: - self.process_words(words) + process_token_list(tokens) def process_words(self, words): """Process a set of 'words' in pyshyacc parlance, which includes 68416 2 patrick.ohly 2016-11-17 16:56:23 +0000 I'll post my preliminary patch on the bitbake mailing list and ask for help there. 68906 3 ross.burton 2016-12-07 16:55:40 +0000 Merged in bitbake d18a74de9ac75ba32f84c40620ca9d47c1ef96a3. 78373 4 oscarx.lopez.arandas 2017-11-28 17:02:00 +0000 to reproduce with poky rev 2.3.1 rc2 6bd890d9e011014cf323e61267f8b256949d44aa the issue this resolved. Set up a build dir, run bitbake -g core-image-minimal => oscar@oscar:~/quemux/build$ bitbake -g core-image-minimal Parsing recipes: 100% |######################################################################################################################| Time: 0:00:25 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 48 skipped, 0 masked, 0 errors. NOTE: There are 5 recipes to be removed from sysroot x86_64, removing... NOTE: There are 1 recipes to be removed from sysroot allarch, removing... NOTE: There are 5 recipes to be removed from sysroot i586, removing... NOTE: There are 7 recipes to be removed from sysroot qemux86, removing... NOTE: Resolving any missing task queue dependencies NOTE: PN build list saved to 'pn-buildlist' NOTE: Task dependencies saved to 'task-depends.dot' NOTE: Flatened recipe dependencies saved to 'recipe-depends.dot' oscar@oscar:~/quemux/build$