3403 2012-11-09 07:46:39 +0000 bitbake parse does not handle } in the beginning of a function line correctly 2013-05-02 07:57:53 +0000 1 1 9 Documentation Reference handbook 1.4 x86 Multiple VERIFIED FIXED http://git.yoctoproject.org/cgit.cgi/yocto-docs/commit/?id=a1c0972083751444a8cc89aa8a026f273d98e23e 1 March 2013: In Progress Review Medium normal 1.4 M3 1 hongxu.jia srifenbark corneliux.stoicescu jessica.zhang kai.kang liezhi.yang poky.bs.watcher poky.doc.watcher poky.watcher Qi.Chen richard.purdie sgw yi.zhao zhenfeng.zhao --- --- --- oldest_to_newest 27319 0 hongxu.jia 2012-11-09 07:46:39 +0000 * Problem Define a function in a the recipe, such as meta/classes/populate_sdk_base.bbclass: fakeroot create_shar() { cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh usage() { echo "test" ######The below "}" causes error###### } EOF } There would be an error because of the "}" before the last EOF: <<Running result>> jiahongxu:build$ bitbake meta-toolchain Loading cache: 100% |###########################################| ETA: 00:00:00 Loaded 1117 entries from dependency cache. ERROR: ParseError at /home/jiahongxu/yocto/poky/meta/classes/popu late_sdk_base.bbclass:127: unparsed line: 'INST_ARCH=$(uname -m | sed -e "s/i[3-6]86/ix86/" -e "s/x86[-_]64/x86_64/")' ERROR: Command execution failed: Exited with 1 This is because the lib/bb/parse/parse_py/BBHandler.py treats the beginning "}" as the end of the function. * Resolution: 1) Check whether the "}" is the end of the function or not would fix the problem. See the patch below for details. 2) Another solution is that don't use the "}" at the beginning of the line inside the function, just keep some spaces before "}". diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 2e0647b..dcf45c6 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py @@ -46,6 +46,20 @@ __addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" ) __def_regexp__ = re.compile( r"def\s+(\w+).*:" ) __python_func_regexp__ = re.compile( r"(\s+.*)|(^$)" ) +# Match the shell function defined inside +# function(){ +# } +# +# or (the "{" in the next line) +# +# function() +# { +# } +__left_curly_bracket_def_func_regexp__ = re.compile( r"([\w]+\s*\(\s*\)\s*{\s*$)|({\s*$)" ) + +# Save the number of the curly bracket pairs which are used for inside +# sub functions. +__curly_bracket_pair_num__ = 0 __infunc__ = "" __inpython__ = False @@ -112,6 +126,7 @@ def handle(fn, d, include): __infunc__ = "" __classname__ = "" __residue__ = [] + __curly_bracket_pair_num__ = 0 if include == 0: @@ -165,16 +180,59 @@ def handle(fn, d, include): return d +# In function body, the following regexps should never be matched, +# if matched, the shellstr is invalid. +def check_shellstr_valid(shellstr): + m = __def_regexp__.match(shellstr) + if m: + return False + + m = __export_func_regexp__.match(shellstr) + if m: + return False + + m = __addtask_regexp__.match(shellstr) + if m: + return False + + m = __addhandler_regexp__.match(shellstr) + if m: + return False + + m = __inherit_regexp__.match(shellstr) + if m: + return False + return True + def feeder(lineno, s, fn, root, statements): - global __func_start_regexp__, __inherit_regexp__, __export_func_regexp__, __addtask_regexp__, __addhandler_regexp__, __def_regexp__, __python_func_regexp__, __inpython__, __infunc__, __body__, classes, bb, __residue__ + global __func_start_regexp__, __inherit_regexp__, __export_func_regexp__, __addtask_regexp__, __addhandler_regexp__, __def_regexp__, __python_func_regexp__, __inpython__, __infunc__, __body__, classes, bb, __residue__, __curly_bracket_pair_num__, __left_curly_bracket_def_func_regexp__ if __infunc__: - if s == '}': + + if not check_shellstr_valid(s): + return ConfHandler.feeder(lineno, s, fn, statements) + + # If matched, it means the "{" is the start of a new shell + # sub function inside the function body. + m = __left_curly_bracket_def_func_regexp__.match(s) + if m: + __curly_bracket_pair_num__ += 1 + + # If matched, it means the "}" is the end of a new shell + # sub function inside the function body. + elif s== '}' and __curly_bracket_pair_num__ > 0: + __curly_bracket_pair_num__ -= 1 + + # If matched, it means the "}" is the end of function body. + elif s == '}' and __curly_bracket_pair_num__ == 0: __body__.append('') ast.handleMethod(statements, fn, lineno, __infunc__, __body__) __infunc__ = "" __body__ = [] - else: - __body__.append(s) + return + + # Except the "}" which is the end of function body, all other + # shell string should be appended to the function body + __body__.append(s) return if __inpython__: diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass index dc715c4..baa0db8 100644 --- a/meta/classes/populate_sdk_base.bbclass +++ b/meta/classes/populate_sdk_base.bbclass @@ -117,7 +117,13 @@ fakeroot tar_sdk() { fakeroot create_shar() { cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh #!/bin/bash +usage() +{ +value=${pwd} +test(){ +} +} INST_ARCH=$(uname -m | sed -e "s/i[3-6]86/ix86/" -e "s/x86[-_]64/x86_64/") SDKMACHINE=$(echo ${SDKMACHINE} | sed -e "s/i[5-6]86/ix86/") -- 1.7.10.4 27763 1 richard.purdie 2012-11-28 08:08:34 +0000 To do this properly we'd effectively need to write a shell parser. Rather than writing our own, I'd much rather use the existing shell parsing code we have in the bitbake tree. The trouble is that if we get this wrong, we'll create subtle bugs where some syntax works but other valid shell syntax does not. I really don't want to get into this as I think we'll end up with more subtle bugs and a hugely complex parser for no real net gain. I therefore think at least for now, stating that you must use indentation is the way to proceed so we should document this problem. 27894 2 hongxu.jia 2012-12-03 12:10:08 +0000 I agree with Richard. We should explain this in document: In bitbake recipe files, indentation is the way to proceed. Here is the example: ----Example file example.bb fakeroot create_shar() { cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh usage() { echo "This is really a ugly resolution." } EOF } 28157 3 srifenbark 2012-12-11 20:56:07 +0000 Currently, there is a section in the dev-manual called "Adding a Package" (http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#usingpoky-extend-addpkg). I would like to add a note about this behaviour right up front where it talks about the user needing to write a recipe. Also, there is a helloworld exmaple in the section that I don't think follows the indentation we are going to suggest. What would the properly indented helloworld file look like? Scott 30171 4 corneliux.stoicescu 2013-02-27 15:03:57 +0000 Hongxu, can you please help Scott wrap up the documentation for this? Thanks! 30221 5 hongxu.jia 2013-02-28 08:01:36 +0000 I think it's better to emphasize an exception about the useage of curly brace in shell function. If use a pair of curly brace in shell function, the colse curly brace must not be located at the leading line(leave some spaves ahead). Such as: There is an exception about the useage of close curly brace in shell function: 1,Invalid------------------ fakeroot create_shar() { cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh usage() { echo "test" ######The below "}" at the leading line causes error###### } EOF } 2, Valid----------------- fakeroot create_shar() { cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh usage() { echo "test" ######The below "}" leaves some spaces is ok###### } EOF } 30346 6 srifenbark 2013-03-01 21:44:14 +0000 Added a note in the overview section describing how to add a package. The section talks about writing recipes. Please review the note at http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#usingpoky-extend-addpkg. Scott 32818 7 jessica.zhang 2013-05-01 15:58:00 +0000 In 1.4 official doc 32835 8 corneliux.stoicescu 2013-05-02 07:57:53 +0000 Verified.