8849 2015-12-22 10:44:31 +0000 do_rootfs error reporting of Python exceptions and variable expansion 2016-03-07 14:11:49 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 2.1 x86 Multiple VERIFIED FIXED Medium+ normal 2.1 M2 1 patrick.ohly markus.lehtonen cristina-danielax.agurida poky.bs.watcher poky.watcher richard.purdie cristina-danielax.agurida --- --- No (bug/feature does not impact docs) oldest_to_newest 57424 0 patrick.ohly 2015-12-22 10:44:31 +0000 This issue is about two separate problems, triggered the same way, so I am only reporting this once. Steps to reproduce: * use current Poky master * set up a normal build with oe-init-build-env * add to local.conf one line: INITRD = "${@'${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE}-${MACHINE}.cpio.gz' if d.getVar('INITRD_IMAGE', True) else ''}" * bitbake core-image-minimal Expected result: * INITRD should be empty and building the image should succeed. Actual result: * Creating the image fails without any useful error description: | NOTE: ###### Generate images ####### | NOTE: Executing mklibs_optimize_image ... | DEBUG: Executing shell function mklibs_optimize_image | DEBUG: Shell function mklibs_optimize_image finished | NOTE: Executing prelink_image ... | DEBUG: Executing shell function prelink_image | Size before prelinking 6860. | Size after prelinking 6948. | DEBUG: Shell function prelink_image finished | NOTE: The image creation groups are: [['ext4', 'tar']] | DEBUG: Python function do_rootfs finished | ERROR: Function failed: do_rootfs ERROR: Task 7 (/work/poky/meta/recipes-core/images/core-image-minimal.bb, do_rootfs) failed with exit code '1' NOTE: Tasks Summary: Attempted 2004 tasks of which 2003 didn't need to be rerun and 1 failed. * There's a parse error for INITRD: $ bitbake -e core-image-minimal | grep INITRD ... # expansion of INITRD threw ExpansionError: Failure expanding variable INITRD, expression was ${@'/tmp/test/tmp/deploy/images/qemux86/${INITRD_IMAGE}-qemux86.cpio.gz' if d.getVar('INITRD_IMAGE', True) else ''} which triggered exception SyntaxError: EOL while scanning string literal (INITRD, line 1) The parse error is also what breaks do_rootfs because INITRD gets expanded when writing the wic environment file. ----------------------------------- My expectation regarding the INITRD expansion is that the expression above is evaluated as Python as-is (with ${INITRD_IMAGE} inside the first string); it is valid Python code. I ran into this while working on a image creation class where INITRD must be set depending on the value of INITRD_IMAGE. I did something similar in the OE-core image-vm.bbclass, but there I had set a default value INITRD_IMAGE and thus did not run into the problem. I'm just not sure anymore whether setting a default for INITRD_IMAGE in an image class which might not even be in use is the right thing to do, so now I tried checking for a non-zero value with d.getVar(). 57434 1 patrick.ohly 2015-12-22 16:54:04 +0000 Another expected result: * If there is an error like this, it should get logged. 58270 2 markus.lehtonen 2016-01-27 13:35:20 +0000 I'm not able to reproduce this. Neither on poky (master) nor oe-core (master). Do you still see this? 58272 3 patrick.ohly 2016-01-27 13:57:41 +0000 bitbake commit d80d39e73223a50fda0090784303d2c57167bb4c avoids the SyntaxError. To reproduce the incomplete error reporting during a build, revert the commit above and try the original steps. It does not make the INITRD expansion work as expected, though, because it simply stops expanding. RP and I agreed that this would require more intrusive changes to bitbake's data_smart.py; there's no bug open for it. 58490 4 markus.lehtonen 2016-02-03 11:39:53 +0000 I tried reverting d80d39e73223a50fda0090784303d2c57167bb4c. Bitbake now seems to correctly exit with an error. With the steps you described below, I get: ~/yocto/poky/build $ bitbake core-image-minimal WARNING: Exception during build_dependencies for INITRD | ETA: 00:00:19 WARNING: Error during finalise of /home/marquiz/yocto/poky/meta/recipes-core/images/core-image-minimal-initramfs.bb WARNING: Exception during build_dependencies for INITRD WARNING: Error during finalise of /home/marquiz/yocto/poky/meta/recipes-core/images/core-image-minimal-mtdutils.bb WARNING: Exception during build_dependencies for INITRD WARNING: Error during finalise of /home/marquiz/yocto/poky/meta/recipes-core/images/core-image-base.bb ERROR: ExpansionError during parsing /home/marquiz/yocto/poky/meta/recipes-core/images/core-image-minimal-initramfs.bb: Failure expanding variable INITRD, expression was ${@'/home/marquiz/yocto/poky/build-8849.2/tmp/deploy/images/qemux86/${INITRD_IMAGE}-qemux86.cpio.gz' if d.getVar('INITRD_IMAGE', True) else ''} which triggered exception SyntaxError: EOL while scanning string literal (INITRD, line 1) WARNING: Exception during build_dependencies for INITRD WARNING: Error during finalise of /home/marquiz/yocto/poky/meta/recipes-core/images/core-image-minimal-dev.bb WARNING: Exception during build_dependencies for INITRD WARNING: Error during finalise of /home/marquiz/yocto/poky/meta/recipes-core/images/core-image-minimal.bb Summary: There were 10 WARNING messages shown. Summary: There was 1 ERROR message shown, returning a non-zero exit code. I submitted a patchset to fix the problem with INITRD variable parsing: http://lists.openembedded.org/pipermail/bitbake-devel/2016-February/007005.html 59063 5 patrick.ohly 2016-02-16 08:42:14 +0000 I think the current bitbake behavior of not expanding this particular INITRD value is acceptable. It is more obvious than the previous EOL syntax error. The error reporting also seems to be fixed. 59065 6 richard.purdie 2016-02-16 08:53:52 +0000 Bitbake change d80d39e73223a50fda0090784303d2c57167bb4c handles the first part of the issue here, the expansion errors no longer raise exceptions, the unexpanded value is just returned which is in keeping with the way the rest of bitbake works. With the change to non-expanded python functions, the expression in question can be used as a function (anonyous or otherwise) and its only the inline version which would be more potentially problematic. Fixing the inline python expansion would mean significant changes to the parsing code to properly handle all the different kinds of nested expressions and would come with a performance penalty. After discussion with Patrick, we both feel this wouldn't be worthwhile at this point. The one piece I finally worried about was errors in the wicenv function not propagating. I tested this with: def foobar(d): bb.warn("Here 1") if d.getVar("BB_CURRENTTASK", True) == "rootfs_wicenv": bb.warn("Here 2") raise bb.build.FuncFailed return "" INITRD = "${@foobar(d)}" which correctly fails during do_rootfs_wicenv, I suspect the recent task re-org there has helped improve the robustness of the error handling paths. I therefore believe this issue can be marked resolved. 59066 7 richard.purdie 2016-02-16 08:54:13 +0000 Resolved as fixed as per comments. 59757 8 cristina-danielax.agurida 2016-03-07 14:11:49 +0000 Verified with master: bd3a1d5bddeab2b327055a61e85423303327fc09