12987 2018-11-07 11:43:25 +0000 Remove variables containing shell substitutions from vardeps of shell function if it cannot be expanded 2019-01-23 16:36:38 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED FIXED Medium+ normal 2.7 M2 1 frederik.sdun kergoth kergoth poky.bs.watcher poky.watcher --- --- Don't know oldest_to_newest 82046 0 4384 frederik.sdun 2018-11-07 11:43:25 +0000 Created attachment 4384 put into classes directory If a shell function uses substitution contaning changing variables the metadata is not deterministic and bitbake yields following error messages: ERROR: When reparsing /home/frederik/src/git/poky/meta/recipes-connectivity/openssl/openssl_1.1.1.bb.do_foobar, the basehash value changed from 828eb1721ee1d58febebc7ab86d02778 to 06428c477fa2268fe4b68541f85e764b. The metadata is not deterministic and this needs to be fixed. ERROR: openssl-1.1.1-r0 do_foobar: Taskhash mismatch 09143e58ed1ef42a04311d69cd4a3a7b versus 36573c975a2aa5b6568e07169bdec6fd for /home/frederik/src/git/poky/meta/recipes-connectivity/openssl/openssl_1.1.1.bb.do_foobar ERROR: Taskhash mismatch 09143e58ed1ef42a04311d69cd4a3a7b versus 36573c975a2aa5b6568e07169bdec6fd for /home/frederik/src/git/poky/meta/recipes-connectivity/openssl/openssl_1.1.1.bb.do_foobar bitbake does not prohibit using special characters like %#% in variable names and above error message may not be obvious and hard to debug. Attached class can be used to reproduce the issue if inherited. 82176 1 kergoth 2018-11-12 15:47:58 +0000 I have a fix for this implemented at https://github.com/openembedded/bitbake/compare/master...kergoth:yocto-12987, but I'm going to do a bit more profiling and performance testing to determine the impact, and may switch to a hasattr approach to avoid the set existing even when none of these were found, as they're relatively rare in our metadata. 82809 2 kergoth 2019-01-16 18:58:46 +0000 After looking at this again, it become obvious that the current prototype implementation is far from ideal since it adds an extra wordlist-to-string, uses another set, and duplicates logic that's already in bitbake's own variable parsing. An easier method is to check for and block references based on the existence of certain parameter modifier characters in build_dependencies instead, but even that doesn't seem ideal, and we can't exclude all parameter modifiers, as some are valid in variable names. For example, we allow '-', but ${foo-bar} has a specific parameter expansion in shell, so that's not ideal either. I think an even better approach than that is actually just to stop trying to expand references to variable names which aren't valid bitbake variable names according to our own parser. That'd avoid adding references to names with #, %, etc, without adding extra logic beyond the regex. I'm testing and benchmarking that option now. 82810 3 kergoth 2019-01-16 20:46:01 +0000 Reworked patch at https://github.com/openembedded/bitbake/compare/master...kergoth:yocto-12987-new, pending submission to the bitbake-devel list shortly, after I complete bitbake-selftest + oe-selftest -a and a bit more benchmarking. 82829 4 kergoth 2019-01-18 16:47:56 +0000 Sent to bitbake-devel for review. 4384 2018-11-07 11:43:25 +0000 2018-11-08 17:01:40 +0000 put into classes directory shell-substitution.bbclass text/plain 154 frederik.sdun TVlfRk9PQkFSUyA9ICJmb28tJHtEQVRBVElNRX0gYmFyLSR7REFURVRJTUV9Igpkb19mb29iYXIo KSB7CiAgICBmb3IgZiBpbiAke01ZX0ZPT0JBUlN9IDsgZG8gCiAgICAgICAgYmFzZT0iJHtmIy0k e0RBVEFUSU1FfX0iCiAgICBkb25lCn0KCmFkZHRhc2sgZm9vYmFyCg==