Created attachment 4384 [details] put into classes directory If a shell function uses substitution contaning changing variables the metadata is not deterministic and bitbake yields following error messages: ERROR: When reparsing /home/frederik/src/git/poky/meta/recipes-connectivity/openssl/openssl_1.1.1.bb.do_foobar, the basehash value changed from 828eb1721ee1d58febebc7ab86d02778 to 06428c477fa2268fe4b68541f85e764b. The metadata is not deterministic and this needs to be fixed. ERROR: openssl-1.1.1-r0 do_foobar: Taskhash mismatch 09143e58ed1ef42a04311d69cd4a3a7b versus 36573c975a2aa5b6568e07169bdec6fd for /home/frederik/src/git/poky/meta/recipes-connectivity/openssl/openssl_1.1.1.bb.do_foobar ERROR: Taskhash mismatch 09143e58ed1ef42a04311d69cd4a3a7b versus 36573c975a2aa5b6568e07169bdec6fd for /home/frederik/src/git/poky/meta/recipes-connectivity/openssl/openssl_1.1.1.bb.do_foobar bitbake does not prohibit using special characters like %#% in variable names and above error message may not be obvious and hard to debug. Attached class can be used to reproduce the issue if inherited.
I have a fix for this implemented at https://github.com/openembedded/bitbake/compare/master...kergoth:yocto-12987, but I'm going to do a bit more profiling and performance testing to determine the impact, and may switch to a hasattr approach to avoid the set existing even when none of these were found, as they're relatively rare in our metadata.
After looking at this again, it become obvious that the current prototype implementation is far from ideal since it adds an extra wordlist-to-string, uses another set, and duplicates logic that's already in bitbake's own variable parsing. An easier method is to check for and block references based on the existence of certain parameter modifier characters in build_dependencies instead, but even that doesn't seem ideal, and we can't exclude all parameter modifiers, as some are valid in variable names. For example, we allow '-', but ${foo-bar} has a specific parameter expansion in shell, so that's not ideal either. I think an even better approach than that is actually just to stop trying to expand references to variable names which aren't valid bitbake variable names according to our own parser. That'd avoid adding references to names with #, %, etc, without adding extra logic beyond the regex. I'm testing and benchmarking that option now.
Reworked patch at https://github.com/openembedded/bitbake/compare/master...kergoth:yocto-12987-new, pending submission to the bitbake-devel list shortly, after I complete bitbake-selftest + oe-selftest -a and a bit more benchmarking.
Sent to bitbake-devel for review.