13375 2019-06-03 23:55:35 +0000 CONFIG_SITE is not consistent across do_configure / do_compile / do_install 2019-09-12 14:59:27 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.6.3 All Multiple RESOLVED FIXED Medium+ normal 2.8 M3 1 dan.callaghan ross.burton meta.mr.watcher meta.watcher randy.macleod --- --- Don't know oldest_to_newest 84051 0 dan.callaghan 2019-06-03 23:55:35 +0000 We noticed a strange problem when building one of our custom recipes. The autotools.bbclass does not fill in CONFIG_SITE consistently across different tasks. We have a custom lua recipe which uses a SYSROOT_PREPROCESS_FUNCS function to drop a config_site file into ${TARGET_SYS}_config_site.d with some autoconf cache variables prepopulated, to help the AX_PROG_LUA macro find lua interpreter paths without executing the target interpreter: SYSROOT_PREPROCESS_FUNCS += "do_siteconfig" do_siteconfig() { mkdir -p ${SYSROOT_DESTDIR}${datadir}/${TARGET_SYS}_config_site.d cat >${SYSROOT_DESTDIR}${datadir}/${TARGET_SYS}_config_site.d/lua_config <<EOF ax_cv_pathless_LUA=lua ax_cv_lua_version=5.1 ax_cv_lua_platform=unknown ax_cv_lua_luadir=${datadir}/lua/5.1 ax_cv_lua_luaexecdir=${libdir}/lua/5.1 EOF } Then we have a recipe for a proprietary package which depends on lua and uses AX_PROG_LUA in its configure.ac. The autotools.bbclass populates CONFIG_SITE based on what it finds in the sysroot, and it correctly sees the lua_config file in addition to all the others, when do_configure runs the ./configure script. configure: loading site script .../poky/meta/site/endian-little configure: loading site script .../poky/meta/site/common-linux configure: loading site script .../poky/meta/site/common-glibc configure: loading site script .../poky/meta/site/x86_64-linux configure: loading site script .../poky/meta/site/common configure: loading site script .../poky/../meta-openembedded/meta-networking/site/endian-little configure: loading site script build/tmp-glibc/work/puma-64-og-linux/ogconfig/1.0+git999-r0/recipe-sysroot/usr/share/x86_64-og-linux_config_site.d/glibc_config configure: loading site script build/tmp-glibc/work/puma-64-og-linux/ogconfig/1.0+git999-r0/recipe-sysroot/usr/share/x86_64-og-linux_config_site.d/lua_config configure: loading site script build/tmp-glibc/work/puma-64-og-linux/ogconfig/1.0+git999-r0/recipe-sysroot/usr/share/x86_64-og-linux_config_site.d/ncurses_config [...] So far so good. However, for local development of the proprietary package, we use externalsrc.bbclass and set EXTERNALSRC to point at a git checkout. The recipe has a native variant, so bitbake will build both $OURPACKAGE and $OURPACKAGE-native -- often concurrently. As a consequence, it can happen that bitbake will do autoreconf for one variant of the recipe while the other variant has already completed do_configure and has moved onto do_compile or do_install. The autoreconf happens in the same source tree, even though the build directories are separate. That means the configure script in the source tree will be touched by both recipes variants, possibly while each is building. automake has a rule to invoke ./config.status --recheck if the configure script changed since it was run. And so that rule can be triggered in do_compile or do_install for the concurrently running recipes. Normally ./config.status --recheck just does a quick run through the cache variables to ensure nothing has changed and then the build continues. And nothing should have changed because bitbake gives us a consistent build environment. But something *does* change: the CONFIG_SITE environment variable is different in do_compile and do_install. It doesn't include any files from ${TARGET_SYS}_config_site.d, and so ./config.status --recheck fails. The reason seems to be because CONFIG_SITE is populated in two different ways in autotools.bbclass. Globally there is: # Space separated list of shell scripts with variables defined to supply test # results for autoconf tests we cannot run at build time. export CONFIG_SITE = "${@siteinfo_get_files(d)}" This is calling siteinfo_get_files with sysrootcache=False which intentionally omits everything from ${TARGET_SYS}_config_site.d in the sysroot. The variable is populated differently for do_configure: do_configure[prefuncs] += "autotools_preconfigure autotools_aclocals ${EXTRACONFFUNCS}" [...] python autotools_aclocals () { # Refresh variable with cache files d.setVar("CONFIG_SITE", siteinfo_get_files(d, sysrootcache=True)) } Here it passes sysrootcache=True and so it correctly picks up files from ${TARGET_SYS}_config_site.d in the sysroot. The end result is that CONFIG_SITE is correct for do_configure but not for do_compile and do_install. 84052 1 4520 dan.callaghan 2019-06-03 23:59:16 +0000 Created attachment 4520 proposed patch I think this patch should fix the issue. It populates CONFIG_SITE the same for do_configure, do_compile, and do_install. But I'm not sure if it's the best solution. Are there any other tasks which might need CONFIG_SITE to be set? 84567 2 ross.burton 2019-08-01 15:18:28 +0000 Patch looks reasonable. Can you post it to the mailing list at openembedded-core@lists.openembedded.org? 84582 3 dan.callaghan 2019-08-05 01:16:14 +0000 Sure, posted: https://patchwork.openembedded.org/patch/163660/ 84940 4 ross.burton 2019-09-12 14:59:27 +0000 Merged in oe-core e60c170b451a4aa561d08bfce97dca05508c2106 4520 2019-06-03 23:59:16 +0000 2019-06-03 23:59:16 +0000 proposed patch 0001-autotools-always-include-config_site.d-files-in-CONF.patch text/plain 1773 dan.callaghan RnJvbSAyN2Q0MzE3ZWVkMWQxY2FhOTFmNmRiM2ViYzg3OTAxMWU0NzdjNDIzIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBEYW4gQ2FsbGFnaGFuIDxkYW4uY2FsbGFnaGFuQG9wZW5nZWFy LmNvbT4KRGF0ZTogVHVlLCA0IEp1biAyMDE5IDA5OjU3OjA1ICsxMDAwClN1YmplY3Q6IFtQQVRD SF0gYXV0b3Rvb2xzOiBhbHdheXMgaW5jbHVkZSBjb25maWdfc2l0ZS5kIGZpbGVzIGluIENPTkZJ R19TSVRFCgpGaXhlczogaHR0cHM6Ly9idWd6aWxsYS55b2N0b3Byb2plY3Qub3JnL3Nob3dfYnVn LmNnaT9pZD0xMzM3NQoKU2lnbmVkLW9mZi1ieTogRGFuIENhbGxhZ2hhbiA8ZGFuLmNhbGxhZ2hh bkBvcGVuZ2Vhci5jb20+Ci0tLQogbWV0YS9jbGFzc2VzL2F1dG90b29scy5iYmNsYXNzIHwgNyAr KysrKy0tCiAxIGZpbGUgY2hhbmdlZCwgNSBpbnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQoK ZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy9hdXRvdG9vbHMuYmJjbGFzcyBiL21ldGEvY2xhc3Nl cy9hdXRvdG9vbHMuYmJjbGFzcwppbmRleCA4NzY4YTZhZDY4Li4zZDIyYWQwMjU1IDEwMDY0NAot LS0gYS9tZXRhL2NsYXNzZXMvYXV0b3Rvb2xzLmJiY2xhc3MKKysrIGIvbWV0YS9jbGFzc2VzL2F1 dG90b29scy5iYmNsYXNzCkBAIC0yNSw3ICsyNSw5IEBAIGluaGVyaXQgc2l0ZWluZm8KIAogIyBT cGFjZSBzZXBhcmF0ZWQgbGlzdCBvZiBzaGVsbCBzY3JpcHRzIHdpdGggdmFyaWFibGVzIGRlZmlu ZWQgdG8gc3VwcGx5IHRlc3QKICMgcmVzdWx0cyBmb3IgYXV0b2NvbmYgdGVzdHMgd2UgY2Fubm90 IHJ1biBhdCBidWlsZCB0aW1lLgotZXhwb3J0IENPTkZJR19TSVRFID0gIiR7QHNpdGVpbmZvX2dl dF9maWxlcyhkKX0iCisjIFRoZSB2YWx1ZSBvZiB0aGlzIHZhcmlhYmxlIGlzIGZpbGxlZCBpbiBp biBhIHByZWZ1bmMgYmVjYXVzZSBpdCBkZXBlbmRzIG9uCisjIHRoZSBjb250ZW50cyBvZiB0aGUg c3lzcm9vdC4KK2V4cG9ydCBDT05GSUdfU0lURQogCiBhY3BhdGhzID89ICJkZWZhdWx0IgogRVhU UkFfQVVUT1JFQ09ORiA9ICItLWV4Y2x1ZGU9YXV0b3BvaW50IgpAQCAtMTMyLDYgKzEzNCw4IEBA IEVYVFJBQ09ORkZVTkNTID8/PSAiIgogRVhUUkFfT0VDT05GX2FwcGVuZCA9ICIgJHtQQUNLQUdF Q09ORklHX0NPTkZBUkdTfSIKIAogZG9fY29uZmlndXJlW3ByZWZ1bmNzXSArPSAiYXV0b3Rvb2xz X3ByZWNvbmZpZ3VyZSBhdXRvdG9vbHNfYWNsb2NhbHMgJHtFWFRSQUNPTkZGVU5DU30iCitkb19j b21waWxlW3ByZWZ1bmNzXSArPSAiYXV0b3Rvb2xzX2FjbG9jYWxzIgorZG9faW5zdGFsbFtwcmVm dW5jc10gKz0gImF1dG90b29sc19hY2xvY2FscyIKIGRvX2NvbmZpZ3VyZVtwb3N0ZnVuY3NdICs9 ICJhdXRvdG9vbHNfcG9zdGNvbmZpZ3VyZSIKIAogQUNMT0NBTERJUiA9ICIke1NUQUdJTkdfREFU QURJUn0vYWNsb2NhbCIKQEAgLTE0MCw3ICsxNDQsNiBAQCBBQ0xPQ0FMRVhUUkFQQVRIX2NsYXNz LXRhcmdldCA9ICIgLUkgJHtTVEFHSU5HX0RBVEFESVJfTkFUSVZFfS9hY2xvY2FsLyIKIEFDTE9D QUxFWFRSQVBBVEhfY2xhc3MtbmF0aXZlc2RrID0gIiAtSSAke1NUQUdJTkdfREFUQURJUl9OQVRJ VkV9L2FjbG9jYWwvIgogCiBweXRob24gYXV0b3Rvb2xzX2FjbG9jYWxzICgpIHsKLSAgICAjIFJl ZnJlc2ggdmFyaWFibGUgd2l0aCBjYWNoZSBmaWxlcwogICAgIGQuc2V0VmFyKCJDT05GSUdfU0lU RSIsIHNpdGVpbmZvX2dldF9maWxlcyhkLCBzeXNyb290Y2FjaGU9VHJ1ZSkpCiB9CiAKLS0gCjIu MjAuMQoK