*problem $ bitbake libcap ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to `audit_open' .../lib64/libpam.so: undefined reference to `audit_close' ...... The libaudit has been available and libpam build with it, after we run another build with previous sstate-cache and without libaudit, the error occurs. *analysis The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues. Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs. *solution - Get all available package names, name the list 'pn_list'. - Get each/specific package's DEPENDS. - Run 'configure --help' on each/specific package and collect probable depends from 'enable/disable/with/without' options, such as '--disable-shadow'-->'shadow'; '--with-pam'-->'pam'. - If the probable depend is in 'pn_list' and not in DEPENDS, we should prompt the user to convert them into PACKAGECONFIG specs. The probable depends may be not exactly match the package name. Such as 'ssl' collected from '--enable-ssl' is not exactly match 'openssl' in oe-core, so we should support partly match. But if partly match is supported, the result will be not accurate, such as 'db' collected from '--enable-db' obviously doesn't belong to package 'dbus-ptest'. In this situation, we have to use human eyes to fillter out the noise. Here is a 'exactly match' example: $ test-dependencies.py libpam Analysis 'configure --help' of libpam PACKAGECONFIG[db] = "--enable-db, --disable-db, db" PACKAGECONFIG[xauth] = "--with-xauth, --without-xauth, xauth" Here is a 'partly match' example: $ test-dependencies.py -s libpam Analysis 'configure --help' of libpam PACKAGECONFIG[shared-mime-info] = "--enable-shared, --disable-shared, shared-mime-info" PACKAGECONFIG[psqlodbc] = "--enable-db, --disable-db, psqlodbc" PACKAGECONFIG[dbench] = "--enable-db, --disable-db, dbench" PACKAGECONFIG[gdb] = "--enable-db, --disable-db, gdb" PACKAGECONFIG[python-gitdb] = "--enable-db, --disable-db, python-gitdb" PACKAGECONFIG[xauth] = "--with-xauth, --without-xauth, xauth" PACKAGECONFIG[xrdb] = "--enable-db, --disable-db, xrdb" PACKAGECONFIG[mysql-connector-odbc] = "--enable-db, --disable-db, mysql-connector-odbc" PACKAGECONFIG[gdbm] = "--enable-db, --disable-db, gdbm" PACKAGECONFIG[picocom] = "--with-pic, --without-pic, picocom" PACKAGECONFIG[dbus-glib] = "--enable-db, --disable-db, dbus-glib" PACKAGECONFIG[libunistring] = "--enable-nis, --disable-nis, libunistring" PACKAGECONFIG[dbus-wait] = "--enable-db, --disable-db, dbus-wait" PACKAGECONFIG[packagegroup-core-eclipse-debug] = "--enable-debug, --disable-debug, packagegroup-core-eclipse-debug" PACKAGECONFIG[libdbi-drivers] = "--enable-db, --disable-db, libdbi-drivers" PACKAGECONFIG[packagegroup-wr-core-db] = "--enable-db, --disable-db, packagegroup-wr-core-db" PACKAGECONFIG[packagegroup-core-kdbg-tune] = "--enable-db, --disable-db, packagegroup-core-kdbg-tune" PACKAGECONFIG[dialog-static] = "--enable-static, --disable-static, dialog-static" PACKAGECONFIG[unixodbc] = "--enable-db, --disable-db, unixodbc" PACKAGECONFIG[dbus-daemon-proxy] = "--enable-db, --disable-db, dbus-daemon-proxy" PACKAGECONFIG[python-dbus] = "--enable-db, --disable-db, python-dbus" PACKAGECONFIG[db] = "--enable-db, --disable-db, db" PACKAGECONFIG[dbus-ptest] = "--enable-db, --disable-db, dbus-ptest" PACKAGECONFIG[vala-dbus-binding-tool] = "--enable-db, --disable-db, vala-dbus-binding-tool" PACKAGECONFIG[mobile-broadband-provider-info] = "--enable-db, --disable-db, mobile-broadband-provider-info" PACKAGECONFIG[libmodbus] = "--enable-db, --disable-db, libmodbus" PACKAGECONFIG[gdb-cross-canadian-arm] = "--enable-db, --disable-db, gdb-cross-canadian-arm" PACKAGECONFIG[libdbi] = "--enable-db, --disable-db, libdbi" PACKAGECONFIG[gdb-cross] = "--enable-db, --disable-db, gdb-cross" PACKAGECONFIG[dbus] = "--enable-db, --disable-db, dbus" PACKAGECONFIG[chrpath] = "--enable-rpath, --disable-rpath, chrpath" PACKAGECONFIG[packagegroup-core-tools-debug] = "--enable-debug, --disable-debug, packagegroup-core-tools-debug" PACKAGECONFIG[acpica] = "--with-pic, --without-pic, acpica"
Have you looked at Martin's script called test-dependencies.sh
"Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs." Note that doesn't catch all optional dependencies, many configure scripts will check for the presence of a library and use it if available without offering an explicit enable/disable option. (this is where Martin's script is useful, because the only way to find these checks is to either rebuild everything or manually audit the configure.ac)
The imcomplete DEPENDS issue It includes two parts: One part is 'missing explicit DEPENDS' which means the package is needed by the given package but not explicit in DEPENDS. There must be a failed build if the needed package doesn't exist in the sysroot. Martin's test-dependencies.sh is a good solution to figure out the missing DEPENDS by comparing and catching build failure after a 3-times building. The other part is 'missing optional DEPENDS' which means the package is optional needed and detected by the configure's test. If the configure script finds the optional package is available, the optional package will be included in the build for the given package. Whether available or not, there will be no build failure. But it causes two aspects problem: - The configure test result of given package is affected by the random build ordering with multiple threads, which means it is affected by the unstable sysroot at that test time. That's why PACKAGECONFIG is used, it explicitly figures out the condition whether an optional DEPENDS is needed or not. - As the previous defect described, when multiple builds share a common sstate-cache, the given package's configure test has been done in the firt build, and not do again in the following build even though the sysroot is different from the first build. As you guys talked about, the prevous 'Way to audit all optional dependencies' solution is not sufficent. And based on Marthin's test-dependencies.sh, I have another idea to share: - Build world repeatedly untill a complete and statble sysroot has been created, and dump the output of each recipes's configure, we name it as OUTPUT_CONFIG1 - Clear the prevous build and build the specific/each recipe once, and dump the output of recipes's configure, we name it as OUTPUT_CONFIG2 - Compare OUTPUT_CONFIG1 and OUTPUT_CONFIG2 to find what options checking have been changed. Keys: - In step 1: the flag of 'the creation of a complete and statble sysroot' is each recipe's configure output is no longer changed than previous build, in this situation we could stop the repeated world build.
Coincidentally I had a completely independent todo list item to store the ./configure --help output in buildhistory so you could compare it. I think I even have a POC implementation around somewhere.
Now that recipe-specific sysroots have been implemented I think this is much less of an issue than it was, but the ideas about automatically generating PACKAGECONFIG and noticing changes in optional dependencies do still make sense as potential enhancements.
Agreed that recipe-specific sysroots solves the non-deterministic problem. As to automatic generation, note that recipetool sort of does this by e.g. finding packages that are dependencies in PKG_CHECK_MODULES() calls. I'm not convinced that automatically generating PACKAGECONFIG from the output of --help is feasible: the semantics of what the option does, and whether it needs to be on, and whether the option is eg --with-glib when our recipe is glib-2.0 means that human review of the available options is needed anyway.
Closing, devtool should be able to determine dependencies on its own. If it can't then devtool should be improved where it's feasible, but "parse --help to infer dependencies" is basically an unsolvable problem.