Premise: I'm not sure if this should be for the autobuilder or some other component. Problem ======= Sometimes, especially when cloning and building, unmet dependency errors appear. Often these errors are not reproducible and restarting the build will temporarily solve the problem. Ex: we saw this error - Makefile:580: recipe for target 'install-dbusserviceDATA' failed when building gumd, but it went away after restarting the build. Analysis ======== This seems to be caused by incomplete recipes, which do not list all the dependency. So the build is prone to random failures, due to races between components: if component A depends on component B but it doesn't explicit such dependency, sometimes they build in a favourable order, sometimes they do not. The presence of various levels of caching tends to mask most of these problems, but they do occasionally surface, especially when the build infrastructure changes somehow. Factors that seem to mostly affect the outcome are: * number of CPUs available for parallel build * memory available to the build system * other loads on the build system * relative time taken for cloning sources and building them (if A depends implicitly on B, but B always takes less time to clone and build, it becomes unlikely that the missing dependency is spotted.) Proposal ======== The autobuilder should be used for periodic builds where no/partial cache/sstate is available and the parameters of the build system (CPU, memory, bandwidth for individual downloads, etc) are randomly changed across builds. This is obviously an empirical solution, however over time it would expose those packages that do not declare their dependency correctly. But any other solution that can expose these faults would be equally acceptable.
Btw, one more reason why I think this is important: whatever dependency is missing between 2 packages can have serious effects on the consistency of the cache, leaving as valid some packages which instead would be invalidated, if the dependency was correct.
FWIW, we do have scripts/test-dependencies.sh which does several builds to uncover undeclared dependencies - see the comments at the top of the file for details. This can take many hours to run, but is being run regularly by Martin Jansa on some of his autobuilders (not the Yocto Project Autobuilder).
From what I see, it seems that the script is too deterministic. The key of my proposal is to introduce significant randomness in the execution of the build (number of CPUs, memory footprint, bandwidth, even randomly nuking certain recipes), so that over time they would expose a larger amount of missing dependency that normally are hidden, due to the typical build sequence. This doesn't imply an increment in the number of test runs, it is an increment of the entropy of the build system, to add more variance to the typical build sequence.
To be honest I have never been a fan of random builds. Randomness means there is a chance that you will never find the issue you're looking for, and you will spend a load of time building things that will duplicate the results of the build testing that we already do. Within the core metadata I think our dependencies are in a fairly good state thanks to the dependency QA checks we have and regular running of test-dependencies.sh. If we want to solve this in a better way however, I think we should be trying to do it as deterministically as possible. There are a number of potential approaches: 1) Instrument the build process (perhaps at a low level) so we know where it's looking and what it has found. This is effectively what swabber was supposed to do for accesses to the host system only, although the implementation made the tool impractical to use. 2) Use recipe-specific sysroots - then there can never be anything more than the declared dependencies in the sysroot for a recipe's build process to find and use. Covered under bug 1560. 3) Look for configure script options reported by ./configure --help that are different to previous builds so we know when new optional dependency checks are introduced. I prototyped this and it might be interesting for knowing when we should add new PACKAGECONFIG options, but it won't solve the issue for optional dependencies that aren't exposed as configure script options (or custom configure processes, or dependencies that come in outside of the configure script).
(In reply to comment #4) > To be honest I have never been a fan of random builds. Randomness means > there is a chance that you will never find the issue you're looking for, and > you will spend a load of time building things that will duplicate the > results of the build testing that we already do. That's not the type of randomness I was proposing. But it's irrelevant. I described the problem and gave also a proposal. If you have a better way to solve the problem is solved, I'm totally cool with it. > Within the core metadata I think our dependencies are in a fairly good state > thanks to the dependency QA checks we have and regular running of > test-dependencies.sh. If we want to solve this in a better way however, I > think we should be trying to do it as deterministically as possible. The problem is that the build sequence is only partially affected by the explicit dependency. You do get other pseudo random factors affecting it: number of cores, memory, bandwidth, etc. So, currently, there is already lots of randomness built in the build process. > There > are a number of potential approaches: > > 1) Instrument the build process (perhaps at a low level) so we know where > it's looking and what it has found. This is effectively what swabber was > supposed to do for accesses to the host system only, although the > implementation made the tool impractical to use. > > 2) Use recipe-specific sysroots - then there can never be anything more than > the declared dependencies in the sysroot for a recipe's build process to > find and use. Covered under bug 1560. > > 3) Look for configure script options reported by ./configure --help that are > different to previous builds so we know when new optional dependency checks > are introduced. I prototyped this and it might be interesting for knowing > when we should add new PACKAGECONFIG options, but it won't solve the issue > for optional dependencies that aren't exposed as configure script options > (or custom configure processes, or dependencies that come in outside of the > configure script). The recipe-specific sysroot seems the more bullet-proof approach, as it will not provide what is not explicitly required. Even if it was a special mode with additional overhead, it would be ok to arrange for periodical special builds. But, as I said, as long as it works, everything is ok.
test-dependencies is exactly what you're after, because it attempts to build every recipe with a totally clean sysroot, so any undeclared dependencies are definitely not present. Also, at a personal level, when developing recipes the script "wipe-sysroot" can be used to efficiently clear the sysroot so you can then do a build to verify the build dependencies. Proposing to close this, between the existing tools (test-dependencies, wipe-sysroot, and developers actually using them for their own layers) and bug 1560 I think we've got this covered.
Closing, as I said earlier we have some tooling to help with most of this.