| Summary: | busybox parallel make race | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez> | ||||
| Component: | core | Assignee: | Juro Bystricky <juro.bystricky> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | High | CC: | meta.mr.watcher, meta.watcher, randy.macleod, ross.burton, sgw | ||||
| Version: | 2.2 | ||||||
| Target Milestone: | 2.2 M4 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Attachments: |
|
||||||
Created attachment 3349 [details]
Complete log taken from AB
Richard has been chasing this. Another bug that may be related: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10129 (In reply to comment #3) > Another bug that may be related: > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=10129 Seems these are not related. I've backported a patch from busybox upstream related to this. I'm still not 100% convinced this fixes all the issues though. There is a discussion on the busybox mailing list about this. Appeared in selftest despite the makefile fix being applied: https://autobuilder.yoctoproject.org/main/builders/nightly-oe-selftest/builds/643/steps/Running%20oe-selftest/logs/stdio The problem is fairly difficult to reproduce. I ran stress test several thousand times and reproduced the failure twice. On the positive side, it is always the same/identical error:
libbb/appletlib.c:164:13: error: 'NUM_APPLETS' undeclared (first use in this function)
while (i < NUM_APPLETS) {
^~~~~~~~~~~
Based on the error, the file NUM_APPLETS.h (where NUM_APPLETS is defined) does not exist at the time of the compilation of the file appletlib.c. (The file NUM_APPLETS.h is generated as needed)
The proposed fix is to explicitly specify the dependency in libbb/Kbuild.src:
-libbb/appletlib.o: include/usage_compressed.h
+libbb/appletlib.o: include/usage_compressed.h include/NUM_APPLETS.h
This can take a while to verify, as the absence of evidence != evidence of absence. First we need to make sure we can reliably reproduce the error, then
using the same stress test not observe the error.
Presently we are trying to enhance the stress test by running parallel builds in multiple Docker containers at the same time. (bitbake busybox -c compile -f)
(Also, looking at the bug, it does not seem to be poky-tiny specific).
This bug is discussed on the Busybox mailing list: https://www.mail-archive.com/busybox@busybox.net/msg23244.html The proposed fix did not solve the problem, if anything, the problem was exacerbated. This means there are two (or more) parallel targets, all of which need the fo generate the file NUM_APPLETS.h, but with different contents. (NUM_APPLETS.h is only re-generated when its contents are modified). I noticed there are some new patches related to this problem on the busybox git repo. I am going to test them. So far we ran stress test with the patch "parallel-make-fix.patch" at least 40,000 times without observing any failure.... (commit f755bab79246e129e60b71f20ea16f36393f40b2) Build is eventually broken for yocto commit f755bab79246e129e60b71f20ea16f36393f40b2.
Build is also eventually broken with Busybox GIT commits addressing the problem:
0dddbc1a59795a77679d8c5ef48a2795cb4705630dddbc1 ("build system: always rewrite NUM_APPLETS.h", 2016-08-23)d8e61bb
d8e61bbf13d0cf38d477255cfd5dc71c5d51d575 ("build system: different fix for include/applet_tables.h/include/NUM_APPLETS.h", 2016-08-21)
Always the same error observed.
Sent a patchset to the mailing list for a review: http://lists.openembedded.org/pipermail/openembedded-core/2016-September/126511.html Sent a patchset to the mailing list for a review: http://lists.openembedded.org/pipermail/openembedded-core/2016-September/126511.html (Also renamed the bug, as this is not musl-gcc nor poky-tiny specific bug but a generic busybox problem) Fixed in oe-core a866a05e2c7d090a77aa6e95339c93e3592703a6. |
There is a failure on the AB [1] (also log attached) related to busybox do_compile task using musl-gcc compile. This is one of the issues observed but full log review is pending | libbb/appletlib.c:164:13: error: 'NUM_APPLETS' undeclared (first use in this function) | while (i < NUM_APPLETS) { [1] https://autobuilder.yoctoproject.org/main/builders/poky-tiny/builds/908