| Summary: | External modules fail to build when kernel is reused from sstate | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Martin Jansa <Martin.Jansa> | ||||
| Component: | kernel | Assignee: | Martin Jansa <Martin.Jansa> | ||||
| Status: | RESOLVED FIXED | QA Contact: | Francisco Pedraza <francisco.j.pedraza.gonzalez> | ||||
| Severity: | normal | ||||||
| Priority: | Medium+ | CC: | jose.perez.carranza, richard.purdie, tom.zanussi | ||||
| Version: | unspecified | ||||||
| Target Milestone: | 2.2 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Attachments: |
|
||||||
|
Description
Martin Jansa
2016-03-29 19:48:16 UTC
Bouncing to Richard to re-assign. This is well over my head, and my hands are full with perf races, and queued kernel merges. I am not able to reproduce build failure mentioned here, but kernel rebuild is 100% reproducible. I have a theory about this. I think that linux-yocto-tiny-kexecboot (from http://git.openembedded.org/meta-openembedded/tree/meta-initramfs/recipes-kernel/linux/linux-yocto-tiny-kexecboot_4.4.bb) is pretty toxic and smashes the kernel shared source directory, then any further use of the directory is broken and for example kernel module builds fail. Could someone try "bitbake linux-yocto; bitbake linux-yocto-tiny-kexecboot; bitbake lttng-modules" and see if that gives the failure in question? I'm seeing these issues even after blacklisting linux-yocto-tiny(-kexecboot) in my world builds: https://github.com/shr-project/jenkins-jobs/commit/c168b7824c2c68bd5894532fc122929d2d9df9a7 This works just fine in my environment: rm -rf tmp/ ; bitbake linux-yocto; bitbake linux-yocto-tiny-kexecboot ; bitbake lttng-modules I didn't notice any build failures. I've noticed that linux-yocto was rebuilt when building lttng-modules though. Martin: Can you point at one of the world builds where this fails and linux-yocto-tiny-kexecboot is blacklisted please? *Something* is getting built which trashes the shared kernel workdir, but what that is, I don't know :/. A command which consistently shows this error is: rm tmp -rf; bitbake linux-yocto-tiny-kexecboot lttng-modules -c cleansstate; bitbake lttng-modules; bitbake lttng-modules -c cleansstate; bitbake linux-yocto-tiny-kexecboot ; bitbake lttng-modules Basically, "bitbake lttng-modules" forces the do_shared_workdir to run (which includes building the base part of the kernel since that isn't stored in sstate). The "bitbake linux-yocto-tiny-kexecboot" then destroys the shared_workdir, then the second build of the module fails as the shared workdir is corrupt. If this isn't in Martin's world builds, the log files of those would be useful as something is likely breaking that directory, we just need to figure out what. I confirm that this command breaks the build the way it's showed in the first comment: rm tmp -rf; bitbake linux-yocto-tiny-kexecboot lttng-modules -c cleansstate; bitbake lttng-modules; bitbake lttng-modules -c cleansstate; bitbake linux-yocto-tiny-kexecboot ; bitbake lttng-modules ... ERROR: lttng-modules-2.7.1+gitAUTOINC+45e0ebd91f-r0 do_make_scripts: Function failed: do_make_scripts (log file is located at /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/lttng-modules/2.7.1+gitAUTOINC+45e0ebd91f-r0/temp/log.do_make_scripts.17071) ERROR: Logfile of failure stored in: /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/lttng-modules/2.7.1+gitAUTOINC+45e0ebd91f-r0/temp/log.do_make_scripts.17071 Log data follows: | DEBUG: Executing shell function do_make_scripts | make: Entering directory '/home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-source' | make[1]: Entering directory '/home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-build-artifacts' | HOSTCC scripts/basic/fixdep | GEN ./Makefile | HOSTCC scripts/kconfig/conf.o | SHIPPED scripts/kconfig/zconf.tab.c | SHIPPED scripts/kconfig/zconf.lex.c | SHIPPED scripts/kconfig/zconf.hash.c | HOSTCC scripts/kconfig/zconf.tab.o | HOSTLD scripts/kconfig/conf | scripts/kconfig/conf --silentoldconfig Kconfig | *** | *** Configuration file ".config" not found! | *** | *** Please run some configurator (e.g. "make oldconfig" or | *** "make menuconfig" or "make xconfig"). | *** | /home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-source/scripts/kconfig/Makefile:37: recipe for target 'silentoldconfig' failed | make[3]: *** [silentoldconfig] Error 1 | /home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-source/Makefile:531: recipe for target 'silentoldconfig' failed | make[2]: *** [silentoldconfig] Error 2 | make[1]: *** No rule to make target 'include/config/auto.conf', needed by 'scripts'. Stop. | make[1]: Leaving directory '/home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-build-artifacts' | Makefile:146: recipe for target 'sub-make' failed | make: *** [sub-make] Error 2 | make: Leaving directory '/home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-source' | WARNING: /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/lttng-modules/2.7.1+gitAUTOINC+45e0ebd91f-r0/temp/run.do_make_scripts.17071:1 exit 2 from 'make CC="x86_64-poky-linux-gcc -fuse-ld=bfd" LD="x86_64-poky-linux-ld.bfd " AR="x86_64-poky-linux-ar " -C /home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-source O=/home/ed/git/yocto/poky/build/tmp/work-shared/qemux86-64/kernel-build-artifacts scripts' | ERROR: Function failed: do_make_scripts (log file is located at /home/ed/git/yocto/poky/build/tmp/work/qemux86_64-poky-linux/lttng-modules/2.7.1+gitAUTOINC+45e0ebd91f-r0/temp/log.do_make_scripts.17071) ERROR: Task 6 (/home/ed/git/yocto/poky/meta/recipes-kernel/lttng/lttng-modules_git.bb, do_make_scripts) failed with exit code '1' NOTE: Tasks Summary: Attempted 409 tasks of which 403 didn't need to be rerun and 1 failed. I don't have a recent log from jenkins build, I was re-testing this manually in the same workspace (which already had the PNBLACKLIST). I'll try to reproduce this again after current jenkins builds are finished. The tiny, tiny-kexecboot builds are definitely breaking things (even the build of the linux-yocto kernel itself), but the manual testing shown some issues even when they weren't built in the same TOPDIR - maybe it could be broken sstate for linux-yocto from previous build where tiny/tiny-kexecboot wasn't blacklisted yet. You shouldn't need to blacklist linux-yocto-tiny as that recipe is very different to linux-yocto-tiny-kexec. Notice that the latter zero's out PROVIDES. If PROVIDES isn't zero'ed, both would provide virtual/kernel and we'd see warnings/errors about multiple duplicate providers. The build is designed so only one kernel owns the shared workdir but that kexec recipe goes specifically out of its way to bypass the checks. I'd propose that linux-yocto-tiny is ok to be left enabled but that linux-yocto-tiny-exec needs rewriting. Closing to stop the e-mails. This seems to work as simple reproducer:
for i in linux-yocto netmap-modules lttng-modules kernel-devsrc cryptodev-module; do
echo "cleansstate $i"
bitbake -c cleansstate $i >>log.$i 2>&1
ls tmp-glibc/work-shared/qemux86/kernel-*
echo "build $i"
bitbake $i >>log.$i 2>&1
ls tmp-glibc/work-shared/qemux86/kernel-*
done
After cleansstate lttng-modules:
tmp-glibc/work-shared/qemux86/kernel-build-artifacts:
arch include kernel-abiversion Module.symvers System.map-4.4.11-yocto-standard
which is ok and expected, but then after building lttng-modules:
tmp-glibc/work-shared/qemux86/kernel-build-artifacts:
include Makefile Module.symvers scripts source
log.lttng-modules attached
Created attachment 3212 [details]
log.lttng-modules
|