Bug 6818

Summary: Kernel srcs are not cleaned up when doing ccleansstate
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Khem Raj <raj.khem>
Component: kernelAssignee: Darren Hart <dvhart>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: bruce.ashfield, dvhart, richard.purdie, tom.zanussi
Version: 1.7   
Target Milestone: 1.8   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Khem Raj 2014-10-07 22:38:48 UTC
I am seeing now

ERROR: The recipe stblinux is trying to install files into a shared area when those files already exist. Those files and their manifest lo
cation are:
   /home/ubuntu/work/dizzy/build-pacexi3v2/tmp/sysroots/pacexi3v2/usr/src/kernel/include/config/auto.conf.cmd
   /home/ubuntu/work/dizzy/build-pacexi3v2/tmp/sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/.conf.cmd
   /home/ubuntu/work/dizzy/build-pacexi3v2/tmp/sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/.conf.o.cmd
Please verify which recipe should provide the above files.
The build has stopped as continuing in this scenario WILL break things, if not now, possibly in the future (we've seen builds fail several
 months later). If the system knew how to recover from this automatically it would however there are several different scenarios which can
 result in this and we don't know which one this is. It may be you have switched providers of something like virtual/kernel (e.g. from lin
ux-yocto to linux-yocto-dev), in that case you need to execute the clean task for both recipes and it will resolve this error. It may be y
ou changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning those recipes should again resolve this error however switching DI
STRO_FEATURES on an existing build directory is not supported, you should really clean out tmp and rebuild (reusing sstate should be safe)
. It could be the overlapping files detected are harmless in which case adding them to SSTATE_DUPWHITELIST may be the correct solution. It
 could also be your build is including two different conflicting versions of things (e.g. bluez 4 and bluez 5 and the correct solution for
 that would be to resolve the conflict. If in doubt, please ask on the mailing list, sharing the error and filelist above.
ERROR: Function failed: sstate_task_postfunc


Then I see that I have external kernel modules so it builds scripts folder of kernel src which is fine but it does not get cleaned up. Following files still remains even after bitbake -ccleansstate virtual/kernel

./sysroots/pacexi3v2/usr/src/
./sysroots/pacexi3v2/usr/src/kernel
./sysroots/pacexi3v2/usr/src/kernel/include
./sysroots/pacexi3v2/usr/src/kernel/include/config
./sysroots/pacexi3v2/usr/src/kernel/include/config/auto.conf.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/.conf.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/zconf.tab.o
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/.conf.o.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/zconf.tab.c
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/zconf.lex.c
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/conf
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/conf.o
./sysroots/pacexi3v2/usr/src/kernel/scripts/kconfig/zconf.hash.c
./sysroots/pacexi3v2/usr/src/kernel/scripts/kallsyms
./sysroots/pacexi3v2/usr/src/kernel/scripts/conmakehash
./sysroots/pacexi3v2/usr/src/kernel/scripts/.conmakehash.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/modpost.o
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/modpost
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/mk_elfconfig
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/.modpost.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/.mk_elfconfig.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/.elfconfig.h.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/.modpost.o.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/file2alias.o
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/empty.o
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/sumversion.o
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/.file2alias.o.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/.empty.o.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/elfconfig.h
./sysroots/pacexi3v2/usr/src/kernel/scripts/mod/.sumversion.o.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/.kallsyms.cmd
./sysroots/pacexi3v2/usr/src/kernel/scripts/basic
./sysroots/pacexi3v2/usr/src/kernel/scripts/basic/fixdep
./sysroots/pacexi3v2/usr/src/kernel/scripts/basic/.fixdep.cmd
ubuntu@nebula:~/work/dizzy/build-pacexi3v2/tmp$ 


Whats interesting is that .conf.o.cmd also belongs kernel-dev too
now when kernel is installed from sstate then it tries to reinstall .conf.o.cmd
but its already there.
Comment 1 Bruce Ashfield 2014-10-10 01:38:36 UTC
FWIW. This has been around for a while. I've been hitting this for 2 releases
(if my memory is correct) with perf builds when switching kernels. Build for
one kernel version, change the preferred version to another. boom. You have
to manually clean the sysroot and then force it to be populated again to build.
Comment 2 Richard Purdie 2014-10-21 13:45:50 UTC
Note that in this case, these looks like the files we generate in the sysroot, its not all the files, just the scripts/* ones. These are generated in the sysroot when we "make scripts" or whatever it is there for other modules.

So somehow we need to find a way to better track and remove these. The trouble is they're not understate control.
Comment 3 Richard Purdie 2015-01-08 21:01:39 UTC
The current code changes mean we'll have two clear directories of kernel source and build artefacts, a clean of any kind of the kernel can clear them out completely with no impact on sstate. Assigning to Darren to handle this.
Comment 4 Darren Hart 2015-01-13 04:57:15 UTC
I'm experimenting with:

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index ecde341..7a788d9 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -40,7 +40,8 @@ python __anonymous () {
 # We need to move these over to STAGING_KERNEL_DIR. We can't just
 # create the symlink in advance as the git fetcher can't cope with
 # the symlink.
-do_unpack[cleandirs] += " ${S} ${STAGING_KERNEL_DIR} ${B}"
+do_unpack[cleandirs] += " ${S} ${STAGING_KERNEL_DIR} ${B} ${STAGING_KERNEL_BUILDDIR}"
+do_clean[cleandirs] += " ${S} ${STAGING_KERNEL_DIR} ${B} ${STAGING_KERNEL_BUILDDIR}"

Sorry to say I'm struggling to display the contents of "do_clean[cleandirs]" to verify what I've done...
Comment 5 Darren Hart 2015-01-13 05:12:03 UTC
OK, the above appears to work, thanks to nrossi for pointing me in the right direction for debugging VarFlags:

python __anonymous() {
    print(d.getVarFlags("do_clean", "cleandirs"))
    print(d.getVarFlags("do_unpack", "cleandirs"))
}