Testing on distro-less (only change in local.conf is inheriting package_rpm, but with package_ipk it's the same, not using rm_work or something like that) openembedded-core: Steps to reproduce: Create 2nd MACHINE with x86_64 architecture, but with extra TUNE_CCARGS. OE @ ~/openembedded-core/meta/conf/machine $ diff -uNr qemux86-64* --- qemux86-64.conf 2011-12-13 14:50:31.937361242 +0100 +++ qemux86-64core.conf 2012-01-18 11:15:08.754183198 +0100 @@ -26,3 +26,5 @@ GLIBC_EXTRA_OECONF = "--with-tls" MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d" + +TUNE_CCARGS += " -mtune=corei7-avx" Don't forget to duplicate qemux86-64 overrides in linux-yocto (otherwise linux-yocto build fails). diff --git a/meta/recipes-kernel/linux/linux-yocto_3.0.bb b/meta/recipes-kernel/linux/linux-yocto_3.0.bb index 0b79bee..a9f4c1b 100644 --- a/meta/recipes-kernel/linux/linux-yocto_3.0.bb +++ b/meta/recipes-kernel/linux/linux-yocto_3.0.bb @@ -4,6 +4,7 @@ require recipes-kernel/linux/linux-yocto.inc KMACHINE = "yocto/standard/base" KMACHINE_qemux86 = "yocto/standard/common-pc/base" KMACHINE_qemux86-64 = "yocto/standard/common-pc-64/base" +KMACHINE_qemux86-64core = "yocto/standard/common-pc-64/base" KMACHINE_qemuppc = "yocto/standard/qemu-ppc32" KMACHINE_qemumips = "yocto/standard/mti-malta32-be" KMACHINE_qemuarm = "yocto/standard/arm-versatile-926ejs" @@ -18,6 +19,7 @@ SRCREV_machine_qemumips ?= "56e4361a1bfc10042d83a7d8a5f96932056e3f35" SRCREV_machine_qemuppc ?= "53e71af30e30162e17e32b2f57e4481f5bc0573e" SRCREV_machine_qemux86 ?= "9e73573b6b227ab2c62e66e63470866d136f8d76" SRCREV_machine_qemux86-64 ?= "3551fd0412965b1f6b0b2f6ba35f7fecdaddb58a" +SRCREV_machine_qemux86-64core ?= "3551fd0412965b1f6b0b2f6ba35f7fecdaddb58a" SRCREV_machine ?= "6f5b11830deba4a6262c8c4abf67e608924f649e" SRCREV_meta ?= "73dafd44ea875df654129b32b2877f342d5573e4" build complete image with qemux86-64 bitbake -k core-image-minimal-mtdutils then switch to qemux86-64core and build again bitbake -k core-image-minimal-mtdutils Expected result: qemux86-64core sysroot is populated from sstate-cache and only machine specific packages like kernel are built Actual result: After reusing sstate in do_populate_sysroot, do_package+do_package_write_rpm is called for every recipe and build takes much longer then expected. It's true that "-mtune=corei7-avx" would cause different binaries, but do_configure/do_compile/do_install were correctly skipped (as its output will be in "shared" deploy/rpm/x86_64/ anyway) so skipping "packaging" steps is correct too. Some logs: bitbake -k core-image-minimal-mtdutils Loaded 1088 entries from dependency cache. OE Build Configuration: BB_VERSION = "1.15.0" TARGET_ARCH = "x86_64" TARGET_OS = "linux" MACHINE = "qemux86-64core" DISTRO = "" DISTRO_VERSION = "oe-core.0" TUNE_FEATURES = "m64" TARGET_FPU = "" meta = "jansa/test:b430ec2fd9ce7c24fd48db1b949b72708493e50f" cooker.log part1 and part2 (because I've forget to duplicate those linux-yocto overrides): http://build.shr-project.org/tests/jama/cooker.log.part1 http://build.shr-project.org/tests/jama/cooker.log.part2 OE @ ~/oe-core/tmp-eglibc $ grep gcc-cross-4 cooker.log.part1 NOTE: package gcc-cross-4.6.2+svnr181430-r20: task do_populate_sysroot: Started NOTE: package gcc-cross-4.6.2+svnr181430-r20: task do_populate_sysroot: Succeeded NOTE: package gcc-cross-4.6.2+svnr181430-r20: task do_package: Started NOTE: package gcc-cross-4.6.2+svnr181430-r20: task do_package: Succeeded NOTE: package gcc-cross-4.6.2+svnr181430-r20: task do_package_write_rpm: Started NOTE: package gcc-cross-4.6.2+svnr181430-r20: task do_package_write_rpm: Succeeded OE @ ~/oe-core/tmp-eglibc $ grep gcc-cross-initial-4 cooker.log.part1 NOTE: package gcc-cross-initial-4.6.2+svnr181430-r20: task do_populate_sysroot: Started NOTE: package gcc-cross-initial-4.6.2+svnr181430-r20: task do_populate_sysroot: Succeeded NOTE: package gcc-cross-initial-4.6.2+svnr181430-r20: task do_package: Started NOTE: package gcc-cross-initial-4.6.2+svnr181430-r20: task do_package: Succeeded NOTE: package gcc-cross-initial-4.6.2+svnr181430-r20: task do_package_write_rpm: Started NOTE: package gcc-cross-initial-4.6.2+svnr181430-r20: task do_package_write_rpm: Succeeded OE @ ~/oe-core/tmp-eglibc $ grep zlib cooker.log.part1 NOTE: Running task 627 of 1660 (ID: 467, /OE/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.5.bb, do_populate_sysroot) NOTE: package zlib-1.2.5-r3: task do_populate_sysroot: Started NOTE: package zlib-1.2.5-r3: task do_populate_sysroot: Succeeded NOTE: Running task 679 of 1660 (ID: 472, /OE/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.5.bb, do_package) NOTE: package zlib-1.2.5-r3: task do_package: Started NOTE: package zlib-1.2.5-r3: task do_package: Succeeded NOTE: Running task 1458 of 1660 (ID: 474, /OE/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.5.bb, do_package_write_rpm) NOTE: package zlib-1.2.5-r3: task do_package_write_rpm: Started NOTE: package zlib-1.2.5-r3: task do_package_write_rpm: Succeeded NOTE: Running noexec task 1606 of 1660 (ID: 471, /OE/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.5.bb, do_package_write) I'll rebuild both machines from scratch again (now with linux-yocto recipe prepared) to get better stats on number of tasks, but I guess the issue is clear now.
To be perfectly honest I'd have expected the populate_sysroot tasks to re-execute too. There should have been a complete recompile as the compiler options changed. Was this with a clean build directory and shared sstate or was tmp left complete between the two machines?
(In reply to comment #1) > To be perfectly honest I'd have expected the populate_sysroot tasks to > re-execute too. There should have been a complete recompile as the compiler > options changed. Was this with a clean build directory and shared sstate or was > tmp left complete between the two machines? populate_sysroot tasks were re-executed (and I've expected them to be as sysroots/qemux86-64core was empty before build). whole tmp (including work) was left between two machines. What's the point of rebuilding it again to "shared" deploy/rpm/x86_64/? End user will probably upgrade from that feed and some packages will be with -mtune=corei7-avx and some without. Actually if I let both qemux86-64 and qemux86-64core finish all do_package task and then create image from it, then qemux86-64 will can get some packages built with -mtune=corei7-avx in image (as well as feed). If the right fix is to not only repackage, but also recompile everything when those 2 MACHINEs are not completely the same, then can we change feed to deploy/${MACHINE}/rpm/x86_64/? Maybe drop distinct PACKAGE_ARCH too? Because building multiple machines which differ only in config name is I guess less common use-case then building slightly different MACHINES (e.g. different CPU but all armv7a-novfp-neon).
Rebuilds from scratch finished http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/qemu-tests is oe-core version used in all 3 builds (current master with 2 patches - qemu-native removed from EXTRA_IMAGEDEPENDS and qemux86-64core/qemux86-65copy machine configs added). All logs are here: http://build.shr-project.org/tests/jama/sstate-issue/ important parts: OE qemux86-64@ ~/oe-core $ time bitbake -k core-image-minimal-mtdutils ... real 87m34.210s user 179m6.308s sys 26m48.553s ... NOTE: Tasks Summary: Attempted 1550 tasks switch to qemux86-64core (with that extra mtune) OE qemux86-64core@ ~/oe-core $ time bitbake -k core-image-minimal-mtdutils ... real 34m6.221s user 31m14.743s sys 15m57.443s ... NOTE: Tasks Summary: Attempted 1550 tasks of which 1204 didn't need to be rerun and all succeeded. switch to qemux86-64copy (exactly the same as qemux86-64) OE qemux86-64copy@ ~/oe-core $ time bitbake -k core-image-minimal-mtdutils ... real 26m12.720s user 18m34.057s sys 6m54.421s ... NOTE: Tasks Summary: Attempted 1550 tasks of which 1443 didn't need to be rerun and all succeeded. Less tasks then qemux86-64core but still quite a few do_package re-runs too (because qemux86-64core was built in between - will try to rebuild again without qemux86-64core).
Even more info: Building images now after building all running all those do_packages is _fast_ see http://build.shr-project.org/tests/jama/sstate-issue/time.txt OE qemux86-64@ ~/oe-core $ time bitbake -k core-image-minimal-mtdutils real 1m2.297s user 1m24.155s sys 0m9.886s OE qemux86-64core@ ~/oe-core $ time bitbake -k core-image-minimal-mtdutils real 0m29.087s user 0m15.591s sys 0m6.254s OE qemux86-64copy@ ~/oe-core $ time bitbake -k core-image-minimal-mtdutils real 0m32.289s user 0m15.293s sys 0m6.055s All 3 happily using .rpm files created some with some without that -mtune: NOTE: Tasks Summary: Attempted 1550 tasks of which 1548 didn't need to be rerun and all succeeded. sstate analysis (sed is used just to make more friendly width): $ ls -lah sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-* | sort |\ sed 's#sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-#sstate-#g' -rw-r--r-- 1 bitbake bitbake 123 Jan 18 16:32 sstate-07e46443bf8ea0c54986a0beb3fd176d_deploy-rpm.tgz -rw-r--r-- 1 bitbake bitbake 123 Jan 18 17:31 sstate-747ceb55fc7054f50602e374f1db7e47_deploy-rpm.tgz -rw-r--r-- 1 bitbake bitbake 123 Jan 18 17:53 sstate-b7b56962515abd3f00ae1a7b7eb7bf50_deploy-rpm.tgz -rw-r--r-- 1 bitbake bitbake 165 Jan 18 15:47 sstate-03a0bf16a6bb514f6a57e0ad9e5abfa0_package.tgz -rw-r--r-- 1 bitbake bitbake 165 Jan 18 17:00 sstate-76752938aa52494b274c0b9ea4e6188e_package.tgz -rw-r--r-- 1 bitbake bitbake 165 Jan 18 17:48 sstate-2cc574da6aec95a856478660bd446b4f_package.tgz -rw-r--r-- 1 bitbake bitbake 19M Jan 18 15:41 sstate-7d71d9ac8fb06ca45e865eafb060fef4_populate-sysroot.tgz -rw-r--r-- 1 bitbake bitbake 19M Jan 18 17:00 sstate-7104e53b94bb5aa7872463318f160e0f_populate-sysroot.tgz -rw-r--r-- 1 bitbake bitbake 19M Jan 18 17:48 sstate-700ebce201a49295ce7d821922137b2c_populate-sysroot.tgz -rw-r--r-- 1 bitbake bitbake 20K Jan 18 15:41 sstate-7d71d9ac8fb06ca45e865eafb060fef4_populate-sysroot.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 20K Jan 18 17:00 sstate-7104e53b94bb5aa7872463318f160e0f_populate-sysroot.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 20K Jan 18 17:48 sstate-700ebce201a49295ce7d821922137b2c_populate-sysroot.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 27K Jan 18 16:32 sstate-07e46443bf8ea0c54986a0beb3fd176d_deploy-rpm.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 27K Jan 18 17:31 sstate-747ceb55fc7054f50602e374f1db7e47_deploy-rpm.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 27K Jan 18 17:53 sstate-b7b56962515abd3f00ae1a7b7eb7bf50_deploy-rpm.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 35K Jan 18 15:47 sstate-12e1629254b60eaa5fc15ea630b23566_populate-lic.tgz -rw-r--r-- 1 bitbake bitbake 78K Jan 18 15:47 sstate-03a0bf16a6bb514f6a57e0ad9e5abfa0_package.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 78K Jan 18 17:00 sstate-76752938aa52494b274c0b9ea4e6188e_package.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 78K Jan 18 17:48 sstate-2cc574da6aec95a856478660bd446b4f_package.tgz.siginfo -rw-r--r-- 1 bitbake bitbake 7.0K Jan 18 15:47 sstate-12e1629254b60eaa5fc15ea630b23566_populate-lic.tgz.siginfo OE qemux86-64copy@ ~/oe-core/sstate-cache $ bitbake-diffsigs \ sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-03a0bf16a6bb514f6a57e0ad9e5abfa0_package.tgz.siginfo \ sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-76752938aa52494b274c0b9ea4e6188e_package.tgz.siginfo Hash for dependent task gcc-cross_4.6.bb.do_install changed from 441dccf89045c54262857d304a887ed4 to 3ed94353a8224e499158774a96174ed6 OE qemux86-64copy@ ~/oe-core/sstate-cache $ bitbake-diffsigs \ sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-76752938aa52494b274c0b9ea4e6188e_package.tgz.siginfo \ sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-2cc574da6aec95a856478660bd446b4f_package.tgz.siginfo Hash for dependent task gcc-cross_4.6.bb.do_install changed from 3ed94353a8224e499158774a96174ed6 to 016b304283abf934d2c60e5a82f02d52 OE qemux86-64copy@ ~/oe-core/sstate-cache $ bitbake-diffsigs \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-7d71d9ac8fb06ca45e865eafb060fef4_populate-sysroot.tgz.siginfo \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-7104e53b94bb5aa7872463318f160e0f_populate-sysroot.tgz.siginfo Hash for dependent task gcc-cross_4.6.bb.do_install changed from 441dccf89045c54262857d304a887ed4 to 3ed94353a8224e499158774a96174ed6 OE qemux86-64copy@ ~/oe-core/sstate-cache $ bitbake-diffsigs \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-7104e53b94bb5aa7872463318f160e0f_populate-sysroot.tgz.siginfo \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-700ebce201a49295ce7d821922137b2c_populate-sysroot.tgz.siginfo Hash for dependent task gcc-cross_4.6.bb.do_install changed from 3ed94353a8224e499158774a96174ed6 to 016b304283abf934d2c60e5a82f02d52 OE qemux86-64copy@ ~/oe-core/sstate-cache $ bitbake-diffsigs \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-07e46443bf8ea0c54986a0beb3fd176d_deploy-rpm.tgz.siginfo \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-747ceb55fc7054f50602e374f1db7e47_deploy-rpm.tgz.siginfo Hash for dependent task gcc-cross_4.6.bb.do_package changed from 03a0bf16a6bb514f6a57e0ad9e5abfa0 to 76752938aa52494b274c0b9ea4e6188e OE qemux86-64copy@ ~/oe-core/sstate-cache $ bitbake-diffsigs \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-747ceb55fc7054f50602e374f1db7e47_deploy-rpm.tgz.siginfo \ > sstate-gcc-cross-x86_64-oe-linux-4.6.2+svnr181430-r20-x86_64_x86_64-2-b7b56962515abd3f00ae1a7b7eb7bf50_deploy-rpm.tgz.siginfo Hash for dependent task gcc-cross_4.6.bb.do_package changed from 76752938aa52494b274c0b9ea4e6188e to 2cc574da6aec95a856478660bd446b4f But there is only 1 do_install (it wasn't re-executed) OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_* ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_compile ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_compile.sigdata.efa8047f415949688c1a4c744b24e276 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_configure ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_configure.sigdata.6c5acab02852fb8193ce74e7fa320878 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_install ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_install.sigdata.441dccf89045c54262857d304a887ed4 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package.qemux86-64 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package.qemux86-64copy ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package.qemux86-64core ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package.sigdata.03a0bf16a6bb514f6a57e0ad9e5abfa0 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package.sigdata.2cc574da6aec95a856478660bd446b4f ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package.sigdata.76752938aa52494b274c0b9ea4e6188e ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package_write ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package_write_rpm ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_package_write_rpm.sigdata.b7b56962515abd3f00ae1a7b7eb7bf50 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_lic ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_lic.sigdata.12e1629254b60eaa5fc15ea630b23566 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_sysroot.qemux86-64 ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_sysroot.qemux86-64copy ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_sysroot.qemux86-64core ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_sysroot.sigdata.700ebce201a49295ce7d821922137b2c ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_sysroot.sigdata.7104e53b94bb5aa7872463318f160e0f ../tmp-eglibc/stamps/x86_64-oe-linux/gcc-cross-4.6.2+svnr181430-r20.do_populate_sysroot.sigdata.7d71d9ac8fb06ca45e865eafb060fef4 So it looks like do_package was re-executed due to hypotetical change in do_install checksum (which actually wasn't run, so nothing changed) and because of this do_package "change" deploy-rpm.tgz.siginfo was also changed. Some sstate statistics: OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 * | wc -l 1216 OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *-x86_64-oe-linux* | wc -l 926 OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *native-x86_64-linux* | wc -l 126 OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *all-oe-linux* | wc -l 14 OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *qemux86_64core-oe-linux* | wc -l 48 OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *qemux86_64copy-oe-linux* | wc -l 48 OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *qemux86_64-oe-linux* | wc -l 54 OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *netbase-qemux86_64-oe-linux* | wc -l <- this explains the difference in last number 6 OE qemux86-64copy@ ~/oe-core/sstate-cache $ expr 926 + 126 + 14 + 48 + 48 + 48 + 6 1216 Important is first group OE qemux86-64copy@ ~/oe-core/sstate-cache $ ls -1 *-x86_64-oe-linux* | sed 's#-x86_64-oe-linux.*##g' | sort -u | wc -l 69 # 69 unique packages with x86_64 arch OE qemux86-64copy@ ~/oe-core/sstate-cache $ for pkg in `ls -1 *-x86_64-oe-linux* | sed 's#-x86_64-oe-linux.*##g' | sort -u`; do COUNT=`ls -1 ${pkg}-x86_64-oe-linux* | wc -l`; echo ${COUNT} ${pkg}; done | sort -n # number of sstate tgz packages + .siginfo files per package 8 sstate-binutils-cross 8 sstate-module-init-tools-cross 10 sstate-bash ... 14 sstate-zlib 20 sstate-dbus 20 sstate-gcc-cross 20 sstate-glib-2.0 20 sstate-libtool-cross 8 is for minimum for package/populate-lic/populate-sysroot/deploy-rpm, more is for duplicated tasks OE qemux86-64copy@ ~/oe-core/sstate-cache $ for task in package populate-lic populate-sysroot deploy-rpm; do for pkg in `ls -1 *-x86_64-oe-linux* | sed 's#-x86_64-oe-linux.*##g' | sort -u`; do COUNT=`ls -1 ${pkg}-x86_64-oe-linux*${task}.tgz | wc -l`; echo ${COUNT} ${pkg} ${task}; done; done | sort -n > task.counts see http://build.shr-project.org/tests/jama/sstate-issue/task.counts OE qemux86-64copy@ ~/oe-core/sstate-cache $ grep -c ^0 task.counts 14 14 recipes are missing populate-sysroot task: bash busybox initscripts module-init-tools modutils-initscripts mtd-utils netbase opkg perl python sysvinit tinylogin udev udev-extraconf OE qemux86-64copy@ ~/oe-core/sstate-cache $ grep -c ^1 task.counts 75 for all 69 unique recipes, populate-lic task is there only once 6 tasks from binutils-cross and module-init-tools-cross were also executed only once: 1 sstate-binutils-cross deploy-rpm 1 sstate-binutils-cross package 1 sstate-binutils-cross populate-sysroot 1 sstate-module-init-tools-cross deploy-rpm 1 sstate-module-init-tools-cross package 1 sstate-module-init-tools-cross populate-sysroot OE qemux86-64copy@ ~/oe-core/sstate-cache $ grep -c ^2 task.counts 173 62 recipes have package and deploy-rpm re-executed twice 49 recipes have populate-sysroot re-executed twice expr 62 + 62 + 49 = 173 OE qemux86-64copy@ ~/oe-core/sstate-cache $ grep -c ^3 task.counts 14 And 5 recipes have their tasks executed 3 times: OE qemux86-64copy@ ~/oe-core/sstate-cache $ grep ^3 task.counts 3 sstate-dbus deploy-rpm 3 sstate-dbus package 3 sstate-dbus populate-sysroot 3 sstate-gcc-cross deploy-rpm 3 sstate-gcc-cross package 3 sstate-gcc-cross populate-sysroot 3 sstate-glib-2.0 deploy-rpm 3 sstate-glib-2.0 package 3 sstate-glib-2.0 populate-sysroot 3 sstate-libtool-cross deploy-rpm 3 sstate-libtool-cross package 3 sstate-libtool-cross populate-sysroot 3 sstate-udev deploy-rpm 3 sstate-udev package OE qemux86-64copy@ ~/oe-core/sstate-cache $ expr 75 + 2 \* 173 + 3 \* 14 463 I hope this shows that situation is not really ideal :).
Martin, you're weaving several different issues into this bug and its hard to try and separate them out but I'll try. Firstly, if you go changing the tune parameters in a given machine, you are expected to use a different PACKAGE_ARCH. If you do that, you will get a different package feed for the different binaries, different WORKDIR and so on. This was always the way the package architectures was intended to work and nothing has changed there. Yes, you as the user changing various variables can create inconsistent package feeds. There are 101 ways you can do that, the simple answer is just don't. We're therefore unlikely to add MACHINE to DEPLOY_DIR or remove PACKAGE_ARCH, please just use it as its intended. For the second part of the problem, I think it comes down to changing "distro" level policy (the tune of a given PACKAGE_ARCH) without cleaning tmp. You'd at least have some protection if you had BasicHash enabled since the stamp files would contain the sstate checksums but this isn't the case. I really think this bug comes down to doing something that is just not supported or intended to work. FWIW, the reason that some tasks re-execute is that MACHINE is part of their stamp and you changed MACHINE. Its understandable that they don't recompile either without basichash or without a PACKAGE_ARCH change.
Just to try and summarise, this looks like a case of two machines using different tunes settings in the same PACKAGE_ARCH namespace. Each tune needs its own namespace, then things will all work as expected. Mixing them up won't work well, as you highlight below but that isn't a surprise and isn't supported.
(In reply to comment #5) > Martin, you're weaving several different issues into this bug and its hard to > try and separate them out but I'll try. > > Firstly, if you go changing the tune parameters in a given machine, you are > expected to use a different PACKAGE_ARCH. If you do that, you will get a > different package feed for the different binaries, different WORKDIR and so on. > This was always the way the package architectures was intended to work and > nothing has changed there. Yes, you as the user changing various variables can > create inconsistent package feeds. There are 101 ways you can do that, the > simple answer is just don't. We're therefore unlikely to add MACHINE to > DEPLOY_DIR or remove PACKAGE_ARCH, please just use it as its intended. OK, so changing TUNE_CCARGS to create slightly different MACHINE wasn't good example, so someone who needs something like qemux86-64core will introduce new tune with PACKAGE_ARCH like x86_64-corei7-avx, right? > For the second part of the problem, I think it comes down to changing "distro" > level policy (the tune of a given PACKAGE_ARCH) without cleaning tmp. You'd at > least have some protection if you had BasicHash enabled since the stamp files > would contain the sstate checksums but this isn't the case. OK I'll retry to build from scratch only qemux86-64 and qemux86-64copy, but from my observation of armv7a-vfp-neon builds (which all include just meta/conf/machine/include/tune-cortexa8.inc) it will also reexecute do_package tasks (and recreate new .ipk files with same content). > I really think this bug comes down to doing something that is just not > supported or intended to work. > > FWIW, the reason that some tasks re-execute is that MACHINE is part of their > stamp and you changed MACHINE. Its understandable that they don't recompile > either without basichash or without a PACKAGE_ARCH change. And this is correct? I understand this for packages with MACHINE_ARCH which have .ipk files also stored in different directories/feeds, but for x86_64/*.ipk I would expect that it's enough to create them once, not for every MACHINE. And from that long log it looks like do_package was reexecuted in all cases just because "Hash for dependent task *.do_install changed from abc to xyz" but do_install was executed only once and there is only one do_install.sigdata file. Sorry for confusing report. It's just because my buildhost is already running nonstop to catch up with metadata changes (building 6 machines - 5 are armv7a-vfp-neon), so I fear that with basichash I won't be able to use oe-core (as reported in http://www.mail-archive.com/openembedded-core@lists.openembedded.org/msg11438.html). And you also said on IRC that do_package shouldn't be reexecuted after MACHINE switch, that's why I expected it's a bug not unsupported expectation (of OE-classic behavior) from me. Cheers,
(In reply to comment #7) > OK I'll retry to build from scratch only qemux86-64 and qemux86-64copy, but > from my observation of armv7a-vfp-neon builds (which all include just > meta/conf/machine/include/tune-cortexa8.inc) it will also reexecute do_package > tasks (and recreate new .ipk files with same content). We have found some bugs in the gcc recipes which was damaging the sstate checksums due to the shared gcc work directory. Those issues at least have been fixed in master. To illustrate how we can investigate this problem faster I was able to: cp qemux86.conf qemux86copy.conf Add the kernel SRCREV for qemux86copy rm tmp/stamps/*/*sigdata* MACHINE=qemux86 bitbake core-image-minimal -S find tmp/stamps/i586-poky-linux/ -name \*sigdata* | sort > list1 MACHINE=qemux86copy bitbake core-image-minimal -S find tmp/stamps/i586-poky-linux/ -name \*sigdata* | sort > list2 A diff of list1 and list2 shows what stamps are different between these two builds. It looks like there are issues in gcc and dbus at least which probably cascade down into some of the other things rebuilding as you report. > > I really think this bug comes down to doing something that is just not > > supported or intended to work. > > > > FWIW, the reason that some tasks re-execute is that MACHINE is part of their > > stamp and you changed MACHINE. Its understandable that they don't recompile > > either without basichash or without a PACKAGE_ARCH change. > > And this is correct? > > I understand this for packages with MACHINE_ARCH which have .ipk files also > stored in different directories/feeds, but for x86_64/*.ipk I would expect > that it's enough to create them once, not for every MACHINE. It should recreate them for each different *tune*. It shouldn't happen more than once per given tune. > And you also said on IRC that do_package shouldn't be reexecuted after MACHINE > switch, that's why I expected it's a bug not unsupported expectation (of > OE-classic behavior) from me. It shouldn't and there is a problem somewhere here, its just getting to it which is the problem. I think the above should give us a good start though.
It's much better with last changes, only 3 packages got do_package/do_deploy_tmp re-executed after switching from qemux86-64 to qemux86-64copy OE qemux86-64copy@ ~/oe-core/sstate-cache $ grep ^2 task.counts 2 sstate-dbus deploy-rpm 2 sstate-dbus package 2 sstate-dbus populate-sysroot 2 sstate-glib-2.0 deploy-rpm 2 sstate-glib-2.0 package 2 sstate-glib-2.0 populate-sysroot 2 sstate-udev deploy-rpm 2 sstate-udev package and sstate packages for some gcc-cross-* packages weren't created at all now (instead of 3 times): 0 sstate-gcc-cross deploy-rpm 0 sstate-gcc-cross package 0 sstate-gcc-cross populate-lic 0 sstate-gcc-cross-initial deploy-rpm 0 sstate-gcc-cross-initial package 0 sstate-gcc-cross-initial populate-lic 0 sstate-gcc-cross-intermediate deploy-rpm 0 sstate-gcc-cross-intermediate package 0 sstate-gcc-cross-intermediate populate-lic
(In reply to comment #9) > It's much better with last changes, only 3 packages got > do_package/do_deploy_tmp re-executed after switching from qemux86-64 to > qemux86-64copy > > OE qemux86-64copy@ ~/oe-core/sstate-cache $ grep ^2 task.counts > 2 sstate-dbus deploy-rpm > 2 sstate-dbus package > 2 sstate-dbus populate-sysroot > 2 sstate-glib-2.0 deploy-rpm > 2 sstate-glib-2.0 package > 2 sstate-glib-2.0 populate-sysroot > 2 sstate-udev deploy-rpm > 2 sstate-udev package I've posted more discussion about this on the mailing list. The above should be fixed with some of the patches I've posted there. For core-image-sato, I'm down to two packages rebuilding, both due to kernel-module dependencies. > and sstate packages for some gcc-cross-* packages weren't created at all now > (instead of 3 times): > 0 sstate-gcc-cross deploy-rpm > 0 sstate-gcc-cross package > 0 sstate-gcc-cross populate-lic > 0 sstate-gcc-cross-initial deploy-rpm > 0 sstate-gcc-cross-initial package > 0 sstate-gcc-cross-initial populate-lic > 0 sstate-gcc-cross-intermediate deploy-rpm > 0 sstate-gcc-cross-intermediate package > 0 sstate-gcc-cross-intermediate populate-lic A recent gcc patch I merged marked some of those tasks as noexec meaning they never get run at all removing the overhead of the empty packages :) If you agree I'd propose we close this bug when the remaining patches merge and then create new bugs for any specific new issues?
The mentioned fixes have now merged ending with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0287f645f1547922a37331c281cdf09807d12b2d. As discussed, any further issues can be reported in separate bugs for the specific problems.