Bug 5970

Summary: sstate signature generator issues
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Martin Jansa <Martin.Jansa>
Component: coreAssignee: Richard Purdie <richard.purdie>
Status: VERIFIED FIXED QA Contact: Daniel Istrate <daniel.alexandrux.istrate>
Severity: enhancement    
Priority: Medium CC: bogdanx.a.voiculescu, jessica.zhang, kergoth, meta.mr.watcher, meta.watcher
Version: 1.7   
Target Milestone: 1.9   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Attachments:
Description Flags
signatures none

Description Martin Jansa 2014-03-14 13:20:20 UTC 
This should in theory have 3 subtasks, but lets first discuss the possible implementations here, because all 3 are related.

1) packagegroups no longer rebuild when debian.bbclass renames one of runtime deps
 19:14 < JaMa> RP: there is small problem with http://git.openembedded.org/openembedded-core/commit/?id=80b065ff46322ec0cad039dfd9eb2d010168dba6
 19:15 < JaMa> RP: I have MACHINE_ARCH packagegroup which includes some component which includes only library and thanks to debian.bbclass it was renamed, but packagegroup isn't rebuilt so it still depends on old name and do_rootfs fails
..
 12:21 < RP> JaMa: hmm, having the packagegroups always rebuilding isn't ideal :/
 12:23 < JaMa> agreed, that's why I was suggesting to use the same PACKAGE_ARCH for them as the packages included in them
 12:23 < JaMa> which also isn't ideal, but is somewhere in the middle
 12:24 < JaMa> rebuilding when the (r)dependencies change, but rebuilding just once per architecture
 12:31 < RP> JaMa: once per tune effectively
 12:32 < RP> it seems sad to have to force them all to that behaviour :(
 12:40 < JaMa> agreed, but at least they are "built" super fast
 12:41 < JaMa> once per tune, unless they are really allarch or MACHINE_ARCH :)
 12:44 < RP> JaMa: how many cases of this problem do you run into?
 12:54 < JaMa> I've backported sstatesig.py change 2 days ago and 2 cases (in 6 different images) since then
 12:55 < JaMa> I've already proposed work around for our cases, where the package with just one library (so possibly renamed by debian.bbclass) is moved from packagegroup to some other recipe RDEPENDS
 12:56 < JaMa> but because such renames don't happen too often, I guess that there is more cases like this, just hard to trigger them
 12:56 < JaMa> and people wouldn't understand why it's happening and how to fix it :/

2) signatures of target recipes now include signatures of native tools, so all signatures are different for 32bit and 64bit builders
 changed in:
 http://git.openembedded.org/openembedded-core/commit/meta/lib/oe/sstatesig.py?id=336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6
 before this commit it was possible to populate SSTATE_MIRROR by 64bit builder and use most sstate archives (for target recipes) from it and build only 32bit native tools locally (or small 32bit build which will populate only native/cross tools from sstate-cache/NATIVELSB) and vice versa.

 Now it's more strict so it rebuilds all target recipes when they depend on changed native/cross tool, but it also means that you need another 32bit builder to populate whole SSTATE_MIRROR.

 It would be great to be able to exclude 32/64 bit difference from native signature when including it in target task signature.

3) better ways to exclude more variables from signatures
 There isn't any good compromise between strict and less strict signatures.

 People want reliable builds (so that all tasks which could be possibly changed are always re-executed) and they also want to prevent rebuilding everything when just some whitespace is added to some globally used .bbclass like package.bbclass).

 There were multiple solutions proposed to tackle this, but we never finished them completely.

 a) OEBasicHashLite:
    http://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/OEBasickHashLite&id=71cf0468787c5f5758d4afad5ab8d41926a952f3
    useful in some cases, but still suffers from full rebuilds when .bbclass is changed

 b) comparing the binary output and stop propagating signature change when the binary result of some task is the same (so cannot influence following tasks in dependency tree)
    great idea from kergoth, but hard to implement in current code base
    http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327

 c) lately I was thinking about different signature handler for "own" tasks and tasks included from dependencies
    some variables are important for recipe A when building recipe A, but recipe B which depends on A doesn't care if A.populate_sysroot was executed with e.g. BUILD_ARCH set to i586 or x86-64

    so we would be able to excluded BUILD_ARCH from signature when looking at it from some other recipe, but still have different signatures for 32bit and 64bit build.

    I haven't tried to implement this, so I cannot estimate how difficult it would be in current codebase. It's true that even current signatures and varflags to modify them are quite complicated
    for most people and helper scripts are needed to detect what will re-execute and why after some metadata change, so having another signature for each task makes it even more difficult, but it can serve
    interesting use-cases

    excluding 32/64 difference from native recipes when looking on them from target ones (hmm maybe not so simple, because we'll probably need to still include them when looking from other -native recipes)

    excluding some variables which produce ABISAFE output - now there are only 2 ways to exclude *whole* signature of some dependency from all (SIGGEN_EXCLUDERECIPES_ABISAFE) or only some (SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS) recipes, it would be great to have simple finer control over what is ABISAFE and what's not.

       if recipe A has DEBUG variable which adds some compiler flag in do_configure, then we need to include DEBUG variable in do_configure signature to rebuild it when changed, but if the developer really knows that A is ABI compatible with or without DEBUG enabled, then adding do_configure[vardepexclude-from-deps] += "DEBUG" would prevent rebuilding all recipes depending on A when just DEBUG variable was changed.

       B depending on A will still include some A signature, so it's rebuilt when e.g. A is upgraded and .so version is increased, but it won't care if A was built with DEBUG enabled or not.

    How does this sound?
Comment 1 Richard Purdie 2014-03-17 11:50:18 UTC 
This may be better as something discussed on a mailing list first :/.

For 2), could you share the output of bitbake-diffsigs between two builds from a 32 and 64 bit builder?

I ask since I don't have a machine handy to check with however I was under the impression that since BUILD_ARCH uses uname, it doesn't actually get hardcoded in as you'd first think...

Getting 2) fixed is probably a release blocker
Comment 2 Martin Jansa 2014-03-20 15:12:06 UTC 
Attached archive contains signatures from 32bit and 64bit builder, all were created with:

distroless setup

bitbake/60969cd62e21e7d4af161bf8504b7643a879c73f

oe-core/ef196434620522affc11b5b1b867386b5d14a4c3 + few commits from git.openembedded.org/openembedded-core-contrib/log/?h=jansa/test

and this command:

oe-core/scripts/sstate-diff-machines.sh --machines="qemux86" --targets="core-image-minimal"

diff BUILD/sstate-diff/1395325096/qemux86/list.M BUILD/sstate-diff/1395325761/qemux86/list.M | grep -v "\-native" | wc -l
1186

bitbake-diffsigs BUILD/sstate-diff/{1395325096,1395325761}/qemux86/i586*/zlib/*do_configure*
Hash for dependent task eglibc_2.19.bb.do_populate_sysroot changed from 45a13342c496ac28d1e9ba91a993d213 to 55205208f0b5b2d490ea428ce84eac6b
Hash for dependent task gcc-cross_4.8.bb.do_populate_sysroot changed from 8eacd38703453c6344992a9d189189d3 to 858f961b602e89a665e6b616cb85186b
Hash for dependent task gcc-runtime_4.8.bb.do_populate_sysroot changed from e95be0dd8d128ddce2ca26341d16db58 to 693b10fec759f606184b77c5a6320ea0

bitbake-diffsigs BUILD/sstate-diff/{1395325096,1395325761}/qemux86/i586*/linux-libc-headers/*do_install*
Hash for dependent task virtual:native:pseudo_1.5.1.bb.do_populate_sysroot changed from c54682aa4d1159583cc85534a23ccf65 to 9ac7653f83466a96f9c420bc57aa5457

bitbake-diffsigs BUILD/sstate-diff/{1395325096,1395325761}/qemux86/*/pseudo-native/*do_install*
basehash changed from 183413d3217e5fc203850e57bc840feb to 4e6d227fc89405cfb950e6f772f36419
Variable SITEINFO_BITS value changed from '32' to '64'
Hash for dependent task virtual:native:pseudo_1.5.1.bb.do_compile changed from eef2bf50a3ffff75359ee537c8e06da1 to 2dc4042a46b2a91ad6e08d5014e3aea2

bitbake-diffsigs BUILD/sstate-diff/{1395325096,1395325761}/qemux86/*/pseudo-native/*do_compile*
basehash changed from f92e830559eef8c4abb8c72ea677ea95 to fe56c23f7410d074fb862a7a90db0de0
Variable SITEINFO_BITS value changed from '32' to '64'

so excluding pseudo-native looked like first option, but still:

bitbake-diffsigs BUILD/sstate-diff/*-no-pseudo/qemux86/*/zlib/*do_configure*
Hash for dependent task eglibc_2.19.bb.do_populate_sysroot changed from 1f5e81e38e1ae8ad7f28975c81340923 to a44f3d0ae4e8fc9483fd92513872d10f
Hash for dependent task gcc-cross_4.8.bb.do_populate_sysroot changed from d931a8bc25a30304e5b9d64a7df695c4 to cfc5dfccb35a7402a70d71ce680c03da
Hash for dependent task gcc-runtime_4.8.bb.do_populate_sysroot changed from b1b406033f2937e4aa83ed8ac09b7b8a to 85cf4155602019e40ff033fc05261c7b

linux-libc-headers are fine now:
$ ls BUILD/sstate-diff/*-no-pseudo/qemux86/i586*/linux-libc-headers/*do_populate_sysroot*
BUILD/sstate-diff/32bit-no-pseudo/qemux86/i586-oe-linux/linux-libc-headers/3.10-r0.do_populate_sysroot.sigdata.d37095a400e80dbdd747a3c7b618f0a0
BUILD/sstate-diff/64bit-no-pseudo/qemux86/i586-oe-linux/linux-libc-headers/3.10-r0.do_populate_sysroot.sigdata.d37095a400e80dbdd747a3c7b618f0a0

binutils-cross are next:
mjansa@rabbit:~/build/build-webos-master$ bitbake-diffsigs BUILD/sstate-diff/*-no-pseudo/qemux86/i586*/eglibc-initial/*do_configure*
Hash for dependent task gcc-cross-initial_4.8.bb.do_populate_sysroot changed from 42c6ed51552de3a901de10158efacd28 to 5337b2c826f5b63e0b300f523a46fba0

mjansa@rabbit:~/build/build-webos-master$ bitbake-diffsigs BUILD/sstate-diff/*-no-pseudo/qemux86/i586*/gcc-cross-initial/*do_configure*
Hash for dependent task binutils-cross_2.24.bb.do_populate_sysroot changed from f356a79546121c85a0c9cf76d6e19fea to 9efe1c2957551c52f0e2683c71b1e3c2

mjansa@rabbit:~/build/build-webos-master$ bitbake-diffsigs BUILD/sstate-diff/*-no-pseudo/qemux86/i586*/binutils-cross/*do_configure*

mjansa@rabbit:~/build/build-webos-master$ bitbake-diffsigs BUILD/sstate-diff/*-no-pseudo/qemux86/i586*/binutils-cross/*do_populate_sysroot*
basehash changed from 53d49524cf408edf32a12239434811dc to e63c0f5bef1a5ac6ca4d8df2439a8275
Variable SSTATE_PKGARCH value changed from 'i686_i586' to 'x86_64_i586'

binutils-cross.do_populate_sysroot
List of dependencies for variable SSTATE_PKGSPEC is set(['PR', 'PV', 'SSTATE_PKGARCH', 'TARGET_VENDOR', 'PACKAGE_ARCH', 'PN', 'TARGET_OS', 'SSTATE_VERSION'])
Variable SSTATE_PKGNAME value is ${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC', True), d.getVar('BB_TASKHASH', True), d)}
List of dependencies for variable SSTATE_PKG is set(['SSTATE_PKGNAME'])

gcc-cross-initial.do_populate_sysroot
List of dependencies for variable SSTATE_PKGSPEC is set(['PR', 'PV', 'SSTATE_PKGARCH', 'TARGET_VENDOR', 'PACKAGE_ARCH', 'PN', 'TARGET_OS', 'SSTATE_VERSION'])
Variable SSTATE_PKGNAME value is ${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC', True), d.getVar('BB_TASKHASH', True), d)}
List of dependencies for variable SSTATE_PKG is set(['SSTATE_PKGNAME'])

So it looks like it should have the same problem in gcc-cross-initial and yes it does (doh, I was comparing just do_configure before)

bitbake-diffsigs BUILD/sstate-diff/*-no-pseudo/qemux86/i586*/gcc-cross-initial/*do_populate_sysroot*
basehash changed from de8b571d65e11f0bc0c86a699c9278ed to aa43a9ffa1e028ac3cf0f5d58ee30056
Variable SSTATE_PKGARCH value changed from 'i686_i586' to 'x86_64_i586'
Hash for dependent task gcc-cross-initial_4.8.bb.do_install changed from add9294217608cb08114c33dc2c82455 to 06b265afa1ae4a216a0bae0f0a7f31b1
Comment 3 Martin Jansa 2014-03-20 15:26:48 UTC 
Created attachment 1848 [details]
signatures
Comment 4 Christopher Larson 2014-08-28 17:52:25 UTC 
At Mentor, we've noticed that SITEINFO_ENDIANNESS, SITEINFO_BITS, and SIZEOF_POINTER will all change native checksums, and therefore ripple through the target recipes depending on them, reducing target sstate reuse between 32 and 64 bit hosts. After adding them to BB_HASHBASE_WHITELIST, we've noticed a massive increase in target sstate use between 32 and 64 bit hosts again, which reduces how much sstate we have to distribute. We haven't had time to submit that to oe-core yet, but I'd be interested in thoughts on it, and if others have run into this.

It's worth noting, for the curious, that BUILD_ARCH changing doesn't change native/cross sstate checksums, because its calculated in ${@}, so just its unexpanded form is in the signature. But BUILD_ARCH ends up included in the sstate filenames, so changes to BUILD_ARCH will result in rebuilds of native/cross but not rebuilds of the target recipes which depend on them, which is just what we want -- it just took me a little bit to realize why it was behaving this way, so it's useful background.
Comment 5 Richard Purdie 2015-07-08 12:01:18 UTC 
1) is addressed by a separate bug now (#7298)

For 2), there are a number of patches on the mailing list in review for this. This includes an automated QA test so that now we've fixed this problem, we ensure it doesn't regress again.

For 3), I understand the intent but I struggle to see how we can implement something like this which will work in reality. If anyone does have a solid proposal and time to work on implementing this, great but I think we'd have to mark such a bug as "FUTURE" right now :(. I think it does belong in a separate bug though (bugs targeting specific issues are easiler to deal with). 

I'm therefore assuming this bug targets 2) from here.
Comment 7 Daniel Istrate 2015-10-09 14:32:31 UTC 
Verified on master: eac61f37e36099f74485dab398b57f3812826d17

Functionality testing is covered by oe-selfttest testcases from sstatetests.