14784 2022-04-12 16:50:18 +0000 Unstable Task Re-use/Setscene logic on populate_lic tasks in a MultiConfig Build 2022-04-14 20:24:38 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED WONTFIX Medium normal 4.1 1 gregorylumen gregorylumen poky.bs.watcher poky.watcher randy.macleod --- --- Don't know oldest_to_newest 93011 0 gregorylumen 2022-04-12 16:50:18 +0000 I have been observing what appears to be some Unstable Logic related to Task Re-use and/or setscene around populate_lic tasks in a MultiConfig Build. The build consists of a custom device image and an initramfs multiconfig (that gets pulled into the device image via INITRAMFS_IMAGE_BUNDLE). (Note, the device image and the initramfs configs have separate TMPDIRs) Both the device image and initramfs have a number of common recipies, including zlib (which was chosen randomly for the purposes of this investigation). I initially noted our SSTATE miss rate was higher than expected, even when no changes had been made. After investigating, it became clear that we were unexpectedly re-running a number of populate_lic tasks. Below is a walk-though of a repro path that highlights the unstable behavior: Step 1: Full clean build Step 2: Check contents of Stamps Directory for zlib populate_lic entries (`find build -path "*/zlib/*populate_lic*.sigdata.*"`) .../build/tmp-initramfs/stamps/cortexa57-poky-linux-musl/zlib/1.2.11-r0.do_populate_lic.sigdata.107ba92d54cb4534ac666d60a9dcbac75ba3a30b20231a91b868091080bfb144 .../build/tmp-qemuarm64-my-device/stamps/cortexa57-my-device-linux/zlib/1.2.11-r0.do_populate_lic.sigdata.107ba92d54cb4534ac666d60a9dcbac75ba3a30b20231a91b868091080bfb144 Step 3: Check contents of SSTATE for zlib populate_lic entries (`find build/sstate-cache -name *:zlib:*populate_lic*.siginfo*) .../build/sstate-cache/10/7b/sstate:zlib::1.2.11:r0::7:107ba92d54cb4534ac666d60a9dcbac75ba3a30b20231a91b868091080bfb144_populate_lic.tgz.siginfo Step 4: Delete everything under the build directory _except_ for the sstate-cache, build Step 5: Check contents of Stamps Directory for zlib populate_lic entries (`find build -path "*/zlib/*populate_lic*.sigdata.*"`) .../build/tmp-initramfs/stamps/cortexa57-poky-linux-musl/zlib/1.2.11-r0.do_populate_lic.sigdata.5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f Step 6: Check contents of SSTATE for zlib populate_lic entries (`find build/sstate-cache -name *:zlib:*populate_lic*.siginfo*) .../build/sstate-cache/10/7b/sstate:zlib::1.2.11:r0::7:107ba92d54cb4534ac666d60a9dcbac75ba3a30b20231a91b868091080bfb144_populate_lic.tgz.siginfo .../build/sstate-cache/5b/cc/sstate:zlib::1.2.11:r0::7:5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f_populate_lic.tgz.siginfo Step 7: Save-off sstate-cache, delete entire build directory (clean) Step 8: Run only populate_lic tasks (`bitbake --runonly=populate_lic my-image`) Step 9: Check contents of Stamps Directory for zlib populate_lic entries (`find build -path "*/zlib/*populate_lic*.sigdata.*"`) .../build/tmp-qemuarm64-my-device/stamps/cortexa57-my-device-linux/zlib/1.2.11-r0.do_populate_lic.sigdata.5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f .../build/tmp-initramfs/stamps/cortexa57-poky-linux-musl/zlib/1.2.11-r0.do_populate_lic.sigdata.5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f Step 10: Check contents of SSTATE for zlib populate_lic entries (`find build/sstate-cache -name *:zlib:*populate_lic*.siginfo*) ../build/sstate-cache/5b/cc/sstate:zlib::1.2.11:r0::7:5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f_populate_lic.tgz.siginfo Step 11: Delete entire build directory, Restore sstate-cache Step 12: Run only populate_lic tasks (`bitbake --runonly=populate_lic my-image`) Step 13: Check contents of Stamps Directory for zlib populate_lic entries (`find build -path "*/zlib/*populate_lic*.sigdata.*"`) ../build/tmp-initramfs/stamps/cortexa57-poky-linux-musl/zlib/1.2.11-r0.do_populate_lic.sigdata.5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f Step 14: Check contents of SSTATE for zlib populate_lic entries (`find build/sstate-cache -name *:zlib:*populate_lic*.siginfo*) ../build/sstate-cache/10/7b/sstate:zlib::1.2.11:r0::7:107ba92d54cb4534ac666d60a9dcbac75ba3a30b20231a91b868091080bfb144_populate_lic.tgz.siginfo ../build/sstate-cache/5b/cc/sstate:zlib::1.2.11:r0::7:5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f_populate_lic.tgz.siginfo This appears to identify 4 different scenarios, (full build clean, full build w/ sstate-cache, runonly=populate_lic clean, runonly=populate_lic w/ sstate-cache) each with different behaviors. 93012 1 gregorylumen 2022-04-12 17:54:52 +0000 I should also add that there also appears to be a different 5th behavior when running `--dump-signatures=printdiff` (though I had to hack up a few changes to get around https://bugzilla.yoctoproject.org/show_bug.cgi?id=14774) where it generates the stampfiles for both hashes: .../build/tmp-initramfs/stamps/cortexa57-poky-linux-musl/zlib/1.2.11-r0.do_populate_lic.sigdata.5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f .../build/tmp-qemuarm64-my-device/stamps/cortexa57-my-device-linux/zlib/1.2.11-r0.do_populate_lic.sigdata.107ba92d54cb4534ac666d60a9dcbac75ba3a30b20231a91b868091080bfb144 93034 2 richard.purdie 2022-04-13 17:00:33 +0000 Do you have hash equivalence enabled or disabled? Can you share the sigdata/siginfo files, or at least run bitbake-diffsigs on them and see why they differ (or if they don't)? 93035 3 richard.purdie 2022-04-13 17:00:57 +0000 Also, which release/branch/revision is this with? 93040 4 gregorylumen 2022-04-13 20:39:45 +0000 (In reply to comment #2) > Do you have hash equivalence enabled or disabled? > Can you share the sigdata/siginfo files, or at least run bitbake-diffsigs on them and see why they differ (or if they don't)? I do not currently have hash equivalence enabled (this is part of an enterprise project, and standing up a hash equivalence server hasn't been a priority yet). As for the delta between the signatures, here is the relevant output from `dump-signatures=printdiff` Task zlib:do_populate_lic couldn't be used from the cache because: We need hash 5bcce361bf9ca855dcecf5bf7c1c0c761f7931daf153a46c95bc148a89276a9f, closest matching task was 107ba92d54cb4534ac666d60a9dcbac75ba3a30b20231a91b868091080bfb144 basehash changed from 5ee77ff415e5e09325ce18d1758473c74ca94043a6030f52beab4310d4e6b9ab to 38930dd057247bd17b0bd5f985ddd3ed43bcedf23c73940e9887898c5d608b03 List of dependencies for variable LIBCEXTENSION changed from '{'ABIEXTENSION'}' to 'set()' Variable LIBCEXTENSION value changed: "[-${@['', '-gnu'][(d.getVar('ABIEXTENSION') or '') != '']}-] {+-musl+}" Variable TARGET_VENDOR value changed from '-my-device' to '-poky' I also ran bitbake -e on the two recipes on question and diffed the output to confirm the source of the deltas. The INCLUDE HISTORY is identical, and aside from timestamps/pathing the root of all of the diffs appear to trace back (as expected) to my distro customization. bitbake -e zlib:do_populate_lic <..snip..> # # $TARGET_VENDOR [3 operations] # set .../work/poky/meta/conf/bitbake.conf:132 # "-oe" # set .../work/poky/meta-poky/conf/distro/poky.conf:11 # "-poky" # set ./meta-my-device/conf/distro/my-device.conf:22 # "-my-device" # pre-expansion value: # "-my-device" TARGET_VENDOR="-my-device" <..snip..> bitbake -e mc:initramfs:zlib:do_populate_lic <..snip..> # $TARGET_VENDOR [2 operations] # set /mnt/vss/_work/1/s/work/poky/meta/conf/bitbake.conf:132 # "-oe" # set /mnt/vss/_work/1/s/work/poky/meta-poky/conf/distro/poky.conf:11 # "-poky" # pre-expansion value: # "-poky" TARGET_VENDOR="-poky" <..snip..> (In reply to comment #3) > Also, which release/branch/revision is this with? All of this is pulling from fairly-recent commits out of honister (Examples were pulled using https://git.yoctoproject.org/poky/commit/?h=honister&id=630d754ea3d7206976b1e0f4489e054cdbc08fe8) 93043 5 gregorylumen 2022-04-13 20:50:32 +0000 Also, to clarify. I have not changed BB_SIGNATURE_HANDLER="OEEquivHash", so bitbake is still starting the local Hash Equivalence server, but I am clearing away hashserv.db between each build, so there is no persistent hash equivalence. 93046 6 richard.purdie 2022-04-14 11:07:53 +0000 Thanks, those details help a lot. I think what is happening is that you have two populate_lic tasks which give identical output. This means that when the second one builds in a multiconfig setup, it notices and changes the hash of the second to have an equivalence with the first. You then delete the equivalence data. A subsequent build therefore can't know that sstate is valid any more as there is now no mapping to it. The bottom line is the hash equivalence data and sstate need to be used together. Rather than delete the data, use a non equivalence siggen. 93062 7 randy.macleod 2022-04-14 14:39:43 +0000 We think this is not a bug as explained by Richard. If you agree please close it or provide additional info. 93076 8 gregorylumen 2022-04-14 15:56:13 +0000 (In reply to comment #6) > Thanks, those details help a lot. > > I think what is happening is that you have two populate_lic tasks which give > identical output. This means that when the second one builds in a > multiconfig setup, it notices and changes the hash of the second to have an > equivalence with the first. > > You then delete the equivalence data. > > A subsequent build therefore can't know that sstate is valid any more as > there is now no mapping to it. > > The bottom line is the hash equivalence data and sstate need to be used > together. Rather than delete the data, use a non equivalence siggen. Let me re-run some tests with a non-equivalence siggen and will report the results back. 93083 9 gregorylumen 2022-04-14 20:24:38 +0000 (In reply to comment #8) > (In reply to comment #6) > > Thanks, those details help a lot. > > > > I think what is happening is that you have two populate_lic tasks which give > > identical output. This means that when the second one builds in a > > multiconfig setup, it notices and changes the hash of the second to have an > > equivalence with the first. > > > > You then delete the equivalence data. > > > > A subsequent build therefore can't know that sstate is valid any more as > > there is now no mapping to it. > > > > The bottom line is the hash equivalence data and sstate need to be used > > together. Rather than delete the data, use a non equivalence siggen. > > Let me re-run some tests with a non-equivalence siggen and will report the > results back. The results of the testing are in. When I switched to `BB_SIGNATURE_HANDLER = "OEBasicHash"` Under all scenarios the device config used one signature (107ba92d) and initramfs config used the other (5bcce361). When I re-enabled "OEEquivHash" and persisted hashserv.db both configs converged (and stayed converged) on one signature (107ba92d in this case, but I would take the bet that I could get it to converge the other way onto 5bcce361) Looking back at the original behavior, I still believe that there is evidence of instability in the Task Re-use/Setscene logic, but given that any such instability is fully and easily mitigated by either using "OEBasicHash" or having a persistent hash-server (and assuming that the choice of hash to converge on is unimportant, which by definition it should be), I think we can safely close this bug, Thanks!