| Summary: | Unnecessary rebuilds due to possibly nondeterministic sstate cache lookup | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Matthias Schiffer <matthias.schiffer> |
| Component: | bitbake | Assignee: | Antonin Godard <antonin.godard> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | antonin.godard, ccasciato, poky.bs.watcher, poky.watcher, pokylinux, randy.macleod, yoann.congal |
| Version: | 5.0.10 | ||
| Target Milestone: | 5.3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
When you remove the build directory but reuse the sstate directory, are you sharing the hash equivalence database between the builds? If not, that would explain this kind of behaviour as that is now required when sharing sstate. Ah, that part was not clear to me from reading the docs - in addition to setting SSTATE_DIR to a common value, we just set the same values as Poky:
BB_HASHSERVE = "auto"
BB_SIGNATURE_HANDLER = "OEEquivHash"
Should PERSISTENT_DIR also be shared, or the whole of ${TOPDIR}/cache? I would have expected that data to be stored within SSTATE_DIR if it is required for its correct operation.
Antonin, It seems the docs need to be update. Do you agree? Yes it sounds like the docs need an update but I'm not entirely sure what and where yet :) Matthias, it looks like you are trying to figure this out, can you: 1. Point to the location in the docs where something is likely missing? 2. Tell me what configuration you ended up using? https://docs.yoctoproject.org/overview-manual/concepts.html#hash-equivalence is a start on hash equivalence but may not be enough. I'd like to see if we need a proper guide for this or if we just need to extend the existing documentation. The documentation I referred to is: - https://docs.yoctoproject.org/overview-manual/concepts.html#shared-state-cache - https://docs.yoctoproject.org/test-manual/understand-autobuilder.html#shared-sstate-dir There is also a list of variables that can be adjusted in local.conf in section https://docs.yoctoproject.org/overview-manual/concepts.html#user-configuration . Here, only SSTATE_DIR is listed, there is no mention of PERSISTENT_DIR. I can confirm that sharing PERSISTENT_DIR in addition to SSTATE_DIR fixes my sstate cache reuse issue (tested by symlinking cache in addition to sstate-cache in my TOPDIR, but I assume that is equivalent to overriding the variables in local.conf or similar) (In reply to Matthias Schiffer from comment #5) > The documentation I referred to is: > > - > https://docs.yoctoproject.org/overview-manual/concepts.html#shared-state- > cache > - > https://docs.yoctoproject.org/test-manual/understand-autobuilder.html#shared- > sstate-dir > > There is also a list of variables that can be adjusted in local.conf in > section > https://docs.yoctoproject.org/overview-manual/concepts.html#user- > configuration . Here, only SSTATE_DIR is listed, there is no mention of > PERSISTENT_DIR. > > I can confirm that sharing PERSISTENT_DIR in addition to SSTATE_DIR fixes my > sstate cache reuse issue (tested by symlinking cache in addition to > sstate-cache in my TOPDIR, but I assume that is equivalent to overriding the > variables in local.conf or similar) Thanks! that's all good information. I've sent a series of patches to improve the docs around PERSISTENT_DIR / sharing the hash equivalence database (you are in copy). How does this relate to https://bugzilla.yoctoproject.org/show_bug.cgi?id=15727 ? (In reply to Matthias Schiffer from comment #5) > I can confirm that sharing PERSISTENT_DIR in addition to SSTATE_DIR fixes my > sstate cache reuse issue (tested by symlinking cache in addition to > sstate-cache in my TOPDIR, but I assume that is equivalent to overriding the > variables in local.conf or similar) So, it turns out PERSISTENT_DIR should _not_ be shared. I guess it worked for you because you didn't have two parallel builds (see the discussion on the list here https://lore.kernel.org/yocto-docs/20250709-update-sstate-dir-docs-v1-0-9114688f33d1@bootlin.com/T/#t). I don't know what your exact setup is, but instead can you try: - setting up a shared hash equivalence server - disabling hash equivalence (by unsetting BB_HASHSERVE) And see if this fixes your issue? Yes, running a local hash equivalence server appears to fix my issue, too. I've added a document on how to setup a local hash equivalence server. This ticket is initially what brought the idea of having such a document. https://git.yoctoproject.org/yocto-docs/commit/?id=4ff998336efdc507de1311e43bf8f4a6258c610a I think this bug can be closed now. If you don't have any other comments, I will close it in the following days. Thanks, Antonin Closing the bug (see https://bugzilla.yoctoproject.org/show_bug.cgi?id=15921#c10). |
I've been trying to debug why I often see rebuilds of the whole toolchain when creating a new build directory even if I have just built an equivalent BSP in another directory and the sstate cache is shared between the two builds (with a recent version of Scarthgap, and a BSP based on meta-ti [1] and meta-tq [2]). It appears that one factor causing this is having two different versions of the same task of the same recipe in the cache, with the same output hash, but different task hashes. Multiconfigs may also be relevant (as that can result in multiple versions of the same recipe being added to the cache in a single build), but I believe the issue can also occur without MC if the same recipe is simply built again after its code has been modified. Looking at the task mc:k3r5:gcc-cross-arm:do_deploy_source_date_epoch in my two different build trees, it generates two different sstate files, but they should be identical: - sstate:gcc-cross-arm:x86_64-tq-eabi:13.3.0:r0:x86_64:12:0dd7cf5da28013c4267be46f97829709945cac262927a65d2ea53dadd312e20e_deploy_source_date_epoch.tar.zst - sstate:gcc-cross-arm:x86_64-tq-eabi:13.3.0:r0:x86_64:12:2db0d740e0900ad2d90ca460d18abaa65c8b262c172932f8f10f1dc9cbd65194_deploy_source_date_epoch.tar.zst Diffing the siginfo files shows the following differences: --- @@ -800,7 +800,7 @@ "mc:k3r5:gcc-source-13.3.0:do_deploy_source_date_epoch" ], "runtaskhashes": { - "mc:k3r5:gcc-source-13.3.0:do_deploy_source_date_epoch": "959982ab272c4a866dd1d33741ac3973c72f564c161b9b2b523906e8d67e5452" + "mc:k3r5:gcc-source-13.3.0:do_deploy_source_date_epoch": "df2992e0f9c5e2ca519a40825fe2d4c653fbe09ba5d0b0c6930a3b200d6edf4c" }, "task": "do_deploy_source_date_epoch", "taskdeps": { @@ -962,9 +962,9 @@ "systemd_user_unitdir" ] }, - "taskhash": "0dd7cf5da28013c4267be46f97829709945cac262927a65d2ea53dadd312e20e", + "taskhash": "2db0d740e0900ad2d90ca460d18abaa65c8b262c172932f8f10f1dc9cbd65194", "taskhash_ignore_tasks": null, - "unihash": "0dd7cf5da28013c4267be46f97829709945cac262927a65d2ea53dadd312e20e", + "unihash": "2db0d740e0900ad2d90ca460d18abaa65c8b262c172932f8f10f1dc9cbd65194", "varvals": { "AR": "${BUILD_AR}", "AS": "${BUILD_AS}", --- So two different versions of mc:k3r5:gcc-source-13.3.0:do_deploy_source_date_epoch have been used: - sstate:gcc-source-13.3.0:all-tq-linux:13.3.0:r0:all:12:959982ab272c4a866dd1d33741ac3973c72f564c161b9b2b523906e8d67e5452_deploy_source_date_epoch.tar.zst - sstate:gcc-source-13.3.0:all-tq-linux:13.3.0:r0:all:12:df2992e0f9c5e2ca519a40825fe2d4c653fbe09ba5d0b0c6930a3b200d6edf4c_deploy_source_date_epoch.tar.zst Diffing their siginfos shows a lot more differences throughout the whole file, so I'm showing only a small excerpt: --- @@ -901,10 +923,10 @@ } }, "runtaskdeps": [ - "mc:k3r5:gcc-source-13.3.0:do_patch" + "gcc-source-13.3.0:do_patch" ], "runtaskhashes": { - "mc:k3r5:gcc-source-13.3.0:do_patch": "d2e18e3897cb9d401dc27b9236274bcf1195ddc2a68662038dc29aabfa095d36" + "gcc-source-13.3.0:do_patch": "d2e18e3897cb9d401dc27b9236274bcf1195ddc2a68662038dc29aabfa095d36" }, "task": "do_deploy_source_date_epoch", "taskdeps": { --- One of these two sstates is not from mc:k3r5:gcc-source-13.3.0:do_deploy_source_date_epoch at all, but from gcc-source-13.3.0:do_deploy_source_date_epoch! And while the output of the two tasks is the same - resulting in a proper reproducible build even with the wrong sstate being used - my understanding is that the different task hash propagates through the entire build, resulting in a full rebuild (of the k3r5 multiconfig in this case, but I've also seen the same issue with the main config). [1] https://git.yoctoproject.org/meta-ti/ [2] https://github.com/tq-systems/meta-tq