Created attachment 4700 [details] sample siginfo files showing the problem Building using SSTATE_MIRRORS providing 99% of the artifacts still occasionally results in the local sstate-cache being loaded with rm_work.tgz.siginfo files. It is likely the cause of a behaviour I have observed even without SSTATE_MIRRORS, which shows virtually all such files to be retransfered when rsync'ing the sstate for public consumption - the SSTATE_MIRRORS makes it easier to characterize, though. bitbake-dumpsigs|sort seems to only show differences in ordering of the dependency lists. See example in attachment: $ ls -l sstate-cache-3/a6/sstate:libsndfile1:dbfp4-shadow-linux:1.0.28:r0:dbfp4:3:a62ff980ff61c8724c91a5a8c281295e_rm_work.tgz.siginfo sstate-cache-3+dev/a6/sstate:libsndfile1:dbfp4-shadow-linux:1.0.28:r0:dbfp4:3:a62ff980ff61c8724c91a5a8c281295e_rm_work.tgz.siginfo -rw-rw-r-- 1 gitlab-runner gitlab-runner 22019 Jul 7 11:52 sstate-cache-3/a6/sstate:libsndfile1:dbfp4-shadow-linux:1.0.28:r0:dbfp4:3:a62ff980ff61c8724c91a5a8c281295e_rm_work.tgz.siginfo -rw-rw-r-- 1 gitlab-runner gitlab-runner 22043 Jul 7 12:52 sstate-cache-3+dev/a6/sstate:libsndfile1:dbfp4-shadow-linux:1.0.28:r0:dbfp4:3:a62ff980ff61c8724c91a5a8c281295e_rm_work.tgz.siginfo $ bitbake-diffsigs ... NOTE: Starting bitbake server... $
Can you give a little more detail about what you believe the issue is here please? I'm struggling to understand what the problem is exactly.
Having different sigdata contents causes extra network traffic when rsyncing sstate. Now if there was a difference for the tools I wouldn't object, but the fact that bitbake-diffsigs claims to see no difference hints for a bug. Additionally, this hides in a haystack those sigdata that really changed: if I exlude *siginfo* from a transfer to check for changes that should not be there, I'll miss those that really change for a reason. Here I have a base sstate from the OS team, and an app layer whose artifacts get into their own sstate. This app sstate should not contain stuff coming from the app layer, though it is cluttered with hundred of siginfo files that look essentially identical to that in SSTATE_MIRROR.
Ok, thanks for the info. Please try and clearly explain the issue in bugs as I didn't understand this one at first. To restate the issue, the siginfo files are not being written deterministically. This leads to the same file being written with different checksums which in turn causes your multiple rsync issue. From a quick look at the files, the issue looks like a sorting issue in the dictionaries stored inside the files. The data is the same, pickle is just changing the order (python dicts are unsorted). I need to look further at that to confirm but it seems like a good place to start debugging from.
This is looking like it is dict ordering and is hard to fix as either we change all the dicts to ordered dicts, created with sorting before pickling, or we change from pickle to json format, both of which are fairly invasive. I do want to clarify that files with the same filename should always be identical as far as diffsigs is concerned. The sstate code simply won't overwrite any existing files for that reason. There should be therefore no need to overwrite files which already exist in an sstate cache (and out autobuilder and default code never does on recent releases).
A less invasive option may be, when the hash is the same as in sstate, to import the pre-existing siginfo and compare it to the one we're about to read. But don't we have a canonical sorting order, used to compute the hash value ?
We do have a sorting used to compute the hashes so the hashes themselves are fine. When we store the data in dicts and write to disk, python dicts are unsorted though so the on disk format is variable :(.
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=4c33a6e52be775265c871aa2d3b87ed23a0d48da