All too frequently, somehow we manage to corrupt the buildhistory git repo. The corruption appears with errors like: https://valkyrie.yoctoproject.org/#/builders/39/builds/518 https://valkyrie.yoctoproject.org/#/builders/20/builds/510 We have some disagreement about whether this is an infrastructure issue, misuse of git or a genuine bug somewhere in git or in the git mirroring tooling being used. Somehow we need to work out how/why this occurs and stop it occurring.
A quick history lesson on what the code is doing and where. buildhistory is being used to track and compare how our output changes over time. We do this for multiple builders (qemux86, qemuarm, qemuarm64 and so on) in two different 'flavours' (normal and alt) against multiple branches. Each config has a unique name which maps to a branchname in poky-buildhistory. At the start of a build our init script is called: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/buildhistory-init which either starts a new repo or does: git fetch --depth=5 $REMOTEREPO refs/heads/$BASEBRANCH:refs/remotes/$BASEBRANCH git reset refs/remotes/$BASEBRANCH --hard We then configure the build: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/setup-config#n99 INHERIT += "buildhistory" BUILDHISTORY_DIR = "%s" BUILDHISTORY_PUSH_REPO = "%s%s %s:%s"' BUILDHISTORY_COMMIT = '1' which basically hands responsibly over to bitbake to push the data at the end of the build. That is this code: https://git.yoctoproject.org/poky/tree/meta/classes/buildhistory.bbclass#n848 which does some git tags, git commit, git gc (hmm) and finally a git push. We do then later produce a history comparison report in the results output: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/collect-results#n23 So the weird thing we do here is make a lot of shallow clones which get pushed back from multiple builds. That should work. Having noticed that "git gc" in there, I am left wondering why we do that. It shouldn't break anything, but... Mirroring on the git.yoctoproject.org side is done from push.yoctoproject.org using https://github.com/mricon/grokmirror
Bulk move to 5.2-M3.
Bulk move to 5.2-M3 and this time I'll actually specify M3!
This may be the upstream mirroring code rather than a YP bug.
Leaving at 5.2-M2 for discussion in bug review meeting.
This happened again on Monday August 19th. Konstantin has set up proactive alerts and I've flagged them so we can hopefully catch corruption before it causes build problems.
buildhistory hasn't experienced a corruption event since the size was reduced. I'm going to close this out for now.