Bug 15665

Summary:	buildhistory git repo keeps getting corrupted
Product:	[Infrastructure] AutoBuilder	Reporter:	Richard Purdie <richard.purdie>
Component:	autobuilder	Assignee:	Unassigned <unassigned>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	infras.ab.watcher, Infras.watcher, mhalstead, paulg, pidge, randy.macleod, yoann.congal
Version:	unspecified
Target Milestone:	5.3 M4
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Richard Purdie 2024-11-26 22:01:25 UTC

All too frequently, somehow we manage to corrupt the buildhistory git repo. The corruption appears with errors like:

https://valkyrie.yoctoproject.org/#/builders/39/builds/518
https://valkyrie.yoctoproject.org/#/builders/20/builds/510

We have some disagreement about whether this is an infrastructure issue, misuse of git or a genuine bug somewhere in git or in the git mirroring tooling being used.

Somehow we need to work out how/why this occurs and stop it occurring.

Comment 1 Richard Purdie 2024-11-26 22:15:53 UTC

A quick history lesson on what the code is doing and where. buildhistory is being used to track and compare how our output changes over time. We do this for multiple builders (qemux86, qemuarm, qemuarm64 and so on) in two different 'flavours' (normal and alt) against multiple branches. Each config has a unique name which maps to a branchname in poky-buildhistory.

At the start of a build our init script is called:

https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/buildhistory-init

which either starts a new repo or does:

git fetch --depth=5 $REMOTEREPO refs/heads/$BASEBRANCH:refs/remotes/$BASEBRANCH
git reset refs/remotes/$BASEBRANCH --hard

We then configure the build:

https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/setup-config#n99

INHERIT += "buildhistory"
BUILDHISTORY_DIR = "%s"
BUILDHISTORY_PUSH_REPO = "%s%s %s:%s"'
BUILDHISTORY_COMMIT = '1'

which basically hands responsibly over to bitbake to push the data at
the end of the build. That is this code:

https://git.yoctoproject.org/poky/tree/meta/classes/buildhistory.bbclass#n848

which does some git tags, git commit, git gc (hmm) and finally a git
push.

We do then later produce a history comparison report in the results output:

https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/collect-results#n23

So the weird thing we do here is make a lot of shallow clones which get
pushed back from multiple builds. That should work. Having noticed that
"git gc" in there, I am left wondering why we do that. It shouldn't
break anything, but...

Mirroring on the git.yoctoproject.org side is done from push.yoctoproject.org using https://github.com/mricon/grokmirror

Comment 2 Randy MacLeod 2025-01-30 16:06:36 UTC

Bulk move to 5.2-M3.

Comment 3 Randy MacLeod 2025-01-30 16:17:04 UTC

Bulk move to 5.2-M3 and this time I'll actually specify M3!

Comment 4 Randy MacLeod 2025-03-27 15:03:27 UTC

This may be the upstream mirroring code rather than a YP bug.

Comment 5 Randy MacLeod 2025-08-07 18:20:34 UTC

Leaving at 5.2-M2 for discussion in bug review meeting.

Comment 6 Michael Halstead 2025-08-21 15:16:07 UTC

This happened again on Monday August 19th. Konstantin has set up proactive alerts and I've flagged them so we can hopefully catch corruption before it causes build problems.

Comment 7 Michael Halstead 2026-01-22 19:09:45 UTC

buildhistory hasn't experienced a corruption event since the size was reduced. I'm going to close this out for now.