14791 2022-04-20 00:47:58 +0000 AB-INT: Step "Initialize buildhistory": Switched to a new branch 'poky/master/qemuarm' failure 2022-08-18 15:08:43 +0000 1 1 10 QA/Testing Build Testing general 0.0.0 x86 Multiple RESOLVED WORKSFORME AB-INT Medium+ normal 4.1 M2 1 leonardo.sandoval mhalstead alexandre.belloni chee.yang.lee leonardo.sandoval luca.ceresoli mhalstead randy.macleod richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 93128 0 leonardo.sandoval 2022-04-20 00:47:58 +0000 ==================================================================================================== Target task qemuarm has 3 steps (1650330766.2: 0.0) ==================================================================================================== hint: Using 'master' as the name for the initial branch. This default branch name hint: is subject to change. To configure the initial branch name to use in all hint: of your new repositories, which will suppress this warning, call: hint: hint: git config --global init.defaultBranch <name> hint: hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and hint: 'development'. The just-created branch can be renamed via this command: hint: hint: git branch -m <name> Initialized empty Git repository in /home/pokybuild/yocto-worker/qemuarm/build/build/buildhistory/.git/ Switched to a new branch 'poky/master/qemuarm' remote: fatal: unable to read 802b378917f28f780d8ace161e421eb2c58f6074 error: git upload-pack: git-pack-objects died with error. fatal: git upload-pack: aborting due to possible repository corruption on the remote side. remote: aborting due to possible repository corruption on the remote side. fatal: protocol error: bad pack header fatal: ambiguous argument 'refs/remotes/poky/master/qemuarm': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' fatal: Failed to resolve 'HEAD' as a valid ref. Using BB_LOGCONFIG=/home/pokybuild/yocto-worker/qemuarm/build/build/../bitbake/contrib/autobuilderlog.json ERROR: Command ['/home/pokybuild/yocto-worker/qemuarm/yocto-autobuilder-helper/scripts/buildhistory-init', '/home/pokybuild/yocto-worker/qemuarm/build/build/buildhistory', 'ssh://git@push.yoctoproject.org/poky-buildhistory', 'poky/master/qemuarm', 'poky/master/qemuarm'] failed program finished with exit code 0 elapsedTime=7.559600 93129 1 leonardo.sandoval 2022-04-20 00:48:33 +0000 centos7-ty-4 93137 2 leonardo.sandoval 2022-04-20 22:33:36 +0000 │ centos7-ty-4 │ Non-SSD │ 93161 3 alexandre.belloni 2022-04-21 17:24:51 +0000 https://autobuilder.yoctoproject.org/typhoon/#builders/53/builds/5084/steps/11/logs/stdio qemuarm stream8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#builders/59/builds/5053/steps/10/logs/stdio qemux86 fedora34-ty-1 93162 4 richard.purdie 2022-04-21 19:35:44 +0000 RP: Last time it happened we could restore the poky-buildhistory repo from git.yoctoproject.org back to push.yoctoproject.org iirc. It does raise the question of why this happens. I know that repo is rather large :/. MH: Last time this happened I ran "git clone --mirror https://git.yoctoproject.org/poky-buildhistory.git" as the gitolite3 user on push.yoctoproject.org, backed up the corrupt repo, and moved that one into place. Then I restored the gitolite hooks. I'll start on that now. MH: I spent quite a lot of time trying to figure out how the corruption occured without success last time. I'd appreciate any help figuring out what happened this time. I assume an interrupted push or maybe a push while repacking. This repo is 12G so anyone who wants a copy is in for a download. RP: 12G! The autobuilder only shallow clones the pieces it needs, appends to it and pushes the heads back. I suspect this pushes the repacking onto the server which probably could get interrupted whilst it is doing that. RP: The data in there is useful but I'm wondering if we can't archive it, which would stop it being repacked all the time and make things a little lighter for a while, it has taken quite a few years to grow to that size! MH: poky-buildhistory is corrupted and repaired fairly regularly. This is likely because shallow clones are hard on the server and interfere with the nightly repack job. Three times now this has caused corruption that needed manual intervention. We are going to set https://git-scm.com/docs/git-config/2.21.0#Documentation/git-config.txt-corefsyncObjectFiles to 1 for now and see if we learn anything useful about what's happening. MH: If the problem persists we will move the current pok-buildhistory to poky-buildhistory-archive and then create a new poky-buildhistory containing only to branch tips to attempt to mitigate the issue. 93167 5 mhalstead 2022-04-21 22:20:54 +0000 I've repaired the repo again and added core.fsyncObjectFiles = 1 to the config to see if that helps at all. This corruption is probably do to load caused by shallow cloning plus the night repack jobs. If it occurs again and we do not have a better solution we will archive the current repo and create a new one only containing branch tips. 93307 6 chee.yang.lee 2022-05-25 01:47:04 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5211/steps/10/logs/stdio tumbleweed-ty-3 qemumips64 93313 7 chee.yang.lee 2022-05-25 14:12:50 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5218/steps/10/logs/stdio ubuntu2004-ty-1 SSD qemumips64 93853 8 randy.macleod 2022-08-18 15:08:43 +0000 We made changes to the git server that seems to have resolved this problem (git updated). Re-open if seen again. YP Triage