| Summary: | AB-INT: Step "Initialize buildhistory": Switched to a new branch 'poky/master/qemuarm' failure | ||
|---|---|---|---|
| Product: | [QA/Testing] Build Testing | Reporter: | Leonardo Sandoval <leonardo.sandoval> |
| Component: | general | Assignee: | Michael Halstead <mhalstead> |
| Status: | RESOLVED WORKSFORME | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alexandre.belloni, chee.yang.lee, leonardo.sandoval, luca.ceresoli, mhalstead, randy.macleod, richard.purdie |
| Version: | 0.0.0 | ||
| Target Milestone: | 4.1 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Leonardo Sandoval
2022-04-20 00:47:58 UTC
centos7-ty-4 │ centos7-ty-4 │ Non-SSD │ https://autobuilder.yoctoproject.org/typhoon/#builders/53/builds/5084/steps/11/logs/stdio qemuarm stream8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#builders/59/builds/5053/steps/10/logs/stdio qemux86 fedora34-ty-1 RP: Last time it happened we could restore the poky-buildhistory repo from git.yoctoproject.org back to push.yoctoproject.org iirc. It does raise the question of why this happens. I know that repo is rather large :/. MH: Last time this happened I ran "git clone --mirror https://git.yoctoproject.org/poky-buildhistory.git" as the gitolite3 user on push.yoctoproject.org, backed up the corrupt repo, and moved that one into place. Then I restored the gitolite hooks. I'll start on that now. MH: I spent quite a lot of time trying to figure out how the corruption occured without success last time. I'd appreciate any help figuring out what happened this time. I assume an interrupted push or maybe a push while repacking. This repo is 12G so anyone who wants a copy is in for a download. RP: 12G! The autobuilder only shallow clones the pieces it needs, appends to it and pushes the heads back. I suspect this pushes the repacking onto the server which probably could get interrupted whilst it is doing that. RP: The data in there is useful but I'm wondering if we can't archive it, which would stop it being repacked all the time and make things a little lighter for a while, it has taken quite a few years to grow to that size! MH: poky-buildhistory is corrupted and repaired fairly regularly. This is likely because shallow clones are hard on the server and interfere with the nightly repack job. Three times now this has caused corruption that needed manual intervention. We are going to set https://git-scm.com/docs/git-config/2.21.0#Documentation/git-config.txt-corefsyncObjectFiles to 1 for now and see if we learn anything useful about what's happening. MH: If the problem persists we will move the current pok-buildhistory to poky-buildhistory-archive and then create a new poky-buildhistory containing only to branch tips to attempt to mitigate the issue. I've repaired the repo again and added core.fsyncObjectFiles = 1 to the config to see if that helps at all. This corruption is probably do to load caused by shallow cloning plus the night repack jobs. If it occurs again and we do not have a better solution we will archive the current repo and create a new one only containing branch tips. https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5211/steps/10/logs/stdio tumbleweed-ty-3 qemumips64 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5218/steps/10/logs/stdio ubuntu2004-ty-1 SSD qemumips64 We made changes to the git server that seems to have resolved this problem (git updated). Re-open if seen again. YP Triage |