Bug 11211

Summary: Git clones to NAS are very slow and have significant impact on build times
Product: [Infrastructure] AutoBuilder Reporter: Joshua Lock <joshuagloe>
Component: autobuilderAssignee: Michael Halstead <mhalstead>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: Medium CC: infras.ab.watcher, Infras.watcher, pidge
Version: 2.4   
Target Milestone: 4.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Joshua Lock 2017-03-22 14:29:28 UTC
We are observing *very* slow git clones of linux-yocto during oe-selftest on both clusters.

In #10612 we moved cloning of poky.git off the NAS, however that workaround isn't tenable for clones made by the build-system itself.

This is easily reproduced by logging into one of the servers and comparing the speed of a clone on the local filesystem vs one on the NAS:


git -c core.fsyncobjectfiles=0 clone --bare --mirror git://git.yoctoproject.org/linux-yocto-4.10.git /home/pokybuild/git.yoctoproject.org.linux-yocto-4.10.git --progress
Receiving objects: 100% (5195778/5195778), 1.04 GiB | 11.62 MiB/s, done.


git -c core.fsyncobjectfiles=0 clone --bare --mirror git://git.yoctoproject.org/linux-yocto-4.10.git /srv/autobuilder/autobuilder.yoctoproject.org/current_sources/rptest/git.yoctoproject.org.linux-yocto-4.10.git --progress
Receiving objects:   0% (45737/5195778), 20.53 MiB | 325.00 KiB/s

Speeds are significantly slower and have a negative impact on the overall build times.
Comment 1 Michael Halstead 2017-04-10 15:20:08 UTC
What restrictions are there on clones made by the build-system itself? They have to be done in the SOURCE_PUBLISH_DIR or SOURCE_DL_DIR specifically?

We need attribute caching turned off to keep our downloads from racing but it looks like we will need attribute caching on for large git clones.
Comment 2 Joshua Lock 2017-04-12 13:38:07 UTC
Lowering the urgency of this and pushing out to 2.4 as we have mitigated the impact of this by removing the need to re-clone the repository on each oe-selftest run.
Comment 3 Michael Halstead 2017-08-12 00:11:37 UTC
The NAS storage is performing very well in testing. All clients are now using nfs 4.2. Does this resolve the issue? 

If not my research indicates that git will always be very slow when working over nfs unless we cache stat data longer. We have seen intermittent build failures in the past when allowing caching of stat data. I don't recall exactly why but I think some processes use that data to avoid race conditions.
Comment 4 Michael Halstead 2018-05-24 13:56:01 UTC
I resolving this bug because our workaround is functional and the direct solutions cause critical issues.