Bug 11211 - Git clones to NAS are very slow and have significant impact on build times
Summary: Git clones to NAS are very slow and have significant impact on build times
Status: RESOLVED FIXED
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: 2.4
Hardware: x86 Multiple
: Medium major
Target Milestone: 4.99
Assignee: Michael Halstead
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-03-22 14:29 UTC by Joshua Lock
Modified: 2018-05-24 13:56 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joshua Lock 2017-03-22 14:29:28 UTC
We are observing *very* slow git clones of linux-yocto during oe-selftest on both clusters.

In #10612 we moved cloning of poky.git off the NAS, however that workaround isn't tenable for clones made by the build-system itself.

This is easily reproduced by logging into one of the servers and comparing the speed of a clone on the local filesystem vs one on the NAS:


git -c core.fsyncobjectfiles=0 clone --bare --mirror git://git.yoctoproject.org/linux-yocto-4.10.git /home/pokybuild/git.yoctoproject.org.linux-yocto-4.10.git --progress
Receiving objects: 100% (5195778/5195778), 1.04 GiB | 11.62 MiB/s, done.


git -c core.fsyncobjectfiles=0 clone --bare --mirror git://git.yoctoproject.org/linux-yocto-4.10.git /srv/autobuilder/autobuilder.yoctoproject.org/current_sources/rptest/git.yoctoproject.org.linux-yocto-4.10.git --progress
Receiving objects:   0% (45737/5195778), 20.53 MiB | 325.00 KiB/s

Speeds are significantly slower and have a negative impact on the overall build times.
Comment 1 Michael Halstead 2017-04-10 15:20:08 UTC
What restrictions are there on clones made by the build-system itself? They have to be done in the SOURCE_PUBLISH_DIR or SOURCE_DL_DIR specifically?

We need attribute caching turned off to keep our downloads from racing but it looks like we will need attribute caching on for large git clones.
Comment 2 Joshua Lock 2017-04-12 13:38:07 UTC
Lowering the urgency of this and pushing out to 2.4 as we have mitigated the impact of this by removing the need to re-clone the repository on each oe-selftest run.
Comment 3 Michael Halstead 2017-08-12 00:11:37 UTC
The NAS storage is performing very well in testing. All clients are now using nfs 4.2. Does this resolve the issue? 

If not my research indicates that git will always be very slow when working over nfs unless we cache stat data longer. We have seen intermittent build failures in the past when allowing caching of stat data. I don't recall exactly why but I think some processes use that data to avoid race conditions.
Comment 4 Michael Halstead 2018-05-24 13:56:01 UTC
I resolving this bug because our workaround is functional and the direct solutions cause critical issues.