<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>11211</bug_id>
          
          <creation_ts>2017-03-22 14:29:28 +0000</creation_ts>
          <short_desc>Git clones to NAS are very slow and have significant impact on build times</short_desc>
          <delta_ts>2018-05-24 13:56:01 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>5</classification_id>
          <classification>Infrastructure</classification>
          <product>AutoBuilder</product>
          <component>autobuilder</component>
          <version>2.4</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium</priority>
          <bug_severity>major</bug_severity>
          <target_milestone>4.99</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Joshua Lock">joshuagloe</reporter>
          <assigned_to name="Michael Halstead">mhalstead</assigned_to>
          <cc>infras.ab.watcher</cc>
    
    <cc>Infras.watcher</cc>
    
    <cc>pidge</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>71574</commentid>
    <comment_count>0</comment_count>
    <who name="Joshua Lock">joshuagloe</who>
    <bug_when>2017-03-22 14:29:28 +0000</bug_when>
    <thetext>We are observing *very* slow git clones of linux-yocto during oe-selftest on both clusters.

In #10612 we moved cloning of poky.git off the NAS, however that workaround isn&apos;t tenable for clones made by the build-system itself.

This is easily reproduced by logging into one of the servers and comparing the speed of a clone on the local filesystem vs one on the NAS:


git -c core.fsyncobjectfiles=0 clone --bare --mirror git://git.yoctoproject.org/linux-yocto-4.10.git /home/pokybuild/git.yoctoproject.org.linux-yocto-4.10.git --progress
Receiving objects: 100% (5195778/5195778), 1.04 GiB | 11.62 MiB/s, done.


git -c core.fsyncobjectfiles=0 clone --bare --mirror git://git.yoctoproject.org/linux-yocto-4.10.git /srv/autobuilder/autobuilder.yoctoproject.org/current_sources/rptest/git.yoctoproject.org.linux-yocto-4.10.git --progress
Receiving objects:   0% (45737/5195778), 20.53 MiB | 325.00 KiB/s

Speeds are significantly slower and have a negative impact on the overall build times.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>72210</commentid>
    <comment_count>1</comment_count>
    <who name="Michael Halstead">mhalstead</who>
    <bug_when>2017-04-10 15:20:08 +0000</bug_when>
    <thetext>What restrictions are there on clones made by the build-system itself? They have to be done in the SOURCE_PUBLISH_DIR or SOURCE_DL_DIR specifically?

We need attribute caching turned off to keep our downloads from racing but it looks like we will need attribute caching on for large git clones.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>72382</commentid>
    <comment_count>2</comment_count>
    <who name="Joshua Lock">joshuagloe</who>
    <bug_when>2017-04-12 13:38:07 +0000</bug_when>
    <thetext>Lowering the urgency of this and pushing out to 2.4 as we have mitigated the impact of this by removing the need to re-clone the repository on each oe-selftest run.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>75824</commentid>
    <comment_count>3</comment_count>
    <who name="Michael Halstead">mhalstead</who>
    <bug_when>2017-08-12 00:11:37 +0000</bug_when>
    <thetext>The NAS storage is performing very well in testing. All clients are now using nfs 4.2. Does this resolve the issue? 

If not my research indicates that git will always be very slow when working over nfs unless we cache stat data longer. We have seen intermittent build failures in the past when allowing caching of stat data. I don&apos;t recall exactly why but I think some processes use that data to avoid race conditions.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>80636</commentid>
    <comment_count>4</comment_count>
    <who name="Michael Halstead">mhalstead</who>
    <bug_when>2018-05-24 13:56:01 +0000</bug_when>
    <thetext>I resolving this bug because our workaround is functional and the direct solutions cause critical issues.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>