<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>14283</bug_id>
          
          <creation_ts>2021-03-07 20:46:34 +0000</creation_ts>
          <short_desc>gitsm fetcher failure when LFS content is present in the main repo</short_desc>
          <delta_ts>2022-06-01 10:36:48 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>BitBake</product>
          <component>bitbake</component>
          <version>5.99</version>
          <rep_platform>All</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>3.4 M1</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Maciej Pijanowski">maciej.pijanowski</reporter>
          <assigned_to name="Richard Purdie">richard.purdie</assigned_to>
          <cc>Martin.Jansa</cc>
    
    <cc>niels</cc>
    
    <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
    
    <cc>randy.macleod</cc>
    
    <cc>richard.purdie</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>89513</commentid>
    <comment_count>0</comment_count>
    <who name="Maciej Pijanowski">maciej.pijanowski</who>
    <bug_when>2021-03-07 20:46:34 +0000</bug_when>
    <thetext>How to reproduce:

```
git clone https://github.com/macpijan/meta-gitsm-lfs-bug.git
SHELL=/bin/bash kas-docker build --target gitsm-lfs-bug-main-repo --task unpack ./meta-gitsm-lfs-bug/kas.yml
```

The error message:

```
ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: gitsm: submodule unpack failed: UnpackError Unpack failure for URL: &apos;gitsm://github.com/macpijan/gitsm-lfs-bug-submodule;protocol=https;name=submodule;subpath=submodule;bareclone=1;nobranch=1&apos;. No up to date source found: clone directory not available or not up to date: /work/build/downloads/git2/github.com.macpijan.gitsm-lfs-bug-submodule; shallow clone not enabled
WARNING: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Failed to fetch URL gitsm://github.com/macpijan/gitsm-lfs-bug-main-repo.git;protocol=https, attempting MIRRORS if available
ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Unpack failure for URL: &apos;gitsm://github.com/macpijan/gitsm-lfs-bug-submodule;protocol=https;name=submodule;subpath=submodule;bareclone=1;nobranch=1&apos;. No up to date source found: clone directory not available or not up to date: /work/build/downloads/git2/github.com.macpijan.gitsm-lfs-bug-submodule; shallow clone not enabled
ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Fetcher failure for URL: &apos;gitsm://github.com/macpijan/gitsm-lfs-bug-main-repo.git;protocol=https&apos;. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /work/build/tmp/work/core2-64-poky-linux/gitsm-lfs-bug-main-repo/1.0+gitAUTOINC+b4be8b8e81-r0/temp/log.do_fetch.2900
ERROR: Task (/repo/recipes-gitsm/gitsm-lfs-bug-main-repo/gitsm-lfs-bug-main-repo_git.bb:do_fetch) failed with exit code &apos;1&apos;
```

Error reason:

* The main repo is fetched before submodules get fetched - the method from git.py is called: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/gitsm.py?id=a7e117c8ed063eb11266de3f696d12a2b7b9708e#n201
* if the main repo contains LFS content, following condition is met and self.unpack() is called: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/git.py?id=a7e117c8ed063eb11266de3f696d12a2b7b9708e#n397
* self in this case refers to gitsm, so the unpack() from gitsm() is called
* it tries to unpack submodules, which were not downloaded yet (they will be downloaded after we exit from the download() method of gitsm.py()

Possible fix - below diff fixes fetching in this scenario, but I&apos;m not certain whether it is a good one or whether it breaks anything else:

```
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e3ba80a3f52a..8f9a32552538 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -162,7 +162,7 @@ class Git(FetchMethod):
         ud.bareclone = ud.parm.get(&quot;bareclone&quot;,&quot;0&quot;) == &quot;1&quot;
         if ud.bareclone:
             ud.nocheckout = 1
-  
+
         ud.unresolvedrev = {}
         branches = ud.parm.get(&quot;branch&quot;, &quot;master&quot;).split(&apos;,&apos;)
         if len(branches) != len(ud.names):
@@ -394,7 +394,7 @@ class Git(FetchMethod):
             tmpdir = tempfile.mkdtemp(dir=d.getVar(&apos;DL_DIR&apos;))
             try:
                 # Do the checkout. This implicitly involves a Git LFS fetch.
-                self.unpack(ud, tmpdir, d)
+                Git.unpack(self, ud, tmpdir, d)
 
                 # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into
                 # the bare clonedir.
```</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>89586</commentid>
    <comment_count>1</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2021-03-11 16:07:29 +0000</bug_when>
    <thetext>Can you please send a patch to the oe-core list.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>89999</commentid>
    <comment_count>2</comment_count>
    <who name="Niels Avonds">niels</who>
    <bug_when>2021-04-19 10:53:26 +0000</bug_when>
    <thetext>I encountered the same issue and applied the same fix. A patch has been submitted here: https://lists.openembedded.org/g/openembedded-core/message/150670</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>90000</commentid>
    <comment_count>3</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2021-04-19 11:03:11 +0000</bug_when>
    <thetext>Thanks Niels!</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>90025</commentid>
    <comment_count>4</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2021-04-20 13:09:37 +0000</bug_when>
    <thetext>http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d12a06cd49b6cc13fcd15e56b5034de5ced642ee</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>93358</commentid>
    <comment_count>5</comment_count>
    <who name="Martin Jansa">Martin.Jansa</who>
    <bug_when>2022-06-01 07:18:04 +0000</bug_when>
    <thetext>There seems to be similar issue in vulkan-samples in oe-core, it&apos;s failing on LFS content in submodules:

http://errors.yoctoproject.org/Errors/Details/657552/

The actual error (just reformatted):
```
Submodule path \&apos;assets\&apos;: checked out \&apos;db4f7ef1213f159a18b782b60c887210151a9c12\&apos;
Submodule path \&apos;third_party/CTPL\&apos;: checked out \&apos;437e135dbd94eb65b45533d9ce8ee28b5bd37b6d\&apos;
Submodule path \&apos;third_party/astc\&apos;: checked out \&apos;4861c3db1ca50eb5f5767d95d1cd1bacb98ac8a5\&apos;
Submodule path \&apos;third_party/cli11\&apos;: checked out \&apos;4af78beef777e313814b4daff70e2da9171a385a\&apos;
Submodule path \&apos;third_party/glfw\&apos;: checked out \&apos;6876cf8d7e0e70dc3e4d7b0224d08312c9f78099\&apos;
Submodule path \&apos;third_party/glm\&apos;: checked out \&apos;fce2abd01ce21063bd25ba67c9318be83bf48813\&apos;
Submodule path \&apos;third_party/glslang\&apos;: checked out \&apos;6baa85f6ceec5fddd2e13b43dd1edc944dd930af\&apos;
Submodule path \&apos;third_party/hwcpipe\&apos;: checked out \&apos;5a0c3d0b527bea96154998f29177f3b449cebd3a\&apos;
Submodule path \&apos;third_party/imgui\&apos;: checked out \&apos;c23a19c26fa9cc778f755e76799f0cafdcb5a13b\&apos;
Submodule path \&apos;third_party/opencl\&apos;: checked out \&apos;1aa1139b58a515877a923cce6b254e09d1b2fb2c\&apos;
Submodule path \&apos;third_party/spdlog\&apos;: checked out \&apos;7bf8f148795d2571b4ae4aa9c6d2a8111fbb7077\&apos;
Submodule path \&apos;third_party/spirv-cross\&apos;: checked out \&apos;84f56d0b43e5a6874de45e11be2bff9dc336fd7e\&apos;
Submodule path \&apos;third_party/stb\&apos;: checked out \&apos;2c2908f50515dcd939f24be261c3ccbcd277bb49\&apos;
Submodule path \&apos;third_party/tinygltf\&apos;: checked out \&apos;c0d0251e2c7f2273a894aa1b125af9313a88eae4\&apos;
Submodule path \&apos;third_party/vma\&apos;: checked out \&apos;f9921aefddee2437cc2e3303d3175bd8ef23e22c\&apos;
Submodule path \&apos;third_party/volk\&apos;: checked out \&apos;b18f3192886981f28d17f8f41c26c5a53ac3b977\&apos;
Submodule path \&apos;third_party/vulkan\&apos;: checked out \&apos;76f00ef6cbb1886eb1162d1fa39bee8b51e22ee8\&apos;
 
Downloading other_lib/ios/Debug-iphoneos/libSDL2.a (8.2 MB)
Error downloading object: other_lib/ios/Debug-iphoneos/libSDL2.a (5045776):
Smudge error: Error downloading other_lib/ios/Debug-iphoneos/libSDL2.a (5045776556ca57f2f7125d89c37a8b94f79cdebdc040d420513f0491787c4ea7):

batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post &apos;https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch&apos;: dial tcp: lookup github.com: Temporary failure in name resolution
 
Errors logged to \&apos;/OE/build/poky/build-st/reproducibleA/tmp/work/core2-64-poky-linux/vulkan-samples/git-r0/git/.git/modules/third_party/ktx/lfs/logs/20220531T194140.927704073.log\&apos;.

Use `git lfs logs last` to view the log.
error: external filter \&apos;git-lfs filter-process\&apos; failed
fatal: other_lib/ios/Debug-iphoneos/libSDL2.a: smudge filter lfs failed
fatal: Unable to checkout \&apos;6a38a0694b4d73d22c5d1e22f865d03545e808ea\&apos; in submodule path \&apos;third_party/ktx\&apos;
```

This was with:
git version 2.35.1
git-lfs/3.1.4 (GitHub; linux amd64; go 1.18.2)

and oe-core/master as of yesterday 18a0c31b33, bitbake 6a346df5

Have anyone else seen this? It should be triggered on autobuilder as well (I triggered it with poky, which I normally don&apos;t use, with just &apos;oe-selftest -r reproducible&apos;, but then reproduced the same with just oe-core), maybe it&apos;s specific to git+git-lfs versions from host? This is from Gentoo. Will retry with some ubuntu as well soon.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>93359</commentid>
    <comment_count>6</comment_count>
    <who name="Martin Jansa">Martin.Jansa</who>
    <bug_when>2022-06-01 10:36:48 +0000</bug_when>
    <thetext>It&apos;s also reproducible in ubuntu-22.10 with:
git version 2.36.1
git-lfs/3.2.0 (GitHub; linux amd64; go 1.18.2)

But it&apos;s somehow related to what I had in .gitconfig (don&apos;t remember when or why I&apos;ve added these, but it was long time ago), looks like first item from https://shuhrat.github.io/programming/git-lfs-tips-and-tricks.html :

[filter &quot;lfs&quot;]
       smudge = git-lfs smudge -- %f
       process = git-lfs filter-process
       required = true
       clean = git-lfs clean -- %f

If I keep only smudge, then the do_unpack takes terribly long, but doesn&apos;t fail:
172.53 seconds vulkan-samples-git-r0/do_fetch
5319.38 seconds vulkan-samples-git-r0/do_unpack

If I remove whole lfs section from .gitconfig then it doesn&apos;t fail and is fast:
13.00 seconds vulkan-samples-git-r0/do_fetch
13.75 seconds vulkan-samples-git-r0/do_unpack

With whole lfs section as shown above (or just smudge+required) it fails after some 90 seconds:
6.94 seconds vulkan-samples-git-r0/do_fetch
84.25 seconds vulkan-samples-git-r0/do_unpack</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>