Bug 14283 - gitsm fetcher failure when LFS content is present in the main repo
Summary: gitsm fetcher failure when LFS content is present in the main repo
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 5.99
Hardware: All Multiple
: Medium+ normal
Target Milestone: 3.4 M1
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2021-03-07 20:46 UTC by Maciej Pijanowski
Modified: 2022-06-01 10:36 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Maciej Pijanowski 2021-03-07 20:46:34 UTC
How to reproduce:

```
git clone https://github.com/macpijan/meta-gitsm-lfs-bug.git
SHELL=/bin/bash kas-docker build --target gitsm-lfs-bug-main-repo --task unpack ./meta-gitsm-lfs-bug/kas.yml
```

The error message:

```
ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: gitsm: submodule unpack failed: UnpackError Unpack failure for URL: 'gitsm://github.com/macpijan/gitsm-lfs-bug-submodule;protocol=https;name=submodule;subpath=submodule;bareclone=1;nobranch=1'. No up to date source found: clone directory not available or not up to date: /work/build/downloads/git2/github.com.macpijan.gitsm-lfs-bug-submodule; shallow clone not enabled
WARNING: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Failed to fetch URL gitsm://github.com/macpijan/gitsm-lfs-bug-main-repo.git;protocol=https, attempting MIRRORS if available
ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Unpack failure for URL: 'gitsm://github.com/macpijan/gitsm-lfs-bug-submodule;protocol=https;name=submodule;subpath=submodule;bareclone=1;nobranch=1'. No up to date source found: clone directory not available or not up to date: /work/build/downloads/git2/github.com.macpijan.gitsm-lfs-bug-submodule; shallow clone not enabled
ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Fetcher failure for URL: 'gitsm://github.com/macpijan/gitsm-lfs-bug-main-repo.git;protocol=https'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /work/build/tmp/work/core2-64-poky-linux/gitsm-lfs-bug-main-repo/1.0+gitAUTOINC+b4be8b8e81-r0/temp/log.do_fetch.2900
ERROR: Task (/repo/recipes-gitsm/gitsm-lfs-bug-main-repo/gitsm-lfs-bug-main-repo_git.bb:do_fetch) failed with exit code '1'
```

Error reason:

* The main repo is fetched before submodules get fetched - the method from git.py is called: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/gitsm.py?id=a7e117c8ed063eb11266de3f696d12a2b7b9708e#n201
* if the main repo contains LFS content, following condition is met and self.unpack() is called: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/git.py?id=a7e117c8ed063eb11266de3f696d12a2b7b9708e#n397
* self in this case refers to gitsm, so the unpack() from gitsm() is called
* it tries to unpack submodules, which were not downloaded yet (they will be downloaded after we exit from the download() method of gitsm.py()

Possible fix - below diff fixes fetching in this scenario, but I'm not certain whether it is a good one or whether it breaks anything else:

```
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e3ba80a3f52a..8f9a32552538 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -162,7 +162,7 @@ class Git(FetchMethod):
         ud.bareclone = ud.parm.get("bareclone","0") == "1"
         if ud.bareclone:
             ud.nocheckout = 1
-  
+
         ud.unresolvedrev = {}
         branches = ud.parm.get("branch", "master").split(',')
         if len(branches) != len(ud.names):
@@ -394,7 +394,7 @@ class Git(FetchMethod):
             tmpdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR'))
             try:
                 # Do the checkout. This implicitly involves a Git LFS fetch.
-                self.unpack(ud, tmpdir, d)
+                Git.unpack(self, ud, tmpdir, d)
 
                 # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into
                 # the bare clonedir.
```
Comment 1 Randy MacLeod 2021-03-11 16:07:29 UTC
Can you please send a patch to the oe-core list.
Comment 2 Niels Avonds 2021-04-19 10:53:26 UTC
I encountered the same issue and applied the same fix. A patch has been submitted here: https://lists.openembedded.org/g/openembedded-core/message/150670
Comment 3 Richard Purdie 2021-04-19 11:03:11 UTC
Thanks Niels!
Comment 5 Martin Jansa 2022-06-01 07:18:04 UTC
There seems to be similar issue in vulkan-samples in oe-core, it's failing on LFS content in submodules:

http://errors.yoctoproject.org/Errors/Details/657552/

The actual error (just reformatted):
```
Submodule path \'assets\': checked out \'db4f7ef1213f159a18b782b60c887210151a9c12\'
Submodule path \'third_party/CTPL\': checked out \'437e135dbd94eb65b45533d9ce8ee28b5bd37b6d\'
Submodule path \'third_party/astc\': checked out \'4861c3db1ca50eb5f5767d95d1cd1bacb98ac8a5\'
Submodule path \'third_party/cli11\': checked out \'4af78beef777e313814b4daff70e2da9171a385a\'
Submodule path \'third_party/glfw\': checked out \'6876cf8d7e0e70dc3e4d7b0224d08312c9f78099\'
Submodule path \'third_party/glm\': checked out \'fce2abd01ce21063bd25ba67c9318be83bf48813\'
Submodule path \'third_party/glslang\': checked out \'6baa85f6ceec5fddd2e13b43dd1edc944dd930af\'
Submodule path \'third_party/hwcpipe\': checked out \'5a0c3d0b527bea96154998f29177f3b449cebd3a\'
Submodule path \'third_party/imgui\': checked out \'c23a19c26fa9cc778f755e76799f0cafdcb5a13b\'
Submodule path \'third_party/opencl\': checked out \'1aa1139b58a515877a923cce6b254e09d1b2fb2c\'
Submodule path \'third_party/spdlog\': checked out \'7bf8f148795d2571b4ae4aa9c6d2a8111fbb7077\'
Submodule path \'third_party/spirv-cross\': checked out \'84f56d0b43e5a6874de45e11be2bff9dc336fd7e\'
Submodule path \'third_party/stb\': checked out \'2c2908f50515dcd939f24be261c3ccbcd277bb49\'
Submodule path \'third_party/tinygltf\': checked out \'c0d0251e2c7f2273a894aa1b125af9313a88eae4\'
Submodule path \'third_party/vma\': checked out \'f9921aefddee2437cc2e3303d3175bd8ef23e22c\'
Submodule path \'third_party/volk\': checked out \'b18f3192886981f28d17f8f41c26c5a53ac3b977\'
Submodule path \'third_party/vulkan\': checked out \'76f00ef6cbb1886eb1162d1fa39bee8b51e22ee8\'
 
Downloading other_lib/ios/Debug-iphoneos/libSDL2.a (8.2 MB)
Error downloading object: other_lib/ios/Debug-iphoneos/libSDL2.a (5045776):
Smudge error: Error downloading other_lib/ios/Debug-iphoneos/libSDL2.a (5045776556ca57f2f7125d89c37a8b94f79cdebdc040d420513f0491787c4ea7):

batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution
 
Errors logged to \'/OE/build/poky/build-st/reproducibleA/tmp/work/core2-64-poky-linux/vulkan-samples/git-r0/git/.git/modules/third_party/ktx/lfs/logs/20220531T194140.927704073.log\'.

Use `git lfs logs last` to view the log.
error: external filter \'git-lfs filter-process\' failed
fatal: other_lib/ios/Debug-iphoneos/libSDL2.a: smudge filter lfs failed
fatal: Unable to checkout \'6a38a0694b4d73d22c5d1e22f865d03545e808ea\' in submodule path \'third_party/ktx\'
```

This was with:
git version 2.35.1
git-lfs/3.1.4 (GitHub; linux amd64; go 1.18.2)

and oe-core/master as of yesterday 18a0c31b33, bitbake 6a346df5

Have anyone else seen this? It should be triggered on autobuilder as well (I triggered it with poky, which I normally don't use, with just 'oe-selftest -r reproducible', but then reproduced the same with just oe-core), maybe it's specific to git+git-lfs versions from host? This is from Gentoo. Will retry with some ubuntu as well soon.
Comment 6 Martin Jansa 2022-06-01 10:36:48 UTC
It's also reproducible in ubuntu-22.10 with:
git version 2.36.1
git-lfs/3.2.0 (GitHub; linux amd64; go 1.18.2)

But it's somehow related to what I had in .gitconfig (don't remember when or why I've added these, but it was long time ago), looks like first item from https://shuhrat.github.io/programming/git-lfs-tips-and-tricks.html :

[filter "lfs"]
       smudge = git-lfs smudge -- %f
       process = git-lfs filter-process
       required = true
       clean = git-lfs clean -- %f

If I keep only smudge, then the do_unpack takes terribly long, but doesn't fail:
172.53 seconds vulkan-samples-git-r0/do_fetch
5319.38 seconds vulkan-samples-git-r0/do_unpack

If I remove whole lfs section from .gitconfig then it doesn't fail and is fast:
13.00 seconds vulkan-samples-git-r0/do_fetch
13.75 seconds vulkan-samples-git-r0/do_unpack

With whole lfs section as shown above (or just smudge+required) it fails after some 90 seconds:
6.94 seconds vulkan-samples-git-r0/do_fetch
84.25 seconds vulkan-samples-git-r0/do_unpack