How to reproduce: ``` git clone https://github.com/macpijan/meta-gitsm-lfs-bug.git SHELL=/bin/bash kas-docker build --target gitsm-lfs-bug-main-repo --task unpack ./meta-gitsm-lfs-bug/kas.yml ``` The error message: ``` ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: gitsm: submodule unpack failed: UnpackError Unpack failure for URL: 'gitsm://github.com/macpijan/gitsm-lfs-bug-submodule;protocol=https;name=submodule;subpath=submodule;bareclone=1;nobranch=1'. No up to date source found: clone directory not available or not up to date: /work/build/downloads/git2/github.com.macpijan.gitsm-lfs-bug-submodule; shallow clone not enabled WARNING: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Failed to fetch URL gitsm://github.com/macpijan/gitsm-lfs-bug-main-repo.git;protocol=https, attempting MIRRORS if available ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Unpack failure for URL: 'gitsm://github.com/macpijan/gitsm-lfs-bug-submodule;protocol=https;name=submodule;subpath=submodule;bareclone=1;nobranch=1'. No up to date source found: clone directory not available or not up to date: /work/build/downloads/git2/github.com.macpijan.gitsm-lfs-bug-submodule; shallow clone not enabled ERROR: gitsm-lfs-bug-main-repo-1.0+gitAUTOINC+b4be8b8e81-r0 do_fetch: Fetcher failure for URL: 'gitsm://github.com/macpijan/gitsm-lfs-bug-main-repo.git;protocol=https'. Unable to fetch URL from any source. ERROR: Logfile of failure stored in: /work/build/tmp/work/core2-64-poky-linux/gitsm-lfs-bug-main-repo/1.0+gitAUTOINC+b4be8b8e81-r0/temp/log.do_fetch.2900 ERROR: Task (/repo/recipes-gitsm/gitsm-lfs-bug-main-repo/gitsm-lfs-bug-main-repo_git.bb:do_fetch) failed with exit code '1' ``` Error reason: * The main repo is fetched before submodules get fetched - the method from git.py is called: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/gitsm.py?id=a7e117c8ed063eb11266de3f696d12a2b7b9708e#n201 * if the main repo contains LFS content, following condition is met and self.unpack() is called: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/git.py?id=a7e117c8ed063eb11266de3f696d12a2b7b9708e#n397 * self in this case refers to gitsm, so the unpack() from gitsm() is called * it tries to unpack submodules, which were not downloaded yet (they will be downloaded after we exit from the download() method of gitsm.py() Possible fix - below diff fixes fetching in this scenario, but I'm not certain whether it is a good one or whether it breaks anything else: ``` diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index e3ba80a3f52a..8f9a32552538 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -162,7 +162,7 @@ class Git(FetchMethod): ud.bareclone = ud.parm.get("bareclone","0") == "1" if ud.bareclone: ud.nocheckout = 1 - + ud.unresolvedrev = {} branches = ud.parm.get("branch", "master").split(',') if len(branches) != len(ud.names): @@ -394,7 +394,7 @@ class Git(FetchMethod): tmpdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) try: # Do the checkout. This implicitly involves a Git LFS fetch. - self.unpack(ud, tmpdir, d) + Git.unpack(self, ud, tmpdir, d) # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into # the bare clonedir. ```
Can you please send a patch to the oe-core list.
I encountered the same issue and applied the same fix. A patch has been submitted here: https://lists.openembedded.org/g/openembedded-core/message/150670
Thanks Niels!
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d12a06cd49b6cc13fcd15e56b5034de5ced642ee
There seems to be similar issue in vulkan-samples in oe-core, it's failing on LFS content in submodules: http://errors.yoctoproject.org/Errors/Details/657552/ The actual error (just reformatted): ``` Submodule path \'assets\': checked out \'db4f7ef1213f159a18b782b60c887210151a9c12\' Submodule path \'third_party/CTPL\': checked out \'437e135dbd94eb65b45533d9ce8ee28b5bd37b6d\' Submodule path \'third_party/astc\': checked out \'4861c3db1ca50eb5f5767d95d1cd1bacb98ac8a5\' Submodule path \'third_party/cli11\': checked out \'4af78beef777e313814b4daff70e2da9171a385a\' Submodule path \'third_party/glfw\': checked out \'6876cf8d7e0e70dc3e4d7b0224d08312c9f78099\' Submodule path \'third_party/glm\': checked out \'fce2abd01ce21063bd25ba67c9318be83bf48813\' Submodule path \'third_party/glslang\': checked out \'6baa85f6ceec5fddd2e13b43dd1edc944dd930af\' Submodule path \'third_party/hwcpipe\': checked out \'5a0c3d0b527bea96154998f29177f3b449cebd3a\' Submodule path \'third_party/imgui\': checked out \'c23a19c26fa9cc778f755e76799f0cafdcb5a13b\' Submodule path \'third_party/opencl\': checked out \'1aa1139b58a515877a923cce6b254e09d1b2fb2c\' Submodule path \'third_party/spdlog\': checked out \'7bf8f148795d2571b4ae4aa9c6d2a8111fbb7077\' Submodule path \'third_party/spirv-cross\': checked out \'84f56d0b43e5a6874de45e11be2bff9dc336fd7e\' Submodule path \'third_party/stb\': checked out \'2c2908f50515dcd939f24be261c3ccbcd277bb49\' Submodule path \'third_party/tinygltf\': checked out \'c0d0251e2c7f2273a894aa1b125af9313a88eae4\' Submodule path \'third_party/vma\': checked out \'f9921aefddee2437cc2e3303d3175bd8ef23e22c\' Submodule path \'third_party/volk\': checked out \'b18f3192886981f28d17f8f41c26c5a53ac3b977\' Submodule path \'third_party/vulkan\': checked out \'76f00ef6cbb1886eb1162d1fa39bee8b51e22ee8\' Downloading other_lib/ios/Debug-iphoneos/libSDL2.a (8.2 MB) Error downloading object: other_lib/ios/Debug-iphoneos/libSDL2.a (5045776): Smudge error: Error downloading other_lib/ios/Debug-iphoneos/libSDL2.a (5045776556ca57f2f7125d89c37a8b94f79cdebdc040d420513f0491787c4ea7): batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution Errors logged to \'/OE/build/poky/build-st/reproducibleA/tmp/work/core2-64-poky-linux/vulkan-samples/git-r0/git/.git/modules/third_party/ktx/lfs/logs/20220531T194140.927704073.log\'. Use `git lfs logs last` to view the log. error: external filter \'git-lfs filter-process\' failed fatal: other_lib/ios/Debug-iphoneos/libSDL2.a: smudge filter lfs failed fatal: Unable to checkout \'6a38a0694b4d73d22c5d1e22f865d03545e808ea\' in submodule path \'third_party/ktx\' ``` This was with: git version 2.35.1 git-lfs/3.1.4 (GitHub; linux amd64; go 1.18.2) and oe-core/master as of yesterday 18a0c31b33, bitbake 6a346df5 Have anyone else seen this? It should be triggered on autobuilder as well (I triggered it with poky, which I normally don't use, with just 'oe-selftest -r reproducible', but then reproduced the same with just oe-core), maybe it's specific to git+git-lfs versions from host? This is from Gentoo. Will retry with some ubuntu as well soon.
It's also reproducible in ubuntu-22.10 with: git version 2.36.1 git-lfs/3.2.0 (GitHub; linux amd64; go 1.18.2) But it's somehow related to what I had in .gitconfig (don't remember when or why I've added these, but it was long time ago), looks like first item from https://shuhrat.github.io/programming/git-lfs-tips-and-tricks.html : [filter "lfs"] smudge = git-lfs smudge -- %f process = git-lfs filter-process required = true clean = git-lfs clean -- %f If I keep only smudge, then the do_unpack takes terribly long, but doesn't fail: 172.53 seconds vulkan-samples-git-r0/do_fetch 5319.38 seconds vulkan-samples-git-r0/do_unpack If I remove whole lfs section from .gitconfig then it doesn't fail and is fast: 13.00 seconds vulkan-samples-git-r0/do_fetch 13.75 seconds vulkan-samples-git-r0/do_unpack With whole lfs section as shown above (or just smudge+required) it fails after some 90 seconds: 6.94 seconds vulkan-samples-git-r0/do_fetch 84.25 seconds vulkan-samples-git-r0/do_unpack