14938 2022-10-17 12:24:43 +0000 gitsm fetcher failure when LFS content is present and git smudge fails 2023-09-26 15:55:39 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 4.3 x86 Multiple RESOLVED WONTFIX Medium+ normal Future 1 Martin.Jansa Martin.Jansa michael.opdenacker poky.bs.watcher poky.watcher randy.macleod --- --- Don't know oldest_to_newest 94190 0 Martin.Jansa 2022-10-17 12:24:43 +0000 I've already reported this behavior in: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14283#c5 but it was already fixed and other people were reporting the same e.g.: https://lists.openembedded.org/g/openembedded-core/topic/93089137#169500 From the #14283: Martin Jansa 2022-06-01 07:18:04 UTC There seems to be similar issue in vulkan-samples in oe-core, it's failing on LFS content in submodules: http://errors.yoctoproject.org/Errors/Details/657552/ The actual error (just reformatted): ``` Submodule path \'assets\': checked out \'db4f7ef1213f159a18b782b60c887210151a9c12\' Submodule path \'third_party/CTPL\': checked out \'437e135dbd94eb65b45533d9ce8ee28b5bd37b6d\' Submodule path \'third_party/astc\': checked out \'4861c3db1ca50eb5f5767d95d1cd1bacb98ac8a5\' Submodule path \'third_party/cli11\': checked out \'4af78beef777e313814b4daff70e2da9171a385a\' Submodule path \'third_party/glfw\': checked out \'6876cf8d7e0e70dc3e4d7b0224d08312c9f78099\' Submodule path \'third_party/glm\': checked out \'fce2abd01ce21063bd25ba67c9318be83bf48813\' Submodule path \'third_party/glslang\': checked out \'6baa85f6ceec5fddd2e13b43dd1edc944dd930af\' Submodule path \'third_party/hwcpipe\': checked out \'5a0c3d0b527bea96154998f29177f3b449cebd3a\' Submodule path \'third_party/imgui\': checked out \'c23a19c26fa9cc778f755e76799f0cafdcb5a13b\' Submodule path \'third_party/opencl\': checked out \'1aa1139b58a515877a923cce6b254e09d1b2fb2c\' Submodule path \'third_party/spdlog\': checked out \'7bf8f148795d2571b4ae4aa9c6d2a8111fbb7077\' Submodule path \'third_party/spirv-cross\': checked out \'84f56d0b43e5a6874de45e11be2bff9dc336fd7e\' Submodule path \'third_party/stb\': checked out \'2c2908f50515dcd939f24be261c3ccbcd277bb49\' Submodule path \'third_party/tinygltf\': checked out \'c0d0251e2c7f2273a894aa1b125af9313a88eae4\' Submodule path \'third_party/vma\': checked out \'f9921aefddee2437cc2e3303d3175bd8ef23e22c\' Submodule path \'third_party/volk\': checked out \'b18f3192886981f28d17f8f41c26c5a53ac3b977\' Submodule path \'third_party/vulkan\': checked out \'76f00ef6cbb1886eb1162d1fa39bee8b51e22ee8\' Downloading other_lib/ios/Debug-iphoneos/libSDL2.a (8.2 MB) Error downloading object: other_lib/ios/Debug-iphoneos/libSDL2.a (5045776): Smudge error: Error downloading other_lib/ios/Debug-iphoneos/libSDL2.a (5045776556ca57f2f7125d89c37a8b94f79cdebdc040d420513f0491787c4ea7): batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution batch response: Post 'https://github.com/KhronosGroup/KTX-Software.git/info/lfs/objects/batch': dial tcp: lookup github.com: Temporary failure in name resolution Errors logged to \'/OE/build/poky/build-st/reproducibleA/tmp/work/core2-64-poky-linux/vulkan-samples/git-r0/git/.git/modules/third_party/ktx/lfs/logs/20220531T194140.927704073.log\'. Use `git lfs logs last` to view the log. error: external filter \'git-lfs filter-process\' failed fatal: other_lib/ios/Debug-iphoneos/libSDL2.a: smudge filter lfs failed fatal: Unable to checkout \'6a38a0694b4d73d22c5d1e22f865d03545e808ea\' in submodule path \'third_party/ktx\' ``` This was with: git version 2.35.1 git-lfs/3.1.4 (GitHub; linux amd64; go 1.18.2) and oe-core/master as of yesterday 18a0c31b33, bitbake 6a346df5 Have anyone else seen this? It should be triggered on autobuilder as well (I triggered it with poky, which I normally don't use, with just 'oe-selftest -r reproducible', but then reproduced the same with just oe-core), maybe it's specific to git+git-lfs versions from host? This is from Gentoo. Will retry with some ubuntu as well soon. Martin Jansa 2022-06-01 10:36:48 UTC It's also reproducible in ubuntu-22.10 with: git version 2.36.1 git-lfs/3.2.0 (GitHub; linux amd64; go 1.18.2) But it's somehow related to what I had in .gitconfig (don't remember when or why I've added these, but it was long time ago), looks like first item from https://shuhrat.github.io/programming/git-lfs-tips-and-tricks.html : [filter "lfs"] smudge = git-lfs smudge -- %f process = git-lfs filter-process required = true clean = git-lfs clean -- %f If I keep only smudge, then the do_unpack takes terribly long, but doesn't fail: 172.53 seconds vulkan-samples-git-r0/do_fetch 5319.38 seconds vulkan-samples-git-r0/do_unpack If I remove whole lfs section from .gitconfig then it doesn't fail and is fast: 13.00 seconds vulkan-samples-git-r0/do_fetch 13.75 seconds vulkan-samples-git-r0/do_unpack With whole lfs section as shown above (or just smudge+required) it fails after some 90 seconds: 6.94 seconds vulkan-samples-git-r0/do_fetch 84.25 seconds vulkan-samples-git-r0/do_unpack And today I found that the .gitconfig issue is even worse, because the same "lfs" config is now added automatically to /etc/gitconfig by git-lfst postinst at least in newer ubuntu: ``` root@ljama:~# rm /etc/gitconfig root@ljama:~# git lfs install --skip-repo --system Git LFS initialized. root@ljama:~# cat /etc/gitconfig [filter "lfs"] clean = git-lfs clean -- %f smudge = git-lfs smudge -- %f process = git-lfs filter-process required = true root@ljama:~# cat /var/lib/dpkg/info/git-lfs.postinst #!/bin/sh set -e # Set up /etc/gitconfig for git-lfs. The --skip-repo option prevents failure if # / is a Git repository with existing non-git-lfs hooks. git lfs install --skip-repo --system > /dev/null 2>&1 ``` according to https://changelogs.ubuntu.com/changelogs/pool/universe/g/git-lfs/git-lfs_3.0.2-1/changelog it was added in: ``` git-lfs (2.6.0-1) unstable; urgency=medium * New upstream release * Bump standards version to 4.2.1 * Add postinst/prerm to set up/remove git-lfs gitconfig ``` 94193 1 Martin.Jansa 2022-10-17 13:38:18 +0000 https://patchwork.yoctoproject.org/project/bitbake/patch/20221017132026.88753-1-Martin.Jansa@gmail.com/ 94195 2 Martin.Jansa 2022-10-17 16:20:41 +0000 for vulkan-samples case we can just explicitly disable lfs: https://patchwork.yoctoproject.org/project/oe-core/patch/20221017140558.632766-1-Martin.Jansa@gmail.com/ 94245 3 Martin.Jansa 2022-10-26 08:10:51 +0000 The bitbake patch itself doesn't resolve all corner cases as documented by added test: https://patchwork.yoctoproject.org/project/bitbake/patch/20221018193746.3599845-1-Martin.Jansa@gmail.com/ I don't know what is expected behavior which would be reproducible (not affected by user's .gitconfig and git-lfs installed or not-installed). 96576 4 Martin.Jansa 2023-09-26 15:55:39 +0000 There were some related changes merged recently: https://git.openembedded.org/bitbake/commit/?id=1f8f21fe024b391d3a6670c64b839db0eee083c3 and as it's still unknown what's the expected behavior, I'll close this as WONTFIX.