16194 2026-03-10 20:34:12 +0000 Git fetcher doesn't redownload/update shallow tarball when lfs=1 2026-10-01 15:48:11 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified All Multiple NEW Medium normal 6.2 1 michalwsieron richard.purdie poky.bs.watcher poky.watcher randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 104574 0 michalwsieron 2026-03-10 20:34:12 +0000 Scenario I am describing is that multiple recipes fetch the same repo, but one of them is not requesting git-lfs data (lfs=0) and another one does (lfs=1). If we have BB_GIT_SHALLOW enabled for those recipes, there is a risk that if do_fetch for recipe *not* requesting git-lfs data (let's call it foo.bb) runs before the other one - bar.bb, do_unpack for bar.bb will fail as it won't be able to retrieve git-lfs data from remote. Here is a "synthetic test" for that scenario: ``` diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 74eb73472..028639ba1 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2600,6 +2600,28 @@ class GitLfsTest(FetcherTest): shutil.rmtree(self.gitdir, ignore_errors=True) fetcher.unpack(self.d.getVar('WORKDIR')) + @skipIfNoGitLFS() + @skipIfNoNetwork() + def test_shallow_tarball_with_lfs(self): + uri = "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master" + + # Use shallow tarballs + self.d.setVar('BB_GIT_SHALLOW', '1') + self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') + + # Download without git-lfs + self.d.setVar('SRC_URI', uri + ";lfs=0") + fetcher, ud = self.fetch() + + # Redownload with git-lfs + self.d.setVar('SRC_URI', uri + ";lfs=1") + fetcher, ud = self.fetch() + + # Attempt unpack without network + bb.utils.disable_network() + fetcher.unpack(self.d.getVar('WORKDIR')) + + class GitURLWithSpacesTest(FetcherTest): test_git_urls = { "git://tfs-example.org:22/tfs/example%20path/example.git;branch=master" : { ``` You should run above test with `bin/bitbake-selftest bb.tests.fetch.GitLfsTest -k test_shallow_tarball_with_lfs`. *Expected behavior*: Test passes, because when bitbake is supposed to download a shallow tarball with git-lfs blobs, it will check whether the git-lfs blobs are actually available and download them if needed. *Actual behavior*: Test fails at the unpack step with git-lfs unable to fetch the blob. I fear that to properly support such case without doing tarball content checks all the time, we would have to split the git-lfs blobs into a separate tarball. Then if a recipe sets lfs=1, both shallow tarball and lfs tarball would be unpacked. It could also be reused for full mirror tarballs, because from what I checked it can also affect them. --- Side topic, I had to use `bb.utils.disable_network()` as I found no other way to disable network access for the unpack functionality in the tests. However it is not a viable way to actually do this as it affects the entire testing process and subsequent network tests will fail. 104595 1 randy.macleod 2026-03-12 14:40:09 +0000 Richard to comment. 104596 2 richard.purdie 2026-03-12 14:40:38 +0000 At a quick guess, I suspect that the tarball names need to reflect whether lfs was enabled or not. If we do that there would be different tarball archives for both and then things should work properly.