Bug 16194

Summary: Git fetcher doesn't redownload/update shallow tarball when lfs=1
Product: [Build System, Metadata & Runtime] BitBake Reporter: Michal Sieron <michalwsieron>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: NEW --- QA Contact:
Severity: normal    
Priority: Medium CC: poky.bs.watcher, poky.watcher, randy.macleod
Version: unspecified   
Target Milestone: 6.2   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Michal Sieron 2026-03-10 20:34:12 UTC
Scenario I am describing is that multiple recipes fetch the same repo, but one of them is not requesting git-lfs data (lfs=0) and another one does (lfs=1).

If we have BB_GIT_SHALLOW enabled for those recipes, there is a risk that if do_fetch for recipe *not* requesting git-lfs data (let's call it foo.bb) runs before the other one - bar.bb, do_unpack for bar.bb will fail as it won't be able to retrieve git-lfs data from remote.

Here is a "synthetic test" for that scenario:
```
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 74eb73472..028639ba1 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2600,6 +2600,28 @@ class GitLfsTest(FetcherTest):
             shutil.rmtree(self.gitdir, ignore_errors=True)
             fetcher.unpack(self.d.getVar('WORKDIR'))
 
+    @skipIfNoGitLFS()
+    @skipIfNoNetwork()
+    def test_shallow_tarball_with_lfs(self):
+        uri = "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master"
+
+        # Use shallow tarballs
+        self.d.setVar('BB_GIT_SHALLOW', '1')
+        self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
+
+        # Download without git-lfs
+        self.d.setVar('SRC_URI', uri + ";lfs=0")
+        fetcher, ud = self.fetch()
+
+        # Redownload with git-lfs
+        self.d.setVar('SRC_URI', uri + ";lfs=1")
+        fetcher, ud = self.fetch()
+
+        # Attempt unpack without network
+        bb.utils.disable_network()
+        fetcher.unpack(self.d.getVar('WORKDIR'))
+
+
 class GitURLWithSpacesTest(FetcherTest):
     test_git_urls = {
         "git://tfs-example.org:22/tfs/example%20path/example.git;branch=master" : {
```

You should run above test with `bin/bitbake-selftest bb.tests.fetch.GitLfsTest -k test_shallow_tarball_with_lfs`.

*Expected behavior*: Test passes, because when bitbake is supposed to download a shallow tarball with git-lfs blobs, it will check whether the git-lfs blobs are actually available and download them if needed.

*Actual behavior*: Test fails at the unpack step with git-lfs unable to fetch the blob.

I fear that to properly support such case without doing tarball content checks all the time, we would have to split the git-lfs blobs into a separate tarball. Then if a recipe sets lfs=1, both shallow tarball and lfs tarball would be unpacked. It could also be reused for full mirror tarballs, because from what I checked it can also affect them.

---

Side topic, I had to use `bb.utils.disable_network()` as I found no other way to disable network access for the unpack functionality in the tests. However it is not a viable way to actually do this as it affects the entire testing process and subsequent network tests will fail.
Comment 1 Randy MacLeod 2026-03-12 14:40:09 UTC
Richard to comment.
Comment 2 Richard Purdie 2026-03-12 14:40:38 UTC
At a quick guess, I suspect that the tarball names need to reflect whether lfs was enabled or not. If we do that there would be different tarball archives for both and then things should work properly.