| Summary: |
Git fetcher doesn't redownload/update shallow tarball when lfs=1 |
| Product: |
[Build System, Metadata & Runtime] BitBake
|
Reporter: |
Michal Sieron <michalwsieron> |
| Component: |
bitbake | Assignee: |
Richard Purdie <richard.purdie> |
| Status: |
NEW
---
|
QA Contact: |
|
| Severity: |
normal
|
|
|
| Priority: |
Medium
|
CC: |
poky.bs.watcher, poky.watcher, randy.macleod
|
| Version: |
unspecified | |
|
| Target Milestone: |
6.2 | |
|
| Hardware: |
All | |
|
| OS: |
Multiple | |
|
| Whiteboard: |
|
|
OS type for building Yocto:
|
---
|
Type of Regression:
|
---
|
|
Verified:
|
|
Documentation change:
|
No (bug/feature does not impact docs)
|
Scenario I am describing is that multiple recipes fetch the same repo, but one of them is not requesting git-lfs data (lfs=0) and another one does (lfs=1). If we have BB_GIT_SHALLOW enabled for those recipes, there is a risk that if do_fetch for recipe *not* requesting git-lfs data (let's call it foo.bb) runs before the other one - bar.bb, do_unpack for bar.bb will fail as it won't be able to retrieve git-lfs data from remote. Here is a "synthetic test" for that scenario: ``` diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 74eb73472..028639ba1 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2600,6 +2600,28 @@ class GitLfsTest(FetcherTest): shutil.rmtree(self.gitdir, ignore_errors=True) fetcher.unpack(self.d.getVar('WORKDIR')) + @skipIfNoGitLFS() + @skipIfNoNetwork() + def test_shallow_tarball_with_lfs(self): + uri = "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master" + + # Use shallow tarballs + self.d.setVar('BB_GIT_SHALLOW', '1') + self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') + + # Download without git-lfs + self.d.setVar('SRC_URI', uri + ";lfs=0") + fetcher, ud = self.fetch() + + # Redownload with git-lfs + self.d.setVar('SRC_URI', uri + ";lfs=1") + fetcher, ud = self.fetch() + + # Attempt unpack without network + bb.utils.disable_network() + fetcher.unpack(self.d.getVar('WORKDIR')) + + class GitURLWithSpacesTest(FetcherTest): test_git_urls = { "git://tfs-example.org:22/tfs/example%20path/example.git;branch=master" : { ``` You should run above test with `bin/bitbake-selftest bb.tests.fetch.GitLfsTest -k test_shallow_tarball_with_lfs`. *Expected behavior*: Test passes, because when bitbake is supposed to download a shallow tarball with git-lfs blobs, it will check whether the git-lfs blobs are actually available and download them if needed. *Actual behavior*: Test fails at the unpack step with git-lfs unable to fetch the blob. I fear that to properly support such case without doing tarball content checks all the time, we would have to split the git-lfs blobs into a separate tarball. Then if a recipe sets lfs=1, both shallow tarball and lfs tarball would be unpacked. It could also be reused for full mirror tarballs, because from what I checked it can also affect them. --- Side topic, I had to use `bb.utils.disable_network()` as I found no other way to disable network access for the unpack functionality in the tests. However it is not a viable way to actually do this as it affects the entire testing process and subsequent network tests will fail.