Bug 7958

We used to have this in the fetcher but in the "fetch v2" rewrite, we dropped support for it. Why? It breaks the mirror tarballs, you simply can't tell from the tarball name which revisions it may or may not contain and if your revision is older, you can't even easily update the repo.

So whilst its a nice idea, it does pose some challenges from the mirror tarball perspective.

We may have to save a different mirror tarball when this option is in use so there can't be any overlap between the two cases.

I've done a little research on this. First, it's worth noting that shallow clones are *much* more capable than they used to be with recent git. You can now pull in a shallow repo, which will add new commits on top of the truncated history just fine, and you can convert a shallow repo to unshallow and vice versa with fetch --unshallow and fetch --depth, respectively.

Next, we can't actually do a shallow clone unless SRCREV is the current HEAD of the specified branch, as the git fetch protocol operates relative to refs, not objects, so you can't shallow clone a commit hash, only the full branch. If we knew the number of commits from SRCREV to branch, we could specify that depth, but we don't have that information until the repository is cloned. That said, once we have that repository, we could construct a shallow repo from it, and tar that up as a shallow mirror tarball to speed up the fetch for that specific case.

Of course, if we had a shallow mirror tarball of the wrong depth, e.g. from a different recipe's clone, we won't be able to determine easily what depth we need, since we wouldn't have the history we need to be able to see how far down the SRCREV is. We'd have to promote it to an unshallow repository in that case, and recreate a new shallow repository for the recipe.

So we'll have to determine exactly what information we want in the tarball filename. If we include depth number, then it won't be shared between multiple recipes in some cases, and we could end up fetching multiple shallow tarballs. And there's the question of, if we include depth number, should the mirror fetching check for higher depth counts that should also include our commit?

We may be best off including the SRCREV+branch in the shallow tarball filename. If that *exact* instance can't be found, then fall back to a non-shallow operation and if BB_MIRROR_TARBALLS is set, reconstruct a shallow tarball from that at that point.

So to sum up, as far as I can tell, direct shallow clones aren't viable, but use of shallow git mirror tarballs is viable, if we work out the details.

My current thoughts on a simpler design than I was previously considering:

Create a new fetcher, e.g. gitshallow://, which doesn't actually clone anything. It does not call out to, or subclass, the Git fetcher, which was my original thought, as doing so complicates matters and then I'd need to potentially support multiple mirrortarballs, which the fetch core can't handle today.

Instead, it will only deal with the shallow mirror tarball:
- urldata_init: set ud.mirrortarball, to let PREMIRRORS and the fetch core handle downloading it from an appropriate mirror.
- download: point localpath at the mirrortarball, as I think it needs to be done here rather than in urldata_init to let the core mirrortarball handling work, and if there's no mirrortarball, raise an appropriate FetchException
- unpack: if the mirror tarball exists, unpack it directly into ${WORKDIR} and set it up there
- build_mirror_data: create a shallow clone of the real git repo under ${DL_DIR}/git2/ as created by the git fetcher, if it exists, and tar it up

Note that it doesn't clone from upstream, ever. Instead, we set up a mirror to handle the fallback to a full git repository, e.g.: MIRRORS += "gitshallow://.*/.* git://HOST/PATH\n".

Since it's the origud's unpack() which is used, rather than the ud's unpack() (that is, GitShallow will get its unpack method called, not Git), so the gitshallow unpack will need to be able to dispatch to the Git fetcher unpack() if the mirrortarball doesn't exist. It'd really be nice if we better supported nested fetch methods, or at least passed to unpack() info about what was actually fetched, mirror-wise, rather than just the source urldata.

Does this seem viable? My original thought was to subclass Git and handle all the fallbacks there, but it gets rather messy, as I need to handle multiple mirror tarballs, and there's the question of what clonedir should be set to, and what localpath should be set to to cover both cases, when shallow and non-shallow repositories end up in completely different places..

Ignore the previous design proposal, it's not viable, since the main ud fetch method has to handle revision checking and whatnot. One could try leaving git:// in the URI and use a premirror to fetch shallow, but it'd end up obeying the main git fetcher's localpath, which would be problematic not wanting to put shallow repositories in that location. I think I may have no choice but to either support multiple mirror tarballs in the fetch core, or let shallow repositories end up in ${DL_DIR}/git2/, but that has its own complications.

Side note for future reference: I don't think we have a test in bb.tests.fetch to confirm that the original fetcher download() is called against a git mirror tarball to unpack it.

I have a functional, though limited prototype which modifies git.py. It has two main issues right now:
1. No fallback to full git mirror tarball, the shallow tarball replaces it.
2. No real control over what history is removed. Currently, each SRCREV is the point where the history stops (each is an entry in .git/shallow), and history after that up to the branch HEAD is retained. We may want a way to prevent history after the srcrevs sticking around, as well as a way to retain history beyond SRCREV, either via a depth setting, or by specifying a revision to go up to (e.g. you could use that to keep the history up to the upstream kernel tag in a bsp kernel repo, so the bsp specific commits stick around, and only that which can be easily pulled from upstream is removed, which would be useful if the kernel git repo isn't public).

To handle the first of those, the fetch core will need to be able to handle multiple mirrortarballs, which should be fairly straightforward. I'm testing with simpler logic in try_mirror_url, so all that should be needed next is to pass mirrortarball into uri_replace, and iterate over the tarballs in build_mirroruris, in theory.

For the second, we likely need a few new settings: 1. boolean to control history retaining after the shallow point, 2. depth, which can be either an integer or a committish/ref.

See https://github.com/openembedded/bitbake/compare/openembedded:master...kergoth:shallow-git-mirror-tarballs for the prototype.

The prototype at https://github.com/openembedded/bitbake/compare/openembedded:master...kergoth:shallow-git-mirror-tarballs now supports multiple mirror tarballs, which addresses the first of the concerns mentioned in the previous comment (though there's not yet a way to control *fetch of / use of* the shallow tarball separately from the *creation of* the shallow tarball, if shallow is enabled, shallow is attempted to be fetched, and the shallow tarball is what's created).

I'm inclined toward having the use of / fetch of the shallow tarball be automatic, and default, as I think it makes a great deal of sense to fetch as little as is needed to get your build going out of the box, and then have url parameters (whose default values come from variables) control whether to create a shallow tarball, and if so, how to control its depth. The one potential concern I have about that is, the depth wouldn't be encoded into the filename, so wouldn't be used when fetching, only when creating, and as a result, my shallow tarball might be different than your shallow tarball.

The functional implementation has been sent to the bitbake list as an RFC.

Updating issue status: the final non-RFC patches are on the list, but merging the is postponed to 2.1 to avoid risk with the 2.0 release, as the fetch code has a lot of corner cases to deal with. Revisit after 2.0 hits.

I've run into multiple problems with the current implementation, when attempting to get it working well with linux-yocto, so back to the drawing board.

I have a reworked implementation of this which allows us to keep all branches around, while still correctly removing the specified history at all points where it intersects the branches. At the moment this approach is slow for certain repositories, so I have a new variable to control whether the branches are kept or filtered. I'd like to determine if there's a more efficient method before submitting the new implementation, however.

The fundamental issue comes down to how shallow is implemented in git. There's a file, $GIT_DIR/shallow which lists commits. These commits are grafted to nothing, that is, their parents are removed. The problem with this is, it's possible to bypass the listed commits and pull in their history but not those commits themselves. For example, if the main branch merged v4.1.7 and v4.1.8, and I make it shallow at v4.1.8, the only history removed would be the commits between v4.1.7 and v4.1.8, so the repo stays huge.

One way to work around this is to graft every single commit (git rev-list v4.1.8) in the history of the specified shallow commits, but this causes very large slowdowns in git operations, so I'm still determining if that's viable.

The way I currently have it implemented, I obtain the parents of a shallow commit, add the shallow commit to $GIT_DIR/shallow, then do a git merge-base of the parents with every single ref (branch or tag) in the repository, to find points where the history of the shallow commit intersects our branches, and then each of those merge bases gets processed the same way as the original shallow commit, until we stop finding points of intersection. So you can see how this would slow down as the # of refs and # of shallow commits increases, plus the amount of intersection points in the history will greatly affect this as well.

I'm hoping there's a more efficient method to find these points of intersection than a ton of merge-base calls, but I need to consult with other folks that know git well to look for input from others on this, and give it some more thought.

I also intend to rethink the variable interface used to enable and configure the mechanism, as I'm not entirely happy with it today.

I'm finally fairly happy with the behavior of this, after having worked out some bugs and kinks, but I intend to rework the variable / configuration interface before re-submitting it to the bitbake list. I expect to do that in the next week or two.

The reworked interface is complete, I have one change I'm debating whether to make, then I just need to do a lot of local testing, and consider whether to create unit tests to attempt to cover the corner cases.

Updated implementation for 2.4: https://github.com/openembedded/bitbake/compare/master...kergoth:shallow-external. Will re-submit via email after 2.3 is split from master.

https://github.com/openembedded/bitbake/compare/master...kergoth:shallow has the current implementation, with more granular commits, fixes for gitsm and gitannex, and unit tests, will submit when master opens for 2.4.

Shallow mirror tarball support for bitbake has been merged, so all that's needed is documentation.

Chris - would you be able to send some raw material / changes for the documentation to Scott Rifenbark <srifenbark@gmail.com> for this? Even very rough form would be fine.