Bug 9825

Summary:	allow alternates with full protocol specs for improving bitbake git functionality
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Alexander Stohr <alexander.stohr>
Component:	bitbake	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED FIXED	QA Contact:
Severity:	enhancement
Priority:	Medium	CC:	leonardo.sandoval.gonzalez, poky.bs.watcher, poky.watcher, randy.macleod, sgw
Version:	unspecified
Target Milestone:	Future
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Yes (doc changes required)

Description Alexander Stohr 2016-06-24 18:44:33 UTC

Example of a equivalent set of git repository access methods:

git://git.gnome.org/mobile-broadband-provider-info
https://git.gnome.org/browse/mobile-broadband-provider-info
ssh://USERNAME@git.gnome.org/git/mobile-broadband-provider-info
(of course there are more variants for other repos...)

people might prefer a certain way of access, e.g. password protected access or always HTTP (due to local firewalls they barely can control) instead of the normal "git" protocol when going for the sources of a certain bitbake package.

there should be the option for specfiying patterns for servers at a central location or even locally (with less wild cards) for a single recipe.

in above set the recipe creator would simply write e.g. the git-protocol path to his scripting variables. but at a central place something like this can be there:

server=git.gnome.org
username= # if you keep this empty then any alternate statement using this item gets discarded
alternate=git://%server%/%package%
alternate=https://%server%/browse/%package%
alternate=ssh://%username%@%server%/git/%package%

further it can be very helpful to occasionally insert a local file path for all git locations (for things that got cloned earlier using "git clone <...>" or if not on the master "git clone --bare --mirror" - it's already supported by the build tools). at the beginning of the git master file it could look like this:

alternate_universal=/home/john.smith/Downloads/git-mirrors/%package%

maybe providing a preferred git protocol or even an order for probing might be of some sense: (it might even make some sense if it is available locally per server in a similar fashion)

priority=ssh,https,http,git # leftmost protocol has highest priority in probing

at the ruleset in the recipes it might be helpful in some cases when such alternates are available as well, as a temporary or even a permanent local/public selection for whatever solution to a task. Example:

SRC_URI="git://git.gnome.org/mobile-broadband-provider-info;alternate=file:///usr/local/src/git-mirrors-shared/for-embedded-board-xy/%package%;alternate=https://%server%/browse/%package%;altrnate=ssh://%username%@%server%/git/%package%"

Obviously if a globally stored rule is reasonable then it should be created first by the script creator, simply for better joint long-term maintenance efforts. but the flexibility for individual tweaking seems to be reasonable as well - thus the global settings might step back in favour for the more local settings.

Comment 1 Richard Purdie 2016-06-30 14:46:59 UTC

This is really complicated because different people want different things, sometimes even depending on their location. Rather than adding more syntax, it would make most sense to use MIRRORS and PREMIRRORS to handle this. That also gives a convenient way of setting policy.

Its likely that MIRRORS doesn't quite have enough magic to handle the parameter part of the urls so that would need to be fixed. We'd also want to add tests to bitbake-selftest for this.

I am worried about putting more complexity into what is already a complex piece of code which is why I don't want to rush into changing this, or add yet more syntax which potentially conflicts with MIRRORS.

Comment 2 Richard Purdie 2021-11-02 21:48:54 UTC

A totally generic "use X, Y, Z" in order of preference isn't possible since not all urls support all protocols and even when they do the form can be different. An example is git.yoctoproject.org where https ans git work but the urls are slightly different.

That said, there are possibilities and remapping urls like this is basically what MIRRORS and PREMIRRORS are for.

The good news is that parameter remapping now also works, so it is possible to take a protocol=git parameter and change it to protocol=https, or add a new parameter in a mirror url and other forms of manipulation too. With that now working I believe this issue is addressed as much as we can ever make it.