Bug 9738

Summary: [PATCH] fetcher: allow git+<protocol>: syntax
Product: [Build System, Metadata & Runtime] BitBake Reporter: Enrico Scholz <enrico.scholz>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED WONTFIX QA Contact:
Severity: enhancement    
Priority: Medium CC: evadeflow, liam.howlett, poky.bs.watcher, poky.watcher, randy.macleod
Version: unspecified   
Target Milestone: 4.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Attachments:
Description Flags
patch
none
patch none

Description Enrico Scholz 2016-06-09 14:14:55 UTC
Created attachment 3223 [details]
patch

Specifying an alternative git respository with a different access
protocol is very difficulty atm because the protocol is part of the
URI parameters which are at end the end of the URI.

E.g. when having

| MY_GIT_REPO = "git://example.com"
| SRC_URI = "${MY_GIT_REPO}/foo.git;protocol=ssh"

and somebody wants to access this over https (e.g. because of firewall
constraints) or over ssh (e.g. because it contains a private branch),
this requires modification of 'SRC_URI'.

With this patch, this can be done by

| MY_GIT_REPO = "git+https://example.com"

or

| MY_GIT_REPO = "git+ssh://private.example.com"
Comment 1 Enrico Scholz 2016-06-09 14:28:52 UTC
Created attachment 3224 [details]
patch

correct patch; previous ones missed half of required changes
Comment 2 Richard Purdie 2016-09-12 10:52:55 UTC
Your patch means special casing the git fetcher and the protocol parameter into generic code. I've also wondered about this problem but I think I'd prefer to fix it by allowing the MIRRORS code to handle remapping the protocol part of the url.
Comment 3 Enrico Scholz 2016-09-12 11:45:17 UTC
Code can be generalized to accept <scm>[+<protocol>]:// syntax. Other SCMs (bzr, clearcase, hg, osc, repo, svn) have this parameter too.

Modifying MIRROR would be too complicated for generic case IMO.
Comment 4 Richard Purdie 2019-03-07 15:39:56 UTC
*** Bug 13211 has been marked as a duplicate of this bug. ***
Comment 5 Richard Purdie 2019-03-07 15:40:28 UTC
*** Bug 3306 has been marked as a duplicate of this bug. ***
Comment 7 Richard Purdie 2021-11-02 21:43:31 UTC
The url remapping code now supports setting parameters in mirror urls so it is possible to change a procotol using the mirror syntax. I've just added a test for it to ensure it keeps working, 

Enrico also shared another way to do that here by patching the fetcher. 

I'm marking this bug as obsolete, it is hard to know what to do as there is a solution here and there is a way to use the mirrors too so that could be fixed, won't fix or obsolete!
Comment 8 Enrico Scholz 2021-11-03 10:11:08 UTC
This does not seem to work.  I have

| PREMIRRORS:prepend = "git\+([^/]+)://.* git://\\2;protocol=\\1 \n"
| SRC_URI = "git+http://git.example.org"

which fails with

| WARNING: Invalid protocol (git\+([^/]+)) in PREMIRRORS: ('git\\+([^/]+)://.*', 'git://\\\\2;protocol=\\\\1')
| ... NoMethodError: Could not find a fetcher which supports the URL: 'git+http://git.example.org'

I am in doubt that mirrors are good way to set the protocol: they add new uri entries but do not replace them.  The original uri might be still tried under some conditions.

Native 'git+<protocol>://' support would be much better.
Comment 9 Richard Purdie 2021-11-03 12:17:18 UTC
(In reply to comment #8)
> This does not seem to work.  I have
> 
> | PREMIRRORS:prepend = "git\+([^/]+)://.* git://\\2;protocol=\\1 \n"
> | SRC_URI = "git+http://git.example.org"
> 
> which fails with
> 
> | WARNING: Invalid protocol (git\+([^/]+)) in PREMIRRORS:
> ('git\\+([^/]+)://.*', 'git://\\\\2;protocol=\\\\1')
> | ... NoMethodError: Could not find a fetcher which supports the URL:
> 'git+http://git.example.org'
> 
> I am in doubt that mirrors are good way to set the protocol: they add new
> uri entries but do not replace them.

They can offer replacement, see:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b3dcb6b1790d14845cc5ad70b0749dd6aed32e7a

> The original uri might be still tried under some conditions.
> 
> Native 'git+<protocol>://' support would be much better.

We have never supported that syntax and I'm not planning to add it as we do not want to support too many different ways of doing things, it just complicates the codebase.

FWIW the regex in the above likely wouldn't work anyway since the pattern substitutions work within the url component they're part of, so the \1 in the parameter wouldn't see the match from the protocol section. Changing that would again significantly complicate the code which I don't believe is necessary.
Comment 10 Enrico Scholz 2021-11-03 12:26:08 UTC
This syntax is already supported by the npmsw fetcher and a common way to express the git protocol.

Writing a non-generic PREMIRROR (e.g. to add alternatives for certain hosts) for git:// uris is not ergonomic because mirror setup is global configuration and expensive to change.