9738 2016-06-09 14:14:55 +0000 [PATCH] fetcher: allow git+<protocol>: syntax 2021-11-03 12:26:08 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED WONTFIX Medium enhancement 4.99 1 enrico.scholz richard.purdie evadeflow liam.howlett poky.bs.watcher poky.watcher randy.macleod --- --- Yes (doc changes required) oldest_to_newest 62812 0 3223 enrico.scholz 2016-06-09 14:14:55 +0000 Created attachment 3223 patch Specifying an alternative git respository with a different access protocol is very difficulty atm because the protocol is part of the URI parameters which are at end the end of the URI. E.g. when having | MY_GIT_REPO = "git://example.com" | SRC_URI = "${MY_GIT_REPO}/foo.git;protocol=ssh" and somebody wants to access this over https (e.g. because of firewall constraints) or over ssh (e.g. because it contains a private branch), this requires modification of 'SRC_URI'. With this patch, this can be done by | MY_GIT_REPO = "git+https://example.com" or | MY_GIT_REPO = "git+ssh://private.example.com" 62813 1 3224 enrico.scholz 2016-06-09 14:28:52 +0000 Created attachment 3224 patch correct patch; previous ones missed half of required changes 66029 2 richard.purdie 2016-09-12 10:52:55 +0000 Your patch means special casing the git fetcher and the protocol parameter into generic code. I've also wondered about this problem but I think I'd prefer to fix it by allowing the MIRRORS code to handle remapping the protocol part of the url. 66031 3 enrico.scholz 2016-09-12 11:45:17 +0000 Code can be generalized to accept <scm>[+<protocol>]:// syntax. Other SCMs (bzr, clearcase, hg, osc, repo, svn) have this parameter too. Modifying MIRROR would be too complicated for generic case IMO. 83204 4 richard.purdie 2019-03-07 15:39:56 +0000 *** Bug 13211 has been marked as a duplicate of this bug. *** 83206 5 richard.purdie 2019-03-07 15:40:28 +0000 *** Bug 3306 has been marked as a duplicate of this bug. *** 83209 6 enrico.scholz 2019-03-07 16:00:10 +0000 fwiw, we are solving this with a .bbclass now (no changes to bitbake itself required): https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/thud/classes/elito-uridecode.bbclass https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/thud/lib/elito/__init__.py 91875 7 richard.purdie 2021-11-02 21:43:31 +0000 The url remapping code now supports setting parameters in mirror urls so it is possible to change a procotol using the mirror syntax. I've just added a test for it to ensure it keeps working, Enrico also shared another way to do that here by patching the fetcher. I'm marking this bug as obsolete, it is hard to know what to do as there is a solution here and there is a way to use the mirrors too so that could be fixed, won't fix or obsolete! 91878 8 enrico.scholz 2021-11-03 10:11:08 +0000 This does not seem to work. I have | PREMIRRORS:prepend = "git\+([^/]+)://.* git://\\2;protocol=\\1 \n" | SRC_URI = "git+http://git.example.org" which fails with | WARNING: Invalid protocol (git\+([^/]+)) in PREMIRRORS: ('git\\+([^/]+)://.*', 'git://\\\\2;protocol=\\\\1') | ... NoMethodError: Could not find a fetcher which supports the URL: 'git+http://git.example.org' I am in doubt that mirrors are good way to set the protocol: they add new uri entries but do not replace them. The original uri might be still tried under some conditions. Native 'git+<protocol>://' support would be much better. 91881 9 richard.purdie 2021-11-03 12:17:18 +0000 (In reply to comment #8) > This does not seem to work. I have > > | PREMIRRORS:prepend = "git\+([^/]+)://.* git://\\2;protocol=\\1 \n" > | SRC_URI = "git+http://git.example.org" > > which fails with > > | WARNING: Invalid protocol (git\+([^/]+)) in PREMIRRORS: > ('git\\+([^/]+)://.*', 'git://\\\\2;protocol=\\\\1') > | ... NoMethodError: Could not find a fetcher which supports the URL: > 'git+http://git.example.org' > > I am in doubt that mirrors are good way to set the protocol: they add new > uri entries but do not replace them. They can offer replacement, see: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=b3dcb6b1790d14845cc5ad70b0749dd6aed32e7a > The original uri might be still tried under some conditions. > > Native 'git+<protocol>://' support would be much better. We have never supported that syntax and I'm not planning to add it as we do not want to support too many different ways of doing things, it just complicates the codebase. FWIW the regex in the above likely wouldn't work anyway since the pattern substitutions work within the url component they're part of, so the \1 in the parameter wouldn't see the match from the protocol section. Changing that would again significantly complicate the code which I don't believe is necessary. 91882 10 enrico.scholz 2021-11-03 12:26:08 +0000 This syntax is already supported by the npmsw fetcher and a common way to express the git protocol. Writing a non-generic PREMIRROR (e.g. to add alternatives for certain hosts) for git:// uris is not ergonomic because mirror setup is global configuration and expensive to change. 3223 2016-06-09 14:14:55 +0000 2016-06-09 14:28:52 +0000 patch 0009-fetcher-allow-git-protocol-syntax.patch application/x-download 1783 enrico.scholz RnJvbSBhNjEzNmM2NGQ2YjQ1NDhiZGViOWM0MjgzOTRiMzgxMTY3YWM3YjUxIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBFbnJpY28gU2Nob2x6IDxlbnJpY28uc2Nob2x6QHNpZ21hLWNo ZW1uaXR6LmRlPgpEYXRlOiBNb24sIDI2IE9jdCAyMDE1IDE5OjMzOjE1ICswMTAwClN1YmplY3Q6 IFtQQVRDSCAwOS8xMF0gZmV0Y2hlcjogYWxsb3cgZ2l0Kzxwcm90b2NvbD46IHN5bnRheAoKU3Bl Y2lmeWluZyBhbiBhbHRlcm5hdGl2ZSBnaXQgcmVzcG9zaXRvcnkgd2l0aCBhIGRpZmZlcmVudCBh Y2Nlc3MKcHJvdG9jb2wgaXMgdmVyeSBkaWZmaWN1bHR5IGF0bSBiZWNhdXNlIHRoZSBwcm90b2Nv bCBpcyBwYXJ0IG9mIHRoZQpVUkkgcGFyYW1ldGVycyB3aGljaCBhcmUgYXQgZW5kIHRoZSBlbmQg b2YgdGhlIFVSSS4KCkUuZy4gd2hlbiBoYXZpbmcKCnwgTVlfR0lUX1JFUE8gPSAiZ2l0Oi8vZXhh bXBsZS5jb20iCnwgU1JDX1VSSSA9ICIke01ZX0dJVF9SRVBPfS9mb28uZ2l0O3Byb3RvY29sPXNz aCIKCmFuZCBzb21lYm9keSB3YW50cyB0byBhY2Nlc3MgdGhpcyBvdmVyIGh0dHBzIChlLmcuIGJl Y2F1c2Ugb2YgZmlyZXdhbGwKY29uc3RyYWludHMpIG9yIG92ZXIgc3NoIChlLmcuIGJlY2F1c2Ug aXQgY29udGFpbnMgYSBwcml2YXRlIGJyYW5jaCksCnRoaXMgcmVxdWlyZXMgbW9kaWZpY2F0aW9u IG9mICdTUkNfVVJJJy4KCldpdGggdGhpcyBwYXRjaCwgdGhpcyBjYW4gYmUgZG9uZSBieQoKfCBN WV9HSVRfUkVQTyA9ICJnaXQraHR0cHM6Ly9leGFtcGxlLmNvbSIKCm9yCgp8IE1ZX0dJVF9SRVBP ID0gImdpdCtzc2g6Ly9wcml2YXRlLmV4YW1wbGUuY29tIgoKU2lnbmVkLW9mZi1ieTogRW5yaWNv IFNjaG9seiA8ZW5yaWNvLnNjaG9sekBzaWdtYS1jaGVtbml0ei5kZT4KLS0tCiBsaWIvYmIvZmV0 Y2gyL19faW5pdF9fLnB5IHwgOCArKysrKysrKwogMSBmaWxlIGNoYW5nZWQsIDggaW5zZXJ0aW9u cygrKQoKZGlmZiAtLWdpdCBhL2xpYi9iYi9mZXRjaDIvX19pbml0X18ucHkgYi9saWIvYmIvZmV0 Y2gyL19faW5pdF9fLnB5CmluZGV4IGFhN2IzNDQuLmU0ZTEwNDUgMTAwNjQ0Ci0tLSBhL2xpYi9i Yi9mZXRjaDIvX19pbml0X18ucHkKKysrIGIvbGliL2JiL2ZldGNoMi9fX2luaXRfXy5weQpAQCAt MjI5LDYgKzIyOSwxMSBAQCBjbGFzcyBVUkkob2JqZWN0KToKICAgICAgICAgdXJscCA9IHVybGxp Yi5wYXJzZS51cmxwYXJzZSh1cmkpCiAgICAgICAgIHNlbGYuc2NoZW1lID0gdXJscC5zY2hlbWUK IAorICAgICAgICBwcm90b2NvbCA9IE5vbmUKKyAgICAgICAgaWYgc2VsZi5zY2hlbWUuc3RhcnRz d2l0aCgnZ2l0KycpOgorICAgICAgICAgICAgcHJvdG9jb2wgPSBzZWxmLnNjaGVtZVs0Ol0KKyAg ICAgICAgICAgIHNlbGYuc2NoZW1lID0gc2VsZi5zY2hlbWVbOjNdCisKICAgICAgICAgcmVwYXJz ZSA9IDAKIAogICAgICAgICAjIENvZXJjZSB1cmxwYXJzZSB0byBtYWtlIFVSSSBzY2hlbWUgdXNl IG5ldGxvYwpAQCAtMjY3LDYgKzI3Miw5IEBAIGNsYXNzIFVSSShvYmplY3QpOgogICAgICAgICBp ZiB1cmxwLnF1ZXJ5OgogICAgICAgICAgICAgc2VsZi5xdWVyeSA9IHNlbGYuX3BhcmFtX3N0cl9z cGxpdCh1cmxwLnF1ZXJ5LCAiJiIpCiAKKyAgICAgICAgaWYgcHJvdG9jb2w6CisgICAgICAgICAg ICBzZWxmLnBhcmFtc1sncHJvdG9jb2wnXSA9IHByb3RvY29sCisKICAgICBkZWYgX19zdHJfXyhz ZWxmKToKICAgICAgICAgdXNlcmluZm8gPSBzZWxmLnVzZXJpbmZvCiAgICAgICAgIGlmIHVzZXJp bmZvOgotLSAKMi41LjUKCg== 3224 2016-06-09 14:28:52 +0000 2016-06-09 14:28:52 +0000 patch 0009-fetcher-allow-git-protocol-syntax.patch text/plain 2374 enrico.scholz RnJvbSAxNjM4NjU4OTJlNWE4MTk2OTk3OTE0NDJhNTBkM2FjMmNlZTAwMjU2IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBFbnJpY28gU2Nob2x6IDxlbnJpY28uc2Nob2x6QHNpZ21hLWNo ZW1uaXR6LmRlPgpEYXRlOiBNb24sIDI2IE9jdCAyMDE1IDE5OjMzOjE1ICswMTAwClN1YmplY3Q6 IFtQQVRDSCAwOS8xMV0gZmV0Y2hlcjogYWxsb3cgZ2l0Kzxwcm90b2NvbD46IHN5bnRheAoKU3Bl Y2lmeWluZyBhbiBhbHRlcm5hdGl2ZSBnaXQgcmVzcG9zaXRvcnkgd2l0aCBhIGRpZmZlcmVudCBh Y2Nlc3MKcHJvdG9jb2wgaXMgdmVyeSBkaWZmaWN1bHR5IGF0bSBiZWNhdXNlIHRoZSBwcm90b2Nv bCBpcyBwYXJ0IG9mIHRoZQpVUkkgcGFyYW1ldGVycyB3aGljaCBhcmUgYXQgZW5kIHRoZSBlbmQg b2YgdGhlIFVSSS4KCkUuZy4gd2hlbiBoYXZpbmcKCnwgTVlfR0lUX1JFUE8gPSAiZ2l0Oi8vZXhh bXBsZS5jb20iCnwgU1JDX1VSSSA9ICIke01ZX0dJVF9SRVBPfS9mb28uZ2l0O3Byb3RvY29sPXNz aCIKCmFuZCBzb21lYm9keSB3YW50cyB0byBhY2Nlc3MgdGhpcyBvdmVyIGh0dHBzIChlLmcuIGJl Y2F1c2Ugb2YgZmlyZXdhbGwKY29uc3RyYWludHMpIG9yIG92ZXIgc3NoIChlLmcuIGJlY2F1c2Ug aXQgY29udGFpbnMgYSBwcml2YXRlIGJyYW5jaCksCnRoaXMgcmVxdWlyZXMgbW9kaWZpY2F0aW9u IG9mICdTUkNfVVJJJy4KCldpdGggdGhpcyBwYXRjaCwgdGhpcyBjYW4gYmUgZG9uZSBieQoKfCBN WV9HSVRfUkVQTyA9ICJnaXQraHR0cHM6Ly9leGFtcGxlLmNvbSIKCm9yCgp8IE1ZX0dJVF9SRVBP ID0gImdpdCtzc2g6Ly9wcml2YXRlLmV4YW1wbGUuY29tIgoKU2lnbmVkLW9mZi1ieTogRW5yaWNv IFNjaG9seiA8ZW5yaWNvLnNjaG9sekBzaWdtYS1jaGVtbml0ei5kZT4KLS0tCiBsaWIvYmIvZmV0 Y2gyL19faW5pdF9fLnB5IHwgMTYgKysrKysrKysrKysrKysrKwogMSBmaWxlIGNoYW5nZWQsIDE2 IGluc2VydGlvbnMoKykKCmRpZmYgLS1naXQgYS9saWIvYmIvZmV0Y2gyL19faW5pdF9fLnB5IGIv bGliL2JiL2ZldGNoMi9fX2luaXRfXy5weQppbmRleCBhYTdiMzQ0Li5lYmMzOThiIDEwMDY0NAot LS0gYS9saWIvYmIvZmV0Y2gyL19faW5pdF9fLnB5CisrKyBiL2xpYi9iYi9mZXRjaDIvX19pbml0 X18ucHkKQEAgLTIyOSw2ICsyMjksMTEgQEAgY2xhc3MgVVJJKG9iamVjdCk6CiAgICAgICAgIHVy bHAgPSB1cmxsaWIucGFyc2UudXJscGFyc2UodXJpKQogICAgICAgICBzZWxmLnNjaGVtZSA9IHVy bHAuc2NoZW1lCiAKKyAgICAgICAgcHJvdG9jb2wgPSBOb25lCisgICAgICAgIGlmIHNlbGYuc2No ZW1lLnN0YXJ0c3dpdGgoJ2dpdCsnKToKKyAgICAgICAgICAgIHByb3RvY29sID0gc2VsZi5zY2hl bWVbNDpdCisgICAgICAgICAgICBzZWxmLnNjaGVtZSA9IHNlbGYuc2NoZW1lWzozXQorCiAgICAg ICAgIHJlcGFyc2UgPSAwCiAKICAgICAgICAgIyBDb2VyY2UgdXJscGFyc2UgdG8gbWFrZSBVUkkg c2NoZW1lIHVzZSBuZXRsb2MKQEAgLTI2Nyw2ICsyNzIsOSBAQCBjbGFzcyBVUkkob2JqZWN0KToK ICAgICAgICAgaWYgdXJscC5xdWVyeToKICAgICAgICAgICAgIHNlbGYucXVlcnkgPSBzZWxmLl9w YXJhbV9zdHJfc3BsaXQodXJscC5xdWVyeSwgIiYiKQogCisgICAgICAgIGlmIHByb3RvY29sOgor ICAgICAgICAgICAgc2VsZi5wYXJhbXNbJ3Byb3RvY29sJ10gPSBwcm90b2NvbAorCiAgICAgZGVm IF9fc3RyX18oc2VsZik6CiAgICAgICAgIHVzZXJpbmZvID0gc2VsZi51c2VyaW5mbwogICAgICAg ICBpZiB1c2VyaW5mbzoKQEAgLTM2Niw2ICszNzQsMTEgQEAgZGVmIGRlY29kZXVybCh1cmwpOgog ICAgIHVzZXIgPSBtLmdyb3VwKCd1c2VyJykKICAgICBwYXJtID0gbS5ncm91cCgncGFybScpCiAK KyAgICBwcm90b2NvbCA9IE5vbmUKKyAgICBpZiB0eXBlLnN0YXJ0c3dpdGgoJ2dpdCsnKToKKyAg ICAgICAgcHJvdG9jb2wgPSB0eXBlWzQ6XQorICAgICAgICB0eXBlID0gdHlwZVs6M10KKwogICAg IGxvY2lkeCA9IGxvY2F0aW9uLmZpbmQoJy8nKQogICAgIGlmIGxvY2lkeCAhPSAtMSBhbmQgdHlw ZS5sb3dlcigpICE9ICdmaWxlJzoKICAgICAgICAgaG9zdCA9IGxvY2F0aW9uWzpsb2NpZHhdCkBA IC0zOTQsNiArNDA3LDkgQEAgZGVmIGRlY29kZXVybCh1cmwpOgogICAgICAgICAgICAgICAgIHMx LCBzMiA9IHMuc3BsaXQoJz0nKQogICAgICAgICAgICAgICAgIHBbczFdID0gczIKIAorICAgIGlm IHByb3RvY29sOgorICAgICAgICBwWydwcm90b2NvbCddID0gcHJvdG9jb2wKKwogICAgIHJldHVy biB0eXBlLCBob3N0LCB1cmxsaWIucGFyc2UudW5xdW90ZShwYXRoKSwgdXNlciwgcHN3ZCwgcAog CiBkZWYgZW5jb2RldXJsKGRlY29kZWQpOgotLSAKMi41LjUKCg==