13278 2019-04-12 11:36:47 +0000 If git protocol doesn't work, you get a tar.gz clone from PREMIRROR which has git protocol origin 2023-10-29 09:22:38 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 2.5.4 x86 Multiple RESOLVED FIXED Medium+ major 4.1 M3 1 ernstp pavel hakan.eriksson pavel poky.bs.watcher poky.watcher randy.macleod richard.purdie --- --- Don't know oldest_to_newest 83527 0 ernstp 2019-04-12 11:36:47 +0000 1) bitbake tries to clone a git with the git protocol. That fails because it's blocked in your corporate network. However http works. 2) The poky PREMIRRORS kick in, and bitbake downloads a git2..tar.gz from downloads.yoctoproject.org 3) Everything works, great! 4) Time passes and a recipe is updated to a new SRCREV that you don't have in your clone, so the git must be fetched again. 5) That fails, and the MIRRORS system gets confused. It looks like do_fetch will clone from using protocol=https from MIRRORS, but do_unpack can't find it. WARNING: prelink-native-1.0+gitAUTOINC+05aeafd053-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/prelink-cross.git;branch=cross_prelink, attempting MIRRORS if available (here it actually does the https clone to a different directory, and no symlink is created. Perhaps because the target already exists... ) ERROR: prelink-native-1.0+gitAUTOINC+05aeafd053-r0 do_unpack: Fetcher failure: fatal: reference is not a tree: 05aeafd053e56356ec8c62f4bb8f7b95bae192f3 One decent workaround is to remove the default poky PREMIRRORS, best combined with your own premirror of course. 83528 1 ernstp 2019-04-12 11:41:37 +0000 I think this check is involved http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/__init__.py?id=b1641008f072ecd7b8843d80e0d3de6c0584129e#n1011 83529 2 4471 ernstp 2019-04-12 13:18:12 +0000 Created attachment 4471 bitbake: fetch2: try_mirror_url(): Replace old git with mirror 83530 3 ernstp 2019-04-12 13:20:51 +0000 Ok so here's a little patch that solves my immediate issue, but I guess there could be a number of scenarios I haven't thought of. I tried to limit it to only the git method to minimize side effects. It looks quite nice in the logs: WARNING: prelink-native-1.0+gitAUTOINC+05aeafd053-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/prelink-cross.git;branch=cross_prelink, attempting MIRRORS if available WARNING: prelink-native-1.0+gitAUTOINC+05aeafd053-r0 do_fetch: Replacing /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.prelink-cross.git with link to /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.git.prelink-cross.git 84639 4 ernstp 2019-08-12 15:35:55 +0000 I sent the patch to bitbake-devel a few months ago also: http://lists.openembedded.org/pipermail/bitbake-devel/2019-May/019998.html 84647 5 randy.macleod 2019-08-12 19:27:15 +0000 Ernst, I've asked someone to review the change. Ping the thread in 2 days if you don't see a reply, please. -- Randy 84722 6 ernstp 2019-08-22 08:34:19 +0000 Extra ping here too. 84895 7 ernstp 2019-09-06 09:19:17 +0000 Saw that I had use for my patch again today: WARNING: pseudo-native-1.9.0+gitAUTOINC+3fa7c853e0-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/pseudo, attempting MIRRORS if available WARNING: pseudo-native-1.9.0+gitAUTOINC+3fa7c853e0-r0 do_fetch: Replacing /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.pseudo with link to /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.git.pseudo 87884 8 ernstp 2020-08-06 07:30:09 +0000 This is still used by everyone at my company every day (the version that was sent to the mailing list with one extra "if" statement). And it helps every time you update to a new yocto release. And to reiterate, it helps when the git protocol is blocked, Yocto fetches a git archive from PREMIRRORS, and then later tries to "git fetch" in that git. There are only a few projects that use the git protocol and not git over http. 89942 9 richard.purdie 2021-04-15 13:21:04 +0000 Sorry about the delays in review on this. I understand the problem but the fix in this form will regress some behaviours. In particular, if you specify an invalid SRCREV it will wipe out the local checkout of the repo and pull mirror tarballs before failing which is a pain when the repo is large (e.g. a kernel) What I think would make sense is to download the mirror tarball and extract it, then check if the revision is there, only then replacing the other repo when we know we'll be successful. 89945 10 ernstp 2021-04-15 13:45:42 +0000 The important details is the subtle difference between the two names you get anyway. git.yoctoproject.org.pseudo git.yoctoproject.org.git.pseudo So an even longer and more accurate description would be: 1) bitbake tries to clone a git with the git protocol. That fails because it's blocked in your corporate network. However http works. 2) The poky PREMIRRORS kick in, and bitbake downloads a git2..tar.gz from downloads.yoctoproject.org 3) Everything works, great! 4) Time passes and a recipe is updated to a new SRCREV that you don't have in your clone, so the git must be fetched again. 5) git fetch fails, as expected because the git protocol is still blocked. 6) The mirror system kicks in successfully and downloads a new tarball of the git archive 7) However this doesn't get used because of the confusion between git.yoctoproject.org.pseudo and git.yoctoproject.org.git.pseudo 93917 11 pavel 2022-08-25 18:04:50 +0000 (In reply to comment #10) > The important details is the subtle difference between the two names you get > anyway. > git.yoctoproject.org.pseudo > git.yoctoproject.org.git.pseudo > > So an even longer and more accurate description would be: > > 1) bitbake tries to clone a git with the git protocol. That fails because > it's blocked in your corporate network. However http works. > 2) The poky PREMIRRORS kick in, and bitbake downloads a git2..tar.gz from > downloads.yoctoproject.org > 3) Everything works, great! > 4) Time passes and a recipe is updated to a new SRCREV that you don't have > in your clone, so the git must be fetched again. > 5) git fetch fails, as expected because the git protocol is still blocked. > 6) The mirror system kicks in successfully and downloads a new tarball of > the git archive > 7) However this doesn't get used because of the confusion between > git.yoctoproject.org.pseudo and git.yoctoproject.org.git.pseudo Hello, There were few fixes in git fetcher and broken tarballs handling to properly handle premirrors and mirror functionality so this issue is fixed most likely. To verify this I've tried to reproduce it with: 1) Block outgoing connections to git port 2) tried to build pseudo WARNING: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/pseudo;branch=oe-core, attempting MIRRORS if available 3) premirrors kicked in. DEBUG: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 --passive-ftp -O /mnt/builds/yocto/build/build/download/git2_git.yoctoproject.org.pseudo.tar.gz.tmp -P /mnt/builds/yocto/build/build/download 'http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.pseudo.tar.gz' 4) pseudo is built 5) delete SRCREV from the mirror and tarball. 6) Tried to build pseudo 7) revision not found: [...] git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat branch --contains 2b4b88eb513335b0ece55fe51854693d9b20de35 --list oe-core 2> /dev/null | wc -l' in /mnt/builds/yocto/build/build/download/git2/git.yoctoproject.org.pseudo 8) git fetch tried to download revision but failed as expected: WARNING: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/pseudo;branch=oe-core, attempting MIRRORS if available 9) wget fetcher kicked in to redownload the mirror tarball: DEBUG: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Fetching http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.pseudo.tar.gz using command '/usr/bin/env wget -t 2 -T 30 --passive-ftp -O /mnt/builds/yocto/build/build/download/git2_git.yoctoproject.org.pseudo.tar.gz.tmp -P /mnt/builds/yocto/build/build/download 'http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.pseudo.tar.gz'' 10) Revision is in place. We're good to go: git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat grep lfs oe-core:.gitattributes | wc -l' in /mnt/builds/yocto/build/build/download/git2/git.yoctoproject.org.pseudo Please let me know if the issue is still exists in master or close this bug report. 93928 12 ernstp 2022-08-26 10:45:15 +0000 Yeah this seems to be fixed now! The symlink I mentioned seem to have been completely removed. 93935 13 richard.purdie 2022-08-29 12:49:06 +0000 Thanks for testing and confirming everyone, nice to have this resolved! :) 4471 2019-04-12 13:18:12 +0000 2023-10-29 09:22:38 +0000 bitbake: fetch2: try_mirror_url(): Replace old git with mirror 0001-bitbake-fetch2-try_mirror_url-Replace-old-git-with-m.patch text/plain 1998 ernstp RnJvbSAzOGQ1NWMwNzk5MTM5NDlkNzY2ODhlNmQxYjc4ZjM2YjgyNTY2ODdmIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiA9P1VURi04P3E/RXJuc3Q9MjBTaj1DMz1CNnN0cmFuZD89IDxl cm5zdC5zam9zdHJhbmRAdmVyaXN1cmUuY29tPgpEYXRlOiBGcmksIDEyIEFwciAyMDE5IDEzOjU0 OjE2ICswMjAwClN1YmplY3Q6IFtQQVRDSF0gYml0YmFrZTogZmV0Y2gyOiB0cnlfbWlycm9yX3Vy bCgpOiBSZXBsYWNlIG9sZCBnaXQgd2l0aAogbWlycm9yCk1JTUUtVmVyc2lvbjogMS4wCkNvbnRl bnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD1VVEYtOApDb250ZW50LVRyYW5zZmVyLUVuY29k aW5nOiA4Yml0CgpJZiB3ZSdyZSB0cnlpbmcgdG8gZmV0Y2ggYSBnaXQsIHdlJ3JlIGluIHRyeV9t aXJyb3JfdXJsIGFuZApvcmlndWQubG9jYWxwYXRoIGlzIGEgZGlyZWN0b3J5LCB0aGF0IG1lYW5z IHRoZSBvcmlnaW5hbCBjbG9uZSBpcyBicm9rZW4KZm9yIHNvbWUgcmVhc29uIGJ1dCB3ZSBmb3Vu ZCBhIG1pcnJvciB0aGF0IHdvcmtzLgpSZW1vdmUgdGhlIG9yaWdpbmFsIGNsb25lIGFuZCByZXBs YWNlIGl0IHdpdGggYSBzeW1saW5rLCBsaWtlIHdoYXQgd291bGQKaG9wZWZ1bGx5IGJlIGRvbmUg b24gYSBmcmVzaCBmZXRjaC4KClsgWW9jdG8gIyAxMzI3OCBdCgpTaWduZWQtb2ZmLWJ5OiBFcm5z dCBTasO2c3RyYW5kIDxlcm5zdC5zam9zdHJhbmRAdmVyaXN1cmUuY29tPgotLS0KIGJpdGJha2Uv bGliL2JiL2ZldGNoMi9fX2luaXRfXy5weSB8IDYgKysrKysrCiAxIGZpbGUgY2hhbmdlZCwgNiBp bnNlcnRpb25zKCspCgpkaWZmIC0tZ2l0IGEvYml0YmFrZS9saWIvYmIvZmV0Y2gyL19faW5pdF9f LnB5IGIvYml0YmFrZS9saWIvYmIvZmV0Y2gyL19faW5pdF9fLnB5CmluZGV4IDkzOWNkY2I5Njku LjhiY2Y4MzBlN2IgMTAwNjQ0Ci0tLSBhL2JpdGJha2UvbGliL2JiL2ZldGNoMi9fX2luaXRfXy5w eQorKysgYi9iaXRiYWtlL2xpYi9iYi9mZXRjaDIvX19pbml0X18ucHkKQEAgLTMzLDYgKzMzLDcg QEAgaWYgJ2dpdCcgbm90IGluIHVybGxpYi5wYXJzZS51c2VzX25ldGxvYzoKICAgICB1cmxsaWIu cGFyc2UudXNlc19uZXRsb2MuYXBwZW5kKCdnaXQnKQogaW1wb3J0IG9wZXJhdG9yCiBpbXBvcnQg Y29sbGVjdGlvbnMKK2ltcG9ydCBzaHV0aWwKIGltcG9ydCBzdWJwcm9jZXNzCiBpbXBvcnQgcGlj a2xlCiBpbXBvcnQgZXJybm8KQEAgLTEwMTgsNiArMTAxOSwxMSBAQCBkZWYgdHJ5X21pcnJvcl91 cmwoZmV0Y2gsIG9yaWd1ZCwgdWQsIGxkLCBjaGVjayA9IEZhbHNlKToKICAgICAgICAgICAgICAg ICBpZiBoYXNhdHRyKG9yaWd1ZC5tZXRob2QsICJidWlsZF9taXJyb3JfZGF0YSIpOgogICAgICAg ICAgICAgICAgICAgICBvcmlndWQubWV0aG9kLmJ1aWxkX21pcnJvcl9kYXRhKG9yaWd1ZCwgbGQp CiAgICAgICAgICAgICByZXR1cm4gb3JpZ3VkLmxvY2FscGF0aAorICAgICAgICBpZiBpc2luc3Rh bmNlKHVkLm1ldGhvZCwgZ2l0LkdpdCkgYW5kIG9zLnBhdGguaXNkaXIob3JpZ3VkLmxvY2FscGF0 aCk6CisgICAgICAgICAgICBsb2dnZXIud2FybmluZygiUmVwbGFjaW5nICVzIHdpdGggbGluayB0 byAlcyIsIG9yaWd1ZC5sb2NhbHBhdGgsIHVkLmxvY2FscGF0aCkKKyAgICAgICAgICAgICMgV2Ug aGF2ZSBhbiBvbGQgZ2l0IGNsb25lIGhlcmUgd2hpY2ggaXMgYnJva2VuIGZvciBzb21lIHJlYXNv biwKKyAgICAgICAgICAgICMgcmVtb3ZlIGl0IGFuZCByZXBsYWNlIGlzIHdpdGggYSBzeW1saW5r LgorICAgICAgICAgICAgc2h1dGlsLnJtdHJlZShvcmlndWQubG9jYWxwYXRoKQogICAgICAgICAj IE90aGVyd2lzZSB0aGUgcmVzdWx0IGlzIGEgbG9jYWwgZmlsZTovLyBhbmQgd2Ugc3ltbGluayB0 byBpdAogICAgICAgICBpZiBub3Qgb3MucGF0aC5leGlzdHMob3JpZ3VkLmxvY2FscGF0aCk6CiAg ICAgICAgICAgICBpZiBvcy5wYXRoLmlzbGluayhvcmlndWQubG9jYWxwYXRoKToKLS0gCjIuMTcu MQoK