Bug 13278

I think this check is involved http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/__init__.py?id=b1641008f072ecd7b8843d80e0d3de6c0584129e#n1011

Created attachment 4471 [details]
bitbake: fetch2: try_mirror_url(): Replace old git with  mirror

Ok so here's a little patch that solves my immediate issue, but I guess there could be a number of scenarios I haven't thought of. I tried to limit it to only the git method to minimize side effects.

It looks quite nice in the logs:

WARNING: prelink-native-1.0+gitAUTOINC+05aeafd053-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/prelink-cross.git;branch=cross_prelink, attempting MIRRORS if available
WARNING: prelink-native-1.0+gitAUTOINC+05aeafd053-r0 do_fetch: Replacing /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.prelink-cross.git with link to /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.git.prelink-cross.git

I sent the patch to bitbake-devel a few months ago also:

http://lists.openembedded.org/pipermail/bitbake-devel/2019-May/019998.html

Ernst, I've asked someone to review the change. Ping the thread in 2 days if you don't see a reply, please. -- Randy

Extra ping here too.

Saw that I had use for my patch again today:

WARNING: pseudo-native-1.9.0+gitAUTOINC+3fa7c853e0-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/pseudo, attempting MIRRORS if available
WARNING: pseudo-native-1.9.0+gitAUTOINC+3fa7c853e0-r0 do_fetch: Replacing /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.pseudo with link to /home/ernsjo/code/cu1/downloads//git2/git.yoctoproject.org.git.pseudo

This is still used by everyone at my company every day (the version that was sent to the mailing list with one extra "if" statement).
And it helps every time you update to a new yocto release.

And to reiterate, it helps when the git protocol is blocked, Yocto fetches a git archive from PREMIRRORS, and then later tries to "git fetch" in that git.

There are only a few projects that use the git protocol and not git over http.

Sorry about the delays in review on this. I understand the problem but the fix in this form will regress some behaviours.

In particular, if you specify an invalid SRCREV it will wipe out the local checkout of the repo and pull mirror tarballs before failing which is a pain when the repo is large (e.g. a kernel)

What I think would make sense is to download the mirror tarball and extract it, then check if the revision is there, only then replacing the other repo when we know we'll be successful.

The important details is the subtle difference between the two names you get anyway.
git.yoctoproject.org.pseudo
git.yoctoproject.org.git.pseudo

So an even longer and more accurate description would be:

1) bitbake tries to clone a git with the git protocol. That fails because it's blocked in your corporate network. However http works.
2) The poky PREMIRRORS kick in, and bitbake downloads a git2..tar.gz from downloads.yoctoproject.org
3) Everything works, great!
4) Time passes and a recipe is updated to a new SRCREV that you don't have in your clone, so the git must be fetched again.
5) git fetch fails, as expected because the git protocol is still blocked.
6) The mirror system kicks in successfully and downloads a new tarball of the git archive
7) However this doesn't get used because of the confusion between git.yoctoproject.org.pseudo and git.yoctoproject.org.git.pseudo

(In reply to comment #10)
> The important details is the subtle difference between the two names you get
> anyway.
> git.yoctoproject.org.pseudo
> git.yoctoproject.org.git.pseudo
> 
> So an even longer and more accurate description would be:
> 
> 1) bitbake tries to clone a git with the git protocol. That fails because
> it's blocked in your corporate network. However http works.
> 2) The poky PREMIRRORS kick in, and bitbake downloads a git2..tar.gz from
> downloads.yoctoproject.org
> 3) Everything works, great!
> 4) Time passes and a recipe is updated to a new SRCREV that you don't have
> in your clone, so the git must be fetched again.
> 5) git fetch fails, as expected because the git protocol is still blocked.
> 6) The mirror system kicks in successfully and downloads a new tarball of
> the git archive
> 7) However this doesn't get used because of the confusion between
> git.yoctoproject.org.pseudo and git.yoctoproject.org.git.pseudo


Hello,

There were few fixes in git fetcher and broken tarballs handling to properly handle premirrors and mirror functionality so this issue is fixed most likely.

To verify this I've tried to reproduce it with:

1) Block outgoing connections to git port
2) tried to build pseudo
WARNING: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/pseudo;branch=oe-core, attempting MIRRORS if available

3) premirrors kicked in.
DEBUG: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 --passive-ftp -O /mnt/builds/yocto/build/build/download/git2_git.yoctoproject.org.pseudo.tar.gz.tmp -P /mnt/builds/yocto/build/build/download 'http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.pseudo.tar.gz'

4) pseudo is built

5) delete SRCREV from the mirror and tarball.

6) Tried to build pseudo

7) revision not found:
[...] git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat branch --contains 2b4b88eb513335b0ece55fe51854693d9b20de35 --list oe-core 2> /dev/null | wc -l' in /mnt/builds/yocto/build/build/download/git2/git.yoctoproject.org.pseudo

8) git fetch tried to download revision but failed as expected:

WARNING: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Failed to fetch URL git://git.yoctoproject.org/pseudo;branch=oe-core, attempting MIRRORS if available

9) wget fetcher kicked in to redownload the mirror tarball:
DEBUG: pseudo-1.9.0+gitAUTOINC+2b4b88eb51-r0 do_fetch: Fetching http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.pseudo.tar.gz using command '/usr/bin/env wget -t 2 -T 30 --passive-ftp -O /mnt/builds/yocto/build/build/download/git2_git.yoctoproject.org.pseudo.tar.gz.tmp -P /mnt/builds/yocto/build/build/download 'http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.pseudo.tar.gz''

10) Revision is in place. We're good to go:
git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat grep lfs oe-core:.gitattributes | wc -l' in /mnt/builds/yocto/build/build/download/git2/git.yoctoproject.org.pseudo

Please let me know if the issue is still exists in master or close this bug report.

Yeah this seems to be fixed now!
The symlink I mentioned seem to have been completely removed.

Thanks for testing and confirming everyone, nice to have this resolved! :)