Bug 9061

Summary: git fetcher inconsistency for working with PREMIRRORS.
Product: [Build System, Metadata & Runtime] BitBake Reporter: Alexander Kanevskiy <alexander.kanevskiy>
Component: bitbakeAssignee: Pavel Zhukov <pavel>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: pascal.bach, pavel, poky.bs.watcher, poky.watcher, randy.macleod
Version: 2.2   
Target Milestone: 4.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Alexander Kanevskiy 2016-02-04 17:06:58 UTC 
Current git fetcher code has strange behavior with PREMIRRORS.
I've noticed that in my test cases, where PREMIRROS are defined, file with repository git://github.com/file/file.git present and has needed version.
What happened in log.do_fetch.*:

$ grep "Trying" log*fetch*31663
DEBUG: Trying Upstream
DEBUG: Trying MIRRORS
$
So, it cleanly tries to fetch from upstream, where corporate proxy misbehave, so it can't reach github, then it tries to access MIRROS, but can't find needed revision, and build eventually fails.

Expectation that fetcher code should work consistently for both HTTP/FTP sources and sources from version control systems:

- get sources from PREMIRRORS, verify that needed sources/revision present there.
- if not, try Upstream.
- if Upstream not available, try MIRRORS.
Comment 1 Richard Purdie 2016-09-22 12:15:26 UTC 
I think I might be able to explain this, there were some comments on the bitbake mailing list which shed some light on what is happening.

If a git repository has already been fetched and is present in DL_DIR, the git fetcher specifically will try and pull changes from upstream rather than try a PREMIRROR. This is because those premirrors typically don't support git and would require a complete mirror tarball download.

Ideally, we should differentiate between complete downloads and git aware mirrors.
Comment 2 Richard Purdie 2016-09-22 12:33:37 UTC 
I suspect we need to split download() into two passes, and update() and then a full download if update isn't possible or implemented...
Comment 3 Pascal Bach 2016-09-22 12:45:09 UTC 
The case I'm having trouble with is if BB_ALLOWED_NETWORKS is set, which allows bitbake to access the internal mirror and the internal git repository, but doesn't allow access to gitlab.com for example.

It might be possible to add a check if the url git tries to fetch is a trusted url or not and decide based on this if we should download the tarball or fetch from the repo. 
But this feels like a hack to me, as it would not cover the case where upstream access would be possible but the repository is down. In this case it should also fall back to the mirror as downloading the tarball is still better then to fail.
Comment 4 Randy MacLeod 2022-04-11 21:51:18 UTC 
It has been years since this defect was opened and instead of moving it along to the next release, I'm looking for an indication that this is actually a problem that we should try to fix. We also typically prefer https:// fetching so perhaps this isn't a problem any more. 

Pascal, could you take a look and comment this week before Thursday 10:30 AM ET?
Comment 5 Pavel Zhukov 2022-05-09 07:13:21 UTC 
(In reply to comment #4)
> It has been years since this defect was opened and instead of moving it
> along to the next release, I'm looking for an indication that this is
> actually a problem that we should try to fix. We also typically prefer
> https:// fetching so perhaps this isn't a problem any more. 
> 
> Pascal, could you take a look and comment this week before Thursday 10:30 AM
> ET?

This should be fixed as part of https://git.openembedded.org/bitbake/commit/?h=master-next&id=bde98b6c4a23db9571caa28a111d8fdf889ccd94 . Taking this bug to test.