Bug 15126

Summary:	Github Mirror does not work if SRC_URI contains downloadfilename
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Quirin Gylstorff <quirin.gylstorff>
Component:	bitbake	Assignee:	Ross Burton <ross.burton>
Status:	NEW ---	QA Contact:
Severity:	normal
Priority:	Medium	CC:	pavel, poky.bs.watcher, poky.watcher, randy.macleod, yoann.congal
Version:	4.2
Target Milestone:	5.99
Hardware:	All
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Yes (doc changes required)

Description Quirin Gylstorff 2023-05-24 13:50:36 UTC

Dear all,

we have the problem that is not possible with d97d62e2 (bitbake-user-manual: add BB_HASH_CODEPARSER_VALS) to use a Github Mirror if the SRC_URI contains a downloadfilename.

I bisected the behavior changed with 96c30007 (fetch2: fix downloadfilename issue with premirror).

As Scott Weaver has written in https://bugzilla.yoctoproject.org/show_bug.cgi?id=13039#c10 the expected and documented behavior was that using a Mirror will the same file name as in the original URL.  downloadfilename should only  change the name of the locally stored file.

With:
SRC_URI = "https://somewhere.org/example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz"
MIRROR = "https://.*", "https:///somemirror.org"

the expected result was:
"https:///somemirror.org/example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz"

but instead the following was given:
"https:///somemirror.org/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz"

The Mirrors I know do not change the name of the file the provide. The  only known exception is https://downloads.yoctoproject.org/mirror/sources/, which does not mirror
the files with the original names but with the names defined in downloadfilename.   

My question is:

 - What is the intended behavior in case of  using a MIRROR with downloadfilename set in SRC_URI?

- If it is the current implementation the documentation needs to adapted.

Kind regards

Quirin Gylstorff
Siemens AG

Comment 1 Ross Burton 2023-05-25 19:10:57 UTC

I'm not entirely sure what you mean by a "Github mirror".  I presume you mean an arbitrary source mirror.

There's definitely two conflicting use-cases here.

The first is people that use a fully populated DL_DIR as a source mirror. This is a documented use case and a source mirror is expected to have a specific structure, for example git repositories can be exposed as tarballs with a specific structure and name.  So in this case a source mirror isn't an arbitrary URL, it's specifically a bitbake-compatible source mirror.

The second use-case is "the upstream server is bad, here is an alternative".  We use this in oe-core, for example the Debian archive URLs have mirror entries which use snapshot.debian.org to handle tarballs which are purged from the archive.

The problem is that these two use-cases have different semantics: in the first the downloadfilename in DL_DIR becomes the name in the mirror. In the latter, the real name not the downloadfilename is expected.

It's not really feasible to support both usecase fully as they're contradictory.  I suspect primarily supporting the first use case and the latter mostly works will be the right thing to do.

Comment 2 Quirin Gylstorff 2023-05-26 06:55:33 UTC

Our use case would be the second where the upstream server is not reachable so take an alternative. 

We did not find any documentation which indicated that we have the current behavior, therefore we were not sure which is the correct. 

Could the documentation adapted?

Comment 3 Pavel Zhukov 2023-05-26 11:35:26 UTC

(In reply to Quirin Gylstorff from comment #2)
> Our use case would be the second where the upstream server is not reachable
> so take an alternative. 
> 
> We did not find any documentation which indicated that we have the current
> behavior, therefore we were not sure which is the correct. 
> 
> Could the documentation adapted?

I think you're looking for this doc: https://docs.yoctoproject.org/dev-manual/efficiently-fetching-sources.html . 
Implementations examples: https://low-level.wiki/mirrors/yocto_sources.html https://www.mndet.net/2018/12/28/yocto-mirrors.html Some of them may be outdated though

Comment 4 Quirin Gylstorff 2023-05-30 07:29:15 UTC

An internal discussed solution was:
If the fetcher does not find the `downloadfilename` on the mirror -> it tries the mirror with the original file name.

Would this be an acceptable solution?