Dear all, we have the problem that is not possible with d97d62e2 (bitbake-user-manual: add BB_HASH_CODEPARSER_VALS) to use a Github Mirror if the SRC_URI contains a downloadfilename. I bisected the behavior changed with 96c30007 (fetch2: fix downloadfilename issue with premirror). As Scott Weaver has written in https://bugzilla.yoctoproject.org/show_bug.cgi?id=13039#c10 the expected and documented behavior was that using a Mirror will the same file name as in the original URL. downloadfilename should only change the name of the locally stored file. With: SRC_URI = "https://somewhere.org/example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz" MIRROR = "https://.*", "https:///somemirror.org" the expected result was: "https:///somemirror.org/example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz" but instead the following was given: "https:///somemirror.org/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz" The Mirrors I know do not change the name of the file the provide. The only known exception is https://downloads.yoctoproject.org/mirror/sources/, which does not mirror the files with the original names but with the names defined in downloadfilename. My question is: - What is the intended behavior in case of using a MIRROR with downloadfilename set in SRC_URI? - If it is the current implementation the documentation needs to adapted. Kind regards Quirin Gylstorff Siemens AG
I'm not entirely sure what you mean by a "Github mirror". I presume you mean an arbitrary source mirror. There's definitely two conflicting use-cases here. The first is people that use a fully populated DL_DIR as a source mirror. This is a documented use case and a source mirror is expected to have a specific structure, for example git repositories can be exposed as tarballs with a specific structure and name. So in this case a source mirror isn't an arbitrary URL, it's specifically a bitbake-compatible source mirror. The second use-case is "the upstream server is bad, here is an alternative". We use this in oe-core, for example the Debian archive URLs have mirror entries which use snapshot.debian.org to handle tarballs which are purged from the archive. The problem is that these two use-cases have different semantics: in the first the downloadfilename in DL_DIR becomes the name in the mirror. In the latter, the real name not the downloadfilename is expected. It's not really feasible to support both usecase fully as they're contradictory. I suspect primarily supporting the first use case and the latter mostly works will be the right thing to do.
Our use case would be the second where the upstream server is not reachable so take an alternative. We did not find any documentation which indicated that we have the current behavior, therefore we were not sure which is the correct. Could the documentation adapted?
(In reply to Quirin Gylstorff from comment #2) > Our use case would be the second where the upstream server is not reachable > so take an alternative. > > We did not find any documentation which indicated that we have the current > behavior, therefore we were not sure which is the correct. > > Could the documentation adapted? I think you're looking for this doc: https://docs.yoctoproject.org/dev-manual/efficiently-fetching-sources.html . Implementations examples: https://low-level.wiki/mirrors/yocto_sources.html https://www.mndet.net/2018/12/28/yocto-mirrors.html Some of them may be outdated though
An internal discussed solution was: If the fetcher does not find the `downloadfilename` on the mirror -> it tries the mirror with the original file name. Would this be an acceptable solution?