Bug 3214

I've already submitted a patch to raise an error explicitly with advice if protocol=git is detected with http:// or https:// URIs (your #1 above); it catches the most common mistake. I'm not convinced we can really apply the check in #2 as this is still a valid URI. #3 might be useful perhaps but that's probably something to fix in the next release.

(In reply to comment #1)
> I've already submitted a patch to raise an error explicitly with advice if
> protocol=git is detected with http:// or https:// URIs (your #1 above); it
> catches the most common mistake. I'm not convinced we can really apply the
> check in #2 as this is still a valid URI. #3 might be useful perhaps but
> that's probably something to fix in the next release.

Is it mandatory for SRC_URIs starting with "git://" to also contain
"protocol=XXX"?  I had the impression that this was optional, and that
bitbake would default to "protocol=git" if omitted[?] If this is the
case, then the check for #1 won't help users who try to change:

> SRC_URI = "git://github.com/zeromq/pyzmq.git;..." [protocol unspecified]

to:

> SRC_URI = "http://github.com/zeromq/pyzmq.git;..." [protocol unspecified]

thinking that this will work just like git works. A gentle
nudge towards the right syntax will, I think, be very much appreciated
by new users.  If "protocol=" is NOT optional when SRC_URI starts with
"git://", then I agree that #1 takes care of just about every posible
case, and the rest of this comment probably doesn't apply. (I assume
that making it mandatory--if it currently is not--would break a lot of
recipes.)

Regarding #2, if someone puts:

> SRC_URI = "http://github.com/zeromq/pyzmq.git;..."

in a recipe, you can be 99% certain he actually meant to type:

> SRC_URI = "git://github.com/zeromq/pyzmq.git;..."

or possibly:

> SRC_URI = "git://github.com/zeromq/pyzmq.git;protocol=http;..."


I think it's reasonable to force them to type this instead:

> SRC_URI = "http://github.com/zeromq/pyzmq.git;protocol=http"

if they REALLY want to download the HTML source of a web page.  The
'protocol=http' would normally be redundant, but the fact that the URL
ends in '.git' is a big enough red flag that making it mandatory in this
case--and throwing a fatal error if it's omitted--seems warranted.

Granted, it feels slightly 'odd' when combined with #3, where a warning
is issued if "protocol=http" is given where it normally isn't needed.
Slap users on the wrist for supplying 'http=' in one case, then slap
them for *not* supplying it in another? It's... subtle. But the whole
business of having git-over-http SRC_URIs start with "git://" is equally
subtle. The SRC_URI syntax is immediately familiar to users, but it's
misleading in the case of git-over-http. It's a kind of 'false friend',
so I think it's reasonable for bitbake to work extra hard to protect
users from misuse of this variable.

protocol= is optional for the git fetcher, so in principle I agree it is possible for a user to change git:// to http:// where protocol= is not specified and therefore not get any kind of warning or error. Unfortunately though whilst it is most likely that .git at the end signifies a git repository, it doesn't guarantee it; and as you point out, .git is often not there in valid git repository URLs at all.

For the wget fetcher (which is responsible for handling http://, https:// and ftp://, protocol= is not used at all, and I would be loath to introduce it just for the sake of preventing a warning that is only somewhat effective.

We've introduced an explicit check and error for http:// + protocol=git which is the most common case of this that I have observed based on questions on the mailing list over the years, and in that case it is almost guaranteed that the user meant git:// + protocol=http. All other cases are less clear and much more difficult to handle practically.

Maybe there's some other way to address the 'quality' of the error message, then? This is what I see if I change "git://" to "http://" and "protocol=" is omitted:

> ERROR: Command Error: exit status: 1  Output:
> Applying patch 0001-Makefile.fwinst-fix-install-breakage-for-FW-images-r.patch
> patching file scripts/Makefile.fwinst
> Hunk #1 FAILED at 27.
> 1 out of 1 hunk FAILED -- rejects in file scripts/Makefile.fwinst
> Patch 0001-Makefile.fwinst-fix-install-breakage-for-FW-images-r.patch does not apply (enforce with -f)
> ERROR: Function failed: patch_do_patch

It just seems a little 'rude'. Is there some other way to make bitbake say the equivalent of: "That source tree you thought you were downloading? I can't find it, so... not only did the patch fail to apply, but.. there's nothing *to* patch!"

The thing is as I mentioned in the other bug report, this is a failure at do_patch. The fetch itself must have actually succeeded - the remote server just sent an html page with no error. There's no intrinsic way we can determine here that the result has not matched up with the user's intent, in the absence of protocol=git and branch= of course.

As far as the error message within do_patch is concerned, we're just repeating what the "patch" utility outputs; I'm not sure if there's a lot that can be done at our level about that if it's not satisfactory.

Fixed in master: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0e6cc44a111ff3189011d503217223745055a643

Issue #5
--------
With my local.conf mods set this way:

> MACHINE = "qemux86"
> CONNECTIVITY_CHECK_URIS=""
> BB_GENERATE_MIRROR_TARBALLS = "1" 
> BB_FETCH_PREMIRRORONLY = "1"
> PREMIRRORS_prepend = "\
>      git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"
> SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/"
> INHERIT += "own-mirrors"

when I execute:

> bitbake -c fetchall universe

the following packages are compiled:

> NOTE: package quilt-native-0.51-r1: task do_compile: Started
> NOTE: package gettext-minimal-native-0.18.1.1-r3: task do_compile: Started
> NOTE: package gnu-config-native-20111111-r1: task do_compile: Started
> NOTE: package m4-native-1.4.16-r2: task do_compile: Started
> NOTE: package autoconf-native-2.68-r7: task do_compile: Started
> NOTE: package automake-native-1.11.2-r3: task do_compile: Started
> NOTE: package libtool-native-2.4.2-r3.0: task do_compile: Started
> NOTE: package pkgconfig-native-0.25-r3: task do_compile: Started
> NOTE: package sqlite3-native-3.7.10-r2: task do_compile: Started
> NOTE: package tar-replacement-native-1.26-r1: task do_compile: Started
> NOTE: package pseudo-native-1.3-r10: task do_compile: Started
> NOTE: package ocf-linux-native-20100325-r3.0: task do_compile: Started
> NOTE: package zlib-native-1.2.6-r1: task do_compile: Started
> NOTE: package pigz-native-2.2.4-r2: task do_compile: Started
> NOTE: package openssl-native-1.0.0i-r0.2: task do_compile: Started
> NOTE: package expat-native-2.0.1-r1: task do_compile: Started
> NOTE: package perl-native-5.14.2-r0: task do_compile: Started
> NOTE: package curl-native-7.24.0-r2: task do_compile: Started
> NOTE: package git-native-1.7.7-r2: task do_compile: Started

This is 'expected behavior', as far as I can tell, I'm merely documenting
precisely what gets built on my machine to support the notion that it may be
worthwhile to pursue adding a 'create-mirror' command that does not build
anything.

Honestly, this is less of a problem for me now that I understand that the
default HTTP premirror(s) can be used as a kind of 'back door' to save my
builds from aborting when run from behind a firewall with missing
dependencies. The whole key for me has been realizing:

1. I need to export http_proxy=http://user:passwd@myproxy.com:8080

2. I have to add a PREMIRRORS_prepend line to explicitly steer bitbake
   to my desired fallback mirror(s)

(#2 may actually be required due to a bug that has been fixed on master[??])

The thing I'm (still) worried about is what happens when I build using layers
that aren't part of Yocto proper, such as meta-ivi and meta-ti. I'm assuming
these contain recipes pointing at sources that are not part of the Yocto HTTP
mirror.  If this is the case, I still find myself wishing for a way to quickly
download all the required dependencies. I suppose this could be sufficient:

> bitbake -c fetchall some-image-name

I'll try to build core-image-sato for my Pandaboard ES this way, using
meta-ti, and see what happens...

(In reply to comment #7)

Sorry, mis-posted this to the wrong issue, please ignore...