3214 2012-10-03 14:04:18 +0000 Emit helpful message when SRC_URI="http://" is used with "protocol=git" 2012-10-04 17:27:20 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 1.3 x86 Multiple RESOLVED FIXED Undecided enhancement --- 1 evadeflow richard.purdie bluelightning poky.bs.watcher poky.watcher --- --- --- oldest_to_newest 26137 0 evadeflow 2012-10-03 14:04:18 +0000 For the motivation for this ticket, see the following mailing list message: http://lists.yoctoproject.org/pipermail/yocto/2012-October/012022.html Basically, SRC_URI's like the following: > SRC_URI = "http://dev.omapzoom.org/pub/scm/integration/kernel ubuntu.git;protocol=git;branch=ti-ubuntu-3.1-1282 \ > file://0001-Makefile.fwinst-fix-install-breakage-for-FW-images-r.patch \ > file://defconfig \ do not work, and they're not supposed to work. Whenever the source is a git repository, SRC_URI should start with 'git://'; if git over http is desired, this should be specified by including "protocol=http" in the SRC_URI string. However, new users of bitbake who are familiar with git are extremely likely to make the mistake of specifying git SRC_URI's that begin with 'http://', because this is how it's done on the git command line. When a user mistakenly enters a SRC_URI that points at a git repository, but begins with 'http://', the error generated seems pretty mysterious: > ERROR: Command Error: exit status: 1 Output: > Applying patch 0001-Makefile.fwinst-fix-install-breakage-for-FW-images-r.patch > patching file scripts/Makefile.fwinst > Hunk #1 FAILED at 27. > 1 out of 1 hunk FAILED -- rejects in file scripts/Makefile.fwinst > Patch 0001-Makefile.fwinst-fix-install-breakage-for-FW-images-r.patch does not apply (enforce with -f) > ERROR: Function failed: patch_do_patch Huh? It would be nice if bitbake would: 1) Exit with an error if SRC_URI contains "protocol=git", but starts with "http://", since this (apparently) does not work under any circumstances and is clearly incorrect 2) Exit with an error if: 1. SRC_URI begins with "http://" 2. The pointed-to URL ends in ".git" (e.g. "http://github.com/foo/project.git"), and 3. SRC_URI Does NOT contain "protocol=http" 3) Issue a warning if SRC_URI begins with "http://" and "protocol=<whatever>" or "branch=<whatever>" are specified [OPTIONAL] The rationale for the last two items is that not all git repositories end in ".git"; this is merely a common convention. It's possible (though VERY unlikely) that someone might want to download a URI that ends in '.git' via http. If this is REALLY what they want, they should (redundantly) specify "protocol=http" to indicate this. If someone specifies a SRC_URI like: > SRC_URI = "http://foo.bar.com/pub/myrepo;protocol=http;branch=mybranch" it's pretty clear that they don't intend to download via http. Even if they only say: > SRC_URI = "http://foo.bar.com/pub/myrepo;protocol=http" it seems like a red flag that "protocol=http" is (redundantly) specified, so a warning seems advisable, in case the user actually intended to point at a git or svn repo. Maybe what I've suggested above isn't ideal, but it definitely seems like error-handling for malformed git SRC_URI's could be tightened up a bit... 26141 1 bluelightning 2012-10-03 14:47:14 +0000 I've already submitted a patch to raise an error explicitly with advice if protocol=git is detected with http:// or https:// URIs (your #1 above); it catches the most common mistake. I'm not convinced we can really apply the check in #2 as this is still a valid URI. #3 might be useful perhaps but that's probably something to fix in the next release. 26146 2 evadeflow 2012-10-03 15:40:06 +0000 (In reply to comment #1) > I've already submitted a patch to raise an error explicitly with advice if > protocol=git is detected with http:// or https:// URIs (your #1 above); it > catches the most common mistake. I'm not convinced we can really apply the > check in #2 as this is still a valid URI. #3 might be useful perhaps but > that's probably something to fix in the next release. Is it mandatory for SRC_URIs starting with "git://" to also contain "protocol=XXX"? I had the impression that this was optional, and that bitbake would default to "protocol=git" if omitted[?] If this is the case, then the check for #1 won't help users who try to change: > SRC_URI = "git://github.com/zeromq/pyzmq.git;..." [protocol unspecified] to: > SRC_URI = "http://github.com/zeromq/pyzmq.git;..." [protocol unspecified] thinking that this will work just like git works. A gentle nudge towards the right syntax will, I think, be very much appreciated by new users. If "protocol=" is NOT optional when SRC_URI starts with "git://", then I agree that #1 takes care of just about every posible case, and the rest of this comment probably doesn't apply. (I assume that making it mandatory--if it currently is not--would break a lot of recipes.) Regarding #2, if someone puts: > SRC_URI = "http://github.com/zeromq/pyzmq.git;..." in a recipe, you can be 99% certain he actually meant to type: > SRC_URI = "git://github.com/zeromq/pyzmq.git;..." or possibly: > SRC_URI = "git://github.com/zeromq/pyzmq.git;protocol=http;..." I think it's reasonable to force them to type this instead: > SRC_URI = "http://github.com/zeromq/pyzmq.git;protocol=http" if they REALLY want to download the HTML source of a web page. The 'protocol=http' would normally be redundant, but the fact that the URL ends in '.git' is a big enough red flag that making it mandatory in this case--and throwing a fatal error if it's omitted--seems warranted. Granted, it feels slightly 'odd' when combined with #3, where a warning is issued if "protocol=http" is given where it normally isn't needed. Slap users on the wrist for supplying 'http=' in one case, then slap them for *not* supplying it in another? It's... subtle. But the whole business of having git-over-http SRC_URIs start with "git://" is equally subtle. The SRC_URI syntax is immediately familiar to users, but it's misleading in the case of git-over-http. It's a kind of 'false friend', so I think it's reasonable for bitbake to work extra hard to protect users from misuse of this variable. 26149 3 bluelightning 2012-10-03 15:51:50 +0000 protocol= is optional for the git fetcher, so in principle I agree it is possible for a user to change git:// to http:// where protocol= is not specified and therefore not get any kind of warning or error. Unfortunately though whilst it is most likely that .git at the end signifies a git repository, it doesn't guarantee it; and as you point out, .git is often not there in valid git repository URLs at all. For the wget fetcher (which is responsible for handling http://, https:// and ftp://, protocol= is not used at all, and I would be loath to introduce it just for the sake of preventing a warning that is only somewhat effective. We've introduced an explicit check and error for http:// + protocol=git which is the most common case of this that I have observed based on questions on the mailing list over the years, and in that case it is almost guaranteed that the user meant git:// + protocol=http. All other cases are less clear and much more difficult to handle practically. 26156 4 evadeflow 2012-10-03 16:08:30 +0000 Maybe there's some other way to address the 'quality' of the error message, then? This is what I see if I change "git://" to "http://" and "protocol=" is omitted: > ERROR: Command Error: exit status: 1 Output: > Applying patch 0001-Makefile.fwinst-fix-install-breakage-for-FW-images-r.patch > patching file scripts/Makefile.fwinst > Hunk #1 FAILED at 27. > 1 out of 1 hunk FAILED -- rejects in file scripts/Makefile.fwinst > Patch 0001-Makefile.fwinst-fix-install-breakage-for-FW-images-r.patch does not apply (enforce with -f) > ERROR: Function failed: patch_do_patch It just seems a little 'rude'. Is there some other way to make bitbake say the equivalent of: "That source tree you thought you were downloading? I can't find it, so... not only did the patch fail to apply, but.. there's nothing *to* patch!" 26160 5 bluelightning 2012-10-03 16:17:15 +0000 The thing is as I mentioned in the other bug report, this is a failure at do_patch. The fetch itself must have actually succeeded - the remote server just sent an html page with no error. There's no intrinsic way we can determine here that the result has not matched up with the user's intent, in the absence of protocol=git and branch= of course. As far as the error message within do_patch is concerned, we're just repeating what the "patch" utility outputs; I'm not sure if there's a lot that can be done at our level about that if it's not satisfactory. 26166 6 richard.purdie 2012-10-03 16:38:26 +0000 Fixed in master: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0e6cc44a111ff3189011d503217223745055a643 26254 7 evadeflow 2012-10-04 17:25:06 +0000 Issue #5 -------- With my local.conf mods set this way: > MACHINE = "qemux86" > CONNECTIVITY_CHECK_URIS="" > BB_GENERATE_MIRROR_TARBALLS = "1" > BB_FETCH_PREMIRRORONLY = "1" > PREMIRRORS_prepend = "\ > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > INHERIT += "own-mirrors" when I execute: > bitbake -c fetchall universe the following packages are compiled: > NOTE: package quilt-native-0.51-r1: task do_compile: Started > NOTE: package gettext-minimal-native-0.18.1.1-r3: task do_compile: Started > NOTE: package gnu-config-native-20111111-r1: task do_compile: Started > NOTE: package m4-native-1.4.16-r2: task do_compile: Started > NOTE: package autoconf-native-2.68-r7: task do_compile: Started > NOTE: package automake-native-1.11.2-r3: task do_compile: Started > NOTE: package libtool-native-2.4.2-r3.0: task do_compile: Started > NOTE: package pkgconfig-native-0.25-r3: task do_compile: Started > NOTE: package sqlite3-native-3.7.10-r2: task do_compile: Started > NOTE: package tar-replacement-native-1.26-r1: task do_compile: Started > NOTE: package pseudo-native-1.3-r10: task do_compile: Started > NOTE: package ocf-linux-native-20100325-r3.0: task do_compile: Started > NOTE: package zlib-native-1.2.6-r1: task do_compile: Started > NOTE: package pigz-native-2.2.4-r2: task do_compile: Started > NOTE: package openssl-native-1.0.0i-r0.2: task do_compile: Started > NOTE: package expat-native-2.0.1-r1: task do_compile: Started > NOTE: package perl-native-5.14.2-r0: task do_compile: Started > NOTE: package curl-native-7.24.0-r2: task do_compile: Started > NOTE: package git-native-1.7.7-r2: task do_compile: Started This is 'expected behavior', as far as I can tell, I'm merely documenting precisely what gets built on my machine to support the notion that it may be worthwhile to pursue adding a 'create-mirror' command that does not build anything. Honestly, this is less of a problem for me now that I understand that the default HTTP premirror(s) can be used as a kind of 'back door' to save my builds from aborting when run from behind a firewall with missing dependencies. The whole key for me has been realizing: 1. I need to export http_proxy=http://user:passwd@myproxy.com:8080 2. I have to add a PREMIRRORS_prepend line to explicitly steer bitbake to my desired fallback mirror(s) (#2 may actually be required due to a bug that has been fixed on master[??]) The thing I'm (still) worried about is what happens when I build using layers that aren't part of Yocto proper, such as meta-ivi and meta-ti. I'm assuming these contain recipes pointing at sources that are not part of the Yocto HTTP mirror. If this is the case, I still find myself wishing for a way to quickly download all the required dependencies. I suppose this could be sufficient: > bitbake -c fetchall some-image-name I'll try to build core-image-sato for my Pandaboard ES this way, using meta-ti, and see what happens... 26255 8 evadeflow 2012-10-04 17:27:20 +0000 (In reply to comment #7) Sorry, mis-posted this to the wrong issue, please ignore...