3222 2012-10-03 16:37:46 +0000 Add command to create a mirror without building anything 2013-10-04 16:10:21 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 1.3 x86 Multiple VERIFIED FIXED Medium enhancement 1.5 1 evadeflow richard.purdie corneliux.stoicescu poky.bs.watcher poky.watcher sgw --- --- No (bug/feature does not impact docs) oldest_to_newest 26165 0 evadeflow 2012-10-03 16:37:46 +0000 My 16-core build server is behind an HTTP-only firewall, and getting all required packages to it is a real pain. Basically, I go offsite and attempt to download everything to a folder on a USB hard drive with: bitbake -c fetchall universe bitbake -c fetchall <image_name> Then I rsync that folder over to my build server. I think the second command is supposed to be redundant, but in practice, it's not, for whatever reason: 'fetchall universe' definitely does not download everything required to build any arbitrary image my source tree is capable of building. And even the above commands aren't fool-proof. Inevitably, my first attempt to build a new image aborts due to some missing resource. Also, 'fetchall' builds quite a lot of support code that isn't strictly needed if all you're trying to do is pull down a bunch of tarballs and repositories. I'd love to see a new bitbake command that does what 'fetchall universe' seems like it should do, which is download every single SRC_URI called out in any recipe in my build tree--without building anything. I understand that there are possible complications with BB_GENERATE_MIRROR_TARBALLS and the version of tar on the host. I would recommend issuing a stern, ALL CAPS warning if the host tar is too old, or perhaps just refusing to run the command. The bottom line for me is: getting *all* the sources I need behind the firewall is just way, way, way harder than it should be. Computers are great at this kind of automated drudgery, so it should be a no-brainer. Why do I keep having to drive to the local public library with my laptop and USB hard drive to pull down "just one more tarball"? In summary, something like this: bitbake -c create-mirror /media/acme/poky-mirror that *just works* would be FANTASTIC. 26167 1 richard.purdie 2012-10-03 16:44:19 +0000 Which "support code that isn't strictly needed" is it building? Its likely your version of git is old/buggy and hence its building git-native which in itself pulls in a signifiant dependency list. Update to a version of git and tar meet the minimum version requirements and it will work as you describe. We're not going down the route of printing ALL CAPS WARNINGS about things we can actually deal non-fatally with ourselves. So as far as that part of this bug report goes, the resolution is WONTFIX. When you say unverse doesn't fetch something that is actually required, can you please give a specific example? This should work and I'd like to understand where it didn't work. Actually, if you're behind an http only firewall, I'd extremely surprised it couldn't get what it needed from the http sources mirror. So I'd appreciate more info about how this second part isn't working. 26171 2 evadeflow 2012-10-03 17:29:03 +0000 (In reply to comment #1) > Which "support code that isn't strictly needed" is it building? Its > likely your version of git is old/buggy and hence its building > git-native which in itself pulls in a signifiant dependency list. It's pseudo that's the problem. This has already been marked NOTABUG (see bug #3062), but perhaps it's a valid enhancement request for 1.4. Also, you mention: > Actually, if you're behind an http only firewall, I'd extremely surprised it > couldn't get what it needed from the http sources mirror. There's an http sources mirror?! :-% Where's that? This would be a *huge* help. If you can tell me how to point bitbake at it, I'll attempt to build core-image-sato behind the firewall and let you know the results. 26173 3 evadeflow 2012-10-03 18:25:27 +0000 (In reply to comment #2) > There's an http sources mirror?! :-% Where's that? Okay, I think I found it: - http://mirrors.kernel.org/yocto/yocto/yocto-1.2/sources/ I'll spin up a build using that soonest... 26174 4 evadeflow 2012-10-03 18:35:59 +0000 > Okay, I think I found it: > > - http://mirrors.kernel.org/yocto/yocto/yocto-1.2/sources/ Or maybe this: - http://downloads.yoctoproject.org/mirror/sources/ (They look substantially the same, but I guess I'll go with the second one, and fall back to the first...) 26184 5 richard.purdie 2012-10-03 22:35:43 +0000 Are you using the poky distro since it should be using this http mirror by default, perhaps not for the initial fetch but certainly falling back to it if the upstreams don't work... 26185 6 evadeflow 2012-10-03 22:55:26 +0000 (In reply to comment #5) > Are you using the poky distro since it should be using this http mirror by > default, perhaps not for the initial fetch but certainly falling back to it > if the upstreams don't work... Confirmed. It *does* fall back to downloading from whatever the default mirror(s) are configured to be, per the documentation. So it appears that PREMIRRORS_prepend line can be omitted, provided http_proxy is set suitably in the environment. I'm in the process of determining if this does, in fact, pull in everything needed to build core-image-sato... 26239 7 evadeflow 2012-10-04 15:02:50 +0000 Sorry for spamming the bug tracker a bit here, but I'd like to report some of the issues I've been encountering while trying to download sources from behind a firewall. This isn't working as smoothly as it should on denzil. The problems I'm having are the reason I suggested adding a new 'create-mirror' command, but maybe a better path is just to fix the issues I'm stubbing my toe on. Issue #1 -------- If I put this in local.conf: > MACHINE = qemux86 > CONNECTIVITY_CHECK_URIS="" > BB_GENERATE_MIRROR_TARBALLS = "1" > BB_FETCH_PREMIRRORONLY = "1" > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > INHERIT += "own-mirrors" and then run: > bitbake -DD -c fetchall universe I get a whole slew of errors along the lines of: > ERROR: Function failed: Network access disabled through BB_NO_NETWORK > but access rquested with command /usr/bin/env wget -t 2 -T 30 -nv > --passive-ftp --no-check-certificate -P > /home/evadeflow/projects/poky-git/build/downloads > 'ftp://ftp.gnu.org/gnu/m4/m4-1.4.16.tar.gz' (for url None) > ERROR: Logfile of failure stored in: /home/evadeflow/projects/poky-git/build/tmp/work/i686-linux/m4-native-1.4.16-r2/temp/log.do_fetch.30723 My expectation was that including the line that says: > BB_FETCH_PREMIRRORONLY = "1" would tell bitbake to not even bother trying to download anything from upstream sources, and just to use its hard-coded default mirrors. This does not work. I'm guessing this is somehow expected behavior, but I find it confusing. If I omit the BB_FETCH_PREMIRRORONLY line, it does download a bunch of stuff, but fails differently (see Issue #2, which I'll add in a bit). It also wastes time trying to download from locations I already know are going to fail: > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/m4/m4-1.4.16.tar.gz > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/libtool/libtool-2.4.2.tar.gz > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/automake/automake-1.11.2.tar.bz2 > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.68.tar.bz2 > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/tar/tar-1.26.tar.bz2 > WARNING: Failed to fetch URL http://www.zlib.net/zlib-1.2.6.tar.bz2 > WARNING: Failed to fetch URL http://zlib.net/pigz/pigz-2.2.4.tar.gz > WARNING: Failed to fetch URL https://fedorahosted.org/releases/e/l/elfutils/elfutils-0.148.tar.bz2 > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/ncurses/ncurses-5.9.tar.gz > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/readline/readline-6.2.tar.gz > WARNING: Failed to fetch URL ftp://xmlsoft.org/libxml2/libxml2-2.7.8.tar.gz > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/bison/bison-2.5.tar.gz > WARNING: Failed to fetch URL ftp://ftp.astron.com/pub/file/file-5.11.tar.gz > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/binutils/binutils-2.22.tar.bz2 > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/gettext/gettext-0.18.1.1.tar.gz > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/gmp/gmp-5.0.4.tar.bz2 > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/gperf/gperf-3.0.4.tar.gz > WARNING: Failed to fetch URL ftp://ftp.gnu.org/gnu/bash/bash-4.2.tar.gz;name=tarball > (etc, etc.) As a new user unfamiliar with bitbake, who just wants to ensure I've captured all the bits needed to build an image, all those warnings are kind of unnerving. How am I supposed to know that I can ignore them? I'd really like some way of simply fetching all required sources, with zero fanfare, that doesn't waste time trying to download from sources I don't want it to, and doesn't output any confusing warnings. I think the closest I can currently get is to put something like this in local.conf: > BB_FETCH_PREMIRRORONLY = "1" > PREMIRRORS_prepend = "\ > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" (I haven't confirmed yet whether the above works, but I think it does.) 26242 8 evadeflow 2012-10-04 15:15:46 +0000 Issue #2 -------- When I put this in my local.conf: > MACHINE = qemux86 > CONNECTIVITY_CHECK_URIS="" > BB_GENERATE_MIRROR_TARBALLS = "1" > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > INHERIT += "own-mirrors" and tried to run: > bitbake -DD -c fetchall universe everything looked to be progressing fine, but then I got into some kind of infinite loop where the following error kept getting printed over and over: > ERROR: Running idle function > Traceback (most recent call last): > File "/home/evadeflow/projects/poky-git/bitbake/lib/bb/cooker.py", line 1134, in buildTargetsIdle(server=<ProcessServer(ProcessServer-1, started)>, rq=<bb.runqueue.RunQueue instance at 0xd66b06c>, abort=False): > try: > > retval = rq.execute_runqueue() > except runqueue.TaskFailure as exc: > File "/home/evadeflow/projects/poky-git/bitbake/lib/bb/runqueue.py", line 974, in RunQueue.execute_runqueue(): > if self.state is runQueueCleanUp: > > self.rqexe.finish() > > File "/home/evadeflow/projects/poky-git/bitbake/lib/bb/runqueue.py", line 1088, in RunQueueExecuteTasks.finish(): > for pipe in self.build_pipes: > > self.build_pipes[pipe].read() > > File "/home/evadeflow/projects/poky-git/bitbake/lib/bb/runqueue.py", line 1808, in runQueuePipe.read(): > while index != -1: > > bb.event.fire_from_worker(self.queue[:index+8], self.d) > self.queue = self.queue[index+8:] > File "/home/evadeflow/projects/poky-git/bitbake/lib/bb/event.py", line 155, in fire_from_worker(event='<event>(ilogging\nLogRecord\np1\n(dp2\nS\'taskpid\'\np3\nI6425\nsS\'threadName\'\np4\nS\'MainThread\'\np5\nsS\'name\'\np6\nS\'BitBake.Util\'\np7\nsS\'thread\'\np8\nI-1215575264\nsS\'created\'\np9\nF1349360300.301301\nsS\'process\'\np10\nI6425\nsS\'processName\'\np11\nS\'ProcessServer-1\'\np12\nsS\'args\'\np13\n(tsS\'module\'\np14\nS\'utils\'\np15\nsS\'filename\'\np16\nS\'utils.py\'\np17\nsS\'levelno\'\np18\nI40\nsS\'exc_text\'\np19\nNsS\'pathname\'\np20\nS\'/home/evadeflow/projects/poky-git/bitbake/lib/bb/utils.py\'\np21\nsS\'lineno\'\np22\nI372\nsS\'msg\'\np23\nS\' data = "<event>" + pickle.dumps(event) + "</event>', d=<bb.data_smart.DataSmart object at 0x8b5720c>): > return > > event = pickle.loads(event[7:-8]) > fire_ui_handlers(event, d) > ValueError: insecure string pickle I suspect this is a known bug in denzil (I vaguely remember seeing a patch with the words "infinite loop" and "premirrors" in the subject)? I mention it here because it is part of what's contributing to my current "Making a local pre-mirror and/or working behind a firewall is *really* hard" mindset. For grins, I tried removing the '-DD' from the invocation and simply running: > bitbake -c fetchall universe This made no difference; I got the same infinite loop. Next I'll try explicitly pre-pending the premirrors I want to use, since I think this worked for me yesterday... 26247 9 evadeflow 2012-10-04 15:51:46 +0000 (In reply to comment #8) > Next I'll try > explicitly pre-pending the premirrors I want to use, since I think this > worked for me yesterday... Confirmed. If my local.conf mods look like this: > MACHINE = qemux86 > CONNECTIVITY_CHECK_URIS="" > BB_GENERATE_MIRROR_TARBALLS = "1" > BB_FETCH_PREMIRRORONLY = "1" > PREMIRRORS_prepend = "\ > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > INHERIT += "own-mirrors" then I'm able to successfully 'download the universe' from behind my firewall in about 25 minutes: > time bitbake -c fetchall universe > ... > NOTE: Tasks Summary: Attempted 2005 tasks of which 64 didn't need to be rerun and all succeeded. > > Summary: There were 12 WARNING messages shown. > bitbake -c fetchall universe 1537.14s user 337.40s system 124% cpu 25:00.71 total This is great news, but I am (unfortunately) not done exploring the rough edges of this process. More to come... [Aside: Why am I fetching the universe? I originally didn't understand that bitbake can and will fall back to http mirrors accessible from behind my company's firewall---provided that http_proxy is set appropriately in the environment. It seemed to happen to me several times that running, e.g., 'bitbake -c fethcall core-image-sato' while offsite didn't pull in all required packages, so I got paranoid. But I'm starting to believe that might have been a red herring, or been due to other causes.] 26250 10 evadeflow 2012-10-04 16:55:41 +0000 Issue #3 -------- This may be a known bug on denzil that has since been fixed, but when I set MACHINE to qemuarm in local.conf like so: > MACHINE = "qemuarm" > CONNECTIVITY_CHECK_URIS="" > BB_GENERATE_MIRROR_TARBALLS = "1" > BB_FETCH_PREMIRRORONLY = "1" > PREMIRRORS_prepend = "\ > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > INHERIT += "own-mirrors" and run: > bitbake -c fetchall universe it chugs for awhile (building pseudo and its dependencies) and then I get the following error: > OE Build Configuration: > BB_VERSION = "1.15.2" > TARGET_ARCH = "arm" > TARGET_OS = "linux-gnueabi" > MACHINE = "qemuarm" > DISTRO = "poky" > DISTRO_VERSION = "1.2.1" > TUNE_FEATURES = "armv5 dsp thumb arm926ejs" > TARGET_FPU = "soft" > meta > meta-yocto = "denzil:f6b83ac65be48a31510cda0df423f53d0f477046" > > NOTE: Resolving any missing task queue dependencies > ERROR: Nothing PROVIDES 'syslinux' > ERROR: syslinux was skipped: incompatible with host arm-poky-linux-gnueabi (not in COMPATIBLE_HOST) > ERROR: Required build target 'self-hosted-image' has no buildable providers. > Missing or unbuildable dependency chain was: ['self-hosted-image', 'syslinux'] > > Summary: There was 1 WARNING message shown. > Summary: There were 2 ERROR messages shown, returning a non-zero exit code. 26257 11 evadeflow 2012-10-04 17:31:29 +0000 Issue #4 -------- With my local.conf mods set this way: > MACHINE = "qemux86" > CONNECTIVITY_CHECK_URIS="" > BB_GENERATE_MIRROR_TARBALLS = "1" > BB_FETCH_PREMIRRORONLY = "1" > PREMIRRORS_prepend = "\ > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > INHERIT += "own-mirrors" I'm able to successfully execute the command: > bitbake -c fetchall universe However, it puts all the downloaded resources in build/downloads, when I expected them to wind up in /home/evadeflow/project/poky-mirror. This is perhaps a bug in my expectations. I can certainly move or copy build/downloads to wherever I want it and delete all the "*.done" files, but I can't imagine I'm the only bitbake user who has to do this. It would be nice if there were some way to get the computer to do it for us. 26258 12 evadeflow 2012-10-04 17:32:02 +0000 Issue #5 -------- With my local.conf mods set this way: > MACHINE = "qemux86" > CONNECTIVITY_CHECK_URIS="" > BB_GENERATE_MIRROR_TARBALLS = "1" > BB_FETCH_PREMIRRORONLY = "1" > PREMIRRORS_prepend = "\ > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > INHERIT += "own-mirrors" when I execute: > bitbake -c fetchall universe the following packages are compiled: > NOTE: package quilt-native-0.51-r1: task do_compile: Started > NOTE: package gettext-minimal-native-0.18.1.1-r3: task do_compile: Started > NOTE: package gnu-config-native-20111111-r1: task do_compile: Started > NOTE: package m4-native-1.4.16-r2: task do_compile: Started > NOTE: package autoconf-native-2.68-r7: task do_compile: Started > NOTE: package automake-native-1.11.2-r3: task do_compile: Started > NOTE: package libtool-native-2.4.2-r3.0: task do_compile: Started > NOTE: package pkgconfig-native-0.25-r3: task do_compile: Started > NOTE: package sqlite3-native-3.7.10-r2: task do_compile: Started > NOTE: package tar-replacement-native-1.26-r1: task do_compile: Started > NOTE: package pseudo-native-1.3-r10: task do_compile: Started > NOTE: package ocf-linux-native-20100325-r3.0: task do_compile: Started > NOTE: package zlib-native-1.2.6-r1: task do_compile: Started > NOTE: package pigz-native-2.2.4-r2: task do_compile: Started > NOTE: package openssl-native-1.0.0i-r0.2: task do_compile: Started > NOTE: package expat-native-2.0.1-r1: task do_compile: Started > NOTE: package perl-native-5.14.2-r0: task do_compile: Started > NOTE: package curl-native-7.24.0-r2: task do_compile: Started > NOTE: package git-native-1.7.7-r2: task do_compile: Started This is 'expected behavior', as far as I can tell, I'm merely documenting precisely what gets built on my machine to support the notion that it may be worthwhile to pursue adding a 'create-mirror' command that does not build anything. Honestly, this is less of a problem for me now that I understand that the default HTTP premirror(s) can be used as a kind of 'back door' to save my builds from aborting when run from behind a firewall with missing dependencies. The whole key for me has been realizing: 1. I need to export http_proxy=http://user:passwd@myproxy.com:8080 2. I have to add a PREMIRRORS_prepend line to explicitly steer bitbake to my desired fallback mirror(s) (#2 may actually be required due to a bug that has been fixed on master[??]) The thing I'm (still) worried about is what happens when I build using layers that aren't part of Yocto proper, such as meta-ivi and meta-ti. I'm assuming these contain recipes pointing at sources that are not part of the Yocto HTTP mirror. If this is the case, I still find myself wishing for a way to quickly download all the required dependencies. I suppose this could be sufficient: > bitbake -c fetchall some-image-name I'll try to build core-image-sato for my Pandaboard ES this way, using meta-ti, and see what happens... 26259 13 sgw 2012-10-04 18:06:14 +0000 (In reply to comment #11) > Issue #4 > -------- > With my local.conf mods set this way: > > > MACHINE = "qemux86" > > CONNECTIVITY_CHECK_URIS="" > > BB_GENERATE_MIRROR_TARBALLS = "1" > > BB_FETCH_PREMIRRORONLY = "1" > > PREMIRRORS_prepend = "\ > > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" > > SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/" > > INHERIT += "own-mirrors" > > I'm able to successfully execute the command: > > > bitbake -c fetchall universe > > However, it puts all the downloaded resources in build/downloads, when I > expected them to wind up in /home/evadeflow/project/poky-mirror. This is > perhaps a bug in my expectations. I can certainly move or copy > build/downloads > to wherever I want it and delete all the "*.done" files, but I can't imagine > I'm the only bitbake user who has to do this. It would be nice if there were > some way to get the computer to do it for us. Wow, you have a lot going on here. I think there might be a miss-understanding of a couple of the above flags. First the BB_GENERATE_MIRROR_TARBALLS is for git related fetchs only, since you are doing everything from the PREMIRROR and getting tarballs, this will not do much for you. It will work when you are connected outside the fire wall and attempt to fetch directly from a git repo. When you set SOURCE_MIRROR_URL and then inherit own-mirrors, you are essentially setting PREMIRRORS to be your file location, it might be better to set the DL_DIR = "/home/evadeflow/project/poky-mirror" and SOURCE_MIRROR_URL = "http://downloads.yoctoproject.org/mirror/sources/" That will cause all the tarballs to be loaded into poky-mirror directory, you can the re-use it via DL_DIR for multiple builds if you want. 26261 14 sgw 2012-10-04 18:07:39 +0000 *** Bug 2287 has been marked as a duplicate of this bug. *** 26263 15 evadeflow 2012-10-04 20:49:46 +0000 (In reply to comment #13) > ... > I think there might be a miss-understanding of a couple of the above flags. > > First the BB_GENERATE_MIRROR_TARBALLS is for git related fetchs only... Really? Wouldn't a better name be BB_GENERATE_GIT_TARBALLS (or something), then? I'm still very new to Yocto/bitbake, so I have no expectations whatsoever about what it means to "download from a mirror". I assumed that the mirrors were simply clones of the required git repos made available via http. Indeed, it appears that this is how it works for svn repos. But now that I look carefully, I see that http://downloads.yoctoproject.org/mirror/sources contains a bunch of tarballs named: > git2_git.denx.de.u-boot.git.tar.gz.done > git2_git.eclipse.org.gitroot.tcf.org.eclipse.tcf.agent.git.tar.gz > git2_git.efficios.com.babeltrace.git.tar.gz.done > git2_git.fedorahosted.org.git.newt.tar.gz > git2_git.freescale.com.ppc.sdk.boot-format.git.tar.gz > git2_git.gnome.org.clutter-box2d.git.tar.gz > git2_git.gnome.org.clutter.tar.gz > git_gitorious.org.x-loader.x-loader.git_6f3a26101303051e0f91b6213735b68ce804e94e.tar.gz > git_moblin.org.repos.users.pippin.prototype.git._8f2cb524fe06555182c25b4ba3202d7b368ac0ce.tar.gz > git_people.freedesktop.org.~jimmac.Darkilouche.git_b4ce102a303d1176107a9d9d28c14f7da78feedd.tar.gz > git_sources.redhat.com.git.systemtap.git.tar.gz > (etc., etc) [Idle curiosity: why do some mirror tarball names start with 'git2_', and others start with 'git_'? Maybe I'm better off not knowing! `:-}] It looks like the (git) content of these mirrors was generated using BB_GENERATE_MIRROR_TARBALLS, so I think you're telling me that BB_FETCH_PREMIRRORONLY and BB_GENERATE_MIRROR_TARBALLS are mutually exclusive options? Thanks for pointing this out! Perhaps bitbake should exit with an error if both of these options are enabled. > When you set SOURCE_MIRROR_URL and then inherit own-mirrors, you are > essentially setting PREMIRRORS to be your file location, it might be better > to set the DL_DIR = "/home/evadeflow/project/poky-mirror" and > SOURCE_MIRROR_URL = "http://downloads.yoctoproject.org/mirror/sources/" > > That will cause all the tarballs to be loaded into poky-mirror directory, > you can the re-use it via DL_DIR for multiple builds if you want. Okay, I totally misunderstood this, thanks for the heads-up. I was thinking that I could get bitbake to both fetch from external premirrors *and* create a local premirror for me in one go. It seems this is not possible---at least not the way I was trying to do it. I wish I could find documentation somewhere that spelled this out in the simplest possible terms. (Or, better yet, perhaps a 'create-mirror' target that would just do it for me, no config required. :-}) While I'm thinking about it, what the heck *is* a 'premirror'? Without any context, it's kind of a meaningless term. Is it just "a list of mirrors checked prior to checking the 'real' mirrors"? This FAQ question: - http://www.yoctoproject.org/docs/current/poky-ref-manual/poky-ref-manual.html#id1252014 would seem to indicate that (quite succinctly, I might add): > When the build system searches for source code, it first tries the > local download directory. If that location fails, Poky tries > PREMIRRORS, the upstream source, and then MIRRORS in that order. Okay, I'm starting to get it. Thanks for your patience as I put it all together. I hope that my out-loud fumbling here is helpful in figuring out how to make bitbake more accessible to new users. If it's not, please let me know and I'll move these musings to the mailing list. (Or go back to mumbling quietly at my desk...) 26264 16 evadeflow 2012-10-04 21:08:40 +0000 (In reply to comment #12) > I'll try to build core-image-sato for my Pandaboard ES this way, using > meta-ti, and see what happens... With local.conf set as follows: > MACHINE = "pandaboard" > BBMASK = "meta-ti/recipes-misc" > CONNECTIVITY_CHECK_URIS="" > BB_FETCH_PREMIRRORONLY = "1" > PREMIRRORS_prepend = "\ > git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ > svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n" When I run the cmd: > bitbake -c fetchall core-sato-image I (not surprisingly) get a few errors: > ERROR: Function failed: Network access disabled through BB_NO_NETWORK but access rquested with command git clone --bare --mirror http://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git /home/evadeflow/projects/poky-git/build/downloads/git2/dev.omapzoom.org.pub.scm.integration.kernel-ubuntu.git (for url None) > ERROR: Function failed: Network access disabled through BB_NO_NETWORK but access rquested with command git clone --bare --mirror git://www.denx.de/git/u-boot.git /home/evadeflow/projects/poky-git/build/downloads/git2/www.denx.de.git.u-boot.git (for url None) I totally expected this, so this was a 'watch it fail' kind of test. One nit: with my new user's eyes, this error message is still a bit confusing. BB_NO_NETWORK? *I* didn't set that variable. A more helpful message would be: > ERROR: Function failed: BB_FETCH_PREMIRRORONLY set, but non-premirror access requested with command git clone --bare --mirror http://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git /home/evadeflow/projects/poky-git/build/downloads/git2/dev.omapzoom.org.pub.scm.integration.kernel-ubuntu.git (for url None) Anyway, if I comment out the line in local.conf that reads BB_FETCH_PREMIRRORONLY="1" and re-run the same command, everything downloads fine, and it takes approximately 19 minutes. Then, when I go to actually build the image with: > bitbake core-image-sato it builds successfully in about two hours! This is awesome! If it does turn out that Issue #2 is due to a bug that has been fixed in 1.3 (can someone confirm this?), and it is therefore no longer necessary to manually add the PREMIRRORS_prepend line, I think I'm pretty happy with the status quo. A 'create-mirror' target/command might be a nice addition, but it is by no means a necessity. It might be more effective to just tighten up the error reporting and documentation. It occurs to me that maybe the reason I was seeing that infinite loop in Issue #2 is the 'weird' combination of options I was using. I'll check this and report back... 26268 17 evadeflow 2012-10-04 22:47:02 +0000 (In reply to comment #16) > It occurs to me that maybe the reason I was seeing that infinite loop in > Issue #2 is the 'weird' combination of options I was using. I'll check > this and report back... FWIW, my tests of this seem to indicate that the weird cocktail of variables I was using are not the cause of the 'Issue #2' behavior. I stripped down my local.conf so that the only modification from the default created by 'source oe-init-build-env' was to add: > CONNECTIVITY_CHECK_URIS="" Then I ran: > bitbake -c fetchall universe and it still repeatably exhibits the same infinite error loop ("ValueError: insecure string pickle"). It seems the only way I can get rid of this is to explicitly prepend to PREMIRRORS. 26292 18 evadeflow 2012-10-05 15:08:54 +0000 (In reply to comment #8) > Issue #2 > -------- > ... > everything looked to be progressing fine, but then I got into some kind > of infinite loop... > ... > I suspect this is a known bug in denzil (I vaguely remember seeing a > patch with the words "infinite loop" and "premirrors" in the subject)? CONFIRMED. This is due to bug #3073. If I manually apply this patch: - http://patchwork.openembedded.org/patch/36547/ in my sandbox, the problem goes away. I'm able to execute: > bitbake -c fetchall universe with no problems from behind my company's HTTP-only firewall, provided my local.conf contains the line: > CONNECTIVITY_CHECK_URIS= "" This fix has been back-ported to denzil on 9/28 (about a week ago), but I didn't have it in my work tree because I haven't yet figured out a way to work around our firewall's blocking of 'git://' repo URIs... 26563 19 evadeflow 2012-10-12 21:52:19 +0000 (In reply to comment #16) > ERROR: Function failed: Network access disabled through BB_NO_NETWORK but access rquested with command git clone --bare --mirror http://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git /home/evadeflow/projects/poky-git/build/downloads/git2/dev.omapzoom.org.pub.scm.integration.kernel-ubuntu.git (for url None) > ERROR: Function failed: Network access disabled through BB_NO_NETWORK but access rquested with command git clone --bare --mirror git://www.denx.de/git/u-boot.git /home/evadeflow/projects/poky-git/build/downloads/git2/www.denx.de.git.u-boot.git (for url None) > > I totally expected this, so this was a 'watch it fail' kind of test. > One nit: with my new user's eyes, this error message is still a bit > confusing. BB_NO_NETWORK? *I* didn't set that variable... I tracked this down to the following code of the Fetch.download() method in bitbake/lib/bb/fetch2/__init.py__ (~line 1185): > if premirroronly: > self.d.setVar("BB_NO_NETWORK", "1") So: when BB_FETCH_PREMIRRORONLY is set, bitbake is setting BB_NO_NETWORK after it has failed to download from any premirror, solely for the side effect this has of causing m.download() to later raise a NetworkAccess exception. This seems slightly sneaky, but I don't yet know the code well enough to know whether this is a one-off hack for convenience, or an instance of a generally-accepted paradigm within the bitbake code base. (I hope it's the former.) 34004 20 richard.purdie 2013-06-20 12:45:17 +0000 There were some problems with fetching everything which were fixed in master with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=5af68d42ba62a2a18d1b0cdacad8e32c2b119255 and http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=de1841d9c2a0ec1d7103335d450b5fbc2273804e so the fetchall command now does really fetch everything a given build target needs. I also improved the warning message mentioned in one of the comments with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=801fbe5a0c1316ac54a3e8ea61c42a74e28bb0ed I'm therefore going to mark this bug as resolved. If there are remaining issues I think we need to split them out into specific bugs for specific issues. 37644 21 corneliux.stoicescu 2013-10-04 16:10:21 +0000 Tested like this: 1) bitbake -c fetchall core-image-sato 2) unplug ethernet cable 3) bitbake core-image-sato The build finished successfully. Marking as VERIFIED on: poky master:a02280f99f05b1c21de2a78de9222a2f62a0dabc