It would at least build subversion-native, maybe git-native depending on the version of git you have. What exactly did it build?

I've had to return to the office, so it's a little hard to confirm. Now when I
try to execute:

  % bitbake --cmd fetchall --continue world

from behind the firewall, I get the error:

  ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
      Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
      Following is the list of potential problems / advisories:
  
      Failed to fetch test data from the network. Please ensure your network is configured correctly.


This isn't surprising, since the failed command was:

 git ls-remote git://git.yoctoproject.org/yocto-firewall-test HEAD


and my company's firewall definitely blocks this. (This is the reason I went
offsite in the first place.)

I'm pretty sure I saw the following targets being built when I was outside the
firewall:

  - apr
  - util-linux
  - openssl
  - python-native

which is when I started to get suspicious. I'll re-run this as soon as I can
and attach the complete output...

If you didn't already, enable BB_NUMBER_THREADS to something higher than one in local.conf. It will enable multiple fetches in parallel.

You can disable the sanity test by setting CONNECTIVITY_CHECK_URIS = "", you can also completely disable use of the network by setting BB_NO_NETWORK = "1".

If you are limited to http, you approach makes sense.

You can also try "bitbake -c fetch universe"

That will pull all recipes available.

Created attachment 764 [details]
Output of bitbake --cmd fetchall world

It seems like --fetchall builds many more targets that strictly necessary...

Really, Bugzilla? :-/  I meant for that attachment to have the following comments:

I re-started this process from scratch by running the following commands:

  % git clone git://git.yoctoproject.org/poky.git
  % cd poky
  % source oe-init-build-env
  % bitbake --cmd fetchall --continue world 2>&1 | tee bitbake-fetchall.log

Unfortunately, I had an appointment and couldn't wait for it to complete, but I'm attaching the output for the first 40 minutes or so.  I'm (still) surprised by the number of targets that get built. It just seems like this can't be right(?) I only want to download everything to sneak it behind the firewall and onto on our nice, big build server. Why is the world getting built on my poor little Linux VM?

Agreed, I can reproduce this locally, there is clearly a problem it should not be building a lot of those things, especially something like grub!

(In reply to comment #7)
> Agreed, I can reproduce this locally, there is clearly a problem it should
> not be building a lot of those things, especially something like grub!

My mistake, I saw grub but did not see it was a fetch, here is the list of things that I saw build after the initial 63 which is needed for pseudo.

Here is the list that builds initially.

gettext-minimal-native-0.18.1.1-r3
libcap-native-2.22-r3
zlib-native-1.2.7-r0
ocf-linux-native-20100325-r3.0
pigz-native-2.2.5-r0
bzip2-native-1.0.6-r5
libgpg-error-native-1.10-r2
lzo-native-2.06-r1
gdbm-native-1.10-r3
expat-native-2.1.0-r0
libtasn1-native-2.13-r0
libgcrypt-native-1.5.0-r1
ncurses-native-5.9-r10.1
readline-native-6.2-r3
util-linux-native-2.21.2-r1
openssl-native-1.0.0j-r15.3
apr-native-1.4.6-r1
apr-util-native-1.4.1-r1
gnutls-native-2.12.20-r8.3
python-native-2.7.3-r0.1
libxml2-native-2.8.0-r2
neon-native-0.29.6-r2
subversion-native-1.7.6-r0


As Richard mentioned, did you set BB_NUMBER_THREADS, that might make a difference.  Also, you sound like you are running in a VM of some kind, what tools are you using? Can you give it multiple cpus?

I believe I'm seeing about the same number of packages being compiled--or would be, if I let it run long enough. And yes, I'm compiling in a VM under VMWare Fusion, running on a MacBook Air. It's not an ideal build environment; but I'm not trying to build, I only want to download the sources to create a pre-mirror. I was surprised that *anything* needed to be built, but it sounds like you're saying this is expected behavior.

Is it really necessary to build Subversion from source just to create a pre-mirror?  It would be nice if there were some way to tell bitbake to just use the system-installed versions of svn/git/etc when doing a fetchall rather than building these things from scratch. Perhaps there's a way to create a recipe to do this, but I'm not familiar enough with the tool set to see how. Do I *really* need to build such a big chunk of the world just to get a copy of the upstream sources and patches?

So building the 23 packages that subversion needs as dependencys is a small fraction of the 800+ packages that are part of the Yocto Project.  We build a native subversion because we rely on features of newer subversion.  

There has been some investigation into trying to determine the version of a given host application (like subversion), in order to not build it, but that has not been implemented yet.

I understand your build issue, if you can set up your Fusion to take advantage of the number of core your MacBook Air has, it possible to speed up that build. If you are running with 1 Core then it will be slow, but if you can increase the number of cores it will speed up dramaticly. On a host with 8 cores for example it takes about 10-15 minutes to download all the upstream source, build and download time.

I understand, thanks for the detailed explanation. In hindsight, I probably should've posted this to the mailing list. I downloaded the universe overnight and it pulled 2041 packages, so this will (I hope) not be a problem for me any longer.  I'll set CONNECTIVITY_CHECK_URIS = "" and BB_NO_NETWORK = "1" as you suggested to make sure I'm not inadvertently pulling in packages that aren't in my mirror, and make trips offsite as necessary to download any that I've missed.

I guess you could leave issue this open as a feature request, or just close it if you have some other way of tracking these things...

> it pulled 2041 packages...

CORRECTION: it pulled 2641 packages. (Probably not important, but posts like this can have a pretty long tail and I didn't want to be off by 600[!] if somebody stumbles across this later...)

Thanks for your comments, I will close this as not a bug, we know about the subversion issue and I think that it's being worked on for the 1.4 (next) release

It seems that creating a symlink from ${COREBASE}/downloads to (e.g.) /home/poky-mirror is not sufficient. When I tried this, I got this error:

> ERROR: ExpansionError during parsing
> /media/MagnetiMarelli/remus/poky-git/meta/recipes-kernel/linux/linux-yocto_3.4.bb:
> Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)}
> which triggered exception NetworkAccess: Network access disabled through
> BB_NO_NETWORK but access requested with command git ls-remote
> git://git.yoctoproject.org/linux-yocto-3.4.git standard/beagleboard (for url
> None) ERROR: Command execution failed: Exited with 1

I'm surprised by this, because this resource does exist in the mirror:

poky-git-ivi% ls -la downloads 
lrwxrwxrwx 1 evadeflow evadeflow 17 Sep  6 10:25 downloads -> ../../poky-mirror/
poky-git-ivi% ls downloads/git2/git.yoctoproject.org.linux-yocto-3.4.git
branches/  description  HEAD    info/     packed-refs
config     FETCH_HEAD   hooks/  objects/  refs/

It seems bitbake doesn't know how to extract content from bare repos in a mirror. (This is a known bug, possibly?) I'm guessing I should have set BB_GENERATE_MIRROR_TARBALLS = "1" when I last went through this exercise.

The following URLs seem to indicate the 'right' way to achieve my desired goal:

  - http://www.mail-archive.com/yocto@yoctoproject.org/msg04906.html
  - https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F

I'll try this and see if it helps...

For the record (and for anybody else who stumbles upon this page), the steps in my last comment worked. I thought I could create a mirror by doing a fetchall and symlinking all my build trees to the same downloads folder. This doesn't work. You have to inherit own-mirrors and set SOURCE_MIRROR_URL as explained in the above links. (I didn't test whether setting BB_GENERATE_MIRROR_TARBALLS = "1" is strictly necessary; but it should be faster, anyway, so I just did it.)

Issue #4
--------
With my local.conf mods set this way:

> MACHINE = "qemux86"
> CONNECTIVITY_CHECK_URIS=""
> BB_GENERATE_MIRROR_TARBALLS = "1" 
> BB_FETCH_PREMIRRORONLY = "1"
> PREMIRRORS_prepend = "\
>      git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
>      svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"
> SOURCE_MIRROR_URL ?= "file:///home/evadeflow/projects/poky-mirror/"
> INHERIT += "own-mirrors"

I'm able to successfully execute the command:

> bitbake -c fetchall universe

However, it puts all the downloaded resources in build/downloads, when I
expected them to wind up in /home/evadeflow/project/poky-mirror.  This is
perhaps a bug in my expectations. I can certainly move or copy build/downloads
to wherever I want it and delete all the "*.done" files, but I can't imagine
I'm the only bitbake user who has to do this. It would be nice if there were
some way to get the computer to do it for us.

(In reply to comment #16)

Gak! Ignore, please. I've change my bugzilla prefs so it stops advancing
to the next bug, so this will (I hope!) not happen again. :-}