The current crownbay kernel build is failing because the git2_ tarball that gets fetched from the autobuilder for the kernel, http://autobuilder.yoctoproject.org/sources/git2_git.pokylinux.org.linux-yocto-2.6.37.tar.gz, is missing a local crownbay branch. I can make the tarball itself available if needed for debugging. This is also the same problem seen a few weeks ago with both the crownbay and jasperforest BSPs - the problem was fixed then by (I guess) removing the old tarballs from the autobuilder. Here's a listing of the branches in the tarball. Looking at it, the remote tracking branches look ok, so it seems to be looking at an up-to-date kernel repo (e.g. eg20t, gma500 and emgd branches were just added very recently), but the local branches are different. trz@elmorro:~/Downloads$ git branch -a * master meta yocto/base yocto/standard/arm-versatile-926ejs yocto/standard/base yocto/standard/beagleboard yocto/standard/common-pc-64/base yocto/standard/common-pc-64/sugarbay yocto/standard/common-pc/atom-pc yocto/standard/common-pc/base yocto/standard/emenlow yocto/standard/fsl-mpc8315e-rdb yocto/standard/mti-malta32-be yocto/standard/mti-malta32-le yocto/standard/preempt-rt/base yocto/standard/preempt-rt/common-pc yocto/standard/preempt-rt/common-pc-64 yocto/standard/qemu-ppc32 yocto/standard/routerstationpro remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/eg20t remotes/origin/yocto/emgd remotes/origin/yocto/gma500 remotes/origin/yocto/standard/arm-versatile-926ejs remotes/origin/yocto/standard/base remotes/origin/yocto/standard/beagleboard remotes/origin/yocto/standard/common-pc-64/base remotes/origin/yocto/standard/common-pc-64/jasperforest remotes/origin/yocto/standard/common-pc-64/sugarbay remotes/origin/yocto/standard/common-pc/atom-pc remotes/origin/yocto/standard/common-pc/base remotes/origin/yocto/standard/crownbay remotes/origin/yocto/standard/emenlow remotes/origin/yocto/standard/fsl-mpc8315e-rdb remotes/origin/yocto/standard/mti-malta32-be remotes/origin/yocto/standard/mti-malta32-le remotes/origin/yocto/standard/preempt-rt/base remotes/origin/yocto/standard/preempt-rt/common-pc remotes/origin/yocto/standard/preempt-rt/common-pc-64 remotes/origin/yocto/standard/qemu-ppc32 remotes/origin/yocto/standard/routerstationpro More info from IRC: (12:03:47 PM) tomz: pidge: I'm seeing problems again with the kernel tarball being served up by the autobuilder - http://autobuilder.yo (12:03:47 PM) tomz: ctoproject.org/sources/git2_git.pokylinux.org.linux-yocto-2.6.37.tar.gz doesn't have the yocto/standard/crownbay branch. (12:04:16 PM) tomz: pidge: the kernel repo looks fine though (12:05:09 PM) tomz: pidge: when we were seeing similar things a few weeks ago, you did something on the autobuilder to fix it - what did you have to do? (12:14:49 PM) pidge: tomz: which buildset? It's an easy fix, I'll just need to hop on and rm -rf that file (12:15:13 PM) tomz: pidge: buildset? (12:16:08 PM) tomz: pidge: also wondering how the tarball gets missing branches ((12:16:10 PM) pidge: is it just on the one autobuilder, is my question. no worries, I'll find out... give me a few moments. (12:16:53 PM) JaMa|Off left the room (quit: Quit: reboot). (12:16:57 PM) pidge: tomz: that's a really good question. it shouldn't unless something is going wrong with: the fetcher, my copying of SOURCEDIR. (12:18:45 PM) pidge: tomz: if you can hold off for a few moments, I want to figure out why this is happening. (12:18:59 PM) tomz: pidge: sure (12:19:57 PM) pidge: hrm.... so the file I have in SOURCEDIR is the same as in /sources. Which means: (12:20:24 PM) pidge: a. when I kick a build it's not updating the tarball (12:21:09 PM) pidge: erm... I can't think of a b.... why would that be happening? (12:22:26 PM) pidge: ok, well, i rm'd the tarball. next build that uses it should repull it. however, I need to root cause this. (12:23:26 PM) tomz: pidge: dunno - I've never seen anything similar when I have the kenrel recipe pointed to a local clone of the repo, which is how I normally work. I guess this should be in bugzilla -should I file a bug for it? (12:24:15 PM) pidge: yes, do so. assign to me. (12:24:26 PM) tomz: pidge: ok, will do
Upgrading this bug since it's blocking investigation of further bugs. I'm trying to investigate Bug 994 (Fix host infection in compile_perf) but can't get to the perf build because the branch can't be found.
do_kernel_checkout() in kernel-yocto.bbclass goes through the results of 'git show-ref' and greps the lines matching 'refs/remotes/origin'. For the repo in git2 this shows: trz@elmorro:~/work/sandbox/build/downloads/git2/git.pokylinux.org.linux-yocto-2.6.37$ git show-ref | egrep refs/remotes/origin 3c0eee3fe6a3a1c745379547c7e7c904aa64f6d5 refs/remotes/origin/master 727fc4769aa920fc5bb3eb9a81bf92d0e6340903 refs/remotes/origin/meta a9d833fda90e2f1257888a97e092135610b5f259 refs/remotes/origin/yocto/base a9d833fda90e2f1257888a97e092135610b5f259 refs/remotes/origin/yocto/eg20t a9d833fda90e2f1257888a97e092135610b5f259 refs/remotes/origin/yocto/emgd a9d833fda90e2f1257888a97e092135610b5f259 refs/remotes/origin/yocto/gma500 e5ca41def834db9d18b28393a45d53a8d18f3d05 refs/remotes/origin/yocto/standard/arm-versatile-926ejs 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/base 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/beagleboard 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/common-pc-64/base 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/common-pc-64/jasperforest 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/common-pc-64/sugarbay 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/common-pc/atom-pc cb0537a40dadea20f12bc10e0986fd2a70290b42 refs/remotes/origin/yocto/standard/common-pc/base 86bd1278bbf11589d87639df17d2f5f42db97b60 refs/remotes/origin/yocto/standard/crownbay 2215a346eb4f9e09053d00bdf61ed999ff80e029 refs/remotes/origin/yocto/standard/emenlow 6861d8a4d58f8aa75997f7678cc454791544507a refs/remotes/origin/yocto/standard/fsl-mpc8315e-rdb 9bbc8e3432406429923fab0de038af5d3e647901 refs/remotes/origin/yocto/standard/mti-malta32-be ec4cce0e3376feb36c00dda41c10457c436600c3 refs/remotes/origin/yocto/standard/mti-malta32-le 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/preempt-rt/base 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/preempt-rt/common-pc 69cfbdf9f1ff461a75e5b77d6d7ba35e97db4cc3 refs/remotes/origin/yocto/standard/preempt-rt/common-pc-64 f0ff494e62bfaa921e844c1ec7fe6cf4a977980a refs/remotes/origin/yocto/standard/qemu-ppc32 8db69980d76e1f863af409e70175963f23de83ab refs/remotes/origin/yocto/standard/routerstationpro The repo in git2 is cloned to the work directory: poky/bitbake/lib/bb/fetch2/git.py:unpack() clones the kernel repo using: git clone -s -n The cloned repo shows this as the output of show-ref: trz@elmorro:/usr/local/test/intel-bsp/build/tmp/work/crownbay-poky-linux/linux-yocto-2.6.37+git1+51c761795b078bf3c80104ed6e73b3d995067384_1+86bd1278bbf11589d87639df17d2f5f42db97b60-r21/linux$ git show-ref | egrep refs/remotes/origin 3c0eee3fe6a3a1c745379547c7e7c904aa64f6d5 refs/remotes/origin/master 5a32d7fe3b817868ebb697d2d883d743903685ae refs/remotes/origin/meta a9d833fda90e2f1257888a97e092135610b5f259 refs/remotes/origin/yocto/base c6d1949b479118d027a6c0f0841247cc0ea697b2 refs/remotes/origin/yocto/standard/arm-versatile-926ejs 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/base 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/beagleboard 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/common-pc-64/base 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/common-pc-64/sugarbay 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/common-pc/atom-pc cb4faf3af527e25feee584f39c3956ca0c9448c8 refs/remotes/origin/yocto/standard/common-pc/base 30a8b903be9cb18ba1fc11242159fa5d7bfdb1cc refs/remotes/origin/yocto/standard/emenlow 68fb0b8d73bbb336ffa4a47bc70ec1eab021b7f6 refs/remotes/origin/yocto/standard/fsl-mpc8315e-rdb cab9f6666b9913136549cee2f5db07134f4d9780 refs/remotes/origin/yocto/standard/mti-malta32-be 44d803669e3adf7ec65cea05e43c8115bc3d3540 refs/remotes/origin/yocto/standard/mti-malta32-le 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/preempt-rt/base 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/preempt-rt/common-pc 343c06b25b221aa7f7019aee83b2d976041fddce refs/remotes/origin/yocto/standard/preempt-rt/common-pc-64 4ab4b22d4768f65cfadccae6d749e4f3769b0afa refs/remotes/origin/yocto/standard/qemu-ppc32 4f1adb92fd08001ec76185c7febea7504c84f082 refs/remotes/origin/yocto/standard/routerstationpro So at this point the branches we're interested in go missing and cause the problem. The problem is that a git clone won't clone a remote branch if there isn't a local branch, and for each of the missing remote branches that do_kernel_checkout() isn't finding, there isn't a corresponding local branch in the clone received from the autobuilder, and therefore the remote branch isn't found by do_kernel_checkout(). Here's an example of the branches gone missing after the clone: $ git branch -a * master meta yocto/base yocto/standard/arm-versatile-926ejs yocto/standard/base yocto/standard/beagleboard yocto/standard/common-pc-64/base yocto/standard/common-pc-64/sugarbay yocto/standard/common-pc/atom-pc yocto/standard/common-pc/base yocto/standard/emenlow yocto/standard/fsl-mpc8315e-rdb yocto/standard/mti-malta32-be yocto/standard/mti-malta32-le yocto/standard/preempt-rt/base yocto/standard/preempt-rt/common-pc yocto/standard/preempt-rt/common-pc-64 yocto/standard/qemu-ppc32 yocto/standard/routerstationpro remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/eg20t remotes/origin/yocto/emgd remotes/origin/yocto/gma500 remotes/origin/yocto/standard/arm-versatile-926ejs remotes/origin/yocto/standard/base remotes/origin/yocto/standard/beagleboard remotes/origin/yocto/standard/common-pc-64/base remotes/origin/yocto/standard/common-pc-64/jasperforest remotes/origin/yocto/standard/common-pc-64/sugarbay remotes/origin/yocto/standard/common-pc/atom-pc remotes/origin/yocto/standard/common-pc/base remotes/origin/yocto/standard/crownbay remotes/origin/yocto/standard/emenlow remotes/origin/yocto/standard/fsl-mpc8315e-rdb remotes/origin/yocto/standard/mti-malta32-be remotes/origin/yocto/standard/mti-malta32-le remotes/origin/yocto/standard/preempt-rt/base remotes/origin/yocto/standard/preempt-rt/common-pc remotes/origin/yocto/standard/preempt-rt/common-pc-64 remotes/origin/yocto/standard/qemu-ppc32 remotes/origin/yocto/standard/routerstationpro $ git clone -s -n git.pokylinux.org.linux-yocto-2.6.37 junk junk$ git branch -a * master remotes/origin/HEAD -> origin/master remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/standard/arm-versatile-926ejs remotes/origin/yocto/standard/base remotes/origin/yocto/standard/beagleboard remotes/origin/yocto/standard/common-pc-64/base remotes/origin/yocto/standard/common-pc-64/sugarbay remotes/origin/yocto/standard/common-pc/atom-pc remotes/origin/yocto/standard/common-pc/base remotes/origin/yocto/standard/emenlow remotes/origin/yocto/standard/fsl-mpc8315e-rdb remotes/origin/yocto/standard/mti-malta32-be remotes/origin/yocto/standard/mti-malta32-le remotes/origin/yocto/standard/preempt-rt/base remotes/origin/yocto/standard/preempt-rt/common-pc remotes/origin/yocto/standard/preempt-rt/common-pc-64 remotes/origin/yocto/standard/qemu-ppc32 remotes/origin/yocto/standard/routerstationpro So because it appears that the fetcher is actually getting the tarball correctly from the autobuilder, but the tarball is defective, missing the proper branches, we need to know how exactly that tarball is being generated by the autobuilder - for me the trail runs dry at the point just before that happens. Also, why some of the remote branches in the tarball have local branches and some don't...
ACK what Tom says. Reproducible test case is to build master, switch to bernard, and then build bernard.
> So because it appears that the fetcher is actually getting the tarball > correctly from the autobuilder, but the tarball is defective, missing the > proper branches, we need to know how exactly that tarball is being generated by > the autobuilder - for me the trail runs dry at the point just before that > happens. Also, why some of the remote branches in the tarball have local > branches and some don't... One note. This is not an autobuilder issue, the autobuilder is just exposing an existing reproducible issue. The fetcher is getting the tarball from the autobuilder fine, it's when the autobuilder does the get_kernel_checkout() and updates it's existing kernel git repo in it's DL_DIR where things are going wrong.
The clone of the git repo in unpack() is exposing the fact that the tarball being served up by the autobuilder is wrong. So can you please point us to the scripts on the autobuilder that create the tarball that is getting supplied to the user - that should point to the cause of the problem.
This should be resolved on the AB by setting MIRRORS = "" PREMIRRORS = "" in auto.conf
I'm reopening this as I shouldn't have closed it. It's just not an ab issue.
linux-yocto dowloaded during the build doesn't have the romley branch, but a local clone of the repo does. kishore@kishore-desktop:/usr/local/src/mga-yocto/build/tmp/work/romley-poky-linux/linux-yocto-2.6.37+git1+e2cabadc059ad8d91f7e4a023106620d93abd23a_1+2b412826bbeb4a16abe2ea74f2456ab880c6e3c1-r18/linux$ git branch -a * master meta yocto/base yocto/eg20t yocto/emgd yocto/gma500 yocto/standard/arm-versatile-926ejs yocto/standard/base yocto/standard/beagleboard yocto/standard/common-pc-64/base yocto/standard/common-pc-64/jasperforest yocto/standard/common-pc-64/sugarbay yocto/standard/common-pc/atom-pc yocto/standard/common-pc/base yocto/standard/crownbay yocto/standard/emenlow yocto/standard/fsl-mpc8315e-rdb yocto/standard/mti-malta32-be yocto/standard/mti-malta32-le yocto/standard/preempt-rt/base yocto/standard/preempt-rt/common-pc yocto/standard/preempt-rt/common-pc-64 yocto/standard/qemu-ppc32 yocto/standard/routerstationpro remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/eg20t remotes/origin/yocto/emgd remotes/origin/yocto/gma500 remotes/origin/yocto/standard/arm-versatile-926ejs remotes/origin/yocto/standard/base remotes/origin/yocto/standard/beagleboard remotes/origin/yocto/standard/common-pc-64/base remotes/origin/yocto/standard/common-pc-64/jasperforest remotes/origin/yocto/standard/common-pc-64/sugarbay remotes/origin/yocto/standard/common-pc/atom-pc remotes/origin/yocto/standard/common-pc/base remotes/origin/yocto/standard/crownbay remotes/origin/yocto/standard/emenlow remotes/origin/yocto/standard/fsl-mpc8315e-rdb remotes/origin/yocto/standard/mti-malta32-be remotes/origin/yocto/standard/mti-malta32-le remotes/origin/yocto/standard/preempt-rt/base remotes/origin/yocto/standard/preempt-rt/common-pc remotes/origin/yocto/standard/preempt-rt/common-pc-64 remotes/origin/yocto/standard/qemu-ppc32 remotes/origin/yocto/standard/routerstationpro
Following kishore's report, I did a from-scratch build of sugarbay and at the same time did a local clone of the kernel git repo. The local clone contained all branches as expected, but the one from the build was missing both the romley and fishriver branches for example. Also the meta branch from the local clone showed the latest commit, 8888d948e0ddbc5f37bd3dc23cc1b398ab230e2e, while the one from the build shows a last commit dating from April 4, ec411d4008dbfc03f609e32123703bcce9227410. git branch -a from the local clone: * master meta yocto/base yocto/eg20t yocto/emgd yocto/gma500 yocto/standard/arm-versatile-926ejs yocto/standard/base yocto/standard/beagleboard yocto/standard/common-pc-64/base yocto/standard/common-pc-64/jasperforest yocto/standard/common-pc-64/romley yocto/standard/common-pc-64/sugarbay yocto/standard/common-pc/atom-pc yocto/standard/common-pc/base yocto/standard/crownbay yocto/standard/emenlow yocto/standard/fishriver yocto/standard/fsl-mpc8315e-rdb yocto/standard/mti-malta32-be yocto/standard/mti-malta32-le yocto/standard/preempt-rt/base yocto/standard/preempt-rt/common-pc yocto/standard/preempt-rt/common-pc-64 yocto/standard/qemu-ppc32 yocto/standard/routerstationpro git branch -a from the build: master meta meta-orig yocto/base yocto/eg20t yocto/emgd yocto/gma500 yocto/standard/arm-versatile-926ejs yocto/standard/base yocto/standard/beagleboard yocto/standard/common-pc-64/base yocto/standard/common-pc-64/jasperforest * yocto/standard/common-pc-64/sugarbay yocto/standard/common-pc-64/sugarbay-orig yocto/standard/common-pc/atom-pc yocto/standard/common-pc/base yocto/standard/crownbay yocto/standard/emenlow yocto/standard/fsl-mpc8315e-rdb yocto/standard/mti-malta32-be yocto/standard/mti-malta32-le yocto/standard/preempt-rt/base yocto/standard/preempt-rt/common-pc yocto/standard/preempt-rt/common-pc-64 yocto/standard/qemu-ppc32 yocto/standard/routerstationpro remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/eg20t remotes/origin/yocto/emgd remotes/origin/yocto/gma500 remotes/origin/yocto/standard/arm-versatile-926ejs remotes/origin/yocto/standard/base remotes/origin/yocto/standard/beagleboard remotes/origin/yocto/standard/common-pc-64/base remotes/origin/yocto/standard/common-pc-64/jasperforest remotes/origin/yocto/standard/common-pc-64/sugarbay remotes/origin/yocto/standard/common-pc/atom-pc remotes/origin/yocto/standard/common-pc/base remotes/origin/yocto/standard/crownbay remotes/origin/yocto/standard/emenlow remotes/origin/yocto/standard/fsl-mpc8315e-rdb remotes/origin/yocto/standard/mti-malta32-be remotes/origin/yocto/standard/mti-malta32-le remotes/origin/yocto/standard/preempt-rt/base remotes/origin/yocto/standard/preempt-rt/common-pc remotes/origin/yocto/standard/preempt-rt/common-pc-64 remotes/origin/yocto/standard/qemu-ppc32 remotes/origin/yocto/standard/routerstationpro
*** Bug 1207 has been marked as a duplicate of this bug. ***
(In reply to comment #7) > This should be resolved on the AB by setting > > MIRRORS = "" > PREMIRRORS = "" > > in auto.conf I've had to undo this fix, as some upstream packages have gone missing. As the mirror is the only place they exist, I am now forced to use the mirror.
From my email on this https://lists.yoctoproject.org/pipermail/yocto/2011-July/001791.html This is how to reproduce this: --------------------------------------------------------------------------- First, I created a local linux-yocto repo and modified meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it: SRCREV_machine = ${AUTOREV} SRCREV_meta = ${AUTOREV} SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta" I do bitbake virtual/kernel -c kernel_checkout -f and the repo is fetched into my DL_DIR, unpacked into tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then do_kernel_checkout runs. At this point, everything is correct. At this point my local SRC_URI repo is: * master meta yocto/base yocto/eg20t yocto/emgd yocto/gma500 .... My clone in DL_DIR is the same. My clone in tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux looks like this: master meta yocto/base yocto/eg20t yocto/emgd yocto/gma500 .... remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/eg20t remotes/origin/yocto/emgd remotes/origin/yocto/gma500 With the local head branches and the remotes being the same. Now, on to where this gets ugly. I commit a new branch to my SRC_URI repo: My SRC_URI: BAD_BRANCH master meta yocto/base yocto/eg20t yocto/emgd yocto/gma500 .... I re-run fetch and my DL_DIR looks like this: * master meta yocto/base yocto/eg20t yocto/emgd yocto/gma500 .... remotes/origin/BAD_BRANCH remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/eg20t remotes/origin/yocto/emgd remotes/origin/yocto/gma500 Notice, the new BAD_BRANCH is now only in remotes. This *should* be ok, since after we do_unpack we run do_kernel_checkout which should copy the refs in remotes to heads. However.... I clean out my workdir and run unpack. A git branch -a of the git dir it unpacks to shows that BAD_BRANCH does not exist: ~/poky/build/tmp/work/qemux86-poky-linux/linux-yocto-2.6.37+git3+94bca94ff11276b4cfffc1818c1defa3051854b2_1+ea7bcd9e408ddfaf5ecd0bcc37956998add58e2d-r20/git> git branch -a * master remotes/origin/HEAD -> origin/master remotes/origin/master remotes/origin/meta remotes/origin/yocto/base remotes/origin/yocto/eg20t remotes/origin/yocto/emgd remotes/origin/yocto/gma500 So, do_kernel_checkout never even gets the chance to fix up the git repo because the git clone never clones the remotes.
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=acc75635b8bab53c07c89c326f590eb387e3db67