Bug 1006

Created attachment 141 [details]
mpc8315erdb boot log

boot log of mpc8315erdb board of night build 20110423-1

Created attachment 142 [details]
routerstationpro boot fail

The same issue also occurs on routerstationpro sato-sdk and minimal image, for more info please see attached file "routerstationpro_boot_fail.log".

Image location:
http://autobuilder.yoctoproject.org/nightly/20110423-1/machines/routerstationpro/mips/

Tried the kernel and dtb from 20110412-1 (last weekly test) and the newly minimal, sato ,sato-sdk images from 20110423-1, the same error also happend.

There's been no changes to the kernel, so this is likely something that has changed
in userspace .. and hence we aren't in a good position to debug.

If the most recent kernel is used with the rootfs from 20110412-1, it boots,
correct ?

I'm not seeing this as something that can be handled in the kernel, but instead
someone who's been changing userspace / dhcp should have a look. But I'll
wait for confirmation about the good/bad booting before making a suggestion.

The kernel of 20110412-1 and 20110423-1  are the same version:
2.6.37.6
the 20110412-1 build version boot normally.

On routerstationpro:
Scenario 1: The recent kernel(20110423-1) and rootfs(20110412-1)
The target can bootup, but it is kernel panic after a while.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yocto (Built by Poky 5.0) 1.0+snapshot-20110413 routerstationpro ttyS0

routerstationpro login: root
login[277]: root login  on `ttyS0'

root@routerstationpro:~# CPU 0 Unable to handle kernel paging request at virtual address 0000020c, epc == 80325718, ra == 802fd91c
Oops[#1]:
Cpu 0
$ 0   : 00000000 804e0000 87e6cd00 00000000
$ 4   : 87ebe0c0 0000000b 00000001 00000000
$ 8   : 00000000 00000000 0000000f 00000005
$12   : ffffffff 803cbfb0 00000001 803cbfa8
$16   : 87ebe0c0 87ebe0c0 87eaa06e 0000000b
$20   : 8042edec 00000001 00000000 8050a00c
$24   : 00000000 80069140                  
$28   : 8048e000 8048fbf8 80509e0c 802fd91c
Hi    : 0015f51c
Lo    : f5a06768
epc   : 80325718 icmp_send+0x58/0x514
    Not tainted
ra    : 802fd91c ip_expire+0x120/0x184
Status: 1000fc03    KERNEL EXL IE 
Cause : 00800008
BadVA : 0000020c
PrId  : 00019374 (MIPS 24Kc)
Modules linked in:
Process swapper (pid: 0, threadinfo=8048e000, task=80490620, tls=00000000)
Stack : 00000025 8048fd3c 8050904c 801e0f4c 8044fc5f 00000004 87e6cd00 00000002
        00000002 0000000a ffffffff ffffffff 00020aff ffffffff 00000001 87ee5158
        87ee5158 87ee5188 80492f90 8007a6a0 8048fd2c 8044fc2c 87ee5158 80490000
        80492f90 00000000 00000000 00000001 00000000 80079870 804f0000 8007f6b8
        80508c4c 00000004 00000001 8044fd24 80508c69 87ee5158 87ee5158 87c23bf0
        ...
Call Trace:
[<80325718>] icmp_send+0x58/0x514
[<802fd91c>] ip_expire+0x120/0x184
[<800892e0>] run_timer_softirq+0x178/0x220
[<80083c08>] __do_softirq+0xb8/0x164
[<80083cfc>] do_softirq+0x48/0x68
[<80061e0c>] ret_from_irq+0x0/0x4
[<80062000>] r4k_wait+0x20/0x40
[<80063ac4>] cpu_idle+0x24/0x44
[<804b1898>] start_kernel+0x318/0x334


Code: afa20018  8c43000c  8c920094 <8c71020c> 8c8300a4  0243182b  1460011f  26430014  8c85009c 
Disabling lock debugging due to kernel taint
Kernel panic - not syncing: Fatal exception in interrupt
eth0: raw intr=00000010 RXPR 
eth0: enable polling mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Scenario 2: the kernel(20110412-1) and the recent rootfs(20110423-1)
The target can *not* be bootup and have the same situation.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

ADDRCONF(NETDEV_UP): eth0: link is not ready
scsi 0:0:0:0: Direct-Access     Generic  STORAGE DEVICE   9451 PQ: 0 ANSI: 0
sd 0:0:0:0: [sda] Attached SCSI removable disk
IP-Config: Complete:
     device=eth0, addr=128.224.178.127, mask=255.255.254.0, gw=128.224.178.1,
     host=128.224.178.127, domain=, nis-domain=(none),
     bootserver=128.224.178.20, rootserver=128.224.178.20, rootpath=
eth0: link up (100Mbps/Full duplex)
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
VFS: Mounted root (nfs filesystem) on device 0:10.
Freeing unused kernel memory: 188k freed
Algorithmics/MIPS FPU Emulator v1.5
INIT: version 2.88 booting
modprobe: FATAL: Could not load /lib/modules/2.6.37.6-yocto-standard-00349-g8db6998/modules.dep: No such file or directory

modprobe: FATAL: Could not load /lib/modules/2.6.37.6-yocto-standard-00349-g8db6998/modules.dep: No such file or directory

Error opening /dev/fb0: No such device or address
Starting udev
Configuring network interfaces... udhcpc (v1.17.3) started
Setting IP address 0.0.0.0 on eth0
Sending discover...
Sending select for 128.224.178.251...
Lease of 128.224.178.251 obtained, lease time 600
nfs: server 128.224.178.20 not responding, still trying
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is very strange, we are saying that these boards fail to boot now, but there
have been zero (literally zero) changes to the kernel itself. This implies that
something in the toolchain (+ libs) or userspace has changed that is triggering this.

We need some non-kernel expertise to help here as well.

While we look into this in more detail, can we get two other final
confirmations:

  - we built and used these extensively in the 1.0 release timeframe, so these boards 
     were stable at this point. Are the 1.0 release images booting stabily on these boards ?

  - do disk or flash based booting show the same behaviour ?

No such error in booting 1.0 release images on mpc8315e-rdb board, the board boot normally.

I tested mpc8315e-rdb board. As Bruce said, it's not the kernel issue, but some userspace change runs dhcp to get a new IP, so nfsroot to be losted. I think userspace people should know why this change.

Because of adding feature to netbase in order to automatically bring up eth0, system ran dhcp to get a new ip address and lost connection to nfsroot.
Disable automatically bring up eth0 and system can boot successfully.

--- a/etc/network/interfaces	2011-04-24 16:42:32.000000000 +0800
+++ b/etc/network/interfaces	2011-04-13 23:36:03.000000000 +0800
@@ -14,7 +14,6 @@
 iface atml0 inet dhcp
 
 # Wired or wireless interfaces
-auto eth0
 iface eth0 inet dhcp
 iface eth1 inet dhcp

It can solve this problem. But I think it needs further research.

Aha! Thanks for the update, right in the area that we figured it
was.

Sounds like we need a board specific override to the interfaces file.

Confirmed here on routerstationpro. Unfortunately it seems that the mechanism [1] that's supposed to stop the netbase scripts from bringing up the interface aren't working. This is strange because the mechanism worked for me on atom-pc.

[1] http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=601464b4974e46d15abd02efd46663efcd530da8

OK, so the problem with the above mentioned mechanism is it relies on /sbin/ip existing, and that file (or symlink) is not present - ip is in /bin. I'm starting to wonder how this could have worked before.

I've sent a proposed patch here:

https://lists.yoctoproject.org/pipermail/poky/2011-April/005745.html

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ea0e29150edb03d785dfca63f220b00d10c6e22e

No such error in booting mpc8315e-rdb board using minimal,sato,sato-sdk images from nightly build 20110506-4, the board can boot success.
git info :
856a5e496b7ddd814f6a94552d02074f0ab30842

(In reply to comment #16)
> No such error in booting mpc8315e-rdb board using minimal,sato,sato-sdk images
> from nightly build 20110506-4, the board can boot success.
> git info :
> 856a5e496b7ddd814f6a94552d02074f0ab30842

the same error happended on the lsb-sdk images from nightly build 20110527-1:
http://autobuilder.pokylinux.org/nightly/20110527-1/machines/mpc8315e-rdb/ppc/
git commit : 4ff7af11ef69849ef9c16f585eae58ac920b222b
branch:  poky/1.1_M1

If this is failing then I don't think it's because of the mechanism that caused the original bug report - looking at the 20110529210000 image from the autobuilder, /bin/ip is present and points to /sbin/ip.iproute2 which is executable so the test "if [ -x /bin/ip -o -x /sbin/ip ]" should be working. Without direct investigation it's hard to debug this further though, and unfortunately my mpc8315e-rdb has still not yet arrived. Are no other machines affected?

Marking NEEDINFO per Paul's last comment.

This issue also occurs on routerstationpro with lsb-sdk(20110529214534) image.

Scenario I:  Comment out the 'auto eth0' in /etc/network/interfaces. 
The routerstationpro can bootup, but it will kernel panic after a while. The same phenomenon as before. 

Scenario II: No modify to /etc/network/interfaces, and add an link 'link -s /sbin/ip.iproute2 /sbin/ip'.
The routerstationpro can bootup and no kernel panic.

More Info:
###################
In core-image-sato-sdk-routerstationpro-20110529141824 image:
1. /bin/ip links to busybox
$ l sdk/bin/ip 
lrwxrwxrwx 1 root root 7 2011-06-03 09:08 sdk/bin/ip -> busybox*

2. There isn't have ip under /sbin/
$ l sdk/sbin/ip*
ls: cannot access sdk/sbin/ip*: No such file or directory


In lsb-sdk-routerstationpro-20110529214534 image:
1. /bin/ip links to /sbin/ip.iproute2.
$ l lsb/bin/ip 
lrwxrwxrwx 1 root root 17 2011-06-03 09:10 lsb/bin/ip -> /sbin/ip.iproute2

2. There isn't have ip under /sbin/.
$ l lsb/sbin/ip*
-rwxr-xr-x 1 root root 242K 2011-05-29 16:34 lsb/sbin/ip.iproute2*
lrwxrwxrwx 1 root root   17 2011-06-03 09:09 lsb/sbin/ipmaddr -> ipmaddr.net-tools*
-rwxr-xr-x 1 root root  17K 2011-05-29 16:31 lsb/sbin/ipmaddr.net-tools*
lrwxrwxrwx 1 root root   18 2011-06-03 09:09 lsb/sbin/iptunnel -> iptunnel.net-tools*
-rwxr-xr-x 1 root root  22K 2011-05-29 16:31 lsb/sbin/iptunnel.net-tools*

(In reply to comment #20)
> This issue also occurs on routerstationpro with lsb-sdk(20110529214534) image.
> 
> Scenario I:  Comment out the 'auto eth0' in /etc/network/interfaces. 
> The routerstationpro can bootup, but it will kernel panic after a while. The
> same phenomenon as before. 
> 
> Scenario II: No modify to /etc/network/interfaces, and add an link 'link -s
> /sbin/ip.iproute2 /sbin/ip'.
For the Scenario II, it is also need to comment out the 'auto eth0' in /etc/network/interfaces, otherwise the target can not bootup. Sorry for this confusion.
> The routerstationpro can bootup and no kernel panic.
>

Just to be clear. I'm not actively working on this, since it isn't in
the kernel. 

Who's looking into the userspace side of things here ? We need to
get their name on the bug.

Sorry, I have been looking into this and I should have directly committed to fixing it since it was my change that seems to have caused it in the first place. Assigning to myself.

I'm still a bit confused as to how this could be happening. Now that I know it still occurs on routerstationpro at least I have that board to try to reproduce it with. Will report back when I know more.

As I suspected this is now a different issue than the original bug - with iproute2 installed in preference to ip from busybox, "ip route get" used by the /etc/network/if-pre-up.d/nfsroot script produces no output. This may be due to [1] and I am testing an upgrade to iproute2 2.6.37 to see if it makes a difference.

[1] http://comments.gmane.org/gmane.linux.network/168538

Confirmed, the update to iproute2 2.6.37 fixes the problem. I have submitted a patch:

http://lists.linuxtogo.org/pipermail/openembedded-core/2011-June/003869.html

After a request to update to the latest version (2.6.38) I have posted a new patch (tested, ip route get works):
http://lists.linuxtogo.org/pipermail/openembedded-core/2011-June/003916.html

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=cd8694d7363f3be2b2c422dad8d2b7deed9774fd

no such error in booting mpc8315e-rdb board using minimal,sato,sato-sdk images
from nightly build 20110617-4, the board can boot success:
http://autobuilder.pokylinux.org/nightly/20110617-4/machines/mpc8315e-rdb/ppc/
git info:
2163461ec94528ecf046a04edc5db3d2dd3a6b8b
git branch: master

verified on nightly build 20110617-4:
http://autobuilder.pokylinux.org/nightly/20110617-4/machines/routerstationpro/mips/
http://autobuilder.pokylinux.org/nightly/20110617-4/machines/mpc8315e-rdb/ppc/

git info:
2163461ec94528ecf046a04edc5db3d2dd3a6b8b
git branch: master