| Summary: | [mpc8315e-rdb] kernel crashes when booting with 1.3 M2 RC1 20120712 build | ||
|---|---|---|---|
| Product: | [Yocto Project Subprojects] Kernel | Reporter: | Yi Zhao <yi.zhao> |
| Component: | linux-yocto | Assignee: | Paul Gortmaker <paulg> |
| Status: | VERIFIED FIXED | QA Contact: | |
| Severity: | critical | ||
| Priority: | High | CC: | bruce.ashfield, dvhart, jiajun.xu, laurentiu.serban, liang.li2, paulg, Qi.Chen, sgw, yi.zhao |
| Version: | 1.3 | ||
| Target Milestone: | 1.3 M3 | ||
| Hardware: | mpc8315e-rdb | ||
| OS: | ppc | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
| Attachments: | |||
|
Description
Yi Zhao
2012-07-17 05:36:02 UTC
I'll look at this, since I did the uprev to 3.4 and it passed on my board. It may be that the test board's u-boot is too old... Created attachment 644 [details]
Log file of using u-boot to reflash over itself.
Be careful when using u-boot to reflash over itself. If you get things wrong, and corrupt flash, you will need an expensive JTAG tool to recover your board!
Note the "md" steps are optional; they just allow the user to see what is stored in ram/flash as a double check on things, since getting it right is critical.
I've copied your kernel and your rootfs image locally and they work fine (I'll attach a log) on my board. I even inserted a USB stick since I saw you had one in your log on your board. Sitll all works OK. I'm left thinking one of two things: 1) old u-boot causing some kind of issue, corrupting RAM and causing ftrace failure (the "Not a trampoline" message is from __ftrace_make_nop call). I am using a new u-boot, based on the recent u-boot changes in OE core. 2) Your hard disk is dying; since the board hangs when a udev thread times out on reading sda4. Odd that we don't see kernel driver error messages on What we can try: A) Getting you to boot an older rootfs/kernel, that was known to work for you. If that now fails, then it is probably #2 (try reading /dev/sda4 once it is booted). If it passes that test, then it is probably #1. B) Upgrading your u-boot. I'll put the prebuilt image I'm using somewhere for you and give you upgrade instructions (actually a log file from when I updated mine, showing all the steps.) Created attachment 645 [details]
Log of successful boot, login, verified kernel version.
Attached is the boot log; From /proc/version, you can see it is your kernel, and the md5sum of libc should be unique to your rootfs image as well. I got them from /export/pxeboot/vlm-boards/17652 (pek-tuxlab)
--------------
17652$md5sum rootfs/core-image-minimal-mpc8315e-rdb-20120712134431.rootfs.tar.bz2
159020e84d0b898804e9b0c5bf18aa73 rootfs/core-image-minimal-mpc8315e-rdb-20120712134431.rootfs.tar.bz2
17652$pwd
/export/pxeboot/vlm-boards/17652
17652$md5sum kernel
7c61a2536e15a50ff94e6af9999e2742 kernel
17652$md5sum dtb
b100dc9ca62d20161b41b8437fb1af4f dtb
17652$
-----------------
I've put the u-boot binary in /var/tmp on pek-tuxlab as well:
17652$md5sum /var/tmp/u-boot-mpc8315e-rdb.bin
ad2373c5a3de8a5dd14ef91db997bf6d /var/tmp/u-boot-mpc8315e-rdb.bin
17652$
(In reply to comment #3) > I've copied your kernel and your rootfs image locally and they work fine > (I'll attach a log) on my board. I even inserted a USB stick since I saw > you had one in your log on your board. Sitll all works OK. > > I'm left thinking one of two things: > > 1) old u-boot causing some kind of issue, corrupting RAM and causing ftrace > failure (the "Not a trampoline" message is from __ftrace_make_nop call). I > am using a new u-boot, based on the recent u-boot changes in OE core. > > 2) Your hard disk is dying; since the board hangs when a udev thread times > out > on reading sda4. Odd that we don't see kernel driver error messages on > > What we can try: > > A) Getting you to boot an older rootfs/kernel, that was known to work for > you. If that now fails, then it is probably #2 (try reading /dev/sda4 once > it is booted). If it passes that test, then it is probably #1. > I tried to boot an older rootfs/kernel with 1.3 M1 (kernel version: Linux mpc8315e-rdb 3.0.32-yocto-standard #1 PREEMPT Wed Jun 13 17:39:56 PDT 2012 ppc GNU/Linux). It didn't fail, but there were some error messages for sda4 when booting: ############ Starting udev udev[77]: starting version 164 talitos e0030000.crypto: hwrng talitos e0030000.crypto: authenc-hmac-sha1-cbc-aes-talitos talitos e0030000.crypto: authenc-hmac-sha1-cbc-3des-talitos talitos e0030000.crypto: authenc-hmac-sha256-cbc-aes-talitos talitos e0030000.crypto: authenc-hmac-sha256-cbc-3des-talitos talitos e0030000.crypto: authenc-hmac-md5-cbc-aes-talitos talitos e0030000.crypto: authenc-hmac-md5-cbc-3des-talitos talitos e0030000.crypto: cbc-aes-talitos talitos e0030000.crypto: cbc-3des-talitos talitos e0030000.crypto: md5-talitos talitos e0030000.crypto: sha1-talitos talitos e0030000.crypto: sha224-talitos talitos e0030000.crypto: sha256-talitos talitos e0030000.crypto: sha384-talitos talitos e0030000.crypto: sha512-talitos EXT3-fs (sda): error: can't find ext3 filesystem on dev sda. EXT2-fs (sda): error: can't find an ext2 filesystem on dev sda. FAT-fs (sda): invalid media value (0x00) FAT-fs (sda): Can't find a valid FAT filesystem EXT3-fs (sdb): error: can't find ext3 filesystem on dev sdb. EXT2-fs (sdb): error: can't find an ext2 filesystem on dev sdb. FAT-fs (sdb): bogus number of reserved sectors FAT-fs (sdb): Can't find a valid FAT filesystem VFS: Can't find a Minix filesystem V1 | V2 | V3 on device sda. FAT-fs (sda): invalid media value (0x00) FAT-fs (sda): Can't find a valid FAT filesystem VFS: Can't find a Minix filesystem V1 | V2 | V3 on device sdb. FAT-fs (sdb): bogus number of reserved sectors FAT-fs (sdb): Can't find a valid FAT filesystem attempt to access beyond end of device sda4: rw=0, want=4, limit=2 EXT3-fs (sda4): error: unable to read superblock attempt to access beyond end of device sda4: rw=0, want=4, limit=2 EXT2-fs (sda4): error: unable to read superblock FAT-fs (sda4): bogus number of reserved sectors FAT-fs (sda4): Can't find a valid FAT filesystem attempt to access beyond end of device sda4: rw=0, want=4, limit=2 MINIX-fs: unable to read superblock FAT-fs (sda4): bogus number of reserved sectors FAT-fs (sda4): Can't find a valid FAT filesystem EXT2-fs (sdb1): warning: mounting unchecked fs, running e2fsck is recommended EXT3-fs: barriers not enabled EXT3-fs (sda1): warning: maximal mount count reached, running e2fsck is recommended kjournald starting. Commit interval 5 seconds EXT3-fs (sda1): using internal journal EXT3-fs (sda1): mounted filesystem with ordered data mode EXT3-fs: barriers not enabled kjournald starting. Commit interval 5 seconds EXT3-fs (sda2): using internal journal EXT3-fs (sda2): mounted filesystem with ordered data mode EXT3-fs: barriers not enabled EXT3-fs (sda3): warning: mounting unchecked fs, running e2fsck is recommended kjournald starting. Commit interval 5 seconds EXT3-fs: barriers not enabled EXT3-fs: barriers not enabled kjournald starting. Commit interval 5 seconds EXT3-fs (sda5): using internal journal EXT3-fs (sda3): using internal journal EXT3-fs (sda3): mounted filesystem with ordered data mode EXT3-fs (sda5): mounted filesystem with ordered data mode EXT3-fs (sda6): warning: maximal mount count reached, running e2fsck is recommended kjournald starting. Commit interval 5 seconds EXT3-fs (sda6): using internal journal EXT3-fs (sda6): mounted filesystem with ordered data mode ############ The sda4 is a extended partition so it can't be mounted. ########### root@mpc8315e-rdb:~# fdisk -l /dev/sda Disk /dev/sda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders, total 312581808 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x09b10400 Device Boot Start End Blocks Id System /dev/sda1 63 58593750 29296844 83 Linux /dev/sda2 58595328 117186559 29295616 83 Linux /dev/sda3 117186560 175781887 29297664 83 Linux /dev/sda4 175781888 312576704 68397408+ 5 Extended /dev/sda5 175781951 238693769 31455909+ 83 Linux /dev/sda6 238695818 312576704 36940443+ 83 Linux ########## I will try it again after upgrading u-boot. > B) Upgrading your u-boot. I'll put the prebuilt image I'm using somewhere > for you and give you upgrade instructions (actually a log file from when I > updated mine, showing all the steps.) We'd like to close on this for M2 if possible. Paul's u-boot fix was merged this morning. Yi Zhao, does this resolve the issue for you? Marking as M2 as we still have one week left of "stabilization" according to the calendar. Given that Yi reports "there were some error messages for sda4 when booting" on the old version (pre-3.4 uprev), I'm reasonably confident he's got a hardware issue. Plus we have the fact that the exact same binary builds boot fine on my hardware. Given that, I'm adjusting the priority accordingly. Yi, the next/other trivial test you could do, is to boot something that gets to a shell; use fdisk to delete and recreate the offending partitions, and then mke2fs the remade partitions. Finally, in the same vein you could put smartmontools in a package list, build that, and get it so that you have "smartctl" on the target rootfs. This would allow you to run something like "smartctl -t long /dev/sda" as a disk sanity test. But at the moment, this is looking like a hardware fail. After updated the u-boot, I re-tested it with *sato-sdk* image (not minimal image). The system could boot and I ran the weekly test on it. All cases could pass. But I found there are 2 issues:
1. There is a module crashing message when booting:
#############
udev[79]: starting version 164
Not a trampoline
WARNING: at /srv/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-ppc/build/build/tmp/work/mpc8315e_rdb-poky-linux/linux-yocto-3.4.4+git1+a8cf77018b0faa0d29f1483ff4e5a2034dc8edd5_1+47d3076aa7e2a640a1dab8d663529a784709f697-r4.0/linux/kernel/trace/ftrace.c:1583
Modules linked in: rng_core(+)
NIP: c008c98c LR: c008d05c CTR: c0409378
REGS: c71c9d20 TRAP: 0700 Not tainted (3.4.4-yocto-standard)
MSR: 00029032 <EE,ME,IR,DR,RI> CR: 22228482 XER: 20000000
TASK = c7185100[133] 'modprobe' THREAD: c71c8000
GPR00: c008d05c c71c9dd0 c7185100 ffffffea c9035008 ffffffff c0394ea0 00000014
GPR08: 00000004 c07be3d5 00000001 0001ffff 22228424 10029140 00020000 1000d394
GPR16: bffd475c c9035008 c07d0000 00000000 c0010bb0 00000000 c078dcf4 c07fb6a8
GPR24: c07cc114 c7a4fca0 c9035b00 00000000 c72c4000 00000000 00000001 c9035008
NIP [c008c98c] ftrace_bug+0xb8/0x240
LR [c008d05c] ftrace_process_locs+0x410/0x5b0
Call Trace:
[c71c9dd0] [c0094e4c] ftrace_now+0x34/0x80 (unreliable)
[c71c9df0] [c008d05c] ftrace_process_locs+0x410/0x5b0
[c71c9e40] [c008ea38] ftrace_module_notify+0x6c/0x80
[c71c9e50] [c05dc8c4] notifier_call_chain+0x78/0xc0
[c71c9e70] [c0049df4] __blocking_notifier_call_chain+0x5c/0x80
[c71c9e90] [c006d6e0] sys_init_module+0x9b8/0x1ae8
[c71c9f40] [c0010298] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xf4a551c
LR = 0xff99fe0
Instruction dump:
7fe4fb78 38630ee8 7fe5fb78 80010024 bb410008 38210020 7c0803a6 4855441c
3d20c07c 3929e3d5 89490003 694a0001 <0f0a0000> 2f8a0000 409e015c 3d20c07d
---[ end trace d62b350335b2e5aa ]---
ftrace failed to modify [<c9035008>] rng_dev_open+0x8/0x50 [rng_core]
actual: 48:00:08:29
talitos e0030000.crypto: hwrng
###############
2. When I created a extended partition and added a logical partition on the harddisk, the udev would print some failed message.
#######
udevadm settle - timeout of 180 seconds reached, the event queue contains:
/sys/devices/e0000000.immr/e0018000.sata/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda4 (1082)
Starting Bootlog daemon: bootlogd.
udevd[79]: worker [88] unexpectedly returned with status 0x0100
udevd[79]: worker [88] failed while handling '/devices/e0000000.immr/e0018000.sata/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda4'
########
There is no such error when only primary partition on the harddisk.
This issue happened on our 2 mpc8315e-rdb boards (barcode 17649, 17652). So I don't think it's just a hardware fail.
I attached the detail boot log.
Created attachment 650 [details]
boot log without extended partition
Here is the log when there is no extended partition on harddisk
Created attachment 651 [details]
boot log with extended partition
Boot log when there is a extended partition on harddisk.
The original issue appears to be resolved. The two issues you describe above should be opened as new bug reports. I'm not sure either warrants a "critical" severity. I suggest marking this bug as resolved+fixed as the uboot upgrade from Paul has resolved the original reported boot issue. Paul, would you agree? 1) The udev issue definitely does look to be independent of the kernel uprev, and it also seems arch independent -- Fedora (x86) is tracking the same bug here: https://bugzilla.redhat.com/show_bug.cgi?id=670964 Based on the two logfiles recently attached, it is almost as if udev is doing something silly like trying to read the short extended partition... Not looking to be a kernel issue, anyway. 2) I've managed to solve (I think) the ftrace problem. It was introduced by the 3.4.3 stable release taking on a cherry pick of 3c75296562f43e ("powerpc: Fix kernel panic during kernel module load") which in turn also requires: ------- commit fd5a42980e1cf327b7240adf5e7b51ea41c23437 Author: roger blofeld <blofeldus@yahoo.com> powerpc/ftrace: Fix assembly trampoline register usage Just like the module loader, ftrace needs to be updated to use r12 instead of r11 with newer gcc's. ------- I can't reproduce the bug locally, but the above seems pretty cut and dry. I will make a request to have the above put in the next 3.4.x stable release. Darren, given the above, I'd say making a new udev bug might make sense, and if you wanted, you could leave this open until the ftrace fix loops around via stable. Or whatever you think, I have no special preference. Yi, can you please open a new bug to track the udev issue? Will leave this bug open to track the ftrace fix Paul referenced, and close once merged (it isn't in linux-yocto-3.4 as of just now). As Paul has been working this and has committed to getting the ftrace patch merged, marking as accepted. (In reply to comment #15) > Yi, can you please open a new bug to track the udev issue? > I have opened a new bug 2855 to track it. > Will leave this bug open to track the ftrace fix Paul referenced, and close > once merged (it isn't in linux-yocto-3.4 as of just now). This was picked up by Greg-kh, so we'll pull it in via the next 3.4.x stable update: http://git.kernel.org/?p=linux/kernel/git/stable/stable-queue.git;a=blob;f=queue-3.4/powerpc-ftrace-fix-assembly-trampoline-register-usage.patch;h=7f10db41986ff32243c8e4ee91349fc188158542;hb=0215b418053819f902759726c7c05c30ab93fc34 OK, there is really nothing more to do here then. We could leave it open and wait for the stable release - but then we'd have to discuss this bug repeatedly as it's targeted for M3. I don't think the ftrace issue merits a new bug. If someone brings it up, I think we can address it via email easily enough. Closing this out as the original issue is resolved and the additional two are adequately accounted for. The kernel has been updated to 3.4.7 in master branch, but the ftrace module crash issue still happens. So I reopen it.
gitinfo: poky/master: 19ff55a53b436d463f9577adaed15be77708f0d2
###################
root@mpc8315e-rdb:~# uname -a
Linux mpc8315e-rdb 3.4.7-yocto-standard #1 PREEMPT Wed Aug 22 09:50:27 PDT 2012 ppc GNU/Linux
#################
The crash log:
###############
Starting udev
udev[83]: starting version 164
Not a trampoline
------------[ cut here ]------------
WARNING: at /srv/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-ppc/build/build/tmp/work/mpc8315e_rdb-poky-linux/linux-yocto-3.4.7+git6+28bcd46af1d592dab39bd8a0891c872454fde8bc_3+f10500e9faad3f9f799137a3159ff5c836c88227-r4.0/linux/kernel/trace/ftrace.c:1583
Modules linked in: rng_core(+)
NIP: c00957c4 LR: c0095e94 CTR: c041240c
REGS: c6c99d20 TRAP: 0700 Not tainted (3.4.7-yocto-standard)
MSR: 00029032 <EE,ME,IR,DR,RI> CR: 22228482 XER: 20000000
TASK = c6d13740[142] 'modprobe' THREAD: c6c98000
GPR00: c0095e94 c6c99dd0 c6d13740 ffffffea c9035008 ffffffff c039df64 00000014
GPR08: 00000004 c07ca496 00000001 0001ffff 22228422 10029140 00020000 bfd3899c
GPR16: 00000000 c9035008 c07e0000 c0010bd0 00000000 00000000 c0799dcc c084a118
GPR24: c07d8114 c7a5fb50 c9035b00 00000000 c6d0d000 00000000 00000001 c9035008
NIP [c00957c4] ftrace_bug+0xb8/0x240
LR [c0095e94] ftrace_process_locs+0x410/0x5b0
Call Trace:
[c6c99dd0] [c009dcb8] ftrace_now+0x34/0x80 (unreliable)
[c6c99df0] [c0095e94] ftrace_process_locs+0x410/0x5b0
[c6c99e40] [c0097870] ftrace_module_notify+0x6c/0x80
[c6c99e50] [c05e59d4] notifier_call_chain+0x78/0xc0
[c6c99e70] [c004a280] __blocking_notifier_call_chain+0x5c/0x80
[c6c99e90] [c006dcd0] sys_init_module+0x9b8/0x1ae8
[c6c99f40] [c00102b8] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff05ac0
LR = 0xfdfa020
Instruction dump:
7fe4fb78 3863aa20 7fe5fb78 80010024 bb410008 38210020 7c0803a6 48554700
3d20c07d 3929a496 89490003 694a0001 <0f0a0000> 2f8a0000 409e015c 3d20c07e
---[ end trace 26c47aef7f811a84 ]---
ftrace failed to modify [<c9035008>] rng_dev_open+0x8/0x50 [rng_core]
actual: 48:00:08:29
############
You've jumped the gun on testing this. It wasn't merged until 3.4.8 and I see your kernel that you've used for testing is 3.4.7 - so you don't have the patch. ~/git/stable/stable-queue$ git grep -l 'Not a trampoline' . releases/3.0.40/powerpc-ftrace-fix-assembly-trampoline-register-usage.patch releases/3.4.8/powerpc-ftrace-fix-assembly-trampoline-register-usage.patch releases/3.5.1/powerpc-ftrace-fix-assembly-trampoline-register-usage.patch ~/git/stable/stable-queue$ Yi, The master branch now uses the 3.4.10 kernel for the routerstationpro. Can you please retest and verify the fix? There is no this issue on 1.3_M4 20120910 build. Branch: poky/1.3_M4 Gitinfo: dd3edc9fbb674682947e8218202fdbeef8635dd5 Image from http://autobuilder.yoctoproject.org/pub/nightly/20120910-5/machines/mpc8315e-rdb/ ################ root@mpc8315e-rdb:~# dmesg |grep ftrace ftrace: allocating 19529 entries in 58 pages root@mpc8315e-rdb:~# root@mpc8315e-rdb:~# uname -a Linux mpc8315e-rdb 3.4.9-yocto-standard #1 PREEMPT Mon Sep 10 18:12:55 PDT 2012 ppc GNU/Linux root@mpc8315e-rdb:~# ################ Verified. |