Bug 4129

Summary: 3.8 Kernel crashes on qemumips
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Khem Raj <raj.khem>
Component: kernelAssignee: Bruce Ashfield <bruce.ashfield>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: alexandrux.palalau, bruce.ashfield, tom.zanussi, yi.zhao
Version: 1.4   
Target Milestone: 1.4 M6   
Hardware: Other   
OS: mips   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Khem Raj 2013-03-27 01:31:06 UTC
I am trying to boot a systemd based image on qemumips and below is the result

Reserved instruction in kernel code[#1]:
Cpu 0
$ 0   : 00000000 80233cc0 c0011000 80000000
$ 4   : c0017400 00000000 87fc9880 87ff8000
$ 8   : 00000000 00000000 00000010 00000001
$12   : 00000000 7ff5de74 00000000 00000000
$16   : 87f3fcc0 c0017400 87ff8000 00000000
$20   : 87ff7000 00000000 87ff7000 00000000
$24   : 00000010 80483580
$28   : 87c22000 87c23e28 004f9340 80233bc8
Hi    : 00000000
Lo    : 00000000
epc   : c0011000 autofs_mount+0x0/0x30 [autofs4]
    Not tainted
ra    : 80233bc8 mount_fs+0x68/0x200
Status: 1000a403    KERNEL EXL IE
Cause : 90800028
PrId  : 00019300 (MIPS 24Kc)
Modules linked in: autofs4
Process systemd (pid: 1, threadinfo=87c22000, task=87c28000, tls=7721e490)
Stack : 809b0000 80253130 0000000b 9484a621 87f5c01d 80159e88 87fc9880 87ff8000
        87f3fcc0 c0017400 87ff8000 87fc9880 87fc9900 8088896c 00000000 802532fc
        00000006 80230ef4 87fc9900 802521a0 00000000 87fc9880 87ff8000 00000000
        c0017400 00000020 87ff8000 80255b68 7ff5dd90 00000019 00000000 00490000
        87ff8000 801e057c 00000000 0000000a 87f5c000 801f670c 87f3f010 87934dd0
        ...
Call Trace:
[<c0011000>] autofs_mount+0x0/0x30 [autofs4]
[<80233bc8>] mount_fs+0x68/0x200
[<802532fc>] vfs_kern_mount+0x6c/0x118
[<80255b68>] do_mount+0x210/0x9d8
[<802563d4>] sys_mount+0xa4/0xec
[<8010c07c>] stack_done+0x20/0x44


Code: (Bad address in epc)

---[ end trace 98c875a5c9825ffa ]---
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
Comment 1 Bruce Ashfield 2013-03-27 01:55:40 UTC
Hmmm. I built and booted this on core-image-minaal and sato several times
today alone.

Can you paste the hashes of your build layers ?
Comment 2 Bruce Ashfield 2013-03-27 02:01:09 UTC
Poky 8.0 (Yocto Project 1.3 Reference Distro) 1.3+snapshot-20130326 qemumips ttyS0

qemumips login: root
root@qemumips:~# uname -a
Linux qemumips 3.8.4-yocto-standard #1 PREEMPT Tue Mar 26 10:26:36 EDT 2013 mips GNU/Linux
root@qemumips:~#
Comment 3 Khem Raj 2013-03-27 02:09:37 UTC
latest OE-Core master from today. But as I said I have systemd in my DISTRO_FEATURES
Comment 4 Bruce Ashfield 2013-03-27 02:11:07 UTC
I missed the systemd comment. Sorry about that. If we can confirm that
this boots without systemd enabled, we can get someone like Ross to
look into this more, since I'm clueless at the moment about systemd.
Comment 5 Yi Zhao 2013-03-27 09:02:06 UTC
Sometimes the kernel crashes on a init based image. Please see bug 4052
Comment 6 Bruce Ashfield 2013-03-27 12:38:28 UTC
Aha. Thanks .. then this is likely a qemu issue (as Richard mentioned in the Tuesday call)
versus kernel or systemd issue. 

I've never seen this, so can't debug it from here. The traces don't have enough information.
Comment 7 Bruce Ashfield 2013-03-27 13:42:49 UTC
khem: out of curiosity .. has this just started happening to you ?

I just pushed a different (better) fix for signal handling on arch/mips that may have
been impacting qemumips after I did the 3.8.4-stable update.

If you check linux-yocto-3.8, and bump your SRCREV to:

http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-3.8/commit/?h=standard/mti-malta32&id=c7297b41016919c95a2c779cafc0aad736f6a400

for qemumips, I'd be interested to hear if this helps, before I send the pull request
to oe-core.
Comment 8 Khem Raj 2013-03-29 14:29:34 UTC
(In reply to comment #7)
> khem: out of curiosity .. has this just started happening to you ?

3.8 never booted with systemd/mips

> 
> I just pushed a different (better) fix for signal handling on arch/mips that
> may have
> been impacting qemumips after I did the 3.8.4-stable update.
> 
> If you check linux-yocto-3.8, and bump your SRCREV to:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-3.8/commit/?h=standard/
> mti-malta32&id=c7297b41016919c95a2c779cafc0aad736f6a400
> 

still fails.


> for qemumips, I'd be interested to hear if this helps, before I send the
> pull request
> to oe-core.

The core-image-minimal with sysvinit boots ok. but with systemd it dies with below message.
Comment 9 Bruce Ashfield 2013-03-30 03:46:38 UTC
This is good (for me) that it wasn't my resolution of the signal handling
breakage that caused the boot problem. But not good overall.

Khem: what's the right/fastest way to enable systemd, it's a distro feature,
right ? I can always try a build here and see.

While the kernel should never trap based on something happening from 
userspace, this looks like something happening specific to the qemu emulation
and an instruction that is being generated .. so debugging it from someone
that knows systemd and it's code base may be easier.
Comment 10 Bruce Ashfield 2013-04-03 19:02:39 UTC
I have a fix for this. There's a module relocation that is impacting qemu mips. We
can fix this via a config tweak.

I've now done many systemd boots. no issues.
Comment 11 Bruce Ashfield 2013-04-10 17:28:09 UTC
This was merged to oe-core.
Comment 12 Bruce Ashfield 2013-04-10 17:28:32 UTC
commit 80ab72bcef4083cd50c348efc1fcae9fe7b9f4c0
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date:   Wed Apr 10 02:58:56 2013 +0000

    linux-yocto/3.8: aufs, config processing, tiny, mips boot fixes
    
    Updating the SRCREVs to fix a number of bugs, boot issues and ktype support
    additions.
Comment 13 Alexandru Palalau 2013-05-01 12:50:43 UTC
Verified on master branch.
Commit ID: c42cc8fb38aee46f4778b9c68412a47ae5919923