Bug 10328

Summary: Kernel oops with linux-yocto-rt 4.8
Product: [Yocto Project Subprojects] Kernel Reporter: California Sullivan <california.l.sullivan>
Component: linux-yoctoAssignee: Paul Gortmaker <paulg>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: richard.purdie, sgw
Version: 2.2   
Target Milestone: 2.2 M4   
Hardware: x86   
OS: x86_64   
Whiteboard: YP 2.2 M4 RC1 BUILD BLOCKING BUG
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
dmesg log
none
local.conf
none
bblayers.conf none

Description California Sullivan 2016-09-23 22:51:00 UTC 
The following oops' occur with the new -rt kernel (branch standard/preempt-rt/intel) with an intel-corei7-64 MACHINE:


[    6.845236] ------------[ cut here ]------------
[    6.845253] ------------[ cut here ]------------
[    6.845277] WARNING: CPU: 1 PID: 491 at /home/pokybuild/yocto-autobuilder/yocto-worker/intel-corei7-64-rt/build/build/tmp/work-shared/intel-cor
ei7-64/kernel-source/kernel/sched/core.c:3384 migrate_disable+0xef/0x100
[    6.845285] Modules linked in:
[    6.845296] CPU: 1 PID: 491 Comm: udhcpc Not tainted 4.8.0-rc7-rt1-yocto-preempt-rt #1
[    6.845301] Hardware name: ADI Minnowboard Turbot D0 PLATFORM/MinnowBoard Turbot, BIOS MNW2MAX1.X64.0090.R01.1603151016 03/15/2016
[    6.845318]  0000000000000000 ffff8800378e7a38 ffffffff814061ef 0000000000000000
[    6.845330]  0000000000000000 ffff8800378e7a78 ffffffff8107b344 00000d38000ce5b4
[    6.845341]  ffff880037a754c0 0000000000000034 0000000000000034 ffffffff820cce40
[    6.845344] Call Trace:
[    6.845362]  [<ffffffff814061ef>] dump_stack+0x4f/0x70
[    6.845374]  [<ffffffff8107b344>] __warn+0xe4/0x100
[    6.845385]  [<ffffffff8107b42d>] warn_slowpath_null+0x1d/0x20
[    6.845393]  [<ffffffff810a0f4f>] migrate_disable+0xef/0x100
[    6.845406]  [<ffffffff810c9a6a>] call_console_drivers.isra.21.constprop.30+0xda/0x160
[    6.845417]  [<ffffffff810cae62>] console_unlock+0x252/0x630
[    6.845428]  [<ffffffff810cb4bf>] vprintk_emit+0x27f/0x500
[    6.845439]  [<ffffffff810a108f>] ? migrate_enable+0x12f/0x170
[    6.845449]  [<ffffffff810cb8af>] vprintk_default+0x1f/0x30
[    6.845459]  [<ffffffff8114e566>] printk+0x48/0x50
[    6.845471]  [<ffffffff81869147>] ? __alloc_skb+0x97/0x1c0
[    6.845479]  [<ffffffff810a108f>] ? migrate_enable+0x12f/0x170
[    6.845489]  [<ffffffff8107b299>] __warn+0x39/0x100
[    6.845499]  [<ffffffff8107b42d>] warn_slowpath_null+0x1d/0x20
[    6.845507]  [<ffffffff810a108f>] migrate_enable+0x12f/0x170
[    6.845519]  [<ffffffff819ed607>] rt_spin_unlock+0x27/0x40
[    6.845528]  [<ffffffff81880725>] __dev_queue_xmit+0x415/0x690
[    6.845538]  [<ffffffff818809b0>] dev_queue_xmit+0x10/0x20
[    6.845549]  [<ffffffff8196c0a7>] packet_sendmsg+0xb87/0x13e0
[    6.845560]  [<ffffffff819eaf1a>] ? wait_for_completion+0x3a/0xd0
[    6.845572]  [<ffffffff810d529c>] ? __wait_rcu_gp+0xac/0xd0
[    6.845588]  [<ffffffff81860ecf>] SYSC_sendto+0xff/0x180
[    6.845603]  [<ffffffff811d5ec5>] ? fd_install+0x25/0x30
[    6.845612]  [<ffffffff8185fd54>] ? sock_map_fd+0x44/0x70
[    6.845624]  [<ffffffff8186156e>] SyS_sendto+0xe/0x10
[    6.845633]  [<ffffffff819ed8db>] entry_SYSCALL_64_fastpath+0x13/0x8f
[    6.845641] ---[ end trace 0000000000000002 ]---
[    7.245277] WARNING: CPU: 1 PID: 491 at /home/pokybuild/yocto-autobuilder/yocto-worker/intel-corei7-64-rt/build/build/tmp/work-shared/intel-cor
ei7-64/kernel-source/kernel/sched/core.c:3415 migrate_enable+0x12f/0x170
[    7.273397] Modules linked in:
[    7.280198] CPU: 1 PID: 491 Comm: udhcpc Tainted: G        W       4.8.0-rc7-rt1-yocto-preempt-rt #1
[    7.280200] Hardware name: ADI Minnowboard Turbot D0 PLATFORM/MinnowBoard Turbot, BIOS MNW2MAX1.X64.0090.R01.1603151016 03/15/2016
[    7.280206]  0000000000000000 ffff8800378e7c18 ffffffff814061ef 0000000000000000
[    7.280211]  0000000000000000 ffff8800378e7c58 ffffffff8107b344 00000d578107cea0
[    7.280215]  ffff880037a754c0 0000000000000002 ffff8800372db600 ffffffff81f00b20
[    7.280216] Call Trace:
[    7.280223]  [<ffffffff814061ef>] dump_stack+0x4f/0x70
[    7.280227]  [<ffffffff8107b344>] __warn+0xe4/0x100
[    7.280231]  [<ffffffff8107b42d>] warn_slowpath_null+0x1d/0x20
[    7.280234]  [<ffffffff810a108f>] migrate_enable+0x12f/0x170
[    7.280239]  [<ffffffff819ed607>] rt_spin_unlock+0x27/0x40
[    7.280242]  [<ffffffff81880725>] __dev_queue_xmit+0x415/0x690
[    7.280246]  [<ffffffff818809b0>] dev_queue_xmit+0x10/0x20
[    7.280250]  [<ffffffff8196c0a7>] packet_sendmsg+0xb87/0x13e0
[    7.280254]  [<ffffffff819eaf1a>] ? wait_for_completion+0x3a/0xd0
[    7.280258]  [<ffffffff810d529c>] ? __wait_rcu_gp+0xac/0xd0
[    7.280264]  [<ffffffff81860ecf>] SYSC_sendto+0xff/0x180
[    7.280269]  [<ffffffff811d5ec5>] ? fd_install+0x25/0x30
[    7.280273]  [<ffffffff8185fd54>] ? sock_map_fd+0x44/0x70
[    7.280277]  [<ffffffff8186156e>] SyS_sendto+0xe/0x10
[    7.280280]  [<ffffffff819ed8db>] entry_SYSCALL_64_fastpath+0x13/0x8f
[    7.280315] ---[ end trace 0000000000000003 ]---

This is from a MinnowBoard Turbot, but I have also reproduced it on the NUC6.
Comment 1 Bruce Ashfield 2016-09-28 17:54:08 UTC 
I don't have any hardware to reproduce this. Qemux86* is booting fine.
Comment 2 Paul Gortmaker 2016-10-01 19:00:27 UTC 
Can you attach your local.conf and bblayers.conf?

And I think you have some local changes in meta-intel that are not on master branch.  Currently there is no 4.8-rt bbappend in meta-intel ; just the 4.1 and 4.4 versions so this currently isn't even a valid bug.

I had to hack around and put COMPATIBLE_MACHINE_corei7-64-intel-common = "${MACHINE}" into my local.conf but now I've not no confidence I'm building and configuring what you were.

Also please include full dmesg output showing the kernel bootargs ; at the moment I'm guessing you were using NFS root due to the relatively early timestamps on the kernel messages -- but again I don't want to have to guess at the relevant steps in attempting to create a reproducer so I can test any prospective fix(es).
Comment 3 California Sullivan 2016-10-03 17:32:38 UTC 
Created attachment 3448 [details]
dmesg log
Comment 4 California Sullivan 2016-10-03 17:33:45 UTC 
Created attachment 3449 [details]
local.conf
Comment 5 California Sullivan 2016-10-03 17:34:24 UTC 
Created attachment 3450 [details]
bblayers.conf
Comment 6 California Sullivan 2016-10-03 17:36:32 UTC 
My changes that add the 4.8 preempt-rt kernel recipe are here: https://git.yoctoproject.org/cgit/cgit.cgi/meta-intel-contrib/log/?h=clsulliv/master-test

This was originally found building on the autobuilder, but I also verified it occurs with the above local.conf and bblayers.conf in a local build.
Comment 7 Paul Gortmaker 2016-10-03 18:36:31 UTC 
I believe I've an understanding of the issue here; it is fixed by dropping a patch from the -rt series as I did here:

https://git.kernel.org/cgit/linux/kernel/git/paulg/4.8-rt-patches.git/commit/?h=v4.8-rt

Bruce has integrated the above change into the 4.8 kernel release he just did.
Comment 8 California Sullivan 2016-10-03 21:44:29 UTC 
As of Bruce's integration patch I can confirm this no longer occurs.

https://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.8/commit/?h=standard/preempt-rt/base&id=cd97b2cf4bc73316204a5148d2b0f5fc5d2ecc19