Bug 8663

Summary: Kernel oops on qemuarm boot
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Mihail Stanciu <stanciux.mihail>
Component: coreAssignee: Aníbal Limón <anibal.limon>
Status: VERIFIED FIXED QA Contact: Mihail Stanciu <stanciux.mihail>
Severity: normal    
Priority: Medium+ CC: benjamin.esquivel, bruce.ashfield, meta.mr.watcher, meta.watcher, richard.purdie
Version: 2.0   
Target Milestone: 2.1 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: Unknown (Hard to categorize)
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
fail log
none
blk-mq: fix race between timeout and freeing request none

Description Mihail Stanciu 2015-11-06 13:52:08 UTC
Tested on master: fc45deac89ef63ca1c44e763c38ced7dfd72cbe1
OS: OpenSUSE

During DISTRO tests the qemuarm buildset failed with kernel oops:
Internal error: Oops: 17 [#1] PREEMPT ARM
Modules linked in:
CPU: 0 PID: 42 Comm: jbd2/vda-8 Not tainted 4.1.8-yocto-standard #1
Hardware name: ARM-Versatile PB
task: c7223300 ti: c725e000 task.ti: c725e000
PC is at blk_mq_tag_to_rq+0x24/0x78
LR is at bt_for_each+0x70/0xc4
pc : [<c0364458>]    lr : [<c0368118>]    psr: 20000113
sp : c725fa18  ip : c725fa30  fp : c725fa2c
r10: c0365ce0  r9 : 00000020  r8 : c7104200
r7 : c70ee5ac  r6 : 00000001  r5 : 00000021  r4 : c7122da0
r3 : c7104000  r2 : 00000000  r1 : 00000021  r0 : 00000000
Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 00093177  Table: 072f4000  DAC: 00000017
Process jbd2/vda-8 (pid: 42, stack limit = 0xc725e190)
Stack: (0xc725fa18 to 0xc7260000)

Suspect this was a hardware failure on the machine, due to not being able to reproduce issue so far.
Left several buildsets to run over the weekend and if issue does not repeat will close bug.
Comment 1 Mihail Stanciu 2015-11-06 14:02:56 UTC
(In reply to comment #0)
> During DISTRO tests the qemuarm buildset failed with kernel oops:

Correction, buildset was nightly-arm.
Comment 2 Mihail Stanciu 2015-11-09 10:15:42 UTC
Was unable to reproduce problem.
Will continue to run builds for another day or two.
Comment 3 Mihail Stanciu 2015-11-12 13:07:53 UTC
Been running builds since this first happened. Haven't been able to reproduce this as of yet. Will continue to run builds at least for a few more days.
Comment 4 Mihail Stanciu 2015-11-13 14:02:51 UTC
So far have run 15 back-to-back builds and I haven't been able to reproduce this.
Comment 5 Aníbal Limón 2015-11-13 21:42:16 UTC
Seems to be that this bug didn't happen again. So if the issue comes again please re-open the bug with panic information and core dump.
Comment 6 Richard Purdie 2015-11-16 11:02:01 UTC
These bugs happen intermittently so I don't think closing this is appropriate. 

Equally, a full kernel stack trace would go a long way to helping try and figure this issue out or at least allow it to be compared with other bugs. Was the error message pasted here to only information available from the crash? 

NEEDINFO might be a better status for the bug.
Comment 7 Aníbal Limón 2015-11-17 16:03:47 UTC
Set to NEEDINFO waiting for new information.
Comment 8 Mihail Stanciu 2015-11-24 09:10:13 UTC
I can provide access to the machine this was running on at the time it failed, but I don't think I can still recover the core dump for this, since the machine has been running other tests on it in the mean time and it has been subjected to at least 1 upgrade/reboot cycle.
Comment 9 Benjamin Esquivel 2015-11-24 17:41:01 UTC
It would be good to have a logic that stops when a given test fails for debugging, I believe that is not a feature of our current testing framework, right Mihail?
Comment 10 Mihail Stanciu 2015-11-25 08:41:18 UTC
(In reply to comment #9)
> It would be good to have a logic that stops when a given test fails for
> debugging, I believe that is not a feature of our current testing framework,
> right Mihail?

As far as I know, we don't have this feature in our framework.

This machine in particular is running tests under an Autobuilder setup, so having something like this wouldn't have helped all that much, since it was scheduled to run several buildsets in sequence. 
To be clear, the logs for the failure are still there and can be accessed, however I'm not sure if the kernel dump (if there is one from qemu) still exists.
Comment 11 Mihail Stanciu 2015-11-25 15:40:35 UTC
Created attachment 2867 [details]
fail log
Comment 12 Aníbal Limón 2015-11-25 15:44:15 UTC
(In reply to comment #11)
> Created attachment 2867 [details]
> fail log

With this log i started investigation...
Comment 13 Aníbal Limón 2015-11-26 18:03:52 UTC
I found a patch [1] that seems to solve the issue but it didn't apply directly to our linux-yocto-4.1.

I started to review the code for figure out if backport the patch is viable.

https://github.com/torvalds/linux/commit/0048b4837affd153897ed1222283492070027aa9
Comment 14 Aníbal Limón 2015-11-27 16:54:21 UTC
Created attachment 2876 [details]
blk-mq: fix race between timeout  and freeing request

I backported the patch and test into qemux86 and qemuarm running testimage for sato.
Comment 15 Aníbal Limón 2015-12-08 16:43:07 UTC
Patch integrated into linux-yocto [1], waiting for SRCREV recipe updates.

[1]http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.1/commit/?h=standard/base&id=2c30f87db7824e90b0b096eee3a5b7f93c84b079
Comment 17 Mihail Stanciu 2016-01-07 10:40:53 UTC
Verified on 2.1 M1: 5e3e2e0cbb0a49986f4653e64c4c8d2b5461645e