| Summary: | Kernel panic for custom qemux86-64 bsp with yocto-bsp | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Meta-yocto | Reporter: | Yi Zhao <yi.zhao> | ||||||||
| Component: | meta-yocto | Assignee: | Alejandro Hernandez <alejandro.hernandez> | ||||||||
| Status: | RESOLVED FIXED | QA Contact: | Yi Zhao <yi.zhao> | ||||||||
| Severity: | normal | ||||||||||
| Priority: | Medium+ | CC: | alejandro.hernandez, bruce.ashfield, california.l.sullivan, jose.perez.carranza, leonardo.sandoval.gonzalez, poky.bs.watcher, poky.watcher, richard.purdie, sgw | ||||||||
| Version: | 2.2 | ||||||||||
| Target Milestone: | 2.2 M4 | ||||||||||
| Hardware: | x86 | ||||||||||
| OS: | Multiple | ||||||||||
| Whiteboard: | |||||||||||
| OS type for building Yocto: | --- | Type of Regression: | Regression (Used to work) | ||||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||||
| Attachments: |
|
||||||||||
|
Description
Yi Zhao
2016-07-26 09:18:44 UTC
Humberto, the problem is still present d57de67357a4cd3da2824f8bd2b02d100f1ac9c3 and its parent 3ddde5f32afe9225994df607f6b8dfcc58f6bdc7, so was introduce before these. (In reply to comment #1) > Humberto, > > the problem is still present d57de67357a4cd3da2824f8bd2b02d100f1ac9c3 and > its parent 3ddde5f32afe9225994df607f6b8dfcc58f6bdc7, so was introduce before > these. BTW, the test was done with the core-image-minimal image. Created attachment 3360 [details] kenerla_panic_full screenshot Ok, so after bisecting I found the guilty commit to be: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=55159ef0567c9c5f47609a18cff8866a17e53173 It seems a little weird since the code change is just to warn the user about the C library not being compatible with the kernel, and even if that was the case, there is no such warning shown. Also, I noticed the attachment is a bit misleading, since it looks like the panic is caused by a pritnk, although its missing some information, the error is actually no valid root filesystem found, I will attach a new screenshot. (In reply to comment #3) > Created attachment 3360 [details] > kenerla_panic_full screenshot > > Ok, so after bisecting I found the guilty commit to be: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/ > ?id=55159ef0567c9c5f47609a18cff8866a17e53173 Nice, this is a good start. > > It seems a little weird since the code change is just to warn the user about > the C library not being compatible with the kernel, and even if that was the > case, there is no such warning shown. > > Also, I noticed the attachment is a bit misleading, since it looks like the > panic is caused by a pritnk, although its missing some information, the > error is actually no valid root filesystem found, I will attach a new > screenshot. Alex, naive question: is the kernel command line changed somehow so the root FS became invalid? Can you try with another working FS using the same kernel? (In reply to comment #4) > Alex, naive question: is the kernel command line changed somehow so the root > FS became invalid? Can you try with another working FS using the same kernel? Nope, I already checked and the kernel cmdline isn't changed, in both cases its when trying to boot is uses /dev/vda, it may be that for some reason /dev/vda isn't found in our case. Created attachment 3361 [details]
kernel config diffs qemux86-64 vs myqemux86-64
I took the diff between the kernel configs for a working qemux86-64 and the myqemux86-64 on the commit where it starts failing, here it is in case someone finds something interesting.
(In reply to comment #6) > Created attachment 3361 [details] > kernel config diffs qemux86-64 vs myqemux86-64 > > I took the diff between the kernel configs for a working qemux86-64 and the > myqemux86-64 on the commit where it starts failing, here it is in case > someone finds something interesting. Actually, the diff should be or should have been between the same machine (myqemux86-64) using two different commits, the one where it starts failing and the one before that, I just did that, and there is nothing different between configs, so the error has to be coming from somewhere else. I found something interesting: When switching commits from the working one, to the first one where it fails, I noticed that it was taking linux-yocto from SSTATE, BUT if I clean the SSTATE on the working commit and then switch to the non working one and rebuild, it was still taking it from SSTATE, which means the following: Yes, 55159ef0567c9c5f47609a18cff8866a17e53173 is the first commit where the bug is present, BUT there could be several commits after that one where the linux-yocto signature is the same, meaning that if we didnt change anything regarding the kernel, it'd have the same signature, which also explains why there is a signature change on this specific commit because the kernel class was indeed modified. Anyway, the point is that this isn't necessarily the guilty commit which presented the bug, to be sure this needs to be re-tested, after cleaning linux-yocto SSTATE on the "guilty" commit After cleaning the shared state and re-testing , I can confirm that this is not the commit causing the issue, although it gives us a good clue, the actual commit where the bug occured must be between this, and the next one that involves somehow changin the kernel, which would invalidate the sstate signature. So, we go back to commit http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d57de67357a4cd3da2824f8bd2b02d100f1ac9c3 It is the first one where this issue is introduced, cleaning the shared states, and doing a: $ bitbake linux-yocto -c deploy fixes the problem, and vice versa, building on its parent 3ddde5f32afe9225994df607f6b8dfcc58f6bdc7 works fine, cleaning the shared states, and doing a $ bitbake linux-yocto -c deploy shows the problem again, so somehow, this commit is causing the error when bulding the kernel. Moving this to M4 due to time constraints. This is fixed on master, likely due to the kernel tools change. Note that runqemu will fail to start with the reproduction steps right now due to the created bsp layer not creating a qemuboot.conf file the runqemu scripts not being able to correctly guess the architecture of the machine. This is currently being worked on. In the mean time, you can apply the following patch (which has a significant error but can be used to verify that this bug is fixed) found here: https://patchwork.openembedded.org/patch/132469/ If you wish to test yourself. |