| Summary: | AB-INT: qemux86/x86-64 hangs intermittently | ||
|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Alexandre Belloni <alexandre.belloni> |
| Component: | general | Assignee: | Unassigned <unassigned> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alexandre.belloni, bruce.ashfield, paulg, randy.macleod |
| Version: | unspecified | ||
| Target Milestone: | 4.3 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Alexandre Belloni
2023-06-08 13:03:51 UTC
The last line before the hang is often: [ 0.267040] Freeing SMP alternatives memory: 52K https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5249/steps/14/logs/stdio oe-selftest-debian debian11-ty-1 Probably came in with 6.1.26 and onwards. Updates to 6.1.30/31 didn't help. https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5121/steps/13/logs/stdio qemux86-64-ptest ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5287/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2004-ty-1 So, I can't recall exactly, but stuff I was sent like this caught my attention: [ 0.307298] Freeing SMP alternatives memory: 52K [ 19.910999] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) My immediate reaction was "What the heck was it doing for 18+ seconds?!?" And so I'll own some of the blame of pointing people at possible changing of code for on-lining CPUs that comes *after* the "Freeing" message. But if we ignore e-mail and go to the AB log, and expand our context backwards in time to where CPU mitigations start and take a wider view.... [ 0.274003] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.275004] Spectre V2 : Mitigation: Retpolines [ 0.276000] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.277000] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT [ 0.278002] Speculative Store Bypass: Vulnerable [ 0.279002] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.280001] MMIO Stale Data: Unknown: No mitigations [ 0.281006] SRBDS: Unknown: Dependent on hypervisor status [ 0.307298] Freeing SMP alternatives memory: 52K [ 19.910999] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) Do you see it? I was focused on the evidence of the forest fire and not looking for the spark that started it. Look closely. Pretty much every step is done in 0.001 -- but then SRDBS fires, and wham-o. It goes from 0.281 to 0.307 - which is like a 25x+ increase. I can't directly compare to my stuff, because I always boot bare-metal with "mitigations=off". The SRBDS workarounds have been mainline long before v6.1 so we can't directly blame that. We aren't really interested in testing all those mitigations from a Yocto perspective, and we've already tried turning SMP off, so I'd suggest AB turning mitigations off with the boot arg. Permanently - even if they don't solve this AB hang quirk. https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5326/steps/14/logs/stdio oe-selftest-fedora fedora37-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5283/steps/14/logs/stdio oe-selftest-debian debian11-ty-3 For the paper trail, we believe the backport of e9523a0d81 to various stable releases is the root cause: https://lkml.org/lkml/2023/6/13/1460 https://lore.kernel.org/all/23fb8ad7-beb0-ae1c-fa5a-a682a57f79b0@grsecurity.net/ Proposed fix: https://lore.kernel.org/all/20230615091830.RxMV2xf_@linutronix.de/ Fixed by: https://git.openembedded.org/openembedded-core/commit/?id=73b7f36e51de 73b7f36e51de linux-yocto/6.1: fix intermittent x86 boot hangs |