qemu seems to hang since the switch to kernel 6.1. We though the stable upgrades fixed it but this is still happening. https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/7249/steps/15/logs/stdio qa-extras2 fedora38-ty-4
The last line before the hang is often: [ 0.267040] Freeing SMP alternatives memory: 52K
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5249/steps/14/logs/stdio oe-selftest-debian debian11-ty-1
Probably came in with 6.1.26 and onwards. Updates to 6.1.30/31 didn't help.
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5121/steps/13/logs/stdio qemux86-64-ptest ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5287/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2004-ty-1
So, I can't recall exactly, but stuff I was sent like this caught my attention: [ 0.307298] Freeing SMP alternatives memory: 52K [ 19.910999] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) My immediate reaction was "What the heck was it doing for 18+ seconds?!?" And so I'll own some of the blame of pointing people at possible changing of code for on-lining CPUs that comes *after* the "Freeing" message. But if we ignore e-mail and go to the AB log, and expand our context backwards in time to where CPU mitigations start and take a wider view.... [ 0.274003] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.275004] Spectre V2 : Mitigation: Retpolines [ 0.276000] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.277000] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT [ 0.278002] Speculative Store Bypass: Vulnerable [ 0.279002] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.280001] MMIO Stale Data: Unknown: No mitigations [ 0.281006] SRBDS: Unknown: Dependent on hypervisor status [ 0.307298] Freeing SMP alternatives memory: 52K [ 19.910999] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) Do you see it? I was focused on the evidence of the forest fire and not looking for the spark that started it. Look closely. Pretty much every step is done in 0.001 -- but then SRDBS fires, and wham-o. It goes from 0.281 to 0.307 - which is like a 25x+ increase. I can't directly compare to my stuff, because I always boot bare-metal with "mitigations=off". The SRBDS workarounds have been mainline long before v6.1 so we can't directly blame that. We aren't really interested in testing all those mitigations from a Yocto perspective, and we've already tried turning SMP off, so I'd suggest AB turning mitigations off with the boot arg. Permanently - even if they don't solve this AB hang quirk.
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5326/steps/14/logs/stdio oe-selftest-fedora fedora37-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5283/steps/14/logs/stdio oe-selftest-debian debian11-ty-3
For the paper trail, we believe the backport of e9523a0d81 to various stable releases is the root cause: https://lkml.org/lkml/2023/6/13/1460 https://lore.kernel.org/all/23fb8ad7-beb0-ae1c-fa5a-a682a57f79b0@grsecurity.net/ Proposed fix: https://lore.kernel.org/all/20230615091830.RxMV2xf_@linutronix.de/
https://lists.openembedded.org/g/openembedded-core/message/182829
Fixed by: https://git.openembedded.org/openembedded-core/commit/?id=73b7f36e51de 73b7f36e51de linux-yocto/6.1: fix intermittent x86 boot hangs