15138 2023-06-08 13:03:51 +0000 AB-INT: qemux86/x86-64 hangs intermittently 2023-07-20 14:53:06 +0000 1 1 10 QA/Testing Runtime Testing general unspecified x86 Multiple RESOLVED FIXED AB-INT Medium+ normal 4.3 M2 1 alexandre.belloni unassigned alexandre.belloni bruce.ashfield paulg randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 95652 0 alexandre.belloni 2023-06-08 13:03:51 +0000 qemu seems to hang since the switch to kernel 6.1. We though the stable upgrades fixed it but this is still happening. https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/7249/steps/15/logs/stdio qa-extras2 fedora38-ty-4 95653 1 alexandre.belloni 2023-06-08 13:04:45 +0000 The last line before the hang is often: [ 0.267040] Freeing SMP alternatives memory: 52K 95654 2 alexandre.belloni 2023-06-08 13:13:56 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5249/steps/14/logs/stdio oe-selftest-debian debian11-ty-1 95660 3 randy.macleod 2023-06-08 14:50:27 +0000 Probably came in with 6.1.26 and onwards. Updates to 6.1.30/31 didn't help. 95684 4 alexandre.belloni 2023-06-08 21:30:17 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5121/steps/13/logs/stdio qemux86-64-ptest ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5287/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2004-ty-1 95701 5 paulg 2023-06-10 23:59:25 +0000 So, I can't recall exactly, but stuff I was sent like this caught my attention: [ 0.307298] Freeing SMP alternatives memory: 52K [ 19.910999] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) My immediate reaction was "What the heck was it doing for 18+ seconds?!?" And so I'll own some of the blame of pointing people at possible changing of code for on-lining CPUs that comes *after* the "Freeing" message. But if we ignore e-mail and go to the AB log, and expand our context backwards in time to where CPU mitigations start and take a wider view.... [ 0.274003] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.275004] Spectre V2 : Mitigation: Retpolines [ 0.276000] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.277000] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT [ 0.278002] Speculative Store Bypass: Vulnerable [ 0.279002] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.280001] MMIO Stale Data: Unknown: No mitigations [ 0.281006] SRBDS: Unknown: Dependent on hypervisor status [ 0.307298] Freeing SMP alternatives memory: 52K [ 19.910999] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) Do you see it? I was focused on the evidence of the forest fire and not looking for the spark that started it. Look closely. Pretty much every step is done in 0.001 -- but then SRDBS fires, and wham-o. It goes from 0.281 to 0.307 - which is like a 25x+ increase. I can't directly compare to my stuff, because I always boot bare-metal with "mitigations=off". The SRBDS workarounds have been mainline long before v6.1 so we can't directly blame that. We aren't really interested in testing all those mitigations from a Yocto perspective, and we've already tried turning SMP off, so I'd suggest AB turning mitigations off with the boot arg. Permanently - even if they don't solve this AB hang quirk. 95712 6 alexandre.belloni 2023-06-14 08:31:12 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5326/steps/14/logs/stdio oe-selftest-fedora fedora37-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5283/steps/14/logs/stdio oe-selftest-debian debian11-ty-3 95742 7 paulg 2023-06-16 14:39:28 +0000 For the paper trail, we believe the backport of e9523a0d81 to various stable releases is the root cause: https://lkml.org/lkml/2023/6/13/1460 https://lore.kernel.org/all/23fb8ad7-beb0-ae1c-fa5a-a682a57f79b0@grsecurity.net/ Proposed fix: https://lore.kernel.org/all/20230615091830.RxMV2xf_@linutronix.de/ 95745 8 paulg 2023-06-16 19:37:13 +0000 https://lists.openembedded.org/g/openembedded-core/message/182829 95999 9 randy.macleod 2023-07-20 14:53:06 +0000 Fixed by: https://git.openembedded.org/openembedded-core/commit/?id=73b7f36e51de 73b7f36e51de linux-yocto/6.1: fix intermittent x86 boot hangs