https://autobuilder.yoctoproject.org/typhoon/#/builders/145/builds/684/steps/12/logs/stdio qemux86-tc on fedora38-ty-1 Linux kernel 6.5.5. Note this is not using kvm. Relevant section of the kernel messages: [ 0.076085] __common_interrupt: 0.119 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] ------------[ cut here ]------------ [ 0.076085] WARNING: CPU: 0 PID: 0 at /arch/x86/kernel/fpu/core.c:769 fpregs_assert_state_consistent+0x36/0x4c [ 0.076085] Modules linked in: [ 0.076085] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.5.5-yocto-standard #1 [ 0.076085] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014 [ 0.076085] EIP: fpregs_assert_state_consistent+0x36/0x4c [ 0.076085] Code: 00 f5 d3 c7 8b 03 f6 c4 40 74 09 8b 5d fc c9 c3 8d 74 26 00 e8 f7 ef cb 00 64 8b 15 0c 57 d3 c7 8d 8b 00 08 00 00 39 d1 74 0a <0f> 0b 8b 5d fc c9 c3 8d 76 00 3b 83 00 08 00 00 75 ee eb cd 66 90 [ 0.076085] EAX: 00000000 EBX: c7a7c540 ECX: c7a7cd40 EDX: 00000000 [ 0.076085] ESI: c7a7c540 EDI: c7a799a0 EBP: c7a79960 ESP: c7a7995c [ 0.076085] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00200082 [ 0.076085] CR0: 80050033 CR2: ffd3a000 CR3: 07d50000 CR4: 001506d0 [ 0.076085] Call Trace: [ 0.076085] ? show_regs+0x4f/0x58 [ 0.076085] ? fpregs_assert_state_consistent+0x36/0x4c [ 0.076085] ? __warn+0x70/0x138 [ 0.076085] ? fpregs_assert_state_consistent+0x36/0x4c [ 0.076085] ? report_bug+0x171/0x1ac [ 0.076085] ? exc_overflow+0x38/0x38 [ 0.076085] ? handle_bug+0x2d/0x58 [ 0.076085] ? exc_invalid_op+0x18/0x54 [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_overflow+0x38/0x38 [ 0.076085] ? fpregs_assert_state_consistent+0x36/0x4c [ 0.076085] ? exc_overflow+0x38/0x38 [ 0.076085] ? fpregs_assert_state_consistent+0x36/0x4c [ 0.076085] exit_to_user_mode_prepare+0x38/0x194 [ 0.076085] ? iret_error+0x30/0x60 [ 0.076085] irqentry_exit_to_user_mode+0x8/0x20 [ 0.076085] irqentry_exit+0x39/0x3c [ 0.076085] common_interrupt+0x30/0x38 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] asm_common_interrupt+0x102/0x140 [ 0.076085] EIP: 0x60 [ 0.076085] EAX: 00000000 EBX: c7a799e8 ECX: 00000000 EDX: 006afc90 [ 0.076085] ESI: 00000001 EDI: c76efb94 EBP: c7a799e0 ESP: 00000001 [ 0.076085] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 99e8 EFLAGS: c7a79a34 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? asm_common_interrupt+0x1/0x140 [ 0.076085] ? exc_device_not_available+0x5c/0x5c [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? __optimize_nops+0x26c/0x27c [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? asm_common_interrupt+0x2/0x140 [ 0.076085] ? asm_common_interrupt+0x11/0x140 [ 0.076085] ? asm_common_interrupt+0x3/0x140 [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? optimize_nops+0x61/0xb0 [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? asm_common_interrupt+0xf/0x140 [ 0.076085] ? asm_common_interrupt+0x1/0x140 [ 0.076085] ? apply_alternatives+0x1e3/0x3dc [ 0.076085] ? asm_exc_debug+0x30/0x30 [ 0.076085] ? alternative_instructions+0x30/0xd4 [ 0.076085] ? arch_cpu_finalize_init+0x48/0x64 [ 0.076085] ? start_kernel+0x347/0x5c4 [ 0.076085] ? i386_start_kernel+0x48/0x48 [ 0.076085] ? startup_32_smp+0x156/0x158 [ 0.076085] ---[ end trace 0000000000000000 ]--- [ 0.076085] BUG: unable to handle page fault for address: 000024c0 [ 0.076085] #PF: user instruction fetch in kernel mode [ 0.076085] #PF: error_code(0x0014) - not-present page [ 0.076085] *pde = 00000000 [ 0.076085] Oops: 0014 [#1] PREEMPT SMP [ 0.076085] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.5.5-yocto-standard #1 [ 0.076085] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014 [ 0.076085] EIP: 0x60 [ 0.076085] Code: Unable to access opcode bytes at 0x36. [ 0.076085] EAX: 00000000 EBX: 00000020 ECX: 00000060 EDX: 006afc90 [ 0.076085] ESI: c7a79f68 EDI: 00000020 EBP: c7a79f20 ESP: c76f2c1c [ 0.076085] DS: 0000 ES: 0000 FS: 0000 GS: 0000 SS: 9eb8 EFLAGS: 00261e92 [ 0.076085] CR0: 80050033 CR2: 000024c0 CR3: 07d50000 CR4: 001506d0 [ 0.076085] Call Trace: [ 0.076085] ? show_regs+0x4f/0x58 [ 0.076085] ? __die+0x1d/0x58 [ 0.076085] ? page_fault_oops+0x16e/0x360 [ 0.076085] ? notifier_call_chain+0x3c/0xb8 [ 0.076085] ? kernelmode_fixup_or_oops.constprop.0+0x7c/0xcc [ 0.076085] ? __bad_area_nosemaphore.constprop.0+0x114/0x1a8 [ 0.076085] ? iret_error+0x4b/0x60 [ 0.076085] ? bad_area_nosemaphore+0xf/0x14 [ 0.076085] ? do_user_addr_fault+0x1dc/0x3d0 [ 0.076085] ? exc_page_fault+0x53/0x130 [ 0.076085] ? pvclock_clocksource_read_nowd+0x130/0x130 [ 0.076085] ? handle_exception+0x133/0x133 [ 0.076085] ? pvclock_clocksource_read_nowd+0x130/0x130 [ 0.076085] ? ct_irq_exit+0x8/0xc [ 0.076085] Modules linked in: [ 0.076085] CR2: 00000000000024c0 [ 0.076085] ---[ end trace 0000000000000000 ]--- [ 0.076085] EIP: 0x60 [ 0.076085] Code: Unable to access opcode bytes at 0x36. [ 0.076085] EAX: 00000000 EBX: 00000020 ECX: 00000060 EDX: 006afc90 [ 0.076085] ESI: c7a79f68 EDI: 00000020 EBP: c7a79f20 ESP: c76f2c1c [ 0.076085] DS: 0000 ES: 0000 FS: 0000 GS: 0000 SS: 9eb8 EFLAGS: 00261e92 [ 0.076085] CR0: 80050033 CR2: 000024c0 CR3: 07d50000 CR4: 001506d0 [ 0.076085] Kernel panic - not syncing: Attempted to kill the idle task! [ 0.076085] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
Happened once before. Usually with x86, no kvm. Bruce Ashfield knows about the issue but isn't working on it much if at all.
https://autobuilder.yoctoproject.org/typhoon/#/builders/145/builds/749/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/145/builds/706
Was there enough build time logged on v6.4.x that we can say this issue came in with the late breaking move from v6.4 to v6.5 with any confidence?
(In reply to Paul Gortmaker from comment #4) > Was there enough build time logged on v6.4.x that we can say this issue came > in with the late breaking move from v6.4 to v6.5 with any confidence? As far as I know we only saw it with 6.5 and never while we had 6.4 there...
So, courtesy of the "fun" ttyS1 bug, I already had what amounts to a stand-alone OE-QA "snippet" from Richard that would let me easily do a boot loop on an existing build that is "as close as possible" to what the OE-QA stuff does - including the launch and connect to the TCP serial port streams. Long story short - I built master with the default conf/local.conf file except for the one line change to select qemu x86 32 for MACHINE. I ran that in a loop for 12h and with RP's OE-QA "snippet" and with KVM explicitly disabled. No joy. After that 12h and 1600+ iterations, I didn't get one instance of FPU issues. Unfortunate, since the estimate on "routine automated testing" was about once in 30 runs. But I guess every non-detection tells you something implicitly? On to other strategies...
So, unable to create one myself, I decided to deconstruct the error log from the case. And if this reads like it has useless details, well it has. I don't know yet where this is going, so I don't have the luxury of revisionist history. I'm just putting down where my thoughts are currently at. Let us look at the crux of what we do have. I have left the printk-time prefixes intentionally. [ 0.031196] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT [ 0.033236] Speculative Store Bypass: Vulnerable [ 0.035323] L1TF: Kernel not compiled for PAE. No mitigation for L1TF [ 0.037600] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.038269] MMIO Stale Data: Unknown: No mitigations [ 0.040413] SRBDS: Unknown: Dependent on hypervisor status [ 0.054633] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.061441] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.062304] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.063653] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.065293] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.076085] __common_interrupt: 0.119 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] __common_interrupt: 0.165 No irq handler for vector [ 0.076085] ------------[ cut here ]------------ [ 0.076085] WARNING: CPU: 0 PID: 0 at /arch/x86/kernel/fpu/core.c:769 fpregs_assert_state_consistent+0x36/0x4c A "proper" boot (from the logs of my slower qemu host) looks something like this: [ 0.289210] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.289352] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.289965] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.290229] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.303087] Freeing SMP alternatives memory: 52K [ 0.303495] pid_max: default: 32768 minimum: 301 [ 0.310188] LSM: initializing lsm=capability,landlock,integrity The times aren't directly comparable since they rely on the speed of the host running qemu and whatever else. And two different host machines. 1st note is that we'd expect "Freeing SMP alternatives" as the next line in the failed case. But we get the 1st "No irq handler" error. Next is that the delta between the last x86/fpu line and the 1st "common_interrupt" error line. It went from 0.065 to 0.076. On the slower "correct" machine, it went from 0.290 to 0.303 - both about 0.01, even though the latter is slower. But don't dwell on that -- it is just to set scale. Look at the common_interrupt lines. Their time is unchanged across the board. So that is in a tight loop and/or we are not advancing time. Where does the common_interrupt message thing come from? (sorry for the largish paste) /* * common_interrupt() handles all normal device IRQ's (the special SMP * cross-CPU interrupts have their own entry points). */ DEFINE_IDTENTRY_IRQ(common_interrupt) { struct pt_regs *old_regs = set_irq_regs(regs); struct irq_desc *desc; /* entry code tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU"); desc = __this_cpu_read(vector_irq[vector]); if (likely(!IS_ERR_OR_NULL(desc))) { handle_irq(desc, regs); } else { ack_APIC_irq(); if (desc == VECTOR_UNUSED) { pr_emerg_ratelimited("%s: %d.%u No irq handler for vector\n", <---------- __func__, smp_processor_id(), vector); } else { __this_cpu_write(vector_irq[vector], VECTOR_UNUSED); } } set_irq_regs(old_regs); } The %s is __common_interrupt (handler name -- obtained by __func__ to avoid the macro obfuscation) ; the core is zero and the vector is 119/165 as printed. In the "obvious" case, it just does "handle_irq() - but if my reading is correct, you get one free pass on having a non assigned IRQ vector. Then VECTOR_UNUSED is assigned and the next time through it screams at you. So you are probably asking why isn't this a function and what the DEFINE_IDTENTRY_IRQ stuff is? It is macro magic in arch/x86/include/asm/idtentry.h that kind of wraps the core irq in preamble and postamble (is that a word?) instrumentation_begin(); \ kvm_set_cpu_l1tf_flush_l1d(); \ run_irq_on_irqstack_cond(__##func, regs, vector); \ <---- actual IRQ instrumentation_end(); \ irqentry_exit(regs, state); I don't think we need to dwell on the wrapper goo at this point. But what do I know? Next we get the WARNING from arch/x86/kernel/fpu/core.c:769 -- what does that look like? /* * If current FPU state according to its tracking (loaded FPU context on this * CPU) is not valid then we must have TIF_NEED_FPU_LOAD set so the context is * loaded on return to userland. */ void fpregs_assert_state_consistent(void) { struct fpu *fpu = ¤t->thread.fpu; if (test_thread_flag(TIF_NEED_FPU_LOAD)) return; WARN_ON_FPU(!fpregs_state_valid(fpu, smp_processor_id())); <------------ splat! } At this point, I'm thinking this last part isn't a problem but a symptom. We did FPU setup - a bunch of core IDT stuff went pear shaped - and whoa! The FPU state is trash. I am so shocked! I'll admit things have changed a lot since I've looked at FPU stuff that was hardwired to irq13 on garbage from many decades ago - so we'll leave it at vague but possible conclusion, and next go data mining in git history...
We've identified several "files of interest" and Richard has a degree of confidence that the issue wasn't there in v6.4 but was in v6.5. Lets combine the two and see what git history shows us. paul@q9400:~/git/linux-yocto$ git log --oneline v6.5 ^v6.4 arch/x86/include/asm/idtentry.h paul@q9400:~/git/linux-yocto$ git log --oneline v6.5 ^v6.4 arch/x86/kernel/fpu/core.c 1f69383b203e x86/fpu: Invalidate FPU state correctly on exec() paul@q9400:~/git/linux-yocto$ git log --oneline v6.5 ^v6.4 arch/x86/kernel/irq.c 504dba50b0c3 x86/irq: Add hardcoded hypervisor interrupts to /proc/stat paul@q9400:~/git/linux-yocto$ OK - one strikeout, and a couple things to look into further. I always like to look at the "container" of what a commit was packaged with - this can tell you a lot about what the submitter was working on and any implicit dependencies it might have had. paul@q9400:~/git/linux-yocto$ git describe --contains 1f69383b203e v6.5~2^2~1 If you don't speak git, here is a crash course. You read right to left, not left to right. A "~" followed by a number N basically means "back N commits from" A caret "^" is a reference to a merge. Most merges are just involving two streams -- that would be ^1 (the maintainer, or Linus in this case) and ^2 (the submitter). You can leave off the "1" - in that "^1" is the same as just "^". So, read the above "backwards" - i.e. right to left. You've got "one commit back (~1) from the 2nd branch (^2) of the merge v6.5~2 -- which is just two commits back from the final tag of v6.5 --- and yes all of these references are valid for git args - you can see the merge with "git show v6.5~2" and similar. So, just what exactly was in that last minute merge, just before the final v6.5 tag? Well, we need another minor git speak detail. A leading caret "^" that has nothing before it is basically a "not" operator. So a common use would be "show me what was in the submitter branch (^2) but NOT (^) all the goop already in mainline parent (^1). paul@q9400:~/git/linux-yocto$ git log --oneline v6.5~2^2 ^v6.5~2^1 2c66ca3949dc x86/fpu: Set X86_FEATURE_OSXSAVE feature after enabling OSXSAVE in CR4 1f69383b203e x86/fpu: Invalidate FPU state correctly on exec() paul@q9400:~/git/linux-yocto$ Yes, the trailing "1" on the command line was redundant, but maybe useful for an example? Anyway, we now know the "git show v6.5~2" merge only brought in two last minute commits. Lets give them both a look: paul@q9400:~/git/linux-yocto$ git show 1f69383b203e | grep Fixes: Fixes: 33344368cb08 ("x86/fpu: Clean up the fpu__clear() variants") paul@q9400:~/git/linux-yocto$ git describe --contains 33344368cb08 v5.14-rc1~52^2~20 Well, how broken can it have been, if the origin breakage was in v5.14??? Shouldn't be appearing in the v6.4 --> v6.5 transition. What about the other one? paul@q9400:~/git/linux-yocto$ git show 2c66ca3949dc | grep Fixes: Fixes: b81fac906a8f ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()") paul@q9400:~/git/linux-yocto$ git describe --contains b81fac906a8f v6.5-rc1~231^2~1 [for those training at home, that is one commit back from the head of the 2nd branch of the merge that was 231 commits prior to the v6.5-rc1 tag] Well that looks a lot more interesting. The commit is in the v6.4 ---> v6.5 window. What is it all about? paul@q9400:~/git/linux-yocto$ git show v6.5-rc1~231^2~1 | head -n 8 commit b81fac906a8f9e682e513ddd95697ec7a20878d4 Author: Thomas Gleixner <tglx@linutronix.de> Date: Wed Jun 14 01:39:46 2023 +0200 x86/fpu: Move FPU initialization into arch_cpu_finalize_init() Initializing the FPU during the early boot process is a pointless exercise. Early boot is convoluted and fragile enough. paul@q9400:~/git/linux-yocto$ Oooh. Early boot FPU init stuff - smells pretty related to the stuff we are seeing in this case!! And that is where the story stops for now - because no word of a lie, that is all I know so far. If I can't get back to it immediately, at least what I know is all here. And maybe someone will appreciate and learn from how the thought process went.
Ha, so this is kind of amusing (and sad). I'd tinkered with some udelay() around the FPU init (no joy), and it is still bugging me why I can't reproduce this. Then I remembered I was using the distro's qemu and decided I should try the one from our sysroot. So as I'm tinkering with hacking runqemu from Richard, and ensuring I am actually running the qemu binary from my sysroot, I'm manually running the command.py from Richard one instance at a time. And since I'm actually watching the screen, and not running it in an endless loop (yet) -- I notice it appears to be stalled, and then it hits my (much shortened) 100s timeout -- woot, I'm thinking just like that - "I've got one!" And then in a directory relatively clear of log files, I see what was happening. My "loop" was this on-the-fly command line: while [ 1 ] ; do ./commands.py ; echo xxx $? xxx ; ls -l *.2 |grep -v ' 267 ' ; sleep 2 ; done Basically a command history copy of what I was using for the ttyS1 issue - filter out the "proper" logs of 267 bytes abd you'll see the log files for the broken instance(s). One problem. You don't get a zero byte ttyS1 log file if the kernel doesn't boot. You don't get any "dot 2" log file at all. Doh!! I go back and look at my earlier 12h run data - which had 1700+ runs before I hit ^C and sure enough a substring of the error is in the boot logs from ttyS0: $grep -l __common * | wc -l 41 If you do the math, that is about 2.4% reproducibility - as good or better than the hateful ttyS1 issue. Which means I should be able to test on either side of the FPU commits that raised an eyebrow earlier, and with a bit of luck I can once again "statistically prove" which commit broke things.
Since I was done for the day after writing the above, I let an overnight test run with the yocto qemu in the sysroot vs. the distro qemu. Resulted in 34 fails in 1624 runs, or about 2.1% -- within expected deviation from the original 12hr run done with the distro qemu.
As it turned out, the poor FPU changes -- even though the FPU commit logs I mentioned above screamed "me me me!" -- appear to be innocent. So I turn my attention to the merges from the tip repository (if you know x86 kernel merge window history, that makes sense). Was all set to find the issue in core or irq or boot merge, but no. The merge that brought in the regression was: 8c69e7afe958 Merge tag 'x86_alternatives_for_v6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip I'm attaching a detailed listing of my "manual bisect" test points, which includes a listing of the alternate commits, if anybody isn't quite sure what "alternates" means in this context. [And if changing executable code at runtime scares you, you aren't alone] That merge in turn contains 12 commits which I also pseudo-bisected, with a preference knowing a fail is easier to prove (time-wise expense) than a "good" commit. Not to bury the lead any worse than I have, it finally landed me at: bad (6/381) 6c480f222128 x86/alternative: Rewrite optimize_nops() some <-------- first *bad* good (0/367) 14e4ec9c3e91 x86/lib/memmove: Decouple ERMS from FSRM So (6/381) means six fails in 381 executions of the qemu boot test. It is unclear at this point if the kernel change is at fault, or qemu (w/o kernel support) is at fault. I'll pass the information found here upwards to the right people, and we'll see where it goes.
Created attachment 4983 [details] Good/bad kernel commits as tested in a manual bisect. Not from an actual run of "git bisect" - knowing where content came from, and knowing "bad" commits are easier to detect than "good" ones, the choice of what commits to test was done manually.
https://lore.kernel.org/all/ZT6narvE+LxX+7Be@windriver.com/
https://autobuilder.yoctoproject.org/typhoon/#/builders/145/builds/932/steps/12/logs/stdio qemux86-tc ubuntu2004-ty-1
Created attachment 4995 [details] dmesg from booting v6.6.0 with "debug-alternates" boot option. I've added a "dmesg" output for a failed boot on v6.6.0 - had to also boot with "ignore-loglevel" because they are at debug loglevel. You can see them in a normal boot after logging logging in and running "dmesg". But in a crash boot, you can't login and they don't end up on the console. :(
Created attachment 4996 [details] dmesg from good (non-failed) boot of same v6.6.0 the exact same way For comparison, a boot with the identical kernel image and identical boot and qemu args (both runs launched from the same automated script). There is definitely a whole bunch more SMP debug-alternates that should be printed after the crash!!
Created attachment 4997 [details] kernel config file from the v6.6 kernel build This is the kernel config file from the v6.6.0 build - it is basically just a carry forward of the default v6.5 file auto-created by the Yocto build system.
Upstream thread where "debug-alternates" was mentioned: https://lore.kernel.org/all/ZT6narvE+LxX+7Be@windriver.com/
With the additional info tglx requested, he was able to spot the flaw in the change. My work here is done. https://lore.kernel.org/lkml/87r0k9ym0y.ffs@tglx/T/#meef1f825891fe4c6d5b102b964cee3cb486f2d68