KVM is not working on Fedora43, resulting in broken tests: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/qemux86/build/layers/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) File "/srv/pokybuild/yocto-worker/qemux86/build/layers/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) File "/srv/pokybuild/yocto-worker/qemux86/build/layers/openembedded-core/meta/lib/oeqa/runtime/cases/oe_syslog.py", line 18, in test_syslog_running status, output = self.target.run(self.tc.target_cmds['ps']) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/pokybuild/yocto-worker/qemux86/build/layers/openembedded-core/meta/lib/oeqa/core/target/ssh.py", line 98, in run status, output = self._run(sshCmd, processTimeout, ignore_status, raw, ignore_ssh_fails) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/pokybuild/yocto-worker/qemux86/build/layers/openembedded-core/meta/lib/oeqa/core/target/ssh.py", line 70, in _run raise AssertionError("ssh exited with status '255' for command " "'%s': this is likely an SSH failure\n%s" % (command, output)) AssertionError: ssh exited with status '255' for command '['ssh', '-l', 'root', '-o', 'ServerAliveCountMax=2', '-o', 'ServerAliveInterval=30', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '192.168.7.6', 'export PATH=/usr/sbin:/sbin:/usr/bin:/bin; ps -ef']': this is likely an SSH failure ssh: connect to host 192.168.7.6 port 22: No route to host
qemux86 fedora43-vk-1 master-next completed at 2025-11-20 08:58:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/2731/steps/15/logs/stdio Richard extracted the logs here: https://autobuilder.yocto.io/pub/failed-builds-data/qemux86-f43/
I'm able to reproduce this intermittently on the fedora43-vk-1 worker - where "intermittently" means: - Logged in as pokybuild user - Using local.conf closely matching qemux86 builder config (see below) - Master branch checked out for oe-core, bitbake, meta-yocto - I'm running `bitbake core-image-minimal bitbake core-image-sato bitbake core-image-sato-sdk` and then `bitbake core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage` - The second command fails for at least one of the images being tested - I've seen core-image-sato, core-image-sato-sdk (or both) fail.
Note that occasionally they all still pass... makes me think there's a race condition for accessing tap devices or something.
I've done 20 runs on fedora43-vk-1 and had five of those runs report at least one failure for an image. Still not sure exactly why, but I've tried to save the logs (with -v passed to bitbake). I'll attach a passing and a failing example. Here is also the list of additions to local.conf, since I forgot to list that earlier: CONF_VERSION = "2" BB_HASHSERVE = 'wss://hashserv-eu.yoctoproject.org/ws' BUILDHISTORY_COMMIT = '1' BUILDHISTORY_DIR = "/srv/pokybuild/yocto-worker/qemux86/build/build/buildhistory" BUILDHISTORY_PUSH_REPO = "-f ssh://git@push.yoctoproject.org/poky-buildhistory poky/master-next/qemux86:poky/master-next/qemux86" DISTRO = "poky" ERROR_QA:remove = "version-going-backwards" EXTRA_IMAGE_FEATURES ?= 'allow-empty-password empty-root-password allow-root-login' IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME' IMAGE_FSTYPES:append = ' wic wic.bmap' IMAGE_INSTALL:append = ' ssh-pregen-hostkeys' INHERIT += "buildhistory" INHERIT += 'image-buildinfo' MACHINE = "qemux86" OE_FRAGMENTS += 'core/yocto-autobuilder/autobuilder core/yocto-autobuilder/autobuilder-resource-constraints' PACKAGE_CLASSES = "package_rpm package_deb package_ipk" SANITY_TESTED_DISTROS = '' SDKMACHINE = "x86_64" SSTATE_DIR ?= '/srv/autobuilder/valkyrie.yocto.io/pub/sstate'
Created attachment 5154 [details] qemux86 fedora 43 testimage verbose log (passing)
Created attachment 5155 [details] qemux86 fedora 43 testimage verbose log (failing, serialdebug)
Created attachment 5156 [details] qemux86 fedora 43 testimage verbose log (failing, ssh)
If I start repeatedly booting core-image-sato, occasionally I see errors like: ALSA: Restoring mixer settings... INIT: Entering runlevel: 5 Configuring network interfaces... Error: ipv4: Address already assigned. Starting system message bus: dbus. Starting Connection Manager Starting OpenBSD Secure Shell server: sshd done. Starting Xserver Starting rpcbind daemon... done. Starting bluetooth: bluetoothd. Starting syslogd/klogd: X.Org X Server 1.21.1.18 X Protocol Version 11, Revision 0 Current Operating System: Linux qemux86 6.16.11-yocto-standard #1 SMP PREEMPT_DYNAMIC Mon Oct 6 13:50:41 UTC 2025 i686 Kernel command line: root=/dev/vda rw ip=192.168.7.4::192.168.7.3:255.255.255.0::eth0:off:8.8.8.8 net.ifnames=0 console=ttyS0 console=ttyS1 oprofile.timer=1 tsc=reliable no_timer_check rcup [ 2.786240] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 [ 2.786895] Bluetooth: BNEP filters: protocol multicast [ 2.787585] Bluetooth: BNEP socket layer initialized Current version of pixman: 0.46.4 Before reporting problems, check http://wiki.x.org to make sure that you have the latest version. Markers: (--) probed, (**) from config file, (==) default setting, (++) from command line, (!!) notice, (II) informational, (WW) warning, (EE) error, (NI) not implemented, (??) unknown. (==) Log file: "/var/log/Xorg.0.log", Time: Thu Nov 27 01:55:11 2025 (==) Using config directory: "/etc/X11/xorg.conf.d" (==) Using system config directory "/usr/share/X11/xorg.conf.d" done * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon [ ok ] Starting Telephony daemon Starting crond: [ 2.807718] Oops: int3: 0000 [#1] SMP [ 2.807722] CPU: 1 UID: 998 PID: 534 Comm: dbus-daemon Not tainted 6.16.11-yocto-standard #1 PREEMPT(full) [ 2.807724] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 2.807725] EIP: __pi___x86_indirect_thunk_edx+0x6/0x20 [ 2.807730] Code: 00 00 00 00 e8 01 00 00 00 cc 89 0c 24 c3 90 2e 8d 74 26 00 2e 8d b4 26 00 00 00 00 2e 8d b4 26 00 00 00 00 e8 01 00 00 00 cc <89> 14 24 c3 90 2e 8d 74 26 00 2e 8d b4 266 [ 2.807731] EAX: 2c8347b4 EBX: c43b1f64 ECX: 00000000 EDX: 00000000 [ 2.807732] ESI: 00000193 EDI: c2790c00 EBP: c43b1f5c ESP: c43b1f40 [ 2.807733] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00000282 [ 2.807735] CR0: 80050033 CR2: af4966f4 CR3: 0217e000 CR4: 001506d0 [ 2.807736] Call Trace: [ 2.807738] __ia32_sys_clock_gettime+0x58/0xb0 [ 2.807741] ia32_sys_call+0x18e8/0x1e90 [ 2.807744] __do_fast_syscall_32+0x62/0x200 [ 2.807745] do_fast_syscall_32+0x29/0x60 [ 2.807746] do_SYSENTER_32+0x15/0x20 [ 2.807747] entry_SYSENTER_32+0x98/0xf9 [ 2.807749] EIP: 0xb7eea579 [ 2.807750] Code: b8 01 10 06 03 74 b4 01 10 07 03 74 b0 01 10 08 03 74 d8 01 00 00 00 00 00 00 00 00 00 00 00 00 00 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 776 [ 2.807751] EAX: ffffffda EBX: 00000001 ECX: bfced24c EDX: b7ee4000 [ 2.807751] ESI: b7e38e2c EDI: bfced2c0 EBP: bfced1f8 ESP: bfced198 [ 2.807752] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000293 [ 2.807753] Modules linked in: bnep [ 2.807755] ---[ end trace 0000000000000000 ]--- [ 2.807756] EIP: __pi___x86_indirect_thunk_edx+0x6/0x20 [ 2.807757] Code: 00 00 00 00 e8 01 00 00 00 cc 89 0c 24 c3 90 2e 8d 74 26 00 2e 8d b4 26 00 00 00 00 2e 8d b4 26 00 00 00 00 e8 01 00 00 00 cc <89> 14 24 c3 90 2e 8d 74 26 00 2e 8d b4 266 [ 2.807758] EAX: 2c8347b4 EBX: c43b1f64 ECX: 00000000 EDX: 00000000 [ 2.807758] ESI: 00000193 EDI: c2790c00 EBP: c43b1f5c ESP: c43b1f40 [ 2.807759] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00000282 [ 2.807760] CR0: 80050033 CR2: af4966f4 CR3: 0217e000 CR4: 001506d0 [ 2.807760] Kernel panic - not syncing: Fatal exception in interrupt [ 2.808375] Kernel Offset: disabled [ 2.830766] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
That's with `runqemu kvm nographic snapshot`.
Seen again after updating to latest master and making several boot attempts with core-image-sato-sdk: [ 2.808722] Bluetooth: BNEP socket layer initialized * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon [ ok ] Starting Telephony daemon Starting crond: OK Starting tcf-agent: OK pci id for fd 12: 1234:1111, driver (null) pci id for fd 13: 1234:1111, driver (null) kmsro: driver missing pci id for fd 13: 1234:1111, driver (null) kmsro: driver missing The XKEYBOARD keymap compiler (xkbcomp) reports: > Warning: Could not resolve keysym XF86RefreshRateToggle > Warning: Could not resolve keysym XF86Accessibility > Warning: Could not resolve keysym XF86DoNotDisturb Errors from xkbcomp are not fatal to the X server D-BUS per-session daemon address is: unix:path=/tmp/dbus-DIFg0ivKDe,guid=ca40107dddc2edf97b957dc469286c21 Using calibration data stored in /etc/pointercal.xinput Using calibration data stored in /etc/pointercal.xinput matchbox: Cant find a keycode for keysym 269025056 matchbox: ignoring key shortcut XF86Calendar=!$contacts matchbox: Cant find a keycode for keysym 2809 matchbox: ignoring key shortcut telephone=!$dates matchbox: Cant find a keycode for keysym 269025050 matchbox: ignoring key shortcut XF86Start=!matchbox-remote -desktop [settings daemon] Forking. run with -n to prevent fork [ 3.116097] Oops: int3: 0000 [#1] SMP [ 3.116101] CPU: 3 UID: 999 PID: 615 Comm: avahi-daemon Not tainted 6.16.11-yocto-standard #1 PREEMPT(full) [ 3.116103] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 3.116104] EIP: __pi___x86_indirect_thunk_edx+0x6/0x20 [ 3.116109] Code: 00 00 00 00 e8 01 00 00 00 cc 89 0c 24 c3 90 2e 8d 74 26 00 2e 8d b4 26 00 00 00 00 2e 8d b4 26 00 00 00 00 e8 01 00 00 00 cc <89> 14 24 c3 90 2e 8d 74 26 00 2e 8d b4 26 00 00 00 00 2e 8d b4 26 [ 3.116110] EAX: 17b674f1 EBX: bfb8bd40 ECX: 00000000 EDX: 00000000 [ 3.116111] ESI: 00000193 EDI: c25e5400 EBP: c2365f5c ESP: c2365f5c [ 3.116112] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00000286 [ 3.116115] CR0: 80050033 CR2: 00490e00 CR3: 04671000 CR4: 001506d0 [ 3.116115] Call Trace: [ 3.116117] __ia32_sys_clock_gettime+0x58/0xb0 [ 3.116120] ia32_sys_call+0x18e8/0x1e90 [ 3.116122] __do_fast_syscall_32+0x62/0x200 [ 3.116123] do_fast_syscall_32+0x29/0x60 [ 3.116124] do_SYSENTER_32+0x15/0x20 [ 3.116125] entry_SYSENTER_32+0x98/0xf9 [ 3.116127] EIP: 0xb7f51579 [ 3.116128] Code: b8 01 10 06 03 74 b4 01 10 07 03 74 b0 01 10 08 03 74 d8 01 00 00 00 00 00 00 00 00 00 00 00 00 00 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 8d 76 [ 3.116129] EAX: ffffffda EBX: 00000000 ECX: bfb8bd40 EDX: b7f4b000 [ 3.116130] ESI: b7e33e2c EDI: 0043ce7c EBP: bfb8bce8 ESP: bfb8bc88 [ 3.116130] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000293 [ 3.116132] Modules linked in: bnep [ 3.116134] ---[ end trace 0000000000000000 ]--- [ 3.116135] EIP: __pi___x86_indirect_thunk_edx+0x6/0x20 [ 3.116136] Code: 00 00 00 00 e8 01 00 00 00 cc 89 0c 24 c3 90 2e 8d 74 26 00 2e 8d b4 26 00 00 00 00 2e 8d b4 26 00 00 00 00 e8 01 00 00 00 cc <89> 14 24 c3 90 2e 8d 74 26 00 2e 8d b4 26 00 00 00 00 2e 8d b4 26 [ 3.116137] EAX: 17b674f1 EBX: bfb8bd40 ECX: 00000000 EDX: 00000000 [ 3.116137] ESI: 00000193 EDI: c25e5400 EBP: c2365f5c ESP: c2365f5c [ 3.116138] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00000286 [ 3.116139] CR0: 80050033 CR2: 00490e00 CR3: 04671000 CR4: 001506d0 [ 3.116139] Kernel panic - not syncing: Fatal exception in interrupt [ 3.116670] Kernel Offset: disabled [ 3.139535] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
Once the fedora fix for sstate contamination is in (#16087), we should retest this, as qemu has meanwhile been upgraded to 10.2.0 (the latest release as of now).
Trevor has been trying but so far has not be able to reproduce the problem on the YP F43 AB.
An urgent issue but limited to Fedora-43 which has been removed from the YP AB so it won't block 6.0-M2. Moving to M3. Trevor may not get time to work in the bug so if you're interested, please check with him. - YP bug triage team
Created attachment 5187 [details] core-image-sato testimage log 1 - short
It seems that most of the logs I've created in testing are too big to upload here. They're available on fedora43-vk-1 under /srv/pokybuild/tgamblin-qemux86/openembedded-core/build/testimage_logs. Note that they're varied in size - some ~20KB and some 1300KB+. In the former case, it seems that only one ptest (e.g. acl) runs and then the VM quits. I don't see an obvious error there suggesting the VM was told to shut down early. The longer files just seem to include the full testimage run, but I haven't taken a close look at all of them yet. An important point is that the same sort of SSH failure mentioned in the first comment of this bug shows up regularly: NOTE: Executing Tasks NOTE: Running task 1464 of 1464 (/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/recipes-sato/images/core-image-sato.bb:do_testimage) NOTE: recipe core-image-sato-1.0-r0: task do_testimage: Started QMP Available for connection at /srv/pokybuild/tgamblin-qemux86/openembedded-core/build/tmp/.y5_6ecsg QMP connected to QEMU at 02/19/26 17:20:50 and took 2.54 seconds QMP released QEMU at 02/19/26 17:20:50 and took 0.02 seconds from connect Traceback (most recent call last): File "/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) File "/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) File "/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) File "/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail self.do_ptestrunner() ~~~~~~~~~~~~~~~~~~~^^ File "/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/lib/oeqa/runtime/cases/ptest.py", line 62, in do_ptestrunner status, output = self.target.run('ptest-runner -t 450 -d \"{}\"'.format(' '.join(ptest_dirs)), 0) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/lib/oeqa/core/target/ssh.py", line 98, in run status, output = self._run(sshCmd, processTimeout, ignore_status, raw, ignore_ssh_fails) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/pokybuild/tgamblin-qemux86/openembedded-core/meta/lib/oeqa/core/target/ssh.py", line 70, in _run raise AssertionError("ssh exited with status '255' for command " "'%s': this is likely an SSH failure\n%s" % (command, output)) AssertionError: ssh exited with status '255' for command '['ssh', '-l', 'root', '-o', 'ServerAliveCountMax=2', '-o', 'ServerAliveInterval=30', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=E RROR', '192.168.7.2', 'export PATH=/usr/sbin:/sbin:/usr/bin:/bin; ptest-runner -t 450 -d "/usr/lib"']': this is likely an SSH failure START: ptest-runner 2026-02-19T17:20 BEGIN: /usr/lib/acl/ptest But as you can see at the end, it does eventually connect and start the ptest. However, in the last log (16), QEMU seems to have hanged. It's not clear to me if it eventually would've started or not, but I wanted to post an update here with what I'd found.
qemux86 reliably fails on fedora43-vk-1 (I ran four builds recently, and they all showed the kernel panic): https://autobuilder.yoctoproject.org/valkyrie/#/workers/65 (note that qemux86-64 does not fail on that worker) At the same time, qemux86 reliably succeeds on fedora43-vk-2: https://autobuilder.yoctoproject.org/valkyrie/#/workers/66 There's a notable difference between CPUs on these workers (/proc/cpuinfo): fedora43-vk-1: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 198 model name : Intel(R) Core(TM) Ultra 7 265 fedora43-vk-2: processor : 0 vendor_id : AuthenticAMD cpu family : 25 model : 17 model name : AMD EPYC 9454P 48-Core Processor It's also notable that the kernel crash is always happening here: [ 2.807736] Call Trace: [ 2.807738] __ia32_sys_clock_gettime+0x58/0xb0
From weekly community discussion, reiterating Alex's latest comment - this may be unique to the fedora43-vk-1 worker. It would be useful information to re-enable the fedora43-vk-2 worker and see if we ever see the failure there.
I've re-enabled the vk-2 worker :) For reference, here's the disassembly of the crash in comment 8: All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: e8 01 00 00 00 call 0xa 9: cc int3 a: 89 0c 24 mov %ecx,(%rsp) d: c3 ret e: 90 nop f: 2e 8d 74 26 00 cs lea 0x0(%rsi,%riz,1),%esi 14: 2e 8d b4 26 00 00 00 cs lea 0x0(%rsi,%riz,1),%esi 1b: 00 1c: 2e 8d b4 26 00 00 00 cs lea 0x0(%rsi,%riz,1),%esi 23: 00 24: e8 01 00 00 00 call 0x2a 29: cc int3 2a:* 89 14 24 mov %edx,(%rsp) <-- trapping instruction 2d: c3 ret 2e: 90 nop 2f: 2e 8d 74 26 00 cs lea 0x0(%rsi,%riz,1),%esi 34: 2e 8d b4 26 00 00 00 cs lea 0x0(%rsi,%riz,1),%esi 3b: 00 3c: 2e cs 3d: 8d .byte 0x8d 3e: b4 26 mov $0x26,%ah
Some further hints from resident kernel developers at LX: you run a 32bit host and do clock_gettime(). you should handle this with vdso but you don't, you do a full syscall so either it is not a "standard" clock such CLOCK_MONOTONIC or something else is broken could you boot with "mitigations=off" ? then backtrace might be simpler the kernel has scripts/decode_stacktrace.sh so you can throw in there __ia32_sys_clock_gettime+0x58/0xb0 if you have the matching vmlinux to the the matching C code
I don't have much to add yet since the last comment, other than that it seems with mitigations=off, the images I build for testing seem to take much longer than usual to boot/run tests...
Created attachment 5190 [details] core-image-sato qemux86 bootlog 12 march 2026 New failure when running testimage on qemux86 on our fedora43-vk-1 worker. Looks different than the gettime one we've seen before.
Created attachment 5191 [details] qemux86 testimage verbose log 12 march 2026 Here is the full testimage log to go with the qemu boot log in the previous comment, generated with: bitbake -vvv -c testimage core-image-sato 2>&1 | tee testimage.log
(In reply to Trevor Gamblin from comment #21) > Created attachment 5190 [details] > core-image-sato qemux86 bootlog 12 march 2026 > > New failure when running testimage on qemux86 on our fedora43-vk-1 worker. > Looks different than the gettime one we've seen before. I'm not sure where is the difference? Bootlog shows the same: [ 2.775584] Call Trace: [ 2.775586] __ia32_sys_clock_gettime+0x58/0xb0 [ 2.775589] ia32_sys_call+0x9f1/0x1eb0
Yes, it is the same - I thought maybe there was a BNEP bug but that doesn't seem to be the real issue.
I had a hunch that since this CPU has two different sets of cores, maybe pinning the testimage/bitbake/QEMU process to certain ones might indicate whether it only happens when we rely on them. I've tried with taskset 0-7 (assuming that the first 8 enumerated cores out of 20 are the performance ones) but I'm not sure yet, will have to test several more times. I'm having a hard time getting detailed per-core info from the host system to tell for sure. This was about the best info: (buildbot-venv) [pokybuild@fedora43-vk-1 build]$ lscpu --extended CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ MHZ 0 0 0 0 0:0:0:0 yes 3700.0000 800.0000 800.0000 1 0 0 1 4:4:1:0 yes 3700.0000 800.0000 1374.0900 2 0 0 2 16:16:4:0 yes 3700.0000 800.0000 800.0000 3 0 0 3 20:20:5:0 yes 3800.0000 800.0000 800.0000 4 0 0 4 24:24:6:0 yes 3800.0000 800.0000 800.0000 5 0 0 5 28:28:7:0 yes 3700.0000 800.0000 800.0000 6 0 0 6 40:40:10:0 yes 3700.0000 800.0000 800.0000 7 0 0 7 44:44:11:0 yes 3700.0000 800.0000 800.0000 8 0 0 8 3:0 yes 3300.0000 800.0000 1235.1429 9 0 0 9 26:26:3:0 yes 3300.0000 800.0000 800.0000 10 0 0 10 3:0 yes 3300.0000 800.0000 800.0000 11 0 0 11 30:30:3:0 yes 3300.0000 800.0000 2119.7581 12 0 0 12 64:64:8:0 yes 4600.0000 800.0000 2481.1941 13 0 0 13 66:66:8:0 yes 4600.0000 800.0000 3863.6230 14 0 0 14 68:68:8:0 yes 3300.0000 800.0000 3671.1531 15 0 0 15 70:70:8:0 yes 3300.0000 800.0000 800.0000 16 0 0 16 72:72:9:0 yes 3300.0000 800.0000 800.0000 17 0 0 17 74:74:9:0 yes 3300.0000 800.0000 3486.3279 18 0 0 18 76:76:9:0 yes 3300.0000 800.0000 800.0000 19 0 0 19 78:78:9:0 yes 3300.0000 800.0000 800.0000 The cache data might be helpful, but I think the min/max CPU frequency is misleading, at least compared to the CPU's info page: https://www.intel.com/content/www/us/en/products/sku/241068/intel-core-ultra-7-processor-265-30m-cache-up-to-5-30-ghz/specifications.html
How to reproduce the crash without too much pain: bitbake core-image-sato-sdk (for qemux86) runqemu kvm nographic snapshot (the following commands inside qemu) wget https://downloads.yoctoproject.org/mirror/sources/cpio-2.15.tar.gz tar xvf cpio-2.15.tar.gz && cd cpio-2.15 while true; do ./configure; make; done Rather sooner than later the above configure/make loop will trigger the kernel crash (usually after a couple minutes or so). It's always in something timing related, e.g. atime_needs_update+0xb1/0x170 or file_update_time_flags+0x3e/0x170 or clock_gettime syscall.
I have narrowed this to retpoline mitigation for spectre v2 cpu vulnerability. The crash refers to __pi___x86_indirect_thunk_edx which is a part of that mitigation. Modern CPUs have better ways to handle the vulnerability, but we're emulating an Ivy Bridge chip from 2011 (via -cpu IvyBridge paramater to qemu), and so guest kernel auto-selects that mitigation, via spectre_v2=auto kernel parameter being the default [1][2]. [1] https://docs.kernel.org/admin-guide/hw-vuln/spectre.html [2] https://docs.kernel.org/admin-guide/kernel-parameters.html How to see what mitigation is in use: root@qemux86:~# cat /sys/devices/system/cpu/vulnerabilities/spectre_v2 Mitigation: Retpolines; STIBP: disabled; RSB filling; PBRSB-eIBRS: Not affected; BHI: Retpoline Then I ran a few experiments: 1. kernel cmdline: spectre_v2=off ---> crash disappears 2. qemu cmdline: -cpu host ---> crash disappears, as kernel auto-selection chooses: Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; PBRSB-eIBRS: Not affected; BHI: BHI_DIS_S which is same as on the fedora 43 host kernel. 3. qemu cmdline: -cpu host, kernel cmdline: spectre_v2=retpoline ---> crash re-appears Possible next steps: - confirm that the issue is indeed 32 bit specific, and does not occur on 64 bit kernels - report the issue upstream (possibly after confirming it with tip of linus tree). One could argue that retpoline is a 'dated' technique for older CPUs that do not have 'ibrs_enhanced' feature (seen in /proc/cpuinfo), and no one's going to use it on Arrow Lake generation, especially if it happens only in 32 bit kernels, but equally there is a bug somewhere, as it does work on other CPUs, and that bug might manifest elsewhere as well. I would need a bit of help with 'report upstream' bit as I have no idea where to start with that.
As discussed on IRC, plugging the details of the bug that we know (so far) into Claude led it to suggest a variety of things (including the attempt to pin the process to a subset of the host cores as I mentioned above, without success). One of them was to try booting with the cpu configured as such: -cpu host,-tsc-deadline,-kvmclock,-kvm_pv_unhalt I didn't want to apply the whole configuration change at once for obivous reasons. Instead (with spectre_v2=retpoline added to the kernel command line in all cases): -cpu host,kvmclock=off - no failure on the cpio test loop for ~30 minutes -cpu host,kvmclock=on - failure after ~7 minutes -cpu host,kvmclock=on,kvm_pv_unhalt=off - failure after ~6 minutes -cpu host,kvmclock=on,kvm_pv_unhalt=on - failure after ~15 minutes Posting this now for utility, but I'll have a follow-up with some tsc-deadline settings added and a second (longer) run at the "passing" case.
re: "I would need a bit of help with 'report upstream' bit as I have no idea where to start with that." My $0.02 advice would be to start with people like tglx and peterz as per kernel's MAINTAINERS file, plus cc'd to an appropriate list from the same file. Be concise and don't send reams of data - just a brief summary of the issue and an indication that you are looking for advice on next steps. If you prefer a more informal approach -- these guys all hang out on #linux-rt on OFTC -- don't let the "RT" part deter you, as it has become a kind of catch-all for that audience. There is also #x86 on OFTC, but that tends to get more into the weeds on specific details.
I ran the loop for 98 minutes with this: -cpu host,kvmclock=off spectre_v2=retpoline added to the kernel command line And the loop was still running. We may at least be able to consider it a workaround for qemux86 (when KVM is enabled) for the time being, and it may be useful data to report as part of the broader triage.
(In reply to Trevor Gamblin from comment #30) > I ran the loop for 98 minutes with this: > > -cpu host,kvmclock=off > spectre_v2=retpoline added to the kernel command line > > And the loop was still running. We may at least be able to consider it a > workaround for qemux86 (when KVM is enabled) for the time being, and it may > be useful data to report as part of the broader triage. Yes, only the workaround wouldn't change the cpu type currently used in qemux86, and just use '-cpu IvyBridge,kvmclock=off'. '-cpu host' will make the guest environment non-reproducible as everyone will see their own host cpu in it. I tested a 64 bit kernel/image; the issue doesn't seem to trigger there.
We should still try to look into what kvmclock kernel driver does (that other clock sources do not) that trips over retpoline. 'dmesg|grep clock' shows a list of clock candidates, and the one that kernel picks. (it's of course different with kvmclock=off)
Definitely. I spent a bit of time trying to learn about the underlying mitigation and what it does, eventually finding this article: https://blogs.oracle.com/linux/understanding-spectre-v2-mitigations-on-x86
While discussing this on the patch review call I did a `git log` on the Linux kernel and searched for `kvmclock`. The first hit looks relevant: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5bb9ac1865123356337a389af935d3913ee917ed. KVM: x86: Return "unsupported" instead of "invalid" on access to unsupported PV MSR Return KVM_MSR_RET_UNSUPPORTED instead of '1' (which for all intents and purposes means "invalid") when rejecting accesses to KVM PV MSRs to adhere to KVM's ABI of allowing host reads and writes of '0' to MSRs that are advertised to userspace via KVM_GET_MSR_INDEX_LIST, even if the vCPU model doesn't support the MSR. E.g. running a QEMU VM with -cpu host,-kvmclock,kvm-pv-enforce-cpuid yields: qemu: error: failed to set MSR 0x12 to 0x0 qemu: target/i386/kvm/kvm.c:3301: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed. There may also be other relevant changes to kvm. I wonder if this is something that is fixed on mainline but not on whatever distro/stable kernel we've tested on.
To test the guest machine with the commit Paul mentioned, I set PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev" and rebuilt: root@qemux86:~# uname -a Linux qemux86 7.0.0-rc4-yoctodev-standard #1 SMP PREEMPT_DYNAMIC Tue Mar 24 18:04:57 UTC 2026 i686 GNU/Linux root@qemux86:~# But I still see the same panic when using the reproducer. As a side note, maybe it's just the way the linux-yocto-dev log is structured, but I don't see that particular commit (5bb9ac1865123356337a389af935d3913ee917ed) associated with any tags in linux-yocto-dev, but I do see it in my clone of the upstream Linux kernel tree: tgamblin@megalith ~/workspace/git/linux (master)$ git tag --contains 5bb9ac1865123356337a389af935d3913ee917ed v7.0-rc1 v7.0-rc2 v7.0-rc3 v7.0-rc4 v7.0-rc5
Note also https://bugzilla.yoctoproject.org/show_bug.cgi?id=16219 - the workaround patch to set kvmclock=off for qemux86 kvm seems to be causing other issues.
Running with mitgations=off: root@qemux86:~# cat /proc/cmdline root=/dev/vda rw ip=192.168.7.2::192.168.7.1:255.255.255.0::eth0:off:8.8.8.8 net.ifnames=0 console=ttyS0 console=ttyS1 oprofile.timer=1 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 mitigations=off swiotlb=0 The reproducer that Alex provided has been running for over an hour, with no panic: [tgamblin@fedora43-vk-1 ~]$ ps aux | grep qemu pokybui+ 39084 0.0 0.0 26048 20720 pts/3 S+ 12:39 0:00 python3 /srv/pokybuild/tgamblin-qemux86/openembedded-core/scripts/runqemu kvm nographic snapshot pokybui+ 39124 80.7 1.0 6165064 688368 pts/3 Sl+ 12:39 106:18 /srv/pokybuild/tgamblin-qemux86/openembedded-core/build/tmp/work/x86_64-linux/qemu-helper-native/1.0/recipe-sysroot-native/usr/bin/qemu-system-i386 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:02 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -drive file=/srv/pokybuild/tgamblin-qemux86/openembedded-core/build/tmp/deploy/images/qemux86/core-image-sato-sdk-qemux86.rootfs-20260401123623.ext4,if=virtio,format=raw -usb -device usb-tablet -usb -device usb-kbd -cpu IvyBridge -machine q35,i8042=off -smp 4 -enable-kvm -m 1024 -snapshot -serial mon:stdio -serial null -nographic -kernel /srv/pokybuild/tgamblin-qemux86/openembedded-core/build/tmp/deploy/images/qemux86/bzImage -append root=/dev/vda rw ip=192.168.7.2::192.168.7.1:255.255.255.0::eth0:off:8.8.8.8 net.ifnames=0 console=ttyS0 console=ttyS1 oprofile.timer=1 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 mitigations=off swiotlb=0 tgamblin 40994 0.0 0.0 6560 2360 pts/4 S+ 14:50 0:00 grep --color=auto qemu
Limited to one intel processor type so lower to Medium+ for 6.1. We really need more info about who/what build hosts are affected. Eric CCed. Eric, I asked internally if we could get our test team to see if this error happens on only one type of intel host of it it's more broadly applicable. If we can help, that would be useful.
After much debugging we now suspect the f43 worker needed the cpu mircocode updating. That has been done and the tests which could reproduce the issue no longer do. We can therefore close this, reopening if it happens again.