We recently closed a bug where we saw intermittent, but reproducible, failures with qemux86 kvm systems: https://bugzilla.yoctoproject.org/show_bug.cgi?id=16074 This was originally thought to be a Fedora- or host-specific issue, but eventually it was determined to be a problem with the microcode version running on the AB. We should implement some sort of check to test the versions we're running and see if we need updates.
parallel-ssh -h ~/yocto-io.hosts -o cpu "cat /proc/cpuinfo | sed -nr '/^$/q; /family|model\s*:|stepping/p' | perl -pe 's{\d+}{sprintf \"0x%02x\", $&}e'" $ find cpu ! -empty -type f -exec md5sum {} + | sort 00ae64d9ed3b17d61c739f6f87af62be cpu/stream9-vk-1.yocto.io 3571cc10fd6ddc211c9629ac652af372 cpu/fedora44-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/alma10-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/alma8-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/alma8-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/alma9-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian11-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian11-vk-3.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-3.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-4.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-5.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-6.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-7.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-8.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian12-vk-9.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian13-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/debian13-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/fedora43-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/opensuse156-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/opensuse160-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/opensuse160-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/rocky8-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/rocky9-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2204-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2204-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2204-vk-3.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2204-vk-4.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2404-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2404-vk-2.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2404-vk-3.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2504-vk-1.yocto.io 3b8e5780e4498cb22b104710fa4da299 cpu/ubuntu2510-vk-1.yocto.io 8fc74545d3eb3f5762ae4c878532582c cpu/perf-alma8-vk.yocto.io 8fc74545d3eb3f5762ae4c878532582c cpu/perf-debian12-vk.yocto.io 8fc74545d3eb3f5762ae4c878532582c cpu/ubuntu2604-vk-1.yocto.io 8fc74545d3eb3f5762ae4c878532582c cpu/ubuntu2604-vk-2.yocto.io so there are 4 different CPU types on different x86 workers. 1 of 0x17 0x31 0x02 1 of 0x06 0xc6 0x00 32 of 0x19 0x11 0x01 4 of 0x19 0x61 0x02 https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amd-ucode/README says the most common can either be runtime upgraded to: Family=0x19 Model=0x11 Stepping=0x01: Patch=0x0a101148 Length=5568 bytes or if the minimum bios firmware is right: Family=0x19 Model=0x11 Stepping=0x01: Patch=0x0a101158 Length=5568 bytes Minimum base ucode version for loading: 0x0a101153 I got dmesg permissions errors for my own user or pokybuild on some workers. ubuntu2404-vk-1.yocto.io:[ 1.219670] microcode: Current revision: 0x0a101148 ubuntu2404-vk-2.yocto.io:[ 1.211318] microcode: Current revision: 0x0a101148 ubuntu2404-vk-3.yocto.io:[ 1.218377] microcode: Current revision: 0x0a101148 ubuntu2510-vk-1.yocto.io:[ 1.080236] microcode: Current revision: 0x0a101148 debian13-vk-1.yocto.io:[ 0.826498] microcode: Current revision: 0x0a101148 debian13-vk-2.yocto.io:[ 0.807246] microcode: Current revision: 0x0a101148 fedora43-vk-2.yocto.io:[ 1.041429] microcode: Current revision: 0x0a101148 fedora44-vk-1.yocto.io:[ 0.634788] microcode: Current revision: 0x0000011b opensuse156-vk-1.yocto.io:[ 1.061631] microcode: Current revision: 0x0a101148 debian11-vk-1.yocto.io:[ 1.330211] microcode: microcode updated early to new patch_level=0x0a101148 debian11-vk-3.yocto.io:[ 1.330145] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-1.yocto.io:[ 1.059011] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-2.yocto.io:[ 1.068483] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-3.yocto.io:[ 1.058041] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-4.yocto.io:[ 1.057809] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-5.yocto.io:[ 1.062869] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-6.yocto.io:[ 1.063429] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-7.yocto.io:[ 1.064893] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-8.yocto.io:[ 1.058509] microcode: microcode updated early to new patch_level=0x0a101148 debian12-vk-9.yocto.io:[ 1.064607] microcode: microcode updated early to new patch_level=0x0a101148 The were some which showed issues cpu1/alma10-vk-1.yocto.io:[ 0.830941] microcode: Current revision: 0x0a101144 cpu1/alma9-vk-2.yocto.io:[ 1.005427] microcode: Current revision: 0x0a101144 cpu1/rocky9-vk-2.yocto.io:[ 0.982961] microcode: Current revision: 0x0a101144 with some errors trying to upgrade to the wrong version given the low base firmware version: alma8-vk-1.yocto.io:[ 1.582169] microcode: CPU0: update failed for patch_level=0x0a101158 alma8-vk-2.yocto.io:[ 2.074576] microcode: CPU0: update failed for patch_level=0x0a101158 rocky8-vk-1.yocto.io:[ 2.123963] microcode: CPU0: update failed for patch_level=0x0a101158 So the main issues look to be alma/rocky/stream. I didn't check the smaller number of other system versions yet.