Bug 15526

Summary: kernel serial interrupt issues with 6.6.32 kernel
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: kernelAssignee: Bruce Ashfield <bruce.ashfield>
Status: RESOLVED OBSOLETE QA Contact:
Severity: normal    
Priority: Medium+ CC: jon.mason, paulg, randy.macleod, ross.burton
Version: 0.0.0   
Target Milestone: 6.0 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
qemu boot log
none
Second serial port log
none
test image log
none
qemux86 boot log on AMD 5950 CPU none

Description Richard Purdie 2024-06-25 12:14:58 UTC
Created attachment 5053 [details]
qemu boot log

https://valkyrie.yoctoproject.org/#/builders/30/builds/48

There is a rather simple looking:

WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into serial console as root using blank password
WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
<<< run_serial(): command timed out after 120 seconds without output >>>

however digging into the failed build shows something nastier going on. See the attached log:


qemux86 login: [    3.722856] __common_interrupt: 2.37 No irq handler for vector
[    3.938777] __common_interrupt: 2.37 No irq handler for vector
[    4.154841] __common_interrupt: 2.37 No irq handler for vector
[    4.370858] __common_interrupt: 2.37 No irq handler for vector
[    4.586744] __common_interrupt: 2.37 No irq handler for vector
[    4.802839] __common_interrupt: 2.37 No irq handler for vector

This is a 6.6.32 kernel on a fedora 40 worker using KVM acceleration for the image  (core-image-sato-sdk)
Comment 1 Richard Purdie 2024-06-25 12:15:19 UTC
Created attachment 5054 [details]
Second serial port log
Comment 2 Richard Purdie 2024-06-25 12:17:40 UTC
Created attachment 5055 [details]
test image log
Comment 3 Richard Purdie 2024-06-25 12:21:07 UTC
Also probably related earlier in the log:

[    0.206386] smpboot: x86: Booting SMP configuration:
[    0.206588] .... node  #0, CPUs:      #1
[    0.084519] ------------[ cut here ]------------
[    0.084519] WARNING: This combination of AMD processors is not suitable for SMP.
[    0.084519] WARNING: CPU: 1 PID: 0 at /arch/x86/kernel/cpu/amd.c:341 init_amd+0xaee/0xbcc
[    0.084519] Modules linked in:
[    0.084519] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.6.32-yocto-standard #1
[    0.084519] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[    0.084519] EIP: init_amd+0xaee/0xbcc
[    0.084519] Code: ff 6a 00 8b 55 c4 b8 1b 00 01 c0 8b 4d c8 e8 e9 82 67 00 58 e9 79 fc ff ff c6 05 56 0b a6 d7 01 68 cc 69 7d d7 e8 4e ec 02 00 <0f> 0b 5e e9 ff fb ff ff 51 b8 21 10 01 c0 89 da 89 f1 e8 bb 82 67
[    0.084519] EAX: 00000044 EBX: 20000000 ECX: 00000000 EDX: d796c18c
[    0.084519] ESI: ef7fb81d EDI: ef7fb7a0 EBP: c11b5f54 ESP: c11b5f10
[    0.084519] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210082
[    0.084519] CR0: 80050033 CR2: 00000000 CR3: 17bbb000 CR4: 00150690
[    0.084519] Call Trace:
[    0.084519]  ? show_regs+0x4f/0x58
[    0.084519]  ? init_amd+0xaee/0xbcc
[    0.084519]  ? __warn+0x70/0x138
[    0.084519]  ? init_amd+0xaee/0xbcc
[    0.084519]  ? report_bug+0x160/0x184
[    0.084519]  ? exc_overflow+0x38/0x38
[    0.084519]  ? handle_bug+0x2d/0x58
[    0.084519]  ? exc_invalid_op+0x18/0x54
[    0.084519]  ? handle_exception+0x133/0x133
[    0.084519]  ? exc_overflow+0x38/0x38
[    0.084519]  ? init_amd+0xaee/0xbcc
[    0.084519]  ? exc_overflow+0x38/0x38
[    0.084519]  ? init_amd+0xaee/0xbcc
[    0.084519]  identify_cpu+0x150/0x7c4
[    0.084519]  identify_secondary_cpu+0x17/0xf0
[    0.084519]  smp_store_cpu_info+0x3c/0x48
[    0.084519]  start_secondary+0x68/0xf4
[    0.084519]  startup_32_smp+0x156/0x158
[    0.084519] ---[ end trace 0000000000000000 ]---
[    0.084519] Disabling lock debugging due to kernel taint
[    0.237606]  #2 #3
[    0.260084] smp: Brought up 1 node, 4 CPUs
[    0.261077] smpboot: Max logical packages: 1
[    0.261583] smpboot: Total of 4 processors activated (33537.02 BogoMIPS)
Comment 4 Richard Purdie 2024-06-25 12:24:08 UTC
I'd note that this is the new cluster and there are AMD systems in there. /proc/cpuinfo from that f40 worker:

processor	: 31
vendor_id	: AuthenticAMD
cpu family	: 25
model		: 97
model name	: AMD Ryzen 9 7950X3D 16-Core Processor
stepping	: 2
microcode	: 0xa601206
cpu MHz		: 400.000
cache size	: 1024 KB
physical id	: 0
siblings	: 32
core id		: 15
cpu cores	: 16
apicid		: 31
initial apicid	: 31
fpu		: yes
fpu_exception	: yes
cpuid level	: 16
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d
bugs		: sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
bogomips	: 8383.88
TLB size	: 3584 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14
Comment 5 Bruce Ashfield 2024-06-25 12:58:44 UTC
I'm starting a new build on a different builder to see if I can reproduce this.

I didn't see anything like it in my sanity testing.
Comment 6 Richard Purdie 2024-06-25 14:19:25 UTC
I added a check for the kernel traceback in parselogs and executing qemux86 on any new valkyrie cluster worker gives that same error:

https://valkyrie.yoctoproject.org/#/builders/30/builds/50 - fedora40
https://valkyrie.yoctoproject.org/#/builders/30/builds/51 - ubuntu2204
https://valkyrie.yoctoproject.org/#/builders/30/builds/52 - debian12

I suspect the interrupt issue is a separate one to the SMP KVM 32 bit traceback. qemux8664 doesn't do this.
Comment 7 Richard Purdie 2024-06-25 14:26:16 UTC
Just to be clear, these failures were on the new cluster which has AMD CPUs rather than Intel ones. I'm not sure why that breaks 32 bit KVM.
Comment 8 Ross Burton 2024-06-25 14:46:54 UTC
Just to confirm: is this limited to qemux86 as it depends on KVM? Does it happen on qemux86-64?
Comment 9 Richard Purdie 2024-06-25 15:01:11 UTC
(In reply to Ross Burton from comment #8)
> Just to confirm: is this limited to qemux86 as it depends on KVM?

I believe so.

> Does it happen on qemux86-64?

Not that I can see.
Comment 10 Jon Mason 2024-06-25 22:21:13 UTC
Created attachment 5056 [details]
qemux86 boot log on AMD 5950 CPU

Datapoint, qemux86 is booting to prompt on my AMD 5950 CPU on Debian 11.
NOTE: is is throwing an error about not enabling SMP in the kernel log.

Running poky commit ac40cb5ee24a8ef44b941c4b9bbf6eec6f61d13c
Comment 11 Richard Purdie 2024-06-26 14:33:51 UTC
arch/x86/kernel/cpu/amd.c: init_amd_k7(struct cpuinfo_x86 *c)

	/*
	 * Don't taint if we are running SMP kernel on a single non-MP
	 * approved Athlon
	 */
	WARN_ONCE(1, "WARNING: This combination of AMD"
		" processors is not suitable for SMP.\n");
	add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_NOW_UNRELIABLE);

https://lists.gnu.org/archive/html/qemu-devel/2010-03/msg01428.html

https://lkml.org/lkml/2010/3/30/397

so the error is harmless and can be ignored. We just need to find out how to disable it.
Comment 12 Randy MacLeod 2024-06-27 14:41:34 UTC
We are about to fix the trackback issue but the underlying IRQ issue remains.
Comment 13 Randy MacLeod 2025-04-24 18:54:45 UTC
Bulk move as requesting during YP BB meeting.
Comment 14 Randy MacLeod 2026-03-05 16:15:00 UTC
Kernel version has changed too much so resolving as obsolete.

Open an new issue if seen again.