Now that we get a bit more details about SSH failures during tests, we can see a lot of them are failing because of test timeout. Let's track them, I hope we will be able to identify some test that have to be fixed. Typical trace: AssertionError: ssh exited with status '255' for command '['ssh', '-l', 'root', '-o', 'ServerAliveCountMax=2', '-o', 'ServerAliveInterval=30', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '192.168.7.6', 'export PATH=/usr/sbin:/sbin:/usr/bin:/bin; stap -v -p4 -m stap_hello --disable-cache -DSTP_NO_VERREL_CHECK -e \'probe oneshot { print("Hello, "); println("SystemTap!") }\'']': this is likely an SSH failure ... Process killed - no output for 900 seconds. Total running time: 910 seconds.
pkgman-non-rpm fedora40-vk-2 master-next completed at 2025-10-10 11:52:28+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2582/steps/22/logs/stdio qemux86-64-ltp debian12-vk-5 master completed at 2025-10-12 02:50:26+00:00 test_posix_groups https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2388/steps/13/logs/stdio pkgman-non-rpm fedora39-vk-1 master completed at 2025-10-13 07:49:43+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2590/steps/22/logs/stdio
pkgman-non-rpm debian12-vk-2 mathieu/master-next completed at 2025-10-14 08:12:24+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2602/steps/19/logs/stdio
pkgman-non-rpm fedora39-vk-1 mathieu/master-next completed at 2025-10-14 10:10:30+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2604/steps/22/logs/stdio pkgman-non-rpm debian11-vk-3 mathieu/master-next completed at 2025-10-15 19:36:28+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2616/steps/22/logs/stdio
About test_stap, I just added a commit in my branch, increasing the timeout to 1200. Let's see how it goes.
pkgman-non-rpm opensuse155-vk-1 master-next completed at 2025-10-16 00:28:59+00:00 https://autobuilder.yoctoproject.org/valkyrie#/builders/67/builds/2619/steps/20/logs/stdio pkgman-non-rpm fedora41-vk-1 mathieu/master-next completed at 2025-10-16 06:47:17+00:00 https://autobuilder.yoctoproject.org/valkyrie#/builders/67/builds/2622/steps/19/logs/stdio pkgman-non-rpm fedora40-vk-3 master completed at 2025-10-17 10:27:55+00:00 https://autobuilder.yoctoproject.org/valkyrie#/builders/67/builds/2630/steps/19/logs/stdio qemuarm64 debian11-vk-3 master-next completed at 2025-10-18 00:12:54+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie#/builders/36/builds/2608/steps/15/logs/stdio qemux86-64-ltp fedora40-vk-4 master completed at 2025-10-21 03:03:29+00:00 test_posix_groups https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2444/steps/13/logs/stdio
pkgman-non-rpm opensuse155-vk-1 mathieu/master-next completed at 2025-10-26 12:51:56+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2662/steps/20/logs/stdio This was in my branch, where the timeout was already increased from 900s to 1200s. I will increase it a bit more, but there is likely something else behind this. qemuarm64-alt rocky9-vk-2 mathieu/master-next completed at 2025-10-26 17:43:23+00:00 test_go_compile https://autobuilder.yoctoproject.org/valkyrie/#/builders/9/builds/2621/steps/14/logs/stdio
qemuriscv64 ubuntu2204-vk-4 master-next completed at 2025-10-30 18:18:36+00:00 test_go_module https://autobuilder.yoctoproject.org/valkyrie/#/builders/45/builds/665/steps/15/logs/stdio pkgman-non-rpm debian12-vk-1 master-next completed at 2025-10-31 15:08:59+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2689/steps/19/logs/stdio qemuarm64 debian12-vk-6 mathieu/master-next completed at 2025-11-02 10:11:43+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/2668/steps/14/logs/stdio
qemuarm64 debian12-vk-3 mathieu/master-next completed at 2025-11-19 10:14:55+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/2745/steps/14/logs/stdio
qemuarm ubuntu2404-vk-3 mathieu/master-next completed at 2025-11-23 22:46:09+00:00 test_go_compile https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/2793/steps/14/logs/stdio qemux86-64-ltp fedora43-vk-1 mathieu/master-next completed at 2025-11-23 09:09:31+00:00 test_ltp_groups https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2587/steps/13/logs/stdio
qemuarm64-alt ubuntu2404-vk-3 master completed at 2025-11-24 05:12:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/9/builds/2749/steps/15/logs/stdio Probably the VM was also dead here. qemux86-64-ltp debian12-vk-9 master-next completed at 2025-11-25 17:19:51+00:00 test_posix_groups https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2597/steps/13/logs/stdio qemuarm64 debian13-vk-2 master-next completed at 2025-11-25 23:54:04+00:00 test_go_compile / test_go_module / test_stap combo https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/2778/steps/15/logs/stdio qemux86-64-ltp alma8-vk-2 master-next completed at 2025-11-25 19:31:38+00:00 test_posix_groups https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2599/steps/14/logs/stdio pkgman-non-rpm debian11-vk-3 master completed at 2025-11-26 02:47:54+00:00 test_stap https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2806/steps/19/logs/stdio Richard captured some logs for that one: https://autobuilder.yocto.io/pub/failed-builds-data/stap-fail-oom/qemu_boot_log.20251126022342.2 Looks like it was an OOM. Machine memory was increased: https://git.openembedded.org/openembedded-core/commit/?id=6b022b441c0cc12075f651182d2a352b9bf99606 qemuarm debian13-vk-2 master completed at 2025-11-26 21:14:21+00:00 test_go_compile / test_go_module https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/2810/steps/15/logs/stdio
pkgman-non-rpm debian13-vk-2 mathieu/master-next completed at 2025-11-30 01:17:06+00:00 test_go_compile https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2830/steps/19/logs/stdio
pkgman-non-rpm rocky9-vk-1 master completed at 2025-12-04 09:56:48+00:00 test_go_compile https://autobuilder.yoctoproject.org/valkyrie/#/builders/67/builds/2847/steps/22/logs/stdio
Richard did increase core-image-sato-sdk VM memory size from 768MB to 1024M. https://git.openembedded.org/openembedded-core/commit/?id=6b022b441c0cc12075f651182d2a352b9bf99606 This should fix stap failures, as they were caused by OOM killer. The ltp ones (test_posix_groups) are only caused by OOM killer, but on core-image-sato. I will probably increase a bit the memory if it happens again. The test_go_compile failures are NOT related to any memory issue: I will try to increase a bit the timeout as a first idea.
I sent a patch to fix the test_go_compile / test_go_module issues and to improve a bit the logs. https://lists.openembedded.org/g/openembedded-core/message/228592 https://lists.openembedded.org/g/openembedded-core/message/228591
qemux86-64-ltp debian12-vk-2 mathieu/master-next completed at 2026-01-29 22:38:13+00:00 test_posix_groups https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2926/steps/13/logs/stdio
test_posix_groups qemux86-64-ltp fedora40-vk-2 master completed at 2026-02-05 03:16:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2951/steps/13/logs/stdio
Created attachment 5180 [details] qemu boot log QEMU boot log of the last failure. OOM killer did trigger, 1024M was not enough.
(In reply to Mathieu Dubois-Briand from comment #17) > Created attachment 5180 [details] > qemu boot log > > QEMU boot log of the last failure. OOM killer did trigger, 1024M was not > enough. Hum, I compared with a successful build, and actually the OOM also triggers in there. https://valkyrie.yocto.io/pub/non-release/20260205-52/testresults/qemux86-64-ltp/core-image-sato/ So probably my assumption that this is memory related is wrong. Test is a bit long, even on successful builds: ltp_compliance.LtpPosixTest.test_posix_groups: PASSED (1234.49s) Comparing test log output, we probably have regular messages, so we should not reach the 300 seconds timeout. I will still try to increase it a bit. Adding a patch in my branch, I will send it after a while if the situation is better.
Mathieu will open a new issue and likely close this one since there have been many problems intermixed in the comments here.
Closing, as there is too many different issues in there. Some issues were fixed. ltp / test_posix_groups will be tracked here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=16161 New tickets will be opened as needed for other issues.