qemumips64-tc alma9-vk-1 https://valkyrie.yoctoproject.org//#/builders/21/builds/226/steps/12/logs/stdio failures: ---- [ui] tests/ui/process/process-sigpipe.rs stdout ---- error: test run failed! status: exit status: 1 command: RUST_TEST_THREADS="96" "/srv/pokybuild/yocto-worker/qemumips64-tc/build/build-st-1702276/tmp/work/mips64r2-poky-linux/rust/1.79.0/rustc-1.79.0-src/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "run" "0" "/srv/pokybuild/yocto-worker/qemumips64-tc/build/build-st-1702276/tmp/work/mips64r2-poky-linux/rust/1.79.0/rustc-1.79.0-src/build/x86_64-unknown-linux-gnu/test/ui/process/process-sigpipe/a" --- stdout ------------------------------- uploaded "/srv/pokybuild/yocto-worker/qemumips64-tc/build/build-st-1702276/tmp/work/mips64r2-poky-linux/rust/1.79.0/rustc-1.79.0-src/build/x86_64-unknown-linux-gnu/test/ui/process/process-sigpipe/a", waiting for result ------------------------------------------ stderr: none
qemuppc-tc alma9-vk-1 https://valkyrie.yoctoproject.org/#/builders/49/builds/177/steps/12/logs/stdio
Rust on these targets is tier 3 in Rust upstream. We need someone who has an interest in these targets to help out directly or at least proivide lots of help Sundeep's team.
qemumips64-tc alma9-vk-1 https://valkyrie.yoctoproject.org/#/builders/21/builds/226/steps/12/logs/stdio
All these runs failed on the same test: > test [ui] tests/ui/process/process-sigpipe.rs ... FAILED The test itself is straightforward. It starts a shell with the command `while echo y; do :; done | head` and expects it to complete within 5 seconds. More details can be found here: https://github.com/rust-lang/rust/blob/master/tests/ui/process/process-sigpipe.rs. I suspect that under heavy system load, the test might not complete in time. I reproduced the failure by simulating load with stress -m 48 on a system with 24 logical cores. To avoid this failure, I see two possible solutions: either increase the timeout (perhaps to 10 seconds) or skip the test.
The proposed fix is now on master branch: https://git.yoctoproject.org/poky/commit/?id=b82bb08f15587a376b2b4da5ff553b630e87283d Lets hope this improves the situation.
Thanks for the fix Peter. Adrian, I'm resolving the defect as fixed. If we see the same symptom, most likely Mathieu will re-open this bug.
qemuppc-tc alma9-vk-1 master completed at 2024-12-23T05:30:31Z https://valkyrie.yoctoproject.org/#/builders/49/builds/680/steps/12/logs/stdio qemuppc-tc alma9-vk-1 mathieu/master-next completed at 2024-12-23T16:35:57Z https://valkyrie.yoctoproject.org/#/builders/49/builds/681/steps/12/logs/stdio
Definitely not the Christmas present I was most hoping for... But it still looks like a simple timing issue. Arguments in favor of this: - Code sets a timeout. - Peter was able to prove that overloading the build machine during this test leads to a timeout. - Also this error looks like a simple timeout: - 13590: test [ui] tests/ui/process/process-sigpipe.rs has been running for a long time - 13600: test [ui] tests/ui/process/process-sigpipe.rs ... FAILED On the other hand ... - does it really only occur on ppc and mips because they have such bad qemu implementations? Or could it be that the same bug that occurs on other architectures is not being reported here? - Did increasing the timeout by a factor of 2 (5000 -> 10000) reduce the occurrence of this error? Is it worth trying to increase the timeout by a factor of 10 instead of just 2, or are we overlooking something else?
Adrian, We've found that timeouts often need to be significantly increased so let's try a 10x increase to see if that resolves the issue. If you also want to also satisfy developers who are working interactively on an idle machine, you can use the original timeout value but retry 10 times, issuing a warning at each stage until finally failing with an error.
qemuppc-tc alma9-vk-1 master completed at 2025-01-05T06:13:11Z https://valkyrie.yoctoproject.org/#/builders/49/builds/730/steps/12/logs/stdio
Sent another patch to increase the timeout by 10 times. First try was with 2 times. https://patchwork.yoctoproject.org/project/oe-core/patch/20250114221701.13257-1-adrian.freihofer@siemens.com/