15625 2024-10-16 09:05:04 +0000 AB-INT: rust process-sigpipe fails on qemu: mips64, ppc 2025-01-15 12:15:52 +0000 1 1 10 QA/Testing Functional (self) Testing general unspecified x86 Multiple RESOLVED FIXED AB-INT Medium+ normal 5.2 M1 1 mathieu.dubois-briand unassigned adrian.freihofer adrian.freihofer chuckwolber peter.tatrai.ext randy.macleod sundeep.kokkonda --- --- No (bug/feature does not impact docs) oldest_to_newest 99878 0 mathieu.dubois-briand 2024-10-16 09:05:04 +0000 qemumips64-tc alma9-vk-1 https://valkyrie.yoctoproject.org//#/builders/21/builds/226/steps/12/logs/stdio failures: ---- [ui] tests/ui/process/process-sigpipe.rs stdout ---- error: test run failed! status: exit status: 1 command: RUST_TEST_THREADS="96" "/srv/pokybuild/yocto-worker/qemumips64-tc/build/build-st-1702276/tmp/work/mips64r2-poky-linux/rust/1.79.0/rustc-1.79.0-src/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "run" "0" "/srv/pokybuild/yocto-worker/qemumips64-tc/build/build-st-1702276/tmp/work/mips64r2-poky-linux/rust/1.79.0/rustc-1.79.0-src/build/x86_64-unknown-linux-gnu/test/ui/process/process-sigpipe/a" --- stdout ------------------------------- uploaded "/srv/pokybuild/yocto-worker/qemumips64-tc/build/build-st-1702276/tmp/work/mips64r2-poky-linux/rust/1.79.0/rustc-1.79.0-src/build/x86_64-unknown-linux-gnu/test/ui/process/process-sigpipe/a", waiting for result ------------------------------------------ stderr: none 99882 1 mathieu.dubois-briand 2024-10-16 15:42:52 +0000 qemuppc-tc alma9-vk-1 https://valkyrie.yoctoproject.org/#/builders/49/builds/177/steps/12/logs/stdio 99906 2 randy.macleod 2024-10-17 14:46:41 +0000 Rust on these targets is tier 3 in Rust upstream. We need someone who has an interest in these targets to help out directly or at least proivide lots of help Sundeep's team. 99940 3 mathieu.dubois-briand 2024-10-18 10:19:54 +0000 qemumips64-tc alma9-vk-1 https://valkyrie.yoctoproject.org/#/builders/21/builds/226/steps/12/logs/stdio 100050 4 peter.tatrai.ext 2024-10-28 20:47:22 +0000 All these runs failed on the same test: > test [ui] tests/ui/process/process-sigpipe.rs ... FAILED The test itself is straightforward. It starts a shell with the command `while echo y; do :; done | head` and expects it to complete within 5 seconds. More details can be found here: https://github.com/rust-lang/rust/blob/master/tests/ui/process/process-sigpipe.rs. I suspect that under heavy system load, the test might not complete in time. I reproduced the failure by simulating load with stress -m 48 on a system with 24 logical cores. To avoid this failure, I see two possible solutions: either increase the timeout (perhaps to 10 seconds) or skip the test. 100151 5 adrian.freihofer 2024-11-14 08:05:29 +0000 The proposed fix is now on master branch: https://git.yoctoproject.org/poky/commit/?id=b82bb08f15587a376b2b4da5ff553b630e87283d Lets hope this improves the situation. 100155 6 randy.macleod 2024-11-14 15:15:21 +0000 Thanks for the fix Peter. Adrian, I'm resolving the defect as fixed. If we see the same symptom, most likely Mathieu will re-open this bug. 100497 7 mathieu.dubois-briand 2024-12-24 09:53:50 +0000 qemuppc-tc alma9-vk-1 master completed at 2024-12-23T05:30:31Z https://valkyrie.yoctoproject.org/#/builders/49/builds/680/steps/12/logs/stdio qemuppc-tc alma9-vk-1 mathieu/master-next completed at 2024-12-23T16:35:57Z https://valkyrie.yoctoproject.org/#/builders/49/builds/681/steps/12/logs/stdio 100500 8 adrian.freihofer 2024-12-24 14:33:42 +0000 Definitely not the Christmas present I was most hoping for... But it still looks like a simple timing issue. Arguments in favor of this: - Code sets a timeout. - Peter was able to prove that overloading the build machine during this test leads to a timeout. - Also this error looks like a simple timeout: - 13590: test [ui] tests/ui/process/process-sigpipe.rs has been running for a long time - 13600: test [ui] tests/ui/process/process-sigpipe.rs ... FAILED On the other hand ... - does it really only occur on ppc and mips because they have such bad qemu implementations? Or could it be that the same bug that occurs on other architectures is not being reported here? - Did increasing the timeout by a factor of 2 (5000 -> 10000) reduce the occurrence of this error? Is it worth trying to increase the timeout by a factor of 10 instead of just 2, or are we overlooking something else? 100502 9 randy.macleod 2024-12-24 19:53:29 +0000 Adrian, We've found that timeouts often need to be significantly increased so let's try a 10x increase to see if that resolves the issue. If you also want to also satisfy developers who are working interactively on an idle machine, you can use the original timeout value but retry 10 times, issuing a warning at each stage until finally failing with an error. 100626 10 mathieu.dubois-briand 2025-01-14 10:07:23 +0000 qemuppc-tc alma9-vk-1 master completed at 2025-01-05T06:13:11Z https://valkyrie.yoctoproject.org/#/builders/49/builds/730/steps/12/logs/stdio 100645 11 adrian.freihofer 2025-01-15 12:15:52 +0000 Sent another patch to increase the timeout by 10 times. First try was with 2 times. https://patchwork.yoctoproject.org/project/oe-core/patch/20250114221701.13257-1-adrian.freihofer@siemens.com/