15423 2024-03-07 09:01:03 +0000 glibc toolchain tests leave stale qemu processes 2026-05-28 14:05:41 +0000 1 1 10 QA/Testing Runtime Testing oeqa/runtime unspecified x86 Multiple ACCEPTED Medium+ normal 6.1 M4 1 richard.purdie sundeep.kokkonda randy.macleod ross.burton sundeep.kokkonda --- --- No (bug/feature does not impact docs) oldest_to_newest 98394 0 richard.purdie 2024-03-07 09:01:03 +0000 Looking at processes left behind on a worker, qemumips and qemuppc have left behind stale qemu processes. We need to find out why and clean these up. 2622734 ? SNl 0:00 /home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-mips -r 5.15 -L /home/pokybuild/yocto-worker/qemu 2623117 ? SNl 0:00 /home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-mips -r 5.15 -L /home/pokybuild/yocto-worker/qemu 2651712 ? I 0:00 [kworker/15:0-events] 2651727 ? I 0:00 [kworker/16:0-mm_percpu_wq] 2808681 ? I 0:00 [kworker/10:0-events] 2854486 ? SNl 0:00 /home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-ppc -r 5.15 -cpu 7400 -L /home/pokybuild/yocto-work 2854919 ? SNl 0:00 /home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-ppc -r 5.15 -cpu 7400 -L /home/pokybuild/yocto-work 2939473 ? I 0:03 [kworker/u113:2-flush-btrfs-1] (buildbot-venv) [pokybuild@fedora38-ty-4 saved-screenshots]$ cat /proc/2622734/cmdline /home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-mips-r5.15-L/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-ELD_LIBRARY_PATH=/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/usr/lib:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/lib/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/elf/ld.so.1--library-path/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/math:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/elf:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/dlfcn:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/nss:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/nis:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/rt:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/resolv:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/mathvec:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/support:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/nptl/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/misc/tst-scm_rights(buildbot-venv) [pokybuild@fedora38-ty-4 saved-screenshots]$ cat /proc/2623117/cmdline /home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-mips-r5.15-L/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-ELD_LIBRARY_PATH=/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/usr/lib:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/lib/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/elf/ld.so.1--library-path/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/math:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/elf:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/dlfcn:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/nss:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/nis:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/rt:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/resolv:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/mathvec:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/support:/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/nptl/home/pokybuild/yocto-worker/qemumips-tc/build/build-st-1278223/tmp/work/mips32r2-poky-linux/glibc-testsuite/2.39+git/build-mips-poky-linux/misc/tst-scm_rights-time64(buildbot-venv) [pokybuild@fedora38-ty-4 saved-screenshots]$ cat /proc/2854486/cmdline /home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-ppc-r5.15-cpu7400-L/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-ELD_LIBRARY_PATH=/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/usr/lib:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/lib/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/elf/ld.so.1--library-path/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/math:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/elf:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/dlfcn:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/nss:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/nis:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/rt:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/resolv:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/mathvec:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/support:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/nptl/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/misc/tst-scm_rights(buildbot-venv) [pokybuild@fedora38-ty-4 saved-screenshots]$ cat /proc/2854919/cmdline /home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-native/usr/bin/qemu-ppc-r5.15-cpu7400-L/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot-ELD_LIBRARY_PATH=/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/usr/lib:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/recipe-sysroot/lib/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/elf/ld.so.1--library-path/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/math:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/elf:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/dlfcn:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/nss:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/nis:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/rt:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/resolv:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/mathvec:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/support:/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/nptl/home/pokybuild/yocto-worker/qemuppc-tc/build/build-st-2351293/tmp/work/ppc7400-poky-linux/glibc-testsuite/2.39+git/build-powerpc-poky-linux/misc/tst-scm_rights-time64(buildbot-venv) [pokybuild@fedora38-ty-4 saved-screenshots]$ 98395 1 richard.purdie 2024-03-07 09:02:03 +0000 The logs show it is misc/tst-scm_rights and misc/tst-scm_rights-time64 98403 2 Yash.Shinde 2024-03-07 15:02:06 +0000 Can you mention the steps to reproduce? I believe it as follows: 1. run glibc oe-selftest i.e oe-selftest -r glibc 2. checking the background processes running from ps (process status) cmd?(if there is any specific cmd to check this, please let me know). Is there is anything else that I should know? 98429 3 randy.macleod 2024-03-07 20:18:36 +0000 Yash, I think that your reproducer should be fine as long as you are testing qemumips or qemuppc. Before you start the build and oe-selftest run, just make sure that your user id does not have any qemu processes running. Then when the oe-selftest is done, us ps to look for a qemu process. Let us know here tomorrow if you can reproduce the issue and if the hung processes were also running either of misc/tst-scm_rights and misc/tst-scm_rights-time64. You may have to RTFM of ps to get long output. You might not be able to reproduce the issue on an ubuntu host but it's probably worth trying that first anyway since those are fast machines. The failure here was with a fedora-38 host machine so you might also use our 24 core fedora-39 system to test as well. 98436 4 Yash.Shinde 2024-03-08 12:10:37 +0000 As suggested by Randy, I checked for the stale qemu processes for qemuppc on ubuntu machine and they were found(misc/tst-scm_rights and misc/tst-scm_rights-time64) after running glibc oe-selftest. $ ps -el | grep qemu 1 S 23306 2592477 1 0 80 0 - 1101979 sigsus ? 00:00:00 qemu-ppc 1 S 23306 2592807 1 0 80 0 - 1101979 sigsus ? 00:00:00 qemu-ppc I was not able to check on fedora machine since there were some runqemu issues with it but I believe it would also reproduce there... I see there are 2 test case files(tst-scm_rights.c and tst-scm_rights-time64.c-> this file has just #include of the former file) at "sysdeps/unix/sysv/linux/" in glibc sources that passes a file descriptor from a subprocess to the parent process, using recvmsg/sendmsg or recvmmsg/sendmmsg. I think this send /rec processes are still not completed while glibc oe-selftest and thus are left as stale qemu processes. 98439 5 randy.macleod 2024-03-08 18:16:39 +0000 Thanks Yash. The high level code is: /* This test passes a file descriptor from a subprocess to the parent process, using recvmsg/sendmsg or recvmmsg/sendmmsg. */ ... static int do_test (void) { one_test (); use_multi_call = true; one_test (); return 0; } The child calls pause() and waits for the parent to check the test results and then terminate the child: /* Terminate the subprocess. */ TEST_COMPARE (kill (pid, SIGUSR1), 0); int status; TEST_COMPARE (xwaitpid (pid, &status, 0), pid); TEST_VERIFY (WIFSIGNALED (status)); TEST_COMPARE (WTERMSIG (status), SIGUSR1); xclose (sockets[0]); xclose (sockets[1]); Does this work when run manually outside of oe-selftest on a native env. Does the process really hang in the oe-selftest mode and if we should fix that or (temporarily) skip the test. 98537 6 Yash.Shinde 2024-03-20 06:42:54 +0000 (In reply to Randy MacLeod from comment #5) > Does this work when run manually outside of oe-selftest on a native env. > Does the process really hang in the oe-selftest mode and if we should fix > that > or (temporarily) skip the test. I tried to reproduce this manually outside of oe-selftest env but it could not reproduce since there some make errors while cross-compiling for ppc target. Meanwhile, I tried to add a process exit method as follows in the test case(sysdeps/unix/sysv/linux/tst-scm_rights.c) file and observed that there were not stale qemu processes left behind. diff --git a/sysdeps/unix/sysv/linux/tst-scm_rights.c b/sysdeps/unix/sysv/linux/tst-scm_rights.c @@ -172,6 +172,7 @@ xclose (sockets[0]); xclose (sockets[1]); + exit(0); } static int Another observation was that when I test the reproducer on newly cloned poky sources in some other directory path or user workspace, the stale qemu processes were not seen. Is this something that occurs sporadically or is it seen regularly? 98713 7 Yash.Shinde 2024-04-05 10:25:33 +0000 Patch to disable the two test cases merged in oe-core master: https://git.yoctoproject.org/poky/commit/?id=8d27d8ff7cc65d2059ca29352b57301ffc35a163 99167 8 randy.macleod 2024-06-11 18:38:15 +0000 Bump to M3, we still need to understand and fix the root cause problem. 100637 9 randy.macleod 2025-01-14 19:52:14 +0000 Bump to a later M3. We still need to understand and fix the root cause problem. 103202 10 ross.burton 2025-11-06 11:00:38 +0000 I just saw this with master. 103204 11 Yash.Shinde 2025-11-06 12:05:10 +0000 Hi Ross, Can you share more details about the recent observation? Does the same tests i.e tst-scm_rights and tst-scm_rights-time64 still leave stale processes for mips/ppc targets (I believe these targets are dropped from regular AB testing) or is there some new observation? These tests were skipped to avoid stale processes issue via the patch: 0023-tests-Skip-2-qemu-tests-that-can-hang-in-oe-selftest.patch . Perhaps, some recent changes to the patch might have caused this issue again.