| Summary: | glibc toolchain tests leave stale qemu processes | ||
|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Richard Purdie <richard.purdie> |
| Component: | oeqa/runtime | Assignee: | Sundeep Kokkonda <sundeep.kokkonda> |
| Status: | ACCEPTED --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | randy.macleod, ross.burton, sundeep.kokkonda |
| Version: | unspecified | ||
| Target Milestone: | 6.1 M4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Richard Purdie
2024-03-07 09:01:03 UTC
The logs show it is misc/tst-scm_rights and misc/tst-scm_rights-time64 Can you mention the steps to reproduce? I believe it as follows: 1. run glibc oe-selftest i.e oe-selftest -r glibc 2. checking the background processes running from ps (process status) cmd?(if there is any specific cmd to check this, please let me know). Is there is anything else that I should know? Yash, I think that your reproducer should be fine as long as you are testing qemumips or qemuppc. Before you start the build and oe-selftest run, just make sure that your user id does not have any qemu processes running. Then when the oe-selftest is done, us ps to look for a qemu process. Let us know here tomorrow if you can reproduce the issue and if the hung processes were also running either of misc/tst-scm_rights and misc/tst-scm_rights-time64. You may have to RTFM of ps to get long output. You might not be able to reproduce the issue on an ubuntu host but it's probably worth trying that first anyway since those are fast machines. The failure here was with a fedora-38 host machine so you might also use our 24 core fedora-39 system to test as well. As suggested by Randy, I checked for the stale qemu processes for qemuppc on ubuntu machine and they were found(misc/tst-scm_rights and misc/tst-scm_rights-time64) after running glibc oe-selftest. $ ps -el | grep qemu 1 S 23306 2592477 1 0 80 0 - 1101979 sigsus ? 00:00:00 qemu-ppc 1 S 23306 2592807 1 0 80 0 - 1101979 sigsus ? 00:00:00 qemu-ppc I was not able to check on fedora machine since there were some runqemu issues with it but I believe it would also reproduce there... I see there are 2 test case files(tst-scm_rights.c and tst-scm_rights-time64.c-> this file has just #include of the former file) at "sysdeps/unix/sysv/linux/" in glibc sources that passes a file descriptor from a subprocess to the parent process, using recvmsg/sendmsg or recvmmsg/sendmmsg. I think this send /rec processes are still not completed while glibc oe-selftest and thus are left as stale qemu processes. Thanks Yash.
The high level code is:
/* This test passes a file descriptor from a subprocess to the parent
process, using recvmsg/sendmsg or recvmmsg/sendmmsg. */
...
static int
do_test (void)
{
one_test ();
use_multi_call = true;
one_test ();
return 0;
}
The child calls pause() and waits for the parent to check the test results and
then terminate the child:
/* Terminate the subprocess. */
TEST_COMPARE (kill (pid, SIGUSR1), 0);
int status;
TEST_COMPARE (xwaitpid (pid, &status, 0), pid);
TEST_VERIFY (WIFSIGNALED (status));
TEST_COMPARE (WTERMSIG (status), SIGUSR1);
xclose (sockets[0]);
xclose (sockets[1]);
Does this work when run manually outside of oe-selftest on a native env.
Does the process really hang in the oe-selftest mode and if we should fix that
or (temporarily) skip the test.
(In reply to Randy MacLeod from comment #5) > Does this work when run manually outside of oe-selftest on a native env. > Does the process really hang in the oe-selftest mode and if we should fix > that > or (temporarily) skip the test. I tried to reproduce this manually outside of oe-selftest env but it could not reproduce since there some make errors while cross-compiling for ppc target. Meanwhile, I tried to add a process exit method as follows in the test case(sysdeps/unix/sysv/linux/tst-scm_rights.c) file and observed that there were not stale qemu processes left behind. diff --git a/sysdeps/unix/sysv/linux/tst-scm_rights.c b/sysdeps/unix/sysv/linux/tst-scm_rights.c @@ -172,6 +172,7 @@ xclose (sockets[0]); xclose (sockets[1]); + exit(0); } static int Another observation was that when I test the reproducer on newly cloned poky sources in some other directory path or user workspace, the stale qemu processes were not seen. Is this something that occurs sporadically or is it seen regularly? Patch to disable the two test cases merged in oe-core master: https://git.yoctoproject.org/poky/commit/?id=8d27d8ff7cc65d2059ca29352b57301ffc35a163 Bump to M3, we still need to understand and fix the root cause problem. Bump to a later M3. We still need to understand and fix the root cause problem. I just saw this with master. Hi Ross, Can you share more details about the recent observation? Does the same tests i.e tst-scm_rights and tst-scm_rights-time64 still leave stale processes for mips/ppc targets (I believe these targets are dropped from regular AB testing) or is there some new observation? These tests were skipped to avoid stale processes issue via the patch: 0023-tests-Skip-2-qemu-tests-that-can-hang-in-oe-selftest.patch . Perhaps, some recent changes to the patch might have caused this issue again. |