| Summary: | valgrind ptest failing on qemux86-64 | ||
|---|---|---|---|
| Product: | [QA/Testing] Package Testing (ptest) | Reporter: | Jon Mason <jon.mason> |
| Component: | ptest | Assignee: | Mingli Yu <mingli.yu> |
| Status: | NEW --- | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | Medium+ | CC: | alexandre.belloni, flowergom, Jagadheesan_Duraisamy, joaomarcos.costa, leonardo.sandoval, raj.khem, randy.macleod |
| Version: | 3.3 | ||
| Target Milestone: | 6.99 | ||
| Hardware: | Other | ||
| OS: | x86_64 | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1777/steps/12/logs/stdio WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'valgrind': ['gdbserver_tests/nlcontrolc']} Cannot run ptests without @expectedFailure as ptests are expected to fail Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1781/steps/12/logs/stdio WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'valgrind': ['gdbserver_tests/nlcontrolc']} Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1785/steps/12/logs/stdio Bitbake still alive (5000s) WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'valgrind': ['gdbserver_tests/nlcontrolc']} Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1789/steps/12/logs/stdio WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'valgrind': ['gdbserver_tests/nlcontrolc']} Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1782/steps/12/logs/stdio Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1795/steps/12/logs/stdio Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1792/steps/12/logs/stdio When I quickly ran locally, maybe not with the most up todate version of poky I see the test was skipped. my head is sitting at f5188da2f12bee03e087c20d90720250e65e2922 SKIP: gdbserver_tests/nlcontrolc reproduced... last comment was on arm silly me. 2021-02-11T20:36 BEGIN: /usr/lib/valgrind/ptest Hide valgrind tests that are non-deterministic Reported at https://bugs.kde.org/show_bug.cgi?id=430321 mv: cannot stat 'drd/tests/bar_bad.vgtest': No such file or directory mv: cannot stat 'drd/tests/bar_bad_xml.vgtest': No such file or directory timed out PASS: gdbserver_tests/hginfo timed out PASS: gdbserver_tests/hgtls PASS: gdbserver_tests/mcblocklistsearch PASS: gdbserver_tests/mcbreak PASS: gdbserver_tests/mcclean_after_fork PASS: gdbserver_tests/mchelp timed out PASS: gdbserver_tests/mcinfcallRU timed out FAIL: gdbserver_tests/mcinfcallWSRU PASS: gdbserver_tests/mcinvokeRU PASS: gdbserver_tests/mcinvokeWS PASS: gdbserver_tests/mcleak PASS: gdbserver_tests/mcmain_pic PASS: gdbserver_tests/mcsignopass PASS: gdbserver_tests/mcsigpass PASS: gdbserver_tests/mcvabits timed out FAIL: gdbserver_tests/mcwatchpoints PASS: gdbserver_tests/mssnapshot timed out FAIL: gdbserver_tests/nlcontrolc PASS: gdbserver_tests/nlfork_chain seems to be a timeout issue... again downgraded to glibc2.32: no timeout failures seems related to the recent glibc2.33 uprev on poky commit: (prior to glibc2.33) commit 03bf6b3ec0aaa24e9f81b3b0024c59f7cc77b410 (HEAD) Author: Mingli Yu <mingli.yu@windriver.com> Date: Thu Feb 4 17:07:14 2021 +0800 timed out FAIL: drd/tests/annotate_barrier === Test Summary === TOTAL: 726 PASSED: 687 FAILED: 1 SKIPPED: 38 DURATION: 1260 nlcontrolc hangs on archlinux with glibc2.33 and valgrind 3.16.1 built from source trying to see if valgrind master fixed this issue clarification: archlinux keeps most up to date packages unlike other distro only valgrind was built from source, every other tool was from their most up-date repo. https://archlinux.org/packages/core/x86_64/glibc/ filed a bug report, a similar bug was reported couple months ago for arm64 https://bugs.kde.org/show_bug.cgi?id=432870 Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1809/steps/12/logs/stdio WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: 'valgrind': ['gdbserver_tests/nlcontrolc']} Cannot run ptests without @expectedFailure as ptests are expected to fail this commit for glibc (present in 2.33 but not 2.32) is causing this test to fail when i rebuilt glibc with a patch to revert this commit, the test *passes*. ``` commit 2433d39b69743f100f972e7886f91a2e21795ef0 Author: Adhemerval Zanella <adhemerval.zanella@linaro.org> Date: Mon Jul 6 16:06:51 2020 -0300 linux: Add time64 select support ``` since this commit is the root cause, dropping that particular patch as a temporary workaround is the easiest solution as it is a high priority bug (until a fix for either valgrind or glibc is implemented) nlcontrolc basically means sending a "ctrl-c" to the process in gdb. the test runs `LOOP` amount of `select` syscall to select which does nothing other than wait for the `TIMEOUT` (4th arg) to expire. Also note that 3 other threads (4 total) are also doing the exact same thing. `TIMEOUT` and `LOOP` are very large numbers, so the test would hang... unless the user sends a ctrl-c with gdb (which basically means causing a SIGTRAP) -> giving control back to gdb. we change TIMEOUT to 0 (for all 4 threads) and let the process run again if we are able to do it successfully, the test would complete much faster and on the next SIGTRAP we send, we should be no longer stuck in the the `select` syscall. Obviously in our bug, we probably weren't able to successfully get out of `select`, maybe because in linux(glibc), `select` actually updates the remaining amount of timeout in the same spot that gdb is trying to overwrite (so another thread/process can read that value) Looking into this interaction: https://man7.org/linux/man-pages/man2/select.2.html: ``` On Linux, select() modifies timeout to reflect the amount of time not slept; most other implementations do not do this. (POSIX.1 permits either behavior.) ``` Update: `LOOP` doesn't even matter, the burn time and the sleep(select timeout) is long enough already. setting the timeout argument to 0 will just make the threads go to burn (also taking a long time) Another update: seems like we have an compiler optimization problem after running the test manually: ./sleepers 1000000000 1000000000 1000000000 BSBSBSBS 1 info threads ``` * 1 Thread 0x7ffff7ddb740 (LWP 340) "sleepers" 0x00007ffff7ecf619 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=<optimized out>) ``` not really sure why it would happen when we add that glibc patch, how does glibc deal with timeout? testing with volatile looking more like a glibc issue (at least for the glibc 2.32->2.33 failure on x86-64) if the timeout parameter gets optimized out by the compiler. There is no reason for the compiler to optimize it out since the timeout parameter gets filled by a value coming from argv. the only reason I suspect that would happen is a problem down in glibc where timeout doesn't get used at all. could it be that valgrind calls it the "wrong" way? very *unlikely* because its an API call defined by the POSIX standard. It might be doing something fishy by changing the last argument during the call (setting `timeout->sec_tv` 0) using gdb. here is syscall trace on glibc2.33 (with the bug present) : 5th argument seems to be garbage (expected to be in sequence t[0], t[1] t[2], t[3]) ``` root@qemux86-64:/usr/lib/valgrind/ptest/gdbserver_tests# valgrind --trace-syscalls=yes ./sleepers 1000000000 1000000000 1000000000 BSBSBSBS 2>&1 | grep pselect6 SYSCALL[365,2](270) sys_pselect6 ( 0, 0x0, 0x0, 0x0, 0x5634e00, 0x0 ) --> [async] ... SYSCALL[365,3](270) sys_pselect6 ( 0, 0x0, 0x0, 0x0, 0x5e35e00, 0x0 ) --> [async] ... SYSCALL[365,1](270) sys_pselect6 ( 0, 0x0, 0x0, 0x0, 0x1fff000a50, 0x0 ) --> [async] ... SYSCALL[365,4](270) sys_pselect6 ( 0, 0x0, 0x0, 0x0, 0x6636e00, 0x0 ) --> [async] ... things to investigate further: * turning off compiler optimization on glibc or valgrind nlcontrolc expects to change the behaviour of a syscall by changing the argument `timeout` during that syscall (while it is blocked and probably getting woken up periodically by the system timer) Here is what i think is the root cause: From looking at that particular glibc commit, this is dependent on the compiler's optimization. - glibc creates a wrapper around the real syscall and converts the timeout type `timeval` to `timespec`, a new variable. Compiler can decide to optimize for space and actually have them be the same variable. nlcontrolc takes advantage of that by modifying it. in newest glibc2.33, they added the time64 support, but compiler is less likely to say that `timeval32` looks like `timespec64` Currently testing this theory if i build 2.32 with no optimization. expected behaviour is a hang in nlcontrolc another way to test it is actually changing the input to the argument to the underlying syscall. expected behaviour is no hang in nlcontrol Here is a working call to sleepers *without* the select update patch in 2.33
My previous assumption about compiler optimization is wrong, it used to directly passes timeout to the underlying syscall.
```
root@qemux86-64:/usr/lib/valgrind/ptest/gdbserver_tests# valgrind --trace-syscalls=yes ./sleepers 1000000000 1000000000 1000000000 BSBSBSBS 2>&1 | grep select
SYSCALL[6021,2](23) sys_select ( 0, 0x0, 0x0, 0x0, 0x4041b0 ) --> [async] ...
SYSCALL[6021,3](23) sys_select ( 0, 0x0, 0x0, 0x0, 0x4041c0 ) --> [async] ...
SYSCALL[6021,1](23) sys_select ( 0, 0x0, 0x0, 0x0, 0x4041a0 ) --> [async] ...
SYSCALL[6021,4](23) sys_select ( 0, 0x0, 0x0, 0x0, 0x4041d0 ) --> [async] ...
```
It used to pass timeout directly to `select`, now it calls `select6`
```
-__select (int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds,
- struct timeval *timeout)
+__select64 (int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds,
+ struct __timeval64 *timeout)
{
-#ifdef __NR_select
- return SYSCALL_CANCEL (select, nfds, readfds, writefds, exceptfds,
- timeout);
```
here is how it calls it now
```
+ r = SYSCALL_CANCEL (pselect6, nfds, readfds, writefds, exceptfds, pts32,
+ NULL);
```
TLDR: ## glibc patch that causes failure ``` commit 2433d39b69743f100f972e7886f91a2e21795ef0 Author: Adhemerval Zanella <adhemerval.zanella@linaro.org> Date: Mon Jul 6 16:06:51 2020 -0300 linux: Add time64 select support ``` ## what changed in this patch to cause it to fail? timeout argument the user passes no longer makes its way to the syscall. Glibc copies it over and converts into a different format to call a different syscall `pselect` the failing test tries to modify said timeout argument to make the syscall end faster. Unfortunately doesn't work. ## what is the fix for this? disable it now in YP ## what about actually fixing this bug though? talking to valgrind about the purpose of nlcontrolc test - can we use a different syscall to sleep for a duration? - what is the exact purpose if this test ## other questions that arise Is glibc setting a new standard? what is the expected libc implementation of select? Grey area... according to `man select` ``` On Linux, select() modifies timeout to reflect the amount of time not slept; most other implementations do not do this. (POSIX.1-2001 permits either behavior.) ``` Did glibc violate the "linux standard"? ## other mentions the same test has been disabled for arm64 for around 6months now. Added Khem to CC since he knows how to spell glibc. :) I thought Alistair was in the YP BZ but he's not. I'll forward this thread to him. Is this commit: commit 2433d39b69743f100f972e7886f91a2e21795ef0 Author: Adhemerval Zanella <adhemerval.zanella@linaro.org> Date: Mon Jul 6 16:06:51 2020 -0300 linux: Add time64 select support The syscall __NR_pselect6_time64 (32-bit) or __NR_pselect6 (64-bit) is used as default. For architectures with __ASSUME_TIME64_SYSCALLS the 32-bit fallback uses __NR_select/__NR__newselect or __NR_pselect6 (it should cover the microblaze case where older kernels do not provide __NR_pselect6). Checked on x86_64-linux-gnu and i686-linux-gnu (on 5.4 and on 4.15 kernel). Reviewed-by: Alistair Francis <alistair.francis@wdc.com> a problem for any other package that you all know of? See Yi's previous comment for a good short summary of what's going wrong. Yi please look at the musl select behaviour/implementation. musl master source code src/select/select.c
tv is timeoutvalue
s and ns are based on tv->tv_sec and tv->tv_usec
```
r = __syscall_cp(SYS_pselect6_time64, n, rfds, wfds, efds,
tv ? ((long long[]){s, ns}) : 0,
``
a new array is created,
so musl select implementation looks to align with the new gibc one
current nlcontrolc should fail also with musl... checking
Please try to reproduce this with gdb master and if it works then we need to search for a relevant fix perhaps try applying https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=439706e6a907b54658714a358b220febc63a0182 For record, happened in: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1821/steps/12/logs/stdio temporary fix to skip test https://git.openembedded.org/openembedded-core/commit/?id=bb991a24952ebf02cf0092c194d2b3a54a8b2e5b (In reply to comment #26) > For record, happened in: > https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1821/steps/ > 12/logs/stdio Also in https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1813/steps/12/logs/stdio khem: built newest gdb (had to remove some patches and depend on gmp) test still fails ``` timed out FAIL: gdbserver_tests/nlcontrolc PASS: gdbserver_tests/nlfork_chain PASS: gdbserver_tests/nlgone_abrt c^C root@qemux86-64:/usr/lib/valgrind/ptest# ^C --version86-64:/usr/lib/valgrind/ptest# gdb GNU gdb (GDB) 11.0.50.20210219-git Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` Another failure event [1]
WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
self.do_ptestrunner()
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
self.fail(failmsg)
AssertionError: Failed ptests:
{'valgrind': ['gdbserver_tests/nlcontrolc']}
[1] https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1811/steps/12/logs/stdio
(In reply to comment #30) > AssertionError: Failed ptests: > {'valgrind': ['gdbserver_tests/nlcontrolc']} > > > [1] > https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1811/steps/ > 12/logs/stdio this failure 9 days ago, the patch was committed around 7 days ago. https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1842 a similar test with 'master-next' 1 day ago AssertionError: Failed ptests: {'python3': ['test_many_processes']} Lowering to M+ since we are skipping the test and working with upstream valgrind and/or gdb (glibc?) devs to get to the root cause and a fix. An other similar failure from before the patch: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1810 And other failures seen: 'valgrind': ['memcheck/tests/linux/timerfd-syscall'] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1556/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1548/steps/12/logs/stdio 'valgrind': ['drd/tests/std_list'] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1550/steps/12/logs/stdio An other valgrind ptest failing:
{'valgrind': ['drd/tests/thread_name', 'drd/tests/thread_name_xml']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1549/steps/10/logs/step1c
https://git.openembedded.org/openembedded-core/commit/?id=aac00b1c8042e41cd6bb1aea8e3033a1c6dd2b05 should make the test pass more frequently, upstream has fixed nlcontrolc by redesigning the test https://sourceware.org/git/?p=valgrind.git;a=commit;h=c79180a3afcf65902e578646c3b716cc749db406 a patch to oe-core is in master-next So far failures happened: 3 times on opensuse151-ty-1 4 times on ubuntu1804-arm-1 2 times on tumbleweed-ty-1 2 times on centos8-ty-1 2 times on debian10-ty-2 2 times on opensuse152-ty-1 2 times on ubuntu1804-ty-2 1 time on ubuntu1804-ty-1 1 time on ubuntu2004-ty-1 1 time on fedora33-ty-1 resolving this... please open new bug for other failures this ticket was for nlcontrolc https://git.openembedded.org/openembedded-core/commit/?id=64f98a731adaf86c9185c90436db1a6eb8e16b42 https://autobuilder.yoctoproject.org/valkyrie/#/builders/73/builds/866/steps/12/logs/stdio nlcontrolc appeared again: {'valgrind': ['gdbserver_tests/mcinfcallWSRU', 'gdbserver_tests/nlcontrolc', 'gdbserver_tests/nlvgdbsigqueue', 'memcheck/tests/sendmsg', 'drd/tests/fork-parallel', 'drd/tests/fork-serial', 'drd/tests/threaded-fork-vcs', 'drd/tests/threaded-fork', 'none/tests/fdleak_ipv4', 'none/tests/file_dclose', 'none/tests/socket_close']} qemux86-64-ptest rocky9-vk-1 Reopening. https://autobuilder.yoctoproject.org/valkyrie/#/builders/61/builds/864/steps/12/logs/stdio Failed ptests: {'valgrind': ['gdbserver_tests/mcinfcallWSRU', 'gdbserver_tests/nlcontrolc', 'gdbserver_tests/nlvgdbsigqueue', 'memcheck/tests/sendmsg', 'drd/tests/fork-parallel', 'drd/tests/fork-serial', 'drd/tests/threaded-fork-vcs', 'drd/tests/threaded-fork', 'none/tests/fdleak_ipv4', 'none/tests/file_dclose', 'none/tests/socket_close']} qemuarm64-ptest ubuntu2204-vk-arm1 This was with glibc-2.41. Mingli is working with valgrind upstream to try to improve the test approach so that we don't have as big a patchset and hopefully one that has fewer if any intermittent problems. We have disabled valgrind ptests on master: https://git.openembedded.org/openembedded-core/commit/?id=8fc31fb235af12530475ec5055f6c00ccfafadfa |
Seeing valgrind ptest failing in multiple runs with the new glibc. For example: WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'valgrind': ['gdbserver_tests/nlcontrolc']} Cannot run ptests without @expectedFailure as ptests are expected to fail RESULTS: RESULTS - ping.PingTest.test_ping: PASSED (0.61s) RESULTS - ssh.SSHTest.test_ssh: PASSED (0.17s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: EXPECTEDFAIL (10957.28s) SUMMARY: core-image-sato-sdk-ptest () - Ran 4 tests in 10958.078s core-image-sato-sdk-ptest - OK - All required tests passed (successes=2, skipped=1, failures=0, errors=0) NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Succeeded NOTE: Tasks Summary: Attempted 910 tasks of which 909 didn't need to be rerun and all succeeded. See https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1769 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1772 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1774