14294 2021-03-09 14:35:25 +0000 AB-INT PTEST: valgrind memcheck/tests/linux/timerfd-syscall ptest intermittent failure 2021-07-23 16:35:47 +0000 1 1 10 QA/Testing Package Testing (ptest) ptest unspecified x86 Multiple RESOLVED INVALID AB-INT AB-INT-TMPFS Medium+ normal 3.4 M3 1 kory.maincent tony.tascioglu alexandre.belloni bluelightning chee.yang.lee flowergom jon.mason naveen.kumar.saini oobitots randy.macleod richard.purdie tony.tascioglu --- --- No (bug/feature does not impact docs) oldest_to_newest 89543 0 kory.maincent 2021-03-09 14:35:25 +0000 New intermittent ptest failure in test core-image-sato, seen in: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1590/steps/12/logs/stdio NOTE: Running task 901 of 901 (/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb:do_testimage) NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Started Bitbake still alive (5000s) Bitbake still alive (10000s) WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'valgrind': ['memcheck/tests/linux/timerfd-syscall']} 89578 1 alexandre.belloni 2021-03-11 14:44:32 +0000 This was on ubuntu1804-arm-1 89665 2 alexandre.belloni 2021-03-19 23:37:20 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1556/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1548/steps/12/logs/stdio 89711 3 oobitots 2021-03-23 15:17:15 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1647 89726 4 yf3yu 2021-03-24 19:36:58 +0000 I can't reproduce this failure I think its due to the load/IO issue version: 3.16.1 === Test Summary === TOTAL: 726 PASSED: 686 FAILED: 0 SKIPPED: 40 DURATION: 1278 leaving it open for now... 89729 5 alexandre.belloni 2021-03-24 22:01:36 +0000 Until now, this only failed on ubuntu1804-arm-1 89798 6 bluelightning 2021-03-29 03:01:17 +0000 Failed again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1662 89818 7 yf3yu 2021-03-29 17:36:18 +0000 this seems to happening frequently on arm and not that intermittent in nature 89829 8 bluelightning 2021-03-30 16:48:11 +0000 Another instance: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1664/steps/12/logs/stdio 89836 9 bluelightning 2021-04-01 02:52:14 +0000 Another one: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1671/steps/12/logs/stdio 89946 10 alexandre.belloni 2021-04-15 14:24:55 +0000 This last onesA were on ubunt1804-arm-1. and the last one did contain the runqemu in tmpfs patch 90007 11 yf3yu 2021-04-19 19:12:56 +0000 fixed(more a workaround), see previous commit. 90008 12 yf3yu 2021-04-19 19:15:36 +0000 Sorry wrong bugzilla link.. reopening although I did submit a patch https://git.openembedded.org/openembedded-core/commit/?id=099313ef541920d4a84b801d9d8788a56ba7ec61 to print out the diff from expected result. 90009 13 yf3yu 2021-04-19 19:15:51 +0000 Sorry wrong bugzilla link.. reopening although I did submit a patch https://git.openembedded.org/openembedded-core/commit/?id=099313ef541920d4a84b801d9d8788a56ba7ec61 to print out the diff from expected result. 90079 14 yf3yu 2021-04-22 21:11:49 +0000 assigning it back to randy, if it happens again, we can get the test output details file at https://autobuilder.yocto.io/pub/non-release/ and make a judgement if the test needs to be disabled for timing/load reasons. 90112 15 naveen.kumar.saini 2021-04-26 03:02:15 +0000 Seen additional failures: AssertionError: Failed ptests: {'valgrind': ['drd/tests/bar_bad']} on master (qemux86-64): woerker: ubuntu1604-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2005 On master-next: worker:fedora32-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2006 90219 16 alexandre.belloni 2021-05-02 22:31:53 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1723/steps/12/logs/stdio qemuarm64 master on ubuntu1804-arm-1 90235 17 alexandre.belloni 2021-05-04 07:52:10 +0000 'valgrind': ['memcheck/tests/leak_cpp_interior'] https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2022/steps/12/logs/stdio qemuarm64 master on ubuntu1804-arm-1 90252 18 alexandre.belloni 2021-05-06 10:36:26 +0000 leak_cpp_interior is now disabled by 984ffe3ab4b7857b6201494cd77f2261bfc6d050 90321 19 chee.yang.lee 2021-05-08 14:45:17 +0000 happen again qemuarm64-ptest worker : ubuntu1804-arm-1 branch_poky : master-next https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1777 90386 20 chee.yang.lee 2021-05-14 08:21:30 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1799/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 90453 21 jon.mason 2021-05-21 11:51:44 +0000 Reoccurence https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1818/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 90519 22 alexandre.belloni 2021-05-27 22:35:41 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1840/steps/12/logs/stdio {'valgrind': ['memcheck/tests/linux/stack_changes']} ubuntu1804-arm-1 qemuarm64-ptest 90530 23 alexandre.belloni 2021-05-29 00:02:38 +0000 {'valgrind': ['memcheck/tests/linux/stack_changes']} https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1843/steps/12/logs/stdio qemuarm64-ptest ubuntu2004-arm-1 90546 24 flowergom 2021-06-02 02:03:43 +0000 Happend again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1853/steps/12/logs/stdio 'valgrind': ['memcheck/tests/linux/stack_changes'] qemuarm64-ptest 90549 25 flowergom 2021-06-02 02:33:23 +0000 Happend again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1852/steps/12/logs/stdio 90551 26 flowergom 2021-06-02 02:37:05 +0000 Happend again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1850/steps/12/logs/stdio 90586 27 flowergom 2021-06-07 10:31:08 +0000 Happend again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1871/steps/12/logs/stdio 90588 28 flowergom 2021-06-07 11:02:41 +0000 Happend again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1872/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2161/steps/12/logs/stdio 90594 29 tony.tascioglu 2021-06-09 16:24:04 +0000 Hello, I want to update this thread with a few findings. First, there are several different tests failing in the autobuilder logs in this thread, I'm not sure if it's due to the same underlying cause between these yet. Furthermore, some of these are failures are on x86_64 and some are on arm64. I don't think it's the same SMP issue that was affecting the helgrind tests, though it's harder to tell without the diffs or logs from the autobuilder. I tried to recreate this error by running stack_changes, timerfd-syscall and fb_test_amd64 in Qemu x86_64 in parallel with SMP enabled for over 12 hours overnight and neither of the tests failed. Arm64 on the other hand has several issues. Testing on qemu_arm64 from an x86_64 host without SMP and with 1G of RAM allocated, stack_changes failed all 10/10 times I ran it on master, and several of the other tests fail randomly a lot more frequently. I am going to try on older versions as well to check if it is a recent change. 90625 30 alexandre.belloni 2021-06-14 13:10:32 +0000 As discussed, the memcheck/tests/linux/stack_changes issue is not intermittent and I split it ou in bug 14430 91014 31 tony.tascioglu 2021-07-23 16:35:47 +0000 This was catch-all bug with several failures grouped together. We've created more-specific targeted bugs. The issues covered here are either covered by those bugs or no longer occurring.