14296 2021-03-09 16:21:09 +0000 AB-INT PTEST: python3 ptest intermittent failure 2022-04-28 14:38:05 +0000 1 1 10 QA/Testing Package Testing (ptest) ptest unspecified x86 Multiple RESOLVED WORKSFORME AB-INT Medium normal 4.1 1 kory.maincent randy.macleod alexandre.belloni bluelightning chee.yang.lee flowergom jon.mason randy.macleod richard.purdie thomas.perrot tim.orling --- --- No (bug/feature does not impact docs) oldest_to_newest 89555 0 kory.maincent 2021-03-09 16:21:09 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1606/steps/12/logs/stdio NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Started Bitbake still alive (5000s) Bitbake still alive (10000s) WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'python3': ['test_wait_integer']} 89565 1 kory.maincent 2021-03-11 14:13:41 +0000 Python3 and lttng-tools intermittent failure. https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1893/steps/13/logs/stdio NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Started Bitbake still alive (5000s) Bitbake still alive (10000s) WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'lttng-tools': ['tools/notification/test_notification_multi_app_28_-_Waiting_message:_error_logged_see_file_content:_notification:_high_4,_/tmp/tmp.XXXXXXXXXX/high_app_output_file_0', 'tools/notification/test_notification_multi_app_29_-_Stop_lttng_tracing_for_session_my_session', 'tools/notification/test_notification_multi_app_30_-_Waiting_message:_error_logged_see_file_content:_notification:_low_4,_/tmp/tmp.XXXXXXXXXX/low_app_output_file_0', 'tools/notification/test_notification_multi_app_31_-_Notifications_4_received', 'tools/notification/test_notification_multi_app_32_-_Waiting_message:_error_logged_see_file_content:_exit:_0,_/tmp/tmp.XXXXXXXXXX/low_app_output_file_0', 'tools/notification/test_notification_multi_app_33_-_Application_for_low_notification_terminated_normally', 'tools/notification/test_notification_multi_app_34_-_Waiting_message:_error_logged_see_file_content:_exit:_0,_/tmp/tmp.XXXXXXXXXX/high_app_output_file_0', 'tools/notification/test_notification_multi_app_35_-_Application_for_high_notification_terminated_normally', 'tools/notification/test_notification_multi_app_36_-_Destroy_session_my_session'], 'python3': ['test_many_processes']} 89582 2 tim.orling 2021-03-11 15:57:32 +0000 > {'python3': ['test_wait_integer']} https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/test/_test_multiprocessing.py#L4718 89584 3 tim.orling 2021-03-11 16:00:11 +0000 > 'python3': ['test_many_processes']} https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/test/_test_multiprocessing.py#L571 89800 4 bluelightning 2021-03-29 03:13:52 +0000 This seems similar (ignoring the non-python3 failures, though they might be indicative of load at the time): https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1662/steps/12/logs/stdio WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'python3': ['test_process_time'], 'tcl': ['interp.test'], 'valgrind': ['memcheck/tests/linux/timerfd-syscall']} 89820 5 yf3yu 2021-03-29 18:48:12 +0000 The load is highish, the test tests for cpu time consumed for a sleep. https://autobuilder.yocto.io/pub/non-release/20210328-5/testresults/qemuarm64-ptest/python3.log OK 0:38:55 load avg: 1.20 [356/425] test_time [SNIP] FAIL: test_process_time (test.test_time.TimeTestCase) PASS: test_pthread_getcpuclockid (test.test_time.TimeTestCase) PASS: test_sleep (test.test_time.TimeTestCase) PASS: test_strftime (test.test_time.TimeTestCase) PASS: test_strftime_bounding_check (test.test_time.TimeTestCase) PASS: test_strftime_format_check (test.test_time.TimeTestCase) PASS: test_strptime (test.test_time.TimeTestCase) PASS: test_strptime_bytes (test.test_time.TimeTestCase) PASS: test_strptime_exception_context (test.test_time.TimeTestCase) PASS: test_thread_time (test.test_time.TimeTestCase) PASS: test_time (test.test_time.TimeTestCase) PASS: test_time_ns_type (test.test_time.TimeTestCase) test test_time failed PASS: test_tzset (test.test_time.TimeTestCase) ====================================================================== FAIL: test_process_time (test.test_time.TimeTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.9/test/test_time.py", line 483, in test_process_time self.assertLess(stop - start, 0.020) AssertionError: 0.06567554000002929 not less than 0.02 ------------------------------------------------------------ 89857 6 yf3yu 2021-04-01 20:19:12 +0000 will send a patch to disable these timing related tests from looking at the stack trace, I don't think they are actual failures, only slightly off from the expected value and the failure is not reproducible locally. 90006 7 yf3yu 2021-04-19 19:12:09 +0000 https://git.openembedded.org/openembedded-core/commit/?id=7c67bc2476b784083acbc7a55ecf3627ec8f2b6b 90010 8 yf3yu 2021-04-19 19:16:56 +0000 the previous commit ripped out the 3 failing tests. 90014 9 randy.macleod 2021-04-19 22:24:58 +0000 Also fixed on 3.3 by: https://git.openembedded.org/openembedded-core/commit/?id=7c67bc2476b784083acbc7a55ecf3627ec8f2b6b $ git tag --contains 7c67bc2476b784083acbc7a55ecf3627ec8f2b6b 2021-04 2021-04-hardknott yocto-3.3 90194 10 alexandre.belloni 2021-04-29 22:19:36 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1696/steps/12/logs/stdio This is still qemuarm64 on ubuntu1804-arm-1. I'm not sure which test failed, the relevant output seems to be: '(test.test_concurrent_futures.ProcessPoolSpawnProcessPoolExecutorTest) ' '... 0.92s ok\n' 'Exception in thread Thread-403:\n' 'Traceback (most recent call last):\n' ' File "/usr/lib/python3.9/threading.py", line 954, in ' '_bootstrap_inner\n' ' self.run()\n' ' File "/usr/lib/python3.9/concurrent/futures/process.py", line ' '317, in run\n' ' result_item, is_broken, cause = ' 'self.wait_result_broken_or_wakeup()\n' ' File "/usr/lib/python3.9/concurrent/futures/process.py", line ' '376, in wait_result_broken_or_wakeup\n' ' worker_sentinels = [p.sentinel for p in ' 'self.processes.values()]\n' ' File "/usr/lib/python3.9/concurrent/futures/process.py", line ' '376, in <listcomp>\n' ' worker_sentinels = [p.sentinel for p in ' 'self.processes.values()]\n' 'RuntimeError: dictionary changed size during iteration\n' '\n' 'ERROR: Exit status is 9\n' 'DURATION: 800\n'} 90198 11 tim.orling 2021-04-29 23:22:43 +0000 Might be similar reason/root cause, but the failing test case--to the best of my ability to find it--is: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/test/test_concurrent_futures.py#L1008 90199 12 tim.orling 2021-04-29 23:28:26 +0000 The key line to me is: 'RuntimeError: dictionary changed size during iteration\n' This implies either a file or data in memory changed during the test execution. 90245 13 alexandre.belloni 2021-05-06 07:58:17 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2056/steps/13/logs/stdio qemux86-64 master-next on centos7-ty-4 90255 14 alexandre.belloni 2021-05-06 11:16:19 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1742/steps/12/logs/stdio qemuarm64 master-next on ubuntu2004-arm-1 90258 15 alexandre.belloni 2021-05-06 11:38:14 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2033/steps/12/logs/stdio qemux86-64 master-next on ubuntu2004-ty-2 90264 16 alexandre.belloni 2021-05-06 12:27:09 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1746/steps/12/logs/stdio qemuarm64 master on ubuntu1804-arm-1 90270 17 alexandre.belloni 2021-05-06 14:37:08 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1756/steps/12/logs/stdio qemuarm64 master-next on ubuntu1804-arm-1 90299 18 randy.macleod 2021-05-06 15:33:23 +0000 http://autobuilder.yocto.io/pub/non-release/20210501-17/testresults/testresult-report.txt testseries | result_id : qemux86-64-ptest | runtime_core-image-sato-sdk-ptest_qemux86-64_20210501191514 ptestresult.python3.test_cancel_futures Alex will go over the recent failures and provide data as to which ptest failed. 90412 19 jon.mason 2021-05-19 00:55:41 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2099/steps/12/logs/stdio qemux86-64-ptest centos8-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2095/steps/12/logs/stdio qemux86-64-ptest centos8-ty-1 90445 20 tvgamblin 2021-05-20 18:55:30 +0000 Latest tests look like OOM failures. From https://autobuilder.yocto.io/pub/non-release/20210516-3/testresults/qemux86-64-ptest/python3.log: ... PASS: test_popen_error_with_stdin_pipe (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_read_stdout_after_process_exit (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_send_signal (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_shell (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_shell_loop_deprecated (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_start_new_session (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_stdin_broken_pipe (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_stdin_not_inheritable (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_stdin_stdout (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) PASS: test_terminate (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) ERROR: Exit status is 9 DURATION: 404 90548 21 flowergom 2021-06-02 02:32:02 +0000 Happend again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1852/steps/12/logs/stdio 90734 22 alexandre.belloni 2021-06-24 11:58:32 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2198/steps/13/logs/stdio {'python3': ['test_shell']} qemux86-64-ptest ubuntu1604-ty-1 90814 23 alexandre.belloni 2021-06-30 23:31:53 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1940/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 90842 24 alexandre.belloni 2021-07-04 13:24:18 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2233/steps/12/logs/stdio qemux86-64-ptest ubuntu1804-ty-1 91225 25 randy.macleod 2021-08-19 15:03:21 +0000 Hasn't happened in months, Trevor will try one more time and then close if he can't reproduce it. 91369 26 randy.macleod 2021-09-09 14:51:16 +0000 Re-open if seen again. 91419 27 alexandre.belloni 2021-09-16 09:02:09 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2130/steps/12/logs/stdio qemuarm64-ptest ubuntu2004-arm-1 91423 28 alexandre.belloni 2021-09-16 09:08:57 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2157/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 91703 29 thomas.perrot 2021-10-11 11:46:58 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2368/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 91704 30 thomas.perrot 2021-10-11 11:52:38 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2656/steps/12/logs/stdio qemux86-64-ptest centos8-ty-1 91741 31 thomas.perrot 2021-10-14 11:19:26 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2396/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'python3': ['test_thread_time'], 'strace': ['count.test']} 91900 32 chee.yang.lee 2021-11-07 05:27:43 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2465/steps/12/logs/stdio {'python3': ['test_control_and_wait']} ubuntu1804-arm-1 qemuarm64-ptest 91918 33 chee.yang.lee 2021-11-09 06:05:41 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2413/steps/12/logs/stdio {'python3': ['test_waitfor_timeout'], 'tcl': ['socket.test']} master next on ubuntu1804-arm-1 qemuarm64-ptest 92025 34 flowergom 2021-11-17 06:24:39 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2549/steps/12/logs/stdio Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 108, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'python3': ['test_thread_time']} Cannot run ptests without @expectedFailure as ptests are expected to fail 92251 35 alexandre.belloni 2022-01-07 16:28:48 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2648/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 93195 36 randy.macleod 2022-04-28 14:38:05 +0000 We have mitigated the load and moved to SSDs and likely upgraded python all of which may have resulted in this failure stopping 4 months ago.