| Summary: | AB-INT PTEST: tcl ptest intermittent failure | ||
|---|---|---|---|
| Product: | [QA/Testing] Package Testing (ptest) | Reporter: | Naveen Saini <naveen.kumar.saini> |
| Component: | ptest | Assignee: | Ross Burton <ross.burton> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alexandre.belloni, bluelightning, chee.yang.lee, flowergom, jon.mason, kory.maincent, leonardo.sandoval, oobitots, randy.macleod, richard.purdie, tim.orling |
| Version: | 3.3 | ||
| Target Milestone: | 3.4 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
Grace, please work on this in the coming weeks. I didn't reproduce the issue locally. The autobuilder log doesn't seem to be of much help. Here's the log:
--------------------------------------------------------------------------------------------------------------------------------------------------------
NOTE: Running task 910 of 910 (/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb:do_testimage)
NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Started
Bitbake still alive (5000s)
WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
self.do_ptestrunner()
File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
self.fail(failmsg)
AssertionError: Failed ptests:
{'tcl': ['socket.test']}
Cannot run ptests without @expectedFailure as ptests are expected to fail
RESULTS:
RESULTS - ping.PingTest.test_ping: PASSED (0.07s)
RESULTS - ssh.SSHTest.test_ssh: PASSED (0.23s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: EXPECTEDFAIL (9679.28s)
SUMMARY:
core-image-sato-sdk-ptest () - Ran 4 tests in 9679.593s
core-image-sato-sdk-ptest - OK - All required tests passed (successes=2, skipped=1, failures=0, errors=0)
NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Succeeded
NOTE: Tasks Summary: Attempted 910 tasks of which 909 didn't need to be rerun and all succeeded.
Summary: There was 1 WARNING message shown.
/usr/lib64/python3.8/subprocess.py:844: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
self.stdout = io.open(c2pread, 'rb', bufsize)
program finished with exit code 0
elapsedTime=9726.330243
--------------------------------------------------------------------------------------------------------------------------------------------------------
the socket.test is doing quite a bit, can you add some logging so that we can find out what part of the tests is failing should it happen again?
We do have access to the autobuilder so we can run tests overnight and we can manually collect logs if needed.
Did you try to do a separate build and/or use stress-ng to generate high cpu/io load while the tests were running?
Another event of the same problem https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1560/steps/12/logs/stdio (In reply to comment #4) > Another event of the same problem > > https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1560/steps/ > 12/logs/stdio Another from the same AB collection https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1842/steps/12/logs/stdio Hi Leonardo, The link you posted is not the failed log for tcl and it should be as expected at some point. BTW, I couldn't reproduce the tcl ptest failure locally. @Mingli, you are right. Somehow, I through these were the same issues. @Mingli, Please add some logging so that we can find out what part of the tests is failing should it happen again. Richard is happy to add such changes to the master-next branch to improve the test results on the autobuilders. An other tcl ptest failed:
AssertionError: Failed ptests:
{'tcl': ['interp.test']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1531/steps/12/logs/stdio
'tcl': ['interp.test'] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1550/steps/12/l ogs/stdio So far, happened: 3 times on ubuntu1804-arm-1 1 time on fedora-ty-1 1 time on ubuntu1804-ty-1 1 time on tumbleweed-ty-3 AssertionError: Failed ptests:
{'tcl': ['timer.test']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1615/steps/12/logs/stdio
Grace, can you send a patch to either increase the timeout or remove the timer test? @Randy,
I'm still looking into this issue.
Currently there are three cases failed as the autobuilder ever reported:
{'tcl': ['socket.test']}
{'tcl': ['interp.test']}
{'tcl': ['timer.test']}
*** Bug 14312 has been marked as a duplicate of this bug. *** qemuarm64-ptest ubuntu1804-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1652 NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Started Bitbake still alive (5000s) Bitbake still alive (10000s) WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'tcl': ['chanio.test', 'interp.test'], 'valgrind': ['callgrind/tests/simwork-branch']} Cannot run ptests without @expectedFailure as ptests are expected to fail RESULTS: RESULTS - ping.PingTest.test_ping: PASSED (0.09s) RESULTS - ssh.SSHTest.test_ssh: PASSED (0.20s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: EXPECTEDFAIL (12885.39s) SUMMARY: core-image-sato-sdk-ptest () - Ran 4 tests in 12885.711s core-image-sato-sdk-ptest - OK - All required tests passed (successes=2, skipped=1, failures=0, errors=0) NOTE: recipe core-image-sato-sdk-ptest-1.0-r0: task do_testimage: Succeeded NOTE: Tasks Summary: Attempted 901 tasks of which 900 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message shown. tcl ptest failed again (qemuarm64):
AssertionError: Failed ptests:
{'tcl': ['interp.test']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1704
Happened again on master-next (qemuarm64):
AssertionError: Failed ptests:
{'tcl': ['interp.test']}
Worker: ubuntu1804-arm-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1711
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2088/steps/12/logs/stdio {'tcl': ['interp.test']} debian10-ty-1 qemux86-64-ptest https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1805/steps/12/logs/stdio {'tcl': ['interp.test']} qemuarm64-ptest ubuntu1804-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1812/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1825/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 Happend again:https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1668/steps/12/logs/stdio Happend again: tcl ptest failure on qemux86-64-ptest https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2141/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2193/steps/12/logs/stdio qemux86-64-ptest opensuse152-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1946/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 Got a patch for interp at least. Hopefully that helps that test case specifically, and if it does the others can have a similar approach taken. At least some of the test cases have changed since this issue was first opened so may not be as prone to timing. Closing as the patches and landed and hopefully they fix the issues. Re-open this or new tickets for future tcl issues. |
tcl ptest failure on qemux86-64 machine (master-next) AssertionError: Failed ptests: {'tcl': ['socket.test']} https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1654