| Summary: | AB-INT PTEST ARM: libevent arm ptest intermittent failure | ||
|---|---|---|---|
| Product: | [QA/Testing] Package Testing (ptest) | Reporter: | Naveen Saini <naveen.kumar.saini> |
| Component: | ptest | Assignee: | Ross Burton <ross.burton> |
| Status: | RESOLVED WORKSFORME | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alexandre.belloni, bluelightning, chee.yang.lee, flowergom, jon.mason, leonardo.sandoval, oobitots, randy.macleod, richard.purdie, ross.burton |
| Version: | 3.3 | ||
| Target Milestone: | 4.0 M4 | ||
| Hardware: | x86 | ||
| OS: | arm64 | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
| Bug Depends on: | 14328 | ||
| Bug Blocks: | |||
|
Description
Naveen Saini
2020-12-28 07:38:47 UTC
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1517/steps/12/logs/stdio WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'libevent': ['[no_events_FAILED]']} Another event on this failure in the AB [1]:
WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
self.do_ptestrunner()
File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
self.fail(failmsg)
AssertionError: Failed ptests:
{'libevent': ['[loopexit_multiple_FAILED]', '[bufferevent_timeout_FAILED]']}
[1] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1532/steps/12/logs/stdio
So far, it only happened on ubuntu1804-arm-1 qemuarm64-ptest ubuntu1804-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1653 Bitbake still alive (5000s) Bitbake still alive (10000s) WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'libevent': ['[persistent_active_timeout_FAILED]']} Similar: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1661/steps/12/logs/stdio WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail self.do_ptestrunner() File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner self.fail(failmsg) AssertionError: Failed ptests: {'libevent': ['[bufferevent_timeout_filter_pair_FAILED]']} Happened on master-next (3.3):
machine: qemuarm64
worker: ubuntu1804-arm-1
AssertionError: Failed ptests:
{'libevent': ['[common_timeout_FAILED]'] ...}
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1699
Ok will likely have to disable these timeout tests until the intermittent root cause is fixed adding 14328 to depends. looking at the source code of the tests, common_timeout generates 100 events that need to timeout to be accurate to +/-50ms obviously this is highly dependent on scheduling and the current load. increasing to a more acceptable +100ms [oe-core][PATCH] libevent: Increase ptest timing tolerance 50 ms -> 100 ms in the mailing list https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1749/steps/12/logs/stdio 'libevent': ['[usleep_FAILED]'] qemuarm64 master-next on ubuntu1804-arm-1 qemuarm64
ubuntu1804-arm-1
{'libevent': ['[common_timeout_FAILED]']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1773/steps/12/logs/stdio
Only happening on arm builders. Can the test be adjusted again or skipped? We think that the latest failure happen after the commit was merged but we're not 100% certain. Let's wait for a month to see if it happens again. I confirm it did https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1935/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1959/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1952/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 'libevent': ['[monotonic_res_FAILED]'] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1975/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2085/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 *** Bug 14549 has been marked as a duplicate of this bug. *** 'libevent': ['[conditions_simple_FAILED]'] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2223/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 'libevent': ['[persistent_timeout_jump_FAILED]'] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2229/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2279/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 'libevent': ['[persistent_active_timeout_FAILED]'] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2258/steps/12/logs/stdio qemuarm64-ptest ubuntu1804-arm-1 'libevent': ['[conditions_simple_FAILED]'], Yi Fan, we'd like Ross to work on this since it's happening on arm. PASS: thread/forking thread/conditions_simple: [forking] FAIL ../libevent-2.1.12-stable/test/regress_thread.c:378 FAIL: [conditions_simple FAILED] [RETRYING conditions_simple (3)] PASS: thread/conditions_simple Hasn't happened in months, perhaps due to: https://git.openembedded.org/openembedded-core/commit/?id=cae0eb03e8e754bd77651fc017b2669f8fd57f75 |