Bug 14163

Summary: AB-INT PTEST ARM: libevent arm ptest intermittent failure
Product: [QA/Testing] Package Testing (ptest) Reporter: Naveen Saini <naveen.kumar.saini>
Component: ptestAssignee: Ross Burton <ross.burton>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium+ CC: alexandre.belloni, bluelightning, chee.yang.lee, flowergom, jon.mason, leonardo.sandoval, oobitots, randy.macleod, richard.purdie, ross.burton
Version: 3.3   
Target Milestone: 4.0 M4   
Hardware: x86   
OS: arm64   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on: 14328    
Bug Blocks:    

Description Naveen Saini 2020-12-28 07:38:47 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1364

AssertionError: Failed ptests:
{'libevent': ['[persistent_active_timeout_FAILED]']}
Comment 1 Minjae Kim 2021-02-11 13:44:17 UTC
Happened again:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1517/steps/12/logs/stdio

WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
    self.fail(failmsg)
AssertionError: Failed ptests:
{'libevent': ['[no_events_FAILED]']}
Comment 2 Leonardo Sandoval 2021-02-23 15:40:19 UTC
Another event on this failure in the AB [1]:

WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
    self.fail(failmsg)
AssertionError: Failed ptests:
{'libevent': ['[loopexit_multiple_FAILED]', '[bufferevent_timeout_FAILED]']}

[1] https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1532/steps/12/logs/stdio
Comment 3 Alexandre Belloni 2021-03-11 14:19:24 UTC
So far, it only happened on ubuntu1804-arm-1
Comment 4 Oleksiy Obitotskyy 2021-03-25 10:56:53 UTC
qemuarm64-ptest ubuntu1804-arm-1

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1653

Bitbake still alive (5000s)
Bitbake still alive (10000s)
WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
    self.fail(failmsg)
AssertionError: Failed ptests:
{'libevent': ['[persistent_active_timeout_FAILED]']}
Comment 5 Paul Eggleton 2021-03-29 03:16:28 UTC
Similar:

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1661/steps/12/logs/stdio

WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
    self.fail(failmsg)
AssertionError: Failed ptests:
{'libevent': ['[bufferevent_timeout_filter_pair_FAILED]']}
Comment 6 Naveen Saini 2021-04-19 08:32:06 UTC
Happened on master-next (3.3):
machine: qemuarm64
worker: ubuntu1804-arm-1

AssertionError: Failed ptests:
{'libevent': ['[common_timeout_FAILED]'] ...}

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1699
Comment 7 Yi Fan Yu 2021-04-20 20:02:41 UTC
Ok will likely have to disable these timeout tests until the intermittent root cause is fixed

adding 14328 to depends.
Comment 8 Yi Fan Yu 2021-04-23 02:40:46 UTC
looking at the source code of the tests,
common_timeout generates 100 events that need to timeout to be accurate to +/-50ms

obviously this is highly dependent on scheduling and the current load.

increasing to a more acceptable +100ms
Comment 9 Yi Fan Yu 2021-04-23 15:05:12 UTC
[oe-core][PATCH] libevent: Increase ptest timing tolerance 50 ms -> 100 ms
in the mailing list
Comment 10 Alexandre Belloni 2021-05-06 14:44:50 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1749/steps/12/logs/stdio

'libevent': ['[usleep_FAILED]']

qemuarm64
master-next on ubuntu1804-arm-1
Comment 11 Chee Yang 2021-05-09 11:50:50 UTC
qemuarm64
ubuntu1804-arm-1
{'libevent': ['[common_timeout_FAILED]']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1773/steps/12/logs/stdio
Comment 12 Randy MacLeod 2021-05-20 15:26:58 UTC
Only happening on arm builders.

Can the test be adjusted again or skipped?
We think that the latest failure happen after the commit was merged but we're not 100% certain.

Let's wait for a month to see if it happens again.
Comment 13 Alexandre Belloni 2021-05-20 15:47:45 UTC
I confirm it did
Comment 14 Alexandre Belloni 2021-06-30 23:01:59 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1935/steps/12/logs/stdio

qemuarm64-ptest ubuntu1804-arm-1
Comment 15 Alexandre Belloni 2021-07-04 13:44:20 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1959/steps/12/logs/stdio

qemuarm64-ptest ubuntu1804-arm-1
Comment 16 Alexandre Belloni 2021-07-04 13:53:33 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1952/steps/12/logs/stdio

qemuarm64-ptest ubuntu1804-arm-1
Comment 17 Alexandre Belloni 2021-07-15 13:34:13 UTC
'libevent': ['[monotonic_res_FAILED]']

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1975/steps/12/logs/stdio
Comment 18 Alexandre Belloni 2021-08-13 12:50:19 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2085/steps/12/logs/stdio

qemuarm64-ptest ubuntu1804-arm-1
Comment 19 Ross Burton 2021-09-16 14:43:02 UTC
*** Bug 14549 has been marked as a duplicate of this bug. ***
Comment 20 Ross Burton 2021-09-16 14:43:24 UTC
'libevent': ['[conditions_simple_FAILED]']

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2223/steps/12/logs/stdio
qemuarm64-ptest ubuntu1804-arm-1




'libevent': ['[persistent_timeout_jump_FAILED]']

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2229/steps/12/logs/stdio
qemuarm64-ptest ubuntu1804-arm-1
Comment 21 Alexandre Belloni 2021-09-23 21:25:58 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2279/steps/12/logs/stdio
qemuarm64-ptest ubuntu1804-arm-1


'libevent': ['[persistent_active_timeout_FAILED]']
Comment 22 Alexandre Belloni 2021-09-23 21:38:12 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2258/steps/12/logs/stdio
qemuarm64-ptest ubuntu1804-arm-1


'libevent': ['[conditions_simple_FAILED]'],
Comment 23 Randy MacLeod 2021-09-30 15:11:51 UTC
Yi Fan, we'd like Ross to work on this since it's happening on arm.
Comment 24 Ross Burton 2021-10-07 12:44:08 UTC
PASS: thread/forking
thread/conditions_simple: [forking]
  FAIL ../libevent-2.1.12-stable/test/regress_thread.c:378
FAIL:   [conditions_simple FAILED]

  [RETRYING conditions_simple (3)]
PASS: thread/conditions_simple
Comment 25 Randy MacLeod 2022-03-31 14:49:48 UTC
Hasn't happened in months, perhaps due to:
https://git.openembedded.org/openembedded-core/commit/?id=cae0eb03e8e754bd77651fc017b2669f8fd57f75