Bug 14388

Summary: AB-INT PTEST: valgrind failed helgrind/tests/hg05_race2
Product: [QA/Testing] Package Testing (ptest) Reporter: Chee Yang <chee.yang.lee>
Component: ptestAssignee: Randy MacLeod <randy.macleod>
Status: RESOLVED WONTFIX QA Contact:
Severity: normal    
Priority: Medium+ CC: alexandre.belloni, jay.shen.teoh, jon.mason, leonardo.sandoval, meta.mr.watcher, meta.watcher, oobitots, randy.macleod, richard.purdie
Version: unspecified   
Target Milestone: ---   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Chee Yang 2021-05-10 03:21:22 UTC
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
    self.fail(failmsg)
AssertionError: Failed ptests:
{'valgrind': ['helgrind/tests/hg05_race2']}

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2069/steps/12/logs/stdio
Comment 3 Chee Yang 2021-05-12 12:15:39 UTC
another occurance with different test
{'valgrind': ['helgrind/tests/tc09_bad_unlock']}
qemux86-64-ptest
ubuntu2004-ty-2

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2080/steps/12/logs/stdio
Comment 4 Jon Mason 2021-05-19 01:25:14 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2096/steps/12/logs/stdio

qemux86-64-ptest
tumbleweed-ty-1
Comment 5 Alexandre Belloni 2021-05-27 13:01:06 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2112/

qemux86-64-ptest
tumbleweed-ty-3
Comment 6 Tony Tascioglu 2021-05-31 16:33:31 UTC
(In reply to comment #5)
> https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2112/
> 
> qemux86-64-ptest
> tumbleweed-ty-3

Note: that link is incomplete, the build log is available at https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2112/steps/12/logs/stdio

Interestingly, it's not just the hg05 test failing this time, I'll look into the drd test as well to see if there's a similarity.
Comment 7 Tony Tascioglu 2021-05-31 16:40:37 UTC
I have contacted upstream about the failing hg05_race2 test, and it looks like it has been happening for a while.

Here is the original upstream report: https://bugs.kde.org/show_bug.cgi?id=345121

I have been able to reproduce the failing test for both hg05 and tc09, running it with their testing harness usually causes it to fail within a few runs, and it consistently produces the invalid output if run using strace -f. Both of the tests fail with outputs differing by 1/2 lines.

The diffs for the invalid output from qemu, an Intel desktop and my Ryzen laptop fail in the same way, so I don't think it's specific to our build process.
That being said, it does have to do with the ordering of threads required for an unwinding step them as per upstream.

I'll see if I can come up with a solution or workaround to stop the failing builds.
Comment 8 Alexandre Belloni 2021-06-16 00:59:01 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2169/steps/12/logs/stdio

qemux86-64-ptest
tumbleweed-ty-3
Comment 9 Alexandre Belloni 2021-07-16 21:56:17 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2274/steps/12/logs/stdio

qemux86-64-ptest debian10-ty-2
Comment 10 Alexandre Belloni 2021-07-22 13:18:59 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2279/steps/12/logs/stdio

qemux86-64-ptest debian10-ty-2

'valgrind': ['helgrind/tests/hg05_race2', 'helgrind/tests/tc09_bad_unlock']
Comment 11 Teoh Jay Shen 2021-08-02 00:10:43 UTC
These two ptests were also failing on QA for yocto-3.3.2.rc2 : 

1. ptestresult.valgrind.helgrind/tests/hg05_race2: PASSED -> FAILED
2. ptestresult.valgrind.helgrind/tests/tc09_bad_unlock: PASSED -> FAILED
Comment 12 Randy MacLeod 2021-08-05 14:44:03 UTC
backport taskset work-around to 3.3 please Tony.
Comment 13 Tony Tascioglu 2021-08-20 13:22:11 UTC
(In reply to comment #12)
> backport taskset work-around to 3.3 please Tony.

The change was already backported to Hardknott (3.3) some time ago:

http://cgit.openembedded.org/openembedded-core/commit/meta/recipes-devtools/valgrind/valgrind/?h=hardknott&id=79ec1d73a107277586d3d8e9c0d46dfc0ac2b0d8

http://cgit.openembedded.org/openembedded-core/commit/meta/recipes-devtools/valgrind/valgrind/?h=hardknott&id=f076edb7515ba2ecfc0adbfdf30ae5a9aa96e231

It's not perfect, and the tests do still fail occasionally, but it does seem to improve the situation.
Comment 14 Chee Yang 2021-11-06 03:41:01 UTC
hardknott

{'valgrind': ['helgrind/tests/hg05_race2', 'helgrind/tests/tc09_bad_unlock']}

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2786/steps/12/logs/stdio
Comment 15 Randy MacLeod 2022-01-06 15:53:22 UTC
need to backport the taskset depenedency to previous releases.
Comment 16 Oleksiy Obitotskyy 2022-02-21 09:52:36 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/3133/steps/13/logs/stdio

qemux86-64-ptest ubuntu1604-ty-1

WARNING: core-image-sato-sdk-ptest-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 25, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 112, in do_ptestrunner
    self.fail(failmsg)
AssertionError: Failed ptests:
{'valgrind': ['helgrind/tests/hg05_race2', 'helgrind/tests/tc09_bad_unlock']}
Comment 17 Leonardo Sandoval 2022-04-20 22:44:17 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/3401/steps/12/logs/stdio

│ ubuntu1604-ty-1  │ Non-SSD │
Comment 18 Richard Purdie 2022-04-21 06:53:22 UTC
(In reply to comment #17)
> https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/3401/steps/
> 12/logs/stdio
> 
> │ ubuntu1604-ty-1  │ Non-SSD │

Just to note, the above was hardknott
Comment 19 Randy MacLeod 2022-04-23 00:34:44 UTC
hardknott is at EOL and I think this particualar issue is worked-around in later branches.