Bug 13368 - mdadm ptest times out
Summary: mdadm ptest times out
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 2.7
Hardware: x86 Multiple
: High normal
Target Milestone: 2.8 M3
Assignee: Changqing Li
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-05-29 15:57 UTC by Richard Purdie
Modified: 2019-09-09 03:27 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2019-05-29 15:57:39 UTC
The ptests for mdadm are timing out rather than running:

https://autobuilder.yocto.io/pub/non-release/20190529-5/testresults/testresult-report.txt

I confirmed from the commandline that the tests do just hang, reason is unknown, its also not known when this started happening. I ruled out gcc8 verses 9 though.
Comment 1 Sangeeta Jain 2019-07-05 06:10:05 UTC
The ptests for mdadm are timing out in 2.7.1 rc1 as well. 

https://autobuilder.yocto.io/pub/releases/yocto-2.7.1.rc1/testresults/testresult-report.txt
Comment 2 Mingli Yu 2019-07-10 09:15:22 UTC
The below two cases make mdadm ptest hang:

/usr/lib64/mdadm/ptest/tests/01r5integ

/usr/lib64/mdadm/ptest/tests/01raid6integ


Both cases have below logic to wait for the array to be fully optimal before proceeding.
while ! (check state 'U*'); do check wait; sleep 0.2; done

BTW, mdadm has lower pass rate in qemu env, it's better to run the mdadm test in the real board.
Comment 4 yeoh ee peng 2019-08-01 02:05:18 UTC
QA team found mdadm timeout in qemuarm64 testresults below:
https://autobuilder.yocto.io/pub/releases/yocto-2.8_M2.rc1/testresults/qemuarm64-ptest/testresults.json

Log:
PASS: /usr/lib/mdadm/ptest/tests/10ddf-sudden-degraded
ERROR: Exit status is 9
DURATION: 736
Comment 5 Mingli Yu 2019-08-08 09:41:43 UTC
seems the test 11spare-migration hangs on qemuarm64.
Comment 6 Randy MacLeod 2019-08-08 15:03:13 UTC
Grace, so are you going to debug the hang?
Comment 7 Randy MacLeod 2019-08-22 14:48:25 UTC
Any news Sandy?
Comment 8 Changqing Li 2019-08-30 06:51:07 UTC
I test on qemux86-64 and qemuarm64,   mdadm timeouted with default cmd: 
ptest-runner mdadm.


But it can completed if I set timeout longger:

ptest-runner mdadm -t 6000


Below is result on qemuarm64:
....
FAIL: /usr/lib/mdadm/ptest/tests/19repair-does-not-destroyFAILED - see //19repair-does-not-destroy.log and //fail19repair-does-not-destroy.log for details
FAIL: /usr/lib/mdadm/ptest/tests/20raid5journalFAILED - see //20raid5journal.log and //fail20raid5journal.log for details
FAIL: /usr/lib/mdadm/ptest/tests/21raid5cacheFAILED - see //21raid5cache.log and //fail21raid5cache.log for details
DURATION: 1788
END: /usr/lib/mdadm/ptest
2019-08-30T06:19
STOP: ptest-runner

@Randy @Richard
As mingli also said before, "mdadm has lower pass rate in qemu env, it's better to run the mdadm test in the real board.", do you really need this test in PTESTS_SLOW?  I am wondering if we only skip slow subcase or remove the whole mdadm test.

if only skip subcase, I will skip 11spare-migration, I  have test on qemuarm64, skip this one can make mdadm test not timeout
Comment 9 Randy MacLeod 2019-09-03 18:26:13 UTC
I don't object to skipping the slow test for 3.0 so you could send that for now.
Best if you briefly explain what the 11spare-migration test is doing and report on how long it takes under qemuarm64.

It would be nice to understand why it's so slow in case that reveals deeper problems. You could run the one test under [sl]trace -o /tmp/mdadm.trc -f -ttt ... to learn more today or under this defect during 3.1.
Comment 10 Randy MacLeod 2019-09-05 14:47:06 UTC
Please skip the test ASAP for M3 and we can downgrade the investigation to not block M3.