Bug 13067

Summary: setupClass failures not being handled correcting for parallel oe-selftest
Product: [QA/Testing] Functional (self) Testing Reporter: Richard Purdie <richard.purdie>
Component: oe-selftestAssignee: Chen Qi <Qi.Chen>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: randy.macleod, ross.burton, tvgamblin
Version: unspecified   
Target Milestone: 3.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2018-12-07 12:51:20 UTC
If we run oe-selftest -r and there is a failure in setupClass, it doesn't get reported to the user clearly. A good example is this:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6f2ef620d90ab39870bb6c02183249e0e1045aeb

which shows a clear error without the -j options but with it, it only shows:

2018-12-05 18:43:24,262 - oe-selftest - INFO - RESULTS - eSDK.oeSDKExtSelfTest.test_image_generation_binary_feeds - Testcase 1603: UNKNOWN
2018-12-05 18:43:24,262 - oe-selftest - INFO - RESULTS - eSDK.oeSDKExtSelfTest.test_install_libraries_headers - Testcase 1602: UNKNOWN

(from https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/55/steps/7/logs/step2d)
Comment 1 Richard Purdie 2018-12-07 12:53:21 UTC
I should note the error is shown in the log but the summary at the end is wrong:

All required tests passed (successes=334, skipped=2, failures=0, errors=0)
Comment 2 Richard Purdie 2018-12-07 17:50:23 UTC
Another example: https://autobuilder.yoctoproject.org/typhoon/#/builders/84/builds/61
Comment 3 Randy MacLeod 2019-11-19 16:30:22 UTC
It's good to get selftests working reliably sooner rather than later so target milestone is 3.1-M1.
Comment 4 Richard Purdie 2020-03-08 10:51:47 UTC
I tested and confirmed that Trevor's recent work did also fix this issue and its no longer present.