Bug 13067 - setupClass failures not being handled correcting for parallel oe-selftest
Summary: setupClass failures not being handled correcting for parallel oe-selftest
Status: RESOLVED FIXED
Alias: None
Product: Functional (self) Testing
Classification: QA/Testing
Component: oe-selftest (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.1 M3
Assignee: Chen Qi
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2018-12-07 12:51 UTC by Richard Purdie
Modified: 2020-03-08 10:51 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2018-12-07 12:51:20 UTC
If we run oe-selftest -r and there is a failure in setupClass, it doesn't get reported to the user clearly. A good example is this:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6f2ef620d90ab39870bb6c02183249e0e1045aeb

which shows a clear error without the -j options but with it, it only shows:

2018-12-05 18:43:24,262 - oe-selftest - INFO - RESULTS - eSDK.oeSDKExtSelfTest.test_image_generation_binary_feeds - Testcase 1603: UNKNOWN
2018-12-05 18:43:24,262 - oe-selftest - INFO - RESULTS - eSDK.oeSDKExtSelfTest.test_install_libraries_headers - Testcase 1602: UNKNOWN

(from https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/55/steps/7/logs/step2d)
Comment 1 Richard Purdie 2018-12-07 12:53:21 UTC
I should note the error is shown in the log but the summary at the end is wrong:

All required tests passed (successes=334, skipped=2, failures=0, errors=0)
Comment 2 Richard Purdie 2018-12-07 17:50:23 UTC
Another example: https://autobuilder.yoctoproject.org/typhoon/#/builders/84/builds/61
Comment 3 Randy MacLeod 2019-11-19 16:30:22 UTC
It's good to get selftests working reliably sooner rather than later so target milestone is 3.1-M1.
Comment 4 Richard Purdie 2020-03-08 10:51:47 UTC
I tested and confirmed that Trevor's recent work did also fix this issue and its no longer present.