Bug 9129

Summary: Enhance Test Suite scanning and execution
Product: [QA/Testing] Test Plans/Suite Reporter: Alex Franco <alejandro.franco>
Component: test-plans/suiteAssignee: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Status: RESOLVED DUPLICATE QA Contact:
Severity: enhancement    
Priority: Medium CC: benjamin.esquivel, corneliux.stoicescu, leonardo.sandoval.gonzalez
Version: unspecified   
Target Milestone: 2.3   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Bug Depends on: 8410    
Bug Blocks:    

Description Alex Franco 2016-02-17 18:17:21 UTC 
Derived from bug 8410

1.- The usage of "auto" in TEST_SUITES.
    When "auto" is listed in TEST_SUITES, a function ran after do_testimage scans the directories '<layer>/lib/oeqa/runtime/', and any tests found are added to the test run list, in whatever order they are found.

2.- How tests are skipped
    Once the tests in TEST_SUITES and the ones scanned if "auto" was present are in the run test list, they are ran one at a time, in order of appearance. First all the ones in TEST_SUITES in the order they were added there, then the ones scanned if "auto" was present, in order of scan.

If any of these tests should be skipped, then those tests themselves will issue their own skip action, either using unittest.skipTest or any of the skip decorators specified in utils/decorators.py, like skipUnlessPassed. These decorators only work if the test list is ordered by dependencies, which is not the case sometimes, and results in errors.

Proposed solution:

Improving how test suites are scanned (2) in order to have an *ordered* test list should guarantee (3) works as designed.
Comment 1 Alex Franco 2016-02-17 20:43:08 UTC 
In short, the goal of this is making sure testimage test suite dependencies are met when testing
Comment 2 Benjamin Esquivel 2016-02-17 20:48:20 UTC 
correct, because currently the dependencies are not guaranteed to run in order.
Comment 3 Leonardo Sandoval Gonzalez 2016-05-25 20:46:00 UTC 
Due to limited resources, moving this to 2.3
Comment 4 Leonardo Sandoval Gonzalez 2016-10-27 21:17:16 UTC 
This work is covered by the new implementation of oeqa, called oeqa2, so marking as duplicate.

*** This bug has been marked as a duplicate of bug 10230 ***