Bug 8750 - Allow oe-selftest to run custom test suites based on different criteria
Summary: Allow oe-selftest to run custom test suites based on different criteria
Status: VERIFIED FIXED
Alias: None
Product: Build Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 2.0
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 2.1 M1
Assignee: Daniel Istrate
QA Contact: Bogdan Alexandru Voiculescu
URL:
Whiteboard:
Depends on:
Blocks: 8647
  Show dependency tree
 
Reported: 2015-12-02 14:33 UTC by Daniel Istrate
Modified: 2016-02-16 10:45 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: New (Never tested)
Verified:
Documentation change: Yes (doc changes required)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Istrate 2015-12-02 14:33:09 UTC 
oe-selftest should be able to run custom test suites based on different criteria like: id (test case id (@testcase)), name (test case name), class, module and tag (@tag(feature=(feature list))).

eg: oe-selftest --run-tests-by tag wic bitake recipetool pr-service
    oe-selftest --run-tests-by name test_x test_y test_z
    oe-selftest --run-tests-by id 1001 1002 185 1902
    oe-selftest --run-tests-by class Signing ImageFeatures
    oe-selftest --run-tests-by module wic signing
Comment 1 Daniel Istrate 2015-12-02 14:43:02 UTC 
Patch sent to ML: 
http://lists.openembedded.org/pipermail/openembedded-core/2015-December/113637.html

This patch covers:

1. Can run custom lists of tests based on different criteria:
   --run-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags>
   eg: --run-tests-by module imagefeatures signing recipetool
       --run-tests-by id 1377 1273 935
       --run-tests-by tag wic sstate bitbake
2. Can list tests based on different criteria:
   --list-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags>
   eg: --list-tests-by module imagefeatures signing recipetool
       --list-tests-by id 1377 1273 935
       --list-tests-by tag wic sstate bitbake
3. Can list all tags that have been set to test cases:
   --list-tags
   The list of tags should be kept as minimal as possible.
   This helps preview the tags used so far.

   To take advantage of the 'tag' feature:
   - add @tag(feature=<>) to testcases
   eg: @tag(feature='signing') for a single tag
       @tag(feature=(('signing', 'sstate')) or
       @tag(feature=['signing', 'sstate']) for multiple tags
Comment 2 Alexandru Georgescu 2015-12-07 08:12:46 UTC 
Moved to M1 since Daniel already submitted a patch for it in M1.
Comment 3 Alexandru Georgescu 2015-12-07 08:13:41 UTC 
Patches submitted.
Comment 4 Daniel Istrate 2015-12-14 07:28:25 UTC 
As of b3d6872bf169a13e572638e3809c50a868246616: patch merged into master.
Comment 5 Alexandru Georgescu 2016-02-16 10:45:00 UTC 
Verified with m2.rc1