| Summary: | Filter proper test cases from the test case manifest file and run the filtered ones | ||
|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Cathy Shen <cathy.shen> |
| Component: | general | Assignee: | Costin <costin.c.constantin> |
| Status: | VERIFIED FIXED | QA Contact: | Cathy Shen <cathy.shen> |
| Severity: | enhancement | ||
| Priority: | Medium+ | CC: | alexandru.c.georgescu, junhuix.zhang |
| Version: | 1.8.1 | ||
| Target Milestone: | 1.9 M3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
|
Description
Cathy Shen
2015-06-05 03:46:30 UTC
Test runner can provide a decorator: addtag. when a case was decorated by this decorator, a tag will be added to this case. then test runner will filter by these tags to run specific cases:
for example:
#foo.py
from oeqa.oetest import oeRuntimeTest
from oeqa.utils.decorators import addtag
@addtag("class")
class TestClass(oeRuntimeTest):
@addtag("foo")
def testfoo(self):
print "testfoo"
@addtag("bar1","bar2")
def testbar(self):
print "testbar"
@addtag("bar1")
def testbaz(self):
print "testbaz"
If user use a tag into conf/local.conf: TEST_SUITES_TAGS = "class", runner will run all test cases: testfoo, testbar, testbaz.
If user use a tag: TEST_SUITES_TAGS = "foo", runner will run test cases: testfoo.
If user use a tag: TEST_SUITES_TAGS = "foo bar1", runner will run two test cases: testfoo,testbar,testbaz.
If user use a tag: TEST_SUITES_TAGS = "bar1&bar2", runner will run test case: testbar.
A test case can add many tags, and user also can use many tags to filter test cases.
If run testcases by an exported test, use -a or --tag to add filter tag(e.g. -a "foo bar1&bar2").
Hi Junhui, The idea looks good. Costin To support complicated situation, we change TEST_SUITES_TAGS is a python expression, It means that multiple tags like "foo bar1" should be "foo or bar1", or is a python keyword.
details:
Test runner provides a decorator: tag. when a case was decorated by this decorator, tags will be added to this case. then test runner will filter cases by these tags. Runner follows below rules:
1. tags can be added to a test class or a test method, test method will inherit tags form its class, also can overwrite those tags.
2. tags can be valued, if not valued, the runner will give it a default value: True
3. test runner follows python expression, when this expression's result is True for a test case, this case will be executed.
4. If the case doesn't have a specific tag, this variable is None.
Some examples:
#foo.py
from oeqa.oetest import oeRuntimeTest
from oeqa.utils.helper import tag
@tag(TestType="Functional Positive")
class TestClass(oeRuntimeTest):
@tag("foo", TestType="Stability", FeatureID="IOTOS-211", debug=5)
def testfoo(self):
print "testfoo"
@tag("bar", FeatureID="IOTOS-212", debug=10)
def testbar(self):
print "testbar"
@tag("baz", FeatureID="IOTOS-213", debug=15)
def testbaz(self):
print "testbaz"
If user use a tag into conf/local.conf: TEST_SUITES_TAGS = "TestType == 'Functional Positive'", runner will run two test cases: testbar, testbaz.
If user use a tag: TEST_SUITES_TAGS = "foo", runner will run test cases: testfoo.
If user use a tag: TEST_SUITES_TAGS = "debug < 11", runner will run two test cases: testfoo,testbar.
If user use a tag: TEST_SUITES_TAGS = "TestType == 'Functional Positive' and debug <= 10", runner will run test cases: testbar.
This patch is merged on poky. Hello, Verified with 75bad1b64bd485a50af69ed1f8663a0dadbb5f9e patch was merged with: fac0d67ec25f357684bcd0c80be282e1bca3adef marking as verified |