Bug 7849 - Filter proper test cases from the test case manifest file and run the filtered ones
Summary: Filter proper test cases from the test case manifest file and run the filtere...
Status: VERIFIED FIXED
Alias: None
Product: Runtime Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 1.8.1
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 1.9 M3
Assignee: Costin
QA Contact: Cathy Shen
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-06-05 03:46 UTC by Cathy Shen
Modified: 2015-09-14 11:22 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Cathy Shen 2015-06-05 03:46:30 UTC
There is request to run specific test cases at runtime based on user input, since for different test cycle, test set is different. E.g. run priority 1 cases only, run cases in a specific component only , run cases with specific tag, etc.

We'll have patch to Yocto to support this.
Comment 1 zjh 2015-06-29 08:30:59 UTC
Test runner can provide a decorator: addtag. when a case was decorated by this decorator, a tag will be added to this case. then test runner will filter by these tags to run specific cases:
for example:
#foo.py
from oeqa.oetest import oeRuntimeTest
from oeqa.utils.decorators import addtag

@addtag("class")
class TestClass(oeRuntimeTest):
    @addtag("foo")
    def testfoo(self):
	    print "testfoo"

	@addtag("bar1","bar2")
	def testbar(self):
	    print "testbar"

	@addtag("bar1")
	def testbaz(self):
	    print "testbaz"

If user use a tag into conf/local.conf: TEST_SUITES_TAGS = "class", runner will run all test cases: testfoo, testbar, testbaz.
If user use a tag: TEST_SUITES_TAGS = "foo", runner will run test cases: testfoo.
If user use a tag: TEST_SUITES_TAGS = "foo bar1", runner will run two test cases: testfoo,testbar,testbaz.
If user use a tag: TEST_SUITES_TAGS = "bar1&bar2", runner will run test case: testbar.
A test case can add many tags, and user also can use many tags to filter test cases.
If run testcases by an exported test, use -a or --tag to add filter tag(e.g. -a "foo bar1&bar2").
Comment 2 Costin 2015-07-09 10:54:31 UTC
Hi Junhui,

The idea looks good.

Costin
Comment 3 zjh 2015-08-26 09:31:22 UTC
To support complicated situation, we change TEST_SUITES_TAGS is a python expression, It means that multiple tags like "foo bar1" should be "foo or bar1", or is a python keyword.
details:
Test runner provides a decorator: tag. when a case was decorated by this decorator, tags will be added to this case. then test runner will filter cases by these tags. Runner follows below rules:
1. tags can be added to a test class or a test method, test method will inherit tags form its class, also can overwrite those tags.
2. tags can be valued, if not valued, the runner will give it a default value: True
3. test runner follows python expression, when this expression's result is True for a test case, this case will be executed.
4. If the case doesn't have a specific tag, this variable is None.

Some examples:
#foo.py
from oeqa.oetest import oeRuntimeTest
from oeqa.utils.helper import tag
@tag(TestType="Functional Positive")
class TestClass(oeRuntimeTest):
    @tag("foo", TestType="Stability", FeatureID="IOTOS-211", debug=5)
    def testfoo(self):
         print "testfoo"
    @tag("bar", FeatureID="IOTOS-212", debug=10)
    def testbar(self):
         print "testbar"
    @tag("baz", FeatureID="IOTOS-213", debug=15)
    def testbaz(self):
         print "testbaz"
If user use a tag into conf/local.conf: TEST_SUITES_TAGS = "TestType == 'Functional Positive'", runner will run two test cases: testbar, testbaz.
If user use a tag: TEST_SUITES_TAGS = "foo", runner will run test cases: testfoo.
If user use a tag: TEST_SUITES_TAGS = "debug < 11", runner will run two test cases: testfoo,testbar.
If user use a tag: TEST_SUITES_TAGS = "TestType == 'Functional Positive' and debug <= 10", runner will run test cases: testbar.
Comment 4 zjh 2015-09-09 07:26:45 UTC
This patch is merged on poky.
Comment 5 Costin 2015-09-14 11:22:20 UTC
Hello,

Verified with 75bad1b64bd485a50af69ed1f8663a0dadbb5f9e
patch was merged with: 	fac0d67ec25f357684bcd0c80be282e1bca3adef
Comment 6 Costin 2015-09-14 11:22:56 UTC
marking as verified