| Summary: | Code coverage: add ability to create coverage-based test exclusion reports | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Benjamin Esquivel <benjamin.esquivel> |
| Component: | Scripts and Tools | Assignee: | Unassigned <unassigned> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | Medium | CC: | humberto.ibarra.lopez, joshuagloe, leonardo.sandoval.gonzalez, mhalstead, randy.macleod, ross.burton |
| Version: | 5.99 | ||
| Target Milestone: | Future | ||
| Hardware: | All | ||
| OS: | Multiple | ||
| Whiteboard: | OB REVIEW | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
|
Description
Benjamin Esquivel
2016-01-13 20:58:41 UTC
Assigning to Humberto, he is leading this area on the development area. Lets consider the following: the value 1 indicates that the executed lines between testi and testj are the same, and value value 0 is the exactly the opposite, so a formula like this may help (too pythonic but the idea is there)
res = [executed lines on testi] and [executed lines on testj]
where if both tests executed the same lines, then
res = []
and if both tests executed completely different lines
res = [executed lines on testi]
thus
value = (len(res) - len([executed lines on testi]))
------------------------------------------
len([executed lines on testi])
we can construct a matrix with the values, just populating the upper right part as seem below:
----------------------------------------------
test1 test2 test3 test4 ......... testn
test1 1 0.9 0 0.1 .......... 0.5
test2 1
test3 1
test4 1
. .
. .
testn 1
----------------------------------------------
The problem is to identify those pair of tests with higher numbers, i.e test1 and test2 on the sample matrix have a high value, basically exercising the same code.
Beto, as discussed today, and to store in other place rather that the blackboard, this would be a better formula for the similarity between two sets A and B(each sets in this case correspond to the executed lines by a specific test):
|(A-B)| - |(B-A)|
-----------------
|A+B|
if A=B
0 + 0
----- = 0
|2A|
if A != B
|A| + |B|
--------- = 1
|A + B|
so, 0 indicates tests are identical and 1 indicates tests are not related at all The operator || indicates the length of the set.
This is low priority for 2.3 and there is no window time to work on this area during 2.3. I think result tool does something like this. We also added fast and slow test groups. I believe this in some form has been addressed. A nice idea but not resource feasible. |