Bug 8679 - oe-selftest: measure test coverage
Summary: oe-selftest: measure test coverage
Status: RESOLVED FIXED
Alias: None
Product: Build Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: unspecified
Hardware: All Multiple
: Medium enhancement
Target Milestone: 2.1 M2
Assignee: Leonardo Sandoval Gonzalez
QA Contact: Daniel Istrate
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-11-11 16:38 UTC by Paul Eggleton
Modified: 2016-01-28 16:00 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
proposed patch to enable coverage on unit tests (5.32 KB, patch)
2015-11-13 22:40 UTC, Leonardo Sandoval Gonzalez 		no flags Details | Diff
provision patch to list classes (and its tests methods) without format (1.13 KB, patch)
2015-11-13 22:44 UTC, Leonardo Sandoval Gonzalez 		no flags Details | Diff
oe-selftest --run-all-tests including coverage (28.48 KB, text/plain)
2015-11-18 14:32 UTC, Leonardo Sandoval Gonzalez 		no flags Details
Coverage per unit test (6.29 MB, application/x-compressed-tar)
2015-11-18 14:37 UTC, Leonardo Sandoval Gonzalez 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Eggleton 2015-11-11 16:38:44 UTC 
It would be useful to have a measurement of test coverage for oe-selftest - in particular for python code in meta/lib/oe, bitbake/lib/bb and scripts.

I think Ed looked into this briefly when we were initially discussing wic testing within oe-selftest.
Comment 1 Ed Bartosh 2015-11-13 09:37:10 UTC 
yes, I briefly looked at it and put it to my TODO list. I can work on it if needed after 22nd of Nov when I'm back from vacation.
Comment 2 Leonardo Sandoval Gonzalez 2015-11-13 15:22:11 UTC 
(In reply to comment #1)
> yes, I briefly looked at it and put it to my TODO list. I can work on it if
> needed after 22nd of Nov when I'm back from vacation.

Ed. I already have a patch and some measurements. I will post them today on this page. In case more work is needed, just let me know.

BTW, all coverage has been done with python coverage 4.0.2.
Comment 3 Leonardo Sandoval Gonzalez 2015-11-13 22:40:30 UTC 
Created attachment 2851 [details]
proposed patch to enable coverage on unit tests
Comment 4 Leonardo Sandoval Gonzalez 2015-11-13 22:44:12 UTC 
Created attachment 2852 [details]
provision patch to list classes (and its tests methods) without format

Once this patch is applied, it can used to get the unit tests and executed each one with coverage:

$oe-selftest --list-classes > list-classes.txt
$ # remove any tests you do want to execute
$ xargs -a list-classes.txt -n1 oe-selftest --coverage --run-tests

Lots of .coverage.* files are created, having coverage PER unit test.
Comment 5 Leonardo Sandoval Gonzalez 2015-11-18 14:32:59 UTC 
Created attachment 2854 [details]
oe-selftest --run-all-tests including coverage

This is a report created by coverage at the end of the run 'oe-selftest --run-all-tests'. In resume, the current coverage for the Poky system is around 42%.
Comment 6 Leonardo Sandoval Gonzalez 2015-11-18 14:37:43 UTC 
Created attachment 2855 [details]
Coverage per unit test

This tarball contains coverage output data PER unit tests. The unit tests taken into account are indicated on list-classes.txt and the coverage configuration is defined on .coveragerc. The coverage was done under the following commit:

fc45dea: Richard Purdie: Tue Nov 3 14:03:03 2015 +0000: build-appliance-image: Update to jethro head revision
Comment 7 Leonardo Sandoval Gonzalez 2015-11-18 15:07:28 UTC 
Patch sent to the ML. Waiting for comments.
Comment 8 Leonardo Sandoval Gonzalez 2016-01-28 16:00:32 UTC 
Initial set of patches are already on master, enabling source code coverage from oe-selftest. This work has been done by Humberto Ibarra.