Bug 8920

after discussing with Ed, a parameter '--coverage-include' would be more appropriate in this case, which would have the same meaning as the --include parameter in coverage:

    --include is a list of file name patterns. If specified, only files matching those patterns will be measured.

Per conversation with Humberto Ibarra, we agree on the following command line arguments:


oe-selftest --coverage <run-command> <run-value>

for example

oe-selftest --coverage include scripts/wic

where <run-command> could be any of [1]

[1] https://coverage.readthedocs.org/en/coverage-4.0.2/config.html#run

I think this approach might have some difficulties that I'd like to discuss with you.

Some examples,

- sub-arguments become positional
- oe-selftest already has his own set of arguments, try seeing how a full line will look like when doing an actual run of tests with coverage.

I want to weigh in on the design of this. I'll look you up.

(In reply to comment #3)
> I think this approach might have some difficulties that I'd like to discuss
> with you.
> 
> Some examples,
> 
> - sub-arguments become positional
> - oe-selftest already has his own set of arguments, try seeing how a full
> line will look like when doing an actual run of tests with coverage.
> 
> I want to weigh in on the design of this. I'll look you up.

There is a similar (in term of syntax) argument:

 --list-tests-by [LIST_TESTS_BY [LIST_TESTS_BY ...]]
                        list-tests-by <name|class|module|id|tag> <list of

so we are basically repeating the patter with coverage.

I like the fact that you make it consistent with what is there already but I think I still have some doubts about the current approach of the options that oe-selftest takes. Let's make a full running example:

$ oe-selftest --run-tests-by name test1 test2 --coverage include /this/path1/ /this/path2/

Questions:
- is that how it should be specified a list of paths? the same way it is specified a list of tests (consecutively)?

- when --coverage obtains more 'run-command's, will you need to add --coverage next-command?

- are you thinking to '--exclude' as well?

(In reply to comment #5)
> I like the fact that you make it consistent with what is there already but I
> think I still have some doubts about the current approach of the options
> that oe-selftest takes. Let's make a full running example:
> 
> $ oe-selftest --run-tests-by name test1 test2 --coverage include
> /this/path1/ /this/path2/
> 
> Questions:
> - is that how it should be specified a list of paths? the same way it is
> specified a list of tests (consecutively)?
> 

just 'comma' separated, thus it will be

$ oe-selftest --run-tests-by name test1 test2 --coverage include
 /this/path1/,/this/path2/

> - when --coverage obtains more 'run-command's, will you need to add
> --coverage next-command?

--coverage expects just ONE run-command, so we cannot include and exclude at the same command line. This is of course a restriction, but in general, if user wants more flexibility, it can write a .coveragerc and oe-selftest would respect that file 
> 
> - are you thinking to '--exclude' as well?

In resume, 

"--coverage": this will look into the .coveragerc, if present, it will use it, if not, it will create a default .coverarc (this is done already)

"--coverage include scripts/wic,bitbake": this time, the .coveragerc run include section will be overwritten (or any other run parameter, like omit), otherwise it will create the .coveragerc and set this value

Reviewing the oe-selftest command line arguments, I realized that there is a need to refactor, thus I filed the following bug

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8938

Following this criteria, we should then follow this pattern:

# execute all, taking into account coverage. By default, --coverage-source contains just those poky folders containing python code (no-metadata)

oe-selftest --coverage

# execute all, just looking into the bitbake folder
oe-selftest --coverage --coverage-source bitbake

# execute all, including just meta-* folders
oe-selftest --coverage --coverage-include 'meta-*'

# execute all, omitting scritps folder
oe-selftest --coverage --coverage-omit scripts

Thus, besides --coverage, there are three more coverage source related arguments:

--coverage--source
--coverage-include
--coverage-omit

An interesting check would be to check these oe-selftest parameters against the coverage standalone tool, for example, these two commands should output the same .coverage file

1. coverage oe-selftest --run-all-tests
2. oe-selftest --run-all-tests --coverage

The same applies to the more refined coverage sub commands.

maybe that is a bit of duplication of work and have to maintain changes in the 2 of them. I agree that a coverage script will be required mainly for coverage reporting/processing but it should not mimic the runner's duties.

(In reply to comment #9)
> maybe that is a bit of duplication of work and have to maintain changes in
> the 2 of them. I agree that a coverage script will be required mainly for
> coverage reporting/processing but it should not mimic the runner's duties.

we should mimic just these (the input data), not the reporting arguments. in fact, we wont expect to add more that these arguments. the reporting stuff should be done with the coverage tool. 

the reason for mimic these is to simple have the same user experience in both, and I believe this is important.

Fixed:

http://lists.openembedded.org/pipermail/openembedded-core/2016-February/117621.html

it should be in progress review instead of resolved

V2

http://lists.openembedded.org/pipermail/openembedded-core/2016-February/117675.html

v3.

http://lists.openembedded.org/pipermail/openembedded-core/2016-February/117793.html