8920 2016-01-13 22:36:13 +0000 oe-selftest: users should be able to specify those folders that 'coverage' should look into 2016-03-03 20:36:38 +0000 1 1 10 QA/Testing Build Testing general 2.0 x86 Multiple RESOLVED FIXED Medium enhancement 2.1 1 leonardo.sandoval.gonzalez humberto.ibarra.lopez benjamin.esquivel eduard.bartosh humberto.ibarra.lopez --- --- No (bug/feature does not impact docs) oldest_to_newest 57799 0 leonardo.sandoval.gonzalez 2016-01-13 22:36:13 +0000 Currently, the source code coverage is done on those files under the meta, meta-*, scripts and bitbake folders. Users should be able to specify the latter through a command line argument (--coverage-source scripts, for example) 57820 1 leonardo.sandoval.gonzalez 2016-01-14 14:46:18 +0000 after discussing with Ed, a parameter '--coverage-include' would be more appropriate in this case, which would have the same meaning as the --include parameter in coverage: --include is a list of file name patterns. If specified, only files matching those patterns will be measured. 57878 2 leonardo.sandoval.gonzalez 2016-01-15 17:31:02 +0000 Per conversation with Humberto Ibarra, we agree on the following command line arguments: oe-selftest --coverage <run-command> <run-value> for example oe-selftest --coverage include scripts/wic where <run-command> could be any of [1] [1] https://coverage.readthedocs.org/en/coverage-4.0.2/config.html#run 57879 3 benjamin.esquivel 2016-01-15 18:15:25 +0000 I think this approach might have some difficulties that I'd like to discuss with you. Some examples, - sub-arguments become positional - oe-selftest already has his own set of arguments, try seeing how a full line will look like when doing an actual run of tests with coverage. I want to weigh in on the design of this. I'll look you up. 57880 4 leonardo.sandoval.gonzalez 2016-01-15 18:22:53 +0000 (In reply to comment #3) > I think this approach might have some difficulties that I'd like to discuss > with you. > > Some examples, > > - sub-arguments become positional > - oe-selftest already has his own set of arguments, try seeing how a full > line will look like when doing an actual run of tests with coverage. > > I want to weigh in on the design of this. I'll look you up. There is a similar (in term of syntax) argument: --list-tests-by [LIST_TESTS_BY [LIST_TESTS_BY ...]] list-tests-by <name|class|module|id|tag> <list of so we are basically repeating the patter with coverage. 57885 5 benjamin.esquivel 2016-01-15 19:25:31 +0000 I like the fact that you make it consistent with what is there already but I think I still have some doubts about the current approach of the options that oe-selftest takes. Let's make a full running example: $ oe-selftest --run-tests-by name test1 test2 --coverage include /this/path1/ /this/path2/ Questions: - is that how it should be specified a list of paths? the same way it is specified a list of tests (consecutively)? - when --coverage obtains more 'run-command's, will you need to add --coverage next-command? - are you thinking to '--exclude' as well? 57886 6 leonardo.sandoval.gonzalez 2016-01-15 19:38:14 +0000 (In reply to comment #5) > I like the fact that you make it consistent with what is there already but I > think I still have some doubts about the current approach of the options > that oe-selftest takes. Let's make a full running example: > > $ oe-selftest --run-tests-by name test1 test2 --coverage include > /this/path1/ /this/path2/ > > Questions: > - is that how it should be specified a list of paths? the same way it is > specified a list of tests (consecutively)? > just 'comma' separated, thus it will be $ oe-selftest --run-tests-by name test1 test2 --coverage include /this/path1/,/this/path2/ > - when --coverage obtains more 'run-command's, will you need to add > --coverage next-command? --coverage expects just ONE run-command, so we cannot include and exclude at the same command line. This is of course a restriction, but in general, if user wants more flexibility, it can write a .coveragerc and oe-selftest would respect that file > > - are you thinking to '--exclude' as well? In resume, "--coverage": this will look into the .coveragerc, if present, it will use it, if not, it will create a default .coverarc (this is done already) "--coverage include scripts/wic,bitbake": this time, the .coveragerc run include section will be overwritten (or any other run parameter, like omit), otherwise it will create the .coveragerc and set this value 57892 7 leonardo.sandoval.gonzalez 2016-01-15 23:05:09 +0000 Reviewing the oe-selftest command line arguments, I realized that there is a need to refactor, thus I filed the following bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=8938 Following this criteria, we should then follow this pattern: # execute all, taking into account coverage. By default, --coverage-source contains just those poky folders containing python code (no-metadata) oe-selftest --coverage # execute all, just looking into the bitbake folder oe-selftest --coverage --coverage-source bitbake # execute all, including just meta-* folders oe-selftest --coverage --coverage-include 'meta-*' # execute all, omitting scritps folder oe-selftest --coverage --coverage-omit scripts Thus, besides --coverage, there are three more coverage source related arguments: --coverage--source --coverage-include --coverage-omit 57925 8 leonardo.sandoval.gonzalez 2016-01-18 17:15:26 +0000 An interesting check would be to check these oe-selftest parameters against the coverage standalone tool, for example, these two commands should output the same .coverage file 1. coverage oe-selftest --run-all-tests 2. oe-selftest --run-all-tests --coverage The same applies to the more refined coverage sub commands. 57930 9 benjamin.esquivel 2016-01-18 18:11:31 +0000 maybe that is a bit of duplication of work and have to maintain changes in the 2 of them. I agree that a coverage script will be required mainly for coverage reporting/processing but it should not mimic the runner's duties. 57932 10 leonardo.sandoval.gonzalez 2016-01-18 18:21:20 +0000 (In reply to comment #9) > maybe that is a bit of duplication of work and have to maintain changes in > the 2 of them. I agree that a coverage script will be required mainly for > coverage reporting/processing but it should not mimic the runner's duties. we should mimic just these (the input data), not the reporting arguments. in fact, we wont expect to add more that these arguments. the reporting stuff should be done with the coverage tool. the reason for mimic these is to simple have the same user experience in both, and I believe this is important. 59121 11 humberto.ibarra.lopez 2016-02-16 20:11:54 +0000 Fixed: http://lists.openembedded.org/pipermail/openembedded-core/2016-February/117621.html 59123 12 leonardo.sandoval.gonzalez 2016-02-16 20:33:05 +0000 it should be in progress review instead of resolved 59173 13 humberto.ibarra.lopez 2016-02-17 21:58:27 +0000 V2 http://lists.openembedded.org/pipermail/openembedded-core/2016-February/117675.html 59287 14 humberto.ibarra.lopez 2016-02-19 21:33:52 +0000 v3. http://lists.openembedded.org/pipermail/openembedded-core/2016-February/117793.html