Bug 8846

Summary: oe-selftest: coverage including non-unittest code
Product: [QA/Testing] Build Testing Reporter: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Component: generalAssignee: Humberto Ibarra <humberto.ibarra.lopez>
Status: RESOLVED FIXED QA Contact: Daniel Istrate <daniel.alexandrux.istrate>
Severity: minor    
Priority: Medium CC: bogdanx.a.voiculescu
Version: 2.0   
Target Milestone: 2.1 M2   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Leonardo Sandoval Gonzalez 2015-12-21 21:12:22 UTC
Source code Coverage starts at early stage, covering execution not related to unit-testing, thus giving a base coverage percentage even if there if unit test is empty. This can lead to biased results.
Comment 1 Leonardo Sandoval Gonzalez 2015-12-21 21:13:52 UTC
Changing owner because he has already created a patch.
Comment 2 Humberto Ibarra 2015-12-21 21:19:40 UTC
Coverage data reported even when no testcase is run:

$ :~/poky/build> oe-selftest --run-tests sstate --coverage
2015-12-21 09:48:17,775 - selftest - INFO - Running bitbake -e to get BBPATH
2015-12-21 09:48:18,942 - selftest - INFO - Checking that everything is in order before running the tests
2015-12-21 09:48:20,132 - selftest - INFO - Running bitbake -p
2015-12-21 09:48:22,223 - selftest - INFO - Coverage is enabled
2015-12-21 09:48:25,755 - selftest - INFO - Loading tests from: oeqa.selftest.sstate
2015-12-21 09:48:25,756 - selftest - INFO - Adding: "include selftest.inc" in local.conf
2015-12-21 09:48:25,756 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
2015-12-21 09:48:25,756 - selftest - INFO - Finished
2015-12-21 09:48:25,756 - selftest - INFO - Coverage configuration file (/home/humberto/poky/build/.coveragerc)
2015-12-21 09:48:25,757 - selftest - INFO - ===========================
2015-12-21 09:48:25,757 - selftest - INFO - 
[run]
data_file = /home/humberto/poky/build/.coverage.sstate
branch = True
source = 
    /home/humberto/poky/meta
    /home/humberto/poky/meta-yocto
    /home/humberto/poky/meta-yocto-bsp
    /home/humberto/poky/meta-selftest
    /home/humberto/poky/scripts
    /home/humberto/poky/bitbake

2015-12-21 09:48:25,760 - selftest - INFO - Coverage Report
2015-12-21 09:48:25,760 - selftest - INFO - ===============
2015-12-21 09:48:31,107 - selftest - INFO - 
Name                                                               Stmts   Miss Branch BrPart  Cover
----------------------------------------------------------------------------------------------------
/home/humberto/poky/bitbake/bin/bitbake                               23      8      4      2    63%
/home/humberto/poky/bitbake/lib/bb/COW.py                            226    142     70      6    32%
/home/humberto/poky/bitbake/lib/bb/__init__.py                        71     16     14      4    74%
/home/humberto/poky/bitbake/lib/bb/build.py                          498    417    170      3    14%
/home/humberto/poky/bitbake/lib/bb/cache.py                          541    470    223      3    10%
/home/humberto/poky/bitbake/lib/bb/checksum.py                        54     32     14      0    32%
/home/humberto/poky/bitbake/lib/bb/codeparser.py                     272    109    109      9    54%
/home/humberto/poky/bitbake/lib/bb/command.py                        255    168     30      0    31%
/home/humberto/poky/bitbake/lib/bb/compat.py                           2      0      0      0   100%
/home/humberto/poky/bitbake/lib/bb/cooker.py                        1524   1308    644     16    12%
/home/humberto/poky/bitbake/lib/bb/cookerdata.py                     233     71     99     26    62%
/home/humberto/poky/bitbake/lib/bb/daemonize.py                       38     28     10      1    23%
/home/humberto/poky/bitbake/lib/bb/data.py                           287    219    134      4    20%
/home/humberto/poky/bitbake/lib/bb/data_smart.py                     726    201    394     41    70%
/home/humberto/poky/bitbake/lib/bb/event.py                          406    193     84     17    49%
/home/humberto/poky/bitbake/lib/bb/exceptions.py                      63     48     26      0    17%
/home/humberto/poky/bitbake/lib/bb/fetch2/__init__.py               1097    923    511      2    11%
/home/humberto/poky/bitbake/lib/bb/fetch2/bzr.py                      68     49     14      0    23%
/home/humberto/poky/bitbake/lib/bb/fetch2/clearcase.py               136    111     32      0    15%
/home/humberto/poky/bitbake/lib/bb/fetch2/cvs.py                     101     89     44      0     8%
/home/humberto/poky/bitbake/lib/bb/fetch2/git.py                     255    223    114      0     9%
/home/humberto/poky/bitbake/lib/bb/fetch2/gitannex.py                 42     29      6      0    27%
/home/humberto/poky/bitbake/lib/bb/fetch2/gitsm.py                    87     73     36      0    11%
/home/humberto/poky/bitbake/lib/bb/fetch2/hg.py                      159    131     56      0    13%
/home/humberto/poky/bitbake/lib/bb/fetch2/local.py                    75     58     26      0    17%
/home/humberto/poky/bitbake/lib/bb/fetch2/osc.py                      80     63     16      0    18%
/home/humberto/poky/bitbake/lib/bb/fetch2/perforce.py                119    100     46      0    12%
/home/humberto/poky/bitbake/lib/bb/fetch2/repo.py                     45     32     10      0    24%
/home/humberto/poky/bitbake/lib/bb/fetch2/sftp.py                     41     27     10      0    27%
/home/humberto/poky/bitbake/lib/bb/fetch2/ssh.py                      42     29      8      0    26%
/home/humberto/poky/bitbake/lib/bb/fetch2/svn.py                     101     78     28      0    18%
/home/humberto/poky/bitbake/lib/bb/fetch2/wget.py                    296    266    112      0     7%
/home/humberto/poky/bitbake/lib/bb/main.py                           229    102     80     21    49%
/home/humberto/poky/bitbake/lib/bb/methodpool.py                       4      0      0      0   100%
/home/humberto/poky/bitbake/lib/bb/monitordisk.py                    162    151     81      0     5%
/home/humberto/poky/bitbake/lib/bb/msg.py                            118     36     36     13    62%
/home/humberto/poky/bitbake/lib/bb/namedtuple_with_abc.py            140    112      8      3    22%
/home/humberto/poky/bitbake/lib/bb/parse/__init__.py                 113     44     44      4    55%
/home/humberto/poky/bitbake/lib/bb/parse/ast.py                      342    130    104      4    60%
/home/humberto/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py       170     14     78     13    88%
/home/humberto/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py     103     12     48     11    83%
/home/humberto/poky/bitbake/lib/bb/parse/parse_py/__init__.py          5      0      0      0   100%
/home/humberto/poky/bitbake/lib/bb/persist_data.py                   122     57     30      3    47%
/home/humberto/poky/bitbake/lib/bb/process.py                        113     81     54      4    22%
/home/humberto/poky/bitbake/lib/bb/providers.py                      229    208    116      0     6%
/home/humberto/poky/bitbake/lib/bb/pysh/__init__.py                    0      0      0      0   100%
/home/humberto/poky/bitbake/lib/bb/pysh/pyshlex.py                   544    452    240      0    12%
/home/humberto/poky/bitbake/lib/bb/pysh/pyshtables.py                 18      0      8      0   100%
/home/humberto/poky/bitbake/lib/bb/pysh/pyshyacc.py                  426    315    134      0    20%
/home/humberto/poky/bitbake/lib/bb/pysh/sherrors.py                   19      0      0      0   100%
/home/humberto/poky/bitbake/lib/bb/runqueue.py                      1558   1427    764      0     6%
/home/humberto/poky/bitbake/lib/bb/server/__init__.py                 32      8      0      0    75%
/home/humberto/poky/bitbake/lib/bb/server/process.py                 186     78     50      7    51%
/home/humberto/poky/bitbake/lib/bb/server/xmlrpc.py                  245    187     56      0    19%
/home/humberto/poky/bitbake/lib/bb/siggen.py                         411    346    214      1    11%
/home/humberto/poky/bitbake/lib/bb/taskdata.py                       424    386    252      0     6%
/home/humberto/poky/bitbake/lib/bb/ui/__init__.py                      0      0      0      0   100%
/home/humberto/poky/bitbake/lib/bb/ui/knotty.py                      443    294    204     45    32%
/home/humberto/poky/bitbake/lib/bb/ui/uievent.py                      88     72     18      0    15%
/home/humberto/poky/bitbake/lib/bb/ui/uihelper.py                     62     45     20      5    27%
/home/humberto/poky/bitbake/lib/bb/utils.py                          900    733    416     19    16%
/home/humberto/poky/bitbake/lib/bs4/__init__.py                      202    161     84      1    15%
/home/humberto/poky/bitbake/lib/bs4/builder/__init__.py              142     70     42      1    42%
/home/humberto/poky/bitbake/lib/bs4/builder/_html5lib.py             173    168     40      0     2%
/home/humberto/poky/bitbake/lib/bs4/builder/_htmlparser.py           146    119     48      1    14%
/home/humberto/poky/bitbake/lib/bs4/builder/_lxml.py                 135    130     48      0     3%
/home/humberto/poky/bitbake/lib/bs4/dammit.py                        250    180    102      0    21%
/home/humberto/poky/bitbake/lib/bs4/element.py                       889    704    445      1    14%
/home/humberto/poky/bitbake/lib/codegen.py                           425    345    130      0    15%
/home/humberto/poky/bitbake/lib/ply/__init__.py                        1      0      0      0   100%
/home/humberto/poky/bitbake/lib/ply/lex.py                           664    605    300      1     6%
/home/humberto/poky/bitbake/lib/ply/yacc.py                         1816    930    818    104    50%
/home/humberto/poky/bitbake/lib/progressbar.py                       194    136     34      0    25%
/home/humberto/poky/bitbake/lib/prserv/__init__.py                    11      5      2      0    46%
/home/humberto/poky/bitbake/lib/prserv/db.py                         179    152     54      1    13%
/home/humberto/poky/bitbake/lib/prserv/serv.py                       330    273     56      1    15%
/home/humberto/poky/bitbake/lib/pyinotify.py                         942    614    319     22    30%
/home/humberto/poky/meta/lib/oe/__init__.py                            2      0      0      0   100%
/home/humberto/poky/meta/lib/oe/cachedpath.py                        144    122     60      0    11%
/home/humberto/poky/meta/lib/oe/data.py                               11      3      4      1    73%
/home/humberto/poky/meta/lib/oe/lsb.py                                68     47     41      4    25%
/home/humberto/poky/meta/lib/oe/maketype.py                           58      9     26      4    85%
/home/humberto/poky/meta/lib/oe/package.py                            88     84     34      0     3%
/home/humberto/poky/meta/lib/oe/packagegroup.py                       27     20     16      0    16%
/home/humberto/poky/meta/lib/oe/path.py                              134    112     58      0    11%
/home/humberto/poky/meta/lib/oe/sstatesig.py                         234    197    110      1    11%
/home/humberto/poky/meta/lib/oe/types.py                              80     53     28      2    27%
/home/humberto/poky/meta/lib/oe/utils.py                             176    135     68      2    19%
/home/humberto/poky/scripts/oepydevshell-internal.py                  73     73     16      0     2%
/home/humberto/poky/scripts/opkg-query-helper.py                      52     52     38      0     0%
/home/humberto/poky/scripts/relocate_sdk.py                          145    145     66      0     0%
/home/humberto/poky/scripts/sysroot-relativelinks.py                  21     21     12      0     0%
----------------------------------------------------------------------------------------------------
TOTAL                                                              22981  16666   9248    434    24%

2015-12-21 09:48:31,107 - selftest - INFO - Removing the include from local.conf
2015-12-21 09:48:31,108 - selftest - INFO - Removing the include from bblayers.conf
Comment 3 Humberto Ibarra 2015-12-23 23:40:55 UTC
Sent patch for review. http://lists.openembedded.org/pipermail/openembedded-core/2015-December/114611.html
Comment 4 Humberto Ibarra 2016-01-13 15:00:00 UTC
Commit: 02d259c1f4d9cf19636867ebcabf8695cd79f797