We should at least have a list of those libraries not being tested by oe-selftest. Once we have this list, we can start filing bugs to test the most important features on the untested code. This tasks is particulary important on this phase, where the python2 to python3 is taking place.
I'd propose to concentrate on test caverage instead. The fact that there is a test for some code doesn't always mean it's properly tested.
Right. I did not mention coverage explicitly but that is a way to partially answer the question, at least on the python files. But this bug is about finding those files with no tests at all. In other words, This bug does not care about the quality of test cases or coverage, just about those showing those untested files.
Just to avoid confusing. This bug tries to find those untested libraries; once found (if any), new bugs will be created. Once bug creation is done, this bug can be resolved.
Sadly running oe-selftest --coverage -a produced empty coverage report. I was going to use coverage report to find out which files are covered. # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -a' # HEAD commit 5c11e365e19357f721c49d076971567e7b64b61b [run] data_file = /home/ed/git/yocto/poky/build/.coverage.20160706T183956 branch = True source = /home/ed/git/yocto/poky/meta /home/ed/git/yocto/poky/meta-yocto /home/ed/git/yocto/poky/meta-yocto-bsp /home/ed/git/yocto/poky/meta-selftest /home/ed/git/yocto/poky/scripts /home/ed/git/yocto/poky/bitbake 2016-07-07 00:49:16,878 - selftest - INFO - Coverage Report 2016-07-07 00:49:16,879 - selftest - INFO - =============== 2016-07-07 00:49:16,938 - selftest - INFO - Name Stmts Miss Cover --------------------------------------------------------------------------- /home/ed/git/yocto/poky/scripts/oepydevshell-internal 73 73 0% /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% --------------------------------------------------------------------------- TOTAL 299 299 0% ---------------------------------------------------------------------- Ran 187 tests in 22160.353s OK (skipped=1)
(In reply to comment #4) > Sadly running oe-selftest --coverage -a produced empty coverage report. I > was going to use coverage report to find out which files are covered. > > # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest > --coverage -a' > # HEAD commit 5c11e365e19357f721c49d076971567e7b64b61b > [run] > data_file = /home/ed/git/yocto/poky/build/.coverage.20160706T183956 > branch = True > source = > /home/ed/git/yocto/poky/meta > /home/ed/git/yocto/poky/meta-yocto > /home/ed/git/yocto/poky/meta-yocto-bsp > /home/ed/git/yocto/poky/meta-selftest > /home/ed/git/yocto/poky/scripts > /home/ed/git/yocto/poky/bitbake > > 2016-07-07 00:49:16,878 - selftest - INFO - Coverage Report > 2016-07-07 00:49:16,879 - selftest - INFO - =============== > 2016-07-07 00:49:16,938 - selftest - INFO - > Name Stmts Miss Cover > --------------------------------------------------------------------------- > /home/ed/git/yocto/poky/scripts/oepydevshell-internal 73 73 0% > /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% > /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% > /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% > --------------------------------------------------------------------------- > TOTAL 299 299 0% > > > ---------------------------------------------------------------------- > Ran 187 tests in 22160.353s > > OK (skipped=1) Let me investigate. Coverage was fully working before. However, since the switch to python3, no testing has been done.
Humberto, please check oe-selftest coverage on latest master.
I ran oe-selftest -a --coverage and got a valid output with all files covered. @Ed can you please elaborate on your failure? What's the coverage version you are using?
I tried to run oe-selftest again and got 0% coverages again: $ oe-selftest --coverage -r wic.Wic.test_build_image_name 2016-07-21 18:55:43,753 - selftest - INFO - Running bitbake -e to get BBPATH 2016-07-21 18:55:45,054 - selftest - INFO - Checking that everything is in order before running the tests 2016-07-21 18:55:46,218 - selftest - INFO - Running bitbake -p 2016-07-21 18:55:49,307 - selftest - INFO - Loading tests from: oeqa.selftest.wic.Wic.test_build_image_name 2016-07-21 18:55:49,314 - selftest - INFO - Adding: "include selftest.inc" in local.conf 2016-07-21 18:55:49,314 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2016-07-21 18:55:49,343 - selftest - INFO - Coverage is enabled 2016-07-21 18:55:50 - test_build_image_name (oeqa.selftest.wic.Wic) Test wic create directdisk --image-name core-image-minimal ... ok 2016-07-21 19:00:01,120 - selftest - INFO - Coverage configuration file (/home/ed/git/yocto/poky/build/.coveragerc) 2016-07-21 19:00:01,121 - selftest - INFO - =========================== 2016-07-21 19:00:01,121 - selftest - INFO - # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -r wic.Wic.test_build_image_name' # HEAD commit 36feb38045b7a2af86ece147fec54b0db3bf491f [run] data_file = /home/ed/git/yocto/poky/build/.coverage.20160721T185549 branch = True source = /home/ed/git/yocto/poky/meta /home/ed/git/yocto/poky/meta-yocto /home/ed/git/yocto/poky/meta-yocto-bsp /home/ed/git/yocto/poky/meta-selftest /home/ed/git/yocto/poky/scripts /home/ed/git/yocto/poky/bitbake 2016-07-21 19:00:01,122 - selftest - INFO - Coverage Report 2016-07-21 19:00:01,122 - selftest - INFO - =============== 2016-07-21 19:00:01,176 - selftest - INFO - Name Stmts Miss Cover --------------------------------------------------------------------------- /home/ed/git/yocto/poky/scripts/oepydevshell-internal 72 72 0% /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% --------------------------------------------------------------------------- TOTAL 298 298 0% ---------------------------------------------------------------------- Ran 1 test in 251.862s OK 2016-07-21 19:00:01,177 - selftest - INFO - Finished 2016-07-21 19:00:01,178 - selftest - INFO - Removing the include from local.conf 2016-07-21 19:00:01,179 - selftest - INFO - Removing the include from bblayers.conf Here is the list of coverage packages installed: $ rpm -qa |grep coverage python3-coverage-3.7.1-2.1.3.x86_64 python-coverage-3.7.1-2.1.4.x86_64
Another try - same result. This time I'm trying to run another test on another machine: e-selftest --coverage -r manifest.VerifyManifest.test_image_manifest_entries 2016-07-22 11:52:02,214 - selftest - INFO - Running bitbake -e to get BBPATH 2016-07-22 11:52:03,029 - selftest - INFO - Checking that everything is in order before running the tests 2016-07-22 11:52:03,854 - selftest - INFO - Running bitbake -p 2016-07-22 11:52:05,634 - selftest - INFO - Loading tests from: oeqa.selftest.manifest.VerifyManifest.test_image_manifest_entries 2016-07-22 11:52:05,635 - selftest - INFO - Adding: "include selftest.inc" in local.conf 2016-07-22 11:52:05,635 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2016-07-22 11:52:05,646 - selftest - INFO - Coverage is enabled 2016-07-22 11:52:06,315 - selftest.base - INFO - VerifyManifest: doing bitbake core-image-minimal as a prerequisite of the test 2016-07-22 11:52:17 - test_image_manifest_entries (oeqa.selftest.manifest.VerifyManifest) Verifying the image manifest entries exist ... ok 2016-07-22 11:52:28,933 - selftest - INFO - Coverage configuration file (/home/ed/git/yocto/poky/build/.coveragerc) 2016-07-22 11:52:28,933 - selftest - INFO - =========================== 2016-07-22 11:52:28,933 - selftest - INFO - # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -r manifest.VerifyManifest.test_image_manifest_entries' # HEAD commit 36feb38045b7a2af86ece147fec54b0db3bf491f [run] data_file = /home/ed/git/yocto/poky/build/.coverage.20160722T115205 branch = True source = /home/ed/git/yocto/poky/meta /home/ed/git/yocto/poky/meta-poky /home/ed/git/yocto/poky/meta-yocto-bsp /home/ed/git/yocto/poky/meta-selftest /home/ed/git/yocto/poky/scripts /home/ed/git/yocto/poky/bitbake 2016-07-22 11:52:28,933 - selftest - INFO - Coverage Report 2016-07-22 11:52:28,933 - selftest - INFO - =============== 2016-07-22 11:52:28,949 - selftest - INFO - Name Stmts Miss Cover --------------------------------------------------------------------------- /home/ed/git/yocto/poky/scripts/oepydevshell-internal 72 72 0% /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% --------------------------------------------------------------------------- TOTAL 298 298 0% ---------------------------------------------------------------------- Ran 1 test in 23.314s OK 2016-07-22 11:52:28,950 - selftest - INFO - Finished 2016-07-22 11:52:28,950 - selftest - INFO - Removing the include from local.conf 2016-07-22 11:52:28,950 - selftest - INFO - Removing the include from bblayers.conf > rpm -qa |grep coverage python3-coverage-3.7.1-3.3.x86_64
It's interesting that the coverage output is the same for all different tests. It always includes 4 files from scripts/ directory.
looks like coverage report contains only files with .py extension from scripts/: > ls -la ../scripts/*.py -rwxr-xr-x 1 ed users 2512 Jul 21 18:42 ../scripts/oepydevshell-internal.py -rwxr-xr-x 1 ed users 2767 Jul 21 18:42 ../scripts/opkg-query-helper.py -rwxr-xr-x 1 ed users 9071 Jul 21 18:42 ../scripts/relocate_sdk.py -rwxr-xr-x 1 ed users 960 Jul 21 18:42 ../scripts/sysroot-relativelinks.py
coverage itself works just fine: > coverage run ../bitbake/bin/bitbake-layers show-layers ... > > coverage report | head -10 Name Stmts Miss Branch BrMiss Cover ----------------------------------------------------------------------------------------------------- /home/ed/git/yocto/poky/bitbake/bin/bitbake-layers 75 13 20 6 80% /home/ed/git/yocto/poky/bitbake/lib/bb/COW 223 144 70 60 30% /home/ed/git/yocto/poky/bitbake/lib/bb/__init__ 72 13 14 5 79% /home/ed/git/yocto/poky/bitbake/lib/bb/build 534 449 188 174 14% /home/ed/git/yocto/poky/bitbake/lib/bb/cache 536 427 228 218 16% /home/ed/git/yocto/poky/bitbake/lib/bb/checksum 89 65 36 36 19% /home/ed/git/yocto/poky/bitbake/lib/bb/codeparser 294 121 128 85 51% /home/ed/git/yocto/poky/bitbake/lib/bb/command 256 168 30 30 31%
It looks like oe-selftest --coverage doesn't work only on OpenSUSE. I run it on Fedora and it produced valid report just fine.
Created attachment 3311 [details] not covered bitbake files
Created attachment 3312 [details] not covered scripts files
here is a simple script to report uncovered python files based on coverage report: $ cat uncovered.sh #!/bin/sh path=$1 if [ -z "$path" ] ; then echo "Usage: ./uncovered <dir>" exit 1 fi if [ ! -d "$path" ] ; then echo "directory $path doesn't exits" exit 1 fi coverage3 report --rcfile=build/.coveragerc | grep "$path" | grep -v '0%$' | grep -v '\.pyc$' | \ xargs file | grep 'Python script' | cut -f1 -d: | sort > /tmp/covered.tmp find $path | grep -v '\.pyc$' | xargs file | grep 'Python script' | cut -f1 -d: | sort > /tmp/all.tmp diff -u /tmp/covered.tmp /tmp/all.tmp | grep "^+$path" | sed -ne 's/+\(.*\)/\1/p' rm /tmp/covered.tmp /tmp/all.tmp I've attached output of the following 2 commands: ./uncovered.sh bitbake/ |grep -v 'bitbake/lib/\(toaster\)\|\(bs4\)\//' |grep -v 'bitbake/lib/bb/tests' ./uncovered.sh scripts/ I hope this should be enough to resolve this ticket. Let me know if anything else is expected.
Ed, Looks good to me. Are you planning to push these into poky/scripts/contrib?
The script has been sent for review: http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124289.html Please note that script code and usage is changed.
the patch has been merged to master
verified on master commit : 6b66e9317f4ec3a69f98f29836aafa35b52f3fc7