9809 2016-06-21 22:01:55 +0000 List python/bitbake libraries that are not being tested by oe-selftest 2016-10-17 17:19:56 +0000 1 1 7 Build System, Metadata & Runtime OE-Core Scripts and Tools 2.2 x86 Multiple VERIFIED FIXED Medium+ normal 2.2 M2 1 leonardo.sandoval.gonzalez eduard.bartosh bluelightning eduard.bartosh jose.perez.carranza mariano.lopez ross.burton sgw jose.perez.carranza --- --- No (bug/feature does not impact docs) oldest_to_newest 63232 0 leonardo.sandoval.gonzalez 2016-06-21 22:01:55 +0000 We should at least have a list of those libraries not being tested by oe-selftest. Once we have this list, we can start filing bugs to test the most important features on the untested code. This tasks is particulary important on this phase, where the python2 to python3 is taking place. 63241 1 eduard.bartosh 2016-06-22 08:16:59 +0000 I'd propose to concentrate on test caverage instead. The fact that there is a test for some code doesn't always mean it's properly tested. 63252 2 leonardo.sandoval.gonzalez 2016-06-22 13:27:28 +0000 Right. I did not mention coverage explicitly but that is a way to partially answer the question, at least on the python files. But this bug is about finding those files with no tests at all. In other words, This bug does not care about the quality of test cases or coverage, just about those showing those untested files. 63294 3 leonardo.sandoval.gonzalez 2016-06-23 15:02:07 +0000 Just to avoid confusing. This bug tries to find those untested libraries; once found (if any), new bugs will be created. Once bug creation is done, this bug can be resolved. 63706 4 eduard.bartosh 2016-07-07 08:38:57 +0000 Sadly running oe-selftest --coverage -a produced empty coverage report. I was going to use coverage report to find out which files are covered. # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -a' # HEAD commit 5c11e365e19357f721c49d076971567e7b64b61b [run] data_file = /home/ed/git/yocto/poky/build/.coverage.20160706T183956 branch = True source = /home/ed/git/yocto/poky/meta /home/ed/git/yocto/poky/meta-yocto /home/ed/git/yocto/poky/meta-yocto-bsp /home/ed/git/yocto/poky/meta-selftest /home/ed/git/yocto/poky/scripts /home/ed/git/yocto/poky/bitbake 2016-07-07 00:49:16,878 - selftest - INFO - Coverage Report 2016-07-07 00:49:16,879 - selftest - INFO - =============== 2016-07-07 00:49:16,938 - selftest - INFO - Name Stmts Miss Cover --------------------------------------------------------------------------- /home/ed/git/yocto/poky/scripts/oepydevshell-internal 73 73 0% /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% --------------------------------------------------------------------------- TOTAL 299 299 0% ---------------------------------------------------------------------- Ran 187 tests in 22160.353s OK (skipped=1) 63717 5 leonardo.sandoval.gonzalez 2016-07-07 14:26:22 +0000 (In reply to comment #4) > Sadly running oe-selftest --coverage -a produced empty coverage report. I > was going to use coverage report to find out which files are covered. > > # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest > --coverage -a' > # HEAD commit 5c11e365e19357f721c49d076971567e7b64b61b > [run] > data_file = /home/ed/git/yocto/poky/build/.coverage.20160706T183956 > branch = True > source = > /home/ed/git/yocto/poky/meta > /home/ed/git/yocto/poky/meta-yocto > /home/ed/git/yocto/poky/meta-yocto-bsp > /home/ed/git/yocto/poky/meta-selftest > /home/ed/git/yocto/poky/scripts > /home/ed/git/yocto/poky/bitbake > > 2016-07-07 00:49:16,878 - selftest - INFO - Coverage Report > 2016-07-07 00:49:16,879 - selftest - INFO - =============== > 2016-07-07 00:49:16,938 - selftest - INFO - > Name Stmts Miss Cover > --------------------------------------------------------------------------- > /home/ed/git/yocto/poky/scripts/oepydevshell-internal 73 73 0% > /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% > /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% > /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% > --------------------------------------------------------------------------- > TOTAL 299 299 0% > > > ---------------------------------------------------------------------- > Ran 187 tests in 22160.353s > > OK (skipped=1) Let me investigate. Coverage was fully working before. However, since the switch to python3, no testing has been done. 63718 6 leonardo.sandoval.gonzalez 2016-07-07 14:33:48 +0000 Humberto, please check oe-selftest coverage on latest master. 63932 7 humberto.ibarra.lopez 2016-07-13 16:59:30 +0000 I ran oe-selftest -a --coverage and got a valid output with all files covered. @Ed can you please elaborate on your failure? What's the coverage version you are using? 64277 8 eduard.bartosh 2016-07-21 16:27:46 +0000 I tried to run oe-selftest again and got 0% coverages again: $ oe-selftest --coverage -r wic.Wic.test_build_image_name 2016-07-21 18:55:43,753 - selftest - INFO - Running bitbake -e to get BBPATH 2016-07-21 18:55:45,054 - selftest - INFO - Checking that everything is in order before running the tests 2016-07-21 18:55:46,218 - selftest - INFO - Running bitbake -p 2016-07-21 18:55:49,307 - selftest - INFO - Loading tests from: oeqa.selftest.wic.Wic.test_build_image_name 2016-07-21 18:55:49,314 - selftest - INFO - Adding: "include selftest.inc" in local.conf 2016-07-21 18:55:49,314 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2016-07-21 18:55:49,343 - selftest - INFO - Coverage is enabled 2016-07-21 18:55:50 - test_build_image_name (oeqa.selftest.wic.Wic) Test wic create directdisk --image-name core-image-minimal ... ok 2016-07-21 19:00:01,120 - selftest - INFO - Coverage configuration file (/home/ed/git/yocto/poky/build/.coveragerc) 2016-07-21 19:00:01,121 - selftest - INFO - =========================== 2016-07-21 19:00:01,121 - selftest - INFO - # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -r wic.Wic.test_build_image_name' # HEAD commit 36feb38045b7a2af86ece147fec54b0db3bf491f [run] data_file = /home/ed/git/yocto/poky/build/.coverage.20160721T185549 branch = True source = /home/ed/git/yocto/poky/meta /home/ed/git/yocto/poky/meta-yocto /home/ed/git/yocto/poky/meta-yocto-bsp /home/ed/git/yocto/poky/meta-selftest /home/ed/git/yocto/poky/scripts /home/ed/git/yocto/poky/bitbake 2016-07-21 19:00:01,122 - selftest - INFO - Coverage Report 2016-07-21 19:00:01,122 - selftest - INFO - =============== 2016-07-21 19:00:01,176 - selftest - INFO - Name Stmts Miss Cover --------------------------------------------------------------------------- /home/ed/git/yocto/poky/scripts/oepydevshell-internal 72 72 0% /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% --------------------------------------------------------------------------- TOTAL 298 298 0% ---------------------------------------------------------------------- Ran 1 test in 251.862s OK 2016-07-21 19:00:01,177 - selftest - INFO - Finished 2016-07-21 19:00:01,178 - selftest - INFO - Removing the include from local.conf 2016-07-21 19:00:01,179 - selftest - INFO - Removing the include from bblayers.conf Here is the list of coverage packages installed: $ rpm -qa |grep coverage python3-coverage-3.7.1-2.1.3.x86_64 python-coverage-3.7.1-2.1.4.x86_64 64327 9 eduard.bartosh 2016-07-22 08:54:19 +0000 Another try - same result. This time I'm trying to run another test on another machine: e-selftest --coverage -r manifest.VerifyManifest.test_image_manifest_entries 2016-07-22 11:52:02,214 - selftest - INFO - Running bitbake -e to get BBPATH 2016-07-22 11:52:03,029 - selftest - INFO - Checking that everything is in order before running the tests 2016-07-22 11:52:03,854 - selftest - INFO - Running bitbake -p 2016-07-22 11:52:05,634 - selftest - INFO - Loading tests from: oeqa.selftest.manifest.VerifyManifest.test_image_manifest_entries 2016-07-22 11:52:05,635 - selftest - INFO - Adding: "include selftest.inc" in local.conf 2016-07-22 11:52:05,635 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2016-07-22 11:52:05,646 - selftest - INFO - Coverage is enabled 2016-07-22 11:52:06,315 - selftest.base - INFO - VerifyManifest: doing bitbake core-image-minimal as a prerequisite of the test 2016-07-22 11:52:17 - test_image_manifest_entries (oeqa.selftest.manifest.VerifyManifest) Verifying the image manifest entries exist ... ok 2016-07-22 11:52:28,933 - selftest - INFO - Coverage configuration file (/home/ed/git/yocto/poky/build/.coveragerc) 2016-07-22 11:52:28,933 - selftest - INFO - =========================== 2016-07-22 11:52:28,933 - selftest - INFO - # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -r manifest.VerifyManifest.test_image_manifest_entries' # HEAD commit 36feb38045b7a2af86ece147fec54b0db3bf491f [run] data_file = /home/ed/git/yocto/poky/build/.coverage.20160722T115205 branch = True source = /home/ed/git/yocto/poky/meta /home/ed/git/yocto/poky/meta-poky /home/ed/git/yocto/poky/meta-yocto-bsp /home/ed/git/yocto/poky/meta-selftest /home/ed/git/yocto/poky/scripts /home/ed/git/yocto/poky/bitbake 2016-07-22 11:52:28,933 - selftest - INFO - Coverage Report 2016-07-22 11:52:28,933 - selftest - INFO - =============== 2016-07-22 11:52:28,949 - selftest - INFO - Name Stmts Miss Cover --------------------------------------------------------------------------- /home/ed/git/yocto/poky/scripts/oepydevshell-internal 72 72 0% /home/ed/git/yocto/poky/scripts/opkg-query-helper 52 52 0% /home/ed/git/yocto/poky/scripts/relocate_sdk 153 153 0% /home/ed/git/yocto/poky/scripts/sysroot-relativelinks 21 21 0% --------------------------------------------------------------------------- TOTAL 298 298 0% ---------------------------------------------------------------------- Ran 1 test in 23.314s OK 2016-07-22 11:52:28,950 - selftest - INFO - Finished 2016-07-22 11:52:28,950 - selftest - INFO - Removing the include from local.conf 2016-07-22 11:52:28,950 - selftest - INFO - Removing the include from bblayers.conf > rpm -qa |grep coverage python3-coverage-3.7.1-3.3.x86_64 64328 10 eduard.bartosh 2016-07-22 09:15:27 +0000 It's interesting that the coverage output is the same for all different tests. It always includes 4 files from scripts/ directory. 64330 11 eduard.bartosh 2016-07-22 09:39:12 +0000 looks like coverage report contains only files with .py extension from scripts/: > ls -la ../scripts/*.py -rwxr-xr-x 1 ed users 2512 Jul 21 18:42 ../scripts/oepydevshell-internal.py -rwxr-xr-x 1 ed users 2767 Jul 21 18:42 ../scripts/opkg-query-helper.py -rwxr-xr-x 1 ed users 9071 Jul 21 18:42 ../scripts/relocate_sdk.py -rwxr-xr-x 1 ed users 960 Jul 21 18:42 ../scripts/sysroot-relativelinks.py 64338 12 eduard.bartosh 2016-07-22 11:55:14 +0000 coverage itself works just fine: > coverage run ../bitbake/bin/bitbake-layers show-layers ... > > coverage report | head -10 Name Stmts Miss Branch BrMiss Cover ----------------------------------------------------------------------------------------------------- /home/ed/git/yocto/poky/bitbake/bin/bitbake-layers 75 13 20 6 80% /home/ed/git/yocto/poky/bitbake/lib/bb/COW 223 144 70 60 30% /home/ed/git/yocto/poky/bitbake/lib/bb/__init__ 72 13 14 5 79% /home/ed/git/yocto/poky/bitbake/lib/bb/build 534 449 188 174 14% /home/ed/git/yocto/poky/bitbake/lib/bb/cache 536 427 228 218 16% /home/ed/git/yocto/poky/bitbake/lib/bb/checksum 89 65 36 36 19% /home/ed/git/yocto/poky/bitbake/lib/bb/codeparser 294 121 128 85 51% /home/ed/git/yocto/poky/bitbake/lib/bb/command 256 168 30 30 31% 64403 13 eduard.bartosh 2016-07-25 07:30:08 +0000 It looks like oe-selftest --coverage doesn't work only on OpenSUSE. I run it on Fedora and it produced valid report just fine. 64404 14 3311 eduard.bartosh 2016-07-25 07:38:10 +0000 Created attachment 3311 not covered bitbake files 64405 15 3312 eduard.bartosh 2016-07-25 07:39:12 +0000 Created attachment 3312 not covered scripts files 64406 16 eduard.bartosh 2016-07-25 07:39:47 +0000 here is a simple script to report uncovered python files based on coverage report: $ cat uncovered.sh #!/bin/sh path=$1 if [ -z "$path" ] ; then echo "Usage: ./uncovered <dir>" exit 1 fi if [ ! -d "$path" ] ; then echo "directory $path doesn't exits" exit 1 fi coverage3 report --rcfile=build/.coveragerc | grep "$path" | grep -v '0%$' | grep -v '\.pyc$' | \ xargs file | grep 'Python script' | cut -f1 -d: | sort > /tmp/covered.tmp find $path | grep -v '\.pyc$' | xargs file | grep 'Python script' | cut -f1 -d: | sort > /tmp/all.tmp diff -u /tmp/covered.tmp /tmp/all.tmp | grep "^+$path" | sed -ne 's/+\(.*\)/\1/p' rm /tmp/covered.tmp /tmp/all.tmp I've attached output of the following 2 commands: ./uncovered.sh bitbake/ |grep -v 'bitbake/lib/\(toaster\)\|\(bs4\)\//' |grep -v 'bitbake/lib/bb/tests' ./uncovered.sh scripts/ I hope this should be enough to resolve this ticket. Let me know if anything else is expected. 64417 17 leonardo.sandoval.gonzalez 2016-07-25 19:14:03 +0000 Ed, Looks good to me. Are you planning to push these into poky/scripts/contrib? 64469 18 eduard.bartosh 2016-07-26 09:58:23 +0000 The script has been sent for review: http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124289.html Please note that script code and usage is changed. 64826 19 eduard.bartosh 2016-08-03 08:47:09 +0000 the patch has been merged to master 65154 20 jose.perez.carranza 2016-08-12 20:51:38 +0000 verified on master commit : 6b66e9317f4ec3a69f98f29836aafa35b52f3fc7 3311 2016-07-25 07:38:10 +0000 2016-07-25 07:38:10 +0000 not covered bitbake files bitbake.uncovered application/octet-stream 1129 eduard.bartosh Yml0YmFrZS9iaW4vYml0YmFrZS1kaWZmc2lncwpiaXRiYWtlL2Jpbi9iaXRiYWtlLWR1bXBzaWcK Yml0YmFrZS9iaW4vYml0YmFrZS1zZWxmdGVzdApiaXRiYWtlL2Jpbi9iaXRiYWtlLXdvcmtlcgpi aXRiYWtlL2Jpbi9iaXRkb2MKYml0YmFrZS9iaW4vdG9hc3Rlci1ldmVudHJlcGxheQpiaXRiYWtl L2NvbnRyaWIvZHVtcF9jYWNoZS5weQpiaXRiYWtlL2xpYi9iYi9jYWNoZV9leHRyYS5weQpiaXRi YWtlL2xpYi9iYi9jb21wYXQucHkKYml0YmFrZS9saWIvYmIvQ09XLnB5CmJpdGJha2UvbGliL2Ji L2RhdGFfc21hcnQucHkKYml0YmFrZS9saWIvYmJsYXllcnMvY29tbW9uLnB5CmJpdGJha2UvbGli L2JibGF5ZXJzL19faW5pdF9fLnB5CmJpdGJha2UvbGliL2JiL21ldGhvZHBvb2wucHkKYml0YmFr ZS9saWIvYmIvcGFyc2UvcGFyc2VfcHkvX19pbml0X18ucHkKYml0YmFrZS9saWIvYmIvcHlzaC9i dWlsdGluLnB5CmJpdGJha2UvbGliL2JiL3B5c2gvaW50ZXJwLnB5CmJpdGJha2UvbGliL2JiL3B5 c2gvbHNwcm9mLnB5CmJpdGJha2UvbGliL2JiL3B5c2gvcHlzaC5weQpiaXRiYWtlL2xpYi9iYi9w eXNoL3NoZXJyb3JzLnB5CmJpdGJha2UvbGliL2JiL3B5c2gvc3VicHJvY2Vzc19maXgucHkKYml0 YmFrZS9saWIvYmIvc2VydmVyL3Byb2Nlc3MucHkKYml0YmFrZS9saWIvYmIvc2hlbGwucHkKYml0 YmFrZS9saWIvYmIvdWkvYnVpbGRpbmZvaGVscGVyLnB5CmJpdGJha2UvbGliL2JiL3VpL2RlcGV4 cC5weQpiaXRiYWtlL2xpYi9iYi91aS9uY3Vyc2VzLnB5CmJpdGJha2UvbGliL2JiL3VpL3RvYXN0 ZXJ1aS5weQpiaXRiYWtlL2xpYi9iczQvZGlhZ25vc2UucHkKYml0YmFrZS9saWIvYnM0L3Rlc3Rp bmcucHkKYml0YmFrZS9saWIvYnM0L3Rlc3RzL3Rlc3RfYnVpbGRlcl9yZWdpc3RyeS5weQpiaXRi YWtlL2xpYi9iczQvdGVzdHMvdGVzdF9odG1sNWxpYi5weQpiaXRiYWtlL2xpYi9iczQvdGVzdHMv dGVzdF9odG1scGFyc2VyLnB5CmJpdGJha2UvbGliL2JzNC90ZXN0cy90ZXN0X2x4bWwucHkKYml0 YmFrZS9saWIvYnM0L3Rlc3RzL3Rlc3Rfc291cC5weQpiaXRiYWtlL2xpYi9iczQvdGVzdHMvdGVz dF90cmVlLnB5CmJpdGJha2UvbGliL3Byb2dyZXNzYmFyL19faW5pdF9fLnB5Cg== 3312 2016-07-25 07:39:12 +0000 2016-07-25 07:39:12 +0000 not covered scripts files scripts.uncovered application/octet-stream 1688 eduard.bartosh c2NyaXB0cy9iaXRiYWtlLXdoYXRjaGFuZ2VkCnNjcmlwdHMvYnVpbGRoaXN0b3J5LWNvbGxlY3Qt c3JjcmV2cwpzY3JpcHRzL2NsZWFudXAtd29ya2RpcgpzY3JpcHRzL2NvbWJvLWxheWVyCnNjcmlw dHMvY29udHJpYi9iYnZhcnMucHkKc2NyaXB0cy9jb250cmliL2RldnRvb2wtc3RyZXNzLnB5CnNj cmlwdHMvY29udHJpYi9ncmFwaC10b29sCnNjcmlwdHMvY29udHJpYi9saXN0LXBhY2thZ2Vjb25m aWctZmxhZ3MucHkKc2NyaXB0cy9jb250cmliL3B5dGhvbi9nZW5lcmF0ZS1tYW5pZmVzdC0yLjcu cHkKc2NyaXB0cy9jb250cmliL3B5dGhvbi9nZW5lcmF0ZS1tYW5pZmVzdC0zLjUucHkKc2NyaXB0 cy9jb250cmliL3ZlcmlmeS1ob21lcGFnZS5weQpzY3JpcHRzL2NwLW5vZXJyb3IKc2NyaXB0cy9n ZW4tbG9ja2Vkc2lnLWNhY2hlCnNjcmlwdHMvbGliL2JzcC9lbmdpbmUucHkKc2NyaXB0cy9saWIv YnNwL2hlbHAucHkKc2NyaXB0cy9saWIvYnNwL2tlcm5lbC5weQpzY3JpcHRzL2xpYi9kZXZ0b29s L3N0YW5kYXJkLnB5CnNjcmlwdHMvbGliL3JlY2lwZXRvb2wvY3JlYXRlX25wbS5weQpzY3JpcHRz L2xpYi93aWMvaGVscC5weQpzY3JpcHRzL2xpYi93aWMvcGFydGl0aW9uLnB5CnNjcmlwdHMvbGli L3dpYy9wbHVnaW5zL2ltYWdlci9kaXJlY3RfcGx1Z2luLnB5CnNjcmlwdHMvbGliL3dpYy9wbHVn aW5zL3NvdXJjZS9pc29pbWFnZS1pc29oeWJyaWQucHkKc2NyaXB0cy9saWIvd2ljL3V0aWxzL2Vy cm9ycy5weQpzY3JpcHRzL2xpYi93aWMvdXRpbHMvc3lzbGludXgucHkKc2NyaXB0cy9sbnIKc2Ny aXB0cy9vZS1idWlsZC1wZXJmLXRlc3QKc2NyaXB0cy9vZS1wdWJsaXNoLXNkawpzY3JpcHRzL29l cHlkZXZzaGVsbC1pbnRlcm5hbC5weQpzY3JpcHRzL29lLXNlbGZ0ZXN0CnNjcmlwdHMvb2UtdHJp bS1zY2hlbWFzCnNjcmlwdHMvb3BrZy1xdWVyeS1oZWxwZXIucHkKc2NyaXB0cy9weWJvb3RjaGFy dGd1aS9weWJvb3RjaGFydGd1aS9iYXRjaC5weQpzY3JpcHRzL3B5Ym9vdGNoYXJ0Z3VpL3B5Ym9v dGNoYXJ0Z3VpL2RyYXcucHkKc2NyaXB0cy9weWJvb3RjaGFydGd1aS9weWJvb3RjaGFydGd1aS9n dWkucHkKc2NyaXB0cy9weWJvb3RjaGFydGd1aS9weWJvb3RjaGFydGd1aS9tYWluLnB5LmluCnNj cmlwdHMvcHlib290Y2hhcnRndWkvcHlib290Y2hhcnRndWkvcGFyc2luZy5weQpzY3JpcHRzL3B5 Ym9vdGNoYXJ0Z3VpL3B5Ym9vdGNoYXJ0Z3VpL3Byb2Nlc3NfdHJlZS5weQpzY3JpcHRzL3B5Ym9v dGNoYXJ0Z3VpL3B5Ym9vdGNoYXJ0Z3VpLnB5CnNjcmlwdHMvcHlib290Y2hhcnRndWkvcHlib290 Y2hhcnRndWkvc2FtcGxlcy5weQpzY3JpcHRzL3B5Ym9vdGNoYXJ0Z3VpL3B5Ym9vdGNoYXJ0Z3Vp L3Rlc3RzL3BhcnNlcl90ZXN0LnB5CnNjcmlwdHMvcHlib290Y2hhcnRndWkvcHlib290Y2hhcnRn dWkvdGVzdHMvcHJvY2Vzc190cmVlX3Rlc3QucHkKc2NyaXB0cy9weXRob25kZXBzCnNjcmlwdHMv cmVsb2NhdGVfc2RrLnB5CnNjcmlwdHMvc2VuZC1lcnJvci1yZXBvcnQKc2NyaXB0cy9zeXNyb290 LXJlbGF0aXZlbGlua3MucHkKc2NyaXB0cy90ZXN0LXJlbW90ZS1pbWFnZQpzY3JpcHRzL3Rpbnkv ZGlyc2l6ZS5weQpzY3JpcHRzL3Rpbnkva3NpemUucHkKc2NyaXB0cy95b2N0by1ic3AKc2NyaXB0 cy95b2N0by1rZXJuZWwKc2NyaXB0cy95b2N0by1sYXllcgo=