Bug 9809

Summary: List python/bitbake libraries that are not being tested by oe-selftest
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Component: Scripts and ToolsAssignee: Ed Bartosh <eduard.bartosh>
Status: VERIFIED FIXED QA Contact: Jose Perez C <jose.perez.carranza>
Severity: normal    
Priority: Medium+ CC: bluelightning, eduard.bartosh, jose.perez.carranza, mariano.lopez, ross.burton, sgw
Version: 2.2   
Target Milestone: 2.2 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
not covered bitbake files
none
not covered scripts files none

Description Leonardo Sandoval Gonzalez 2016-06-21 22:01:55 UTC 
We should at least have a list of those libraries not being tested by oe-selftest. Once we have this list, we can start filing bugs to test the most important features on the untested code.

This tasks is particulary important on this phase, where the python2 to python3 is taking place.
Comment 1 Ed Bartosh 2016-06-22 08:16:59 UTC 
I'd propose to concentrate on test caverage instead. The fact that there is a test for some code doesn't always mean it's properly tested.
Comment 2 Leonardo Sandoval Gonzalez 2016-06-22 13:27:28 UTC 
Right. I did not mention coverage explicitly but that is a way to partially answer the question, at least on the python files. But this bug is about finding those files with no tests at all. In other words, This bug does not care about the quality of test cases or coverage, just about those showing those untested files.
Comment 3 Leonardo Sandoval Gonzalez 2016-06-23 15:02:07 UTC 
Just to avoid confusing. This bug tries to find those untested libraries;  once found (if any), new bugs will be created. Once bug creation is done, this bug can be resolved.
Comment 4 Ed Bartosh 2016-07-07 08:38:57 UTC 
Sadly running oe-selftest --coverage -a produced empty coverage report. I was going to use coverage report to find out which files are covered.

# Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -a'
# HEAD commit 5c11e365e19357f721c49d076971567e7b64b61b
[run]
data_file = /home/ed/git/yocto/poky/build/.coverage.20160706T183956
branch = True
source = 
    /home/ed/git/yocto/poky/meta
    /home/ed/git/yocto/poky/meta-yocto
    /home/ed/git/yocto/poky/meta-yocto-bsp
    /home/ed/git/yocto/poky/meta-selftest
    /home/ed/git/yocto/poky/scripts
    /home/ed/git/yocto/poky/bitbake

2016-07-07 00:49:16,878 - selftest - INFO - Coverage Report
2016-07-07 00:49:16,879 - selftest - INFO - ===============
2016-07-07 00:49:16,938 - selftest - INFO - 
Name                                                    Stmts   Miss  Cover
---------------------------------------------------------------------------
/home/ed/git/yocto/poky/scripts/oepydevshell-internal      73     73     0%
/home/ed/git/yocto/poky/scripts/opkg-query-helper          52     52     0%
/home/ed/git/yocto/poky/scripts/relocate_sdk              153    153     0%
/home/ed/git/yocto/poky/scripts/sysroot-relativelinks      21     21     0%
---------------------------------------------------------------------------
TOTAL                                                     299    299     0%


----------------------------------------------------------------------
Ran 187 tests in 22160.353s

OK (skipped=1)
Comment 5 Leonardo Sandoval Gonzalez 2016-07-07 14:26:22 UTC 
(In reply to comment #4)
> Sadly running oe-selftest --coverage -a produced empty coverage report. I
> was going to use coverage report to find out which files are covered.
> 
> # Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest
> --coverage -a'
> # HEAD commit 5c11e365e19357f721c49d076971567e7b64b61b
> [run]
> data_file = /home/ed/git/yocto/poky/build/.coverage.20160706T183956
> branch = True
> source = 
>     /home/ed/git/yocto/poky/meta
>     /home/ed/git/yocto/poky/meta-yocto
>     /home/ed/git/yocto/poky/meta-yocto-bsp
>     /home/ed/git/yocto/poky/meta-selftest
>     /home/ed/git/yocto/poky/scripts
>     /home/ed/git/yocto/poky/bitbake
> 
> 2016-07-07 00:49:16,878 - selftest - INFO - Coverage Report
> 2016-07-07 00:49:16,879 - selftest - INFO - ===============
> 2016-07-07 00:49:16,938 - selftest - INFO - 
> Name                                                    Stmts   Miss  Cover
> ---------------------------------------------------------------------------
> /home/ed/git/yocto/poky/scripts/oepydevshell-internal      73     73     0%
> /home/ed/git/yocto/poky/scripts/opkg-query-helper          52     52     0%
> /home/ed/git/yocto/poky/scripts/relocate_sdk              153    153     0%
> /home/ed/git/yocto/poky/scripts/sysroot-relativelinks      21     21     0%
> ---------------------------------------------------------------------------
> TOTAL                                                     299    299     0%
> 
> 
> ----------------------------------------------------------------------
> Ran 187 tests in 22160.353s
> 
> OK (skipped=1)

Let me investigate. Coverage was fully working before. However, since the switch to python3, no testing has been done.
Comment 6 Leonardo Sandoval Gonzalez 2016-07-07 14:33:48 UTC 
Humberto, please check oe-selftest coverage on latest master.
Comment 7 Humberto Ibarra 2016-07-13 16:59:30 UTC 
I ran oe-selftest -a --coverage and got a valid output with all files covered.

@Ed can you please elaborate on your failure? What's the coverage version you are using?
Comment 8 Ed Bartosh 2016-07-21 16:27:46 UTC 
I tried to run oe-selftest again and got 0% coverages again:

$ oe-selftest --coverage -r wic.Wic.test_build_image_name
2016-07-21 18:55:43,753 - selftest - INFO - Running bitbake -e to get BBPATH
2016-07-21 18:55:45,054 - selftest - INFO - Checking that everything is in order before running the tests
2016-07-21 18:55:46,218 - selftest - INFO - Running bitbake -p
2016-07-21 18:55:49,307 - selftest - INFO - Loading tests from: oeqa.selftest.wic.Wic.test_build_image_name
2016-07-21 18:55:49,314 - selftest - INFO - Adding: "include selftest.inc" in local.conf
2016-07-21 18:55:49,314 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2016-07-21 18:55:49,343 - selftest - INFO - Coverage is enabled
2016-07-21 18:55:50 - test_build_image_name (oeqa.selftest.wic.Wic)
Test wic create directdisk --image-name core-image-minimal ... ok
2016-07-21 19:00:01,120 - selftest - INFO - Coverage configuration file (/home/ed/git/yocto/poky/build/.coveragerc)
2016-07-21 19:00:01,121 - selftest - INFO - ===========================
2016-07-21 19:00:01,121 - selftest - INFO - 
# Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -r wic.Wic.test_build_image_name'
# HEAD commit 36feb38045b7a2af86ece147fec54b0db3bf491f
[run]
data_file = /home/ed/git/yocto/poky/build/.coverage.20160721T185549
branch = True
source = 
    /home/ed/git/yocto/poky/meta
    /home/ed/git/yocto/poky/meta-yocto
    /home/ed/git/yocto/poky/meta-yocto-bsp
    /home/ed/git/yocto/poky/meta-selftest
    /home/ed/git/yocto/poky/scripts
    /home/ed/git/yocto/poky/bitbake

2016-07-21 19:00:01,122 - selftest - INFO - Coverage Report
2016-07-21 19:00:01,122 - selftest - INFO - ===============
2016-07-21 19:00:01,176 - selftest - INFO - 
Name                                                    Stmts   Miss  Cover
---------------------------------------------------------------------------
/home/ed/git/yocto/poky/scripts/oepydevshell-internal      72     72     0%
/home/ed/git/yocto/poky/scripts/opkg-query-helper          52     52     0%
/home/ed/git/yocto/poky/scripts/relocate_sdk              153    153     0%
/home/ed/git/yocto/poky/scripts/sysroot-relativelinks      21     21     0%
---------------------------------------------------------------------------
TOTAL                                                     298    298     0%


----------------------------------------------------------------------
Ran 1 test in 251.862s

OK
2016-07-21 19:00:01,177 - selftest - INFO - Finished
2016-07-21 19:00:01,178 - selftest - INFO - Removing the include from local.conf
2016-07-21 19:00:01,179 - selftest - INFO - Removing the include from bblayers.conf




Here is the list of coverage packages installed:
$ rpm -qa |grep coverage
python3-coverage-3.7.1-2.1.3.x86_64
python-coverage-3.7.1-2.1.4.x86_64
Comment 9 Ed Bartosh 2016-07-22 08:54:19 UTC 
Another try - same result. This time I'm trying to run another test on another machine:

e-selftest --coverage -r manifest.VerifyManifest.test_image_manifest_entries
2016-07-22 11:52:02,214 - selftest - INFO - Running bitbake -e to get BBPATH
2016-07-22 11:52:03,029 - selftest - INFO - Checking that everything is in order before running the tests
2016-07-22 11:52:03,854 - selftest - INFO - Running bitbake -p
2016-07-22 11:52:05,634 - selftest - INFO - Loading tests from: oeqa.selftest.manifest.VerifyManifest.test_image_manifest_entries
2016-07-22 11:52:05,635 - selftest - INFO - Adding: "include selftest.inc" in local.conf
2016-07-22 11:52:05,635 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2016-07-22 11:52:05,646 - selftest - INFO - Coverage is enabled
2016-07-22 11:52:06,315 - selftest.base - INFO - VerifyManifest: doing bitbake core-image-minimal as a prerequisite of the test
2016-07-22 11:52:17 - test_image_manifest_entries (oeqa.selftest.manifest.VerifyManifest)
Verifying the image manifest entries exist ... ok
2016-07-22 11:52:28,933 - selftest - INFO - Coverage configuration file (/home/ed/git/yocto/poky/build/.coveragerc)
2016-07-22 11:52:28,933 - selftest - INFO - ===========================
2016-07-22 11:52:28,933 - selftest - INFO - 
# Generated with command '/home/ed/git/yocto/poky/scripts/oe-selftest --coverage -r manifest.VerifyManifest.test_image_manifest_entries'
# HEAD commit 36feb38045b7a2af86ece147fec54b0db3bf491f
[run]
data_file = /home/ed/git/yocto/poky/build/.coverage.20160722T115205
branch = True
source = 
    /home/ed/git/yocto/poky/meta
    /home/ed/git/yocto/poky/meta-poky
    /home/ed/git/yocto/poky/meta-yocto-bsp
    /home/ed/git/yocto/poky/meta-selftest
    /home/ed/git/yocto/poky/scripts
    /home/ed/git/yocto/poky/bitbake

2016-07-22 11:52:28,933 - selftest - INFO - Coverage Report
2016-07-22 11:52:28,933 - selftest - INFO - ===============
2016-07-22 11:52:28,949 - selftest - INFO - 
Name                                                    Stmts   Miss  Cover
---------------------------------------------------------------------------
/home/ed/git/yocto/poky/scripts/oepydevshell-internal      72     72     0%
/home/ed/git/yocto/poky/scripts/opkg-query-helper          52     52     0%
/home/ed/git/yocto/poky/scripts/relocate_sdk              153    153     0%
/home/ed/git/yocto/poky/scripts/sysroot-relativelinks      21     21     0%
---------------------------------------------------------------------------
TOTAL                                                     298    298     0%


----------------------------------------------------------------------
Ran 1 test in 23.314s

OK
2016-07-22 11:52:28,950 - selftest - INFO - Finished
2016-07-22 11:52:28,950 - selftest - INFO - Removing the include from local.conf
2016-07-22 11:52:28,950 - selftest - INFO - Removing the include from bblayers.conf


> rpm -qa |grep coverage
python3-coverage-3.7.1-3.3.x86_64
Comment 10 Ed Bartosh 2016-07-22 09:15:27 UTC 
It's interesting that the coverage output is the same for all different tests. It always includes 4 files from scripts/ directory.
Comment 11 Ed Bartosh 2016-07-22 09:39:12 UTC 
looks like coverage report contains only files with .py extension from scripts/:

> ls -la ../scripts/*.py
-rwxr-xr-x 1 ed users 2512 Jul 21 18:42 ../scripts/oepydevshell-internal.py
-rwxr-xr-x 1 ed users 2767 Jul 21 18:42 ../scripts/opkg-query-helper.py
-rwxr-xr-x 1 ed users 9071 Jul 21 18:42 ../scripts/relocate_sdk.py
-rwxr-xr-x 1 ed users  960 Jul 21 18:42 ../scripts/sysroot-relativelinks.py
Comment 12 Ed Bartosh 2016-07-22 11:55:14 UTC 
coverage itself works just fine:

> coverage run ../bitbake/bin/bitbake-layers show-layers
...
> > coverage report | head -10
Name                                                                Stmts   Miss Branch BrMiss  Cover
-----------------------------------------------------------------------------------------------------
/home/ed/git/yocto/poky/bitbake/bin/bitbake-layers                     75     13     20      6    80%
/home/ed/git/yocto/poky/bitbake/lib/bb/COW                            223    144     70     60    30%
/home/ed/git/yocto/poky/bitbake/lib/bb/__init__                        72     13     14      5    79%
/home/ed/git/yocto/poky/bitbake/lib/bb/build                          534    449    188    174    14%
/home/ed/git/yocto/poky/bitbake/lib/bb/cache                          536    427    228    218    16%
/home/ed/git/yocto/poky/bitbake/lib/bb/checksum                        89     65     36     36    19%
/home/ed/git/yocto/poky/bitbake/lib/bb/codeparser                     294    121    128     85    51%
/home/ed/git/yocto/poky/bitbake/lib/bb/command                        256    168     30     30    31%
Comment 13 Ed Bartosh 2016-07-25 07:30:08 UTC 
It looks like oe-selftest --coverage doesn't work only on OpenSUSE.

I run it on Fedora and it produced valid report just fine.
Comment 14 Ed Bartosh 2016-07-25 07:38:10 UTC 
Created attachment 3311 [details]
not covered bitbake files
Comment 15 Ed Bartosh 2016-07-25 07:39:12 UTC 
Created attachment 3312 [details]
not covered scripts files
Comment 16 Ed Bartosh 2016-07-25 07:39:47 UTC 
here is a simple script to report uncovered python files based on coverage report:

$ cat uncovered.sh 
#!/bin/sh
path=$1

if [ -z "$path" ] ; then
     echo "Usage: ./uncovered <dir>"
     exit 1
fi

if [ ! -d "$path" ] ; then
    echo "directory $path doesn't exits"
    exit 1
fi

coverage3 report --rcfile=build/.coveragerc | grep "$path" | grep -v '0%$' | grep -v '\.pyc$' | \
                 xargs file | grep 'Python script' | cut -f1 -d: | sort > /tmp/covered.tmp
find $path | grep -v '\.pyc$' | xargs file | grep 'Python script' | cut -f1 -d: | sort > /tmp/all.tmp
diff -u /tmp/covered.tmp /tmp/all.tmp | grep "^+$path" | sed -ne 's/+\(.*\)/\1/p'
rm /tmp/covered.tmp /tmp/all.tmp


I've attached output of the following 2 commands:

./uncovered.sh bitbake/ |grep -v 'bitbake/lib/\(toaster\)\|\(bs4\)\//' |grep -v 'bitbake/lib/bb/tests'

./uncovered.sh scripts/


I hope this should be enough to resolve this ticket. Let me know if anything else is expected.
Comment 17 Leonardo Sandoval Gonzalez 2016-07-25 19:14:03 UTC 
Ed,

Looks good to me. Are you planning to push these into poky/scripts/contrib?
Comment 18 Ed Bartosh 2016-07-26 09:58:23 UTC 
The script has been sent for review: http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124289.html

Please note that script code and usage is changed.
Comment 19 Ed Bartosh 2016-08-03 08:47:09 UTC 
the patch has been merged to master
Comment 20 Jose Perez C 2016-08-12 20:51:38 UTC 
verified on master commit : 6b66e9317f4ec3a69f98f29836aafa35b52f3fc7