When python3 unittest-xml-reporting is installed (i.e. "import xmlrunner" works), then oe-selftest produces a directory called "TestResults_<date>_<number>" with individual .xml files. We should enable that in our CI system by installing unittest-xml-reporting for Python3 and publish the resulting .xml files so that Jenkins shows the same way as DAFT test results. unittest-xml-reporting wasn't packaged for Debian, but installation with "sudo pip3 install unittest-xml-reporting" worked fine.
python3-unittest-xml-reporting is pre-packaged for opensuse, adding it to Docker image is now a commit in PR#205
test run of #205 where python3-unittest-xml-reporting was added, is here: https://ostroproject.org/jenkins/job/intel-iot-refkit_pull-requests/1668/consoleFull but I don't see any directory called "TestResults_<date>_<number>" on worker, did I miss something about enabling this? Where exactly should this directory appear, on main level, under build/ or somewhere else?
explanation is, python3-unittest-xml-reporting is not enough: ImportError: No module named 'six' with python3-six added as well, import xmlrunner works. I will create a new PR (as 205 got dedicated) and try.
1st part verified, PR build created TestResults_ directory with 5 xml-files.
Test results parsed and published by jenkins: https://ostroproject.org/jenkins/job/intel-iot-refkit_pull-requests/1781/testReport/ (result of testing PR#210) The TEST-*.xml files are also accessible under "Build Artifacts" section of that job.
(In reply to comment #5) > Test results parsed and published by jenkins: > > https://ostroproject.org/jenkins/job/intel-iot-refkit_pull-requests/1781/ > testReport/ > > (result of testing PR#210) > > The TEST-*.xml files are also accessible under "Build Artifacts" > section of that job. Looks nice! Is it possible to add the results in the email summary too?
PR#210 now has email report part. I was thinking, what to write as name of image for these tests, initially I used "post-build", but that's not really image name, please propose better if yo have ideas. I guess Device=qemu is correctly expressed. In current run of PR#210 that part of report shows like: Image: post-build Device: qemu Total:16 Pass:15 Fail:0 Skip:1 Error:0 N/A:0 Run rate:93% Pass rate of total:93% Pass rate of exec:100% This part will be shown as first, as post-build testing is first to complete.
(In reply to comment #7) > I was thinking, what to write as name of image for these tests, > initially I used "post-build", but that's not really image name, > please propose better if yo have ideas. > I guess Device=qemu is correctly expressed. We don't necessarily have to follow the same pattern. Some of the self-tests do use qemu but not all so Device=qemu is not 100% correct. How about simply oe-selftest: [pre|post]-build Total:16 Pass:15 Fail:0 Skip:1 Error:0 N/A:0 Run rate:93% Pass rate of total:93% Pass rate of exec:100%
I had overlooked pre-test totally, there are XML files produced also. https://ostroproject.org/jenkins/job/intel-iot-refkit_pull-requests/1893/console (near end of output) is example of how Summary will look like, if PR#210 gets merged as it is now.
PR#210 was merged and this is now functional
whats the known method to get python3 xmlrunner on Ubuntu 16.04.2 ? package-install based solution is preferred, but I dont find such packaged. There is python-xmlrunner package but it only provides xmlrunner for python2. Installing python3-junitxml and/or python3-unittest2 does not help.
(In reply to comment #11) > whats the known method to get python3 xmlrunner on Ubuntu 16.04.2 ? Install the python3-pip package, then run (as root) "pip3 install unitest-xml-reporting".