Bug 11012 - oe-selftest.log is not reporting failures when using xmlrunner
Summary: oe-selftest.log is not reporting failures when using xmlrunner
Status: VERIFIED FIXED
Alias: None
Product: Build Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 2.2
Hardware: x86 Multiple
: High normal
Target Milestone: 2.3 M3
Assignee: Benjamin Esquivel
QA Contact: Jose Perez C
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-02-03 15:06 UTC by Mariano Lopez
Modified: 2017-02-21 15:33 UTC (History)
1 user (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Mariano Lopez 2017-02-03 15:06:57 UTC 
If xmlrunner is enabled and a test fails, the log (results-oe-selftest.log) will mark test as PASSED. Steps to reproduce this:

1. Install xmlrunner (unittest-xml-reporting)
2. Checkout poky to 0ce888c4dc76b6e7ec09beb6ad74b85a87a54f7c revision
3. Add meta-selftest to bblayers.conf
4. Run 'oe-selftest -r sstatetests.SStateTests.test_sstate_sametune_samesigs'
5. results-oe-selftest.log will have the test as PASSED even as you saw it FAIL
Comment 1 Mariano Lopez 2017-02-03 15:26:24 UTC 
The revision is incorrect, it should be:

f431f766a0c5cd2c4545deff5c56e9918117d325
Comment 2 Benjamin Esquivel 2017-02-03 15:51:08 UTC 
I verified it fails, when oe-selftest writes to the log it writes a mistaken result.
Comment 3 Benjamin Esquivel 2017-02-13 02:59:34 UTC 
review sent, can be seen at:

https://patchwork.openembedded.org/patch/137058/

and at:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=besquive/bug-11012

Mariano, would you check it?
Comment 5 Jose Perez C 2017-02-21 15:33:52 UTC 
Tested in GDC local AB and is working correctly, all the logs generated are consistent.