Bug 11012

Summary: oe-selftest.log is not reporting failures when using xmlrunner
Product: [QA/Testing] Build Testing Reporter: Mariano Lopez <mariano.lopez>
Component: generalAssignee: Benjamin Esquivel <benjamin.esquivel>
Status: VERIFIED FIXED QA Contact: Jose Perez C <jose.perez.carranza>
Severity: normal    
Priority: High CC: benjamin.esquivel
Version: 2.2   
Target Milestone: 2.3 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Mariano Lopez 2017-02-03 15:06:57 UTC 
If xmlrunner is enabled and a test fails, the log (results-oe-selftest.log) will mark test as PASSED. Steps to reproduce this:

1. Install xmlrunner (unittest-xml-reporting)
2. Checkout poky to 0ce888c4dc76b6e7ec09beb6ad74b85a87a54f7c revision
3. Add meta-selftest to bblayers.conf
4. Run 'oe-selftest -r sstatetests.SStateTests.test_sstate_sametune_samesigs'
5. results-oe-selftest.log will have the test as PASSED even as you saw it FAIL
Comment 1 Mariano Lopez 2017-02-03 15:26:24 UTC 
The revision is incorrect, it should be:

f431f766a0c5cd2c4545deff5c56e9918117d325
Comment 2 Benjamin Esquivel 2017-02-03 15:51:08 UTC 
I verified it fails, when oe-selftest writes to the log it writes a mistaken result.
Comment 3 Benjamin Esquivel 2017-02-13 02:59:34 UTC 
review sent, can be seen at:

https://patchwork.openembedded.org/patch/137058/

and at:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=besquive/bug-11012

Mariano, would you check it?
Comment 5 Jose Perez C 2017-02-21 15:33:52 UTC 
Tested in GDC local AB and is working correctly, all the logs generated are consistent.