Bug 13663

Summary: oeqa doesn't handle skipped classes of tests correctly
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: coreAssignee: Trevor Gamblin <tvgamblin>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: liezhi.yang, meta.mr.watcher, meta.watcher, randy.macleod, tim.orling, tvgamblin
Version: 3.0   
Target Milestone: 3.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2019-11-28 12:19:51 UTC 
A simple example is to take a machine without python3-cairo installed and run "e-selftest -r oescripts.OEPybootchartguyTests". If you run without -j, you'll see the counts are wrong, 4 tests were marked as SKIPPED but the count says 1. With -j, the test status is marked as UNKNOWN.
Comment 1 Richard Purdie 2019-11-28 12:20:27 UTC 
Without -j showing incorrect skipped count:

pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r oescripts.OEPybootchartguyTests 
2019-11-28 12:14:50,179 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-11-28 12:14:56,400 - oe-selftest - INFO - Adding layer libraries:
2019-11-28 12:14:56,400 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib
2019-11-28 12:14:56,401 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib
2019-11-28 12:14:56,401 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib
2019-11-28 12:14:56,403 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-11-28 12:14:58,426 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build/conf/local.conf
2019-11-28 12:14:58,427 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-11-28 12:14:58,428 - oe-selftest - INFO - skipped 'Python module cairo is not present'
2019-11-28 12:14:58,428 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-28 12:14:58,429 - oe-selftest - INFO - Ran 0 tests in 0.003s
2019-11-28 12:14:58,429 - oe-selftest - INFO - OK
2019-11-28 12:14:58,429 - oe-selftest - INFO -  (skipped=1)
2019-11-28 12:15:04,734 - oe-selftest - INFO - RESULTS:
2019-11-28 12:15:04,735 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED
2019-11-28 12:15:04,735 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED
2019-11-28 12:15:04,735 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED
2019-11-28 12:15:04,735 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED
2019-11-28 12:15:04,741 - oe-selftest - INFO - SUMMARY:
2019-11-28 12:15:04,741 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.005s
2019-11-28 12:15:04,741 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=1, failures=0, errors=0)
Comment 2 Richard Purdie 2019-11-28 12:21:28 UTC 
Broken output with UNKNOWN from output wiht -j:

pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2
2019-11-28 12:14:17,367 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-11-28 12:14:23,499 - oe-selftest - INFO - Adding layer libraries:
2019-11-28 12:14:23,499 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib
2019-11-28 12:14:23,500 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib
2019-11-28 12:14:23,500 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib
2019-11-28 12:14:23,503 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-11-28 12:14:28,385 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-55667/conf/local.conf
2019-11-28 12:14:28,386 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
skip: setUpClass (oescripts.OEPybootchartguyTests) [
Python module cairo is not present
]
2019-11-28 12:14:28,408 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-28 12:14:28,409 - oe-selftest - INFO - Ran 0 tests in 2.335s
2019-11-28 12:14:28,409 - oe-selftest - INFO - OK
2019-11-28 12:14:34,701 - oe-selftest - INFO - RESULTS:
2019-11-28 12:14:34,701 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: UNKNOWN
2019-11-28 12:14:34,702 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: UNKNOWN
2019-11-28 12:14:34,702 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: UNKNOWN
2019-11-28 12:14:34,702 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: UNKNOWN
2019-11-28 12:14:34,714 - oe-selftest - INFO - SUMMARY:
2019-11-28 12:14:34,715 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 2.725s
2019-11-28 12:14:34,715 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=0, failures=0, errors=0)
Comment 3 Richard Purdie 2019-11-28 12:24:09 UTC 
Failure could also be reproduced by using an invalid python module name. Aim of the bug is to fix the counts and status with and without the -j option.
Comment 4 Trevor Gamblin 2019-12-02 03:40:03 UTC 
Been looking at this off and on this weekend. The unittest framework is unfamiliar to me so I've been spending a fair amount of that time just understanding how everything's working.

I found this thread on another project that looks to be a similar issue:
https://github.com/CleanCut/green/issues/159

Here's what I see with some debug print calls added to meta/lib/oeqa/core/runner.py:

2019-12-01 18:50:48,027 - oe-selftest - INFO - SUMMARY:
2019-12-01 18:50:48,027 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.005s
in wasSuccessful
len(self.failures): 0
len(self.errors): 0
len(self.skipped): 1
self.skipped: [(<ErrorHolder description='setUpClass (oescripts.OEPybootchartguyTests)'>, 'Python module cairo is not present')]
wasSuccessful: True

So self.skipped is getting set once to the "error" log, instead of getting the right count of skips with the case IDs. That sort of lines up with what I see in the link above.

For reference, the start of the OEScriptTests class in meta/lib/oeqa/selftest/cases/oescripts.py (with line numbers):

36 class OEScriptTests(OESelftestTestCase):
37 
38     @classmethod
39     def setUpClass(cls):
40         super(OEScriptTests, cls).setUpClass()
41         try:
42             import cairo
43         except ImportError:
44             raise unittest.SkipTest('Python module cairo is not present')


Will try the solution in the GitHub link in the AM.
Comment 5 Trevor Gamblin 2019-12-02 15:20:47 UTC 
Found a fix for the skip counts:

2019-12-02 07:03:31,931 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_svg_output (oescripts.OEPybootchartguyTests)
2019-12-02 07:03:31,933 - oe-selftest - INFO -  ... skipped 'Python module cairo is not present'
2019-12-02 07:03:31,934 - oe-selftest - INFO - Python module cairo is not present
2019-12-02 07:03:31,934 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-02 07:03:31,934 - oe-selftest - INFO - Ran 4 tests in 29.850s
2019-12-02 07:03:31,935 - oe-selftest - INFO - OK
2019-12-02 07:03:31,935 - oe-selftest - INFO -  (skipped=4)
2019-12-02 07:03:34,566 - oe-selftest - INFO - RESULTS:
2019-12-02 07:03:34,567 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED (0.00s)
2019-12-02 07:03:34,567 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED (0.00s)
2019-12-02 07:03:34,567 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED (0.00s)
2019-12-02 07:03:34,567 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED (0.00s)
2019-12-02 07:03:34,569 - oe-selftest - INFO - SUMMARY:
2019-12-02 07:03:34,569 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 29.850s
2019-12-02 07:03:34,569 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=4, failures=0, errors=0)
tgamblin@ala-lpggp2$

The solution is indeed what I mentioned in the previous comment, based on the link - the try/except in setUpClass() is being treated by unittest as an issue with setting up the entire OEPybootchartguyTests class. The crude-but-functional way to get it to count is to remove that try/except verbatim into each of the test functions in OEPybootchartguyTests. 

Still looking at it.
Comment 6 Trevor Gamblin 2019-12-02 18:47:14 UTC 
Moving the try/except fixes the UNKNOWN results issue, too:

tgamblin@ala-lpggp2$oe-selftest -r oescripts.OEPybootchartguyTests -j 2
2019-12-02 10:24:10,309 - oe-selftest - INFO - Adding layer libraries:
2019-12-02 10:24:10,309 - oe-selftest - INFO -  /ala-lpggp22/tgamblin/oe-core.git/meta/lib
2019-12-02 10:24:10,309 - oe-selftest - INFO -  /ala-lpggp22/tgamblin/oe-core.git/meta-selftest/lib
2019-12-02 10:24:10,310 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-02 10:24:13,172 - oe-selftest - INFO - Adding: "include selftest.inc" in /ala-lpggp22/tgamblin/build-st-80515/conf/local.conf
2019-12-02 10:24:13,173 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-02 10:45:32,050 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_help (subunit.RemotedTestCase)
2019-12-02 10:45:32,051 - oe-selftest - INFO -  ... skipped 'Python module cairo is not present\n'
2019-12-02 10:45:32,051 - oe-selftest - INFO - 0: 1/4 1/4 (0.01s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_help)
2019-12-02 10:45:32,051 - oe-selftest - INFO - Python module cairo is not present

2019-12-02 10:45:32,056 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output (subunit.RemotedTestCase)
2019-12-02 10:45:32,056 - oe-selftest - INFO -  ... skipped 'Python module cairo is not present\n'
2019-12-02 10:45:32,057 - oe-selftest - INFO - 0: 2/4 2/4 (0.01s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output)
2019-12-02 10:45:32,057 - oe-selftest - INFO - Python module cairo is not present

2019-12-02 10:45:32,062 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output (subunit.RemotedTestCase)
2019-12-02 10:45:32,062 - oe-selftest - INFO -  ... skipped 'Python module cairo is not present\n'
2019-12-02 10:45:32,063 - oe-selftest - INFO - 0: 3/4 3/4 (0.01s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output)
2019-12-02 10:45:32,063 - oe-selftest - INFO - Python module cairo is not present

2019-12-02 10:45:32,068 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output (subunit.RemotedTestCase)
2019-12-02 10:45:32,068 - oe-selftest - INFO -  ... skipped 'Python module cairo is not present\n'
2019-12-02 10:45:32,069 - oe-selftest - INFO - 0: 4/4 4/4 (0.01s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output)
2019-12-02 10:45:32,069 - oe-selftest - INFO - Python module cairo is not present

2019-12-02 10:46:14,080 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-02 10:46:14,080 - oe-selftest - INFO - Ran 4 tests in 1322.348s
2019-12-02 10:46:14,080 - oe-selftest - INFO - OK
2019-12-02 10:46:14,080 - oe-selftest - INFO -  (skipped=4)
2019-12-02 10:46:17,423 - oe-selftest - INFO - RESULTS:
2019-12-02 10:46:17,424 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED (0.01s)
2019-12-02 10:46:17,424 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED (0.01s)
2019-12-02 10:46:17,424 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED (0.01s)
2019-12-02 10:46:17,425 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED (0.01s)
2019-12-02 10:46:17,426 - oe-selftest - INFO - SUMMARY:
2019-12-02 10:46:17,426 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 1322.494s
2019-12-02 10:46:17,426 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=4, failures=0, errors=0)
tgamblin@ala-lpggp2$
Comment 7 Trevor Gamblin 2019-12-03 18:47:14 UTC 
Just doing one last round of testing of the patch before submission.
Comment 8 Trevor Gamblin 2019-12-04 21:12:22 UTC 
The issue is earlier in the process than the _getTestResultDetails method. Here are the logs with print("rtype: {0}".format(getattr(self, rtype))) added at the top of the for loop, and print("At the end, printing UNKNOWN") just before the return of UNKNOWN:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests                                                                                                                                          [49/1968]
2019-12-04 16:05:30,695 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-04 16:05:30,696 - oe-selftest - INFO - Adding layer libraries:                      
2019-12-04 16:05:30,696 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib                          
2019-12-04 16:05:30,696 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib                                                                                                                                                  
2019-12-04 16:05:30,697 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-04 16:05:32,931 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build/conf/local.conf                                                                                                                         
2019-12-04 16:05:32,931 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf                                                                                                                                                
2019-12-04 16:05:32,932 - oe-selftest - INFO - skipped 'Python module cairo is not present'                                                                                                                                                   
2019-12-04 16:05:32,933 - oe-selftest - INFO - ----------------------------------------------------------------------                                                                                                                         
2019-12-04 16:05:32,933 - oe-selftest - INFO - Ran 0 tests in 0.003s
2019-12-04 16:05:32,933 - oe-selftest - INFO - OK                                                                      
2019-12-04 16:05:32,933 - oe-selftest - INFO -  (skipped=1)                                                                                                                                                                                   
2019-12-04 16:05:36,103 - oe-selftest - INFO - RESULTS:
rtype: []                                                                                                              
rtype: []                                    
rtype: [(<ErrorHolder description='setUpClass (oescripts.OEPybootchartguyTests)'>, 'Python module cairo is not present')]
rtype: []
rtype: []
rtype: [(<ErrorHolder description='setUpClass (oescripts.OEPybootchartguyTests)'>, 'Python module cairo is not present')]
rtype: []
rtype: []
rtype: [(<ErrorHolder description='setUpClass (oescripts.OEPybootchartguyTests)'>, 'Python module cairo is not present')]
rtype: []
rtype: []
rtype: [(<ErrorHolder description='setUpClass (oescripts.OEPybootchartguyTests)'>, 'Python module cairo is not present')]
2019-12-04 16:05:36,104 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED
2019-12-04 16:05:36,104 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED
2019-12-04 16:05:36,105 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED
2019-12-04 16:05:36,105 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED
2019-12-04 16:05:36,113 - oe-selftest - INFO - SUMMARY:
2019-12-04 16:05:36,113 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.004s
2019-12-04 16:05:36,113 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=1, failures=0, errors=0)

And with -j 2:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2                                                                                                                                      [7/1959]
2019-12-04 16:05:54,299 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build                                                                                                                                                           
2019-12-04 16:05:54,299 - oe-selftest - INFO - Adding layer libraries:                                                                                                                                                                        
2019-12-04 16:05:54,299 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib                                                                                                                                                           
2019-12-04 16:05:54,299 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib                                                                                                                                                  
2019-12-04 16:05:54,300 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable                                                                                                                                 
2019-12-04 16:05:59,116 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build-st-3328437/conf/local.conf
2019-12-04 16:05:59,117 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf                         
skip: setUpClass (oescripts.OEPybootchartguyTests) [                                                                   
Python module cairo is not present                                                                                     
]                                                                                                                      
2019-12-04 16:05:59,151 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-04 16:05:59,152 - oe-selftest - INFO - Ran 0 tests in 2.455s
2019-12-04 16:05:59,152 - oe-selftest - INFO - OK      
2019-12-04 16:06:02,338 - oe-selftest - INFO - RESULTS:
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
At the end, printing UNKNOWN
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
At the end, printing UNKNOWN
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
rtype: []                   
At the end, printing UNKNOWN
rtype: []
rtype: []
rtype: []
rtype: []
rtype: []
rtype: []
At the end, printing UNKNOWN
2019-12-04 16:06:02,339 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: UNKNOWN
2019-12-04 16:06:02,339 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: UNKNOWN
2019-12-04 16:06:02,339 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: UNKNOWN
2019-12-04 16:06:02,340 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: UNKNOWN
2019-12-04 16:06:02,347 - oe-selftest - INFO - SUMMARY:
2019-12-04 16:06:02,348 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 2.608s
2019-12-04 16:06:02,348 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=0, failures=0, errors=0)
Comment 9 Trevor Gamblin 2019-12-05 01:43:16 UTC 
I think doing this in meta/lib/oeqa/selftest/cases/oescripts.py is actually our best bet:

37 @unittest.skipUnless("cairo" in sys.modules, "Python cairo module is not present")
38 class OEScriptTests(OESelftestTestCase):

It's likely not the most general we could get, and requires manual modification for this bug (and any future skip count issues), but it's directly from unittest and is (IMO) the more correct way to handle checking for a required module (in this case cairo) than using the try/except block.

I'll submit another patch with a cover letter mirroring this tomorrow morning.
Comment 10 Trevor Gamblin 2019-12-05 16:50:35 UTC 
After a discussion yesterday about this fix and a little more
investigation, I think the problem is ultimately that the technique used
to check for the cairo module in 785bb108ce wasn't using the unittest
framework where it ought to be. Specifically, unittest has decorators
for use with both tests and classes such as unittest.skipUnless that
allow us to do what was being done in oescripts.py with

|try:
|    import cairo
|except ImportError:
|    raise unittest.SkipTest('Python module cairo is not present')

This mirrors a similar issue found in another project, where SkipTest
inside setUpClass was being treated as a class setup failure (see
https://github.com/CleanCut/green/issues/159). Instead of trying to
mimic their fix (which could possibly be done with a new decorator,
but in one of the OE selftest parent classes), it's simpler to add an
import of importlib, then decorate OEScriptTests with

|@unittest.skipUnless(importlib.util.find_spec("cairo"), "Python cairo module is not present")

and follow a similar procedure for any other classes that may need a
conditional import or three. This also happens to be much cleaner than
the original patch that I had proposed, again because it uses the
unittest framework's skipUnless method, rather than setting a class
variable that then gets checked by each of the individual test methods
in turn (which had the same functional result for this test, however).

What I haven't been able to determine is why the test logs include the
following for the non-parallel and parallel processes (-j option) runs,
respectively:

|rtype: [(<ErrorHolder description='setUpClass (oescripts.OEPybootchartguyTests)'>, 'Python module cairo is not present')]

|skip: setUpClass (oescripts.OEPybootchartguyTests) [                                                                   
|Python module cairo is not present                                                                                     
|]   

(see https://bugzilla.yoctoproject.org/show_bug.cgi?id=13663#c8). I
suspect that this is from the inconsistency in handling the raise
inside setUpClass between the two oe-selftest calls. However, I still
think that the issue is the way that the import check was handled
starting with commit 785bb108ce.
Comment 11 Trevor Gamblin 2019-12-05 18:19:42 UTC 
With raise Exception("broken") in setUpClass, cairo is available, using unittest.skipUnless decorator:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests
2019-12-05 12:18:26,800 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 12:18:26,801 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 12:18:26,801 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib                                                                                                                                                           
2019-12-05 12:18:26,801 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib                                                                                                                                                  
2019-12-05 12:18:26,803 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable                                                                                                                                 
2019-12-05 12:18:29,048 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build/conf/local.conf                                                                                                                         
2019-12-05 12:18:29,049 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-05 12:19:05,911 - oe-selftest - INFO - test_pybootchartguy_help (oescripts.OEPybootchartguyTests)
2019-12-05 12:19:05,975 - oe-selftest - INFO -  ... ok                                                                                                                                                                                        
2019-12-05 12:19:05,976 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_pdf_output (oescripts.OEPybootchartguyTests)
2019-12-05 12:19:06,055 - oe-selftest - INFO -  ... ok                                                                 
2019-12-05 12:19:06,056 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_png_output (oescripts.OEPybootchartguyTests)
2019-12-05 12:19:06,155 - oe-selftest - INFO -  ... ok
2019-12-05 12:19:06,156 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_svg_output (oescripts.OEPybootchartguyTests)
2019-12-05 12:19:06,237 - oe-selftest - INFO -  ... ok
2019-12-05 12:19:06,239 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 12:19:06,239 - oe-selftest - INFO - Ran 4 tests in 37.191s
2019-12-05 12:19:06,239 - oe-selftest - INFO - OK
2019-12-05 12:19:09,400 - oe-selftest - INFO - RESULTS:
2019-12-05 12:19:09,401 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED (0.06s)
2019-12-05 12:19:09,401 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED (0.08s)
2019-12-05 12:19:09,401 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED (0.10s)
2019-12-05 12:19:09,402 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED (0.08s)
2019-12-05 12:19:09,415 - oe-selftest - INFO - SUMMARY:
2019-12-05 12:19:09,415 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 37.192s
2019-12-05 12:19:09,415 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=4, skipped=0, failures=0, errors=0)

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2
2019-12-05 12:19:26,820 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 12:19:26,820 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 12:19:26,820 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 12:19:26,820 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 12:19:26,822 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-05 12:19:31,651 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build-st-1681787/conf/local.conf
2019-12-05 12:19:31,652 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-05 12:56:49,456 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_help (subunit.RemotedTestCase)
2019-12-05 12:56:49,473 - oe-selftest - INFO -  ... ok
2019-12-05 12:56:49,473 - oe-selftest - INFO - 0: 1/4 1/4 (0.15s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_help)
2019-12-05 12:56:50,504 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output (subunit.RemotedTestCase)
2019-12-05 12:56:50,504 - oe-selftest - INFO -  ... ok
2019-12-05 12:56:50,504 - oe-selftest - INFO - 0: 2/4 2/4 (1.05s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output)
2019-12-05 12:56:51,724 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output (subunit.RemotedTestCase)
2019-12-05 12:56:51,725 - oe-selftest - INFO -  ... ok
2019-12-05 12:56:51,725 - oe-selftest - INFO - 0: 3/4 3/4 (1.22s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output)
2019-12-05 12:56:52,270 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output (subunit.RemotedTestCase)
2019-12-05 12:56:52,270 - oe-selftest - INFO -  ... ok
2019-12-05 12:56:52,271 - oe-selftest - INFO - 0: 4/4 4/4 (0.55s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output)
2019-12-05 12:57:22,688 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 12:57:22,688 - oe-selftest - INFO - Ran 4 tests in 2273.477s
2019-12-05 12:57:22,689 - oe-selftest - INFO - OK
2019-12-05 12:57:26,095 - oe-selftest - INFO - RESULTS:
2019-12-05 12:57:26,096 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED (0.16s)
2019-12-05 12:57:26,096 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED (1.05s)
2019-12-05 12:57:26,096 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED (1.22s)
2019-12-05 12:57:26,097 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED (0.55s)
2019-12-05 12:57:26,110 - oe-selftest - INFO - SUMMARY:
2019-12-05 12:57:26,110 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 2273.629s
2019-12-05 12:57:26,110 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=4, skipped=0, failures=0, errors=0)
Comment 12 Trevor Gamblin 2019-12-05 19:03:04 UTC 
With raise Exception("broken") in setUpClass, cairo is available, without unittest.skipUnless decorator:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests
2019-12-05 13:20:25,666 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 13:20:25,666 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 13:20:25,666 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 13:20:25,666 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 13:20:25,667 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-05 13:20:27,901 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build/conf/local.conf
2019-12-05 13:20:27,901 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-05 13:21:05,074 - oe-selftest - INFO - test_pybootchartguy_help (oescripts.OEPybootchartguyTests)
2019-12-05 13:21:05,135 - oe-selftest - INFO -  ... ok
2019-12-05 13:21:05,135 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_pdf_output (oescripts.OEPybootchartguyTests)
2019-12-05 13:21:05,212 - oe-selftest - INFO -  ... ok
2019-12-05 13:21:05,213 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_png_output (oescripts.OEPybootchartguyTests)
2019-12-05 13:21:05,309 - oe-selftest - INFO -  ... ok
2019-12-05 13:21:05,310 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_svg_output (oescripts.OEPybootchartguyTests)
2019-12-05 13:21:05,389 - oe-selftest - INFO -  ... ok
2019-12-05 13:21:05,390 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 13:21:05,390 - oe-selftest - INFO - Ran 4 tests in 37.490s
2019-12-05 13:21:05,390 - oe-selftest - INFO - OK
2019-12-05 13:21:08,565 - oe-selftest - INFO - RESULTS:
2019-12-05 13:21:08,566 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED (0.06s)
2019-12-05 13:21:08,566 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED (0.08s)
2019-12-05 13:21:08,567 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED (0.10s)
2019-12-05 13:21:08,567 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED (0.08s)
2019-12-05 13:21:08,579 - oe-selftest - INFO - SUMMARY:
2019-12-05 13:21:08,579 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 37.491s
2019-12-05 13:21:08,579 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=4, skipped=0, failures=0, errors=0)

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2
2019-12-05 13:24:01,467 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 13:24:01,468 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 13:24:01,468 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 13:24:01,468 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 13:24:01,469 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-05 13:24:06,320 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build-st-3123066/conf/local.conf
2019-12-05 13:24:06,321 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-05 14:00:20,403 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_help (subunit.RemotedTestCase)
2019-12-05 14:00:20,404 - oe-selftest - INFO -  ... ok
2019-12-05 14:00:20,404 - oe-selftest - INFO - 0: 1/4 1/4 (0.07s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_help)
2019-12-05 14:00:20,997 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output (subunit.RemotedTestCase)
2019-12-05 14:00:20,997 - oe-selftest - INFO -  ... ok
2019-12-05 14:00:20,997 - oe-selftest - INFO - 0: 2/4 2/4 (0.59s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output)
2019-12-05 14:00:22,143 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output (subunit.RemotedTestCase)
2019-12-05 14:00:22,144 - oe-selftest - INFO -  ... ok
2019-12-05 14:00:22,144 - oe-selftest - INFO - 0: 3/4 3/4 (1.15s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output)
2019-12-05 14:00:22,685 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output (subunit.RemotedTestCase)
2019-12-05 14:00:22,685 - oe-selftest - INFO -  ... ok
2019-12-05 14:00:22,686 - oe-selftest - INFO - 0: 4/4 4/4 (0.54s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output)
2019-12-05 14:00:52,180 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 14:00:52,181 - oe-selftest - INFO - Ran 4 tests in 2208.282s
2019-12-05 14:00:52,181 - oe-selftest - INFO - OK
2019-12-05 14:00:55,610 - oe-selftest - INFO - RESULTS:
2019-12-05 14:00:55,611 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED (0.07s)
2019-12-05 14:00:55,612 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED (0.59s)
2019-12-05 14:00:55,612 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED (1.15s)
2019-12-05 14:00:55,612 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED (0.54s)
2019-12-05 14:00:55,623 - oe-selftest - INFO - SUMMARY:
2019-12-05 14:00:55,623 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 2208.468s
2019-12-05 14:00:55,623 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=4, skipped=0, failures=0, errors=0)
Comment 13 Trevor Gamblin 2019-12-05 19:19:40 UTC 
With raise Exception("broken") in setUpClass, cairo is not available, with unittest.skipUnless decorator:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests                                                                                                                                                   
2019-12-05 14:09:16,191 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 14:09:16,191 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 14:09:16,192 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib                                                                                                                                                           
2019-12-05 14:09:16,192 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib      
2019-12-05 14:09:16,193 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable                                                                                                                                 
2019-12-05 14:09:18,428 - oe-selftest - INFO - test_pybootchartguy_help (oescripts.OEPybootchartguyTests)
2019-12-05 14:09:18,429 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'
2019-12-05 14:09:18,429 - oe-selftest - INFO - Python cairo module is not present                                    
2019-12-05 14:09:18,429 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_pdf_output (oescripts.OEPybootchartguyTests)
2019-12-05 14:09:18,430 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'
2019-12-05 14:09:18,430 - oe-selftest - INFO - Python cairo module is not present
2019-12-05 14:09:18,430 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_png_output (oescripts.OEPybootchartguyTests)
2019-12-05 14:09:18,430 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'                                                                                                                                              
2019-12-05 14:09:18,430 - oe-selftest - INFO - Python cairo module is not present                                                                                                                                                             
2019-12-05 14:09:18,430 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_svg_output (oescripts.OEPybootchartguyTests)                         
2019-12-05 14:09:18,431 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'                                                                                                                                              
2019-12-05 14:09:18,431 - oe-selftest - INFO - Python cairo module is not present
2019-12-05 14:09:18,431 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 14:09:18,431 - oe-selftest - INFO - Ran 4 tests in 0.003s                                                                                                                                                                          
2019-12-05 14:09:18,432 - oe-selftest - INFO - OK                                                                                                                                                                                             
2019-12-05 14:09:18,432 - oe-selftest - INFO -  (skipped=4) 
2019-12-05 14:09:21,608 - oe-selftest - INFO - RESULTS:
2019-12-05 14:09:21,610 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED (0.00s)
2019-12-05 14:09:21,610 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED (0.00s)
2019-12-05 14:09:21,610 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED (0.00s)
2019-12-05 14:09:21,610 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED (0.00s)
2019-12-05 14:09:21,624 - oe-selftest - INFO - SUMMARY:
2019-12-05 14:09:21,624 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 0.004s
2019-12-05 14:09:21,624 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=4, failures=0, errors=0)


[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2
2019-12-05 14:09:54,849 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 14:09:54,849 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 14:09:54,849 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 14:09:54,850 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 14:09:54,851 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-05 14:09:59,659 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_help (subunit.RemotedTestCase)
2019-12-05 14:09:59,660 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 14:09:59,660 - oe-selftest - INFO - 0: 1/4 1/4 (0.00s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_help)
2019-12-05 14:09:59,660 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 14:09:59,661 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output (subunit.RemotedTestCase)
2019-12-05 14:09:59,661 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 14:09:59,661 - oe-selftest - INFO - 0: 2/4 2/4 (0.00s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output)
2019-12-05 14:09:59,661 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 14:09:59,662 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output (subunit.RemotedTestCase)
2019-12-05 14:09:59,662 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 14:09:59,662 - oe-selftest - INFO - 0: 3/4 3/4 (0.00s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output)
2019-12-05 14:09:59,662 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 14:09:59,663 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output (subunit.RemotedTestCase)
2019-12-05 14:09:59,663 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 14:09:59,663 - oe-selftest - INFO - 0: 4/4 4/4 (0.00s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output)
2019-12-05 14:09:59,663 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 14:09:59,689 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 14:09:59,689 - oe-selftest - INFO - Ran 4 tests in 2.451s
2019-12-05 14:09:59,690 - oe-selftest - INFO - OK
2019-12-05 14:09:59,690 - oe-selftest - INFO -  (skipped=4)
2019-12-05 14:10:03,015 - oe-selftest - INFO - RESULTS:
2019-12-05 14:10:03,016 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED (0.00s)
2019-12-05 14:10:03,017 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED (0.00s)
2019-12-05 14:10:03,017 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED (0.00s)
2019-12-05 14:10:03,017 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED (0.01s)
2019-12-05 14:10:03,029 - oe-selftest - INFO - SUMMARY:
2019-12-05 14:10:03,030 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 2.603s
2019-12-05 14:10:03,030 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=4, failures=0, errors=0)
Comment 14 Trevor Gamblin 2019-12-05 19:21:48 UTC 
With raise Exception("broken") in setUpClass, cairo is not available, without unittest.skipUnless decorator:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests
2019-12-05 14:18:42,080 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 14:18:42,080 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 14:18:42,080 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 14:18:42,081 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 14:18:42,082 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-05 14:18:44,322 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build/conf/local.conf
2019-12-05 14:18:44,323 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-05 14:18:44,324 - oe-selftest - INFO - ERROR
2019-12-05 14:18:44,325 - oe-selftest - INFO - ======================================================================
2019-12-05 14:18:44,325 - oe-selftest - INFO - ERROR: setUpClass (oescripts.OEPybootchartguyTests)
2019-12-05 14:18:44,325 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 14:18:44,325 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 44, in setUpClass
    import cairo
ModuleNotFoundError: No module named 'cairo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/core/case.py", line 39, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 46, in setUpClass
    raise Exception("broken")
Exception: broken

2019-12-05 14:18:44,326 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 14:18:44,326 - oe-selftest - INFO - Ran 0 tests in 0.004s
2019-12-05 14:18:44,326 - oe-selftest - INFO - FAILED
2019-12-05 14:18:44,326 - oe-selftest - INFO -  (errors=1)
2019-12-05 14:18:47,514 - oe-selftest - INFO - RESULTS:
2019-12-05 14:18:47,515 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: ERROR
2019-12-05 14:18:47,515 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: ERROR
2019-12-05 14:18:47,515 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: ERROR
2019-12-05 14:18:47,515 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: ERROR
2019-12-05 14:18:47,528 - oe-selftest - INFO - SUMMARY:
2019-12-05 14:18:47,529 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.006s
2019-12-05 14:18:47,529 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)


[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2
2019-12-05 14:20:42,916 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 14:20:42,916 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 14:20:42,916 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 14:20:42,916 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 14:20:42,917 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-05 14:20:47,797 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build-st-364659/conf/local.conf
2019-12-05 14:20:47,798 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-05 14:20:47,834 - oe-selftest - INFO - setUpClass (oescripts.OEPybootchartguyTests) [ (subunit.RemotedTestCase)
2019-12-05 14:20:47,835 - oe-selftest - INFO -  ... ERROR
2019-12-05 14:20:47,835 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 44, in setUpClass
    import cairo
ModuleNotFoundError: No module named 'cairo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/core/case.py", line 39, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 46, in setUpClass
    raise Exception("broken")
Exception: broken


2019-12-05 14:20:47,836 - oe-selftest - INFO - ======================================================================
2019-12-05 14:20:47,836 - oe-selftest - INFO - ERROR: setUpClass (oescripts.OEPybootchartguyTests) [ (subunit.RemotedTestCase)
2019-12-05 14:20:47,836 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 14:20:47,836 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 44, in setUpClass
    import cairo
ModuleNotFoundError: No module named 'cairo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/core/case.py", line 39, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 46, in setUpClass
    raise Exception("broken")
Exception: broken


2019-12-05 14:20:47,836 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 14:20:47,836 - oe-selftest - INFO - Ran 1 test in 2.509s
2019-12-05 14:20:47,837 - oe-selftest - INFO - FAILED
2019-12-05 14:20:47,837 - oe-selftest - INFO -  (errors=1)
2019-12-05 14:20:51,158 - oe-selftest - INFO - RESULTS:
2019-12-05 14:20:51,159 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: ERROR
2019-12-05 14:20:51,159 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: ERROR
2019-12-05 14:20:51,160 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: ERROR
2019-12-05 14:20:51,160 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: ERROR
2019-12-05 14:20:51,174 - oe-selftest - INFO - SUMMARY:
2019-12-05 14:20:51,174 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2.686s
2019-12-05 14:20:51,174 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)
Comment 15 Trevor Gamblin 2019-12-05 19:32:15 UTC 
Comments 11-14 were tested with the following change to OEScriptTests in 
meta/lib/oeqa/selftest/cases/oescripts.py (comment # removed or added as necessary for the tests. Same for python3-cairo):

37 #@unittest.skipUnless(importlib.util.find_spec("cairo"), "Python cairo module is not present")
38 class OEScriptTests(OESelftestTestCase):
39 
40     @classmethod
41     def setUpClass(cls):
42         super(OEScriptTests, cls).setUpClass()
43         try:
44             import cairo
45         except:
46             raise Exception("broken")
47         bitbake("core-image-minimal -c rootfs -f")
48         cls.tmpdir = get_bb_var('TMPDIR')
49         cls.buildstats = cls.tmpdir + "/buildstats/" + sorted(os.listdir(cls.tmpdir + "/buildstats"))[-1]
50 
51     scripts_dir = os.path.join(get_bb_var('COREBASE'), 'scripts')
Comment 16 Richard Purdie 2019-12-05 21:20:32 UTC 
Trevor, I think you misunderstand me.

I was asking what happens if we put a general:

raise Exception("broken")

in setUpClass(), ignoring anything to do with cairo, i.e. make it always fail.

What I want to ensure is that exceptions in class setup are correctly being mapped back to the tests that should fail/skip.
Comment 17 Trevor Gamblin 2019-12-05 21:43:52 UTC 
Ah, my bad. With that change, it counts a single error (for the class setup), but displays ERROR for each of the tests. This seems correct to me, but is it what you're expecting, or do you think it should report four errors?

No decorator:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests                                                                                                                                                   
2019-12-05 16:30:36,854 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build                                                                                                                                                           
2019-12-05 16:30:36,855 - oe-selftest - INFO - Adding layer libraries:                                                                                                                                                                        
2019-12-05 16:30:36,855 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib                                                                                                                                                           
2019-12-05 16:30:36,855 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib                                                                                                                                                  
2019-12-05 16:30:36,857 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable                                                                                                                                 
2019-12-05 16:30:39,084 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build/conf/local.conf                                                                                                                         
2019-12-05 16:30:39,085 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf                                                                                                                                                
2019-12-05 16:30:39,086 - oe-selftest - INFO - ERROR   
2019-12-05 16:30:39,086 - oe-selftest - INFO - ======================================================================
2019-12-05 16:30:39,086 - oe-selftest - INFO - ERROR: setUpClass (oescripts.OEPybootchartguyTests)                                                                                                                                            
2019-12-05 16:30:39,087 - oe-selftest - INFO - ----------------------------------------------------------------------  
2019-12-05 16:30:39,087 - oe-selftest - INFO - Traceback (most recent call last):                                      
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/core/case.py", line 39, in _oeSetUpClass                              
    clss.setUpClassMethod()                                                                                            
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 43, in setUpClass
    raise Exception("broken")                                                                                          
Exception: broken                                                                                                                                                                                                                             
                                                                                                                       
2019-12-05 16:30:39,087 - oe-selftest - INFO - ----------------------------------------------------------------------                                                                                                                         
2019-12-05 16:30:39,087 - oe-selftest - INFO - Ran 0 tests in 0.003s                                                                                                                                                                          
2019-12-05 16:30:39,087 - oe-selftest - INFO - FAILED                                                                                                                                                                                         
2019-12-05 16:30:39,088 - oe-selftest - INFO -  (errors=1)                                                                                                                                                                                    
2019-12-05 16:30:42,269 - oe-selftest - INFO - RESULTS:                                                                
2019-12-05 16:30:42,270 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: ERROR                                                                                                                      
2019-12-05 16:30:42,270 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: ERROR
2019-12-05 16:30:42,270 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: ERROR
2019-12-05 16:30:42,271 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: ERROR                                                                                              
2019-12-05 16:30:42,284 - oe-selftest - INFO - SUMMARY:                                                                
2019-12-05 16:30:42,285 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.005s                                                                                                                                                         
2019-12-05 16:30:42,285 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)


[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2                                                                                                                                              
2019-12-05 16:30:58,054 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build                                                                                                                                                           
2019-12-05 16:30:58,055 - oe-selftest - INFO - Adding layer libraries:                                   
2019-12-05 16:30:58,055 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib                                                                                                                                                           
2019-12-05 16:30:58,055 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 16:30:58,056 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable                                                                                                                                 
2019-12-05 16:31:02,924 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/tgamblin/build-st-365111/conf/local.conf
2019-12-05 16:31:02,925 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-12-05 16:31:02,958 - oe-selftest - INFO - setUpClass (oescripts.OEPybootchartguyTests) [ (subunit.RemotedTestCase)          
2019-12-05 16:31:02,958 - oe-selftest - INFO -  ... ERROR                                                              
2019-12-05 16:31:02,959 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/core/case.py", line 39, in _oeSetUpClass                                                                                                                                                     
    clss.setUpClassMethod()
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 43, in setUpClass
    raise Exception("broken")
Exception: broken


2019-12-05 16:31:02,959 - oe-selftest - INFO - ======================================================================
2019-12-05 16:31:02,959 - oe-selftest - INFO - ERROR: setUpClass (oescripts.OEPybootchartguyTests) [ (subunit.RemotedTestCase)
2019-12-05 16:31:02,960 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 16:31:02,960 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/core/case.py", line 39, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/tgamblin/oe-core.git/meta/lib/oeqa/selftest/cases/oescripts.py", line 43, in setUpClass
    raise Exception("broken")
Exception: broken


2019-12-05 16:31:02,960 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 16:31:02,960 - oe-selftest - INFO - Ran 1 test in 2.491s
2019-12-05 16:31:02,960 - oe-selftest - INFO - FAILED
2019-12-05 16:31:02,960 - oe-selftest - INFO -  (errors=1)
2019-12-05 16:31:06,302 - oe-selftest - INFO - RESULTS:
2019-12-05 16:31:06,303 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: ERROR
2019-12-05 16:31:06,303 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: ERROR
2019-12-05 16:31:06,304 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: ERROR
2019-12-05 16:31:06,304 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: ERROR
2019-12-05 16:31:06,317 - oe-selftest - INFO - SUMMARY:
2019-12-05 16:31:06,317 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2.645s
2019-12-05 16:31:06,318 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)

With decorator:

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests                                                                                                                                                   
2019-12-05 16:31:30,161 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build               
2019-12-05 16:31:30,162 - oe-selftest - INFO - Adding layer libraries:                                                                                                                                                                        
2019-12-05 16:31:30,162 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 16:31:30,162 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 16:31:30,163 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable                                                                                                                                 
2019-12-05 16:31:32,390 - oe-selftest - INFO - test_pybootchartguy_help (oescripts.OEPybootchartguyTests)
2019-12-05 16:31:32,390 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'                                                                                                                                              
2019-12-05 16:31:32,391 - oe-selftest - INFO - Python cairo module is not present
2019-12-05 16:31:32,391 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_pdf_output (oescripts.OEPybootchartguyTests)
2019-12-05 16:31:32,391 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'                     
2019-12-05 16:31:32,391 - oe-selftest - INFO - Python cairo module is not present
2019-12-05 16:31:32,391 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_png_output (oescripts.OEPybootchartguyTests)
2019-12-05 16:31:32,391 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'
2019-12-05 16:31:32,392 - oe-selftest - INFO - Python cairo module is not present
2019-12-05 16:31:32,392 - oe-selftest - INFO - test_pybootchartguy_to_generate_build_svg_output (oescripts.OEPybootchartguyTests) 
2019-12-05 16:31:32,392 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present'                                                                                                                                              
2019-12-05 16:31:32,392 - oe-selftest - INFO - Python cairo module is not present                                                                                                                                                             
2019-12-05 16:31:32,392 - oe-selftest - INFO - ----------------------------------------------------------------------                                                                                                                         
2019-12-05 16:31:32,393 - oe-selftest - INFO - Ran 4 tests in 0.003s
2019-12-05 16:31:32,393 - oe-selftest - INFO - OK                                                                      
2019-12-05 16:31:32,393 - oe-selftest - INFO -  (skipped=4)                                                                                                                                                                                   
2019-12-05 16:31:35,594 - oe-selftest - INFO - RESULTS:
2019-12-05 16:31:35,595 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED (0.00s)
2019-12-05 16:31:35,595 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED (0.00s)
2019-12-05 16:31:35,596 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED (0.00s)
2019-12-05 16:31:35,596 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED (0.00s)
2019-12-05 16:31:35,610 - oe-selftest - INFO - SUMMARY:
2019-12-05 16:31:35,610 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 0.004s
2019-12-05 16:31:35,610 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=4, failures=0, errors=0)

[tgamblin@yow-tgamblin-fedora1 oe-core.git]$ oe-selftest -r oescripts.OEPybootchartguyTests -j 2
2019-12-05 16:31:47,705 - oe-selftest - INFO - Changing cwd to /home/tgamblin/build
2019-12-05 16:31:47,705 - oe-selftest - INFO - Adding layer libraries:
2019-12-05 16:31:47,706 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta/lib
2019-12-05 16:31:47,706 - oe-selftest - INFO -  /home/tgamblin/oe-core.git/meta-selftest/lib
2019-12-05 16:31:47,707 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-12-05 16:31:52,512 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_help (subunit.RemotedTestCase)
2019-12-05 16:31:52,514 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 16:31:52,514 - oe-selftest - INFO - 0: 1/4 1/4 (0.00s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_help)
2019-12-05 16:31:52,514 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 16:31:52,515 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output (subunit.RemotedTestCase)
2019-12-05 16:31:52,515 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 16:31:52,515 - oe-selftest - INFO - 0: 2/4 2/4 (0.00s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output)
2019-12-05 16:31:52,516 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 16:31:52,516 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output (subunit.RemotedTestCase)
2019-12-05 16:31:52,516 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 16:31:52,517 - oe-selftest - INFO - 0: 3/4 3/4 (0.00s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output)
2019-12-05 16:31:52,517 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 16:31:52,517 - oe-selftest - INFO - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output (subunit.RemotedTestCase)
2019-12-05 16:31:52,518 - oe-selftest - INFO -  ... skipped 'Python cairo module is not present\n'
2019-12-05 16:31:52,518 - oe-selftest - INFO - 0: 4/4 4/4 (0.01s) (oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output)
2019-12-05 16:31:52,518 - oe-selftest - INFO - Python cairo module is not present

2019-12-05 16:31:52,549 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-05 16:31:52,550 - oe-selftest - INFO - Ran 4 tests in 2.453s
2019-12-05 16:31:52,550 - oe-selftest - INFO - OK
2019-12-05 16:31:52,550 - oe-selftest - INFO -  (skipped=4) 
2019-12-05 16:31:55,725 - oe-selftest - INFO - RESULTS:
2019-12-05 16:31:55,726 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_help: SKIPPED (0.00s)
2019-12-05 16:31:55,727 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: SKIPPED (0.00s)
2019-12-05 16:31:55,727 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: SKIPPED (0.01s)
2019-12-05 16:31:55,727 - oe-selftest - INFO - RESULTS - oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: SKIPPED (0.01s)
2019-12-05 16:31:55,741 - oe-selftest - INFO - SUMMARY:
2019-12-05 16:31:55,741 - oe-selftest - INFO - oe-selftest () - Ran 4 tests in 2.605s
2019-12-05 16:31:55,741 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=0, skipped=4, failures=0, errors=0)
Comment 18 Trevor Gamblin 2020-02-18 15:03:27 UTC 
Patch submitted for review.
Comment 19 Trevor Gamblin 2020-02-20 14:24:55 UTC 
There are two patches in place to fix this. This one fixes the concurrencytest.py behavior where subunit couldn't effectively handle a "skip" happening outside of an individual test (i.e. it occurred inside setUpClass()):

https://git.openembedded.org/openembedded-core/commit/?id=9b8734b584d6e8d9c32ff2a721b29f3f3e61cca7

This patch adds some extra handling for OEScriptTests, so that it knows to skip the entire class of tests if python3-cairo is not present on the host:

https://git.openembedded.org/openembedded-core/commit/?id=10c2216cb6fb61c74babd8bd17cb3481754b7475