Bug 11632

Summary: oe-selftest doesn't run all tests
Product: [QA/Testing] Runtime Testing Reporter: Ross Burton <ross.burton>
Component: generalAssignee: Aníbal Limón <anibal.limon>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: mikko.ylinen
Version: 2.2   
Target Milestone: 2.4 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
oe-selftest multiple runs none

Description Ross Burton 2017-06-08 10:50:56 UTC 
$ oe-selftest  -r oelib
2017-06-08 11:44:47,713 - oe-selftest - INFO - Adding layer libraries:
2017-06-08 11:44:47,713 - oe-selftest - INFO - 	/home/ross/Yocto/poky/meta/lib
2017-06-08 11:44:47,713 - oe-selftest - INFO - 	/home/ross/Yocto/poky/meta-yocto-bsp/lib
2017-06-08 11:44:47,713 - oe-selftest - INFO - 	/home/ross/Yocto/poky/meta-selftest/lib
2017-06-08 11:44:47,715 - oe-selftest - INFO - Running bitbake -p
2017-06-08 11:44:50,408 - oe-selftest - INFO -
2017-06-08 11:44:50,409 - oe-selftest - INFO - ----------------------------------------------------------------------
2017-06-08 11:44:50,409 - oe-selftest - INFO - Ran 0 tests in 0.001s
2017-06-08 11:44:50,409 - oe-selftest - INFO -
2017-06-08 11:44:50,409 - oe-selftest - INFO - OK
2017-06-08 11:44:50,409 - oe-selftest - INFO - RESULTS:
2017-06-08 11:44:50,410 - oe-selftest - INFO - SUMMARY:
2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.002s
2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest - OK - All required tests passed

oe-selftest -l says that there are ~20 tests in that module.
Comment 1 Aníbal Limón 2017-06-08 15:48:12 UTC 
(In reply to comment #0)
> $ oe-selftest  -r oelib
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - Adding layer libraries:
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - 
> /home/ross/Yocto/poky/meta/lib
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - 
> /home/ross/Yocto/poky/meta-yocto-bsp/lib
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - 
> /home/ross/Yocto/poky/meta-selftest/lib
> 2017-06-08 11:44:47,715 - oe-selftest - INFO - Running bitbake -p
> 2017-06-08 11:44:50,408 - oe-selftest - INFO -
> 2017-06-08 11:44:50,409 - oe-selftest - INFO -
> ----------------------------------------------------------------------
> 2017-06-08 11:44:50,409 - oe-selftest - INFO - Ran 0 tests in 0.001s
> 2017-06-08 11:44:50,409 - oe-selftest - INFO -
> 2017-06-08 11:44:50,409 - oe-selftest - INFO - OK
> 2017-06-08 11:44:50,409 - oe-selftest - INFO - RESULTS:
> 2017-06-08 11:44:50,410 - oe-selftest - INFO - SUMMARY:
> 2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest () - Ran 0 tests
> in 0.002s
> 2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest - OK - All
> required tests passed
> 
> oe-selftest -l says that there are ~20 tests in that module.

The problem here is the current logic only works for:

<module>.[class].[test_name]

And the module names only accepts a single one, for example:

bbtests.BitbakeTests.test_bbappend_order

Versus:

oelib.buildhistory.TestBlobParsing.test_compare_dict_blobs

I will modify the filtering to allow [1] module names with more than one sub package,

Cheers,
Anibal

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/core/loader.py#n168
Comment 3 Mikko Ylinen 2017-06-09 13:42:05 UTC 
Does that help the problem where with "oe-selftest -r module1 module2 module3" only module1 gets run?
Comment 4 Aníbal Limón 2017-06-09 15:09:53 UTC 
Created attachment 3808 [details]
oe-selftest multiple runs
Comment 5 Aníbal Limón 2017-06-09 15:11:06 UTC 
(In reply to comment #3)
> Does that help the problem where with "oe-selftest -r module1 module2
> module3" only module1 gets run?

That is expected to work, i ran [1] and works:

$ oe-selftest -r oelib lic_checksum

[1] https://bugzilla.yoctoproject.org/attachment.cgi?id=3808
Comment 6 Mikko Ylinen 2017-06-09 19:39:52 UTC 
(In reply to comment #5)
> (In reply to comment #3)
> > Does that help the problem where with "oe-selftest -r module1 module2
> > module3" only module1 gets run?
> 
> That is expected to work, i ran [1] and works:
> 

Yes, looks like I had another s/-/_/ error. All good.