Bug 11632 - oe-selftest doesn't run all tests
Summary: oe-selftest doesn't run all tests
Status: RESOLVED FIXED
Alias: None
Product: Runtime Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 2.2
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.4 M2
Assignee: Aníbal Limón
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-06-08 10:50 UTC by Ross Burton
Modified: 2017-06-12 15:53 UTC (History)
1 user (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
oe-selftest multiple runs (6.67 KB, text/plain)
2017-06-09 15:09 UTC, Aníbal Limón 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Ross Burton 2017-06-08 10:50:56 UTC 
$ oe-selftest  -r oelib
2017-06-08 11:44:47,713 - oe-selftest - INFO - Adding layer libraries:
2017-06-08 11:44:47,713 - oe-selftest - INFO - 	/home/ross/Yocto/poky/meta/lib
2017-06-08 11:44:47,713 - oe-selftest - INFO - 	/home/ross/Yocto/poky/meta-yocto-bsp/lib
2017-06-08 11:44:47,713 - oe-selftest - INFO - 	/home/ross/Yocto/poky/meta-selftest/lib
2017-06-08 11:44:47,715 - oe-selftest - INFO - Running bitbake -p
2017-06-08 11:44:50,408 - oe-selftest - INFO -
2017-06-08 11:44:50,409 - oe-selftest - INFO - ----------------------------------------------------------------------
2017-06-08 11:44:50,409 - oe-selftest - INFO - Ran 0 tests in 0.001s
2017-06-08 11:44:50,409 - oe-selftest - INFO -
2017-06-08 11:44:50,409 - oe-selftest - INFO - OK
2017-06-08 11:44:50,409 - oe-selftest - INFO - RESULTS:
2017-06-08 11:44:50,410 - oe-selftest - INFO - SUMMARY:
2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.002s
2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest - OK - All required tests passed

oe-selftest -l says that there are ~20 tests in that module.
Comment 1 Aníbal Limón 2017-06-08 15:48:12 UTC 
(In reply to comment #0)
> $ oe-selftest  -r oelib
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - Adding layer libraries:
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - 
> /home/ross/Yocto/poky/meta/lib
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - 
> /home/ross/Yocto/poky/meta-yocto-bsp/lib
> 2017-06-08 11:44:47,713 - oe-selftest - INFO - 
> /home/ross/Yocto/poky/meta-selftest/lib
> 2017-06-08 11:44:47,715 - oe-selftest - INFO - Running bitbake -p
> 2017-06-08 11:44:50,408 - oe-selftest - INFO -
> 2017-06-08 11:44:50,409 - oe-selftest - INFO -
> ----------------------------------------------------------------------
> 2017-06-08 11:44:50,409 - oe-selftest - INFO - Ran 0 tests in 0.001s
> 2017-06-08 11:44:50,409 - oe-selftest - INFO -
> 2017-06-08 11:44:50,409 - oe-selftest - INFO - OK
> 2017-06-08 11:44:50,409 - oe-selftest - INFO - RESULTS:
> 2017-06-08 11:44:50,410 - oe-selftest - INFO - SUMMARY:
> 2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest () - Ran 0 tests
> in 0.002s
> 2017-06-08 11:44:50,410 - oe-selftest - INFO - oe-selftest - OK - All
> required tests passed
> 
> oe-selftest -l says that there are ~20 tests in that module.

The problem here is the current logic only works for:

<module>.[class].[test_name]

And the module names only accepts a single one, for example:

bbtests.BitbakeTests.test_bbappend_order

Versus:

oelib.buildhistory.TestBlobParsing.test_compare_dict_blobs

I will modify the filtering to allow [1] module names with more than one sub package,

Cheers,
Anibal

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/core/loader.py#n168
Comment 3 Mikko Ylinen 2017-06-09 13:42:05 UTC 
Does that help the problem where with "oe-selftest -r module1 module2 module3" only module1 gets run?
Comment 4 Aníbal Limón 2017-06-09 15:09:53 UTC 
Created attachment 3808 [details]
oe-selftest multiple runs
Comment 5 Aníbal Limón 2017-06-09 15:11:06 UTC 
(In reply to comment #3)
> Does that help the problem where with "oe-selftest -r module1 module2
> module3" only module1 gets run?

That is expected to work, i ran [1] and works:

$ oe-selftest -r oelib lic_checksum

[1] https://bugzilla.yoctoproject.org/attachment.cgi?id=3808
Comment 6 Mikko Ylinen 2017-06-09 19:39:52 UTC 
(In reply to comment #5)
> (In reply to comment #3)
> > Does that help the problem where with "oe-selftest -r module1 module2
> > module3" only module1 gets run?
> 
> That is expected to work, i ran [1] and works:
> 

Yes, looks like I had another s/-/_/ error. All good.