Bug 11756

Summary:	oe-selftest: --run-tests ignores test order
Product:	[QA/Testing] Build Testing	Reporter:	Patrick Ohly <patrick.ohly>
Component:	general	Assignee:	Sangeeta Jain <sangeeta.jain>
Status:	RESOLVED OBSOLETE	QA Contact:
Severity:	normal
Priority:	Medium	CC:	jair.de.jesus.gonzalez.plascencia, juan.m.cruz.alcaraz, sangeeta.jain
Version:	2.4
Target Milestone:	4.99
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Patrick Ohly 2017-07-05 06:45:32 UTC

Sometimes it may be useful to run tests in a certain order, for example fast ones first to abort early when there are failures.

This used to work, but with the current runner tests seem to run in the order in which they are discovered, not in the order in which they are given on the command line.

Example in refkit:

$ oe-selftest --list-tests | grep compat
...
2017-07-05 08:43:50,339 - oe-selftest - INFO - None	None		refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit
2017-07-05 08:43:50,339 - oe-selftest - INFO - None	None		refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit_computervision
2017-07-05 08:43:50,339 - oe-selftest - INFO - None	None		refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit_core
2017-07-05 08:43:50,339 - oe-selftest - INFO - None	None		refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit_gateway
2017-07-05 08:43:50,339 - oe-selftest - INFO - None	None		refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit_industrial

$ oe-selftest -r refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit_computervision refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit
...
2017-07-05 08:31:03,354 - oe-selftest - INFO -   test_compat_meta_refkit (refkit_poky.TestRefkitPokySignatures) ... OK (100.494s)
...
2017-07-05 08:32:06,405 - oe-selftest - INFO -   test_compat_meta_refkit_computervision (refkit_poky.TestRefkitPokySignatures) ... OK (63.051s)
2017-07-05 08:32:06,406 - oe-selftest - INFO - 
2017-07-05 08:32:06,406 - oe-selftest - INFO - ----------------------------------------------------------------------
2017-07-05 08:32:06,406 - oe-selftest - INFO - Ran 2 tests in 166.189s
2017-07-05 08:32:06,406 - oe-selftest - INFO - 
2017-07-05 08:32:06,406 - oe-selftest - INFO - OK
2017-07-05 08:32:06,407 - oe-selftest - INFO - 
2017-07-05 08:32:06,407 - oe-selftest - INFO - Generating XML reports...
2017-07-05 08:32:06,416 - oe-selftest - INFO - RESULTS:
2017-07-05 08:32:06,416 - oe-selftest - INFO - RESULTS - refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit - Testcase -1: PASSED
2017-07-05 08:32:06,416 - oe-selftest - INFO - RESULTS - refkit_poky.TestRefkitPokySignatures.test_compat_meta_refkit_computervision - Testcase -1
...

Comment 1 Patrick Ohly 2017-07-10 12:57:26 UTC

Probably related and might be fixed at the same time, so let me just add it here: invoking "oe-selftest -r foobar" success instead of failing with an error about "foobar" not being found.

$ oe-selftest -r foobar
2017-07-10 14:56:47,643 - oe-selftest - INFO - Adding layer libraries:
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/meta-intel-iot-security/meta-security-smack/lib
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/meta-intel-iot-security/meta-integrity/lib
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/meta-security-tools/lib
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/intel-iot-refkit//meta-refkit/lib
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/intel-iot-refkit/openembedded-core/meta/lib
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/intel-iot-refkit/openembedded-core/meta-selftest/lib
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/intel-iot-refkit//meta-refkit-core/lib
2017-07-10 14:56:47,643 - oe-selftest - INFO - 	/fast/work/intel-iot-refkit//meta-iotqa/lib
2017-07-10 14:56:47,644 - oe-selftest - INFO - 	/fast/work/intel-iot-refkit/meta-security-isafw/lib
2017-07-10 14:56:47,644 - oe-selftest - INFO - 	/fast/work/meta-swupd/lib
2017-07-10 14:56:47,644 - oe-selftest - INFO - 	/fast/work/meta-intel-iot-security/meta-security-framework/lib
2017-07-10 14:56:47,648 - oe-selftest - INFO - Running: bitbake -p
2017-07-10 14:56:49,818 - oe-selftest - INFO - Loading cache...done.
2017-07-10 14:56:51,009 - oe-selftest - INFO - Loaded 3538 entries from dependency cache.
2017-07-10 14:56:52,109 - oe-selftest - INFO - Parsing recipes...done.
2017-07-10 14:56:52,518 - oe-selftest - INFO - Parsing of 2626 .bb files complete (2622 cached, 4 parsed). 3542 targets, 419 skipped, 0 masked, 0 errors.
2017-07-10 14:56:52,623 - oe-selftest - INFO - 
2017-07-10 14:56:52,623 - oe-selftest - INFO - ----------------------------------------------------------------------
2017-07-10 14:56:52,623 - oe-selftest - INFO - Ran 0 tests in 0.001s
2017-07-10 14:56:52,623 - oe-selftest - INFO - 
2017-07-10 14:56:52,624 - oe-selftest - INFO - OK
2017-07-10 14:56:52,624 - oe-selftest - INFO - RESULTS:
2017-07-10 14:56:52,624 - oe-selftest - INFO - SUMMARY:
2017-07-10 14:56:52,624 - oe-selftest - INFO - oe-selftest () - Ran 0 tests in 0.002s
2017-07-10 14:56:52,624 - oe-selftest - INFO - oe-selftest - OK - All required tests passed

Comment 2 Aníbal Limón 2017-07-10 15:08:53 UTC

(In reply to comment #1)
> Probably related and might be fixed at the same time, so let me just add it
> here: invoking "oe-selftest -r foobar" success instead of failing with an
> error about "foobar" not being found.
> 
> $ oe-selftest -r foobar
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - Adding layer libraries:
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/meta-intel-iot-security/meta-security-smack/lib
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/meta-intel-iot-security/meta-integrity/lib
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/meta-security-tools/lib
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/intel-iot-refkit//meta-refkit/lib
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/intel-iot-refkit/openembedded-core/meta/lib
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/intel-iot-refkit/openembedded-core/meta-selftest/lib
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/intel-iot-refkit//meta-refkit-core/lib
> 2017-07-10 14:56:47,643 - oe-selftest - INFO - 
> /fast/work/intel-iot-refkit//meta-iotqa/lib
> 2017-07-10 14:56:47,644 - oe-selftest - INFO - 
> /fast/work/intel-iot-refkit/meta-security-isafw/lib
> 2017-07-10 14:56:47,644 - oe-selftest - INFO - 	/fast/work/meta-swupd/lib
> 2017-07-10 14:56:47,644 - oe-selftest - INFO - 
> /fast/work/meta-intel-iot-security/meta-security-framework/lib
> 2017-07-10 14:56:47,648 - oe-selftest - INFO - Running: bitbake -p
> 2017-07-10 14:56:49,818 - oe-selftest - INFO - Loading cache...done.
> 2017-07-10 14:56:51,009 - oe-selftest - INFO - Loaded 3538 entries from
> dependency cache.
> 2017-07-10 14:56:52,109 - oe-selftest - INFO - Parsing recipes...done.
> 2017-07-10 14:56:52,518 - oe-selftest - INFO - Parsing of 2626 .bb files
> complete (2622 cached, 4 parsed). 3542 targets, 419 skipped, 0 masked, 0
> errors.
> 2017-07-10 14:56:52,623 - oe-selftest - INFO - 
> 2017-07-10 14:56:52,623 - oe-selftest - INFO -
> ----------------------------------------------------------------------
> 2017-07-10 14:56:52,623 - oe-selftest - INFO - Ran 0 tests in 0.001s
> 2017-07-10 14:56:52,623 - oe-selftest - INFO - 
> 2017-07-10 14:56:52,624 - oe-selftest - INFO - OK
> 2017-07-10 14:56:52,624 - oe-selftest - INFO - RESULTS:
> 2017-07-10 14:56:52,624 - oe-selftest - INFO - SUMMARY:
> 2017-07-10 14:56:52,624 - oe-selftest - INFO - oe-selftest () - Ran 0 tests
> in 0.002s
> 2017-07-10 14:56:52,624 - oe-selftest - INFO - oe-selftest - OK - All
> required tests passed

Hi Patrick,

We have a separate bug for this, see,

https://bugzilla.yoctoproject.org/show_bug.cgi?id=11645

Anibal

Comment 3 Jair Gonzalez 2017-09-14 21:16:12 UTC

The change is not trivial, as the current OEQA framework extends the upstream unittest library which uses recursion to discover the test cases, and sorts them on alphabetical order by default. It is possible to change the default unittest behavior and re-order the test cases afterwards on the way specified on the command line, but some other aspects need to be considered. For example, the framework and the oe-selftest tool allow to specify modules, classes or test cases, or any combination of them. Also, with the inclusion of dependency decorators, the order of test cases is re-arranged according to their dependencies. Finally, the framework also allows the possibility to specify a module manifest file to list the tests to execute, so it may be necessary to re-order the tests in a similar way as specified in the file if that option is used, to have a consistent behavior.
Considering we will be changing the default unittest behavior, it also may be convenient to have a new option in the tool to allow to specify the required order.
According to this, and this item's priority, I'll modify its target milestone to 2.5 M1.

Comment 4 Sangeeta Jain 2019-10-03 07:41:56 UTC

With current testcases in oeqa/runtime, the order in which test cases are run doesn't really matter.