| Summary: | TEST_SUITES="auto" is not specified for all images regularly built by the autobuilders | ||
|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Benjamin Esquivel <benjamin.esquivel> |
| Component: | general | Assignee: | Alex Franco <alejandro.franco> |
| Status: | RESOLVED FIXED | QA Contact: | Cristina Agurida <cristina-danielax.agurida> |
| Severity: | enhancement | ||
| Priority: | Medium+ | CC: | alexandru.c.georgescu, richard.purdie |
| Version: | unspecified | ||
| Target Milestone: | 2.1 M3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
| Bug Depends on: | |||
| Bug Blocks: | 9129 | ||
|
Description
Benjamin Esquivel
2015-09-28 17:11:18 UTC
(In reply to comment #0) > core-image-minimal-dev is one of the cases. > > And the real problem is that the non--specified targets take one test set > that assumes ssh which is not true for especially the *minimal*-based. > > One of the changes could be to make unspecified targets to be based off of > core-image-minimal (just ping test) (RP suggestion) This seems more like an Autobuilder request Hello Alex, I filed this under automated runtime testing and not autobuilder because I saw this while testing manually. I mentioned autobuilders in the subject just to reduce the number of targets that should receive the benefit of having an automatic suite specified. Right now, doing this manually: $ bitbake core-image-minimal-dev -c testimage leads to a failure because it tries to execute ssh and other tests that fail. There are several components to this bug: 1.- The existence of DEFAULT_TEST_SUITES for specific images. I am currently implementing this, and I am adding default test suites for: There are several components to this bug:
1.- The existence of DEFAULT_TEST_SUITES for specific images.
Solution: I am currently implementing this, and I am adding default test suites for:
- core-image-minimal-dev
- core-image-full-cmdline
- core-image-x11
- core-image-lsb
2.- The usage of "auto" in TEST_SUITES.
When "auto" is listed in TEST_SUITES, a function ran after do_testimage scans the directories '<layer>/lib/oeqa/runtime/', any tests found are added to the test run list, in whatever order they are found.
3.- How tests are skipped
Once the tests in TEST_SUITES and the ones scanned if "auto" was present are in the run test list, they are ran one at a time, in order of appearance. First all the ones in TEST_SUITES in the order they were added there, then the ones scanned if "auto" was present, in order of scan.
If any of these tests should be skipped, the test will issue its own skip either using unittest.skipTest or any of the skip decorators specified in utils/decorators.py, like skipUnlessPassed. These decorators only work if the test list is ordered by dependencies, which is not the case sometimes, and results in errors.
Proposed solutions for (2) and (3):
Improving (2) in order to have an ordered test list should guarantee (3) works as designed.
An alternative, perhaps ill-advised or incomplete solution, is skipping whole TestCases in (3) rather than individual tests. This has the downside that tests will be skip either if their dependencies have failed, been skipped OR simply haven't ran yet - this might be undesirable as it could hide errors and make debugging more difficult.
Either way, my goal for this bug is implementing a solution to (1), as solving (2) and (3) is probably scope for a new bug!
* Sent patches that address issue #1 previously mentioned. The series contains two patches plus the cover letter. [1] * A branch was created on poky-contrib to test this. [2] * Currently being tested in an internal autobuilder [1] http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/76451 [2] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=franco%2Ftestsuites My current conclusions for this bug: * "auto" should be used when the intention is scanning for all test suites * default test suites should be enough to cover basic image testing * improving the scanning and execution flow of test suites is a matter for a new bug, as it may be extensive work due to requiring a new model of test suite discovery, queuing, and execution (In reply to comment #6) > My current conclusions for this bug: > > * "auto" should be used when the intention is scanning for all test suites > * default test suites should be enough to cover basic image testing > * improving the scanning and execution flow of test suites is a matter for a > new bug, as it may be extensive work due to requiring a new model of test > suite discovery, queuing, and execution Added new bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=9129 for this work New dev branch is http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=franco%2Ftestsuites_2 Patches sent to ml |