Bug 9756

Summary:	Plan clean up and improvements for oeqa
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Joshua Lock <joshuagloe>
Component:	core	Assignee:	Aníbal Limón <anibal.limon>
Status:	RESOLVED FIXED	QA Contact:
Severity:	enhancement
Priority:	Medium+	CC:	anibal.limon, benjamin.esquivel, bluelightning, leonardo.sandoval.gonzalez, meta.mr.watcher, meta.watcher
Version:	2.2
Target Milestone:	2.2 M3
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Joshua Lock 2016-06-10 16:10:40 UTC

oeqa has several deficiencies which make it difficult to work with and maintain, including:
* dependencies are scattered around the code base
* test-loading only supports one folder deep
* test dependency order is non-deterministic and doesn't work well
* module relations/dependencies are unclear

We'd like to move to a new design which extends the python unittest module and better supports test dependencies and ordering and provides a clear high-level API addressing the concerns above.

In order to do that we first need to perform a review of the oeqa codebase and plan the work involved in moving forward with the new design.

This task will be complete when a new set of tickets are filed covering the tasks involved in improving oeqa.

Comment 1 Aníbal Limón 2016-07-06 21:58:08 UTC

This plan needs to take into account XML output see,

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9682

Comment 2 Benjamin Esquivel 2016-07-12 20:24:29 UTC

Re-assigning to Edwin and moving to M3 as this will not make it for 2.2 M2.

Comment 3 Aníbal Limón 2016-07-14 20:46:17 UTC

I finish the first draft of the plan, some tasks related to migration needs to define a set of step for do it but the other depend tasks needs to be complete for provide details of migration, i will load the bugs for cover the next tasks:

Runner:

- Implement OETestRunner with the ability for output logs in JUnit XML and
RAW text based on configuration parameter.
- Implement OEQemuTestRunner extending OETestRunner, this will start, stop
qemu with certain configuration.

Decorators:
- Implement OETimeout decorator, this will send a SIGALARM when some time expires.
- Implement OETag decorator, like OETestID this will hold a string for identify
some test case, also the ability to list OETestCase's by tag.

TestContext:
- Refactor/split oeqa/oetest.py: Currently this class have all the TestContext implementations
SDK, Image, SDKExt, the refactor needs to take into account don't depend on BItbake datastore.
- Implement/Review the support for load test suites in different layers.

Structure:
- Restructure code for avoid direct dependencies on Bitbake data store, the QA
framework needs to be implemented in a way to made useful outside Bitbake env.
- Audit and move/merge general helper modules outside oeqa/utils like httpserver,
logparser, git, ftools, network to avoid duplcated functionality, may be already exists
functions/method's in oe/lib or bitbake/lib.

Migration (steps to be defined):
- test{image,sdk,sdkext): Support the new oeqa framework.
- testexport: With changes of made independent oeqa framework of Bitbake, refactor current
testexport functionality.
- oe-selftest: Support the new oeqa framework.

Comment 4 Aníbal Limón 2016-09-06 22:40:37 UTC

Master bug filed [1], i'll continue to load the bugs and add as dependants of the master bug.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=10230