oeqa has several deficiencies which make it difficult to work with and maintain, including: * dependencies are scattered around the code base * test-loading only supports one folder deep * test dependency order is non-deterministic and doesn't work well * module relations/dependencies are unclear We'd like to move to a new design which extends the python unittest module and better supports test dependencies and ordering and provides a clear high-level API addressing the concerns above. In order to do that we first need to perform a review of the oeqa codebase and plan the work involved in moving forward with the new design. This task will be complete when a new set of tickets are filed covering the tasks involved in improving oeqa.
This plan needs to take into account XML output see, https://bugzilla.yoctoproject.org/show_bug.cgi?id=9682
Re-assigning to Edwin and moving to M3 as this will not make it for 2.2 M2.
I finish the first draft of the plan, some tasks related to migration needs to define a set of step for do it but the other depend tasks needs to be complete for provide details of migration, i will load the bugs for cover the next tasks: Runner: - Implement OETestRunner with the ability for output logs in JUnit XML and RAW text based on configuration parameter. - Implement OEQemuTestRunner extending OETestRunner, this will start, stop qemu with certain configuration. Decorators: - Implement OETimeout decorator, this will send a SIGALARM when some time expires. - Implement OETag decorator, like OETestID this will hold a string for identify some test case, also the ability to list OETestCase's by tag. TestContext: - Refactor/split oeqa/oetest.py: Currently this class have all the TestContext implementations SDK, Image, SDKExt, the refactor needs to take into account don't depend on BItbake datastore. - Implement/Review the support for load test suites in different layers. Structure: - Restructure code for avoid direct dependencies on Bitbake data store, the QA framework needs to be implemented in a way to made useful outside Bitbake env. - Audit and move/merge general helper modules outside oeqa/utils like httpserver, logparser, git, ftools, network to avoid duplcated functionality, may be already exists functions/method's in oe/lib or bitbake/lib. Migration (steps to be defined): - test{image,sdk,sdkext): Support the new oeqa framework. - testexport: With changes of made independent oeqa framework of Bitbake, refactor current testexport functionality. - oe-selftest: Support the new oeqa framework.
Master bug filed [1], i'll continue to load the bugs and add as dependants of the master bug. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=10230