Bug 9678 - Include automated test execution and reporting in Autobuilder
Summary: Include automated test execution and reporting in Autobuilder
Status: RESOLVED INVALID
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: 2.2
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 4.99
Assignee: Aaron Chan
QA Contact:
URL: http://centaurus01.png.intel.com:8010...
Whiteboard:
: 10580 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-05-26 20:11 UTC by Benjamin Esquivel
Modified: 2018-05-28 02:07 UTC (History)
11 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Benjamin Esquivel 2016-05-26 20:11:55 UTC
The purpose of this bug is to add functionality to the AB for executing the automated test suites.

The tests targeted for inclusion are: testimage, oe-selftest, performance measurements, etc.

One requirement is that all the automated tests that QA runs in a release test cycle can be run at the Autobuilder.

For reporting, initially the target will be to report back to the Testopia session that was setup for the QA cycle. Later will be required another bug for reporting to a test report store that is being tracked at bug 9559.

A text test report should exist at the Autobuilder reporting step for the cases where communication to the test report tool fails. There should be a way to see the results at the execution instance which is the AB in this case.

Test selection will be pre-set for this bug (all tests, per-component test AB job separation). Later, another bug needs to take care of being able to select which tests will be run (perhaps via a field in the forced builds page?) 

Anibal has been working on this for weeks now but here's the bug to track that work.
Comment 1 Leonardo Sandoval Gonzalez 2016-06-23 19:55:36 UTC
My fault, moving it to M1 again. CCB should agree on any milestone movement.
Comment 2 Leonardo Sandoval Gonzalez 2016-06-30 20:27:33 UTC
Demo created (http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/) but it needs more work in order to close this bug.
Comment 3 Benjamin Esquivel 2016-07-15 17:17:18 UTC
This becomes a High if moved to 2.2 M3
Comment 4 Leonardo Sandoval Gonzalez 2016-08-25 17:23:11 UTC
This work wont be ready for this release. Moving to next release, expecting more involvement from QA team.
Comment 5 Leonardo Sandoval Gonzalez 2017-01-25 21:55:05 UTC
Anibal, can you please comment on this one. Ideas to split it?
Comment 6 Aníbal Limón 2017-03-28 18:00:09 UTC
(In reply to comment #5)
> Anibal, can you please comment on this one. Ideas to split it?

There are two needed tasks for complete this,

1) Finish the testopia_update script.
2) Add the buildstep into the Autobuilder for automatic report (call testopia_update).

Currently Jose Perez done the 1) part so the only remaining part will be add the buildstep.

I'll change the owner to Jose Perez and now the split isn't needed.
Comment 7 Leonardo Sandoval Gonzalez 2017-06-02 19:45:38 UTC
*** Bug 10580 has been marked as a duplicate of this bug. ***
Comment 8 Aaron Chan 2018-01-25 04:58:27 UTC
With reference to http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/

Base on the usage model of the testopia_update.py script, it appears that the test cases update is much faster manually compare to updating the test cases result using the script sourcing with the results logfiles. 

In some cases, the results updating results into Testopia appeas to be locking up, unresponsive for several hours and unable to fully complete the result publishing even when running on a bare-metal server. 

Hence the testopia_update.py script require some effort to debug and analysis the issue am facing.

On other areas of enhancement which was tested in autobuilder.

1. Added a patch UploadTestopia.py as an automated buildstep option
2. Modify the buildset-config jobs to adopt the buildstep in item 1.
3. Changes to the job specific interface to accept "Test Run ID" as string 
   parameter to invoke the testopia_update script to upload the auto testcases

Issue Blocking:

1. Testopia_update.py script does not work as per expected
2. Logfiles can be generated in various paths, different logfile names, and 
   some job configuration requires multiple logfiles of various test shares 
   the same test run ID in Testopia.
Comment 9 Aaron Chan 2018-05-28 02:07:39 UTC
This ticket is back dated 2 years ago and the direction has been moved to
test cases management to a different platform whereby Testopia will be retired replacing by other Test case management tool by the community.

Hence, this ticket is no longer valid and the status is now changed to Never Fix. A new ticket should be raised indicating the same motivation and assign to
the dedicated owner currently working on the enhancements.