Bug 12181

Summary:	QA: Archive ptest results in git
Product:	[QA/Testing] Build Testing	Reporter:	Joshua Lock <joshuagloe>
Component:	general	Assignee:	yeoh ee peng <ee.peng.yeoh>
Status:	RESOLVED FIXED	QA Contact:
Severity:	enhancement
Priority:	Medium+	CC:	ee.peng.yeoh, jair.de.jesus.gonzalez.plascencia, liezhi.yang, randy.macleod, richard.purdie
Version:	2.4
Target Milestone:	4.99
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Joshua Lock 2017-10-04 13:22:36 UTC

Per the test plan[1] we currently store pTest results in the wiki[2] and generate comparisons using the ptest-parser in qa-tools[3].

We're increasingly moving QA data to git and using associated scripts that can generate reports or perform comparisons on the data. Examples of this trend are buildhistory and performance data.

It would be desirable to store pTest results this way also and be able to generate comparative reports based on the results archived in git.

Ideally we'd have:
a) a yp-qa-ptest-results (or similarly named) repo on git.yoctoproject.org
b) automated execution of the ptests on a dut (triggered by QA autobuilder)
c) scripts to generate reports comparing the ptest results from two builds

b and c should probably have separate bugs

1. https://wiki.yoctoproject.org/wiki/BSP_Test_Plan#pTest
2. https://wiki.yoctoproject.org/wiki/Ptest/archive
3. https://wiki.yoctoproject.org/wiki/Qa-tools#Ptest_Parser

Comment 1 Richard Purdie 2017-10-05 15:14:31 UTC

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 ?

Comment 2 Randy MacLeod 2017-10-05 15:16:12 UTC

Added Robert since he worked on the commit that adds ptest results to buildhistory that Richard just referenced above.

Comment 3 Joshua Lock 2017-10-05 16:00:17 UTC

(In reply to comment #1)
> http://git.yoctoproject.org/cgit.cgi/poky/commit/
> ?id=5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 ?

This is definitely part of the solution, it means we don't need a separate repository. I need to look closer, some extra tooling may or may not be required.

There were more patches in that series which don't appear to have been merged, they will be useful to resolve this bug.

https://patchwork.openembedded.org/series/8033/#

Comment 4 Joshua Lock 2017-10-05 16:02:29 UTC

(In reply to comment #3)
> (In reply to comment #1)
> > http://git.yoctoproject.org/cgit.cgi/poky/commit/
> > ?id=5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 ?
> 
> This is definitely part of the solution, it means we don't need a separate
> repository. I need to look closer, some extra tooling may or may not be
> required.
> 
> There were more patches in that series which don't appear to have been
> merged, they will be useful to resolve this bug.
> 
> https://patchwork.openembedded.org/series/8033/#

Or I'm an idiot, looks like they were all merged

Comment 5 Jair Gonzalez 2017-11-21 23:36:35 UTC

I've created bug 12372 to track the progress in the implementation of automated execution of pTest.

Comment 6 yeoh ee peng 2018-11-01 08:14:28 UTC

As point out by Richard and Randy, Robert had enabled ptest from testimage to archive test result to buildhistory git repository.

Had verified that this was working with below configuration, ptest result from testimage was stored into buildhistory git repository.  
TEST_SUITES = "ping ssh xorg ptest"
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

Closing this ticket.