Bug 12372

Summary:	Automate the execution of pTest
Product:	[QA/Testing] Build Testing	Reporter:	Jair Gonzalez <jair.de.jesus.gonzalez.plascencia>
Component:	general	Assignee:	Yang Wang <yang.wang>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium	CC:	aaron.chun.yew.chan, akuster, konrad.scherer, randy.macleod, richard.purdie, ross.burton
Version:	2.5
Target Milestone:	4.99
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Jair Gonzalez 2017-11-21 23:28:20 UTC

The PtestRunnerTest class, an existing runtime class in meta/lib/oeqa/runtime/cases/ptest.py already allows to call the ptest-runner script using testimage, parse its results and save them in the logs. However, considering this test needs to be executed on actual hardware, the main obstacle to automate the complete process is to be able to flash and bring up the devices automatically. Though, once the LAVA system allows to do this, it may be possible to complete the work on this area and complete automating the pTest execution.

The purpose of this item is to track the progress and completion of this task.

Comment 1 Randy MacLeod 2018-07-12 19:33:39 UTC

Yang uses Lava quite a bit so he'll work on this enhancement in the coming weeks.
We need to work out an acceptable approach with the goal being that the ptest results and stats analysis is in OE code and LAVA takes care of the hardware boot and control. I'm sure there's more detail to work out via say a brief RFC email to the oe-core list.

Comment 2 Yang Wang 2018-07-12 19:37:15 UTC

This item is on my TODO list now, will update the status in next few weeks.

Comment 3 Aaron Chan 2018-08-23 05:23:11 UTC

We are using LAVA to execute core-image-sato-sdk test cases on hardware but
we had not run any pTest on hardware but rather trying to integrate
phoronix test suite into the image. This PoC has been done x86-64 hardware 
specifics and soon it will be roll out to the community.

Comment 4 Richard Purdie 2018-08-24 08:55:00 UTC

There has been some work to allow lava and autobuilder2 to interact. The work is still being developed but the patches are are:

https://github.com/lab-github/yocto-autobuilder2/commits/linaro/lava
https://github.com/lab-github/yocto-autobuilder-helper/commits/linaro/lava

which most of the supporting code going into the -helper. This work should be submitted soon and could form a base for sending jobs to lava then reporting the results back into the autobuilder for ptest.

Comment 5 Yang Wang 2018-08-24 15:24:05 UTC

Thanks for the comment.
I quickly looked into the code, it tries to make the helper code common and set LAVA server, bsps and device types as variables.

We are doing some similar work:
https://github.com/WindRiver-OpenSourceLabs/ci-scripts

Currently, we haven't test it much with public LAVA server and their board farm, but we do have plan to make it work as soon as we can.

Comment 6 Ross Burton 2019-05-20 15:22:17 UTC

Closing as we run ptest automatically now.

Comment 7 Randy MacLeod 2019-05-23 15:28:53 UTC

Ross can you add a link to a commit or defect for that work please?