Bug 5554

Summary: Add abstraction for running tests on real hardware
Product: [QA/Testing] Runtime Testing Reporter: Paul Eggleton <bluelightning>
Component: generalAssignee: Stefan Stanacar <sstncr>
Status: VERIFIED FIXED QA Contact: Bogdan Alexandru Voiculescu <bogdanx.a.voiculescu>
Severity: enhancement    
Priority: Medium+ CC: bluelightning, bogdanx.a.voiculescu, jessica.zhang, srifenbark
Version: 1.6   
Target Milestone: 1.6 M1   
Hardware: All   
OS: Multiple   
Whiteboard: 18 April 2014: Setting documentation flag to "Done".
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Done (doc changes complete)
Bug Depends on:    
Bug Blocks: 1596    

Description Paul Eggleton 2013-11-18 16:55:22 UTC 
Insert some abstraction between the request to run a test and the launching of it in QEMU so that we can also run the test on real hardware. How the test should be run should be able to be set via a variable, and the variable's value should map to a python class with the implementation of how the image should be deployed and booted.

We should create two initial implementations:
1) A QEMU implementation (does what it does currently)
2) A very simple remote implementation which does not deploy the image but runs the tests on a remote machine via ssh to a specified IP address.

In the current code there's a fair amount of isolation here already so this shouldn't be too tricky.
Comment 1 Stefan Stanacar 2013-11-25 17:56:02 UTC 
Patches on the ML.
Comment 3 Scott Rifenbark 2014-04-08 20:31:10 UTC 
Hi, 

I have added several areas to address this documentation part of the bug based on Stefan's email he sent me.  The new sections need a good review.  I had some difficulty unwrapping the raw information into individual sections so please look it over and comment:

http://www.yoctoproject.org/docs/1.6/dev-manual/dev-manual.html#performing-automated-runtime-testing - This is the main section on running tests in the dev-manual.  It now needs to cover qemu and actual hardware.  I suspect we need to work on this section some more.

These next links are to the new variables.

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#var-TEST_TARGET

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#TEST_TARGET_IP

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#TEST_SERVER_IP

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#TEST_LOG_DIR

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#TEST_EXPORT_ONLY

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#TEST_EXPORT_DIR
Comment 4 Scott Rifenbark 2014-04-10 19:12:38 UTC 
Hi, 

I realized that this area in the manual also needed adjusting based on the ability to now run tests on hardware - http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#ref-classes-testimage.  I basically removed the note that indicated the class only supported running tests with QEMU, excluding hardware. 

Please add this link to the review as well. 

Scott
Comment 5 Scott Rifenbark 2014-04-11 15:50:38 UTC 
Hi, 

I incorporated both your sets of comments (Paul and Stefan) and sent out emails with notes.  Can you guys look the changes over to approve?

http://www.yoctoproject.org/docs/1.6/dev-manual/dev-manual.html#hardware-image-enabling-tests

In that link above, there is one item I need more information on from Stefan.  I have your name in there... you will see it.

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#var-TEST_IMAGE

http://www.yoctoproject.org/docs/1.6/ref-manual/ref-manual.html#var-TEST_TARGET

Thanks, 
Scott
Comment 6 Scott Rifenbark 2014-04-18 16:08:41 UTC 
Hi, 

I worked with Stefan here to update the section on running tests.  We are good now with it all.  See http://www.yoctoproject.org/docs/1.6/dev-manual/dev-manual.html#performing-automated-runtime-testing.

Setting the doc flag to "Done".

Scott
Comment 7 Bogdan Alexandru Voiculescu 2014-08-19 11:40:58 UTC 
Verified.