Bug 6370

Summary: QA/oe-selftest: add test suite for recipe testing
Product: [QA/Testing] Build Testing Reporter: Stoicescu Cornel <corneliux.stoicescu>
Component: generalAssignee: Richard Purdie <richard.purdie>
Status: NEW --- QA Contact:
Severity: enhancement    
Priority: Medium CC: alexandru.c.georgescu, benjamin.esquivel, bill.randle, bluelightning, markus.boos, randy.macleod, richard.purdie, sgw
Version: 1.7   
Target Milestone: 6.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Bug Depends on: 8332, 9085, 9931    
Bug Blocks:    
Attachments:
Description Flags
test-recipe demo none

Description Stoicescu Cornel 2014-05-23 12:01:53 UTC 
Implement a test suite that will contain generic recipe tests. The test suite can be run after modifications have been made to a recipe to check for issues.

Here is the initial proposal from the mailing list and community comments:

- build the recipe with all major architectures(qemux86, qemux86-64, qemuarm, qemuppc, qemumips)
- rebuild the recipe from sstate(with or without a sstate file for the recipe)
- perform cleaning operations on the recipe(cleansstate, clean, cleanall)
- force all major tasks on the recipe (bitbake -C <task> <recipe>)
- selectively use each combination of .bbappend files with the recipe; all the combinations should not break the recipe build.
- we could also create mini test suites just for some of these tests like testing only the rebuild from sstate.

Community input:
- boot in qemu and do ptest suite
Comment 1 Paul Eggleton 2015-10-28 10:13:40 UTC 
This perhaps ties in with bug 4078.
Comment 2 Paul Eggleton 2015-10-28 14:26:48 UTC 
I suspect this isn't going to fit into oe-selftest, since it's going to need to take the recipe to test as input.

It's also worth noting that the automatic upgrade helper already does some of this.
Comment 3 Benjamin Esquivel 2015-11-05 23:15:30 UTC 
This bug is a dependency of 8648 which is targeted by 2.1 M1, it means we need to change this to the same milestone to make it happen.
Comment 4 Daniel Istrate 2015-12-02 10:16:20 UTC 
Hi Benjamin,

I haven't had the time to take care of this one, as my main focus is now on #8647.

If this is blocking you, and you have the resources, please feel free to change it's status accordingly.

Thanks,
--Daniel
Comment 5 Benjamin Esquivel 2015-12-02 16:34:15 UTC 
Thanks for the comment Daniel, will check that option.
Comment 6 Daniel Istrate 2016-01-19 09:51:51 UTC 
This is going to be a separate tool 'test-recipe' that will run the 'recipe tests'. 

All these tests will require as argument the recipe to test on.

The script: scripts/test-recipe
The tests:  meta/lib/oeqa/recipe-tests/

Proposed usage:
test-recipe --recipe recipe_x --run-tests test_1 test_2
test-recipe --recipe recipe_x --run-tests all
test-recipe --list-tests

test-recipe will create an environment variable TESTRECIPE that each test will know about it. This way we can pass arguments to tests.
Comment 7 Benjamin Esquivel 2016-02-09 15:54:28 UTC 
Per our conversation between Paul and I, it would be useful to know whether this enhancement is a matter of adding more QA checks (http://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-qa-checks) or extending them.

I believe the current design is set for being a command line callable set of tests and if that's the case, I wonder where in the automation will they fit, is the current design to tie them to oe-selftest?
Comment 9 Daniel Istrate 2016-02-26 14:57:04 UTC 
Created attachment 3006 [details]
test-recipe demo
Comment 10 Benjamin Esquivel 2016-11-03 22:17:02 UTC 
I'm taking this bug since I need to see what is going to happen with it.
Comment 11 Benjamin Esquivel 2016-11-04 14:39:04 UTC 
Chatting with Paul we decided this enhancement request will not get attention in 2.3 so setting its target milestone to future.