Implement a test suite that will contain generic recipe tests. The test suite can be run after modifications have been made to a recipe to check for issues. Here is the initial proposal from the mailing list and community comments: - build the recipe with all major architectures(qemux86, qemux86-64, qemuarm, qemuppc, qemumips) - rebuild the recipe from sstate(with or without a sstate file for the recipe) - perform cleaning operations on the recipe(cleansstate, clean, cleanall) - force all major tasks on the recipe (bitbake -C <task> <recipe>) - selectively use each combination of .bbappend files with the recipe; all the combinations should not break the recipe build. - we could also create mini test suites just for some of these tests like testing only the rebuild from sstate. Community input: - boot in qemu and do ptest suite
This perhaps ties in with bug 4078.
I suspect this isn't going to fit into oe-selftest, since it's going to need to take the recipe to test as input. It's also worth noting that the automatic upgrade helper already does some of this.
This bug is a dependency of 8648 which is targeted by 2.1 M1, it means we need to change this to the same milestone to make it happen.
Hi Benjamin, I haven't had the time to take care of this one, as my main focus is now on #8647. If this is blocking you, and you have the resources, please feel free to change it's status accordingly. Thanks, --Daniel
Thanks for the comment Daniel, will check that option.
This is going to be a separate tool 'test-recipe' that will run the 'recipe tests'. All these tests will require as argument the recipe to test on. The script: scripts/test-recipe The tests: meta/lib/oeqa/recipe-tests/ Proposed usage: test-recipe --recipe recipe_x --run-tests test_1 test_2 test-recipe --recipe recipe_x --run-tests all test-recipe --list-tests test-recipe will create an environment variable TESTRECIPE that each test will know about it. This way we can pass arguments to tests.
Per our conversation between Paul and I, it would be useful to know whether this enhancement is a matter of adding more QA checks (http://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-qa-checks) or extending them. I believe the current design is set for being a command line callable set of tests and if that's the case, I wonder where in the automation will they fit, is the current design to tie them to oe-selftest?
Patch sent to ML: http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118170.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118171.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118172.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118173.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118174.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118175.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118176.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118177.html http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118178.html Also available at: https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=distrate/bug_6370_testrecipe_final This tool shares many functionality with oe-selftest. It requires an additional parameter '--recipes <recipe>' to know on which recipe to run tests against. Tests are located in meta/lib/oeqa/recipetests/ --Daniel
Created attachment 3006 [details] test-recipe demo
I'm taking this bug since I need to see what is going to happen with it.
Chatting with Paul we decided this enhancement request will not get attention in 2.3 so setting its target milestone to future.
http://lists.openembedded.org/pipermail/openembedded-core/2016-February/118175.html has good ideas