Following discussions about automation for build time tests we should have a script/framwork/infrastructure/whatever that helps us automating certain build tests. I'm calling these buildtime tests triggerd by an external script - they shouldn't be called by bitbake or ran post-build/rootfs. The idea is some kind of oe-selftest script similar bitbake-selftest which would run "bitbake -g or --someoption, change some metadata, bitbake X, check something" type scenarios (like PR service, error output, etc) or "bitbake-layers show_appends/<...>" type and yocto-bsp tools. The script: - it targets short tests (not long builds) - it will be a standalone target/build on the AB - should reside in scripts/ - should call the bitbake <target> or bitbake tools, create configs, check results - should use real data (as possible) for tests not dummy/constructed data (check the fetch example for bitbake-selftest to see what I mean) e.g when testing something like bitbake-layers use the actual layers defined in bblayers.conf don't feed the tests dummy data. - it should be a simple matter of running oe-selftest in a build dir and get some results (similar to bitbake-selftest) - it should provide an way to select different tests or run them all - preferably use python unittest for writing tests Its purpose is to automate certain kind of build tests: https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=142 https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=143 https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=93 https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=144 https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=146 https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=147 https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=286 Tests which are NOT meant to be automated with this tool: - any test that required booting an image (runtime tests) - long builds - tests that check stuff in rootfs or build dir e.g https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=297 Feel free to comment/rewrite part/all of the above.
This will be moved to 1.6.
Work started. Current status: -Initial implementation sent for review on oe-core mailing list. -Working on automating current manual tests; will adapt finished tests if script changes. The current goal is to cover as many manual tests and used the saved time to focus on more automation tasks in the future. After current manual tests are 'dealt with', some test cases can be expanded as needed to cover more features. Also new tests will be added then.
Script + meta-selftest layer + a bunch of tests merged in master, some of the commits are: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=645dd61cd21b2b1c9d9c927447c68841685a1adf http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b9afc0d262160a493a55c9ced6cd285ef29f8feb http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=1611b474b69228597b46590db5fe8532b71147df
Need Documentation.
I am working on finishing the socumentation for this. We now have a wiki page that describes how oe-selftest works without going into much detail: http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded I am also working on adding docstrings to test modules that describe the tests. This should be ready by the end of this week.
Wrong link :( Here is the good one: https://wiki.yoctoproject.org/wiki/Oe-selftest
Hi, Can I assume that these tests are not for the general YP documentation? And, that the wiki will suffice? Thanks, Scott
Yes, the wiki should suffice. If you need any more information I will be happy to help.
The wiki page at https://wiki.yoctoproject.org/wiki/Oe-selftest suffices for the documentation portion of this bug. I am setting it to "Done." Scott
Verified. selftest is already running on AB.