10887 2017-01-06 15:14:03 +0000 Add specific QA strategy section to the manual 2017-04-28 19:05:49 +0000 1 1 9 Documentation Development Manual development unspecified x86 Multiple RESOLVED FIXED 10 April 2017: RESOLVED Medium normal 2.3 M4 1 richard.purdie srifenbark benjamin.esquivel joshuagloe libertad.gonzalez.de.la.cruz srifenbark libertad.gonzalez.de.la.cruz --- --- Done (doc changes complete) oldest_to_newest 69574 0 richard.purdie 2017-01-06 15:14:03 +0000 Looking at the manual there is no specific section about how the project is actually tested. Some pieces are documented but there is no summary. I therefore propose: """ Yotco Project Testing and QA strategy One of the key benefits the Yocto Project brings is that as well as having build tools, we add in testing strategies around this so that as well as being able to build customised Linux, you're also able to test and validate it. This QA/testing infrastructure is in fact woven into the project to the point its core developers take some of it for granted. Roughly speaking there are the following pieces: bitbake-selftest - A standalone command which runs unittesting on key pieces of bitbake and the bitbake fetchers. sanity.bbclass - This class is automatically included by default and checks the users environment for missing tools (e.g. gcc) or common misconfigurations (e.g. MACHINE set incorrectly) insane.bbclass - This class checks the generated output from builds for sanity. For example, if building for an ARM target, were ARM binaries generated? If PPC ones were, there is a problem! testimage.bbclass - Performs runtime testing of images after they're built. Usually used with qemu to boot the images and check the combined runtime result boots and functions but can also take the IP address of a machine to test. ptest - These are packages which package up the tests of a given piece of software, allowing them to be run within a target image. oe-selftest - Used to test cases where combinations of bitbake invocations need to be tested so the tests need to be outside of the buildsystem itself. Runs all tests by default or specific tests or test-suites can be selected. Originally, much of this testing was done manually but significant effort has been made in automating the tests so that more people can use them and we can run them faster and more efficiently. The project's main autobuilder (autobuilder.yoctoproject.org) publically tests the code in OE-Core, Poky, and BitBake both for the current state of master and also testing submitted patches, usually in batches in ross/mut (master-under-test) or master-next branches. This ensures in a publicly visible way that all of the main supposed architectures and recipes in OE-Core build and function. Various features such as multilib, sub architectures like x32, poky-tiny, musl, no-x11 and many more are tested as part of this, along with bitbake-selftest and oe-selftest. It does take the autobuilder workers several hours to validate any given code revision. The autobuilder workers are non-homogeneous meaning we get regular testing across a variety of distros, the autobuilder is limited to only testing qemu based setups and not real hardware. In addition to the autobuilder's tests, the QA team also performs testing on a variety of platforms, including some real hardware ones to ensure that things work as expected there. """ and we can then link to the various pieces we do have documentation for. There may also be further documentation needed if this highlights some gaps which can go into separate bugs. 69586 1 benjamin.esquivel 2017-01-06 19:28:02 +0000 I support adding this section to the manual. It might make sense to try to add a structure along with the already documented pieces. 69688 2 srifenbark 2017-01-10 21:58:53 +0000 I set some parameters for this effort in estimated days and target milestone. Also ACCEPTED the bug. I will start looking at this and see where to best fit it in the existing dev-manual organization. One thing that struck me while reading over Richard's draft was inclusion of details such as "ross/mut". Do we want to do that? Scott 71475 3 srifenbark 2017-03-20 18:21:12 +0000 Richard - Can you comment on Comment 2? Thanks, Scott 71527 4 richard.purdie 2017-03-21 17:08:31 +0000 (In reply to comment #2) > I set some parameters for this effort in estimated days and target > milestone. Also ACCEPTED the bug. I will start looking at this and see > where to best fit it in the existing dev-manual organization. > > One thing that struck me while reading over Richard's draft was inclusion of > details such as "ross/mut". Do we want to do that? I think we want to state that we do batch testing on user created branches and the two current most common examples are master-next (in poky) and ross/mut (in poky-contrib) but others may be used as circumstances require. 71723 5 sjolley.yp.pm 2017-03-24 17:56:33 +0000 It appears the question has been answered. 71908 6 srifenbark 2017-03-30 15:05:00 +0000 Hi, See this new section that overviews the testing infrastructure. http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#testing-and-quality-assurance Note that I cross-referenced the new section with a small paragraph in the dev-manual's "Performing Automated Runtime Testing" section to point to QA and testing infrastructure information. Please comment as needed. Thanks, Scott 72230 7 srifenbark 2017-04-10 18:18:34 +0000 Hi, No review since March. Marking RESOLVED and putting the doc flag to "done." Scott