4137 2013-03-27 22:39:25 +0000 1.6 Test Automation Tracking Bug 2014-04-04 15:44:01 +0000 1 1 10 QA/Testing Build Testing general unspecified All Multiple RESOLVED FIXED High enhancement 1.6 2081 3894 4433 4435 4462 4463 4464 4740 5432 4691 1 sstncr sstncr corneliux.stoicescu david.c.stewart richard.purdie sgw venkatarg --- --- No (bug/feature does not impact docs) oldest_to_newest 31255 0 sstncr 2013-03-27 22:39:25 +0000 Autobuilder is not only a build system, it is used also as a QA tool and using the new AutoBuilder code and better runtime tests (qemuimagetest) we should do more QA on the AB (automate as many tests as possible). That implies more buildsets with different configurations that should cover our tests and more runtime checks. Tools like buildhistory (or other bitbake classes like sanity and insane) can be used/extended to fill the gaps (were buildsets and runtime tests aren't enough). 31355 1 richard.purdie 2013-04-02 11:50:20 +0000 This bug needs to be split out into specific deliverables. 31627 2 david.c.stewart 2013-04-05 23:01:36 +0000 I'm promoting this to a P1 feature for 1.5. As Richard says, it needs to be split up into the deliverables which we will target for various milestones in the 1.5 development cycle 37403 3 sstncr 2013-10-01 13:37:17 +0000 Update on the progress of this master bug: My initial description was to broad/general and the dependent bugs aren't the whole story (some of them aren't actually applicable anymore) so I'm gonna try to provide some background on what happened since this bug has been filed: Bug 3922 - Add sanity checks for Build appliance - Not done, not gonna happen too soon (it was initially targeted for the old qemuimagetest implementation, needs rework) Bug 4114 - Add ptest integration - Not done, not gonna happen too soon. It was somewhat reworked for the new testimage, needs more work. Bug 4433 - Run bitbake-selftest on the autobuilder - Done, still there. Bug 4463 - Add multilib builds to AB with opkg package manager - Done, still there. Bug 4464 - Add systemd enabled images to Autobuilder nightly target - Done, still there. Bug 4435 - Rework qemuimagetest implementation. Done and used on the AB (with #4838). Since it was merged new tests and fixes went it. Right now: For a standard core-image-sato the default suite has 38 test methods For a standard core-image-sato-sdk the default suite has 45 test methods. The theoretical maxium usable is 65 tests (no image is suitable for running all the tests on it, it would have to be a Frankestein core-image-sato-sdk with multilib and x32 and systemd, service, pax-utils and pam packages/features installed). AB has special targets for these purposes. http://autobuilder.yoctoproject.org:8011/builders/nightly-x86/builds/312/steps/Running%20Sanity%20Tests/logs/stdio Related to 4435 - Bug 4878 - Add ability for testimage to find tests in other layers Other stuff related to AutoBuilder/QA for which I didn't find a bug filed (and probably I missed some of them): - Add core-image-sato-sdk builds to sanity tests on AB http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder/commit/?id=9835ea5c2e522385bb3450f75ebd99280cdc55f8 - New AB scenarios since bug has been filed: http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=fa9f98bc36cd1e3d0c6b93340e9e13c9b145d7c9 http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=72bba944ae54534e359930cd9b4e25ce07bad063 http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=5e54f023fa13f6786ff8331fa2a123c56a925ddd http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/? id=77d3ba451e089c77a210cc4d957f9365e913fe9d As I did a poor job describing the feature initially :(, and I haven't said anything about specifics deliverable of this bug (or numbers) I have no idea on how to measure "increase the usage of AB as a QA tool". Sure thing it increased, not sure what the target was, so I guess the bugs stays IN PROGRESS forever. 38425 4 richard.purdie 2013-11-01 13:31:34 +0000 Retargeting this bug to track our 1.6 automated testing goals. This bugs is for the test automation efforts *except* those related to hardware test automation which is tracked separately in #1596. 42224 5 sstncr 2014-04-03 13:42:41 +0000 There are still 3 sub-bugs added to this tracking bug that have been moved to 1.7 cycle, move this one as appropiate. Btw, this bug did had more sub-bugs added over time than originally it had. So once those 3 (2082, 3922, 4114) are closed we should close this one and not add anymore sub-bugs (or else will never close it :)). 42305 6 richard.purdie 2014-04-04 15:44:01 +0000 These were the targets for 1.6 and overall we believe they were met so we can resolve this bug. For 1.7 we will create new ones specific to 1.7.