Ptests should be ran, and results interpreted, as part of automatic testing.
From Richard's recent comments, this would need us to: * Enable ptest to run from the qemuimagetests * Find a way to collect up the ptest results (log into buildhistory so we can compare tests results to previous runs?). Important detail is when test results which did pass start failing.
I'll take shot at this. I've started something here: http://git.yoctoproject.org/cgit.cgi/poky-contrib/?h=stefans%2Fqemu-ptest1 specifically the first part: enable ptest to run from the qemuimagetests I've tested a core-image-sato with the following in local.conf: DISTRO_FEATURES_append = " ptest" IMAGE_FEATURES_append = " ptest-pkgs tools-sdk" IMAGE_INSTALL_append = " ptest-runner dbus bash glib-2.0" IMAGETEST = "qemu" TEST_SCEN = "standalone:ptest" I have some issues: - running ptest takes a long time, right know I've used 850 seconds as a timeout but that's not enough. I want to add support in qemuimagetest for passing arguments to runqemu (arguments like kvm and qemuparams="-m 1024"), and that should help. - there are tests which fail and are known to fail - how to exclude them. - I have no idea on how to integrate the results in buildhistory from qemutests. Help?
Yes, ptests can take a very long time. And obviously the time varies for different targets too. In our system we use a timeout per line instead of per command, so that as long as the tests produce output they don't time out. The ptests recipes in dylan still have some bugs, causing more test failures than necessary (and possibly timeouts). I will submit patches for that shortly. As for integrating with qemuimagetest, I don't know. We are using an expect script to boot and run all targets in buildbot, both qemu and hardware boards. We run this in autobuilder and parse the logs to extract results. I think we should have a discussion about how to implement automatic testing in a way so it is easy for people to set up and run auto tests in their projects and on their hardware.
(In reply to comment #3) > Yes, ptests can take a very long time. And obviously the time varies for > different targets too. In our system we use a timeout per line instead of > per command, so that as long as the tests produce output they don't time out. > Hmm, this might be a good idea. I'll think about it. > The ptests recipes in dylan still have some bugs, causing more test failures > than necessary (and possibly timeouts). I will submit patches for that > shortly. > Ok, great. > As for integrating with qemuimagetest, I don't know. We are using an expect > script to boot and run all targets in buildbot, both qemu and hardware > boards. We run this in autobuilder and parse the logs to extract results. > The thing is we don't want to tie people to the autobuilder to run these tests. Some people might use other build systems (like jenkins). That's way we want the tests to be run as part of qemuimagetest. These way they can be fired off from a command line too. The bad thing is that doesn't integrate with real hardware. After some changes to qemuimagetest went in master: - we can now pass qemuparams="-m 1024" and that helps a lot - a timeout can be specified to Test_SSH now, before it was limited to 50 seconds. That helped me with the long running part and timeouts. Given that and the top 2 commits from http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/qemu-ptest2 now I can succesfully run a sato image and get the log file. If the image it's built with INHERIT += "buildhistory" the ptest log will be copied in buildhistory dir. I'm wondering if that's enough for tracking. > I think we should have a discussion about how to implement automatic testing > in a way so it is easy for people to set up and run auto tests in their > projects and on their hardware. I think that will be great.
Unfortunately I don't think anything usable will make it in 1.5, so I'm bumping the target milestone. Also, as the qemuimagetest is obsolete by testimage.bbclass, I've make an attempt at a skeleton implementation here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=stefans/run-ptest1&id=b9e86394684f2c3809b3539cbe6b67e28e4d96cf It just runs ptest-runner and copies the log to buildhistory dir but it doesn't do any "check" of the results. The good part is that testimage already had support for restart the target with more memory (but now qemu have 256 by default so it's okay) and for setting longer or no timeouts. And if testimage will know about real hardware this will just work. (either from an AB setup or fired off from command line).
Ptest runtime test has been developed and it's waiting review
Code merged to master http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=7c16a538bc57116b87cf1912e7f4a460b38e015c
Verified on commit 8ac8eca2e3bd8c78e2b31ea974930ed0243258a3.