6921 2014-10-31 20:21:24 +0000 ptest automated testing says passed when it failed 2015-02-19 16:19:40 +0000 1 1 10 QA/Testing Runtime Testing general 1.7 All Multiple RESOLVED FIXED Medium normal 1.8 M2 1 ross.burton georgex.l.musat alexandru.c.georgescu georgex.l.musat --- --- No (bug/feature does not impact docs) oldest_to_newest 46713 0 ross.burton 2014-10-31 20:21:24 +0000 So I added _ptest to TEST_SUITES and ptest-pkgs to IMAGE_FEATURES, built and image and ran the tests. test_ptestrunner (oeqa.runtime._ptest.PtestRunnerTest) ... NOTE: Installing complementary packages ... ok Lies. It either failed to install ptest-runner and assumed it worked, or it's eating the numerous errors that should have happened: root@qemux86-64:/usr/lib/libpcre/ptest# ./run-ptest make[1]: Entering directory '/usr/lib/libpcre/ptest' PASS: pcre_stringpiece_unittest FAIL: RunTest PASS: RunGrepTest ============================================================================ Testsuite summary for PCRE 8.34 ============================================================================ # TOTAL: 3 # PASS: 2 # SKIP: 0 # XFAIL: 0 # FAIL: 1 # XPASS: 0 # ERROR: 0 ============================================================================ See ./test-suite.log ============================================================================ Makefile:73: recipe for target 'test-suite.log' failed make[1]: *** [test-suite.log] Error 1 make[1]: Leaving directory '/usr/lib/libpcre/ptest' Makefile:159: recipe for target 'check-TESTS' failed make: *** [check-TESTS] Error 2 46714 1 ross.burton 2014-10-31 22:02:43 +0000 During an *in progress* ptest run: root@qemux86-64:~# grep FAIL: /tmp/ptest.log | wc -l 399 An hour later, it finished: core-image-sato - Ran 24 tests in 5099.061s core-image-sato - OK - All required tests passed (skipped=1) Lies! :) 46783 2 georgex.l.musat 2014-11-06 09:34:56 +0000 Hello Ross, To do a ptest succesfully I build a ptest capable image using these configurations in local.conf: IMAGE_FEATURES_append = " ptest-pkgs" IMAGE_INSTALL_append = " ptest-runner" INHERIT += "buildhistory" INHERIT += "testimage" TEST_SUITES = " _ptest" Buildhistory is mandatory and that test should fail without it. I also see that you skipped ptest-runner image install. But you are right, it seems that a check to see if ptest-runner actually ran succesfully is missing. 46784 3 georgex.l.musat 2014-11-06 09:45:53 +0000 Also I need to underline here that is a difference between the ptest unittest running succesfully and the actual ptest output, that needs to be analyzed separately. Ex: tmp/work/genericx86_64-poky-linux/core-image-sato-sdk/1.0-r0/testimage/ptest-20141106084139.log That is then parsed into: tmp/work/genericx86_64-poky-linux/core-image-sato-sdk/1.0-r0/core-image-sato-sdk-1.0/results$ ls fail.acl fail.attr fail.perl fail.quilt For example we could make the unittest fail every time a "fail.*" file is resulted but I don't think that the test will ever pass with this condition. 48938 4 ross.burton 2015-02-19 16:03:35 +0000 Lucian, is "oeqa/ptest: Fixed complementary package install detection and added ptest-runner exit code check" the fix for this? (oe-core b214c7b901fe0fad081d7a97f9e91cc218c66bd6) 48942 5 ross.burton 2015-02-19 16:19:40 +0000 Yes, it is.