Bug 6921

Summary: ptest automated testing says passed when it failed
Product: [QA/Testing] Runtime Testing Reporter: Ross Burton <ross.burton>
Component: generalAssignee: Lucian Musat <georgex.l.musat>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: alexandru.c.georgescu, georgex.l.musat
Version: 1.7   
Target Milestone: 1.8 M2   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Ross Burton 2014-10-31 20:21:24 UTC 
So I added _ptest to TEST_SUITES and ptest-pkgs to IMAGE_FEATURES, built and image and ran the tests.

test_ptestrunner (oeqa.runtime._ptest.PtestRunnerTest) ... NOTE: Installing complementary packages ...
ok

Lies.  It either failed to install ptest-runner and assumed it worked, or it's eating the numerous errors that should have happened:

root@qemux86-64:/usr/lib/libpcre/ptest# ./run-ptest
make[1]: Entering directory '/usr/lib/libpcre/ptest'
PASS: pcre_stringpiece_unittest
FAIL: RunTest
PASS: RunGrepTest
============================================================================
Testsuite summary for PCRE 8.34
============================================================================
# TOTAL: 3
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
============================================================================
Makefile:73: recipe for target 'test-suite.log' failed
make[1]: *** [test-suite.log] Error 1
make[1]: Leaving directory '/usr/lib/libpcre/ptest'
Makefile:159: recipe for target 'check-TESTS' failed
make: *** [check-TESTS] Error 2
Comment 1 Ross Burton 2014-10-31 22:02:43 UTC 
During an *in progress* ptest run:

root@qemux86-64:~# grep FAIL: /tmp/ptest.log  | wc -l
399

An hour later, it finished:

core-image-sato - Ran 24 tests in 5099.061s
core-image-sato - OK - All required tests passed (skipped=1)

Lies! :)
Comment 2 Lucian Musat 2014-11-06 09:34:56 UTC 
Hello Ross,

To do a ptest succesfully I build a ptest capable image using these configurations in local.conf:

IMAGE_FEATURES_append = " ptest-pkgs"
IMAGE_INSTALL_append = " ptest-runner"
INHERIT += "buildhistory"
INHERIT += "testimage"
TEST_SUITES = " _ptest"

Buildhistory is mandatory and that test should fail without it. I also see that you skipped ptest-runner image install.

But you are right, it seems that a check to see if ptest-runner actually ran succesfully is missing.
Comment 3 Lucian Musat 2014-11-06 09:45:53 UTC 
Also I need to underline here that is a difference between the ptest unittest running succesfully and the actual ptest output, that needs to be analyzed separately. 

Ex:

tmp/work/genericx86_64-poky-linux/core-image-sato-sdk/1.0-r0/testimage/ptest-20141106084139.log

That is then parsed into:
tmp/work/genericx86_64-poky-linux/core-image-sato-sdk/1.0-r0/core-image-sato-sdk-1.0/results$ ls
fail.acl  fail.attr  fail.perl  fail.quilt

For example we could make the unittest fail every time a "fail.*" file is resulted but I don't think that the test will ever pass with this condition.
Comment 4 Ross Burton 2015-02-19 16:03:35 UTC 
Lucian, is "oeqa/ptest: Fixed complementary package install detection and added ptest-runner exit code check" the fix for this?

(oe-core b214c7b901fe0fad081d7a97f9e91cc218c66bd6)
Comment 5 Ross Burton 2015-02-19 16:19:40 UTC 
Yes, it is.