Bug 12441 - ptest comparison script is not showing correct results
Summary: ptest comparison script is not showing correct results
Status: RESOLVED FIXED
Alias: None
Product: Test Plans/Suite
Classification: QA/Testing
Component: test-plans/suite (show other bugs)
Version: 2.3.3
Hardware: x86 Multiple
: Medium+ critical
Target Milestone: 2.7 M2
Assignee: Sudhir Sharma
QA Contact: yeoh ee peng
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-12-21 02:47 UTC by Libertad
Modified: 2019-01-30 09:53 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments
ptest_2.3.3.rc1_compared_2.3.2.rc1 (33.47 KB, image/png)
2018-01-19 06:23 UTC, yeoh ee peng
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Libertad 2017-12-21 02:47:30 UTC
OVERVIEW:

After ptest is executed, it gets compared with the previous release. The script is not comparing correctly since in 2.3.2 rc1 the python test ran with a pass rate of 99+ and in the comparison with 2.3.3 rc1, the results show that passrate was 0% in 2.3.2 rc1.

Compare the ptest report on both wikis.

https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1#pTest_for_genericx86-64_on_NUC

https://wiki.yoctoproject.org/wiki/WW38_-_2017-09-20-_Full_Test_Cycle_-_2.3.2_rc1#pTest_for_genericx86-64_on_NUC
Comment 1 yeoh ee peng 2018-01-19 06:23:08 UTC
Created attachment 4180 [details]
ptest_2.3.3.rc1_compared_2.3.2.rc1
Comment 2 yeoh ee peng 2018-01-19 06:25:00 UTC
Executed the ptest_parser.py, could not reproduce the issue. 
Attached the ptest result that show the python result is working as expected.
Comment 3 yeoh ee peng 2019-01-03 05:42:28 UTC
ptest comparison script is not showing correct results.
For QA 2.7 M1 rc1, the ptest summary from ptest.log (https://wiki.yoctoproject.org/wiki/Ptest_fe43e599327f26b399c1883205e988236a5e51bd) show:
=== Test Summary ===
TOTAL: 579
PASSED: 316
FAILED: 251
SKIPPED: 12
END: /usr/lib/valgrind/ptest

But the qa-tools/scripts/ptest-parser.py show: 
TOTAL (Test Ran): 567
PASSED: 304

Need to fix this an rerun ptest comparison script.
Comment 4 yeoh ee peng 2019-01-16 06:55:36 UTC
QA team was working on this.
Comment 5 yeoh ee peng 2019-01-30 09:53:33 UTC
Fixed developed by Sudhir and patch pushed to qa-tools
http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/commit/?id=e1a81ebc332014b71a680df2742bf950ce542295