| Summary: | ptest: modify the result chart so that the current release commit is compared with the previous commit release | ||
|---|---|---|---|
| Product: | [QA/Testing] Test Plans/Suite | Reporter: | Libertad <libertad.gonzalez.de.la.cruz> |
| Component: | test-plans/suite | Assignee: | Jose Perez C <jose.perez.carranza> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | High | CC: | juro.bystricky, richard.purdie |
| Version: | 2.5 | ||
| Target Milestone: | 2.5 M1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Yes (doc changes required) | |
|
Description
Libertad
2017-09-15 20:17:08 UTC
This was a process issue, now a wiki is created [1] to track consistently all the results and a section of QA was added on ptest wiki [2] to write down the final process. 1- https://wiki.yoctoproject.org/wiki/Ptest/archive 2- https://wiki.yoctoproject.org/wiki/Ptest#QA When I look at (for example) https://wiki.yoctoproject.org/wiki/Ptest_5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 I see for libxml2: Ran Passed Failed libxml2 39 36 3 So I guess you determine number of passed/failed by counting of occurrences of strings PASS:/FAIL: However, the log file contains: Total 2656 tests, 2 errors, 0 leaks I realize that for comparison with the past results we need to use same parsing of logs, but I think we should also include (a new column?) the totals reported by the test program as well. (In reply to comment #2) > When I look at (for example) > https://wiki.yoctoproject.org/wiki/ > Ptest_5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 > > I see for libxml2: > Ran Passed Failed > libxml2 39 36 3 > > So I guess you determine number of passed/failed by counting of occurrences > of strings PASS:/FAIL: > > However, the log file contains: > > Total 2656 tests, 2 errors, 0 leaks > > I realize that for comparison with the past results we need to use same > parsing of logs, but I think we should also include (a new column?) the > totals reported by the test program as well. The main problem is that there is no standard output/results across the packages, someones give the total as libxml2, someones a "summary" and another ones does not give any info, the only common report across all the packages is PASS: / FAIL: hence for reporting purposes we only use PASS/FAIL criteria. Package Test (pTest) process executed by QA team was documented [1] to avoid discrepancies on reports, also a script was developed [2] to create the report wiki automatically and avoid error on the analysis of the results. This resulted with a new proposal of automated and storage process that will be addressed on Bug 12181. Based on above statement this bug can be closed, all the extra enhancement will be tracked on Bug 12181, 1- https://wiki.yoctoproject.org/wiki/BSP_Test_Plan#pTest 2- http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/tree/scripts/ptest-parser.py |