| Summary: | New QA process issues | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [QA/Testing] Test Plans/Suite | Reporter: | Richard Purdie <richard.purdie> | ||||
| Component: | test-plans/suite | Assignee: | Apoorv <apoorv.sangal> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | High | CC: | akuster, bluelightning, ee.peng.yeoh, randy.macleod, ross.burton, sangeeta.jain, tracey.m.erway | ||||
| Version: | unspecified | ||||||
| Target Milestone: | 2.8 M2 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | Yes (doc changes required) | |||||
| Attachments: |
|
||||||
|
Description
Richard Purdie
2019-03-28 15:47:46 UTC
for ptest bugs, https://bugzilla.yoctoproject.org/show_bug.cgi?id=13238 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13239 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13240 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13241 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13242 these were reported on the basis on results from ptest run by Intel QA team. But these results were found to be inconsistent and inaccurate in follow up investigations, hence these bugs are marked as "NOT A BUG". Final ptest bugs for 2.7 M3 RC1 are files as follows: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13253 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13254 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13249 Also, we need to identify "timeout" tests. These are show in the main report with a T against them (e.g. strace, python3, lttng-tools). These should be specific bugs making it clear a timeout occurred with the test. Right now I don't think the QA process is flagging those up and it should. QA team was not flagging "timeout" issues so far. We take it as an AR and start reporting timeout issues. For 2.7 M3 RC1, 3 timeout issues reported: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13255 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13256 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13257 (In reply to comment #0) > Now that we're running through the new QA process there are a few problems > we're running into. > > Stephen and myself have no visibility into who is running tests, what > progress is being made or when we can expect results (or which results to > block on). This can be resolved once we can upload test results on "yocto-testresults" repo. You can see what tests are completed and results are available. > > At a bare minimum I'd propose we need to have "ack" replies to the "QA > start" email from anyone who is planning to run tests saying what tests they > plan to run and when they'll send the results back. This would mean we know > what to wait for. > Agree on this. We can send you an email for our plan of QA cycle specifying what test we are going to run on which build. Created attachment 4457 [details]
2.7 m3 rc1 report from intel and windriver
Sent the new QA process to yocto mailing list. Here was the link https://lists.yoctoproject.org/pipermail/yocto/2019-April/044661.html For new QA process trial run (2.7 m3 rc1), extra test results from intel and windriver team are compiled and sent to Richard for review. Attached the QA result report for extra test results from intel and windriver. We will create new sub-bug to track unresolved activities then close the defect. Documented the new QA process: https://wiki.yoctoproject.org/wiki/New_QA_process Attached a link at main QA page to the new QA process page: https://wiki.yoctoproject.org/wiki/QA#New_QA_Process (In reply to comment #7) > Documented the new QA process: > https://wiki.yoctoproject.org/wiki/New_QA_process > > Attached a link at main QA page to the new QA process page: > https://wiki.yoctoproject.org/wiki/QA#New_QA_Process thank you The resulttool is command line tool available inside poky/scripts, it was developed to manage test result files for both automated tests (eg. oeqa/runtime, oeqa/selftest, oeqa/sdk, etc) and manual tests (eg. oeqa/manual). It was developed as an alternative tool to replace test result storage and reporting using Testopia and Wiki Page. https://wiki.yoctoproject.org/wiki/Resulttool QA wiki page has now been updated with the new process. |