Now that we're running through the new QA process there are a few problems we're running into. Stephen and myself have no visibility into who is running tests, what progress is being made or when we can expect results (or which results to block on). At a bare minimum I'd propose we need to have "ack" replies to the "QA start" email from anyone who is planning to run tests saying what tests they plan to run and when they'll send the results back. This would mean we know what to wait for. This needs to be documented publicly. I've not seen the new QA process being talked about publicly with the community yet either? This doesn't help with progress updates, not sure what we can do there. We're also seeing problems with the way ptest issues are being reported. Examples are bugs https://bugzilla.yoctoproject.org/show_bug.cgi?id=13238 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13239 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13240 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13241 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13242 . 13242 is puzzling as all the tests are passing (which is good) The test results reported here differ quite substantially to those from other builds, e.g. the report in: https://autobuilder.yocto.io/pub/non-release/20190326-8/testresults/testresult-report.txt which has one test failure for python3. Its unclear if the reported bugs are in QA locally run tests or tests run on the public autobuilder. The logs shown should probably include the high level ptest summary and the actual log output from the ptests. We spent quite some time in the triage meeting trying to work out where the problems may be and who should try and do something about them. Its certainly clear that we need to improve the reporting process though to include more information and to report things correctly. Also, we need to identify "timeout" tests. These are show in the main report with a T against them (e.g. strace, python3, lttng-tools). These should be specific bugs making it clear a timeout occurred with the test. Right now I don't think the QA process is flagging those up and it should.
for ptest bugs, https://bugzilla.yoctoproject.org/show_bug.cgi?id=13238 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13239 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13240 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13241 , https://bugzilla.yoctoproject.org/show_bug.cgi?id=13242 these were reported on the basis on results from ptest run by Intel QA team. But these results were found to be inconsistent and inaccurate in follow up investigations, hence these bugs are marked as "NOT A BUG". Final ptest bugs for 2.7 M3 RC1 are files as follows: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13253 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13254 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13249
Also, we need to identify "timeout" tests. These are show in the main report with a T against them (e.g. strace, python3, lttng-tools). These should be specific bugs making it clear a timeout occurred with the test. Right now I don't think the QA process is flagging those up and it should. QA team was not flagging "timeout" issues so far. We take it as an AR and start reporting timeout issues. For 2.7 M3 RC1, 3 timeout issues reported: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13255 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13256 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13257
(In reply to comment #0) > Now that we're running through the new QA process there are a few problems > we're running into. > > Stephen and myself have no visibility into who is running tests, what > progress is being made or when we can expect results (or which results to > block on). This can be resolved once we can upload test results on "yocto-testresults" repo. You can see what tests are completed and results are available. > > At a bare minimum I'd propose we need to have "ack" replies to the "QA > start" email from anyone who is planning to run tests saying what tests they > plan to run and when they'll send the results back. This would mean we know > what to wait for. > Agree on this. We can send you an email for our plan of QA cycle specifying what test we are going to run on which build.
Created attachment 4457 [details] 2.7 m3 rc1 report from intel and windriver
Sent the new QA process to yocto mailing list. Here was the link https://lists.yoctoproject.org/pipermail/yocto/2019-April/044661.html For new QA process trial run (2.7 m3 rc1), extra test results from intel and windriver team are compiled and sent to Richard for review. Attached the QA result report for extra test results from intel and windriver.
We will create new sub-bug to track unresolved activities then close the defect.
Documented the new QA process: https://wiki.yoctoproject.org/wiki/New_QA_process Attached a link at main QA page to the new QA process page: https://wiki.yoctoproject.org/wiki/QA#New_QA_Process
(In reply to comment #7) > Documented the new QA process: > https://wiki.yoctoproject.org/wiki/New_QA_process > > Attached a link at main QA page to the new QA process page: > https://wiki.yoctoproject.org/wiki/QA#New_QA_Process thank you
The resulttool is command line tool available inside poky/scripts, it was developed to manage test result files for both automated tests (eg. oeqa/runtime, oeqa/selftest, oeqa/sdk, etc) and manual tests (eg. oeqa/manual). It was developed as an alternative tool to replace test result storage and reporting using Testopia and Wiki Page. https://wiki.yoctoproject.org/wiki/Resulttool
QA wiki page has now been updated with the new process.