Bug 15262

Summary: Improve patchtest testresult messages
Product: [Yocto Project Subprojects] Patchwork/Patchtest Reporter: Trevor Gamblin <tgamblin>
Component: PatchtestAssignee: Simone Weiß <simone.p.weiss>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: randy.macleod, simone.p.weiss
Version: 5.0   
Target Milestone: 5.0 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)

Description Trevor Gamblin 2023-10-27 11:51:41 UTC 
Patchtest provides a simple set of PASS/FAIL/SKIP results with names of tests that were run, and (if failed or skipped) a short explanation of why and what to do. This is useful but could still be more user-friendly, especially when it comes to the automated service providing feedback on the mailing list. A non-exhaustive list of possible improvements:

- Better test indicators (not just a reformatted version of the test names), including what the tests are looking for
- Better suggestions for failing patches
- Links to the contributor guide or other docs
- (If/when the test suite is expanded) Direct feedback on how variables, syntax, etc. are being misused
Comment 1 Trevor Gamblin 2024-02-15 19:58:41 UTC 
Something that recently came up was a desire to remove all of the unittest-specific text that gets printed with the results, e.g.:

----------------------------------------------------------------------
Ran 21 tests in 2.715s

OK

In theory, this should be catchable by redirecting the stderr output from unittest into a variable that is otherwise ignored, although that's kind of hacky. A better way would be to implement a custom test framework for patchtest, but that's obviously a lot more complicated. I've been wondering about pytest and whether that might be better too, but haven't had time to really look and see what the changes would involve.

There might be another way that I've missed...
Comment 2 Simone Weiß 2024-02-15 20:21:39 UTC 
pytest should ideally also be more performant, which could help with overhead added when  extending patchtests like the one mentioned here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15400#c1