We run ptest builds in every a-full build, but the regressions are only reported on milestones. This always results in a number of regressions being announced and a six week window of changes where the change could have happened. If the ptest autobuilder job performed the regression analysis itself, it could at least warn if any tests regressed and we'd only have a few patches to review.
The autobuilder does track the results into a git repository so we should be able to interrogate that git repo with a script to find out when regressions occur even if the autobuilder doesn't do this directly/automatically.
To find out what regressions occur, we can use 'resuttool regression' feature. For more details, please see https://wiki.yoctoproject.org/wiki/Resulttool#regression-git_example. I don't see a need of a separate script for ptest regression. If anything else, please help to share more info.
I agree we should try and use this functionality but I would like to see this automated so that the autobuilder automatically generates a reliable list of regressions. It does have some code to do this but the reports don't look reliable and it doesn't have a concept of understanding the last release or milestone to compare against.
We need someone to work on this bug soon so we can tell when ptests regress.
Alexis is working on this so in progress state.
Some good progress made but moving to M3 since it's not done yet.
A first integration to allow checks against previous release has been done, but after testing within a real release, some issues appear : - 4.2_M2 regression report is pretty huge and many generation errors are loggued in autobuilder - https://autobuilder.yocto.io/pub/releases/yocto-4.2_M2.rc2/testresults/testresult-regressions-report.txt - https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4844/steps/29/logs/stdio - some known failures/regression are not properly reported - https://lore.kernel.org/yocto/01010185f1a4b508-6e5da1df-b525-4798-ad94-58ea11a96b76-000000@us-west-2.amazonses.com/T/#t - we need to display in the regression report against which revision it has been generated I will look at all those points.
Good progress mostly done but we need to do some analysis of the 4.2 regression report so moving to 4.3 M1.
Developed and integrated. Still needs some real testing (last test was 4.3_M1 but another issue in test results prevented us from evaluating result)
This failed to generate a report for 4.3 M3. The build step "passed" (worryingly) but there is an error for the regression report: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5865/steps/29/logs/stdio
Issues which have prevented report 4.3_M3 from being generated have been fixed. Draft report generated after fix shows the following need: - add a max number of regression for tests pairs generating a lot of errors (e.g. : display only the 30 first regressions), but keeping count of total regression for current tests pair.
I think this feature base has been set and his now running for each release through the autobuilder, so I would like to mark this as resolved. There are still some possible adjustments but it should now be mentioned as improvements in dedicated issues. If no one object, I will close this in a few days