14065 2020-09-28 09:40:28 +0000 Automated ptest regression testing 2024-01-08 08:24:27 +0000 1 1 10 QA/Testing Package Testing (ptest) ptest unspecified All Multiple RESOLVED FIXED High major 5.0 M2 1 ross.burton alexis.lothore alexandre.belloni chi.xu randy.macleod richard.purdie tgamblin tim.orling --- --- No (bug/feature does not impact docs) oldest_to_newest 88251 0 ross.burton 2020-09-28 09:40:28 +0000 We run ptest builds in every a-full build, but the regressions are only reported on milestones. This always results in a number of regressions being announced and a six week window of changes where the change could have happened. If the ptest autobuilder job performed the regression analysis itself, it could at least warn if any tests regressed and we'd only have a few patches to review. 88282 1 richard.purdie 2020-10-01 07:51:44 +0000 The autobuilder does track the results into a git repository so we should be able to interrogate that git repo with a script to find out when regressions occur even if the autobuilder doesn't do this directly/automatically. 89020 2 sangeeta.jain 2021-01-26 02:25:32 +0000 To find out what regressions occur, we can use 'resuttool regression' feature. For more details, please see https://wiki.yoctoproject.org/wiki/Resulttool#regression-git_example. I don't see a need of a separate script for ptest regression. If anything else, please help to share more info. 89065 3 richard.purdie 2021-01-28 15:54:13 +0000 I agree we should try and use this functionality but I would like to see this automated so that the autobuilder automatically generates a reliable list of regressions. It does have some code to do this but the reports don't look reliable and it doesn't have a concept of understanding the last release or milestone to compare against. 93210 4 randy.macleod 2022-04-28 15:17:55 +0000 We need someone to work on this bug soon so we can tell when ptests regress. 94705 5 randy.macleod 2023-01-12 15:56:12 +0000 Alexis is working on this so in progress state. 94813 6 randy.macleod 2023-01-26 15:53:07 +0000 Some good progress made but moving to M3 since it's not done yet. 94824 7 alexis.lothore 2023-01-27 14:22:51 +0000 A first integration to allow checks against previous release has been done, but after testing within a real release, some issues appear : - 4.2_M2 regression report is pretty huge and many generation errors are loggued in autobuilder - https://autobuilder.yocto.io/pub/releases/yocto-4.2_M2.rc2/testresults/testresult-regressions-report.txt - https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4844/steps/29/logs/stdio - some known failures/regression are not properly reported - https://lore.kernel.org/yocto/01010185f1a4b508-6e5da1df-b525-4798-ad94-58ea11a96b76-000000@us-west-2.amazonses.com/T/#t - we need to display in the regression report against which revision it has been generated I will look at all those points. 95309 8 randy.macleod 2023-04-06 14:56:02 +0000 Good progress mostly done but we need to do some analysis of the 4.2 regression report so moving to 4.3 M1. 95714 9 alexis.lothore 2023-06-14 12:29:36 +0000 Developed and integrated. Still needs some real testing (last test was 4.3_M1 but another issue in test results prevented us from evaluating result) 96411 10 richard.purdie 2023-09-14 15:04:25 +0000 This failed to generate a report for 4.3 M3. The build step "passed" (worryingly) but there is an error for the regression report: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5865/steps/29/logs/stdio 96676 11 alexis.lothore 2023-10-12 07:08:40 +0000 Issues which have prevented report 4.3_M3 from being generated have been fixed. Draft report generated after fix shows the following need: - add a max number of regression for tests pairs generating a lot of errors (e.g. : display only the 30 first regressions), but keeping count of total regression for current tests pair. 97740 12 alexis.lothore 2024-01-05 09:07:28 +0000 I think this feature base has been set and his now running for each release through the autobuilder, so I would like to mark this as resolved. There are still some possible adjustments but it should now be mentioned as improvements in dedicated issues. If no one object, I will close this in a few days