15364 2024-01-23 13:47:44 +0000 glibc/gcc test suite causes false positives in regression report 2026-05-28 14:02:23 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain 4.3 x86 Multiple NEW Medium+ normal 6.1 M3 1 ross.burton Harish.Sadineni meta.mr.watcher meta.watcher randy.macleod sundeep.kokkonda Yash.Shinde --- --- No (bug/feature does not impact docs) oldest_to_newest 97951 0 ross.burton 2024-01-23 13:47:44 +0000 The QA regression report has many false positives. Specifically if we look at the 4.3.2 report: http://downloads.yoctoproject.org/releases/yocto/yocto-4.3.2/testresults/testresult-regressions-report.txt We see many regressions of the form: ptestresult.gcc-g++-user.g++.dg/vect/pr21218.cc -mloongson-mmi -std=c++20 (test for excess errors): PASS -> No matching test result ptestresult.gcc-g++-user.g++.dg/vect/pr106841.cc -mloongson-mmi -std=c++98 (test for excess errors): PASS -> No matching test result ptestresult.gcc-user.gcc.dg/weak/weak-2.c scan-assembler weak[^ \t]*[ \t]_?ffoo1e: PASS -> No matching test result Whilst it's possible that the test suite removed a lot of test cases, my hunch is that the test report isn't being formatted correctly so the test case names are not accurate, so the regression report is basically comparing random lines of the log files. 99642 1 randy.macleod 2024-09-11 19:00:50 +0000 Harish Is this issue fixed now? If so please resolve and include a link to the fix. Thanks. 100064 2 randy.macleod 2024-10-30 17:24:56 +0000 Harish, Thanks for moving the bug to 5.2. Since this is a Medium+ importance bug, it should target a specific milestone and I picked M2 since we should improve the tests as soon as we can in part to have impressive results in the YP Summit presentation! ;-) 100859 3 randy.macleod 2025-02-06 15:15:48 +0000 Still happened in : https://downloads.yoctoproject.org/releases/yocto/yocto-5.1.2//testresults/testresult-regressions-report.txt Harish, is this still a problem? Do you have a link to a more recent log proving that the mangled logs are better? 102407 4 randy.macleod 2025-07-03 15:38:55 +0000 Still issues listed: https://downloads.yoctoproject.org/releases/yocto/yocto-5.2.1/testresults/testresult-regressions-report.txt Harish, is this defect being worked on for 5.3? Do you have an idea of how to proceed? If not please explain what you have done so far to try to reproduce the issue. 102420 5 Harish.Sadineni 2025-07-04 10:55:44 +0000 I’ve reviewed the regressions reported from GCC. The 20 regressions listed under ptestresult.gcc-libitm-user are not actual regressions. In the result log, these tests include the full build path in their output. Since the build paths differ between the two builds, the results appear different, but the tests themselves are unchanged. Some of the other reported regressions are caused by program timed out.: WARNING messages. These timeouts don’t occur consistently across builds, so the warnings are not reliable indicators of regressions. In the case of the glibc tests, the results were inconsistent between two consecutive builds. We need to investigate further to understand the cause of these deviations. 104862 6 Harish.Sadineni 2026-03-30 18:12:48 +0000 The false positives in the gcc-libitm test case results in regression report were fixed by the following commit: https://git.openembedded.org/openembedded-core/commit/?id=b91f6cec2b248209fcf7b81d7e32ba0f4f2b1ec9 Currently working on analyzing the remaining false postives.