Bug 15364

Summary:	glibc/gcc test suite causes false positives in regression report
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Ross Burton <ross.burton>
Component:	devtools / tool chain	Assignee:	Harish Sadineni <Harish.Sadineni>
Status:	NEW ---	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	meta.mr.watcher, meta.watcher, randy.macleod, sundeep.kokkonda, Yash.Shinde
Version:	4.3
Target Milestone:	6.1 M3
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Ross Burton 2024-01-23 13:47:44 UTC

The QA regression report has many false positives.  Specifically if we look at the 4.3.2 report:

http://downloads.yoctoproject.org/releases/yocto/yocto-4.3.2/testresults/testresult-regressions-report.txt

We see many regressions of the form:

    ptestresult.gcc-g++-user.g++.dg/vect/pr21218.cc  -mloongson-mmi -std=c++20 (test for excess errors): PASS -> No matching test result

    ptestresult.gcc-g++-user.g++.dg/vect/pr106841.cc  -mloongson-mmi -std=c++98 (test for excess errors): PASS -> No matching test result

    ptestresult.gcc-user.gcc.dg/weak/weak-2.c scan-assembler weak[^ \t]*[ \t]_?ffoo1e: PASS -> No matching test result


Whilst it's possible that the test suite removed a lot of test cases, my hunch is that the test report isn't being formatted correctly so the test case names are not accurate, so the regression report is basically comparing random lines of the log files.

Comment 1 Randy MacLeod 2024-09-11 19:00:50 UTC

Harish 
Is this issue fixed now? If so please resolve and include a link to the fix.
Thanks.

Comment 2 Randy MacLeod 2024-10-30 17:24:56 UTC

Harish,
Thanks for moving the bug to 5.2. 
Since this is a Medium+ importance bug, it should target a specific milestone and I picked M2 since we should improve the tests as soon as we can in part to have impressive results in the YP Summit presentation! ;-)

Comment 3 Randy MacLeod 2025-02-06 15:15:48 UTC

Still happened in :

https://downloads.yoctoproject.org/releases/yocto/yocto-5.1.2//testresults/testresult-regressions-report.txt

Harish, is this still a problem? 
Do you have a link to a more recent log proving that the mangled logs are better?

Comment 4 Randy MacLeod 2025-07-03 15:38:55 UTC

Still issues listed:
   https://downloads.yoctoproject.org/releases/yocto/yocto-5.2.1/testresults/testresult-regressions-report.txt

Harish, is this defect being worked on for 5.3?
Do you have an idea of how to proceed? If not please explain what you have done so far to try to reproduce the issue.

Comment 5 Harish Sadineni 2025-07-04 10:55:44 UTC

I’ve reviewed the regressions reported from GCC.
 
The 20 regressions listed under ptestresult.gcc-libitm-user are not actual regressions. In the result log, these tests include the full build path in their output. Since the build paths differ between the two builds, the results appear different, but the tests themselves are unchanged.
 
Some of the other reported regressions are caused by program timed out.: WARNING messages. These timeouts don’t occur consistently across builds, so the warnings are not reliable indicators of regressions.
 
In the case of the glibc tests, the results were inconsistent between two consecutive builds. We need to investigate further to understand the cause of these deviations.

Comment 6 Harish Sadineni 2026-03-30 18:12:48 UTC

The false positives in the gcc-libitm test case results in regression report were fixed by the following commit:
https://git.openembedded.org/openembedded-core/commit/?id=b91f6cec2b248209fcf7b81d7e32ba0f4f2b1ec9
 
Currently working on analyzing the remaining false postives.