15990 2025-10-06 12:18:18 +0000 AB-INT: perf reproducibility issues 2025-10-23 15:02:16 +0000 1 1 10 QA/Testing Functional (self) Testing general unspecified x86 Multiple RESOLVED FIXED AB-INT High normal 5.3 M3 1 mathieu.dubois-briand bruce.ashfield mathieu.dubois-briand raj.khem randy.macleod richard.purdie ross.burton --- --- No (bug/feature does not impact docs) oldest_to_newest 102886 0 mathieu.dubois-briand 2025-10-06 12:18:18 +0000 A bit surprised we didn't open a ticket earlier, but I didn't find any. We have some reproducibility issues on perf since a few weeks ago. Diffoscope of the last failure: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251005-7z0pvbd7/packages/diff-html/ 102887 1 mathieu.dubois-briand 2025-10-06 12:24:17 +0000 reproducible fedora40-vk-4 mathieu/master-next completed at 2025-10-03 11:33:17+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2629/steps/13/logs/stdio reproducible rocky8-vk-1 mathieu/master-next completed at 2025-10-04 12:43:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2638/steps/14/logs/stdio reproducible rocky9-vk-1 master-next completed at 2025-10-04 17:45:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2640/steps/13/logs/stdio reproducible debian12-vk-5 mathieu/master-next completed at 2025-10-05 15:01:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2646/steps/13/logs/stdio 102889 2 mathieu.dubois-briand 2025-10-07 15:20:19 +0000 reproducible opensuse155-vk-1 mathieu/master-next completed at 2025-10-07 14:49:07+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2661/steps/14/logs/stdio 102894 3 randy.macleod 2025-10-09 14:37:18 +0000 Mathieu to review the diffoscope output, share a list of diffoscope links and summarize. 102897 4 mathieu.dubois-briand 2025-10-09 15:28:59 +0000 Adding links to diffoscope output of above builds. To my eyes they all look similar, except for the comments below. But of course, maybe I'm not looking where I should be. 2629: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251003-sc_8e8re/packages/diff-html/ 2638: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251004-jyj_i26v/packages/diff-html/ 2661: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251007-8nsi9bod/packages/diff-html/ This one has a lot of diffoscope errors, such as: error from `readelf --wide --decompress --string-dump=.debug_line_str {}`: readelf: Error: Unable to find program interpreter name I'm not sure why we have these errors, but I believe they explain all differences between this diff and other diffs. 2640: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251004-2bkiobql/packages/diff-html/ 2646: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251005-7z0pvbd7/packages/diff-html/ There is also some linux-firmware diffs in there, but this is for an unrelated issue. 102898 5 richard.purdie 2025-10-09 15:41:34 +0000 On opensuse155-vk-1, in one of the reproducibility builds that failed, I ran a new perf build. In that build, the build now matched the output from sstate. This means the issue we're looking for is not host specific and instead is more likely some kind of race. 102923 6 mathieu.dubois-briand 2025-10-13 12:10:50 +0000 reproducible ubuntu2204-vk-4 mathieu/master-next completed at 2025-10-08 14:55:15+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251008-5nrkq_kk https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2669/steps/13/logs/stdio reproducible debian11-vk-1 mathieu/master-next completed at 2025-10-08 20:15:56+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251008-4ngzd46v https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2672/steps/13/logs/stdio reproducible debian11-vk-1 mathieu/master-next completed at 2025-10-10 08:08:15+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251010-icw5k6pi https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2690/steps/13/logs/stdio reproducible alma9-vk-2 mathieu/master-next completed at 2025-10-11 16:45:13+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251011-e79ey_zg https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2694/steps/13/logs/stdio reproducible ubuntu2504-vk-1 mathieu/master-next completed at 2025-10-12 10:22:56+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251012-erarnb7o https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2696/steps/13/logs/stdio reproducible debian12-vk-7 mathieu/master-next completed at 2025-10-12 20:37:46+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251012-3e8rcpqj https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2697/steps/13/logs/stdio reproducible ubuntu2504-vk-1 mathieu/master-next completed at 2025-10-13 08:09:47+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251013-ykd1xzn_ https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2703/steps/13/logs/stdio 102930 7 mathieu.dubois-briand 2025-10-14 13:33:24 +0000 reproducible fedora41-vk-1 master completed at 2025-10-14 03:01:18+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251014-o68feedu https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2710/steps/13/logs/stdio 102943 8 mathieu.dubois-briand 2025-10-15 15:25:14 +0000 reproducible debian11-vk-3 master completed at 2025-10-15 02:28:09+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-mq50aegy https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2719/steps/13/logs/stdio reproducible rocky9-vk-3 mathieu/master-next completed at 2025-10-15 11:21:36+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-t2_kk44w https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2721/steps/13/logs/stdio reproducible ubuntu2204-vk-3 mathieu/master-next completed at 2025-10-15 13:49:12+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-n5gu_ll_ https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2722/steps/13/logs/stdio 102949 9 mathieu.dubois-briand 2025-10-16 14:19:18 +0000 reproducible alma8-vk-2 master-next completed at 2025-10-15 20:42:02+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-liktjj43 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2725/steps/14/logs/stdio 102973 10 randy.macleod 2025-10-16 15:18:37 +0000 *** Bug 15978 has been marked as a duplicate of this bug. *** 102990 11 ross.burton 2025-10-17 15:35:09 +0000 Hopefully this has been fixed with "perf: Tweak reproducibility fix" (oe-core 647589fab1fb73a985cbba4c1bec2004831508ca). 103048 12 ross.burton 2025-10-23 15:02:16 +0000 Hasn't repeated, so I think we're good.