A bit surprised we didn't open a ticket earlier, but I didn't find any. We have some reproducibility issues on perf since a few weeks ago. Diffoscope of the last failure: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251005-7z0pvbd7/packages/diff-html/
reproducible fedora40-vk-4 mathieu/master-next completed at 2025-10-03 11:33:17+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2629/steps/13/logs/stdio reproducible rocky8-vk-1 mathieu/master-next completed at 2025-10-04 12:43:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2638/steps/14/logs/stdio reproducible rocky9-vk-1 master-next completed at 2025-10-04 17:45:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2640/steps/13/logs/stdio reproducible debian12-vk-5 mathieu/master-next completed at 2025-10-05 15:01:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2646/steps/13/logs/stdio
reproducible opensuse155-vk-1 mathieu/master-next completed at 2025-10-07 14:49:07+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2661/steps/14/logs/stdio
Mathieu to review the diffoscope output, share a list of diffoscope links and summarize.
Adding links to diffoscope output of above builds. To my eyes they all look similar, except for the comments below. But of course, maybe I'm not looking where I should be. 2629: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251003-sc_8e8re/packages/diff-html/ 2638: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251004-jyj_i26v/packages/diff-html/ 2661: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251007-8nsi9bod/packages/diff-html/ This one has a lot of diffoscope errors, such as: error from `readelf --wide --decompress --string-dump=.debug_line_str {}`: readelf: Error: Unable to find program interpreter name I'm not sure why we have these errors, but I believe they explain all differences between this diff and other diffs. 2640: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251004-2bkiobql/packages/diff-html/ 2646: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251005-7z0pvbd7/packages/diff-html/ There is also some linux-firmware diffs in there, but this is for an unrelated issue.
On opensuse155-vk-1, in one of the reproducibility builds that failed, I ran a new perf build. In that build, the build now matched the output from sstate. This means the issue we're looking for is not host specific and instead is more likely some kind of race.
reproducible ubuntu2204-vk-4 mathieu/master-next completed at 2025-10-08 14:55:15+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251008-5nrkq_kk https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2669/steps/13/logs/stdio reproducible debian11-vk-1 mathieu/master-next completed at 2025-10-08 20:15:56+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251008-4ngzd46v https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2672/steps/13/logs/stdio reproducible debian11-vk-1 mathieu/master-next completed at 2025-10-10 08:08:15+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251010-icw5k6pi https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2690/steps/13/logs/stdio reproducible alma9-vk-2 mathieu/master-next completed at 2025-10-11 16:45:13+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251011-e79ey_zg https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2694/steps/13/logs/stdio reproducible ubuntu2504-vk-1 mathieu/master-next completed at 2025-10-12 10:22:56+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251012-erarnb7o https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2696/steps/13/logs/stdio reproducible debian12-vk-7 mathieu/master-next completed at 2025-10-12 20:37:46+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251012-3e8rcpqj https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2697/steps/13/logs/stdio reproducible ubuntu2504-vk-1 mathieu/master-next completed at 2025-10-13 08:09:47+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251013-ykd1xzn_ https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2703/steps/13/logs/stdio
reproducible fedora41-vk-1 master completed at 2025-10-14 03:01:18+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251014-o68feedu https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2710/steps/13/logs/stdio
reproducible debian11-vk-3 master completed at 2025-10-15 02:28:09+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-mq50aegy https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2719/steps/13/logs/stdio reproducible rocky9-vk-3 mathieu/master-next completed at 2025-10-15 11:21:36+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-t2_kk44w https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2721/steps/13/logs/stdio reproducible ubuntu2204-vk-3 mathieu/master-next completed at 2025-10-15 13:49:12+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-n5gu_ll_ https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2722/steps/13/logs/stdio
reproducible alma8-vk-2 master-next completed at 2025-10-15 20:42:02+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20251015-liktjj43 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/2725/steps/14/logs/stdio
*** Bug 15978 has been marked as a duplicate of this bug. ***
Hopefully this has been fixed with "perf: Tweak reproducibility fix" (oe-core 647589fab1fb73a985cbba4c1bec2004831508ca).
Hasn't repeated, so I think we're good.