16376 2026-07-28 08:00:56 +0000 AB-INT: build system architecture reproducibility issue 2026-10-19 07:36:02 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest unspecified All Multiple NEW AB-INT Medium+ normal 6.1 M3 1 mathieu.dubois-briand unassigned ccasciato randy.macleod richard.purdie sundeep.kokkonda SunilKumar.Dora --- --- No (bug/feature does not impact docs) oldest_to_newest 106353 0 mathieu.dubois-briand 2026-07-28 08:00:56 +0000 I've been having these reproducibility issues on my branch in the past few days. It does happen with a few packages, particularly with diffutils and m4. Here is the worst example I can find: AssertionError: The following deb packages are different and not in exclusion list: /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/diffutils-ptest_3.12-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/gcc-runtime-src_16.1.0-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/m4-ptest_1.4.21-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/openssl-ptest_3.5.7-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/python3-core_3.14.6-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/python3-misc_3.14.6-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/python3-tests_3.14.6-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/sed-ptest_4.10-r0_amd64.deb Corresponding diffoscope output can be found here: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-96wb10by/packages/diff-html/ The exact difference varies from one recipe to another, but basically, it all goes around the build triplet. E.g. "build·=·aarch64-unknown-linux-gnu" on one side and "build·=·x86_64-pc-linux-gnu" on another. This is probably because this data hit the cache first by being built on an arm64 host. This is rare, as x86 hosts are generally quicker, but it might happen. Then we run the reproducibility test, always on x86, and we can observe a difference. Relevant message on the mailing list: https://lore.kernel.org/openembedded-core/8773d1dc4d109da78fa0ffd91bc5667a5f3544d2.camel@linuxfoundation.org/T/#u 106355 1 mathieu.dubois-briand 2026-07-28 08:02:31 +0000 reproducible alma10-vk-1 master&mathieu/master-next-test3 completed at 2026-07-26 11:02:45+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-96wb10by https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4487/steps/13/logs/stdio reproducible debian12-vk-5 mathieu/master-next-test1&master&mathieu/master-next completed at 2026-07-26 07:35:28+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-q_v8oqs2 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4482/steps/13/logs/stdio reproducible stream9-vk-1 mathieu/master-next-test2&master&mathieu/master-next completed at 2026-07-26 08:05:29+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-0895cxdf https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4483/steps/14/logs/stdio reproducible alma9-vk-2 master&mathieu/master-next-test3&mathieu/master-next completed at 2026-07-26 07:00:05+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-vco1pkzk https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4484/steps/14/logs/stdio reproducible stream9-vk-1 mathieu/master-next-test1&master completed at 2026-07-26 10:46:28+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-8vf57bvb https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4485/steps/14/logs/stdio reproducible opensuse160-vk-1 mathieu/master-next-tests&master completed at 2026-07-26 13:44:04+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-htbjbdb4 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4493/steps/13/logs/stdio reproducible ubuntu2404-vk-2 mathieu/master-next-test2&master completed at 2026-07-26 16:57:58+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-9wb1l3t4 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4498/steps/13/logs/stdio reproducible debian12-vk-9 mathieu/master-next completed at 2026-07-25 16:41:34+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260725-qsmm1lvf https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4475/steps/13/logs/stdio reproducible ubuntu2404-vk-1 mathieu/master-next completed at 2026-07-25 18:21:43+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260725-e87zna44 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4476/steps/13/logs/stdio reproducible debian12-vk-1 mathieu/master-next completed at 2026-07-25 20:04:03+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260725-8g5b5qtp https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4477/steps/13/logs/stdio 106371 2 richard.purdie 2026-07-29 20:27:02 +0000 I added a new target to the autobuilder which can test and reproduce this: https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/1/steps/13/logs/stdio leading to: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260729-j6j_xpuw/packages/diff-html/ 106378 3 mathieu.dubois-briand 2026-07-30 07:58:54 +0000 reproducible-arm ubuntu2404-vk-arm1 master completed at 2026-07-30 06:25:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/1/steps/13/logs/stdio reproducible-cross opensuse160-vk-1 master completed at 2026-07-29 20:09:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/1/steps/13/logs/stdio 106394 4 richard.purdie 2026-07-30 15:54:08 +0000 Looking at the diff output above (and for the other new autobuilder jobs that failed), we can see there are ptest/Makefile issues as well as a problematic looking kernel package output issue which varies depending on the buildhost architecture. We need to split these into different categories and maybe create separate bugs for any issues which are going to need in depth specific work. Help in fixing any of the individual issues is welcome and hopefully we can parallel process the issues to resolve things overall. 106402 5 mathieu.dubois-briand 2026-07-31 12:12:36 +0000 reproducible-arm-cross stream10-vk-arm1 master completed at 2026-07-30 13:23:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/1/steps/13/logs/stdio 106487 6 mathieu.dubois-briand 2026-08-11 05:36:47 +0000 reproducible-arm ubuntu2404-vk-arm1 master completed at 2026-08-11 04:11:28+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/2/steps/15/logs/stdio reproducible-cross fedora44-vk-1 master completed at 2026-08-11 03:02:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/2/steps/15/logs/stdio 106500 7 mathieu.dubois-briand 2026-08-12 06:21:50 +0000 reproducible-arm-cross stream10-vk-arm1 master completed at 2026-08-11 15:27:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/2/steps/15/logs/stdio 106639 8 randy.macleod 2026-08-25 15:39:44 +0000 Richard explained in the YP Tech call, that many if not most of these issues are due to an intermittent reproducibility issue with rust. The issue would be more frequent except that the x86 builders are more common in the AB and they are generally faster. Therefore, when there is an update, an x86 worker generally finishes first and pushes it's update into the sstate-cache. Since the reproducibility tests only run (currently) on x86, there usually isn't a problem. 106723 9 SunilKumar.Dora 2026-09-03 18:45:45 +0000 While debugging the rust packages from the reproducible-cross runs I noticed the git/git-dbg failures have a separate cause: git records the build host's `uname -m` in the binary. In the diffoscope report from builder 120 build #1 (2026-07-29, https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260729-j6j_xpuw/packages/diff-html/), git's .rodata shows: -0x001b1780 6769742d 00000000 61617263 68363400 git-....aarch64. +0x001b1780 6769742d 00000000 7838365f 36340000 git-....x86_64.. git's Makefile defaults HOST_CPU to the build machine's `uname -m` and compiles it in as GIT_HOST_CPU, which is the "cpu:" line of `git version --build-options`: https://github.com/git/git/blob/v2.55.0/Makefile#L1686 Upstream documents HOST_CPU as the value to set when cross compiling, but meta/recipes-devtools/git/git_2.55.0.bb never sets it, so we always get the fallback. And since "aarch64" and "x86_64" are of different lengths, every section offset after that string shifts. git/git-dbg differ in all the runs I checked (builders 119/120/121, before and after the rust SVH fix, on both x86 and arm hosts), so this is independent of the rust work. The fix should be a one-liner in git_2.55.0.bb: EXTRA_OEMAKE += "HOST_CPU=${TARGET_ARCH}" TARGET_ARCH is BUILD_ARCH for native and SDK_ARCH for nativesdk, so one line should cover all the variants. I'm verifying with a local qemuarm64 build and will send the patch to the list once confirmed. 106734 10 richard.purdie 2026-09-04 16:43:06 +0000 That looks like a good catch Sunil! We know there are some other issues, it just seemed easiest to filter out the rust ones first so we could more easily see what was left! 106838 11 SunilKumar.Dora 2026-09-15 14:44:41 +0000 Fixes for the rust and git parts of this are now on the list. Rust series: ------------ https://lists.openembedded.org/g/openembedded-core/message/245843 https://lists.openembedded.org/g/openembedded-core/message/245842 https://lists.openembedded.org/g/openembedded-core/message/245844 https://lists.openembedded.org/g/openembedded-core/message/245845 https://lists.openembedded.org/g/openembedded-core/message/245846 https://lists.openembedded.org/g/openembedded-core/message/245847 Patchwork series: https://patchwork.yoctoproject.org/project/oe-core/list/?series=51088 Git fix, sent separately: https://lists.openembedded.org/g/openembedded-core/message/245837 Tested on the autobuilder from a contrib branch: The x86 reproducible builder came back green with zero differing packages, the rust test suite passed, and on both arm-hosted reproducible builders rust, rust-dbg, rust-rustdoc and libstd-rs-dev are gone from the failing list. 106914 12 antonin.godard 2026-09-23 14:55:13 +0000 reproducible debian12-vk-3 agodard/master-next-bis&agodard/master-next completed at 2026-09-15 19:32:10+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260915-vm462bt0 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4991/steps/15/logs/stdio reproducible-arm alma10-vk-arm1 master&master-next completed at 2026-09-17 21:48:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/10/steps/15/logs/stdio reproducible-arm-cross ubuntu2404-vk-arm2 master&master-next completed at 2026-09-17 22:42:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/10/steps/15/logs/stdio reproducible-cross alma8-vk-1 master&master-next completed at 2026-09-17 11:44:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/9/steps/16/logs/stdio reproducible-cross opensuse156-vk-1 master&master-next completed at 2026-09-17 23:10:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/10/steps/16/logs/stdio reproducible-cross debian12-vk-4 master&master-next completed at 2026-09-18 23:42:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/11/steps/15/logs/stdio reproducible-arm-cross alma10-vk-arm1 master&master-next completed at 2026-09-19 11:12:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/11/steps/15/logs/stdio reproducible-arm ubuntu2404-vk-arm2 master&master-next completed at 2026-09-19 09:34:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/11/steps/15/logs/stdio reproducible opensuse160-vk-1 master&master-next completed at 2026-09-19 21:44:06+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260919-l3njlk0p https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/5022/steps/15/logs/stdio 106921 13 antonin.godard 2026-09-24 07:49:58 +0000 reproducible-arm-cross alma10-vk-arm1 master&master-next completed at 2026-09-24 02:20:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/12/steps/15/logs/stdio reproducible-arm debian13-vk-arm1 master&master-next completed at 2026-09-24 01:22:45+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/12/steps/15/logs/stdio 106943 14 SunilKumar.Dora 2026-09-27 04:36:41 +0000 I am working with cargo upstream to fix remain rust packages differ we are seeing on Autobuilder. 106945 15 SunilKumar.Dora 2026-09-27 14:14:02 +0000 A quick status on the ARM host failures I'm working on: libopkg1/opkg-dbg: This one goes back to opkg's switch from autotools to CMake last year. autotools used host_cpu for the built-in default arch, which is the CPU opkg runs on. The CMake port uses CMAKE_HOST_SYSTEM_PROCESSOR instead, and that's the build machine. So the x86 worker puts "x86_64" into the package and the ARM worker puts aarch64". Nobody noticed before since: our images always ship arch.conf, but the ARM workers catch it. I've sent a fix to the opkg list (bug 15784): https://lists.yoctoproject.org/g/opkg/message/205 curl-dev and ruby: The build machine's triplet ends up in files we ship (the configure line in curl-config, the build_* entries in ruby's rbconfig.rb). The fix strips them in the recipes. It's on the autobuilder(Testing on contrib branch) now and I'll send it once that's clean. rust-rustdoc: cargo mixes host-only dependencies (build scripts, proc-macros) into the target crate's -C metadata. Upstream PR: https://github.com/rust-lang/cargo/pull/17522 Still working on the rest and will send the oe-core patches as these get reviewed. 106946 16 SunilKumar.Dora 2026-09-27 15:56:33 +0000 curl and ruby are now sent to oe-core, after the autobuilder runs came back clean for curl-dev and ruby on the ARM builders: - https://lists.openembedded.org/g/openembedded-core/message/246688 - https://lists.openembedded.org/g/openembedded-core/message/246689 106949 17 mathieu.dubois-briand 2026-09-28 11:53:10 +0000 reproducible-arm ubuntu2404-vk-arm1 master-next&master completed at 2026-09-24 22:02:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/13/steps/15/logs/stdio reproducible-arm-cross stream10-vk-arm1 master-next&master completed at 2026-09-24 21:59:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/13/steps/15/logs/stdio reproducible-cross rocky9-vk-2 master-next&master completed at 2026-09-24 19:03:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/13/steps/16/logs/stdio 106966 18 SunilKumar.Dora 2026-09-30 12:00:33 +0000 opkg - sent to oe-core. https://lists.openembedded.org/g/openembedded-core/message/246961 107108 19 mathieu.dubois-briand 2026-10-19 07:36:02 +0000 reproducible-arm ubuntu2404-vk-arm2 master completed at 2026-10-17 23:26:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/16/steps/15/logs/stdio reproducible-arm-cross debian13-vk-arm1 master completed at 2026-10-17 22:33:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/16/steps/15/logs/stdio reproducible-cross rocky9-vk-2 master completed at 2026-10-17 21:01:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/15/steps/16/logs/stdio