| Summary: | AB-INT: build system architecture reproducibility issue | ||
|---|---|---|---|
| Product: | [QA/Testing] Functional (self) Testing | Reporter: | Mathieu Dubois-Briand <mathieu.dubois-briand> |
| Component: | oe-selftest | Assignee: | Unassigned <unassigned> |
| Status: | NEW --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | ccasciato, randy.macleod, richard.purdie, sundeep.kokkonda, SunilKumar.Dora |
| Version: | unspecified | ||
| Target Milestone: | 6.1 M3 | ||
| Hardware: | All | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Mathieu Dubois-Briand
2026-07-28 08:00:56 UTC
reproducible alma10-vk-1 master&mathieu/master-next-test3 completed at 2026-07-26 11:02:45+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-96wb10by https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4487/steps/13/logs/stdio reproducible debian12-vk-5 mathieu/master-next-test1&master&mathieu/master-next completed at 2026-07-26 07:35:28+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-q_v8oqs2 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4482/steps/13/logs/stdio reproducible stream9-vk-1 mathieu/master-next-test2&master&mathieu/master-next completed at 2026-07-26 08:05:29+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-0895cxdf https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4483/steps/14/logs/stdio reproducible alma9-vk-2 master&mathieu/master-next-test3&mathieu/master-next completed at 2026-07-26 07:00:05+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-vco1pkzk https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4484/steps/14/logs/stdio reproducible stream9-vk-1 mathieu/master-next-test1&master completed at 2026-07-26 10:46:28+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-8vf57bvb https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4485/steps/14/logs/stdio reproducible opensuse160-vk-1 mathieu/master-next-tests&master completed at 2026-07-26 13:44:04+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-htbjbdb4 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4493/steps/13/logs/stdio reproducible ubuntu2404-vk-2 mathieu/master-next-test2&master completed at 2026-07-26 16:57:58+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260726-9wb1l3t4 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4498/steps/13/logs/stdio reproducible debian12-vk-9 mathieu/master-next completed at 2026-07-25 16:41:34+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260725-qsmm1lvf https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4475/steps/13/logs/stdio reproducible ubuntu2404-vk-1 mathieu/master-next completed at 2026-07-25 18:21:43+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260725-e87zna44 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4476/steps/13/logs/stdio reproducible debian12-vk-1 mathieu/master-next completed at 2026-07-25 20:04:03+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260725-8g5b5qtp https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4477/steps/13/logs/stdio I added a new target to the autobuilder which can test and reproduce this: https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/1/steps/13/logs/stdio leading to: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260729-j6j_xpuw/packages/diff-html/ reproducible-arm ubuntu2404-vk-arm1 master completed at 2026-07-30 06:25:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/1/steps/13/logs/stdio reproducible-cross opensuse160-vk-1 master completed at 2026-07-29 20:09:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/1/steps/13/logs/stdio Looking at the diff output above (and for the other new autobuilder jobs that failed), we can see there are ptest/Makefile issues as well as a problematic looking kernel package output issue which varies depending on the buildhost architecture. We need to split these into different categories and maybe create separate bugs for any issues which are going to need in depth specific work. Help in fixing any of the individual issues is welcome and hopefully we can parallel process the issues to resolve things overall. reproducible-arm-cross stream10-vk-arm1 master completed at 2026-07-30 13:23:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/1/steps/13/logs/stdio reproducible-arm ubuntu2404-vk-arm1 master completed at 2026-08-11 04:11:28+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/2/steps/15/logs/stdio reproducible-cross fedora44-vk-1 master completed at 2026-08-11 03:02:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/2/steps/15/logs/stdio reproducible-arm-cross stream10-vk-arm1 master completed at 2026-08-11 15:27:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/2/steps/15/logs/stdio Richard explained in the YP Tech call, that many if not most of these issues are due to an intermittent reproducibility issue with rust. The issue would be more frequent except that the x86 builders are more common in the AB and they are generally faster. Therefore, when there is an update, an x86 worker generally finishes first and pushes it's update into the sstate-cache. Since the reproducibility tests only run (currently) on x86, there usually isn't a problem. While debugging the rust packages from the reproducible-cross runs I noticed the git/git-dbg failures have a separate cause: git records the build host's `uname -m` in the binary. In the diffoscope report from builder 120 build #1 (2026-07-29, https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260729-j6j_xpuw/packages/diff-html/), git's .rodata shows: -0x001b1780 6769742d 00000000 61617263 68363400 git-....aarch64. +0x001b1780 6769742d 00000000 7838365f 36340000 git-....x86_64.. git's Makefile defaults HOST_CPU to the build machine's `uname -m` and compiles it in as GIT_HOST_CPU, which is the "cpu:" line of `git version --build-options`: https://github.com/git/git/blob/v2.55.0/Makefile#L1686 Upstream documents HOST_CPU as the value to set when cross compiling, but meta/recipes-devtools/git/git_2.55.0.bb never sets it, so we always get the fallback. And since "aarch64" and "x86_64" are of different lengths, every section offset after that string shifts. git/git-dbg differ in all the runs I checked (builders 119/120/121, before and after the rust SVH fix, on both x86 and arm hosts), so this is independent of the rust work. The fix should be a one-liner in git_2.55.0.bb: EXTRA_OEMAKE += "HOST_CPU=${TARGET_ARCH}" TARGET_ARCH is BUILD_ARCH for native and SDK_ARCH for nativesdk, so one line should cover all the variants. I'm verifying with a local qemuarm64 build and will send the patch to the list once confirmed. That looks like a good catch Sunil! We know there are some other issues, it just seemed easiest to filter out the rust ones first so we could more easily see what was left! Fixes for the rust and git parts of this are now on the list. Rust series: ------------ https://lists.openembedded.org/g/openembedded-core/message/245843 https://lists.openembedded.org/g/openembedded-core/message/245842 https://lists.openembedded.org/g/openembedded-core/message/245844 https://lists.openembedded.org/g/openembedded-core/message/245845 https://lists.openembedded.org/g/openembedded-core/message/245846 https://lists.openembedded.org/g/openembedded-core/message/245847 Patchwork series: https://patchwork.yoctoproject.org/project/oe-core/list/?series=51088 Git fix, sent separately: https://lists.openembedded.org/g/openembedded-core/message/245837 Tested on the autobuilder from a contrib branch: The x86 reproducible builder came back green with zero differing packages, the rust test suite passed, and on both arm-hosted reproducible builders rust, rust-dbg, rust-rustdoc and libstd-rs-dev are gone from the failing list. reproducible debian12-vk-3 agodard/master-next-bis&agodard/master-next completed at 2026-09-15 19:32:10+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260915-vm462bt0 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4991/steps/15/logs/stdio reproducible-arm alma10-vk-arm1 master&master-next completed at 2026-09-17 21:48:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/10/steps/15/logs/stdio reproducible-arm-cross ubuntu2404-vk-arm2 master&master-next completed at 2026-09-17 22:42:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/10/steps/15/logs/stdio reproducible-cross alma8-vk-1 master&master-next completed at 2026-09-17 11:44:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/9/steps/16/logs/stdio reproducible-cross opensuse156-vk-1 master&master-next completed at 2026-09-17 23:10:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/10/steps/16/logs/stdio reproducible-cross debian12-vk-4 master&master-next completed at 2026-09-18 23:42:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/11/steps/15/logs/stdio reproducible-arm-cross alma10-vk-arm1 master&master-next completed at 2026-09-19 11:12:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/11/steps/15/logs/stdio reproducible-arm ubuntu2404-vk-arm2 master&master-next completed at 2026-09-19 09:34:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/11/steps/15/logs/stdio reproducible opensuse160-vk-1 master&master-next completed at 2026-09-19 21:44:06+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260919-l3njlk0p https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/5022/steps/15/logs/stdio reproducible-arm-cross alma10-vk-arm1 master&master-next completed at 2026-09-24 02:20:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/12/steps/15/logs/stdio reproducible-arm debian13-vk-arm1 master&master-next completed at 2026-09-24 01:22:45+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/12/steps/15/logs/stdio I am working with cargo upstream to fix remain rust packages differ we are seeing on Autobuilder. A quick status on the ARM host failures I'm working on: libopkg1/opkg-dbg: This one goes back to opkg's switch from autotools to CMake last year. autotools used host_cpu for the built-in default arch, which is the CPU opkg runs on. The CMake port uses CMAKE_HOST_SYSTEM_PROCESSOR instead, and that's the build machine. So the x86 worker puts "x86_64" into the package and the ARM worker puts aarch64". Nobody noticed before since: our images always ship arch.conf, but the ARM workers catch it. I've sent a fix to the opkg list (bug 15784): https://lists.yoctoproject.org/g/opkg/message/205 curl-dev and ruby: The build machine's triplet ends up in files we ship (the configure line in curl-config, the build_* entries in ruby's rbconfig.rb). The fix strips them in the recipes. It's on the autobuilder(Testing on contrib branch) now and I'll send it once that's clean. rust-rustdoc: cargo mixes host-only dependencies (build scripts, proc-macros) into the target crate's -C metadata. Upstream PR: https://github.com/rust-lang/cargo/pull/17522 Still working on the rest and will send the oe-core patches as these get reviewed. curl and ruby are now sent to oe-core, after the autobuilder runs came back clean for curl-dev and ruby on the ARM builders: - https://lists.openembedded.org/g/openembedded-core/message/246688 - https://lists.openembedded.org/g/openembedded-core/message/246689 reproducible-arm ubuntu2404-vk-arm1 master-next&master completed at 2026-09-24 22:02:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/13/steps/15/logs/stdio reproducible-arm-cross stream10-vk-arm1 master-next&master completed at 2026-09-24 21:59:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/13/steps/15/logs/stdio reproducible-cross rocky9-vk-2 master-next&master completed at 2026-09-24 19:03:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/13/steps/16/logs/stdio opkg - sent to oe-core. https://lists.openembedded.org/g/openembedded-core/message/246961 reproducible-arm ubuntu2404-vk-arm2 master completed at 2026-10-17 23:26:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/16/steps/15/logs/stdio reproducible-arm-cross debian13-vk-arm1 master completed at 2026-10-17 22:33:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/16/steps/15/logs/stdio reproducible-cross rocky9-vk-2 master completed at 2026-10-17 21:01:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/15/steps/16/logs/stdio |