14875 2022-08-09 13:15:10 +0000 reproducibility failures in rust 2025-11-07 23:39:27 +0000 1 1 10 QA/Testing Build Testing general unspecified x86 Multiple RESOLVED FIXED Medium+ normal 4.3 M4 1 richard.purdie sundeep.kokkonda alex.kiernan ccasciato frederic.martinsons randy.macleod richard.purdie sundeep.kokkonda --- --- No (bug/feature does not impact docs) oldest_to_newest 93734 0 richard.purdie 2022-08-09 13:15:10 +0000 We've had to disable rust and rust-dbg from the reproducibility tests since the binaries generated are not always identical. An example are: http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220803-alsw4xhu/packages/reproducibleA/tmp/deploy/deb/core2-64/rust_1.62.0-r0_amd64.deb http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220803-alsw4xhu/packages/reproducibleB/tmp/deploy/deb/core2-64/rust_1.62.0-r0_amd64.deb from https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1279 (cancelled as diffoscope can't cope with the size of these). This is a different issue to the reproducibility patch which is also applied to the rust recipe. 93768 1 randy.macleod 2022-08-11 14:39:52 +0000 Need to fix this so we can continue to report that oe-core is 100% reproducible. 93916 2 randy.macleod 2022-08-25 16:07:46 +0000 Richard, you and Sundeep were talking about this bug today and it seemed that it was only happening on ubu-18.04 but the link below is alma9 only. Were there other build logs to look at or should Sundeep be testing an alma9 docker container? If it is only ubuntu, let me knwo what host is involved and I'll collect the packages installed to see if that's a factor. The Wind River shared servers tend to have a superset of the packages installed on the YP AB nodes. 94122 3 sundeep.kokkonda 2022-10-10 04:38:56 +0000 Hello, This reproducible issue is happening because of the change in the path of the build directory, with the same path name - multiple builds are generating the identical binaries. The change in build path making differences in the generated object files, static libs (.rlib), .rmeta files, fingerprint/hash data & json outputs in bootstrapping & stage-0 builds which are in turn affecting final binaries generated in stage-1. This issue is reported in rust community https://github.com/rust-lang/rust/issues/102299 (detailed technical analysis can be found here). Also, this looks a known issue to rust community, I found a few other similar 'open' issues reported in the past, for eg. https://github.com/rust-lang/cargo/issues/5505, https://github.com/rust-lang/cargo/issues/8140. However, I analyzed issue a bit deeper and tried to fix/workaround the issue by using the option like '--remap-path-prefix' & by removing the metadata crate dependencies but still the final binaries are different. 94128 4 sundeep.kokkonda 2022-10-11 04:07:28 +0000 As per rust community feedback (https://github.com/rust-lang/rust/issues/102299#issuecomment-1273906259) - "The build paths are embedded in the rust binaries, this is a known limitation at the moment and projects like Arch Linux that are currently doing reproducible builds use a standardized build path to reproduce their rust binaries (and the rust compiler itself)". So, for builds to be reproducible the build path should be same. 94155 5 randy.macleod 2022-10-13 17:49:50 +0000 Sundeep, thanks for working on this and consulting with upstream. I suspect that we'd carry a local patch if you can get some guidance from upstream on how to eliminate the build paths. 94593 6 randy.macleod 2022-12-22 16:46:36 +0000 Sundeep. Moved to M2. If it helps, you could document what you know and what each step is that you take to narrow down the issue as discussed in our call earlier today. 94729 7 alex.kiernan 2023-01-16 12:18:12 +0000 I suspect https://github.com/rust-lang/rust/issues/98185 may be relevant (there's a patch inside the embedded .cargo/config.toml in the rustc-source tarball). 94730 8 sundeep.kokkonda 2023-01-16 14:23:47 +0000 Hi Alex, Thanks for your inputs. The .cargo/config.toml mentioned to use the crates from 'vendor' directory. The 'vendor' directory has the 'fixed version' of crates available local to the rust sources. Removing the usage of this .cargo/config.toml causes the cargo to fetch the latest available version of crates from network and build is aborted by the '--frozen' compiler option which forces the compiler to use the crate versions fixed in Cargo.lock file. So, the .cargo/config.toml cannot be removed. The https://github.com/rust-lang/rust/issues/98185 discussion says the hash is changing when the crates are compiled using the local crates and --remap-path-prefix is not working in such cases. I am trying to compile a crate using the sources from 'vendor' directory crates to ensure the hash is changing between the builds. If the hash is changed may be we can say the issue is due to the usage of local crates. Let me know if you've any comments on this approach... 94731 9 alex.kiernan 2023-01-16 14:29:33 +0000 Most of the vendored sources are wired in using these parts of .cargo/config.toml: [source.crates-io] replace-with = "vendored-sources" [source.vendored-sources] directory = "vendor" but I wonder if this piece: [source."https://github.com/bjorn3/rust-ar.git"] git = "https://github.com/bjorn3/rust-ar.git" branch = "do_not_remove_cg_clif_ranlib" replace-with = "vendored-sources" leads to the non-reproducibility we see. Though my recollection when I added the comment was that this was a [patch."..."] section and it's clearly not, so may not apply to the upstream ticket. 94735 10 sundeep.kokkonda 2023-01-17 09:19:14 +0000 The https://github.com/bjorn3/rust-ar.git page says that 'rust-ar' is a rust library for encoding/decoding Unix archive (.a) files. There is no directory with 'vendored-sources' or 'rust-ar' in the tarball, so, I did not understood what is getting patched/replaced with the sources given in the link. However, I removed below code and tried a build, still the reproducibility issue is occuring. [source."https://github.com/bjorn3/rust-ar.git"] git = "https://github.com/bjorn3/rust-ar.git" branch = "do_not_remove_cg_clif_ranlib" replace-with = "vendored-sources" Also, I gave 2 builds with identical path and compared the generated binaries & those are identical i.e., these lines are not having any impact on code generation. 94737 11 alex.kiernan 2023-01-17 10:01:58 +0000 It's referenced here: compiler/rustc_codegen_cranelift/Cargo.toml 21:ar = { git = "https://github.com/bjorn3/rust-ar.git", branch = "do_not_remove_cg_clif_ranlib" } and the vendored directory is vendor/ar. But if removing those doesn't change the reproducibility, it doesn't seem like this is a useful line of inquiry after all. 95120 12 sundeep.kokkonda 2023-03-13 13:37:41 +0000 This issue is occurring only when local crates from 'vendor' directory are used. I am able to reproduce the same reproducible issue as in Yocto with rust tarball sources when compiled the sources from 'vendor' directory inside the rust tarball. (When crates are pulled from network the issue is not reproducible in rust tarball). We must enable the flag 'vendor = true' in config.toml in rust tarball inorder to use the sources from 'vendor' directory, when this flag disabled crates are pulled from network and issue is not reproducible. This bug when using local crates was already raised in rust community https://github.com/rust-lang/rust/issues/98185 Binary diff in rust tarball: :/tar/rustc-1.67.0-src-> diff -ur buildX/x86_64-unknown-linux-gnu/stage2 buildY/x86_64-unknown-linux-gnu/stage2 Binary files buildX/x86_64-unknown-linux-gnu/stage2/lib/librustc_driver-c96e53ba8e842355.so and buildY/x86_64-unknown-linux-gnu/stage2/lib/librustc_driver-c96e53ba8e842355.so differ :/tar/rustc-1.67.0-src-> 96251 13 frederic.martinsons 2023-08-14 07:10:45 +0000 Good news, back in may an RFC about making path trim per default has been merged: https://github.com/rust-lang/rfcs/pull/3127 This may make our reproducible issues disappeared or be less likely , it impacts both cargo (https://github.com/rust-lang/cargo/issues/12137) and rustc (https://github.com/rust-lang/rust/issues/111540) but the implementation is not yet done. I'll watch the two issues to be notified about the progress 96253 14 frederic.martinsons 2023-08-14 07:12:26 +0000 *** Bug 15090 has been marked as a duplicate of this bug. *** 96673 15 sundeep.kokkonda 2023-10-11 14:34:04 +0000 Issue fixed: https://git.openembedded.org/openembedded-core/commit/?id=6ae62259afbbe861ed74211dab18a27b8c8d8b7a 96674 16 sundeep.kokkonda 2023-10-11 14:48:59 +0000 Fix committed to master - https://git.openembedded.org/openembedded-core/commit/?id=6ae62259afbbe861ed74211dab18a27b8c8d8b7a