Bug 14875

Need to fix this so we can continue to report that oe-core is 100% reproducible.

Richard, you and Sundeep were talking about this bug today and it seemed that it was only happening on ubu-18.04 but the link below is alma9 only. Were there other build logs to look at or should Sundeep be testing an alma9 docker container? If it is only ubuntu, let me knwo what host is involved and I'll collect the packages installed to see if that's a factor. The Wind River shared servers tend to have a superset of the packages installed on the YP AB nodes.

Hello,

This reproducible issue is happening because of the change in the path of the build directory, with the same path name - multiple builds are generating the identical binaries.
The change in build path making differences in the generated object files, static libs (.rlib), .rmeta files, fingerprint/hash data & json outputs in bootstrapping & stage-0 builds which are in turn affecting final binaries generated in stage-1.

This issue is reported in rust community https://github.com/rust-lang/rust/issues/102299 (detailed technical analysis can be found here).
Also, this looks a known issue to rust community, I found a few other similar 'open' issues reported in the past, for eg. https://github.com/rust-lang/cargo/issues/5505, https://github.com/rust-lang/cargo/issues/8140.

However, I analyzed issue a bit deeper and tried to fix/workaround the issue by using the option like '--remap-path-prefix' & by removing the metadata crate dependencies but still the final binaries are different.

As per rust community feedback (https://github.com/rust-lang/rust/issues/102299#issuecomment-1273906259) - 
"The build paths are embedded in the rust binaries, this is a known limitation at the moment and projects like Arch Linux that are currently doing reproducible builds use a standardized build path to reproduce their rust binaries (and the rust compiler itself)".
So, for builds to be reproducible the build path should be same.

Sundeep, thanks for working on this and consulting with upstream.
I suspect that we'd carry a local patch if you can get some guidance from upstream on how to eliminate the build paths.

Sundeep. Moved to M2. 
If it helps, you could document what you know and what each step is that you take to narrow down the issue as discussed in our call earlier today.

I suspect https://github.com/rust-lang/rust/issues/98185 may be relevant (there's a patch inside the embedded .cargo/config.toml in the rustc-source tarball).

Hi Alex,

Thanks for your inputs.

The .cargo/config.toml mentioned to use the crates from 'vendor' directory. The 'vendor' directory has the 'fixed version' of crates available local to the rust sources.
Removing the usage of this .cargo/config.toml causes the cargo to fetch the latest available version of crates from network and build is aborted by the '--frozen' compiler option which forces the compiler to use the crate versions fixed in Cargo.lock file.
So, the .cargo/config.toml cannot be removed.

The https://github.com/rust-lang/rust/issues/98185 discussion says the hash is changing when the crates are compiled using the local crates and --remap-path-prefix is not working in such cases.
I am trying to compile a crate using the sources from 'vendor' directory crates to ensure the hash is changing between the builds.
If the hash is changed may be we can say the issue is due to the usage of local crates.

Let me know if you've any comments on this approach...

Most of the vendored sources are wired in using these parts of .cargo/config.toml:

  [source.crates-io]
  replace-with = "vendored-sources"

  [source.vendored-sources]
  directory = "vendor"

but I wonder if this piece:

  [source."https://github.com/bjorn3/rust-ar.git"]
  git = "https://github.com/bjorn3/rust-ar.git"
  branch = "do_not_remove_cg_clif_ranlib"
  replace-with = "vendored-sources"

leads to the non-reproducibility we see.

Though my recollection when I added the comment was that this was a [patch."..."] section and it's clearly not, so may not apply to the upstream ticket.

The https://github.com/bjorn3/rust-ar.git page says that 'rust-ar' is a rust library for encoding/decoding Unix archive (.a) files.
There is no directory with 'vendored-sources' or 'rust-ar' in the tarball, so, I did not understood what is getting patched/replaced with the sources given in the link. 

However, I removed below code and tried a build, still the reproducibility issue is occuring.

  [source."https://github.com/bjorn3/rust-ar.git"]
  git = "https://github.com/bjorn3/rust-ar.git"
  branch = "do_not_remove_cg_clif_ranlib"
  replace-with = "vendored-sources"

Also, I gave 2 builds with identical path and compared the generated binaries & those are identical i.e., these lines are not having any impact on code generation.

It's referenced here:

  compiler/rustc_codegen_cranelift/Cargo.toml
  21:ar = { git = "https://github.com/bjorn3/rust-ar.git", branch = "do_not_remove_cg_clif_ranlib" }

and the vendored directory is vendor/ar.

But if removing those doesn't change the reproducibility, it doesn't seem like this is a useful line of inquiry after all.

This issue is occurring only when local crates from 'vendor' directory are used.

I am able to reproduce the same reproducible issue as in Yocto with rust tarball sources when compiled the sources from 'vendor' directory inside the rust tarball. (When crates are pulled from network the issue is not reproducible in rust tarball).
We must enable the flag 'vendor = true' in config.toml in rust tarball inorder to use the sources from 'vendor' directory, when this flag disabled crates are pulled from network and issue is not reproducible.

This bug when using local crates was already raised in rust community https://github.com/rust-lang/rust/issues/98185

Binary diff in rust tarball:
:/tar/rustc-1.67.0-src-> diff -ur buildX/x86_64-unknown-linux-gnu/stage2 buildY/x86_64-unknown-linux-gnu/stage2
Binary files buildX/x86_64-unknown-linux-gnu/stage2/lib/librustc_driver-c96e53ba8e842355.so and buildY/x86_64-unknown-linux-gnu/stage2/lib/librustc_driver-c96e53ba8e842355.so differ
:/tar/rustc-1.67.0-src->

Good news, back in may an RFC about making path trim per default has been merged: https://github.com/rust-lang/rfcs/pull/3127


This may make our reproducible issues disappeared or be less likely , it impacts both cargo (https://github.com/rust-lang/cargo/issues/12137) and rustc (https://github.com/rust-lang/rust/issues/111540) but the implementation is not yet done.

I'll watch the two issues to be notified about the progress

*** Bug 15090 has been marked as a duplicate of this bug. ***

Issue fixed:
https://git.openembedded.org/openembedded-core/commit/?id=6ae62259afbbe861ed74211dab18a27b8c8d8b7a

Fix committed to master - https://git.openembedded.org/openembedded-core/commit/?id=6ae62259afbbe861ed74211dab18a27b8c8d8b7a