Overview: rust cargo traverses parent directories looking for .cargo/config.toml until it reaches $HOME/.cargo/config.toml outside the yocto build, even though $CARGO_HOME is set. Steps to reproduce: Create $HOME/.cargo/config.toml with [target.x86_64-unknown-linux-gnu] linker = "clang" # or any arbitrary command would probably work here. Then "bitbake cargo-native". (I'm building openbmc head, though see no reason plain poky would behave differently). Expected: Builds OK. Actual: Build fails with Building [ ] 7/329: libc(build.rs), pkg-confi...^M Running `rustc --crate-name build_script_main --edition=2018 /home/matt/tmp/build/openbmc/tmp/work/x86_64-linux/cargo-native/1.75.0/rustc-1.75.0-src/vendor/typenum/build/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=077bb2f562415289 -C extra-filename=-077bb2f562415289 --out-dir /home/matt/tmp/build/openbmc/tmp/work/x86_64-linux/cargo-native/1.75.0/build/target/release/build/typenum-077bb2f562415289 -C linker=clang -L dependency=/home/matt/tmp/build/openbmc/tmp/work/x86_64-linux/cargo-native/1.75.0/build/target/release/deps --cap-lints allow` error: linker `clang` not found Investigating further with strace, it turns out cargo is traversing all directories starting in the build directory, looking for .cargo/config.toml. It eventually gets outside the yocto build tree and loads the config from $HOME/.cargo/config.toml, where it finds the linker="clang" argument. I can't see any way to get cargo to avoid it (without patching), my current workaround is to run the build in "firejail --blacklist=$HOME/.cargo". I personally don't need a solution to this at the moment, but am filing this bug in case anyone else hits the same problem. There's an existing cargo issue https://github.com/rust-lang/cargo/issues/11045 , closed as a duplicate of the more general issue https://github.com/rust-lang/cargo/issues/9769 . I don't think any of the existing cargo PRs solve the problem though.
Sundeep, please have someone in the WR toolchain team look this bug.
I tried reproducing the issue on poky with below config in $HOME/.cargo/config.toml. [target.x86_64-unknown-linux-gnu] # also with [target.x86_64-poky-linux-gnu] linker = "clang" The "bitbake cargo-native/cargo/rust" are build successfully. Can you give more info to reproduce the issue like 'any specific Yocto branch/conf changes'... etc?
I've just checked with a fresh poky checkout, I can reproduce it there. Is your build directory inside the home directory? Let me know any other details that might help. commit b94b85bef0b6375be71be6002521e147906080e5 (HEAD -> master, origin/master, origin/HEAD) Author: Yoann Congal <yoann.congal@smile.fr> Date: Tue Nov 5 23:42:13 2024 +0100 source in $HOME/3rd/oe/poky . ./oe-init-build-env $HOME/build/poky2 Build Configuration: BB_VERSION = "2.9.1" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "x86_64-poky-linux" MACHINE = "qemux86-64" DISTRO = "poky" DISTRO_VERSION = "5.1" TUNE_FEATURES = "m64 core2" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:b94b85bef0b6375be71be6002521e147906080e5" bitbake cargo-native ... | Running `rustc --crate-name subtle --edition=2018 /home/matt/tmp/build/poky2/tmp/work/x86_64-linux/cargo-native/1.79.0/rustc-1.79.0-src/vendor/subtle-2.5.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="i128"' -C metadata=b2c1a17508589118 -C extra-filename=-b2c1a17508589118 --out-dir /home/matt/tmp/build/poky2/tmp/work/x86_64-linux/cargo-native/1.79.0/build/target/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C linker=clang -L dependency=/home/matt/tmp/build/poky2/tmp/work/x86_64-linux/cargo-native/1.79.0/build/target/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/matt/tmp/build/poky2/tmp/work/x86_64-linux/cargo-native/1.79.0/build/target/release/deps --cap-lints allow -L /home/matt/tmp/build/poky2/tmp/work/x86_64-linux/cargo-native/1.79.0/recipe-sysroot-native/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib --remap-path-prefix=/home/matt/tmp/build/poky2/tmp/work/x86_64-linux/cargo-native/1.79.0=/usr/src/debug/cargo-native/1.79.0` | error: linker `clang` not found $HOME/.cargo/config.toml [target.x86_64-unknown-linux-gnu] linker = "clang"
The $HOME/.cargo/config.toml is the default config and the hierarchy of the toml as described in - https://doc.rust-lang.org/cargo/reference/config.html#hierarchical-structure For the linker issue, in the $HOME/.cargo/config.toml, for 'linker' give the absolute path. [target.x86_64-unknown-linux-gnu] linker = "/usr/bin/lld" The rustc will find the linker but you'll get the following error due to missing options/libs etc. | error: linking with `/usr/bin/lld` failed: exit status: 1 | = note: lld: error: unable to find library -lgcc_s | lld: error: unable to find library -lutil | lld: error: unable to find library -lrt | lld: error: unable to find library -lpthread | lld: error: unable to find library -lm | lld: error: unable to find library -ldl | lld: error: unable to find library -lc | error: could not compile `typenum` (build script) due to 1 previous error The Yocto build has a python wrapper for linker 'target-rust-ccld', which was implemented with all required options & libs for linking. Path of linker in build dir - .../build/tmp/work/x86_64-linux/cargo-native/1.79.0/wrapper/target-rust-ccld
The problem here is that the yocto build shouldn't be reading files outside of the build directory at all.
This is not an issue with Yocto environment but the behavior of Cargo. "Cargo probes configuration all the way down from the current working directory to / root , so you might still end up reading the ~/.cargo/config.toml if you're in somewhere under your home directory." See - https://github.com/rust-lang/cargo/issues/12519#issuecomment-1682477126 I am able able to see the issue when the poky is some where on $HOME directory. And, the same build works when the poky in other than $HOME dir. But, I still have a question about the significance of setting cargo_home (let me check this with community for clarification).
Bump to M2. Any news Sundeep? I agree with Matt that we should ensure that cargo in our environment never goes outside of it's project directory and therefore should never touch $HOME/.cargo/config.toml
Hi Randy, As explained here - https://doc.rust-lang.org/cargo/reference/config.html#hierarchical-structure Cargo looks for configuration files in the current directory and all parent directories. When the poky is in $HOME path the global config file also will be read by the cargo. This is not an issue in Yocto/Poky but happening due to the cargo config hierarchy. Issue will not be there when the poky to moved to other than $HOME path. So, it is not a bug.
As Matt said ealier, we can't have a Yocto build failing or behaving differently due to different $HOME/.cargo/config.toml files. We either need to add our own project level config.toml or worst case, past cargo to change limit it's search. Am I missing something Sundeep?
typo: **patch** cargo to change limit it's search.
We can add global project level config.toml file using the CARGO_HOME env variable. Since project will have many other toml files so cargo has an hierarchical structure while parsing the local toml files. The hierarchy will continue till $HOME (https://doc.rust-lang.org/cargo/reference/config.html#hierarchical-structure) To prevent Cargo from using $HOME/.cargo/config.toml, the project should be located outside of the $HOME directory path.
Sundeep, Your link about hierarchical structure says: $CARGO_HOME/config.toml which defaults to: <window's or unix home dir> The key word for me is defaults. Note that CARGO_HOME is defined in: https://doc.rust-lang.org/cargo/guide/cargo-home.html The “Cargo home” functions as a download and source cache. When building a crate, Cargo stores downloaded build dependencies in the Cargo home. You can alter the location of the Cargo home by setting the CARGO_HOME environmental variable. For a Yocto build to be reproducible, we can't allow this default. Can't we set the CARGO_HOME environment variable to be the: <project_dir>/tmp/work/<arch>/.cargo or for cargo-native <project_dir>/tmp/work/<build_arch> and some other directory for the SDK? This just a general idea of how to solve this bug and you should look at where we are getting the cargo downloads and source caches now before picking a directory.
Move to M3. Sundeep, please reply to my previous comment (12): https://bugzilla.yoctoproject.org/show_bug.cgi?id=15637#c12 If there's need, we can talk about this problem on the email list and/or during the YP tech call next week.
**Summary:** 1. Setting CARGO_HOME to a project-level directory (as suggested in comment #12) was tried but does not work. CARGO_HOME/config.toml has the lowest precedence in Cargo's config hierarchy — it is consulted after the full CWD→/ (root) directory scan, so it cannot override configs found earlier in the traversal. 2. A local Yocto patch to limit Cargo's config search to the Yocto build directory was attempted. This hits a fundamental blocker: the Cargo snapshot binary (a pre-built binary downloaded during the build process) still contains the old/default search logic. To work around this, we had to clone and build a patched Cargo snapshot separately, then use that to build the final Cargo — which significantly increases build time and disk usage and is not viable to bring into Yocto. 3. Limiting the search to $CARGO_HOME also requires a local patch to Cargo's source and fails for two reasons: - The same snapshot binary blocker as above. - Several required .toml files for the Rust build exist outside $CARGO_HOME, causing additional build failures. **Upstream Status:** All related upstream PRs remain unmerged. The Cargo team has stated they prefer an RFC-based solution over incremental fixes. No resolution is expected in the near term. - #7894 (CARGO_CONFIG_PATH env var): https://github.com/rust-lang/cargo/pull/7894 - #8652 (--ignore-local-config flag): https://github.com/rust-lang/cargo/pull/8652 - #9769 (meta issue): https://github.com/rust-lang/cargo/issues/9769 **Workarounds (both confirmed working):** - Place the Poky/build directory outside of $HOME - Remove or rename $HOME/.cargo/config.toml With the above workarounds, Yocto builds are reproducible. **Recommendation:** All viable fix paths have been exhausted. Proposing to close this as Won't Fix with the above workarounds documented. Should be revisited if upstream Cargo resolves this via the RFC process.
Hemanth, Thanks for the clear summary and upstream status and proposed resolution. Since we don't have a viable solution, we can add a check for the existence of and/or contents of:~/.cargo/config.toml in meta/classes-global/sanity.bbclass and issue a warning to start and an error later.
We had a setup with adding an internal cache like this: $HOME/.cargo/config.toml [source.local-cache-remote] registry = "sparse+https://artifactory/api/cargo/cargo-remote/index/" [source.crates-io] replace-with = "local-cache-remote" But that caused errors like this when building cargo-native: Caused by: failed to query replaced source registry `crates-io` Caused by: attempting to make an HTTP request, but --frozen was specified Because it collided with the Yocto crates-io override. So a somewhat similar problem.
(In reply to Ernst Persson from comment #16) > We had a setup with adding an internal cache like this: > > $HOME/.cargo/config.toml > > [source.local-cache-remote] > registry = "sparse+https://artifactory/api/cargo/cargo-remote/index/" > > [source.crates-io] > replace-with = "local-cache-remote" > > > But that caused errors like this when building cargo-native: > > Caused by: > failed to query replaced source registry `crates-io` > > Caused by: > attempting to make an HTTP request, but --frozen was specified > > Because it collided with the Yocto crates-io override. > > So a somewhat similar problem. Hi Ernst, Thanks for sharing this case. Just to confirm — does this issue get resolved when applying the workarounds mentioned in comment #14 (i.e. moving the build directory outside $HOME or removing/renaming $HOME/.cargo/config.toml)? I’m trying to understand whether this is another instance of the same underlying problem (Cargo picking up configuration from $HOME), or if it needs to be handled separately.
(In reply to Hemanth Kumar M D from comment #17) > (In reply to Ernst Persson from comment #16) > > We had a setup with adding an internal cache like this: > > > > $HOME/.cargo/config.toml > > > > [source.local-cache-remote] > > registry = "sparse+https://artifactory/api/cargo/cargo-remote/index/" > > > > [source.crates-io] > > replace-with = "local-cache-remote" > > > > > > But that caused errors like this when building cargo-native: > > > > Caused by: > > failed to query replaced source registry `crates-io` > > > > Caused by: > > attempting to make an HTTP request, but --frozen was specified > > > > Because it collided with the Yocto crates-io override. > > > > So a somewhat similar problem. > > > > Hi Ernst, > > Thanks for sharing this case. > > Just to confirm — does this issue get resolved when applying the > workarounds mentioned in comment #14 (i.e. moving the build directory > outside $HOME or removing/renaming $HOME/.cargo/config.toml)? > > I’m trying to understand whether this is another instance of the same > underlying problem (Cargo picking up configuration from $HOME), or if > it needs to be handled separately.
(In reply to Hemanth Kumar M D from comment #14) > **Summary:** > > 1. Setting CARGO_HOME to a project-level directory (as suggested in comment > #12) was tried but does not work. CARGO_HOME/config.toml has the lowest > precedence in Cargo's config hierarchy — it is consulted after the full > CWD→/ (root) directory scan, so it cannot override configs found earlier in > the traversal. > > 2. A local Yocto patch to limit Cargo's config search to the Yocto build > directory was attempted. This hits a fundamental blocker: the Cargo snapshot > binary (a pre-built binary downloaded during the build process) still > contains the old/default search logic. To work around this, we had to clone > and build a patched Cargo snapshot separately, then use that to build the > final Cargo — which significantly increases build time and disk usage and is > not viable to bring into Yocto. > > 3. Limiting the search to $CARGO_HOME also requires a local patch to Cargo's > source and fails for two reasons: > - The same snapshot binary blocker as above. > - Several required .toml files for the Rust build exist outside > $CARGO_HOME, causing additional build failures. > > **Upstream Status:** > All related upstream PRs remain unmerged. The Cargo team has stated they > prefer an RFC-based solution over incremental fixes. No resolution is > expected in the near term. > - #7894 (CARGO_CONFIG_PATH env var): > https://github.com/rust-lang/cargo/pull/7894 > - #8652 (--ignore-local-config flag): > https://github.com/rust-lang/cargo/pull/8652 > - #9769 (meta issue): https://github.com/rust-lang/cargo/issues/9769 > > **Workarounds (both confirmed working):** > - Place the Poky/build directory outside of $HOME > - Remove or rename $HOME/.cargo/config.toml > > With the above workarounds, Yocto builds are reproducible. > > **Recommendation:** > All viable fix paths have been exhausted. Proposing to close this as Won't > Fix with the above workarounds documented. Should be revisited if upstream > Cargo resolves this via the RFC process. Hi Hemanth, In my opinion the issue needs to be fixed from the roots, Do you know any effort on Cargo community to make this configurable?. What is the time that takes to build cargo after the patched version?, of course in which system reference like Kernel or Chromium are good references. So build is the next step and we have sstate-cache/ccache a note about this build process taking time and cargo users can be aware based on documentation. Cheers!, Anibal
(In reply to Aníbal Limón from comment #19) > (In reply to Hemanth Kumar M D from comment #14) > > **Summary:** > > > > 1. Setting CARGO_HOME to a project-level directory (as suggested in comment > > #12) was tried but does not work. CARGO_HOME/config.toml has the lowest > > precedence in Cargo's config hierarchy — it is consulted after the full > > CWD→/ (root) directory scan, so it cannot override configs found earlier in > > the traversal. > > > > 2. A local Yocto patch to limit Cargo's config search to the Yocto build > > directory was attempted. This hits a fundamental blocker: the Cargo snapshot > > binary (a pre-built binary downloaded during the build process) still > > contains the old/default search logic. To work around this, we had to clone > > and build a patched Cargo snapshot separately, then use that to build the > > final Cargo — which significantly increases build time and disk usage and is > > not viable to bring into Yocto. > > > > 3. Limiting the search to $CARGO_HOME also requires a local patch to Cargo's > > source and fails for two reasons: > > - The same snapshot binary blocker as above. > > - Several required .toml files for the Rust build exist outside > > $CARGO_HOME, causing additional build failures. > > > > **Upstream Status:** > > All related upstream PRs remain unmerged. The Cargo team has stated they > > prefer an RFC-based solution over incremental fixes. No resolution is > > expected in the near term. > > - #7894 (CARGO_CONFIG_PATH env var): > > https://github.com/rust-lang/cargo/pull/7894 > > - #8652 (--ignore-local-config flag): > > https://github.com/rust-lang/cargo/pull/8652 > > - #9769 (meta issue): https://github.com/rust-lang/cargo/issues/9769 > > > > **Workarounds (both confirmed working):** > > - Place the Poky/build directory outside of $HOME > > - Remove or rename $HOME/.cargo/config.toml > > > > With the above workarounds, Yocto builds are reproducible. > > > > **Recommendation:** > > All viable fix paths have been exhausted. Proposing to close this as Won't > > Fix with the above workarounds documented. Should be revisited if upstream > > Cargo resolves this via the RFC process. > > Hi Hemanth, > > In my opinion the issue needs to be fixed from the roots, > > Do you know any effort on Cargo community to make this configurable?. > > What is the time that takes to build cargo after the patched version?, of > course in which system reference like Kernel or Chromium are good references. > So build is the next step and we have sstate-cache/ccache a note about this > build process taking time and cargo users can be aware based on > documentation. > > Cheers!, > Anibal Just did a quick search looks like a very deep design issue, with multiple breakage points in cargo. https://github.com/rust-lang/cargo/issues?q=is%3Aissue%20state%3Aopen%20HOME With affectations on different parts from cache, build, target, etc.
Anibal, Hemanth, Sundeep, I still think that since we don't have a viable fix and that may take a very long time, we should use the sanity mechanism to warn users. We should also take advantage or our upstream connections, David Wood specifically, to get upstream Rust developers thinking about our use case. David recently pinged us about how Rust and Yocto are doing so I'd like Sundeep to reply to that email thread, and point David at this issue and/or the upstream equivalent.
Sure Randy, I have the same opinion to convey this to David. I'll do it.
The issue is communicated to DavidWood. As of now the workarounds mentioned in https://bugzilla.yoctoproject.org/show_bug.cgi?id=15637#c14 are the only available solution. Issue is closing (will be reviewed in future if cargo come up with any solution).
A sanity check as in https://bugzilla.yoctoproject.org/show_bug.cgi?id=15637#c15 need to be implemented.
Richard suggests that we get upstream to change cargo to ignore any config in the user's home directory by an environment option or a command-line flag. Also moved from 6.1-M4 to M2 since reproducible builds is an important feature.
Hi Randy, A similar request has already been raised upstream in the Cargo project. The relevant issues are: 1) https://github.com/rust-lang/cargo/issues/7621 — Add flag to ignore all parent directory configs. 2) https://github.com/rust-lang/cargo/issues/8643 — Add --ignore-local-config flag. 3) https://github.com/rust-lang/cargo/issues/9769 — Meta issue tracking all config search problems.
Sanity check is implemented and merged to oe-core master: https://git.openembedded.org/openembedded-core/commit/?id=77516d3c4fe4291eb947c2f7c636a362c041c911