| Summary: | Investigate implementing shared-source for Rust | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Randy MacLeod <randy.macleod> |
| Component: | devtools / tool chain | Assignee: | Deepesh Varatharajan <deepesh.varatharajan> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | Medium | CC: | deepesh.varatharajan, meta.mr.watcher, meta.watcher, ross.burton, sundeep.kokkonda, yoann.congal |
| Version: | 5.3 | ||
| Target Milestone: | 6.1 M3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
|
Description
Randy MacLeod
2025-03-24 17:44:08 UTC
Sundeep
I decided to check if the rust build script (x.py) can be run from
outside the SRCDIR to see if a shared dir is even possible. It is.
You can build rust, out of tree:
❯ tar xf /.../downloads/rustc-1.85.1-src.tar.xz
❯ mkdir b1
❯ ls
b1 rustc-1.85.1-src
❯ cd b1/
❯ ../rustc-1.85.1-src/x.py
INFO: Downloading and building bootstrap before processing --help command.
See src/bootstrap/README.md for help with common commands.
downloading https://static.rust-lang.org/dist/2025-01-09/rust-std-1.84.0-x86_64-unknown-linux-gnu.tar.xz
####################################################################################################################################################################################### 100.0%
downloading https://static.rust-lang.org/dist/2025-01-09/rustc-1.84.0-x86_64-unknown-linux-gnu.tar.xz
####################################################################################################################################################################################### 100.0%
downloading https://static.rust-lang.org/dist/2025-01-09/cargo-1.84.0-x86_64-unknown-linux-gnu.tar.xz
####################################################################################################################################################################################### 100.0%
extracting /media/rmacleod/gitter/rmacleod/src/lang/rust/1.85.1/b1/build/cache/2025-01-09/rust-std-1.84.0-x86_64-unknown-linux-gnu.tar.xz
extracting /media/rmacleod/gitter/rmacleod/src/lang/rust/1.85.1/b1/build/cache/2025-01-09/cargo-1.84.0-x86_64-unknown-linux-gnu.tar.xz
extracting /media/rmacleod/gitter/rmacleod/src/lang/rust/1.85.1/b1/build/cache/2025-01-09/rustc-1.84.0-x86_64-unknown-linux-gnu.tar.xz
Building bootstrap
Compiling proc-macro2 v1.0.89
Compiling unicode-ident v1.0.13
Compiling memchr v2.7.4
Compiling version_check v0.9.5
...
< eventually prints help text>
❯ ../rustc-1.85.1-src/x.py build
...
Compiling rustc_transmute v0.0.0 (/.../rust/1.85.1/rustc-1.85.1-src/compiler/rustc_transmute)
...
Finished `release` profile [optimized] target(s) in 32.75s
WARNING: you have not made a `config.toml`
HELP: consider running `./x.py setup` or copying `config.example.toml` by running `cp config.example.toml config.toml`
NOTE: this message was printed twice to make it more likely to be seen
Build completed successfully in 0:04:21
It seems to be working. Right Sundeep?
Note this is the first time in a while that I'm building rust src outside of bitbake so please let me know if there's a more sensible check.
So, next steps are:
1. determine what stages need the bootstrap and can it be shared if needed.
2. figure out how do we structure the recipes to do the src sharing
(see gcc-shared for what RP says is a bad example)
3. Calculate the download time/space savings and decide if it's worth it.
../Randy
The rust release tarball is very large, so as part of this work, we should look at alternatives to using the main tarball. For 1.88 (rust-native unpack): the rustc-1.88.0-src.tar.xz file is 271MB and the unpacked source is ~4 GB. 2 GB is under the vendor dir, with 750 MB being windows related and not useful. some of the vendor directory source may already be in oe-core as a recipe. The github rust-1.88 tarball: https://github.com/rust-lang/rust/archive/refs/tags/1.88.0.tar.gz is only 31 MB compressed, and 332 MB unpacked. Of course it doesn't contain the llvm-clone which we are hoping to get rid of, tools, docs, and other mostly useful stuff from the src dir. The llvm-clone includes llvm, clang, lldb, compiler-rt, and more. We need to explain this problem to upstream and see if other people are concerned about the large default distribution Some ncdu screens: --- /...nux/rust-native/1.88.0/sources --- /.. 3.9 GiB [##########] /rustc-1.88.0-src 548.7 MiB [# ] /rust-snapshot-components 128.0 KiB [ ] 0001-Define-more-...unknown-linux-g.patch 44.0 KiB [ ] 0001-riscv32-Defi...r-time64-varian.patch 12.0 KiB [ ] rust-oe-selftest.patch 12.0 KiB [ ] revert-link-std-s..._driver-feature.patch 12.0 KiB [ ] repro-issue-fix-w...c-crate-hashmap.patch 4.0 KiB [ ] 0001-src-core-bui...ff-lto-for-rust.patch 4.0 KiB [ ] 0001-libunwind-Us...arget-attribute.patch 4.0 KiB [ ] 0001-Disable-libu...cture-unwinding.patch 4.0 KiB [ ] oeqa-selftest-Inc...cess-sigpipe-ru.patch 4.0 KiB [ ] triagebot.patch --- /...e/1.88.0/sources/rustc-1.88.0-src --- /.. 2.0 GiB [##########] /vendor 1.7 GiB [######## ] /src 174.6 MiB [ ] /tests 45.9 MiB [ ] /library 37.7 MiB [ ] /compiler 804.0 KiB [ ] RELEASES.md 144.0 KiB [ ] Cargo.lock 124.0 KiB [ ] /LICENSES 44.0 KiB [ ] bootstrap.example.toml 12.0 KiB [ ] LICENSE-APACHE 12.0 KiB [ ] license-metadata.json 8.0 KiB [ ] REUSE.toml --- /...0/sources/rustc-1.88.0-src/vendor --- 121.9 MiB [##########] /windows-0.58.0 121.7 MiB [######### ] /windows-0.57.0 115.4 MiB [######### ] /windows-0.59.0 113.4 MiB [######### ] /windows-0.61.1 53.7 MiB [#### ] /winapi-x86_64-pc-windows-gnu-0.4.0 51.5 MiB [#### ] /winapi-i686-pc-windows-gnu-0.4.0 50.9 MiB [#### ] /openssl-src-300.4.2+3.4.1 41.6 MiB [### ] /plotters-0.3.1 31.4 MiB [## ] /windows-bindgen-0.61.0 30.5 MiB [## ] /windows-sys-0.48.0 27.0 MiB [## ] /openssl-src-111.17.0+1.1.1m ... 2.1 MiB [ ] /syn-1.0.85 2.1 MiB [ ] /syn-1.0.86 2.1 MiB [ ] /syn-1.0.89 2.0 MiB [ ] /idna-0.1.5 2.0 MiB [ ] /object-0.36.7 ... $ ls /.../rustc-1.88.0-src/vendor | wc -l 1703 --- /...88.0/sources/rustc-1.88.0-src/src --- /.. 1.0 GiB [##########] /llvm-project 685.1 MiB [###### ] /tools 46.3 MiB [ ] /doc 4.0 MiB [ ] /librustdoc 3.0 MiB [ ] /etc 1.6 MiB [ ] /bootstrap 1.3 MiB [ ] /ci 84.0 KiB [ ] /build_helper 68.0 KiB [ ] /rustdoc-json-types 56.0 KiB [ ] /rustc-std-workspace 48.0 KiB [ ] stage0 8.0 KiB [ ] /gcc 4.0 KiB [ ] README.md 4.0 KiB [ ] version --- /...rustc-1.88.0-src/src/llvm-project --- /.. 409.1 MiB [##########] /clang 390.4 MiB [######### ] /llvm 86.2 MiB [## ] /lldb 50.8 MiB [# ] /compiler-rt 50.2 MiB [# ] /clang-tools-extra 22.8 MiB [ ] /lld 1.0 MiB [ ] /libunwind 72.0 KiB [ ] /runtimes 72.0 KiB [ ] /cmake 40.0 KiB [ ] /llvm-libgcc --- /...ources/rustc-1.88.0-src/src/tools --- /.. 391.1 MiB [##########] /enzyme 194.7 MiB [#### ] /rustc-perf 28.6 MiB [ ] /cargo 24.0 MiB [ ] /clippy 23.9 MiB [ ] /rust-analyzer 10.0 MiB [ ] /rustfmt 10.0 MiB [ ] /miri 624.0 KiB [ ] /compiletest ... $ ls .../rustc-1.88.0-src/src/tools/ | wc -l 48 |