Bug 16276

Summary: Crate fetcher hardcodes URLs
Product: [Build System, Metadata & Runtime] BitBake Reporter: Ross Burton <ross.burton>
Component: bitbakeAssignee: Minsung Cho <ms98.cho>
Status: NEW --- QA Contact:
Severity: enhancement    
Priority: Medium CC: ms98.cho, paul, poky.bs.watcher, poky.watcher, randy.macleod, sundeep.kokkonda
Version: 6.0   
Target Milestone: 6.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Attachments:
Description Flags
fetch2/crate: support configurable registry and index URLs none

Description Ross Burton 2026-05-01 12:06:12 UTC 
The crate fetcher hardcodes behaviour: if the host is crates.io then it hits the CDN directly, otherwise it uses the API URLs to download and list versions.

This is suboptimal, as different crate registries may have different URL structures.

As per https://doc.rust-lang.org/cargo/reference/registry-index.html, the root of a registry should contain config.json that will give us the URLs to use:

$ curl https://index.crates.io/config.json
{
  "dl": "https://static.crates.io/crates",
  "api": "https://crates.io"
}

I suspect we don't want to look up this file on every hit, but we might need to have a way of expressing this information in a configuration variable of some sort so that the crate://host/ URLs can map from the abstract hostname to concrete URLs without having to hardcode.

Whether this is academic or not remains to be seen, we need to find a non-crates.io registry to compare URLs on.
Comment 1 Paul Barker 2026-05-07 14:35:26 UTC 
As an example to compare to, for crates published on my Forgejo instance, https://forge.betafive.net/api/packages/pub/cargo/config.json contains:

{
  "dl": "https://forge.betafive.net/api/packages/pub/cargo/api/v1/crates",
  "api": "https://forge.betafive.net/api/packages/pub/cargo",
  "auth-required": false
}
Comment 2 Randy MacLeod 2026-05-07 14:38:12 UTC 
A nice improvement but not urgent.
Comment 3 Minsung Cho 2026-05-14 05:24:22 UTC 
Created attachment 5215 [details]
fetch2/crate: support configurable registry and index URLs

This adds CRATE_REGISTRY_URL[host] and CRATE_INDEX_URL[host] support with {crate}, {version}, and {index_path} placeholders. The default crates.io behavior is preserved, while API-style custom registries and custom sparse registries are supported. Sparse registries are tracked explicitly so latest-version checks use the Cargo sparse index parser.

Verification:
- BB_SKIP_NETTESTS=yes python3 -m unittest bb.tests.fetch.CrateTest
  - Ran 12 tests, OK (5 skipped)
- crates.io regression tests passed
- python3 -m py_compile lib/bb/fetch2/crate.py lib/bb/tests/fetch.py
- git diff --check HEAD

Also added a dummy local sparse registry test that serves a generated crate archive and sparse index via HTTPService, then verifies latest-version lookup, checksum validation, download, and unpack.
Comment 4 Paul Barker 2026-05-14 14:41:36 UTC 
Hi Minsung Cho,

Thanks for taking a look at this and attaching a patch! Are you able to send this to the bitbake-devel@lists.openembedded.org mailing list via `git send-email` or `b4 send` (https://b4.docs.kernel.org/en/latest/contributor/send.html, does not need a working SMTP server)?

Can we namespace the variables with BB_ prefixes? i.e. BB_CRATE_REGISTRY_URL and BB_CRATE_INDEX_URL.
Comment 5 Minsung Cho 2026-05-14 20:11:43 UTC 
Thanks for reviewing it so quick!
I sent the new patch to the mailing list.

Changes from the original submission:
- Renamed the new variables to use the requested BB_ namespace.
- BB_CRATE_REGISTRY_URL[host]
- BB_CRATE_INDEX_URL[host]