The crate fetcher hardcodes behaviour: if the host is crates.io then it hits the CDN directly, otherwise it uses the API URLs to download and list versions. This is suboptimal, as different crate registries may have different URL structures. As per https://doc.rust-lang.org/cargo/reference/registry-index.html, the root of a registry should contain config.json that will give us the URLs to use: $ curl https://index.crates.io/config.json { "dl": "https://static.crates.io/crates", "api": "https://crates.io" } I suspect we don't want to look up this file on every hit, but we might need to have a way of expressing this information in a configuration variable of some sort so that the crate://host/ URLs can map from the abstract hostname to concrete URLs without having to hardcode. Whether this is academic or not remains to be seen, we need to find a non-crates.io registry to compare URLs on.
As an example to compare to, for crates published on my Forgejo instance, https://forge.betafive.net/api/packages/pub/cargo/config.json contains: { "dl": "https://forge.betafive.net/api/packages/pub/cargo/api/v1/crates", "api": "https://forge.betafive.net/api/packages/pub/cargo", "auth-required": false }
A nice improvement but not urgent.
Created attachment 5215 [details] fetch2/crate: support configurable registry and index URLs This adds CRATE_REGISTRY_URL[host] and CRATE_INDEX_URL[host] support with {crate}, {version}, and {index_path} placeholders. The default crates.io behavior is preserved, while API-style custom registries and custom sparse registries are supported. Sparse registries are tracked explicitly so latest-version checks use the Cargo sparse index parser. Verification: - BB_SKIP_NETTESTS=yes python3 -m unittest bb.tests.fetch.CrateTest - Ran 12 tests, OK (5 skipped) - crates.io regression tests passed - python3 -m py_compile lib/bb/fetch2/crate.py lib/bb/tests/fetch.py - git diff --check HEAD Also added a dummy local sparse registry test that serves a generated crate archive and sparse index via HTTPService, then verifies latest-version lookup, checksum validation, download, and unpack.
Hi Minsung Cho, Thanks for taking a look at this and attaching a patch! Are you able to send this to the bitbake-devel@lists.openembedded.org mailing list via `git send-email` or `b4 send` (https://b4.docs.kernel.org/en/latest/contributor/send.html, does not need a working SMTP server)? Can we namespace the variables with BB_ prefixes? i.e. BB_CRATE_REGISTRY_URL and BB_CRATE_INDEX_URL.
Thanks for reviewing it so quick! I sent the new patch to the mailing list. Changes from the original submission: - Renamed the new variables to use the requested BB_ namespace. - BB_CRATE_REGISTRY_URL[host] - BB_CRATE_INDEX_URL[host]