Bug 15909

Summary: crate://crates.io is not a valid download_location for SPDX 2.2
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Billie Alsup <balsup>
Component: oe-core otherAssignee: Jayasurya Maganuru <Maganuru.Jayasurya>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: ccasciato, JPEWhacker, Maganuru.Jayasurya, randy.macleod, sundeep.kokkonda
Version: 5.0.15   
Target Milestone: 5.3 M4   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
Workaround for crate fetcher URLs being invalid SPDX URLs none

Description Billie Alsup 2025-06-18 04:36:25 UTC 
After moving from dunfell to scarthgap, I am seeing validation errors in the generated SPDX files, e.g.

[sbom] package download_location must be a valid URL or download location according to the specification, but is: crate://crates.io/windows_i686_gnu/0.52.3
[sbom] package download_location must be a valid URL or download location according to the specification, but is: crate://crates.io/windows_aarch64_msvc/0.52.3
[sbom] package download_location must be a valid URL or download location according to the specification, but is: crate://crates.io/windows_aarch64_gnullvm/0.52.3

I am also finding that SPDX generation behavior has changed between dunfell and scarthgap, with SPDX files not being generated where expected.  This seems related to certain tasks not being built if they are only runtime dependencies (possibly related to https://git.openembedded.org/openembedded-core/commit/?id=220d29a5abb5be8b80c64d8f2f48949bc8c534a3 ).  A workaround seems to be explicitly adding 

--runall create_runtime_spdx --runall create_spdx

to the bitbake command (similar to the workaround suggested in the given url to use --runall build)
Comment 1 Billie Alsup 2025-06-18 04:37:41 UTC 
I presume create-spdx class should massage the url similar to how the fetcher does it, e.g.

        if host == 'crates.io':
            host = 'crates.io/api/v1/crates'

        ud.url = "https://%s/%s/%s/download" % (host, name, version)
Comment 2 Billie Alsup 2025-06-18 15:36:25 UTC 
Should the fetcher itself support a function to translate a given (fetcher) URL into a list of download locations?  This problem first presented itself with gitsm, and there is a translation in fetch_data_to_uri (although it seems to simply ignore the submodules).  There are bound to be fetchers in the future that do their own thing, so supporting an optional fetcher method to return a list of download locations (or perhaps mappings between download locations and sets of downloaded files) would seem to make sense, rather than putting exceptions into spdx_common.py.
Comment 3 Randy MacLeod 2025-06-19 14:40:46 UTC 
Internally spdx has mappings of varioius URLS.
Rust wasn't in dunfell so it's likely that there's just a missing mapping for crates.io.

So yes, we should have a translation function as Billie suggested.
Can you do that Billie ?

-- YP bug triage
Comment 4 Billie Alsup 2025-06-23 15:52:59 UTC 
Created attachment 5124 [details]
Workaround for crate fetcher URLs being invalid SPDX URLs
Comment 5 Billie Alsup 2025-06-23 15:55:18 UTC 
Let me get the required approvals here, and then I'll take a look at a long term solution.  In the meantime, I've attached our scarthgap workaround.
Comment 6 Randy MacLeod 2025-06-26 17:53:32 UTC 
Hi Billie,

Thanks for the patch / workaround. 
It might help someone who happens to read this defect.

I'll move this bug to accepted and hopefully it won't take too long for you to get local approval to contribute to Yocto so the patch can be merged to master and then back-ported as needed.
Comment 7 Billie Alsup 2025-08-05 15:21:31 UTC 
I am still in the waiting for business approval state (which is unlikely to come until I finish some internal projects).  The given workaround was sufficient for us to proceed, and deprioritized work to have a generic solution going forward.
Comment 8 Randy MacLeod 2025-08-14 16:24:23 UTC 
Joshua, 
Should Sundeep have someone submit the work-around or is there a better approach?
Comment 9 Joshua Watt 2025-09-18 15:04:45 UTC 
Yes, the munging of the URI is correct for crates, but please do it in the fetch_data_to_uri function in spdx_common.py. Ideally, the actual URI comes from the fetcher and we just need to know to use that for crates (or others)
Comment 10 Joshua Watt 2025-09-25 15:07:07 UTC 
See above comment
Comment 11 Jayasurya Maganuru 2025-10-14 13:00:58 UTC 
(In reply to Joshua Watt from comment #10)
> See above comment

Hi Joshua, 

I've made the equivalent changes of the workaround, to fetch_data_to_uri function of spdx_common.py and manually verified the generated spdx.json (for the recipe python3-bcrypt).

The software_downloadLocation is now correctly set to a valid URL, e.g.:
"software_downloadLocation": "https://crates.io/api/v1/crates/windows_i686_gnu/0.48.5/download"
instead of the invalid crate://crates.io/... format previously used for Rust crates.

Should I be validating the spdx.json output using any specific tool before sending a patch to poky master? Or is this manual check sufficient to move forward?