| Summary: | gitsm+https: is not supported by SPDX 2.2 nor 2.3 specifications | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Billie Alsup <balsup> | ||||||
| Component: | oe-core other | Assignee: | Joshua Watt <JPEWhacker> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | Medium+ | CC: | balsup, randy.macleod | ||||||
| Version: | 5.0 | ||||||||
| Target Milestone: | 5.1 M4 | ||||||||
| Hardware: | All | ||||||||
| OS: | Multiple | ||||||||
| Whiteboard: | |||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||
| Verified: | Documentation change: | Don't know | |||||||
| Attachments: |
|
||||||||
In the short term, we can do a simple remap of gitsm+http:// to just git+http:// which will make the SPDX data not wrong; as far as dealing with the actual submodules, that will take a little more work and thinking. I believe we wouldn't actually need to list the submodules as DEPENDS, since should be scanning the source files contained in the submodules, even if we don't recognize that they are distinct submodules. Add them as explicit dependencies would require some pretty complicated code. We have extracted some code from the gitsm fetcher as suggested, and this is doing the bare minimum. I see that files are not actually listed in the generated SPDX file, so having the additional packages is good. It's not clear to me that having these as explicit independent packages which you could add to DEPENDS will work in the general case, but I certainly don't claim to be an expert in the area. If they were explicit DEPENDS then you would have access to more metadata (e.g. license/copyright/supplier). I will attach a sample output for packages for ovmf recipe, and try to attach the changes we made for same later today. Created attachment 5068 [details]
Updated recipe-ovmf.spdx.json with package per submodule
I think this can be improved by, for example, setting the name from the URL basename perhaps.
Created attachment 5069 [details]
Proof of concept reusing git.py and gitsm.py functions
This is a proof of concept we are using locally. It could be improved to give the package a better name for example, or possibly by looking for license files in the checked out submodule.
This is fixed in 6ecf89c75b1a74515266085acc5d3621a0fb2fa1, at least to the extent that it is no longer objectively wrong. If we would like an improvement to the SPDX code to report submodules in a different way, please open a new bug. Thanks |
gitsm+https: is not a valid URL type for SPDX 2.2 nor SPDX 2.3 specifications. gitsm is the git submodule fetcher nomenclature. Using gitsm as the SPDX download location is wrong not only from the specification point of view, but from a practical standpoint, it means that the submodules are not listed in the SPDX file. The submodules should be treated more like DEPENDS, in that they are their own unique subprojects, possibly with different maintainers, download locations, licenses, copyrights, etc. So I believe that each submodule should be represented as a distinct package within the recipe’s SPDX file. Currently this is not the case. Note that we don't necessarily know all of this information for each submodule, so unknown fields may end up with values of None or NoAssertion. For example, recipe-ovmf references two packages, one of which is the erroneous SPDXRef-Download-ovmf-1 with download location using gitsm+https: nomenclature. I believe that this should actually show up as git+https: nomenclature for the SPDXRef-Download-ovmf-1 package, but there should be additional packages within the SPDX file, as enumerated through git submodule, and/or parsing of .gitmodules. $ git submodule b64af41c3276f97f0e181920400ee056b9c88037 ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 (b64af41) f4153a09f87cbb9c826d8fc12c74642bb2d879ea BaseTools/Source/C/BrotliCompress/brotli (v1.0.9-35-gf4153a0) 8c89224991adff88d53cd380f42a2baa36f91454 CryptoPkg/Library/MbedTlsLib/mbedtls (v3.3.0) de90e54bbe82e5be4fb9608b6f5c308bb837d355 CryptoPkg/Library/OpensslLib/openssl (openssl-3.0.9) f4153a09f87cbb9c826d8fc12c74642bb2d879ea MdeModulePkg/Library/BrotliCustomDecompressLib/brotli (v1.0.9-35-gf4153a0) abfc8ff81df4067f309032467785e06975678f0d MdeModulePkg/Universal/RegularExpressionDxe/oniguruma (v6.9.4_mark1) cfff805481bdea27f900c32698171286542b8d3c MdePkg/Library/BaseFdtLib/libfdt (v1.6.1-3-gcfff805) 370b5944c046bab043dd8b133727b2135af7747a MdePkg/Library/MipiSysTLib/mipisyst (v1.1+edk2) e9ebfa7e77a6bee77df44e096b100e7131044059 RedfishPkg/Library/JsonLib/jansson (v2.13.1) 1cc9cde3448cdd2e000886a26acf1caac2db7cf1 UnitTestFrameworkPkg/Library/CmockaLib/cmocka (cmocka-1.1.5-23-g1cc9cde) 86add13493e5c881d7e4ba77fb91c1f57752b3a4 UnitTestFrameworkPkg/Library/GoogleTestLib/googletest (release-1.8.0-2983-g86add134) 83d4e1ebef3588fae48b69a7352cc21801cb70bc UnitTestFrameworkPkg/Library/SubhookLib/subhook (v0.8.2-12-g83d4e1e) $ $ cat .gitmodules [submodule "CryptoPkg/Library/OpensslLib/openssl"] path = CryptoPkg/Library/OpensslLib/openssl url = https://github.com/openssl/openssl [submodule "SoftFloat"] path = ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 url = https://github.com/ucb-bar/berkeley-softfloat-3.git [submodule "UnitTestFrameworkPkg/Library/CmockaLib/cmocka"] path = UnitTestFrameworkPkg/Library/CmockaLib/cmocka url = https://github.com/tianocore/edk2-cmocka.git [submodule "MdeModulePkg/Universal/RegularExpressionDxe/oniguruma"] path = MdeModulePkg/Universal/RegularExpressionDxe/oniguruma url = https://github.com/kkos/oniguruma [submodule "MdeModulePkg/Library/BrotliCustomDecompressLib/brotli"] path = MdeModulePkg/Library/BrotliCustomDecompressLib/brotli url = https://github.com/google/brotli [submodule "BaseTools/Source/C/BrotliCompress/brotli"] path = BaseTools/Source/C/BrotliCompress/brotli url = https://github.com/google/brotli ignore = untracked [submodule "RedfishPkg/Library/JsonLib/jansson"] path = RedfishPkg/Library/JsonLib/jansson url = https://github.com/akheron/jansson [submodule "UnitTestFrameworkPkg/Library/GoogleTestLib/googletest"] path = UnitTestFrameworkPkg/Library/GoogleTestLib/googletest url = https://github.com/google/googletest.git [submodule "UnitTestFrameworkPkg/Library/SubhookLib/subhook"] path = UnitTestFrameworkPkg/Library/SubhookLib/subhook url = https://github.com/Zeex/subhook.git [submodule "MdePkg/Library/BaseFdtLib/libfdt"] path = MdePkg/Library/BaseFdtLib/libfdt url = https://github.com/devicetree-org/pylibfdt.git [submodule "MdePkg/Library/MipiSysTLib/mipisyst"] path = MdePkg/Library/MipiSysTLib/mipisyst url = https://github.com/MIPI-Alliance/public-mipi-sys-t.git [submodule "CryptoPkg/Library/MbedTlsLib/mbedtls"] path = CryptoPkg/Library/MbedTlsLib/mbedtls url = https://github.com/ARMmbed/mbedtls $ I propose that create-spdx.bbclass iterate the submodules and/or .gitmodules in a manner similar to the gitsm fetcher, and create distinct packages for each submodule, with distinct files, and add appropriate relationships between the packages and files.