The NVD data format is upgrading to json 5.0 in late Spring. https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record
I am going to have to deal with this for my Employer so maybe I can help in migrate the checker.
CVE consumers should also be getting familiar with the CVE JSON 5.0 format. They will begin seeing JSON 5.0 CVE Records published on the CVE website in the JSON 5.0 format by late spring 2022. As a result of the transition to JSON 5.0, beginning in summer 2022, the CVE List download options currently provided by the CVE Program (i.e., CSV, HTML, XML and CVRF) will no longer be supported. After the JSON 4.0 retirement date (which will be announced/confirmed in spring 2022), the CVE List (as of that date) in all JSON 4.0 supported formats will be available in an archive and the CVE List will be downloadable only in JSON 5.0 format.
My thought process on this is to figure out 4.0 -> 5.0 delta as it applies to the bbclass. We may want to define a variable which sets the version to use during the transition phase so we can expand the testing in the community prior to permanently changing. it gives us a fail back if NVD runs into problems and allow the sharing of code changes.
I'll take a look at this. It's a little unclear on the NVD website where a new JSON API is. I see an announcement for deprecating a SOAP API, but it just points to the endpoint already used by cve-update-db-native.
Looks like some 5.0 json data is now available. https://github.com/CVEProject/cvelistV5/tree/master/review_set
Updated Transition Schedule See Transition Bulletin #2 for descriptions of activities 1-6; descriptions for activities 7 and 8 are above. Item Activity Schedule 1 CVE Services 2.1 Community Testing in the Testing Instance Late February/early March 2022 2 CVE Services 2.1 Community Penetration Testing Mid-to-late March 2022 (will run for approximately 3 weeks) 3 CNA Community Review of Historical JSON 5.0 Records January through April 2022 4 CVE Services 2.1/JSON 5.0 Soft Deployment Mid-to-late April 2022 (will span approximately 3 weeks) 5 CVE Services 2.1/JSON 5.0 Hard Deployment Late April 2022 6 JSON 4.0 Retirement July 2022 (3 months after JSON 5.0 is made available to CVE downstream users) 7 CVE List GitHub Submission Pilot Retirement July 2022 (3 months after JSON 5.0 is made available to CVE downstream users) 8 CVE List Download Format Changes July 2022 (3 months after JSON 5.0 is made available to CVE downstream users)
Bulletin Number 2 CVE Services Transition — February 11, 2022 https://cveproject.github.io/automation-transition#bulletin-number-2
(In reply to comment #4) > I'll take a look at this. It's a little unclear on the NVD website where a > new JSON API is. I see an announcement for deprecating a SOAP API, but it > just points to the endpoint already used by cve-update-db-native. The submission web i/f is migrating too to version 2.1 but we don't do anything with that from a the checker side.
So my reading of this is that this is a change to the cve.org (previously mitre) feeds. Our tool fetches from nvd.nist.gov which has its own feeds and doesn't talk about the format changing: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED We should evaluate whether we should be fetching from cve.org instead of nist, but this isn't as important as it seemed.
Marta, Based on your comments sent to the oe-core email list, it seems that you might be a better owner for this issue.
So since I left that comment the page has been updated and now talks about the feeds being retired in late 2023... https://nvd.nist.gov/developers/api-workflows talks about the new suggested workflow. Marta: is this something you'll be willing to work on, or should I add it to my todo list?
Ross, Yes, I plan to work on it. The first step will be to find out if it is better to go for CVE 5 or NVD API 2.0.
I got curious/distracted and hacked a bit: https://pastebin.com/1ZHregqF Conclusion so far is that the API is nice, but slow. A full fetch of the old JSON files took me just under two minutes, but a full fetch with the v2 JSON REST API has been running for a several minutes and it's still going. That said, the incremental fetch should be a lot more reliable so that's a tradeoff that I'm fine with.
15 minutes for a full fetch with the new API!
15 minutes for me too. I've send a message to NVD asking if they might considers a downloadable version. Looked at the CVE 5.0 too. The schema itself contains enough information. However, there is no feed yet. And the 4.0 version does not contain the information we need.
I've just figured out that you can use the API without a key. That simplifies things *a lot*. There's only a different load limiter.
That's useful. I guess we should support people using it without and without a key.
Yes, with an option to give the key. I can now merge two code bases into one. Without the need of mirroring, it will be easier to deploy.
Hi all. I found this thread after researching how much of an issue the NVD dumping their json feeds might be for me. If this is going in Sept 2023: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED Does anyone know if there is a way to replicate it, using the API? Thanks.
The new fetcher has been merged http://git.openembedded.org/openembedded-core/commit/meta/recipes-core/meta/cve-update-nvd2-native.bb?id=fb62c4c3dbca4e58f7ce6cf29d4b630a06411a97
(In reply to Bob from comment #19) > Hi all. I found this thread after researching how much of an issue the NVD > dumping their json feeds might be for me. > > If this is going in Sept 2023: > > https://nvd.nist.gov/vuln/data-feeds#JSON_FEED > > Does anyone know if there is a way to replicate it, using the API? > > Thanks. Bob, do you have a reason to dump the feed separately when the new class is available?