14745 2022-02-24 21:25:06 +0000 cve-checker update to support NVD json 5.0 format 2023-04-25 08:07:03 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 4.0 x86 Multiple RESOLVED FIXED Medium+ major 4.2 M4 1 akuster808 rybczynska akuster david.reyna glimsdal meta.mr.watcher meta.watcher Qi.Chen ralphs randy.macleod richard.purdie ross.burton theslothuk --- --- Don't know oldest_to_newest 92669 0 akuster808 2022-02-24 21:25:06 +0000 The NVD data format is upgrading to json 5.0 in late Spring. https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record 92670 1 akuster808 2022-02-24 21:26:45 +0000 I am going to have to deal with this for my Employer so maybe I can help in migrate the checker. 92698 2 akuster808 2022-03-01 19:57:05 +0000 CVE consumers should also be getting familiar with the CVE JSON 5.0 format. They will begin seeing JSON 5.0 CVE Records published on the CVE website in the JSON 5.0 format by late spring 2022. As a result of the transition to JSON 5.0, beginning in summer 2022, the CVE List download options currently provided by the CVE Program (i.e., CSV, HTML, XML and CVRF) will no longer be supported. After the JSON 4.0 retirement date (which will be announced/confirmed in spring 2022), the CVE List (as of that date) in all JSON 4.0 supported formats will be available in an archive and the CVE List will be downloadable only in JSON 5.0 format. 92724 3 akuster808 2022-03-03 20:17:53 +0000 My thought process on this is to figure out 4.0 -> 5.0 delta as it applies to the bbclass. We may want to define a variable which sets the version to use during the transition phase so we can expand the testing in the community prior to permanently changing. it gives us a fail back if NVD runs into problems and allow the sharing of code changes. 92741 4 glimsdal 2022-03-08 17:59:06 +0000 I'll take a look at this. It's a little unclear on the NVD website where a new JSON API is. I see an announcement for deprecating a SOAP API, but it just points to the endpoint already used by cve-update-db-native. 92956 5 akuster808 2022-04-07 21:41:05 +0000 Looks like some 5.0 json data is now available. https://github.com/CVEProject/cvelistV5/tree/master/review_set 92957 6 akuster808 2022-04-07 21:41:19 +0000 Updated Transition Schedule See Transition Bulletin #2 for descriptions of activities 1-6; descriptions for activities 7 and 8 are above. Item Activity Schedule 1 CVE Services 2.1 Community Testing in the Testing Instance Late February/early March 2022 2 CVE Services 2.1 Community Penetration Testing Mid-to-late March 2022 (will run for approximately 3 weeks) 3 CNA Community Review of Historical JSON 5.0 Records January through April 2022 4 CVE Services 2.1/JSON 5.0 Soft Deployment Mid-to-late April 2022 (will span approximately 3 weeks) 5 CVE Services 2.1/JSON 5.0 Hard Deployment Late April 2022 6 JSON 4.0 Retirement July 2022 (3 months after JSON 5.0 is made available to CVE downstream users) 7 CVE List GitHub Submission Pilot Retirement July 2022 (3 months after JSON 5.0 is made available to CVE downstream users) 8 CVE List Download Format Changes July 2022 (3 months after JSON 5.0 is made available to CVE downstream users) 92958 7 akuster808 2022-04-07 21:42:27 +0000 Bulletin Number 2 CVE Services Transition — February 11, 2022 https://cveproject.github.io/automation-transition#bulletin-number-2 92959 8 akuster808 2022-04-07 21:44:35 +0000 (In reply to comment #4) > I'll take a look at this. It's a little unclear on the NVD website where a > new JSON API is. I see an announcement for deprecating a SOAP API, but it > just points to the endpoint already used by cve-update-db-native. The submission web i/f is migrating too to version 2.1 but we don't do anything with that from a the checker side. 92961 9 ross.burton 2022-04-08 15:19:01 +0000 So my reading of this is that this is a change to the cve.org (previously mitre) feeds. Our tool fetches from nvd.nist.gov which has its own feeds and doesn't talk about the format changing: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED We should evaluate whether we should be fetching from cve.org instead of nist, but this isn't as important as it seemed. 94624 10 randy.macleod 2023-01-05 15:53:03 +0000 Marta, Based on your comments sent to the oe-core email list, it seems that you might be a better owner for this issue. 94628 11 ross.burton 2023-01-05 16:50:16 +0000 So since I left that comment the page has been updated and now talks about the feeds being retired in late 2023... https://nvd.nist.gov/developers/api-workflows talks about the new suggested workflow. Marta: is this something you'll be willing to work on, or should I add it to my todo list? 94629 12 rybczynska 2023-01-05 17:09:40 +0000 Ross, Yes, I plan to work on it. The first step will be to find out if it is better to go for CVE 5 or NVD API 2.0. 94630 13 ross.burton 2023-01-05 18:26:09 +0000 I got curious/distracted and hacked a bit: https://pastebin.com/1ZHregqF Conclusion so far is that the API is nice, but slow. A full fetch of the old JSON files took me just under two minutes, but a full fetch with the v2 JSON REST API has been running for a several minutes and it's still going. That said, the incremental fetch should be a lot more reliable so that's a tradeoff that I'm fine with. 94631 14 ross.burton 2023-01-05 18:42:44 +0000 15 minutes for a full fetch with the new API! 94637 15 rybczynska 2023-01-06 16:07:37 +0000 15 minutes for me too. I've send a message to NVD asking if they might considers a downloadable version. Looked at the CVE 5.0 too. The schema itself contains enough information. However, there is no feed yet. And the 4.0 version does not contain the information we need. 94930 16 rybczynska 2023-02-15 11:13:30 +0000 I've just figured out that you can use the API without a key. That simplifies things *a lot*. There's only a different load limiter. 94931 17 ross.burton 2023-02-16 11:24:17 +0000 That's useful. I guess we should support people using it without and without a key. 94932 18 rybczynska 2023-02-16 11:26:44 +0000 Yes, with an option to give the key. I can now merge two code bases into one. Without the need of mirroring, it will be easier to deploy. 95317 19 theslothuk 2023-04-07 12:59:15 +0000 Hi all. I found this thread after researching how much of an issue the NVD dumping their json feeds might be for me. If this is going in Sept 2023: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED Does anyone know if there is a way to replicate it, using the API? Thanks. 95450 20 rybczynska 2023-04-25 08:06:14 +0000 The new fetcher has been merged http://git.openembedded.org/openembedded-core/commit/meta/recipes-core/meta/cve-update-nvd2-native.bb?id=fb62c4c3dbca4e58f7ce6cf29d4b630a06411a97 95451 21 rybczynska 2023-04-25 08:07:03 +0000 (In reply to Bob from comment #19) > Hi all. I found this thread after researching how much of an issue the NVD > dumping their json feeds might be for me. > > If this is going in Sept 2023: > > https://nvd.nist.gov/vuln/data-feeds#JSON_FEED > > Does anyone know if there is a way to replicate it, using the API? > > Thanks. Bob, do you have a reason to dump the feed separately when the new class is available?