13095 2018-12-19 14:01:02 +0000 Use Django ORM instead of sqlite directly 2023-10-18 20:55:20 +0000 1 1 6 Yocto Project Subprojects Security Response Tool General unspecified x86 Multiple RESOLVED NOTABUG Low normal 4.3 M4 1 ross.burton david.reyna randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 82536 0 ross.burton 2018-12-19 14:01:02 +0000 The NIST/MITRE/etc fetching scripts write to the sqlite data directly, using a generated Python file to expose name/column index mappings. Instead, these tools should just import the database model and use the standard Django interfaces, so they don't rely on the underlying data model. 82539 1 david.reyna 2018-12-19 18:41:48 +0000 These database actions were originally written in Django. They were explicitly moved to command line scripts away from Django, because: 1) The scripts literally run an order of magnitude faster that the Django wrappers. What takes an hour now took half a day or more before. 2) The command line scripts are much more compatible with background update scripts and CRON jobs, a crucial requirement. - David 84429 2 ross.burton 2019-07-16 09:13:39 +0000 Coming back to this, to productise srtool sqlite really isn't up to scratch. If the update hooks were just management hooks then calling them from cron is trivial. 85135 3 ross.burton 2019-10-01 15:08:54 +0000 Another good reason to use the ORM directly is that it lets us switch from sqlite to postresql. 96721 4 david.reyna 2023-10-18 20:55:20 +0000 Per my previous comment, the pure Django implementation is impossibly slow for the magnitude of CVE and defect records. To accommodate the underlying database model for these backend scripts, we have added the script "bin/common/srtool_sqa.py" to provide the needed abstraction. The databases SQLite, Postgres, and MySQL are supported. In addition, we have provided a migration script to move data from the original SQLite to for example Postgres. The migration script also works the other way, for example Postgres to SQLite. The reason is that is it imperative to be able to back up the database in case of errors or host crashes. Postgres does not have a model for backing up its data base other that generating SQL statements. This migration script does exactly that but puts it into an SQLite database, which given that it is one file is easy to preserve remotely, plus it is a form that can be directly accessed bu the SRTool. David