Bug 13207

Summary: Toaster broken with recent SQLite
Product: [Build System, Metadata & Runtime] Toaster Reporter: BogDan Vatra <bogdan>
Component: toasterAssignee: David Reyna <david.reyna>
Status: RESOLVED FIXED QA Contact: Toaster default assignee <toaster>
Severity: major    
Priority: Medium+ CC: amber.n.elliot, jessica.zhang, stanciux.mihail, tim.orling
Version: 2.6.2   
Target Milestone: 3.1 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
poky/bitbake/lib/toaster/orm/fixtures/poky.xml
none
Workaround patch for Django 1.11 none

Description BogDan Vatra 2019-02-27 11:32:52 UTC
Hi,

I tried toaster from *master* and *thud* branches and in both branches it fails to initialize properly the database, below is the log.
The problem is that in the admin area (http://127.0.0.1:8000/admin/orm/) everything is empty, there are no Bitbake versions, no Layer_ versions and no Releases, and creating a new project won't work.


[quote]
....
  Applying sessions.0001_initial... OK

Build configuration saved
Loading default settings
Installed 7 object(s) from 1 fixture(s)
Loading poky configuration
Failure while trying to setup toaster: Problem installing fixtures: no such table: orm_build__old
Traceback (most recent call last):
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: no such table: orm_build__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/yocto/poky/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py", line 84, in _verify_be
    call_command("loaddata", "poky")
  File "/home/yocto/.local/lib/python3.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 115, in loaddata
    connection.check_constraints(table_names=table_names)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 286, in check_constraints
    column_name, referenced_column_name,
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/yocto/.local/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: Problem installing fixtures: no such table: orm_build__old
Starting webserver...
Toaster development webserver started at http://localhost:8000

You can now run 'bitbake <target>' on the command line and monitor your build in Toaster.
You can also use a Toaster project to configure and run a build.
[/quote]
Comment 1 David Reyna 2019-04-15 04:17:08 UTC
This is quite mysterious.

The Toaster database is failing when loading the "poky.xml" fixture file, but there is no reference to "orm_build" in that file, let alone "orm_build__old", nor have we ever put one in since we do not preset data in the build table at startup.

The notation "orm_build__old" in Django speak generally means the "old" structure within the "orm_build" class, which does not exist in the schema (there is at most a non-db transient member called old_state).

The fact that the orm structure ends up empty is because the initialization process failed at the fixture process phase, so no data was ever fetched from teh Layer Index.

I am wondering if the local copy of "orm/fixtures/poky.xml" was somehow corrupted. 

Please attached a copy of "orm/fixtures/poky.xml" so that we can see.
Comment 2 BogDan Vatra 2019-04-15 06:33:12 UTC
Created attachment 4479 [details]
poky/bitbake/lib/toaster/orm/fixtures/poky.xml

Attached the poky/bitbake/lib/toaster/orm/fixtures/poky.xml file, I also tar.xz the entire folder here https://drive.google.com/open?id=1Z1LGe0n76nTorSreskF4ZA_8eht0ILvL .The archive will be available for a few weeks.
Comment 3 David Reyna 2019-05-23 05:47:20 UTC
Thanks, I got the archive.
Comment 4 David Reyna 2019-11-07 16:12:50 UTC
This defect is due to the fact that the latest SQLite breaks Django, which is why I could not reproduce it.

* [Django] #29182: SQLite 3.26 breaks database migration ForeignKey constraint, leaving <table_name>__old in db schema

A patch set is in progress.

- David
Comment 5 David Reyna 2019-11-07 16:15:11 UTC
Note that this also requires an update to Django 2.2, since the Django community will not backport the patch to Django 1.11.

https://groups.google.com/forum/#!topic/django-updates/htVqH24DImg
https://github.com/django/django/pull/11986
https://github.com/django/django/commit/c8ffdbe514b55ff5c9a2b8cb8bbdf2d3978c188f
https://github.com/django/django/pull/10733#issuecomment-456731434

The simple patch to Django 1.11 is included in the above links.
Comment 6 Paul Eggleton 2019-12-16 02:23:58 UTC
Created attachment 4602 [details]
Workaround patch for Django 1.11

I've been hitting this recently. As a quick shortcut, I've attached a patch which you can apply to Django 1.11 in your virtualenv (adapted from the rejected backport patch linked above):

cd /path/to/virtual_env_dir/lib/python*/site-packages
patch -p1 < /path/to/django_sqlite326_fix.patch
Comment 7 Tim Orling 2020-02-01 15:08:26 UTC
Django 1.11 LTS is EOL April 2020. Time to move to 2.2 LTS, which is supported until April 2022.

https://www.djangoproject.com/download/
Comment 8 David Reyna 2020-03-30 18:33:47 UTC
commit 4dabdbe11ddc47a885c7f8cc6b3f26439d40d77c