Created attachment 3130 [details] browser console output Steps to reproduce: 1. Create a Toaster project 2. Build an image (something big might give you a better chance to reproduce the problem: core-image-sato, for example, although I've seen it with core-image-minimal as well) 3. When the build completes, go to the "New custom image" page 4. Search for the image you built in step 2 and click its "customise" button 5. Enter a name for your new custom image and click the "Create custom image" button Nothing happens. The dialog is not dismissed, and no error is shown, but the custom image is created (you can see it if you navigate to the "custom images" page). I've attached the browser console log output. I have not been able to reproduce this reliable (sometimes it happens, sometimes it doesn't), but it seems to happen more often when you select a big image as the base image, or when you have a build running at the time of creating the custom image.
As discussed in the Toaster weekly meeting, this appears to be caused by database locking (again). The solution discussed involved serialising db access, possibly using Celery.
An update on work done on this so far: * I refactored buildinfohelper to channel all database writes through a DatabaseWriter class. * I looked into using Celery to queue database writes. To do this, I made use of redislite as the Celery broker. This means that a Redis server can be embedded into Toaster, rather than requiring a separate Redis server start. I then reworked DatabaseWriter to push database write tasks onto a Celery queue. Issues which have come up so far: (1) My naive approach was to attempt to let Celery handle database writes asynchronously. This doesn't work, of course, as if buildinfohelper writes record A to the database then records B, C, D afterwards (for example), records B, C and D may depend on A (e.g. if adding tasks to a build). This means that the database writes can't all be delayed, but have to be done in batches. Celery provides an API for batching related tasks together using chain(), but I haven't attempted this yet, mainly because of (2)... (2) redislite seems buggy and unreliable. It will occasionally randomly crash, for some reason pulling running apps on my laptop down with it (e.g. Chrome, rhythmbox). It also hogs so many system resources that bash can become unable to fork processes (I've never seen any other program do this). For these reasons, I don't think redislite is a reliable or practical solution, as it could introduce numerous other problems. The alternative to redislite is to use Redis proper. However, this is not trivial to set up, and couldn't be easily done within our current Toaster startup sequence. It would require a user to download and configure Redis themselves. Also, if we need a Redis server to be set up, we may as well get a user to set up MySQL, as this will fix the database locking issues anyway, without needing Celery. The only other stable Celery broker is RabbitMQ, which also requires some setup. There are unstable brokers, but these have big warnings and caveats listed (http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html). So, my conclusion is that we have a few options: A. The only reliable way to use Celery would be to install a proper, standalone server as a broker. We would also need to do more work to modify Toaster so that it batches related database writes sensibly. B. As stated above, the db locking we see could also be solved by setting up MySQL. This is no more complicated than setting up Celery + Redis/RabbitMQ. C. We could create our own solution within Toaster to deal with this (I had a thought about running a separate REST API or Python app which all db writes are routed through). This way we could maintain Toaster as a standalone app which uses SQLite and doesn't need any complicated setup for Celery/MySQL etc.
I'm upping my estimate, as I'm not sure the approach I've taken so far will make much difference to db locking. Django is perfectly capable of causing db locking to happen on its own, even if other processes accessing the database are replaced or removed. I think I need to restart. It might make more sense to deal with this in steps instead: 1. Figure out a way to compare two database states so it's possible to determine whether changes made during refactoring of buildinfohelper are breaking stuff. I missed this step on my first pass, which made it difficult to figure out how something was broken when things started going wrong. 2. Remove runbuilds, replacing its functionality with something in the web app part of Toaster. 3. Provide a REST API for Toaster models using Django REST Framework. 4. Modify buildinfohelper so that it doesn't do any direct db access, and only uses the REST API. By following these steps, we are improving Toaster's architecture and making buildinfohelper easier to refactor, even if we fail to get rid of database locking. I have made progress towards parts 3 and 4 of this, but given the complexity and fragility of buildinfohelper, I think I need to take a more cautious approach, so that I can be sure that I don't break anything as I go along. The work I've done so far looked promising, but I ended up running into problems which were very complex to debug, as I had no frame of reference to understand why stuff was breaking. Completing 1, perhaps by writing a script which can compare the records in two SQLite databases for a particular build target (e.g. core-image-minimal), would enable steps 3 and 4 to be carried out more carefully.
The log file provided by Belen indicates on the occasion when the error occurred and she took the log, database locking caused a 500 error from the server. This would certainly produce the behaviour she noted. Moving to MySQL for the back-end database should alleviate most, if not all, database locking issues due to using SQLite as a back-end.
As a work-around, I will delay all transitions between pages so that the database can "settle" (after database writes) before it's queried for the new page content.
Submitted a patch to just show the loading spinner when creating a custom image: https://lists.yoctoproject.org/pipermail/toaster/2016-July/004998.html
Moving to M3 as the review highlighted an issue which needs to be resolved (both the spinner on the button and the page spinner show at the same time, and we only want one of these two to be visible).
Submitted for review: https://lists.yoctoproject.org/pipermail/toaster/2016-July/005047.html