Bug 8918

Summary: Change the way we handle queued builds
Product: [Build System, Metadata & Runtime] Toaster Reporter: Belen Barros Pena <belen.barros.pena>
Component: toasterAssignee: Ed Bartosh <eduard.bartosh>
Status: RESOLVED FIXED QA Contact: Libertad <libertad.gonzalez.de.la.cruz>
Severity: normal    
Priority: Medium CC: belen.barros.pena, elliot.smith, jessica.zhang, stanciux.mihail
Version: 2.0   
Target Milestone: 2.2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Belen Barros Pena 2016-01-13 17:53:23 UTC 
Toaster allows you to start as many builds you want. If the build directory for a build you start is busy, Toaster will keep the build queued and won't start it until the build directory becomes available.

This is neat. The problem is that currently we try to work out if the build directory is available by querying the database periodically. This causes database locking issues in sqlite, because while a build is running Toaster continuously writes build data to the database, and so we have 2 processes trying to access the database at the same time.

We should improve the way we handle the queued builds to avoid database locking.
Comment 1 Elliot Smith 2016-01-20 08:26:34 UTC 
This was discussed in the Toaster weekly meeting 13th January 2016.

Brian suggested that instead of polling the db (our current approach), we look at the queue of build requests when a build completes.

To get the first build going, we just run it as soon as it is requested, providing there are no ongoing builds. If there are ongoing builds, we add the new request to the queue and trigger it when the current build completes. Django signals might be a way of listening for build completion.

Note that the underlying issue is the one we ran across when upgrading to Django 1.8: we have multiple processes accessing a SQLite database in parallel, contending for locks on the database file. A longer-term goal might be to change how we access the database, so that we have a single point of access when using SQLite, while still allowing multiple points of access for other database. This would require *significant* reworking of the code base, though.

Another side issue is that we have separate build requests and build tables. This seems unnecessary, and makes it complicated for us to do things like showing queued requests (as we have to combine records from two database tables). There's no reason we couldn't just get rid of build requests and have Build objects with a "queued" status instead. Again, this might make the above simpler.
Comment 2 Ed Bartosh 2016-09-12 12:13:22 UTC 
Using django signals for this is impossible as they work only in one django process. We'll be sending POSIX signal SIGUSR1 to runbuilds and will poll database only when signal received.
Comment 3 Ed Bartosh 2016-09-13 13:31:34 UTC 
Implemented in this patchset: https://lists.yoctoproject.org/pipermail/toaster/2016-September/005179.html

Please, review
Comment 4 Ed Bartosh 2016-09-22 08:15:42 UTC 
The fix has been merged into poky master.