14765 2022-03-19 02:48:55 +0000 toaster: race condition for end-of-build 2022-03-21 20:55:30 +0000 1 1 7 Build System, Metadata & Runtime Toaster toaster 3.4.3 x86 Multiple RESOLVED FIXED High critical 4.0 M4 1 david.reyna david.reyna toaster --- --- No (bug/feature does not impact docs) oldest_to_newest 92818 0 david.reyna 2022-03-19 02:48:55 +0000 With the introduction of Django-3, there is a new database commit race condition. At the end of short builds (quilt-native, re-build of a previous build), the behavior is the same as before. However for long builds, the event handler for "bb.event.BuildCompleted" sets the build record's status to 'success', but that change does not make it to the actual database, resulting in a 'finished' build visually stuck at 100%. The solution is to introduce this code: def update_build_stats_and_outcome(self, build, errors, warnings, taskfailures): ... # We force a sync point here to force the outcome status commit, # which resolves a race condition with the build completion takedown transaction.set_autocommit(True) transaction.set_autocommit(False) The problem is set up by the fact that Toaster has SQLite has autocommit normally turned off so that Toaster can wait to catch up with the flood of events and avoid a database lock. But in this case, if the commit queue is too long as per a full build, the build worker threads could be killed before the queue is finish and lose late commits like this one. 92826 1 david.reyna 2022-03-21 20:55:30 +0000 Resolved. Commit: c71a9c89e3c23ba1f1eec06c0183ee1d7148041d