Bug 14765

Summary: toaster: race condition for end-of-build
Product: [Build System, Metadata & Runtime] Toaster Reporter: David Reyna <david.reyna>
Component: toasterAssignee: David Reyna <david.reyna>
Status: RESOLVED FIXED QA Contact: Toaster default assignee <toaster>
Severity: critical    
Priority: High    
Version: 3.4.3   
Target Milestone: 4.0 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description David Reyna 2022-03-19 02:48:55 UTC
With the introduction of Django-3, there is a new database commit race condition.

At the end of short builds (quilt-native, re-build of a previous build), the behavior is the same as before.

However for long builds, the event handler for "bb.event.BuildCompleted" sets the build record's status to 'success', but that change does not make it to the actual database, resulting in a 'finished' build visually stuck at 100%.

The solution is to introduce this code:

    def update_build_stats_and_outcome(self, build, errors, warnings, taskfailures):
        ...

        # We force a sync point here to force the outcome status commit,
        # which resolves a race condition with the build completion takedown
        transaction.set_autocommit(True)
        transaction.set_autocommit(False)

The problem is set up by the fact that Toaster has SQLite has autocommit normally turned off so that Toaster can wait to catch up with the flood of events and avoid a database lock. But in this case, if the commit queue is too long as per a full build, the build worker threads could be killed before the queue is finish and lose late commits like this one.
Comment 1 David Reyna 2022-03-21 20:55:30 UTC
Resolved.

Commit: c71a9c89e3c23ba1f1eec06c0183ee1d7148041d