Bug 14765 - toaster: race condition for end-of-build
Summary: toaster: race condition for end-of-build
Status: RESOLVED FIXED
Alias: None
Product: Toaster
Classification: Build System, Metadata & Runtime
Component: toaster (show other bugs)
Version: 3.4.3
Hardware: x86 Multiple
: High critical
Target Milestone: 4.0 M4
Assignee: David Reyna
QA Contact: Toaster default assignee
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2022-03-19 02:48 UTC by David Reyna
Modified: 2022-03-21 20:55 UTC (History)
0 users

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Reyna 2022-03-19 02:48:55 UTC
With the introduction of Django-3, there is a new database commit race condition.

At the end of short builds (quilt-native, re-build of a previous build), the behavior is the same as before.

However for long builds, the event handler for "bb.event.BuildCompleted" sets the build record's status to 'success', but that change does not make it to the actual database, resulting in a 'finished' build visually stuck at 100%.

The solution is to introduce this code:

    def update_build_stats_and_outcome(self, build, errors, warnings, taskfailures):
        ...

        # We force a sync point here to force the outcome status commit,
        # which resolves a race condition with the build completion takedown
        transaction.set_autocommit(True)
        transaction.set_autocommit(False)

The problem is set up by the fact that Toaster has SQLite has autocommit normally turned off so that Toaster can wait to catch up with the flood of events and avoid a database lock. But in this case, if the commit queue is too long as per a full build, the build worker threads could be killed before the queue is finish and lose late commits like this one.
Comment 1 David Reyna 2022-03-21 20:55:30 UTC
Resolved.

Commit: c71a9c89e3c23ba1f1eec06c0183ee1d7148041d