Bug 16129

I believe I've tracked the issue down to this error in the logs:

retrying <function LogsConnectorComponent.compressLog.<locals>._thd_replace_chunks_by_new_grouped_chunk at 0x7fa5982ca3e0> after sql error (MySQLdb.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction')
	sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction')


I appears that background compression of logchunks is creating deadlocked transactions for active builds. There is no graceful way to break the deadlock and the controller must be killed.

I've set transaction-isolation = READ-COMMITTED in the database default settings which might solve our deadlock issues. Locking only the rows involved in the transaction should allow new logs to continue being written while old logs are compressed. Given our database usage pattern we don't need the protections of the more conservative default REPEATABLE READ transaction isolation.

If deadlocks continue we can disable Buildbot's logchunk compression and switch to a different method. Setting c['logCompressionLimit'] = False and moving the table data into a compressed filesystem or allowing MariaDB's native compression options to transparently handle the task.

I've set this to High / 6.0_M1 as we need green builds on the autobuilder before we can move forward with the milestone release. Hopefully Michael's change should have fixed things, we need to continue to monitor and see if issues reoccur.

Thanks Michael, sounds like a good find!

From before Michael fix:
Stable build hit: whinlatter a-full fedora40-vk-3 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3038
Froze, then got canceled.

Not seen since Michael fixed this on Jan 12. 
Re-open if seen again.

Seen again:
kirkstone a-full fedora39-vk-1 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/29/builds/3067
kirkstone qemuarm64-ptest ubuntu2404-vk-arm1 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/61/builds/2902

On #yp-infra:
[07:39:24] <halstead> This is not the same deadlock error as before.
[...]
[08:20:22] <halstead> I don't think I'll be able to track down the issue tonight. I've gone ahead and restarted the cluster. I selectively cancelled builds and the rest are already in process again.

Happened again around 2026-01-19T08:45:00+00:00

Infrastucture problem, move to M2.

Michael migrated the controller to a different distro and to a postgres database backend instead of MariaDB. This hopefully resolves the issue, we can reopen or create a new issue if things recur.

Reopening. Even though the database deadlocks are not an issue after the switch to PostgreSQL we are still locking up with similar symptoms.

This is due to log compression threads exiting without reporting that back to the main reactor thread.

I've added "c['logCompressionLimit'] = False" to ~/yocto-controller/master.cfg to disable real time log compression which should avoid the issue at the cost of disk space. We can compress logs in a batch during maintenance windows.

I've attempted to fix the issue and submitted a pull request at https://github.com/buildbot/buildbot/pull/8823. I expect I've made errors but hopefully the buildbot devs can correct for them and we can solve this issue permanently.

(In reply to Michael Halstead from comment #12)
> I've attempted to fix the issue and submitted a pull request at
> https://github.com/buildbot/buildbot/pull/8823. I expect I've made errors
> but hopefully the buildbot devs can correct for them and we can solve this
> issue permanently.

Upstream devs fixed this bug with :
db.logs: fix _async_iter_on_pool faulty implementation causing deadlocks by tdesveaux · Pull Request #8826 · buildbot/buildbot
https://github.com/buildbot/buildbot/pull/8826

Now merged on buildbot/master and deployed on https://autobuilder.yoctoproject.org/valkyrie/ since 2026-03-09

Reopen if that happens again.