Bug 16243

Summary:	Exception Maximum PB reference count exceeded.
Product:	[Infrastructure] AutoBuilder	Reporter:	Michael Halstead <mhalstead>
Component:	autobuilder	Assignee:	Unassigned <unassigned>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	infras.ab.watcher, Infras.watcher, paul, randy.macleod
Version:	6.0
Target Milestone:	6.1 M1
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Michael Halstead 2026-04-12 01:53:16 UTC

After 5 days or fedora43-vk-2 worker is exhausting the MAX_BROKER_REFS=1024 limit.

We see exceptions like,

https://autobuilder.yoctoproject.org/valkyrie/#/builders/25/builds/3583
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3595
https://autobuilder.yoctoproject.org/valkyrie/#/builders/10/builds/3525
https://autobuilder.yoctoproject.org/valkyrie/#/builders/15/builds/3481
https://autobuilder.yoctoproject.org/valkyrie/#/builders/56/builds/1390

I'm not sure why it is that work specifically.

Comment 1 Michael Halstead 2026-04-12 01:56:34 UTC

I've added a cronjob that runs every three hours. It will restart any idle worker resetting the MAX_BROKER_REFS.

Ideally we'd track down why we are leaking refs and stop that.

Alternatively we could look into switching from pb to msgpack which does not have limits like this. 

Let's watch and see how this approach works out.

Comment 2 Michael Halstead 2026-05-19 17:07:07 UTC

BROKER_REF exhaustion appears to be fixed.

The restart job was causing a race between restarts and the job schedulers. I’ve moved it so that it starts 8 minutes after the hour, which should immediately resolve most of the issues.

We’re also going to update the restart script so that it pauses idle workers, waits 5 minutes, then checks state before taking further action.

Comment 3 Michael Halstead 2026-05-19 18:32:32 UTC

Script changes are up at https://git.yoctoproject.org/yocto-autobuilder-helper/diff/scripts/restart-if-idle.sh and this problem should be worked around indefinitely.