16243 2026-04-12 01:53:16 +0000 Exception Maximum PB reference count exceeded. 2026-05-19 18:32:32 +0000 1 1 5 Infrastructure AutoBuilder autobuilder 6.0 x86 Multiple RESOLVED FIXED Medium+ normal 6.1 M1 1 mhalstead unassigned infras.ab.watcher Infras.watcher paul randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 105047 0 mhalstead 2026-04-12 01:53:16 +0000 After 5 days or fedora43-vk-2 worker is exhausting the MAX_BROKER_REFS=1024 limit. We see exceptions like, https://autobuilder.yoctoproject.org/valkyrie/#/builders/25/builds/3583 https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3595 https://autobuilder.yoctoproject.org/valkyrie/#/builders/10/builds/3525 https://autobuilder.yoctoproject.org/valkyrie/#/builders/15/builds/3481 https://autobuilder.yoctoproject.org/valkyrie/#/builders/56/builds/1390 I'm not sure why it is that work specifically. 105048 1 mhalstead 2026-04-12 01:56:34 +0000 I've added a cronjob that runs every three hours. It will restart any idle worker resetting the MAX_BROKER_REFS. Ideally we'd track down why we are leaking refs and stop that. Alternatively we could look into switching from pb to msgpack which does not have limits like this. Let's watch and see how this approach works out. 105495 2 mhalstead 2026-05-19 17:07:07 +0000 BROKER_REF exhaustion appears to be fixed. The restart job was causing a race between restarts and the job schedulers. I’ve moved it so that it starts 8 minutes after the hour, which should immediately resolve most of the issues. We’re also going to update the restart script so that it pauses idle workers, waits 5 minutes, then checks state before taking further action. 105496 3 mhalstead 2026-05-19 18:32:32 +0000 Script changes are up at https://git.yoctoproject.org/yocto-autobuilder-helper/diff/scripts/restart-if-idle.sh and this problem should be worked around indefinitely.