After 5 days or fedora43-vk-2 worker is exhausting the MAX_BROKER_REFS=1024 limit. We see exceptions like, https://autobuilder.yoctoproject.org/valkyrie/#/builders/25/builds/3583 https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3595 https://autobuilder.yoctoproject.org/valkyrie/#/builders/10/builds/3525 https://autobuilder.yoctoproject.org/valkyrie/#/builders/15/builds/3481 https://autobuilder.yoctoproject.org/valkyrie/#/builders/56/builds/1390 I'm not sure why it is that work specifically.
I've added a cronjob that runs every three hours. It will restart any idle worker resetting the MAX_BROKER_REFS. Ideally we'd track down why we are leaking refs and stop that. Alternatively we could look into switching from pb to msgpack which does not have limits like this. Let's watch and see how this approach works out.
BROKER_REF exhaustion appears to be fixed. The restart job was causing a race between restarts and the job schedulers. I’ve moved it so that it starts 8 minutes after the hour, which should immediately resolve most of the issues. We’re also going to update the restart script so that it pauses idle workers, waits 5 minutes, then checks state before taking further action.
Script changes are up at https://git.yoctoproject.org/yocto-autobuilder-helper/diff/scripts/restart-if-idle.sh and this problem should be worked around indefinitely.