Keeping track of builders per conf file is a pain. Figure out a better way to do this.
The new autobuilder codebase means this problem no longer exists.