Bug 9821

Summary:	create a configuration for a deployable autobuilder cluster
Product:	[Infrastructure] AutoBuilder	Reporter:	Benjamin Esquivel <benjamin.esquivel>
Component:	autobuilder	Assignee:	Unassigned <unassigned>
Status:	RESOLVED FIXED	QA Contact:
Severity:	enhancement
Priority:	Medium	CC:	anibal.limon, benjamin.esquivel, bluelightning, edwin.plauchu.camacho, infras.ab.watcher, Infras.watcher, joshuagloe, konrad.scherer, pidge, randy.macleod, sgw, tvgamblin, yoann.congal
Version:	unspecified
Target Milestone:	Future
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Benjamin Esquivel 2016-06-23 21:46:05 UTC

The proposal is to have a set of instructions to deploy an AB cluster via provisioning tools like Puppet, Ansible, etc. One option is to leave workers fixed in number and distro or it could be parametric over a set of supported distros.

This work includes creating reference images for the controller and the workers as well as the instructions for deployment.

The goal of this task is to facilitate AB replication.

Comment 1 Konrad Scherer 2016-06-30 15:10:16 UTC

I manage the internal WindRiver Linux build cluster and a lot of my work overlaps with the AB. I have built up Puppet code for provisioning bare-metal builders that does things like setup ntp, install host packages, install monitoring, etc.

I also have Dockerfiles and docker images which are used to build WRLinux. Using Docker in this way has increased our host coverage a lot while reducing the administration overhead by giving configuring all the builders in the same way.

I use Mesos and a custom scheduler to automate the builds and share workloads efficiently across the entire cluster of machines. I use Docker to create a local test setup of the production environment for development and testing. It is important to have a system that scales up and down easily.

I was hesitant to share this earlier because it was in early stages, but the system has matured and has been solid for over a year now. The cluster does over 2000 builds a day over 5 releases and 1500 configurations.

It still has rough edges as infrastructure projects tend to have, but I would be happy to share any and all pieces of my work that you feel would be useful.

Comment 2 Benjamin Esquivel 2016-06-30 15:38:30 UTC

Konrad, you seem to be very close to what we're looking with this entry. The success of this task is measured by having short and concise instructions to replicate a simple AB setup. If you are a couple of steps ahead with this and you have time to write instructions for replicating it, would be very much appreciated. We would test them.

Comment 3 Joshua Lock 2018-04-10 03:02:17 UTC

Should be much simpler with the new model of using vanilla upstream buildbot with a repo containing configuration and extensions.

Comment 4 Randy MacLeod 2026-07-23 15:25:49 UTC

Fixed by the bitbake-setup work.

Comment 5 Yoann Congal 2026-07-23 15:30:00 UTC

(In reply to Randy MacLeod from comment #4)
> Fixed by the bitbake-setup work.

This comment was on the wrong bug.

This bug is actually closed by this docker-compose configuration:
https://git.yoctoproject.org/yocto-autobuilder2/tree/docker/README.md?id=4a7eacd4ff4231989902ff39840cf0b8a95e2958