We'd like to be able to build older releases and maintain releases for longer, e.g. an LTS release. To do this we need to be able to run in a specific distro environment in some cases. At the same time our existing non-homogenous autobuilder worker setup is valuable. This would imply container based workers. Is there some way could support containers for some builds on the autobuilder without complex/invasive installations?
The autobuilder workers would need to have Docker Engine [1] or other container runtime installed. We should fail gracefully if docker is not present or is not running. Until we know we need something different, start with the poky-container [2,3] as the execution unit. Depending on how they are used, they may be able to build more than one task. These containers have multiple flavors of supported distros [4]. The length of time those flavors are supported may need to be increased. Currently, they are phased out with the distro release goes EOL or can no longer be built. Collateral could be placed into escrow if even longer support is needed (this should be done by ISVs, not YP?). Use docker-py [5,6] Client API [7] to create/control the containers. This would need to be added to the python dependencies for AB2 workers. Perhaps this can be configurable/optional. We should fail gracefully if we cannot connect to the docker socket. Probably will need changes in yocto-autobuilder2/builders.py [8], schedulers.py [9], workers.py [10] and likely (a) new script(s) in yocto-autobuilder-helper [11]. QEMU in the container is complicated and might just be a huge bag of landmines, so we may benefit from running QEMU outside the container via libvirt or similar approach. This might require changes in runqemu or testimage to allow for new "remote" workflows (where QEMU would seem to be remote because it is outside the container environment). Alternatively, simply passing in the right devices (e.g. /dev/net/tun) or bind mounted volumes may be enough to make the in-container-built qemu-native work. [1] https://docs.docker.com/engine/ [2] https://github.com/crops/poky-container [3] https://hub.docker.com/r/crops/poky [4] https://github.com/crops/yocto-dockerfiles/tree/master/dockerfiles [5] https://github.com/docker/docker-py [6] https://pypi.org/project/docker-py/ [7] https://docker-py.readthedocs.io/en/stable/client.html [8] http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder2/tree/builders.py [9] http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder2/tree/schedulers.py [10] http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder2/tree/workers.py [11] http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/scripts
> QEMU in the container is complicated and might just be a huge bag of > landmines, so we may benefit from running QEMU outside the container via > libvirt or similar approach. This might require changes in runqemu or > testimage to allow for new "remote" workflows (where QEMU would seem to be > remote because it is outside the container environment). Alternatively, > simply passing in the right devices (e.g. /dev/net/tun) or bind mounted > volumes may be enough to make the in-container-built qemu-native work. > Using slirp with runqemu automatically forwards port 22 in the guest to port 2222 on the host. This means as long as testimage knows to use port 2222, it should "just work". If you want to use kvm for the x86 images, then you would have to bind mount /dev/kvm in the container as well as allow for privileged access.
runqemu and testimage bring a couple of wrinkles to the container approach, especially when one is trying to use 'runqemu kvm gl' for performance. It is fairly common to have '/dev/kvm' be owned by 'kvm' group, '/dev/net/tun' owned by 'netdev' group and '/dev/dri/card%n' owned by 'video' group. These devices need to be bind mounted into the containers and the container user needs to be a member of the owning groups. This should give limited privileges to the container user, rather than a more vulnerable approach of running a privileged container (which essentially runs as root--bad mojo).
While this is technologically feasible, the amount of change required to the autobuilders and autobuilder code might be more than desired for perhaps an over-engineered solution. For now, we will focus on the "much simpler" approach of adding gcc-native to buildtools seems like it will "solve" the immediate support issues on CentOS 7.
NOTE: installing docker-ce and always having the docker daemon running is not the only solution. Podman provides the "same" functionality without the daemon overhead. However, podman is not currently packaged for all our supported distros, meaning it would require building for many of the autobuilders (notably Debian and Ubuntu).
Buildtools tarballs work well so there's no pressure to work on this. MOving to Future.
The buildtools approach seems to be working well, we don't need o revisit this.