Bug 7698

With a buildbot refactor occuring I'd like to wait a bit on this to see what happens for 1.0 buildbot

Was floating around as unassigned. Think this probably belongs to Joshua, as a first pass, at least.

Moving to a newer upstream buildbot is important both to benefit from core improvements, such as plugin support and the latest integration hooks (i.e. for github), and to benefit from any security fixes.

What makes such an upgrade more difficult is that our ~4 year old fork of autobuilder-0.8.8 carries several customisations. We need need to figure out what we want/need to keep and whether we can split any of those features out into plugins.

In addition upstream buildbot has changed enough that we will need to rewrite much of our custom code, i.e. the way buildsteps are implemented changed in 0.9 and old-style buildsteps are no longer supported in releases newer than 0.9.x

# Notable changes we've made

- UI to cancel all pending builds
- customised web status plugin for YP
- yoctogit fetcher that supports multiple code bases in order to maintain local trees

# Files changed

- buildbot/process/botmaster
- buildbot/status/web/templates/[build|builders|forms|grid|grid_macros|grid_transposed|layout].html
- buildbot/status/web/[authz|base|build|grid|master].py
- buildbot/steps/source/yoctogit.py
- buildbot/steps/[transfer|trigger].py

# Upstream changes with affect yocto-autobuilder

* Built-in web server replaces the webstatus plugin: https://docs.buildbot.net/current/manual/cfg-www.html
  At first blush it looks like this should better suit our needs moving forwards as the web server supports plugins and custom templates, meaning it should be easier to customise.

* New-style buildsteps
Will likely require rewriting all of the 56 buildsteps in lib/python2.7/site-packages/autobuilder/buildsteps

* https://docs.buildbot.net/current/manual/customization.html#writing-new-buildsteps
* https://docs.buildbot.net/current/manual/new-style-steps.html

* New-style plugins

Plugins should enable us to separate out more of our customisations into easier to maintain packages. It will likely also mean we need to rewrite the YoctoMailer and wikilog 

* https://docs.buildbot.net/current/manual/plugins.html
* https://docs.buildbot.net/current/developer/plugins-publish.html
* https://docs.buildbot.net/current/manual/customization.html#writing-new-status-plugins

There's something of a migration guide available from upstream Buildbot: http://docs.buildbot.net/current/manual/installation/nine-upgrade.html

Interestingly enough Buildbot 0.8.9 (the next release after we forked) added two items from our notable changes:
* the git source supports an optional 'reference' argument:
"reference
    (optional): use the specified string as a path to a reference repository on the local machine. Git will try to grab objects from this path first instead of the main repository, if they exist."
* the ability to cancel some or all pending builds

Another relevant discovery, the new UI in Buildbot "nine" no longer includes the grid and tgrid views. We make heavy use of the latter!

Upstream buildbot are collecting feedback on why the grid view is needed in order to design a new dashboard which provides equivalent functionality:

https://github.com/buildbot/buildbot/issues/3084

A combination of bug #11455 and a layer setup tool built in to OE/BB (enabling us to drop our custom yoctogit source fetcher) would enable us to proceed with this.

Resolved by the new infrastructure architecture of helper scripts (yocto-autobuilder-helper) with a more recent buildbot with minimal configuration (yocto-autobuilder2).

We still need to come up with a plan for tracking, validating and moving to newer buildbot as it releases. There have already been several releases in 2018.