Bug 9340

Summary:	Improve tooling around BuildLog and SWAT process
Product:	[Infrastructure] AutoBuilder	Reporter:	Joshua Lock <joshuagloe>
Component:	autobuilder	Assignee:	Joshua Lock <joshuagloe>
Status:	RESOLVED FIXED	QA Contact:
Severity:	enhancement
Priority:	High	CC:	infras.ab.watcher, Infras.watcher, jose.perez.carranza
Version:	2.1
Target Milestone:	2.2 M1
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Joshua Lock 2016-03-24 14:25:52 UTC

The BuildLog[1] page on the Yocto Project wiki is very useful for SWAT coordination but adds manual process overheard to update it. 

With the MediWiki Edit API[2] we should be able to automatically create an entry in the BuildLog (most likely only when a checkbox is checked) and then whoever triggered the build can more easily edit the auto created markup on the wiki page to add more detail as required.

1. https://wiki.yoctoproject.org/wiki/BuildLog
2. https://www.mediawiki.org/wiki/API:Edit

Comment 1 Joshua Lock 2016-03-31 16:29:47 UTC

Stepping back a bit and re-framing this issue, we're trying to improve the SWAT process and automate pain points to make it easier for people who start builds to hand-over to SWAT.

In short, for this item, we need a way to communicate info about a build as the build happens:
- why the build was started / intent of build
- what actions does SWAT need to take
- what errors occurred & what has been done to address them (BZ entries, patches, etc)

This information can't be automatically generated in full because it needs manually updating as the build progresses and this the BuildLog page will need to support manual markup.

A possible solution is to implement a continually running daemon that monitors the autobuilders JSON API & tracks builds it has seen. This daemon would then add a new header and some skeleton/boilerplate information to the top of the BuildLog wiki page when a new build comes along that it hasn't seen. 

(Of course, the daemon also needs to accommodate for wiki pages being locked if the markup is being modified).

Comment 2 Joshua Lock 2016-04-01 16:05:14 UTC

An example of the autobuilder JSON API:

https://autobuilder.yoctoproject.org/main/json/builders/nightly/builds/_all?as_text=1

Comment 3 Joshua Lock 2016-05-05 22:07:37 UTC

I have a simple script that will poll the Autobuilder API (currently at 1min intervals) and post a skeletal entry to the BuildLog wiki page when a new build has been fired.

There are doubtless improvements that can be made to both the code and the "design" (such as using the Last-Modified HTTP header, if we can get the Autobuilder JSON API to include that) — but now I've worked out how to update the wiki page, I think the next step should be determining how we would want to deploy such a tool.

Comment 4 Joshua Lock 2016-05-06 13:42:41 UTC

Beth suggested the following, with regards making the script I have deployable to the autobuilder infrastructure:
* have the script called from the builder startup script, in a similar fashion as the janitor script
* have the script live in the yocto-autobuilder/bin
* read the configuration file live in yocto-autobuilder/etc
* read/write the cache file at yocto-autobuilder/

Comment 5 Joshua Lock 2016-05-09 15:14:23 UTC

Patch has been submitted and accepted to yocto-autobuilder, however there are some teething issues with rolling it out on the AB cluster. I will mark this task as resolved when the solution is working on the production AB cluster.

Comment 6 Joshua Lock 2016-05-10 21:09:42 UTC

The buildlogger should now be active on the YP Autobuilder