Bug 10926

Summary:	Buildhistory fails to write to repo on new cluster
Product:	[Infrastructure] AutoBuilder	Reporter:	Stephano Cetola <stephano>
Component:	autobuilder	Assignee:	Stephano Cetola <stephano>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	infras.ab.watcher, Infras.watcher, joshuagloe, leonardo.sandoval.gonzalez, pidge
Version:	unspecified
Target Milestone:	2.4 M2
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)
Bug Depends on:
Bug Blocks:	11814

Description Stephano Cetola 2017-01-14 00:20:52 UTC

While the builds all look like they are succeeding, and the buildhistory logs appear to be written, the following error is being displayed at the end of each build:

NOTE: Writing buildhistory
fatal: '//home/pokybuild/buildhistory' does not appear to be a git repository
fatal: Could not read from remote repository.

e.g.
https://autobuilder.yocto.io/builders/nightly-x86-lsb/builds/136/steps/BuildImages/logs/stdio

Comment 1 Joshua Lock 2017-01-16 12:50:54 UTC

I believe there are two issues here:

1) there's a configuration issue in that the autobuilder configuration points to a non-existent repository as the buildhistory remote
2) when the buildhistory push fails the buildstep isn't marked as failed

Comment 2 Joshua Lock 2017-04-13 10:56:53 UTC

*** Bug 11334 has been marked as a duplicate of this bug. ***

Comment 3 Joshua Lock 2017-04-13 15:22:43 UTC

Too late to patch AB for 2.3 at this point, let's punt to 2.4 M1

Comment 4 Stephano Cetola 2017-07-11 22:44:18 UTC

I changed the configs on all the slaves and for some reason had to restart the controller to get the right variables to populate. I'm not sure what was going on there, but I'll keep an eye on the CreateAutoConf steps to be sure this is fixed. So far that step has completed on debian-testing just fine with nightly-arm.

The bit that was failing was the git ls-remote:

https://autobuilder.yocto.io/builders/nightly-arm/builds/400/steps/CreateAutoConf/logs/stdio
(git ls-remote --exit-code file:////home/pokybuild/buildhistory)

Note that BUILD_HISTORY_DIR was set correctly there after I updated the config, but was being ignored for some reason. I restarted the controller and it seems to work now:

https://autobuilder.yocto.io/builders/nightly-arm/builds/401/steps/CreateAutoConf/logs/stdio
(git ls-remote --exit-code ssh://git@git.yoctoproject.org/poky-buildhistory)

One thing we have to decide is if we want this to be a hard error in the future. Currently it does not fail because, as mentioned, the git push is buried in an if statement with no error handling around it, so it simply never falls into the "then" clause. See CreateAutoConf.py around line 266.

I'll leave this as "in progress review" until we have time to discuss how we want to deal with these types of failures.