Bug 12460

Summary:	Change in push.yoctoproject.org fingerprint causing connection timeout and autobuilder hang
Product:	[Infrastructure] AutoBuilder	Reporter:	Rebecca Chang <rebecca.swee.fun.chang>
Component:	autobuilder	Assignee:	Michael Halstead <mhalstead>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	infras.ab.watcher, Infras.watcher, joshuagloe, pidge, richard.purdie
Version:	unspecified
Target Milestone:	2.5 M2
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Rebecca Chang 2018-01-02 02:03:23 UTC

We need to investigate and root cause the issue in push.yoctoproject.org that causes autobuilder hang with command timed out: 1200 seconds without output.

RP has done a diagnostic on the failure manually:
----- snip -----
[rpurdie@fedora26 ~]$ sudo -iu pokybuild
[pokybuild@fedora26 ~]$ cd /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build
[pokybuild@fedora26 build]$ pwd=`pwd`;if [ ! -d /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb ];then mkdir -p /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;git init /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb; fi;rm -rf /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;mkdir -p /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;git init /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;if git ls-remote --exit-code ssh://git@push.yoctoproject.org/poky-buildhistory refs/heads/poky/master-next/nightly-x86-lsb> /dev/null;then git push -q ssh://git@push.yoctoproject.org/poky-buildhistory :poky/master-next/nightly-x86-lsb;fi;if ! git ls-remote --exit-code ssh://git@push.yoctoproject.org/poky-buildhistory refs/heads/poky/master-next/nightly-x86-lsb > /dev/null;then cd /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb; echo 'Initializing Repo' >> README;git checkout -b poky/master-next/nightly-x86-lsb;git add README; git commit -s -m 'Initializing Repo';git push -q ssh://git@push.yoctoproject.org/poky-buildhistory poky/master-next/nightly-x86-lsb:poky/master-next/nightly-x86-lsb;fi
Initialized empty Git repository in /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb/.git/
The authenticity of host 'push.yoctoproject.org (140.211.169.56)' can't be established.
RSA key fingerprint is SHA256:UG8/tDXRT6y2kewUBO0lSQG6raErMJsqOU+p1wzol/I.
RSA key fingerprint is MD5:66:93:3e:a3:f6:60:a6:9e:07:c6:1b:b5:7d:e6:b1:d6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'push.yoctoproject.org' (RSA) to the list of known hosts.
Switched to a new branch 'poky/master-next/nightly-x86-lsb'
----- snip -----

It appears that it is waiting for user's input to confirm about the change in fingerprint.

However, we are seeing two more failures on MIPS and PPC.
https://autobuilder.yocto.io/builders/nightly-mips-lsb/builds/656/steps/CreateAutoConf/logs/stdio

https://autobuilder.yocto.io/builders/nightly-ppc-lsb/builds/669/steps/CreateAutoConf/logs/stdio

Comment 1 Michael Halstead 2018-01-31 22:07:05 UTC

The fingerprint did not change but the hostname changed from git.yoctoproject.org (RO) to push.yoctoproject.org (RW) and needed to be accepted again.

This is resolved on all the public workers. Some internal workers may still hit this issue.

Add the following to ~/.ssh/known_hosts or run a git push command and accept the new hostname at the prompt.

push.yoctoproject.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAux9kpB6l6UVdVBDhAWSkoH1VDuf/m9gxRU3g/h0a0BINoeIigev9pI8DRzwIm/dI5+2VRVZuzC0Pq8XZyHTqdKEcw92g2Xh84qfZZivh37XErx9cr//HmBr6/rQ30VvQ6EpRhMrJyG1qvrROUtpu7buRp7XxD/Soa0tiHY72afmXHfKulG2bxOOwPieoCiVWXa+r/21NHY9YfCQxvWoHNRV+g8XrAA8Go611Q2gtYgM8X5Fn71Ehy4JM3Je+FIlUOV0zkL9PkSdRA5og4Eb3rQdgEz1++p6dO82PmgTTzns/+SIgzJBxBHzCi94GbOOAzpwgVFlRFqWIcjE7RWnaNw==