Bug 12460 - Change in push.yoctoproject.org fingerprint causing connection timeout and autobuilder hang
Summary: Change in push.yoctoproject.org fingerprint causing connection timeout and au...
Status: RESOLVED FIXED
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.5 M2
Assignee: Michael Halstead
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2018-01-02 02:03 UTC by Rebecca Chang
Modified: 2018-01-31 22:07 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Rebecca Chang 2018-01-02 02:03:23 UTC
We need to investigate and root cause the issue in push.yoctoproject.org that causes autobuilder hang with command timed out: 1200 seconds without output.

RP has done a diagnostic on the failure manually:
----- snip -----
[rpurdie@fedora26 ~]$ sudo -iu pokybuild
[pokybuild@fedora26 ~]$ cd /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build
[pokybuild@fedora26 build]$ pwd=`pwd`;if [ ! -d /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb ];then mkdir -p /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;git init /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb; fi;rm -rf /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;mkdir -p /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;git init /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb;if git ls-remote --exit-code ssh://git@push.yoctoproject.org/poky-buildhistory refs/heads/poky/master-next/nightly-x86-lsb> /dev/null;then git push -q ssh://git@push.yoctoproject.org/poky-buildhistory :poky/master-next/nightly-x86-lsb;fi;if ! git ls-remote --exit-code ssh://git@push.yoctoproject.org/poky-buildhistory refs/heads/poky/master-next/nightly-x86-lsb > /dev/null;then cd /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb; echo 'Initializing Repo' >> README;git checkout -b poky/master-next/nightly-x86-lsb;git add README; git commit -s -m 'Initializing Repo';git push -q ssh://git@push.yoctoproject.org/poky-buildhistory poky/master-next/nightly-x86-lsb:poky/master-next/nightly-x86-lsb;fi
Initialized empty Git repository in /srv/autobuilder/buildhistory/poky/master-next/nightly-x86-lsb/.git/
The authenticity of host 'push.yoctoproject.org (140.211.169.56)' can't be established.
RSA key fingerprint is SHA256:UG8/tDXRT6y2kewUBO0lSQG6raErMJsqOU+p1wzol/I.
RSA key fingerprint is MD5:66:93:3e:a3:f6:60:a6:9e:07:c6:1b:b5:7d:e6:b1:d6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'push.yoctoproject.org' (RSA) to the list of known hosts.
Switched to a new branch 'poky/master-next/nightly-x86-lsb'
----- snip -----

It appears that it is waiting for user's input to confirm about the change in fingerprint.

However, we are seeing two more failures on MIPS and PPC.
https://autobuilder.yocto.io/builders/nightly-mips-lsb/builds/656/steps/CreateAutoConf/logs/stdio

https://autobuilder.yocto.io/builders/nightly-ppc-lsb/builds/669/steps/CreateAutoConf/logs/stdio
Comment 1 Michael Halstead 2018-01-31 22:07:05 UTC
The fingerprint did not change but the hostname changed from git.yoctoproject.org (RO) to push.yoctoproject.org (RW) and needed to be accepted again.

This is resolved on all the public workers. Some internal workers may still hit this issue.

Add the following to ~/.ssh/known_hosts or run a git push command and accept the new hostname at the prompt.

push.yoctoproject.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAux9kpB6l6UVdVBDhAWSkoH1VDuf/m9gxRU3g/h0a0BINoeIigev9pI8DRzwIm/dI5+2VRVZuzC0Pq8XZyHTqdKEcw92g2Xh84qfZZivh37XErx9cr//HmBr6/rQ30VvQ6EpRhMrJyG1qvrROUtpu7buRp7XxD/Soa0tiHY72afmXHfKulG2bxOOwPieoCiVWXa+r/21NHY9YfCQxvWoHNRV+g8XrAA8Go611Q2gtYgM8X5Fn71Ehy4JM3Je+FIlUOV0zkL9PkSdRA5og4Eb3rQdgEz1++p6dO82PmgTTzns/+SIgzJBxBHzCi94GbOOAzpwgVFlRFqWIcjE7RWnaNw==