For some reason the AB is failing to push into git buildhistory repo (master-next branch) [1], there are two concerns here, 1. In CreateAutoConf buildstep it fails to push but the step is marked as succeed [2][3]. 2. May be the buildhistory code needs to be more explicit in the failures?, some kind of sanitizing. [4] [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-buildhistory/ [2] https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/builds/933/steps/CreateAutoConf/logs/stdio [3] https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/912/steps/CreateAutoConf/logs/stdio [4] https://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/870/steps/Running%20Sanity%20Tests/logs/stdio
Michael, #2 and #3 seem to be infrastructure issues, rather than AB code base. Feel free to push it back if I'm wrong.
(In reply to comment #1) > Michael, #2 and #3 seem to be infrastructure issues, rather than AB code > base. Feel free to push it back if I'm wrong. If it was an infrastructure issue it's ok for me to close the bug but also the CreateAutoConf needs to fail when can't push to some repo see my case number 1.
Because of the way passive ftp works, opening up high ports at random, our security appliance is unhappy. Options: a) either turn off security appliance b) setup proxy server in DMZ and setup proxy settings on all builders c) recode recipes to use http or active ftp b is preference We may want to think about sun-setting FTP, if possible.
Ugh, ignore comment #3. That was meant for bug #10284
I've just re-reviewed these logs and realised that this may be a bug in the code. Both [2] and [3] are errors trying to manipulate refs which include master-next, only buildhistory for master-next branches shouldn't be pushed to the buildhistory repository due to it being listed in the BUILD_HISTORY_THROWAWAY_WHITELIST, i.e. from the YP AB config: BUILD_HISTORY_THROWAWAY_WHITELIST = "{'git://git.yoctoproject.org/poky':['master-next']}"
I think we've got a few issues here. 1) Based on my reading of the yocto-autobuilder code and commit history we're expecting master-next branches to already exist in the poky-buildhistory. CreateAutoConf tries to delete those branches from the remote before pushing a new, empty, version when building from the master-next branch of the poky repository. The failure in [2] is an attempt to delete a non-existing branch on the remote failing. 2) The failure in [3] seems to be an SSH host verification failure when communicating with the remote. 3) The push failures in [4] are because the --exit-code option the git ls-remote command in CreateAutoConf is in the wrong position: $ git ls-remote git://git.yoctoproject.org/poky-buildhistory refs/heads/master-next --exit-code $ echo $? 0 $ git ls-remote --exit-code git://git.yoctoproject.org/poky-buildhistory refs/heads/master-next $ echo $? 2 The code as it stands is akin to the former, whereas it must match the ordering of the latter — that is move the options before the arguments. I believe the real issue here is that master-next branches aren't being pushed, due to the master-next branch having not been initialised and pushed because the exit code of the ls-remote command is always 0 with the current argument order. I think 3) is the real issue that needs addressing to resolve this bug, I'm preparing a patch for this now.
(In reply to comment #6) > 3) The push failures in [4] are because the --exit-code option the git > ls-remote command in CreateAutoConf is in the wrong position: > > $ git ls-remote git://git.yoctoproject.org/poky-buildhistory > refs/heads/master-next --exit-code > $ echo $? > 0 > > $ git ls-remote --exit-code git://git.yoctoproject.org/poky-buildhistory > refs/heads/master-next > $ echo $? > 2 > > The code as it stands is akin to the former, whereas it must match the > ordering of the latter — that is move the options before the arguments. > > I believe the real issue here is that master-next branches aren't being > pushed, due to the master-next branch having not been initialised and pushed > because the exit code of the ls-remote command is always 0 with the current > argument order. Further the output of the ls-remote is piped into a cut, followed by a sed. This results in us not checking the return code of the ls-remote command with !.
I've pushed some patches to the yocto-autobuilder repository which seem to partially resolve this issue, unfortunately I'm now seeing Host key verification failures on some workers in the cluster. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. This failure has been seen on fedora24 and opensuse132 so far. centos7, debian8, fedora23 and ubuntu1404 haven't yet displayed the issue.
(In reply to comment #8) > I've pushed some patches to the yocto-autobuilder repository which seem to > partially resolve this issue, unfortunately I'm now seeing Host key > verification failures on some workers in the cluster. There are still some bugs in the command sequence we generate, I have a patch to address this which I'll test tomorrow and hopefully deploy. > Host key verification failed. > fatal: Could not read from remote repository. > Please make sure you have the correct access rights > and the repository exists. > > This failure has been seen on fedora24 and opensuse132 so far. centos7, > debian8, fedora23 and ubuntu1404 haven't yet displayed the issue. Still seeing this failure on ubuntu1404 and ubuntu1604, I'm working with Michael to understand it.
(In reply to comment #9) > (In reply to comment #8) > > I've pushed some patches to the yocto-autobuilder repository which seem to > > partially resolve this issue, unfortunately I'm now seeing Host key > > verification failures on some workers in the cluster. > > There are still some bugs in the command sequence we generate, I have a > patch to address this which I'll test tomorrow and hopefully deploy. The patch passed testing on a local builder so is pushed & deployed to the main cluster.
(In reply to comment #9) > (In reply to comment #8) > > Host key verification failed. > > fatal: Could not read from remote repository. > > Please make sure you have the correct access rights > > and the repository exists. > > > > This failure has been seen on fedora24 and opensuse132 so far. centos7, > > debian8, fedora23 and ubuntu1404 haven't yet displayed the issue. > > Still seeing this failure on ubuntu1404 and ubuntu1604, I'm working with > Michael to understand it. Still seeing this on ubuntu1604, i.e. https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64-lsb/builds/963/steps/CreateAutoConf/logs/stdio https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64-lsb/builds/963/steps/BuildImages/logs/stdio
We're not sure why the host key verifications failures are happening. Michael is deploying a change to the cluster to make the known_hosts files immutable and to ensure the that git.yoctoproject.org is in the system-wide /etc/ssh/ssh_known_hosts
This appears to be working based on recent master and master-next builds on the AB. All of the builders identified in this bug as having host key verification failures have successfully pushed to the poky-buildhistory repo today.