10284 2016-09-15 08:28:31 +0000 FTP failures on new cluster resulting in bitbake-selfcheck and checkuri failures 2016-10-14 02:03:17 +0000 1 1 5 Infrastructure AutoBuilder autobuilder 2.2 x86 Multiple RESOLVED FIXED High normal 2.2 M4 1 joshuagloe mhalstead infras.ab.watcher Infras.watcher pidge richard.purdie sjolley.yp.pm --- --- No (bug/feature does not impact docs) oldest_to_newest 66191 0 joshuagloe 2016-09-15 08:28:31 +0000 This is most easily observed in the failures of an otherwise green build #59[1]. The nightly-checkuri contains several log[2] entries like: ERROR: alsa-utils-scripts-1.1.2-r0 do_checkuri: Function failed: Fetcher failure for URL: 'ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2'. URL ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2 doesn't work ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/build/tmp/work/i586-poky-linux/alsa-utils-scripts/1.1.2-r0/temp/log.do_checkuri.39012 Log data follows: | DEBUG: Executing python function do_checkuri | DEBUG: Testing URL ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2 | DEBUG: checkstatus: trying again | DEBUG: checkstatus() urlopen failed: <urlopen error ftp error: TimeoutError(110, 'Connection timed out')> | DEBUG: Python function do_checkuri finished | ERROR: Function failed: Fetcher failure for URL: 'ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2'. URL ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2 doesn't work NOTE: recipe alsa-utils-scripts-1.1.2-r0: task do_checkuri: Failed 1. http://autobuilder.yocto.io:8010/builders/nightly/builds/59 2. http://autobuilder.yocto.io:8010/builders/nightly-checkuri/builds/37/steps/BuildImages/logs/stdio 66590 1 joshuagloe 2016-09-26 16:17:13 +0000 Because of the way passive ftp works, opening up high ports at random, our security appliance is unhappy. Options: a) either turn off security appliance b) setup proxy server in DMZ and setup proxy settings on all builders c) recode recipes to use http or active ftp b is preference We may want to think about sun-setting FTP, if possible. 66949 2 mhalstead 2016-10-05 17:00:56 +0000 I've updated settings on the firewall and FTP failures have generally abated. While I still see failures when running wget in a loop the current state is good enough. The failures may be on the remote side based on the number of failures. Let's mark this working for now. 67040 3 richard.purdie 2016-10-07 15:14:47 +0000 Sadly the last builds on the autobuilder showed this issue. Not sure why but we're going to need to look into this further. 67250 4 mhalstead 2016-10-14 02:03:17 +0000 I tried disabling the security appliance and the errors continued. I've started to suspect routing or mtu issues. For now I've set up an ftp proxy on another network and enabled it for all workers except ubuntu1510 which is currently busy. I'll update it next. This should solve the problem at least for now.