Bug 10284

Summary: FTP failures on new cluster resulting in bitbake-selfcheck and checkuri failures
Product: [Infrastructure] AutoBuilder Reporter: Joshua Lock <joshuagloe>
Component: autobuilderAssignee: Michael Halstead <mhalstead>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: infras.ab.watcher, Infras.watcher, pidge, richard.purdie, sjolley.yp.pm
Version: 2.2   
Target Milestone: 2.2 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Joshua Lock 2016-09-15 08:28:31 UTC 
This is most easily observed in the failures of an otherwise green build #59[1].

The nightly-checkuri contains several log[2] entries like:

ERROR: alsa-utils-scripts-1.1.2-r0 do_checkuri: Function failed: Fetcher failure for URL: 'ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2'. URL ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2 doesn't work
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/build/tmp/work/i586-poky-linux/alsa-utils-scripts/1.1.2-r0/temp/log.do_checkuri.39012
Log data follows:
| DEBUG: Executing python function do_checkuri
| DEBUG: Testing URL ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2
| DEBUG: checkstatus: trying again
| DEBUG: checkstatus() urlopen failed: <urlopen error ftp error: TimeoutError(110, 'Connection timed out')>
| DEBUG: Python function do_checkuri finished
| ERROR: Function failed: Fetcher failure for URL: 'ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2'. URL ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.1.2.tar.bz2 doesn't work
NOTE: recipe alsa-utils-scripts-1.1.2-r0: task do_checkuri: Failed


1. http://autobuilder.yocto.io:8010/builders/nightly/builds/59
2. http://autobuilder.yocto.io:8010/builders/nightly-checkuri/builds/37/steps/BuildImages/logs/stdio
Comment 1 Joshua Lock 2016-09-26 16:17:13 UTC 
Because of the way passive ftp works, opening up high ports at random, our security appliance is unhappy.

Options:
a) either turn off security appliance
b) setup proxy server in DMZ and setup proxy settings on all builders
c) recode recipes to use http or active ftp

b is preference

We may want to think about sun-setting FTP, if possible.
Comment 2 Michael Halstead 2016-10-05 17:00:56 UTC 
I've updated settings on the firewall and FTP failures have generally abated. While I still see failures when running wget in a loop the current state is good enough. The failures may be on the remote side based on the number of failures. Let's mark this working for now.
Comment 3 Richard Purdie 2016-10-07 15:14:47 UTC 
Sadly the last builds on the autobuilder showed this issue. Not sure why but we're going to need to look into this further.
Comment 4 Michael Halstead 2016-10-14 02:03:17 UTC 
I tried disabling the security appliance and the errors continued. I've started to suspect routing or mtu issues.

For now I've set up an ftp proxy on another network and enabled it for all workers except ubuntu1510 which is currently busy. I'll update it next.

This should solve the problem at least for now.