Bug 7903

Summary: Random build-appliance download problems
Product: [Infrastructure] AutoBuilder Reporter: Richard Purdie <richard.purdie>
Component: autobuilderAssignee: Richard Purdie <richard.purdie>
Status: VERIFIED FIXED QA Contact: Cristina Agurida <cristina-danielax.agurida>
Severity: normal    
Priority: Medium+ CC: alexandru.c.georgescu, elizabeth.flanagan, infras.ab.watcher, Infras.watcher, mhalstead
Version: 1.8   
Target Milestone: 1.9 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2015-06-18 08:09:26 UTC
We've seen several failures in build-appliance, e.g.:

https://autobuilder.yoctoproject.org/main/builders/build-appliance/builds/336/steps/BuildImages_1/logs/stdio

and

https://autobuilder.yoctoproject.org/main/builders/build-appliance/builds/335

which seem to be a race on the downloads directory.

The build creates a download mirror and then uses this to put sources into the build-appliance image. Looking on debian7 (the first failure above):

pokybuild@debian7:~/yocto-autobuilder/yocto-worker/build-appliance/build$ ls -la /home/pokybuild/yocto-autobuilder/yocto-worker/build-appliance/build/build/downloads/man-1.6g.tar.gz
lrwxrwxrwx 1 pokybuild users 76 Jun 17 03:07 /home/pokybuild/yocto-autobuilder/yocto-worker/build-appliance/build/build/downloads/man-1.6g.tar.gz -> /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz
pokybuild@debian7:~/yocto-autobuilder/yocto-worker/build-appliance/build$ ls -la /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz
-rw-r--r-- 1 pokybuild users 256906 May 18  2012 /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz

so the file is a symlink to the main download directory and some some reason, it couldn't follow that symlink for a short while. I suspect this is something more to do with the autobuilder infrastructure than the build, maybe with the way the download directories are updated? Adding Beth/Michael to cc on the bug to see if we can figure out why this is happening.

With each of these intermittent failures, I'm losing a lot of time trying to figure out if its a real issue or not. We therefore need to get to the bottom of some of these.
Comment 1 Michael Halstead 2015-06-19 05:18:31 UTC
It looks like /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz really has been present since May 18  2012 and I don't see any errors on the NAS. So that end should be fine.

So it looks like we are running stat on the path where the symlink will be before the symlink exists.
Comment 3 Beth Flanagan 2015-06-24 12:58:02 UTC
Halstead.

I have a "maybe" idea what is going on, but I'll need to modify some stuff and I don't want to do it right before M1. Could the current_sources clean up script is being a bit too diligent in the middle of the build? 

I think maybe changing:
SOURCE_MIRROR_URL = "file:////srv/www/vhosts/autobuilder.yoctoproject.org/current_sources"

to

SOURCE_MIRROR_URL = "file:////srv/www/vhosts/autobuilder.yoctoproject.org/sources"

would halt that issue as we never clean up sources, just rsync to it. So, yeah. Like, the file is obviously there, but catching this in the act is a bit difficult.

Thoughts? I know the nas logs showed no issues.
Comment 6 Cristina Agurida 2015-07-30 08:25:50 UTC
Set as verified.