Bug 7903 - Random build-appliance download problems
Summary: Random build-appliance download problems
Status: VERIFIED FIXED
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: 1.8
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 1.9 M2
Assignee: Richard Purdie
QA Contact: Cristina Agurida
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-06-18 08:09 UTC by Richard Purdie
Modified: 2015-07-30 08:25 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2015-06-18 08:09:26 UTC
We've seen several failures in build-appliance, e.g.:

https://autobuilder.yoctoproject.org/main/builders/build-appliance/builds/336/steps/BuildImages_1/logs/stdio

and

https://autobuilder.yoctoproject.org/main/builders/build-appliance/builds/335

which seem to be a race on the downloads directory.

The build creates a download mirror and then uses this to put sources into the build-appliance image. Looking on debian7 (the first failure above):

pokybuild@debian7:~/yocto-autobuilder/yocto-worker/build-appliance/build$ ls -la /home/pokybuild/yocto-autobuilder/yocto-worker/build-appliance/build/build/downloads/man-1.6g.tar.gz
lrwxrwxrwx 1 pokybuild users 76 Jun 17 03:07 /home/pokybuild/yocto-autobuilder/yocto-worker/build-appliance/build/build/downloads/man-1.6g.tar.gz -> /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz
pokybuild@debian7:~/yocto-autobuilder/yocto-worker/build-appliance/build$ ls -la /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz
-rw-r--r-- 1 pokybuild users 256906 May 18  2012 /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz

so the file is a symlink to the main download directory and some some reason, it couldn't follow that symlink for a short while. I suspect this is something more to do with the autobuilder infrastructure than the build, maybe with the way the download directories are updated? Adding Beth/Michael to cc on the bug to see if we can figure out why this is happening.

With each of these intermittent failures, I'm losing a lot of time trying to figure out if its a real issue or not. We therefore need to get to the bottom of some of these.
Comment 1 Michael Halstead 2015-06-19 05:18:31 UTC
It looks like /srv/www/vhosts/autobuilder.yoctoproject.org/current_sources/man-1.6g.tar.gz really has been present since May 18  2012 and I don't see any errors on the NAS. So that end should be fine.

So it looks like we are running stat on the path where the symlink will be before the symlink exists.
Comment 3 Beth Flanagan 2015-06-24 12:58:02 UTC
Halstead.

I have a "maybe" idea what is going on, but I'll need to modify some stuff and I don't want to do it right before M1. Could the current_sources clean up script is being a bit too diligent in the middle of the build? 

I think maybe changing:
SOURCE_MIRROR_URL = "file:////srv/www/vhosts/autobuilder.yoctoproject.org/current_sources"

to

SOURCE_MIRROR_URL = "file:////srv/www/vhosts/autobuilder.yoctoproject.org/sources"

would halt that issue as we never clean up sources, just rsync to it. So, yeah. Like, the file is obviously there, but catching this in the act is a bit difficult.

Thoughts? I know the nas logs showed no issues.
Comment 6 Cristina Agurida 2015-07-30 08:25:50 UTC
Set as verified.