Bug 9315

Summary: nightly-world & nightly-world-lsb failed at PublishingArtifacts
Product: [Infrastructure] AutoBuilder Reporter: Joshua Lock <joshuagloe>
Component: autobuilderAssignee: Bill Randle <bill.randle>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: elizabeth.flanagan, infras.ab.watcher, Infras.watcher, jku
Version: 2.1   
Target Milestone: 2.2 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
nightly-world log file
none
nightly-world-lsb log file none

Description Joshua Lock 2016-03-21 12:15:42 UTC
Created attachment 3047 [details]
nightly-world log file

The master build last night failed in nightly-world and nightly-world-lsb at the PublishingArtifacts stage as it refuses to copy over existing artefacts.

It's possible this is because a previous build was aborted? 
(the Fedora22 machine hung, causing the worker to be lost and the nightly build to fail).

For nightly-world the failure was:

cp: cannot create regular file ‘/srv/www/vhosts/autobuilder.yoctoproject.org/pub/nightly/20160320-2/deb/x86/lib32-gawk-locale-ca_4.1.3-r0_i386.deb’: File exists

For nightly-world-lsb there are many more of a similar pattern. Logs are attached.
Comment 1 Joshua Lock 2016-03-21 12:16:06 UTC
Created attachment 3048 [details]
nightly-world-lsb log file
Comment 2 Beth Flanagan 2016-03-21 16:14:49 UTC
Halstead,

This looks like something with the NFS server. We've seen this before, any thoughts?
Comment 3 Michael Halstead 2016-03-29 18:24:33 UTC
I do not see anything problematic in the logs on the NFS server. A cursory search makes me think this is a race condition between 2 cp processes. 

I suspect it is something like:

cp #1 checks if a file exists: doesn't exist
cp #2 checks if a file exists: doesn't exist
cp #1 creates the new file
cp #2 tries to create the file but it already exists, it prints this error

Is there a case where multiple copy processes would be launched copying into 20160320-2/deb/x86/ ?
Comment 4 Jussi Kukkonen 2016-06-17 10:53:15 UTC
(In reply to comment #3)
> I do not see anything problematic in the logs on the NFS server. A cursory
> search makes me think this is a race condition between 2 cp processes. 
> 
> I suspect it is something like:
> 
> cp #1 checks if a file exists: doesn't exist
> cp #2 checks if a file exists: doesn't exist
> cp #1 creates the new file
> cp #2 tries to create the file but it already exists, it prints this error

I can't think of how else this warning could appear ("cp -R" should overwrite existing files AFAIK) but doesn't that sound like an unlikely race condition to hit over 8000 times in one world build? I check random packages on the list and every time I found all three (ipk,rpm,deb) versions listed with a warning... as if this happened to every package.


Incidentally I wonder if we should actually us "cp --no-clobber": should "publishing artifacts" ever overwrite files?
Comment 5 Bill Randle 2016-06-28 22:38:51 UTC
Changed the buildset-config for nightly-world-lsb so it does not publish artifacts. This eliminates the potential name conflict.

Commit id: 69843cdf1eda6a1b52fa52eb4824ed8a52b9ae20