Bug 10448

Summary:	Package files already existed at Publishing Artifacts step
Product:	[Infrastructure] AutoBuilder	Reporter:	Paul Eggleton <bluelightning>
Component:	autobuilder	Assignee:	Joshua Lock <joshuagloe>
Status:	RESOLVED WONTFIX	QA Contact:
Severity:	major
Priority:	Medium+	CC:	infras.ab.watcher, Infras.watcher, pidge, richard.purdie
Version:	unspecified
Target Milestone:	2.3 M4
Hardware:	x86
OS:	x86_64
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Paul Eggleton 2016-10-18 07:27:59 UTC

During the "Publishing Artifacts" step of the 2.2 rc4 build in nightly-arm64 a bunch of nativesdk deb packages apparently already existed at the destination which shouldn't be possible:

--------- snip ---------
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-locale-it_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-locale-de_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-locale-vi_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-doc_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-liblzma5_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-dbg_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-locale-pl_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-locale-cs_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-dev_5.2.2-r0.0_amd64.deb': File exists
cp: cannot create regular file '/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases/yocto-2.2.rc4/deb/x86_64-nativesdk/nativesdk-xz-locale-fr_5.2.2-r0.0_amd64.deb': File exists
--------- snip ---------

Link to the log:

https://autobuilder.yoctoproject.org/main/builders/nightly-arm64/builds/586/steps/Publishing%20Artifacts/logs/stdio

The previous build was interrupted which may have something to do with it, but still that shouldn't have caused this.

Comment 1 Richard Purdie 2016-10-21 07:07:42 UTC

I have a suspicion that multiple builds write this same set of artefacts. If one writes them, then another, one will simply overwrite the other. If two write them at the same time, I suspect one set of cp commmands will error as the file didn't exist when it started the operation but does exist part way through.

Some simple tests with some cp commands running in parallel will likely confirm how cp behaves in these cases.

A -n option to the cp commands on the autobuilder will likely expose the full extent of the problem (but will make it fail badly).

Comment 2 Joshua Lock 2016-12-08 22:44:24 UTC

(In reply to comment #1)
> I have a suspicion that multiple builds write this same set of artefacts. If
> one writes them, then another, one will simply overwrite the other. If two
> write them at the same time, I suspect one set of cp commmands will error as
> the file didn't exist when it started the operation but does exist part way
> through.

I suspect you're right. All of the architecture builders (nightly-arm, nightly-arm64, nightly-mips, nightly-mips64, nightly-ppc, nightly-x86-64, nightly-x86) and nightly-world publish the package artefacts for each package-manager backend, if two of those happen to be scheduled to PublishArtifacts at the same time we could easily run into this issue.

Comment 3 Joshua Lock 2017-03-13 14:38:38 UTC

We haven't seen this for a while, but let's push to M4. We understand what's occurring but I don't have a clean solution yet.

Comment 4 Joshua Lock 2017-03-30 15:11:46 UTC

We haven't seen this for a while. As we know what the issue is we'll aim to resolve it as part of bug #10437.