Bug 12026

Summary:	Refkit clone sometimes fails with artefact missing errors
Product:	[Infrastructure] AutoBuilder	Reporter:	Richard Purdie <richard.purdie>
Component:	autobuilder	Assignee:	Michael Halstead <mhalstead>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	High	CC:	infras.ab.watcher, Infras.watcher, joshuagloe, mikko.ylinen, pidge, randy.macleod, todor.minchev
Version:	2.2
Target Milestone:	2.4 M4
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Richard Purdie 2017-09-01 07:36:56 UTC

All too often we see failures like:
https://autobuilder.yocto.io/builders/nightly/builds/513/steps/Git%20checkout%20of%20refkit/logs/stdio

with errors like:

cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/4716790f0a4d90a55556816404e19aa7e04877’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/2928bfbc52bd4b2105e59fa089f6a28954a5fb’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/c4857c780d0ea427de870b72bf70b49f36b837’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/11375694d7b530d17978943eb3ce9a26f841f3’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/2bf2c4776aee763cbbb2bee50a4c731d1fdccd’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/9ac2a25a2e8ef74633f6087bc6a35997429380’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/fd34a60aa9757e5af58208a88574d3a1a0c3cc’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/52/c71df4c28d4a535993007ba46047b94f0aac35’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/84/0df7771e37e209a56ec44b6d809e49ddf3df13’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/84/c3d7d31682366a0f1d5ac6767e9495643b7264’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/84/983c5aeaef977c1ba3f292d8f45286fa1c7812’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/84/0c08acb72a97bc36a4126ebb378ec83efa0ee6’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/e0’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/8f’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/ee’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/04’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/12’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/d8’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/77’: No such file or directory
cp: cannot stat ‘/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/source/git://git.yoctoproject.org/intel-iot-refkit/.git/modules/openembedded-core/objects/58’: No such file or directory
elapsedTime=8.794371

I have a theory that this happens when we run a "git fetch", a "git gc" runs in the background at the same time as the cp command and this then causes the step to fail. Larger repos have a higher chance of hitting the issue but its a fundamental problem with our approach of cloning these things. From "man git gc":

       Some git commands may automatically run git gc; see the --auto flag below for details. If you know what you’re doing and all you want is to disable this behavior
       permanently without further considerations, just do:

           $ git config --global gc.auto 0

We may want to consider this on the AB workers?

Comment 1 Richard Purdie 2017-09-01 07:37:45 UTC

We'd obviously need to make this part of the AB worker setup instructions

Comment 2 Mikko Ylinen 2017-09-01 18:09:50 UTC

*** Bug 12035 has been marked as a duplicate of this bug. ***

Comment 3 Joshua Lock 2017-09-07 15:18:38 UTC

I've made the change Richard suggested on all (a.y.io and a.y.org) workers. I'd like to observe some successes then work with Michael to ensure that change becomes part of the standard worker setup.

Comment 4 Michael Halstead 2017-09-20 16:07:16 UTC

May I add this config change to the standard setup and close the bug?

Comment 5 Joshua Lock 2017-09-20 19:59:21 UTC

(In reply to comment #4)
> May I add this config change to the standard setup and close the bug?

Please do, thanks Michael!

Comment 6 Michael Halstead 2017-09-20 22:08:48 UTC

Updated the docs.