Bug 9182

Summary: Build race in perf plugin_scsi.o
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Ross Burton <ross.burton>
Component: kernelAssignee: Bruce Ashfield <bruce.ashfield>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: bruce.ashfield, tom.zanussi
Version: 2.1   
Target Milestone: 2.1 M4   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: Regression (Used to work)
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
do_compile
none
do_install none

Description Ross Burton 2016-02-29 15:58:12 UTC
The autobuilder is failing like this recently:

  CC       /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/perf/1.0-r9/perf-1.0/plugin_scsi.o
  CC       /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/perf/1.0-r9/perf-1.0/plugin_scsi.o
  LD       /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/perf/1.0-r9/perf-1.0/plugin_scsi-in.o
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/perf/1.0-r9/perf-1.0/plugin_scsi.o: file not recognized: File truncated
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work-shared/qemux86-64/kernel-source/tools/build/Makefile.build:121: recipe for target '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/perf/1.0-r9/perf-1.0/plugin_scsi-in.o' failed

http://errors.yoctoproject.org/Errors/Details/38137/

Note how it appears to be building plugin-scsi.o twice so my hunch is that the second CC and the LD are racing for plugin_scsi.o.

Also I'm curious why this happens in do_install() instead of do_compile().
Comment 1 Ross Burton 2016-03-01 15:26:09 UTC
More races in today's AB run (http://errors.yoctoproject.org/Errors/Details/40623/)

install: cannot stat '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm64/build/build/tmp/work/qemuarm64-poky-linux/perf/1.0-r9/perf-1.0/plugin_kvm.so': No such file or directory
install: cannot stat '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm64/build/build/tmp/work/qemuarm64-poky-linux/perf/1.0-r9/perf-1.0/plugin_scsi.so': No such file or directory
install: cannot stat '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm64/build/build/tmp/work/qemuarm64-poky-linux/perf/1.0-r9/perf-1.0/plugin_cfg80211.so': No such file or directory
Makefile:264: recipe for target 'install_plugins' failed

As before there are multiple LINK lines in the build running in parallel.
Comment 2 Ross Burton 2016-03-18 16:21:21 UTC
Two more fails last night on the AB...
Comment 3 Bruce Ashfield 2016-03-21 23:14:18 UTC
I've had no luck reproducing this, and am no closer to solving it. Looking at another sstate issue though, and perhaps they are related.
Comment 4 Ross Burton 2016-03-21 23:17:36 UTC
I can't replicate on demand but it looks like it may be from the calling of fresh makes from the perf directory to ../traceevents/.  If two sub-makes are fired in parallel then they'll be unaware and race over building those sources.
Comment 5 Bruce Ashfield 2016-03-21 23:19:17 UTC
(In reply to comment #4)
> I can't replicate on demand but it looks like it may be from the calling of
> fresh makes from the perf directory to ../traceevents/.  If two sub-makes
> are fired in parallel then they'll be unaware and race over building those
> sources.

I've also been looking at it as a parallel build issue. The latest kernel has updates in this area, and I'll scan them to see if there's anything I can backport. I'd rather not go -j1 for this.
Comment 6 Bruce Ashfield 2016-03-30 17:40:15 UTC
I'm running perf builds with -j24 in a tight loop, and haven't had this break. So it is time to change tactics.

The key may be question posed earlier in the bug. Why is compilation happening during the install rule.

In my local tests, that isn't the case. So there's something that is trigger those builds .. and leading to our race.

Do the builders use rm_work ? Something else ?

Can we get access to the do_compile and do_log of the broken and some working builds ? Mine have zero compilation in the install phase .. again, we need to figure that out as a first step.
Comment 7 Ross Burton 2016-04-05 14:49:44 UTC
Created attachment 3093 [details]
do_compile
Comment 8 Ross Burton 2016-04-05 14:50:32 UTC
Created attachment 3094 [details]
do_install

Following from the previous do_compile, this is a do_install which shows compilation happening.

rm_work isn't enabled.  The build succeeded without problems.
Comment 9 Ross Burton 2016-04-19 12:58:46 UTC
It seems that I can stop it building in do_install by passing DESTDIR to make all.  This might work around the race? (patch sent)
Comment 10 Ross Burton 2016-04-28 15:13:04 UTC
Fixed in oe-core 76c473dbe9e6a1eb8bca89f26cf29b41ca18d680.