Bug 7418 - Perf/kernel-devsrc race
Summary: Perf/kernel-devsrc race
Status: RESOLVED WORKSFORME
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: kernel (show other bugs)
Version: 1.8
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.2 M3
Assignee: Bruce Ashfield
QA Contact:
URL:
Whiteboard: Waiting for 4.7 release
Depends on:
Blocks:
 
Reported: 2015-03-06 16:42 UTC by Ross Burton
Modified: 2016-07-27 17:31 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Showing difference in kernel-devsrc rootfs contents (227.69 KB, image/png)
2015-06-11 17:23 UTC, Richard Purdie
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ross Burton 2015-03-06 16:42:46 UTC
This just happened on the autobuilder:

Log data follows:
| DEBUG: Executing shell function do_install
| 0 blocks
| cpio: ./tools/perf/.perf.dev.null: Cannot stat: No such file or directory
| 0 blocks
| WARNING: exit code 2 from a shell command.
| ERROR: Function failed: do_install (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/tmp/work/imx28evk-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/log.do_install.23110)
NOTE: recipe kernel-devsrc-1.0-r0: task do_install: Failed
ERROR: Task 502 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/meta/recipes-kernel/linux/kernel-devsrc.bb, do_install) failed with exit code '1'

kernel-devsrc:do_install() failed because ./tools/perf/.perf.dev.null disappeared during the cpio.

My guess is that perf is writing files into the same directory that kernel-devsrc is copying from (and that perf is in the wrong).
Comment 1 Bruce Ashfield 2015-03-11 19:44:11 UTC
accepting this. But without being able to reproduce it here, we'll need a generic solution
that likely makes a copy of these files earlier on in the process.
Comment 2 Bruce Ashfield 2015-04-15 15:31:21 UTC
This should be fixed when: "[PATCH] kernel.bbclass: Fix race condition"
is merged to oe-core [YOCTO: 7321].

Bumping to 1.9, since that's where that patch will land.
Comment 3 Richard Purdie 2015-06-11 17:23:03 UTC
We actually have a couple more problems with perf and kernel-devsrc. Firstly, the sed changes in perf do_configure may or may not make it into the kernel-devsrc tarball depending on the task execution order. Examples would be the CC changes to Makefile.

Secondly, the sed changes can apply multiple times for util/debug.h. See my attached screenshot which has repackaged builds a few times.
Comment 4 Richard Purdie 2015-06-11 17:23:34 UTC
Created attachment 2544 [details]
Showing difference in kernel-devsrc rootfs contents
Comment 5 Bruce Ashfield 2015-06-16 03:41:18 UTC
I'll revisit this change with the 4.1 kernel update. I knew the
sed changes would be applied multiple times, it just wasn't clear
if it would cause issues. We can add a detect, and only do the 
modification if the change hasn't already been made.

Being unable to patch the kernel to fix perf issues .. is definitely
a pain.

(In reply to comment #4)
> Created attachment 2544 [details]
> Showing difference in kernel-devsrc rootfs contents
Comment 6 Bruce Ashfield 2016-01-08 15:01:12 UTC
ross: are we still seeing this periodically on the autobuilder ?
Comment 7 Ross Burton 2016-01-26 14:27:28 UTC
The race that causes failures appears to have been fixed, but Richard has evidence that there's a number of non-fatal races remaining.
Comment 8 Bruce Ashfield 2016-02-08 22:05:55 UTC
(In reply to comment #7)
> The race that causes failures appears to have been fixed, but Richard has
> evidence that there's a number of non-fatal races remaining.

Are there pointers to the new logs ? I've never seen this in any form in my builds.
Comment 9 Bruce Ashfield 2016-02-18 17:06:43 UTC
I'm not able to reproduce this, and am looking for a pointer to any new logs.
Comment 10 Bruce Ashfield 2016-03-21 23:24:47 UTC
Ross: is this the same as #9182 ? I'll consolidate them if you agree that it looks similar .. or is this really centred around install and not a race in the build/link steps ?
Comment 11 Ross Burton 2016-03-21 23:31:27 UTC
They're definitely distinct, 9182 is a strict compile race where two compiles are running on the same object, this is more "interesting".
Comment 12 Bruce Ashfield 2016-03-21 23:32:22 UTC
ok. moving back to "in design", which means "Bruce is trying to see if he can make it happen" :)
Comment 13 Bruce Ashfield 2016-07-11 16:50:28 UTC
Doing a quick bounce on this bug to find out if it is still being seen.

We can retest against the 4.7+ kernel, when it becomes the 2.2 default.
Comment 14 Ross Burton 2016-07-21 15:27:52 UTC
I don't recall seeing this for a while now.
Comment 15 Bruce Ashfield 2016-07-27 17:31:34 UTC
We are going to take this opportunity and close this for now. Magic isn't good, but since we haven't seen it in a while, and we didn't reproduce it .. I'm going to close it.

When (if) it comes back .. we'll obviously revisit it.