Bug 12228 - Race between do_make_sccripts() and recipes using the scripts it creates
Summary: Race between do_make_sccripts() and recipes using the scripts it creates
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 2.3.2
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.5 M3
Assignee: Bruce Ashfield
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-10-16 11:34 UTC by Peter Kjellerstedt
Modified: 2018-02-28 01:58 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
initial implementation patch (3.45 KB, patch)
2017-10-17 15:47 UTC, Bruce Ashfield
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Kjellerstedt 2017-10-16 11:34:15 UTC
Back in 2011, a lock (${TMPDIR}/kernel-scripts.lock) was introduced to protect do_make_scripts() from racing with itself when executed from different module recipes. The race was due to some of the kernel scripts always being rebuilt. 

Unfortunately this lock is not enough, since there is still a race between the scripts being rebuilt for one recipe and them being used by another. I.e., if do_make_script() is being executed for one kernel module recipe while another kernel module recipe is compiling, then there is a tiny chance that, e.g., modpost has been rewritten but not yet made executable by the first task at the same time as the second task tries to execute modpost, resulting in an error like this:

  /bin/sh: scripts/mod/modpost: Permission denied

My suggestion for a solution is to move do_make_scripts() from modules-base.bbclass to a recipe of its own (e.g., kernel-scripts) and then let modules.bbclass add a dependency on that recipe instead of adding the make_scripts task to the task list for the inheriting recipe. That way there should only ever be one recipe and task that creates the kernel scripts, and no race exists any more (which also means the lock file is no longer needed).
Comment 1 Bruce Ashfield 2017-10-17 12:25:47 UTC
Joe Slater has already sent a patch to do this in the past. I'll try and dig it up from the mailing list.
Comment 2 Bruce Ashfield 2017-10-17 15:45:35 UTC
Attached is the patch to do this.

I tested it here, and it works. I can re-submit to the list shortly, it would be nice to know if it works for Peter has well.

I'm sure it'll take a few revisions to get in, but the concept is the same.
Comment 3 Bruce Ashfield 2017-10-17 15:47:11 UTC
Created attachment 4086 [details]
initial implementation patch
Comment 4 Peter Kjellerstedt 2017-10-17 20:28:17 UTC
The attached patch looks more or less exactly as what I had in mind. The only comment I have is that I believe that the dependency:

do_compile[depends] += "virtual/kernel:do_compile_kernelmodules"

in the new recipe should be:

do_compile[depends] += "virtual/kernel:do_shared_workdir"

When it comes to testing, I have no real test case. I have only seen it happen once in all the time I have used OE, and based my report on my analysis of the code after it happened.
Comment 5 Peter Kjellerstedt 2017-11-03 20:44:36 UTC
The patch creates the make-mod-scripts recipe in meta-skeleton, but that shall of course be in meta...
Comment 6 Peter Kjellerstedt 2017-11-04 00:01:57 UTC
And the make-mod-scripts recipe needs to inherit pkgconfig.
Comment 7 Bruce Ashfield 2017-11-16 14:29:51 UTC
Thanks for the comments. I'm back on this now and will do some tweaks before posting a new version