Bug 8127

(In reply to comment #0)
> It's not really clear to me what the purpose of running do_shared_workdir
> before do_compile_kernelmodules is. It doesn't seem like
> do_compile_kernelmodules needs work-shared at all.

Its more that do_shared_workdir shouldn't really need compile_kernelmodules, except that you've just proved that in fact it does.

do_shared_workdir can be thought of as a "make the kernel source code available" task buts its turning into a "make all the generated objects available" too :(.

> My preliminary testing showed no ill effects for me when changing the
> offending line to
> >addtask shared_workdir after do_compile after do_compile_kernelmodules before do_install
> 
> This solution obviously still precludes linking out-of-tree modules against
> other out-of-tree modules. But I'm not sure that this would even have been
> possible in pre-fido times.

The bigger issue here is that we tried changing things so that we don't have large tarballs of the kernel source in sstate. This means the kernel always extracts from source rather than being populated from sstate.

With your change, the downside is that if you try building a new kernel module in a clean tmpdir, the code will now not only recompile the kernel but all the kernel modules too. There are some simpler workloads (e.g. building perf) where we don't really want this compile overhead, let alone the module compile overhead, just to have kernel source available.

Clearly something needs more thought here although I'm struggling to think exactly how we can fix this in a way which doesn’t kill performance for everyone.

(In reply to comment #1)
> (In reply to comment #0)
> > It's not really clear to me what the purpose of running do_shared_workdir
> > before do_compile_kernelmodules is. It doesn't seem like
> > do_compile_kernelmodules needs work-shared at all.
> 
> Its more that do_shared_workdir shouldn't really need compile_kernelmodules,
> except that you've just proved that in fact it does.
> 
> do_shared_workdir can be thought of as a "make the kernel source code
> available" task buts its turning into a "make all the generated objects
> available" too :(.
> 
> > My preliminary testing showed no ill effects for me when changing the
> > offending line to
> > >addtask shared_workdir after do_compile after do_compile_kernelmodules before do_install
> > 
> > This solution obviously still precludes linking out-of-tree modules against
> > other out-of-tree modules. But I'm not sure that this would even have been
> > possible in pre-fido times.
> 
> The bigger issue here is that we tried changing things so that we don't have
> large tarballs of the kernel source in sstate. This means the kernel always
> extracts from source rather than being populated from sstate.
> 
> With your change, the downside is that if you try building a new kernel
> module in a clean tmpdir, the code will now not only recompile the kernel
> but all the kernel modules too. There are some simpler workloads (e.g.
> building perf) where we don't really want this compile overhead, let alone
> the module compile overhead, just to have kernel source available.
> 

That was my first thought as well. We really didn't want to wait through
a kernel + modules build to have the shared workdir available.


> Clearly something needs more thought here although I'm struggling to think
> exactly how we can fix this in a way which doesn’t kill performance for
> everyone.

I know we considered (and proposed a patch, but due to locking/race
concerns it didn't merge) simply adding a second copy of the symbols
after the modules built. That is what we've been doing internally, and
nothing has blown up yet.

In a scenario like that, an external module could depend on 
do_compile_kernelmodules, take the overhead and then get the updated copy.

But as we said before, we need to make sure that nothing is actively
accessing the existing symbols (tar, etc) or we'll start getting all
sorts of failures.

Is there a lock file we can use to make sure that a task can get 
exclusive access to the shared dir ?

Finally getting to this after clearing my backlog. Moving
the state to "in design".

Hi,
Is there an update on this?

(In reply to comment #4)
> Hi,
> Is there an update on this?

A patch was proposed in November on the mailing list (not
by me) to mitigate the issue while development proceeds with
this larger feature.

I do not know which kernel this is being tested on. But on one of the BSPs I am looking at I use the 3.14 kernel which exhibited the same sort of behavior and using a patch from the mainline kernel fixed the issue for me.
http://www.spinics.net/lists/linux-kbuild/msg09204.html

(In reply to comment #6)
> I do not know which kernel this is being tested on. But on one of the BSPs I
> am looking at I use the 3.14 kernel which exhibited the same sort of
> behavior and using a patch from the mainline kernel fixed the issue for me.
> http://www.spinics.net/lists/linux-kbuild/msg09204.html

In our case, the fix is build system related (and with the placement
of the symbols file after kernel modules have been built), versus
a problem specific to the kernel, or the kernel build.

Marking this case as resolved, since this was merged.

--------------

commit a9cc27e6916e5affe8b0cc431c3e89abd7681643
Author: Jens Rehsack <rehsack@gmail.com>
Date:   Thu Nov 12 17:13:15 2015 +0100

    kernel: fix race condition between compile_kernelmodules and shared_workdir
    
    Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=8127 which causes
    [ 1225.089323] 8189es: Unknown symbol cfg80211_scan_done (err -22)
    [ 1225.095916] 8189es: no symbol version for cfg80211_remain_on_channel_expired
    when loading external compiled 8189es module.
    
    (From OE-Core rev: afcea61e8eb39234d336c706fdfd4680dea7c060)
    
    Signed-off-by: Jens Rehsack <sno@netbsd.org>
    Signed-off-by: Ross Burton <ross.burton@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

:100644 100644 5e8b6cf34318... b75a4621807a... M        meta/classes/kernel.bbclass

---------

This introduces a longer build time for things like perf, but was the accepted short term solution.

If we want to fix the build time issues for perf, we can re-work the dependency in another bugzilla.

Marking this as verified since no one complained after the fix was merged.