Bug 241 - Can't build kernel modules outside of virtual/kernel recipe
Summary: Can't build kernel modules outside of virtual/kernel recipe
Status: RESOLVED FIXED
Alias: None
Product: Meta-yocto
Classification: Build System, Metadata & Runtime
Component: meta-yocto (show other bugs)
Version: 1.0
Hardware: x86 Multiple
: Medium major
Target Milestone: 1.0 M4
Assignee: Darren Hart
QA Contact:
URL:
Whiteboard: Patches out for review (16/Mar/2011)
Depends on:
Blocks:
 
Reported: 2010-08-27 19:03 UTC by Gary Thomas
Modified: 2011-03-24 05:30 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gary Thomas 2010-08-27 19:03:19 UTC
I have a recipe which builds a Linux kernel module outside of the kernel package itself.
Sadly, because of this over exuberant clause in kernel.bbclass, this can't work as most
of these programs are needed when building modules:

        # Remove the following binaries which cause strip errors
        # during do_package for cross-compiled platforms
        bin_files="arch/powerpc/boot/addnote arch/powerpc/boot/hack-coff \
                arch/powerpc/boot/mktree scripts/bin2c scripts/conmakehash \
                scripts/ihex2fw scripts/kallsyms scripts/pnmtologo scripts/basic/docproc \
                scripts/basic/fixdep scripts/basic/hash scripts/dtc/dtc \
                scripts/genksyms/genksyms scripts/kconfig/conf scripts/mod/mk_elfconfig \
                scripts/mod/modpost"
        rm -rf $kerneldir/scripts/*.o
        rm -rf $kerneldir/scripts/basic/*.o
        rm -rf $kerneldir/scripts/kconfig/*.o
        rm -rf $kerneldir/scripts/mod/*.o
        rm -rf $kerneldir/scripts/dtc/*.o
        for entry in $bin_files; do
                rm -f $kerneldir/$entry
        done


The removal if ${bin_files} is incorrect as most of these are needed when building modules outside the main build envuronment.
Comment 1 Richard Purdie 2010-09-16 07:39:34 UTC
Some feedback I gave on the mailing list about why these are being removed:

Imagine you built the kernel on a 32 bit machine and a shared state
package is generated of what's gone into the sysroot. You then switch to
64 bit machine, the checksums match and it decides to use the package.
The result would be running 32 bit binaries on a 64 bit machine.

My point is that target packages containing native binaries is a *very*
bad idea.

This is why they're being removed there so from that point of view its
correct. It does cause problems for external modules though.

At this point in time I don't have a good answer to this. Effectively
the kernel needs to generate a cross package :/
Comment 2 Richard Purdie 2010-09-16 07:43:23 UTC
Also note that the emenlow machine makes this work by forcing those "script"
binaries to be generated by directly making them first. This may be an
acceptable soltuion if included in the module bbclass file for all modules.
Comment 3 Richard Purdie 2010-10-06 07:50:21 UTC
> Richard, you made a comment about how the emenlow machine works around
> this, but I'm not seeing how this is done in the recipes/conf files.
> 
> Any hints?  Maybe I can see how to extend this for all targets.

http://git.pokylinux.org/cgit/cgit.cgi/poky/tree/meta-emenlow/packages/psb-kernel-source/psb-kernel-source-4.42.0/build.patch

See the change to psb-kernel-source-4.42.0/Makefile
Comment 4 Darren Hart 2011-02-16 13:37:03 UTC
Interesting. I wonder if a linux-*-binaries-native would be a good package to generate. Any module package could depend on it, and it would avoid the sstate problem RP mentioned.
Comment 5 Darren Hart 2011-02-23 12:50:39 UTC
I've discussed this with Richard and I'll be looking into adding a new independent task that builds a native "sstate package" for the necessary tools. This should get triggered on a change in host architecture.
Comment 6 Darren Hart 2011-02-28 20:55:43 UTC
I've dug around in this enough to have an idea of the problem now. The kernel tree is staged to $STAGING_KERNEL_DIR where other recipes can reference it. Modules will typically add something like "-C $STAGING_KERNEL_DIR}" where they'll have access to the build objects, headers, and the "hostprogs" (such as the binaries under scripts/*).

So, in order to make these available while avoiding the host arch traps Richard points out, we would need to be able to prepare a directory with all of what $STAGING_KERNEL_DIR contains, plus the hostprogs. We obviously don't want to have to rebuild the kernel everytime, so I see a couple approaches we could take.

1) Provide some kind of $STAGING_KERNEL_HOST_DIR which is a copy (hardlinks?) of $STAGING_KERNEL_DIR plus the hostprogs. All modules would need to use this new dir to build. This would also be a new sstate task, do_hostprogs.

2) We take Richard's earlier suggestion and simply have module.bbclass have something like a do_compile_prepend() function which builds all the hostprogs. None of these are particularly large and performance would only be an issue if an image built 100 or so out of tree module recipes. Perhaps this could somehow be captured in some kind of shared state for all decendents of module.bbclass - I don't know sstate well enough to know if this is doable.

I was working toward #1 above, but am starting to feel it's just not going to produce an elegant solution. I see lots of manual directory population (which will require maintenance every time a new hostprog is introduced) and complicated sstate magic. #2 seems like it could be implemented in about 10 lines of do_compile_prepend() in module.bbclass, which could later be optimized should the need arise.

Thoughts?
Comment 7 Gary Thomas 2011-03-01 08:56:53 UTC
Here's the workaround I ended up using.  It's ugly (and marked as such), but it might give you ideas what infrastructure is missing/needed.  FYI, this came from an OE recipe I adapted for the DSP support on the OMAP3530.  The major part of the work around is that some tools which are used when building modules, plus the kernel tree itself, are needed during this step.

do_compile() {
    unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS

    case ${SOC_FAMILY} in 
    dm365)
            modules="cmem edma irq";;
    omap3)
            modules="cmem sdma";;
        dm355)
            modules="cmem edma irq";;
    *)
            modules="cmem"
    esac

    # Insure host tools are available
    # HORRIBLE HACK
    # Sadly, I can't see how to automate these builds since the normal
    # way to do it using the Makefile ends up building target executables
    # not useful host tools.
    #   make CROSS_COMPILE= -C ${STAGING_KERNEL_DIR}/scripts/basic fixdep
    #   make CROSS_COMPILE= -C ${STAGING_KERNEL_DIR} SUBDIRS=scripts
    cd ${STAGING_KERNEL_DIR}/scripts/basic
    cc -o fixdep fixdep.c
    cd ${STAGING_KERNEL_DIR}/scripts/mod
    cc -o modpost modpost.c file2alias.c sumversion.c

    for module in $modules ; do
    cd ${S}/packages/ti/sdo/linuxutils/$module
    make \
      LINUXKERNEL_INSTALL_DIR="${STAGING_KERNEL_DIR}" \
      MVTOOL_PREFIX="${TARGET_PREFIX}" \
      UCTOOL_PREFIX="${TARGET_PREFIX}" \
      clean debug release
    done
}
Comment 8 Darren Hart 2011-03-01 23:58:28 UTC
I've coded up a solution in module.bbclass that I think is much more elegant than the new native hostprogs sstate task I was pursuing. The module gets built and it successfully loads in a qemux86 image. My last hurtle is finding out why when my example module is included in an image do all the other kernel modules disappear from the rootfs :/ If I can't sort that out via IRC in the morning, I'll post the patch to the list tomorrow and seek help.
Comment 9 Darren Hart 2011-03-03 21:54:53 UTC
Patches posted to poky@yoctoproject.org and openembedded-core@lists.openembedded.org. Waiting for review and pull.
Comment 10 Darren Hart 2011-03-15 09:30:13 UTC
This last solution doesn't fully address the problem. Building with rm work will remove the work dir, which the sysroots kernel tree relied on. I have another patch out for review on poky and oe-core which is experimental and a lot more invasive. Reopening until this is completely resolved.