Bug 6444

When I looked into this in the past, the idea also crossed my mind.

A few clarifying points/questions:
  
  - The kernel module build itself is fast (in my builds), but the packaging is not.

  - Would an optional (emphasis on optional) single kernel-modules package (versus the current many packages
    that are pulled in via the kernel-modules package RDEPENDS) speed things up ? In my test it did show some
    promise. It could PROVIDE all the individual module packages to keep compatibility with existing recipes RDEPENDS.

  - If you don't do a single large kernel module build, I was benchmarking the packaging process and saw a 5 or 6 
    minute delay when a single do_package_split() was called.  Has anyone else seen similar in their benchmarking ?
    If that really is so expensive, is there a way to streamline it ? The thousands of files in the kernel tree seem to be
    triggering the pathalogical and really expensive cost of the operations.

If you look into where the kernel build spends its time, its the *huge* kernel-dev package and sstate files that actually cause the problems. If we ditch the "external module" pieces, I suspect you will be pleasantly surprised how fast it builds. The many small modules are not the speed bottleneck, at least in any measurement I've ever made (and I have made some so I know what I'm talking about here).

Before:

1386.73user 263.17system 8:53.65elapsed 309%CPU (0avgtext+0avgdata 783288maxresident)k
195600inputs+11697048outputs (812major+60094900minor)pagefaults 0swaps

do_bundle_initramfs: Elapsed time: 0.04 seconds 
do_clean: Elapsed time: 8.53 seconds 
do_cleansstate: Elapsed time: 0.53 seconds 
do_compile: Elapsed time: 74.53 seconds 
do_compile_kernelmodules: Elapsed time: 22.09 seconds 
do_configure: Elapsed time: 0.87 seconds 
do_deploy: Elapsed time: 18.95 seconds 
do_fetch: Elapsed time: 0.07 seconds 
do_install: Elapsed time: 26.43 seconds 
do_kernel_checkout: Elapsed time: 4.43 seconds 
do_kernel_configcheck: Elapsed time: 5.61 seconds 
do_kernel_configme: Elapsed time: 12.61 seconds 
do_kernel_link_vmlinux: Elapsed time: 0.04 seconds 
do_package: Elapsed time: 152.61 seconds 
do_packagedata: Elapsed time: 0.59 seconds 
do_package_write_rpm: Elapsed time: 128.54 seconds 
do_patch: Elapsed time: 26.33 seconds 
do_populate_lic: Elapsed time: 0.25 seconds 
do_populate_sysroot: Elapsed time: 7.06 seconds 
do_sizecheck: Elapsed time: 0.03 seconds 
do_strip: Elapsed time: 0.03 seconds 
do_uboot_mkimage: Elapsed time: 0.03 seconds 
do_unpack: Elapsed time: 45.56 seconds 
do_validate_branches: Elapsed time: 10.59 seconds 

After:

1196.84user 187.46system 5:02.99elapsed 456%CPU (0avgtext+0avgdata 782904maxresident)k
392inputs+11470416outputs (0major+53738114minor)pagefaults 0swaps

 do_bundle_initramfs: Elapsed time: 0.03 seconds 
 do_compile: Elapsed time: 72.73 seconds 
 do_compile_kernelmodules: Elapsed time: 21.09 seconds 
 do_configure: Elapsed time: 0.90 seconds 
 do_deploy: Elapsed time: 17.00 seconds 
 do_fetch: Elapsed time: 0.05 seconds 
 do_install: Elapsed time: 3.76 seconds 
 do_kernel_checkout: Elapsed time: 4.34 seconds 
 do_kernel_configcheck: Elapsed time: 5.50 seconds 
 do_kernel_configme: Elapsed time: 12.00 seconds 
 do_kernel_link_vmlinux: Elapsed time: 0.03 seconds 
 do_package: Elapsed time: 34.22 seconds 
 do_packagedata: Elapsed time: 2.74 seconds 
 do_package_write_rpm: Elapsed time: 43.92 seconds 
 do_patch: Elapsed time: 25.61 seconds 
 do_populate_lic: Elapsed time: 0.25 seconds 
 do_populate_sysroot: Elapsed time: 8.35 seconds 
 do_sizecheck: Elapsed time: 0.03 seconds 
 do_strip: Elapsed time: 0.03 seconds 
 do_uboot_mkimage: Elapsed time: 0.04 seconds 
 do_unpack: Elapsed time: 42.50 seconds 
 do_validate_branches: Elapsed time: 11.14 seconds

Created attachment 2000 [details]
Hacky patch tested for performance

As you can see, if we skip installing the source in the install step we take the build time from 9 minutes to 5 minutes. do_package 152s -> 34s however there is much less time spent handling *huge* sstate tarballs which accounts for a lot of the time too. I'm slightly surprised that doesn't show up on these timings from buildstats, need to check into why as do_poulate_sysroot should be miles faster (its probably due to it being a postfunc).

Obviously this source is needed for some things and this patch as is would break perf. We should do something like the shared workdir we use for gcc but for the kernel (machine specific) which would unbreak kernel module and things like perf. Alternatively, we could just checkout the sources we need in those tasks.

I also have to wonder why do_unpack is so slow and whether we're at the point we can relax/remove some of the checks from things like validate_branches.

(In reply to comment #6)
> Obviously this source is needed for some things and this patch as is would
> break perf. We should do something like the shared workdir we use for gcc
> but for the kernel (machine specific) which would unbreak kernel module and
> things like perf. Alternatively, we could just checkout the sources we need
> in those tasks.
> 
> I also have to wonder why do_unpack is so slow and whether we're at the
> point we can relax/remove some of the checks from things like
> validate_branches.

On the validate_branches step .. yes, it will be relaxed shortly. I have a series prepped, but
it broke a couple of use cases that I'm sorting out. So I'll be able to save some time there.

I swear do_package takes longer than 3 minutes for me. RP, can you describe your specific config/command for generating the data in Comment 3? I'd like to reproduce in a typical build for my usecases and compare.

The shared workdir sounds like a good idea given the duration of do_unpack currently - that's a big tree.

I do like the idea of the external module support being a separate recipe rather than a config option for the main recipe. This make it easier to build later and include in a package feed without changing the DISTRO configuration or the RECIPE configuration in local-only hacks.

To be discussed internally with Lucian and Costin.

We did end up doing this in the 1.8 cycle and changed things so that on target kernel module builds are handled by a separate recipe now. We also moved the kernel to using a shared directory for the build process which simplified the amount of copying going on in the kernel tasks. Since we implemented the performance ideas I'm aware of, I believe this is resolved now.

Darren, the data in Comment 3 was something like "bitbake virtual/kernel; bitbake virtual/kernel -c clean; time bitbake virtual/kernel" and then looking at the files in tmp/buildstats/<build>/linux-yocto/* and taking the Eslapsed time entries from those files.