3517 2012-11-30 05:43:53 +0000 kernel: reduce packaging and build system overhead 2013-04-11 11:48:42 +0000 1 1 7 Build System, Metadata & Runtime OE-Core kernel 1.4 x86 Multiple VERIFIED FIXED Medium enhancement 1.4 M5 1 bruce.ashfield bruce.ashfield bruce.ashfield dvhart Martin.Jansa sstncr tom.zanussi --- --- --- oldest_to_newest 27843 0 bruce.ashfield 2012-11-30 05:43:53 +0000 Another significant barrier to entry for the advanced kernel developer is the time required to build, package and deploy the kernel, as an example of a typical bitbake linux-yocto, we see the following: do_compile (2 min) do_compile_kernelmodules (30 sec) do_install (3 min) do_package (7 min) do_package_write (4 min) So, out of 17 minutes total for 'bitbake linux-yocto', 14 minutes is install and package overhead, which is significant. Reducing this overhead to 20-30% over a native build cycle: > make bzImage modules > make install modules_install Would be a great achievement and make working with the kernel from within the build system a much better experience. Current ideas for reducing the time are: - do not individually package each kernel module, but only package the one meta package "kernel-modules", but include a provides for each individual module in the meta package to keep compatibility with existing dependencies. This would be optional, since it trades of the streamlining of the rootfs via individual module dependencies. But there are others who control the size of the rootfs by controlling the kernel config and only build modules that they want on the final rootfs. The savings from this are yet to be measured - Reduce the amount of kernel source copying and i/o involved with the staging and deploy phases. Using tar and pipes or other techniques to hold files in memory are considerations - Document items like not install the kernel image to the rootfs to reduce the rootfs generation time. - Document workflows for faster development, but this should not be the primary solution to this feature. - There are common build system elements to this like rpm and other package backend overhead that should be measured, but not fixed, as part of this feature. But if the overhead is incurred within the kernel classes, it should be fixed as part of this feature. 27857 1 dvhart 2012-11-30 16:51:55 +0000 Bruce, assigning to you as you said you were looking into it in parallel with Tom's more general performance analysis of the overall system. 27858 2 dvhart 2012-11-30 16:52:52 +0000 This may need to be broken into multiple bugs, as I don't see how this one can really be sized/estimated in its current form. 28426 3 tom.zanussi 2012-12-21 20:39:48 +0000 I added some timestamping code to the bitbake log functions and also added some bbnote calls to do_package, to get a bit more breakdown there. Here are the results of [trz@empanada build]$ time bitbake linux-yocto real 19m49.871s user 24m9.529s sys 7m43.978s do_kernel_checkout: 6 sec 12:11:24 12:11:30 do_kernel_configme: 30 sec 12:12:21 12:12:51 do_kernel_configcheck: 7 sec 12:13:35 12:13:42 do_patch: 49 sec 12:11:31 12:12:20 do_configure: 44 sec 12:12:51 12:13:35 do_compile: 2:08 min 12:13:42 12:15:50 do_compile_kernelmodules: 37 sec 12:15:50 12:16:27 do_install: 3:14 min 12:16:27 12:19:41 do_package: 7:39 min 12:19:42 perform_packagecopy: 9 sec 12:19:42 12:19:51 split_and_strip_files: 13 sec 12:19:51 12:20:04 populate_packages: 2:36 min 12:20:05 12:22:41 package_do_filedeps: 1:46 min 12:22:41 12:24:27 package_do_shlibs: 18 sec 12:24:27 12:24:45 package_do_pkgconfig: 8 sec 12:24:45 12:24:53 emit_pkgdata: 10 sec 12:24:53 12:25:03 do_package_qa: 24 sec 12:25:03 12:25:27 sstate_task_postfunc: 1:54 min 12:25:27 sstate_create_package: 57 sec 12:26:24 12:27:21 12:27:21 12:27:21 do_package_write_rpm: 3:43 min 12:27:21 BUILDSPEC: 3:27 min 12:27:32: Executing shell function BUILDSPEC 12:30:59: Shell function BUILDSPEC finished 12:31:04 do_populate_sysroot: 23 sec 12:19:42 12:20:05 Again, the packaging steps take more than half the time. Also, trying to figure out why do_install takes so long, when it happens almost instantaneously by hand, to see if it's a good case to analyze, not really - turns out mostly the copy and pruning of the tree. linux yocto do_install: The actual modules_install is very fast: NOTE 15:52:42: start modules_install NOTE 15:52:44: end modules_install The copy of the entire source tree takes 1:45 min: NOTE 15:52:44: start install 4 bbnote "start install 4" # # Copy the entire source tree. In case an external build directory is # used, copy the build directory over first, then copy over the source # dir. This ensures the original Makefiles are used and not the # redirecting Makefiles in the build directory. # # work and sysroots can be on different partitions, so we can't rely on # hardlinking, unfortunately. # cp -fR * $kerneldir cp .config $kerneldir if [ "${S}" != "${B}" ]; then cp -fR ${S}/* $kerneldir fi install -m 0644 ${KERNEL_OUTPUT} $kerneldir/${KERNEL_IMAGETYPE} install -m 0644 System.map $kerneldir/System.map-${KERNEL_VERSION} NOTE 15:54:29: start install 5 cleaning and removing files not needed takes 1:41 min: NOTE 15:54:29: start install 5 bbnote "start install 5" # # Clean and remove files not needed for building modules. # Some distributions go through a lot more trouble to strip out # unecessary headers, for now, we just prune the obvious bits. # # We don't want to leave host-arch binaries in /sysroots, so # we clean the scripts dir while leaving the generated config # and include files. # oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean make -C $kerneldir _mrproper_scripts find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $ker\ neldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \; find $kerneldir/Documentation -name "*.txt" -exec rm '{}' \; NOTE 15:56:10: start install 6 28503 4 dvhart 2013-01-02 17:08:38 +0000 One thing we could do is create a hardlink copy of the sources first, prune that, and then perform the regular copy. That might save time by not copying files we will later remove. 28504 5 dvhart 2013-01-02 17:09:31 +0000 If we can combine the find commands together, that might also save some time. 28956 6 bruce.ashfield 2013-01-18 17:45:34 +0000 I've received a patch that addresses some of the install time issues. Testing it here. 29133 7 Martin.Jansa 2013-01-24 13:44:12 +0000 Please don't make one big kernel-modules package. Right now each module can have extra postinst with autoload or extra kernel parameters provided from MACHINE.conf. That would get more ugly with one bit kernel-modules. Also that extra diskspace needed for all modules is significant on many devices, now I can enable any extra module in kernel config when even single user asks for it, if it gets packaged as one big package then I'll have to decide if it's worth it putting that module on every rootfs just to make one use happy. 29134 8 bruce.ashfield 2013-01-24 13:56:24 +0000 Martin: no worries, we would never break the existing behaviour in such a drastic way. Both the use case you describe and a big kernel modules package should be able to co-exist and meet different needs. Anything that happens would be optional, and not mandatory with existing techniques being maintained. 30250 9 bruce.ashfield 2013-02-28 14:10:07 +0000 With the cpio kernel changes, Peter Seebach's pseudo updates and RPs general packaging updates for 1.4, we have likely come as far as we can with this for the 1.4 release. Bumping this to M1.5 to take a final look at performance before closing the bugzilla entry. 30840 10 bruce.ashfield 2013-03-19 13:00:12 +0000 This has been resolved via a collection of changes: - packaging changes by Richard - pseudo changes by seebs - kernel file handling changes by others 31925 11 sstncr 2013-04-11 11:48:42 +0000 Marking as Verified. The data here https://wiki.yoctoproject.org/wiki/Performance_Test#1.4_Perfomance_Data shows a clear improvement from about 16:30 min in M1 to 11:20 min in M5.