6444 2014-06-13 11:40:49 +0000 Move on target kernel module support to separate recipe? (Improve Performance and developer experience) 2015-04-30 15:27:46 +0000 1 1 7 Build System, Metadata & Runtime OE-Core kernel 1.7 x86 Multiple RESOLVED FIXED Medium+ enhancement 1.8 1 richard.purdie dvhart alexandru.c.georgescu bruce.ashfield costin.c.constantin georgex.l.musat nitin.a.kamble tom.zanussi alexandru.c.georgescu --- --- Yes (doc changes required) oldest_to_newest 43917 0 richard.purdie 2014-06-13 11:40:49 +0000 We pay a huge performance cost for allowing module builds in the main kernel recipe. I'd like to consider the idea of separating out the module build support into a separate kernel build. This would mean two different kernel unpack loops but would mean the main kernel recipe should build much faster and its sstate footprint would be much lower. The pain is then put onto the people using the external modules either on target or as separate recipes. I'm open to opinions on this but thought I'd at least file the idea... 43922 1 bruce.ashfield 2014-06-13 13:00:03 +0000 When I looked into this in the past, the idea also crossed my mind. A few clarifying points/questions: - The kernel module build itself is fast (in my builds), but the packaging is not. - Would an optional (emphasis on optional) single kernel-modules package (versus the current many packages that are pulled in via the kernel-modules package RDEPENDS) speed things up ? In my test it did show some promise. It could PROVIDE all the individual module packages to keep compatibility with existing recipes RDEPENDS. - If you don't do a single large kernel module build, I was benchmarking the packaging process and saw a 5 or 6 minute delay when a single do_package_split() was called. Has anyone else seen similar in their benchmarking ? If that really is so expensive, is there a way to streamline it ? The thousands of files in the kernel tree seem to be triggering the pathalogical and really expensive cost of the operations. 43923 2 richard.purdie 2014-06-13 13:12:02 +0000 If you look into where the kernel build spends its time, its the *huge* kernel-dev package and sstate files that actually cause the problems. If we ditch the "external module" pieces, I suspect you will be pleasantly surprised how fast it builds. The many small modules are not the speed bottleneck, at least in any measurement I've ever made (and I have made some so I know what I'm talking about here). 43935 3 richard.purdie 2014-06-13 13:38:31 +0000 Before: 1386.73user 263.17system 8:53.65elapsed 309%CPU (0avgtext+0avgdata 783288maxresident)k 195600inputs+11697048outputs (812major+60094900minor)pagefaults 0swaps do_bundle_initramfs: Elapsed time: 0.04 seconds do_clean: Elapsed time: 8.53 seconds do_cleansstate: Elapsed time: 0.53 seconds do_compile: Elapsed time: 74.53 seconds do_compile_kernelmodules: Elapsed time: 22.09 seconds do_configure: Elapsed time: 0.87 seconds do_deploy: Elapsed time: 18.95 seconds do_fetch: Elapsed time: 0.07 seconds do_install: Elapsed time: 26.43 seconds do_kernel_checkout: Elapsed time: 4.43 seconds do_kernel_configcheck: Elapsed time: 5.61 seconds do_kernel_configme: Elapsed time: 12.61 seconds do_kernel_link_vmlinux: Elapsed time: 0.04 seconds do_package: Elapsed time: 152.61 seconds do_packagedata: Elapsed time: 0.59 seconds do_package_write_rpm: Elapsed time: 128.54 seconds do_patch: Elapsed time: 26.33 seconds do_populate_lic: Elapsed time: 0.25 seconds do_populate_sysroot: Elapsed time: 7.06 seconds do_sizecheck: Elapsed time: 0.03 seconds do_strip: Elapsed time: 0.03 seconds do_uboot_mkimage: Elapsed time: 0.03 seconds do_unpack: Elapsed time: 45.56 seconds do_validate_branches: Elapsed time: 10.59 seconds After: 1196.84user 187.46system 5:02.99elapsed 456%CPU (0avgtext+0avgdata 782904maxresident)k 392inputs+11470416outputs (0major+53738114minor)pagefaults 0swaps do_bundle_initramfs: Elapsed time: 0.03 seconds do_compile: Elapsed time: 72.73 seconds do_compile_kernelmodules: Elapsed time: 21.09 seconds do_configure: Elapsed time: 0.90 seconds do_deploy: Elapsed time: 17.00 seconds do_fetch: Elapsed time: 0.05 seconds do_install: Elapsed time: 3.76 seconds do_kernel_checkout: Elapsed time: 4.34 seconds do_kernel_configcheck: Elapsed time: 5.50 seconds do_kernel_configme: Elapsed time: 12.00 seconds do_kernel_link_vmlinux: Elapsed time: 0.03 seconds do_package: Elapsed time: 34.22 seconds do_packagedata: Elapsed time: 2.74 seconds do_package_write_rpm: Elapsed time: 43.92 seconds do_patch: Elapsed time: 25.61 seconds do_populate_lic: Elapsed time: 0.25 seconds do_populate_sysroot: Elapsed time: 8.35 seconds do_sizecheck: Elapsed time: 0.03 seconds do_strip: Elapsed time: 0.03 seconds do_uboot_mkimage: Elapsed time: 0.04 seconds do_unpack: Elapsed time: 42.50 seconds do_validate_branches: Elapsed time: 11.14 seconds 43936 4 2000 richard.purdie 2014-06-13 13:39:00 +0000 Created attachment 2000 Hacky patch tested for performance 43937 5 richard.purdie 2014-06-13 13:42:24 +0000 As you can see, if we skip installing the source in the install step we take the build time from 9 minutes to 5 minutes. do_package 152s -> 34s however there is much less time spent handling *huge* sstate tarballs which accounts for a lot of the time too. I'm slightly surprised that doesn't show up on these timings from buildstats, need to check into why as do_poulate_sysroot should be miles faster (its probably due to it being a postfunc). 43938 6 richard.purdie 2014-06-13 13:44:10 +0000 Obviously this source is needed for some things and this patch as is would break perf. We should do something like the shared workdir we use for gcc but for the kernel (machine specific) which would unbreak kernel module and things like perf. Alternatively, we could just checkout the sources we need in those tasks. I also have to wonder why do_unpack is so slow and whether we're at the point we can relax/remove some of the checks from things like validate_branches. 43939 7 bruce.ashfield 2014-06-13 13:53:27 +0000 (In reply to comment #6) > Obviously this source is needed for some things and this patch as is would > break perf. We should do something like the shared workdir we use for gcc > but for the kernel (machine specific) which would unbreak kernel module and > things like perf. Alternatively, we could just checkout the sources we need > in those tasks. > > I also have to wonder why do_unpack is so slow and whether we're at the > point we can relax/remove some of the checks from things like > validate_branches. On the validate_branches step .. yes, it will be relaxed shortly. I have a series prepped, but it broke a couple of use cases that I'm sorting out. So I'll be able to save some time there. 43953 8 dvhart 2014-06-13 22:07:51 +0000 I swear do_package takes longer than 3 minutes for me. RP, can you describe your specific config/command for generating the data in Comment 3? I'd like to reproduce in a typical build for my usecases and compare. The shared workdir sounds like a good idea given the duration of do_unpack currently - that's a big tree. I do like the idea of the external module support being a separate recipe rather than a config option for the main recipe. This make it easier to build later and include in a package feed without changing the DISTRO configuration or the RECIPE configuration in local-only hacks. 47094 9 alexandru.c.georgescu 2014-11-20 09:16:16 +0000 To be discussed internally with Lucian and Costin. 50666 10 richard.purdie 2015-04-30 15:25:29 +0000 We did end up doing this in the 1.8 cycle and changed things so that on target kernel module builds are handled by a separate recipe now. We also moved the kernel to using a shared directory for the build process which simplified the amount of copying going on in the kernel tasks. Since we implemented the performance ideas I'm aware of, I believe this is resolved now. 50667 11 richard.purdie 2015-04-30 15:27:46 +0000 Darren, the data in Comment 3 was something like "bitbake virtual/kernel; bitbake virtual/kernel -c clean; time bitbake virtual/kernel" and then looking at the files in tmp/buildstats/<build>/linux-yocto/* and taking the Eslapsed time entries from those files. 2000 2014-06-13 13:39:00 +0000 2014-06-13 13:39:00 +0000 Hacky patch tested for performance kernel.patch text/plain 3503 richard.purdie ZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy9rZXJuZWwuYmJjbGFzcyBiL21ldGEvY2xhc3Nlcy9r ZXJuZWwuYmJjbGFzcwppbmRleCA2ZWQxY2I3Li44YTRmY2MxIDEwMDY0NAotLS0gYS9tZXRhL2Ns YXNzZXMva2VybmVsLmJiY2xhc3MKKysrIGIvbWV0YS9jbGFzc2VzL2tlcm5lbC5iYmNsYXNzCkBA IC0yMjYsMjAgKzIyNiw3IEBAIGtlcm5lbF9kb19pbnN0YWxsKCkgewogCiAJZWNobyAiJHtLRVJO RUxfSU1BR0VfQkFTRV9OQU1FfSIgPiRrZXJuZWxkaXIva2VybmVsLWltYWdlLW5hbWUKIAotCSMK LQkjIENvcHkgdGhlIGVudGlyZSBzb3VyY2UgdHJlZS4gSW4gY2FzZSBhbiBleHRlcm5hbCBidWls ZCBkaXJlY3RvcnkgaXMKLQkjIHVzZWQsIGNvcHkgdGhlIGJ1aWxkIGRpcmVjdG9yeSBvdmVyIGZp cnN0LCB0aGVuIGNvcHkgb3ZlciB0aGUgc291cmNlCi0JIyBkaXIuIFRoaXMgZW5zdXJlcyB0aGUg b3JpZ2luYWwgTWFrZWZpbGVzIGFyZSB1c2VkIGFuZCBub3QgdGhlCi0JIyByZWRpcmVjdGluZyBN YWtlZmlsZXMgaW4gdGhlIGJ1aWxkIGRpcmVjdG9yeS4KLQkjCi0JZmluZCAuIC1kZXB0aCAtbm90 IC1uYW1lICIqLmNtZCIgLW5vdCAtbmFtZSAiKi5vIiAtbm90IC1wYXRoICIuL0RvY3VtZW50YXRp b24qIiAtbm90IC1wYXRoICIuL3NvdXJjZSoiIC1ub3QgLXBhdGggIi4vLioiIC1wcmludDAgfCBj cGlvIC0tbnVsbCAtcGRsdSAka2VybmVsZGlyCiAJY3AgLmNvbmZpZyAka2VybmVsZGlyCi0JaWYg WyAiJHtTfSIgIT0gIiR7Qn0iIF07IHRoZW4KLQkJcHdkPSIkUFdEIgotCQljZCAiJHtTfSIKLQkJ ZmluZCAuIC1kZXB0aCAtbm90IC1wYXRoICIuL0RvY3VtZW50YXRpb24qIiAtbm90IC1wYXRoICIu Ly4qIiAtcHJpbnQwIHwgY3BpbyAtLW51bGwgLXBkbHUgJGtlcm5lbGRpcgotCQljZCAiJHB3ZCIK LQlmaQogCiAJIyBUZXN0IHRvIGVuc3VyZSB0aGF0IHRoZSBvdXRwdXQgZmlsZSBhbmQgaW1hZ2Ug dHlwZSBhcmUgbm90IGFjdHVhbGx5CiAJIyB0aGUgc2FtZSBmaWxlLiBJZiBoYXJkbGlua2luZyBp cyB1c2VkLCB0aGV5IHdpbGwgYmUgdGhlIHNhbWUsIGFuZCB0aGVyZSdzCkBAIC0yNTEsNjEgKzIz OCw2IEBAIGtlcm5lbF9kb19pbnN0YWxsKCkgewogICAgICAgICBta2RpciAka2VybmVsZGlyL0Rv Y3VtZW50YXRpb24KICAgICAgICAgdG91Y2ggJGtlcm5lbGRpci9Eb2N1bWVudGF0aW9uL01ha2Vm aWxlCiAKLQkjCi0JIyBDbGVhbiBhbmQgcmVtb3ZlIGZpbGVzIG5vdCBuZWVkZWQgZm9yIGJ1aWxk aW5nIG1vZHVsZXMuCi0JIyBTb21lIGRpc3RyaWJ1dGlvbnMgZ28gdGhyb3VnaCBhIGxvdCBtb3Jl IHRyb3VibGUgdG8gc3RyaXAgb3V0Ci0JIyB1bmVjZXNzYXJ5IGhlYWRlcnMsIGZvciBub3csIHdl IGp1c3QgcHJ1bmUgdGhlIG9idmlvdXMgYml0cy4KLQkjCi0JIyBXZSBkb24ndCB3YW50IHRvIGxl YXZlIGhvc3QtYXJjaCBiaW5hcmllcyBpbiAvc3lzcm9vdHMsIHNvCi0JIyB3ZSBjbGVhbiB0aGUg c2NyaXB0cyBkaXIgd2hpbGUgbGVhdmluZyB0aGUgZ2VuZXJhdGVkIGNvbmZpZwotCSMgYW5kIGlu Y2x1ZGUgZmlsZXMuCi0JIwotCW9lX3J1bm1ha2UgLUMgJGtlcm5lbGRpciBDQz0iJHtLRVJORUxf Q0N9IiBMRD0iJHtLRVJORUxfTER9IiBjbGVhbiBfbXJwcm9wZXJfc2NyaXB0cyAKLQotCSMgaGlk ZSBkaXJlY3RvcmllcyB0aGF0IHNob3VsZG4ndCBoYXZlIHRoZWlyIC5jLCBzIGFuZCBTIGZpbGVz IGRlbGV0ZWQKLQlmb3IgZCBpbiB0b29scyBzY3JpcHRzIGxpYjsgZG8KLQkJbXYgJGtlcm5lbGRp ci8kZCAka2VybmVsZGlyLy4kZAotCWRvbmUKLQotCSMgZGVsZXRlIC5jLCAucyBhbmQgLlMgZmls ZXMsIHVubGVzcyB3ZSBoaWQgYSBkaXJlY3RvcnkgYXMgLjxkaXI+LiBUaGlzIHRlY2huaXF1ZSBp cyAKLQkjIG11Y2ggZmFzdGVyIHRoYW4gZmluZCAtcHJ1bmUgYW5kIC1leGVjCi0JZmluZCAka2Vy bmVsZGlyIC1ub3QgLXBhdGggJyovXC4qJyAtdHlwZSBmIC1uYW1lICIqLltjc1NdIiAtZGVsZXRl Ci0KLQkjIHB1dCB0aGUgaGlkZGVuIGRpcnMgYmFjawotCWZvciBkIGluIHRvb2xzIHNjcmlwdHMg bGliOyBkbwotCQltdiAka2VybmVsZGlyLy4kZCAka2VybmVsZGlyLyRkCi0JZG9uZQotCi0JIyBB cyBvZiBMaW51eCBrZXJuZWwgdmVyc2lvbiAzLjAuMSwgdGhlIGNsZWFuIHRhcmdldCByZW1vdmVz Ci0JIyBhcmNoL3Bvd2VycGMvbGliL2NydHNhdnJlcy5vIHdoaWNoIGlzIHByZXNlbnQgaW4KLQkj IEtCVUlMRF9MREZMQUdTX01PRFVMRSwgbWFraW5nIGl0IHJlcXVpcmVkIHRvIGJ1aWxkIGV4dGVy bmFsIG1vZHVsZXMuCi0JaWYgWyAke0FSQ0h9ID0gInBvd2VycGMiIF07IHRoZW4KLQkJY3AgLWwg YXJjaC9wb3dlcnBjL2xpYi9jcnRzYXZyZXMubyAka2VybmVsZGlyL2FyY2gvcG93ZXJwYy9saWIv Y3J0c2F2cmVzLm8KLQlmaQotCi0JIyBOZWNlc3NhcnkgZm9yIGJ1aWxkaW5nIG1vZHVsZXMgbGlr ZSBjb21wYXQtd2lyZWxlc3MuCi0JaWYgWyAtZiBpbmNsdWRlL2dlbmVyYXRlZC9ib3VuZHMuaCBd OyB0aGVuCi0JCWNwIC1sIGluY2x1ZGUvZ2VuZXJhdGVkL2JvdW5kcy5oICRrZXJuZWxkaXIvaW5j bHVkZS9nZW5lcmF0ZWQvYm91bmRzLmgKLQlmaQotCWlmIFsgLWQgYXJjaC8ke0FSQ0h9L2luY2x1 ZGUvZ2VuZXJhdGVkIF07IHRoZW4KLQkJbWtkaXIgLXAgJGtlcm5lbGRpci9hcmNoLyR7QVJDSH0v aW5jbHVkZS9nZW5lcmF0ZWQvCi0JCWNwIC1mbFIgYXJjaC8ke0FSQ0h9L2luY2x1ZGUvZ2VuZXJh dGVkLyogJGtlcm5lbGRpci9hcmNoLyR7QVJDSH0vaW5jbHVkZS9nZW5lcmF0ZWQvCi0JZmkKLQot CSMgUmVtb3ZlIHRoZSBmb2xsb3dpbmcgYmluYXJpZXMgd2hpY2ggY2F1c2Ugc3RyaXAgb3IgYXJj aCBRQSBlcnJvcnMKLQkjIGR1cmluZyBkb19wYWNrYWdlIGZvciBjcm9zcy1jb21waWxlZCBwbGF0 Zm9ybXMKLQliaW5fZmlsZXM9ImFyY2gvcG93ZXJwYy9ib290L2FkZG5vdGUgYXJjaC9wb3dlcnBj L2Jvb3QvaGFjay1jb2ZmIFwKLQkgICAgICAgICAgIGFyY2gvcG93ZXJwYy9ib290L21rdHJlZSBz Y3JpcHRzL2tjb25maWcvemNvbmYudGFiLm8gXAotCQkgICBzY3JpcHRzL2tjb25maWcvY29uZi5v IHNjcmlwdHMva2NvbmZpZy9reGdldHRleHQubyIKLQlmb3IgZW50cnkgaW4gJGJpbl9maWxlczsg ZG8KLQkJcm0gLWYgJGtlcm5lbGRpci8kZW50cnkKLQlkb25lCi0KLQkjIGtlcm5lbHMgPDIuNi4z MCBkb24ndCBoYXZlICRrZXJuZWxkaXIvdG9vbHMgZGlyZWN0b3J5IHNvIHdlIGNoZWNrIGlmIGl0 IGV4aXN0cyBiZWZvcmUgY2FsbGluZyBzZWQKLQlpZiBbIC1mICRrZXJuZWxkaXIvdG9vbHMvcGVy Zi9NYWtlZmlsZSBdOyB0aGVuCi0JCSMgRml4IFNMQU5HX0lOQyBmb3Igc2xhbmcuaAotCQlzZWQg LWkgJ3MjLUkvdXNyL2luY2x1ZGUvc2xhbmcjLUk9L3Vzci9pbmNsdWRlL3NsYW5nI2cnICRrZXJu ZWxkaXIvdG9vbHMvcGVyZi9NYWtlZmlsZQotCWZpCiB9CiBkb19pbnN0YWxsW3ByZWZ1bmNzXSAr PSAicGFja2FnZV9nZXRfYXV0b19wciIKIAo=