13113 2019-01-03 07:09:13 +0000 Accelerate out-of-tree kernel module built in eSDK 2019-01-11 02:12:52 +0000 1 1 6 Yocto Project Subprojects eSDK eSDK 2.7 x86 Multiple RESOLVED WONTFIX Undecided enhancement --- 1 zhangzl2013 bluelightning bruce.ashfield randy.macleod richard.purdie francisco.j.pedraza.gonzalez --- --- No (bug/feature does not impact docs) oldest_to_newest 82640 0 zhangzl2013 2019-01-03 07:09:13 +0000 When building out-of-tree kernel modules in eSDK, it will always trigger a new rebuilding of kernel, which consumes lots of time. I have tried to add virtual/kernel:do_shared_workdir to sstate cache and ship it with eSDK. On the eSDK side, run `bitbake linux-yocto -c shared_workdir` returns immediatlely, but `bitbake hello-mod` still triggers tasks from linux-yocto:do_fetch. Adding this enhancement, it may need more time to generate the eSDK installer, but it will save considerably time at the eSDK user's side. 82653 1 richard.purdie 2019-01-03 16:25:34 +0000 As discussed on the mailing list, this is not something we plan to support. There is no technical reason that do_shared_workdir couldn't be an sstate task however the combination of the kernel source and the kernel compiled artifacts is large and results in a large sstate object. This means the time to build the kernel increases substantially. Using the the object in a new build, or in an eSDK then results in having to transfer a large object over a network which is also slow unless its a local transfer so there is a double hit. We made a decision as a project to just rebuild these pieces if/as needed rather than take the hit on the performance of the base kernel builds. I appreciate that doesn't work for every workload but it is the tradeoff we decided to make. If should be possible to patch do_shared_work to be an sstate task, however that code path will be quite different from the current behavior. We don't have the build resources to be able to test these two different code paths (does devtoool work with both, do external modules work with both etc). We're therefore unlikely to merge such a patch. I've cc'd Bruce in case he has any different viewpoint on this. 82665 2 richard.purdie 2019-01-03 22:52:06 +0000 Just to additionally note, there was a change submitted which makes it easier to implement this: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=aa83cb5264554dd1963042f905688afc796e98d0 I decided to merge that to at least make this option easier to implement. 82696 3 bruce.ashfield 2019-01-07 13:55:55 +0000 Just documenting that my view is the same as RPs. What made it into the tree looks appropriate. 82748 4 4421 zhangzl2013 2019-01-11 02:12:52 +0000 Created attachment 4421 add STAGING_KERNEL_BUILDDIR and STAGING_KERNEL_DIR to eSDK For anyone that may need this acceleration, this patch is for your reference. Apply this patch as well as http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=aa83cb5264554dd1963042f905688afc796e98d0. My testing result: With patch: Orig: =========== ===== build and export eSDK: ---------------------- real 72m28.549s real 69m17.119s user 653m12.390s user 645m38.162s sys 154m17.175s sys 153m22.593s eSDK installer size: -------------------- 4.6G 4.4G eSDK installation: ------------------ real 6m40.714s real 8m7.331s user 24m19.822s user 25m41.045s sys 4m17.492s sys 4m23.538s build hello-mod in eSDK: ------------------------ real 0m31.966s real 21m7.686s user 3m48.485s user 35m10.894s sys 0m7.827s sys 15m3.837s 4421 2019-01-11 02:12:52 +0000 2019-01-11 02:12:52 +0000 add STAGING_KERNEL_BUILDDIR and STAGING_KERNEL_DIR to eSDK 0001-eSDK-sstate-kernel-stuffs-and-export-to-eSDK.patch text/plain 2220 zhangzl2013 RnJvbSA5MzIzMmYzYWU5NmYzZGRkNTQ2OGY0ZTkzZmZmZjAxNTkxOWMyZDBhIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBaaGFvbG9uZyBaaGFuZyA8emhhbmd6bDIwMTNAMTI2LmNvbT4K RGF0ZTogVGh1LCAxMCBKYW4gMjAxOSAxMzo1NDo0OSArMDgwMApTdWJqZWN0OiBbUEFUQ0hdIGVT REs6IHNzdGF0ZSBrZXJuZWwgc3R1ZmZzIGFuZCBleHBvcnQgdG8gZVNESwoKU2lnbmVkLW9mZi1i eTogWmhhb2xvbmcgWmhhbmcgPHpoYW5nemwyMDEzQDEyNi5jb20+Ci0tLQogbWV0YS9jbGFzc2Vz L2tlcm5lbC5iYmNsYXNzICAgICAgICAgICB8IDEzICsrKysrKysrKysrLS0KIG1ldGEvY2xhc3Nl cy9wb3B1bGF0ZV9zZGtfZXh0LmJiY2xhc3MgfCAgMiArKwogMiBmaWxlcyBjaGFuZ2VkLCAxMyBp bnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy9r ZXJuZWwuYmJjbGFzcyBiL21ldGEvY2xhc3Nlcy9rZXJuZWwuYmJjbGFzcwppbmRleCA3OGQ2YzMw Li45YjkxNjg4IDEwMDY0NAotLS0gYS9tZXRhL2NsYXNzZXMva2VybmVsLmJiY2xhc3MKKysrIGIv bWV0YS9jbGFzc2VzL2tlcm5lbC5iYmNsYXNzCkBAIC0zNDEsNiArMzQxLDEzIEBAIGRvX2NvbXBp bGVfa2VybmVsbW9kdWxlcygpIHsKIAlmaQogfQogYWRkdGFzayBjb21waWxlX2tlcm5lbG1vZHVs ZXMgYWZ0ZXIgZG9fY29tcGlsZSBiZWZvcmUgZG9fc3RyaXAKK2FkZHRhc2sgY29tcGlsZV9rZXJu ZWxtb2R1bGVzX3NldHNjZW5lCisKK1NTVEFURVRBU0tTICs9ICJkb19jb21waWxlX2tlcm5lbG1v ZHVsZXMiCitweXRob24gZG9fY29tcGlsZV9rZXJuZWxtb2R1bGVzX3NldHNjZW5lICgpIHsKKyAg ICBzc3RhdGVfc2V0c2NlbmUoZCkKK30KK2RvX2NvbXBpbGVfa2VybmVsbW9kdWxlc1tzc3RhdGUt cGxhaW5kaXJzXSA9ICIke1NUQUdJTkdfS0VSTkVMX0JVSUxERElSfSIKIAoga2VybmVsX2RvX2lu c3RhbGwoKSB7CiAJIwpAQCAtNDE0LDggKzQyMSw5IEBAIGRvX2tlcm5lbF92ZXJzaW9uX3Nhbml0 eV9jaGVjaygpIHsKIGFkZHRhc2sgc2hhcmVkX3dvcmtkaXIgYWZ0ZXIgZG9fY29tcGlsZSBiZWZv cmUgZG9fY29tcGlsZV9rZXJuZWxtb2R1bGVzCiBhZGR0YXNrIHNoYXJlZF93b3JrZGlyX3NldHNj ZW5lCiAKLWRvX3NoYXJlZF93b3JrZGlyX3NldHNjZW5lICgpIHsKLQlleGl0IDEKK1NTVEFURVRB U0tTICs9ICJkb19zaGFyZWRfd29ya2RpciIKK3B5dGhvbiBkb19zaGFyZWRfd29ya2Rpcl9zZXRz Y2VuZSAoKSB7CisgICAgc3N0YXRlX3NldHNjZW5lKGQpCiB9CiAKIGVtaXRfZGVwbW9kX3BrZ2Rh dGEoKSB7CkBAIC00MjcsNiArNDM1LDcgQEAgZW1pdF9kZXBtb2RfcGtnZGF0YSgpIHsKIAogUEFD S0FHRUZVTkNTICs9ICJlbWl0X2RlcG1vZF9wa2dkYXRhIgogCitkb19zaGFyZWRfd29ya2Rpcltz c3RhdGUtcGxhaW5kaXJzXSA9ICIke1NUQUdJTkdfS0VSTkVMX0RJUn0iCiBkb19zaGFyZWRfd29y a2RpcltjbGVhbmRpcnNdICs9ICIgJHtTVEFHSU5HX0tFUk5FTF9CVUlMRERJUn0iCiBkb19zaGFy ZWRfd29ya2RpciAoKSB7CiAJY2QgJHtCfQpkaWZmIC0tZ2l0IGEvbWV0YS9jbGFzc2VzL3BvcHVs YXRlX3Nka19leHQuYmJjbGFzcyBiL21ldGEvY2xhc3Nlcy9wb3B1bGF0ZV9zZGtfZXh0LmJiY2xh c3MKaW5kZXggZTFiYmE0OS4uNjFiZDMyMCAxMDA2NDQKLS0tIGEvbWV0YS9jbGFzc2VzL3BvcHVs YXRlX3Nka19leHQuYmJjbGFzcworKysgYi9tZXRhL2NsYXNzZXMvcG9wdWxhdGVfc2RrX2V4dC5i YmNsYXNzCkBAIC01OSw2ICs1OSw4IEBAIGRlZiBnZXRfc2RrX2luc3RhbGxfdGFyZ2V0cyhkLCBp bWFnZXNfb25seT1GYWxzZSk6CiAgICAgICAgIGlmIGQuZ2V0VmFyKCdTREtfSU5DTFVERV9UT09M Q0hBSU4nKSA9PSAnMSc6CiAgICAgICAgICAgICBzZGtfaW5zdGFsbF90YXJnZXRzICs9ICcgbWV0 YS1leHRzZGstdG9vbGNoYWluOmRvX3BvcHVsYXRlX3N5c3Jvb3QnCiAKKyAgICBzZGtfaW5zdGFs bF90YXJnZXRzICs9ICIgdmlydHVhbC9rZXJuZWw6ZG9fc2hhcmVkX3dvcmtkaXIiCisgICAgc2Rr X2luc3RhbGxfdGFyZ2V0cyArPSAiIHZpcnR1YWwva2VybmVsOmRvX2NvbXBpbGVfa2VybmVsbW9k dWxlcyIKICAgICByZXR1cm4gc2RrX2luc3RhbGxfdGFyZ2V0cwogCiBnZXRfc2RrX2luc3RhbGxf dGFyZ2V0c1t2YXJkZXBzZXhjbHVkZV0gPSAiQkJfVEFTS0RFUERBVEEiCi0tIAoxLjkuMQoK