11009 2017-02-02 17:27:48 +0000 patchelf inflates filesize and causes performance problems 2017-03-07 20:53:52 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain 2.2 x86 Multiple RESOLVED FIXED Medium+ normal 2.3 M3 11123 1 richard.purdie stephano meta.mr.watcher meta.watcher randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 70393 0 richard.purdie 2017-02-02 17:27:48 +0000 When we switched to recipe specific sysroots (rss), performance took a nose dive. Its easy to blame rss but it turns out not to be entirely at fault. Three configurations are compared here: a) Pre-RSS (revision 45df694a9f472ac2f684aadac4d864c3dfdc48a7) b) Post-RSS (revision 226a508da955439b881b2f0a544a3aee76e59919) c) as b) with a tweak to patchelf Overall build times: a) 22794.25user 2687.88system 30:32.84elapsed 1390%CPU (0avgtext+0avgdata 919056maxresident)k b) 22677.25user 3238.79system 36:16.68elapsed 1190%CPU (0avgtext+0avgdata 918896maxresident)k c) 23571.84user 3383.65system 31:36.83elapsed 1421%CPU (0avgtext+0avgdata 919068maxresident)k For the overall build and sstate directories, du -s shows: a) 3992588 build-pre-rss/sstate-cache 30804484 build-pre-rss/tmp b) 4013272 build-with-rss/sstate-cache 36519084 build-with-rss/tmp c) 4014744 build-with-rss2/sstate-cache 35336960 build-with-rss2/tmp However more worryingly: $ du -s build-pre-rss/tmp/sysroots/ 2506092 build-pre-rss/tmp/sysroots/ $ du -s build-with-rss/tmp/sysroots-components/ 3790712 build-with-rss/tmp/sysroots-components/ $ du -s build-with-rss2/tmp/sysroots-components/ 2467544 build-with-rss2/tmp/sysroots-components/ These numbers *should* be equivalent but as you can see, b) is ~1.2GB larger. The reason turned out to be patchelf. Taking a specific binary from a specific recipe, bc from bc-native, in a) its 82kb (stripped) yet in b) its 2.17MB. $ ./patchelf --set-interpreter /bin/rp bc warning: working around a Linux kernel bug by creating a hole of 2084864 bytes in ‘bc’ https://github.com/NixOS/patchelf/blob/master/src/patchelf.cc#L710 shows that this "hole" is just padded zeros using memset, its not a proper sparse hole. My quick hack for c) is: Subject: Make patchelf files sparse Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass index 410fb72..0d1063a 100644 --- a/meta/classes/uninative.bbclass +++ b/meta/classes/uninative.bbclass @@ -121,11 +121,8 @@ python uninative_changeinterp () { if not elf.isDynamic(): continue - try: - subprocess.check_output(("patchelf-uninative", "--set-interpreter", - d.getVar("UNINATIVE_LOADER"), f), - stderr=subprocess.STDOUT) - except subprocess.CalledProcessError as e: - bb.fatal("'%s' failed with exit code %d and the following output:\n%s" % - (e.cmd, e.returncode, e.output)) + subprocess.check_output(("patchelf-uninative", "--set-interpreter", d.getVar("UNINATIVE_LOADER"), f), stderr=subprocess.STDOUT) + subprocess.check_output(("cp", "--sparse=always", f, f + ".sparse"), stderr=subprocess.STDOUT) + os.unlink(f) + os.rename(f + ".sparse", f) } which after patchelf does a cp --sparse=always on the file to turn the padding into sparseness, after which we get the performance back which seems linked to sysroot sizes. Ideally, rather than my hack, we'd teach patchelf to seek() when it writes out the file, creating sparseness out the box and removing the need for my workaround. We'd need to ship a new uninative with the new patchelf to fix this all properly. In the meantime given the performance improvement, I'm tempted to add my workaround. 70650 1 stephano 2017-02-12 17:13:48 +0000 I think the fix here is to store the hole size and location, then call a function from writeFile that uses lseek, at least that's my running theory. A couple questions: 1. Why are we producing binary executables with the e_type set to DYN? 2. PatchELF is working around a kernel bug, but doesn't mention the actual bug it is working around. Any ideas, or shall I assume that is a rabbit hole? 71082 2 richard.purdie 2017-03-06 11:51:49 +0000 (In reply to comment #1) > I think the fix here is to store the hole size and location, then call a > function from writeFile that uses lseek, at least that's my running theory. > A couple questions: > > 1. Why are we producing binary executables with the e_type set to DYN? > 2. PatchELF is working around a kernel bug, but doesn't mention the actual > bug it is working around. Any ideas, or shall I assume that is a rabbit hole? For 1, see bug 11123 which I added as a DEPENDS. These two issues appear related. 71092 3 richard.purdie 2017-03-06 15:49:51 +0000 (In reply to comment #2) > (In reply to comment #1) > > I think the fix here is to store the hole size and location, then call a > > function from writeFile that uses lseek, at least that's my running theory. > > A couple questions: > > > > 1. Why are we producing binary executables with the e_type set to DYN? > > 2. PatchELF is working around a kernel bug, but doesn't mention the actual > > bug it is working around. Any ideas, or shall I assume that is a rabbit hole? The bug seems to be described in: https://github.com/NixOS/patchelf/commit/1a64cf7edc39b93a37beb0665de8f0c6cab90f51 71093 4 richard.purdie 2017-03-06 17:10:57 +0000 To update further, the kernel bug described here does still exist, I was able to test/reproduce it. 71100 5 3643 richard.purdie 2017-03-07 00:09:13 +0000 Created attachment 3643 Patch for the kernel to allow holes in PT_LOAD sections, fixing the bug patchelf referrs to 71113 6 richard.purdie 2017-03-07 20:53:52 +0000 The fix for #11123 also fixes this since it removes the hole code: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c4901328fe5cf912c0965e5b011b64a95a9bcb9d 3643 2017-03-07 00:09:13 +0000 2017-03-07 00:09:13 +0000 Patch for the kernel to allow holes in PT_LOAD sections, fixing the bug patchelf referrs to f1b71802a8b908c3606e5bfe9b779608a5cfbc96.patch text/plain 4613 richard.purdie RnJvbSBmMWI3MTgwMmE4YjkwOGMzNjA2ZTViZmU5Yjc3OTYwOGE1Y2ZiYzk2IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBSaWNoYXJkIFB1cmRpZSA8cmljaGFyZC5wdXJkaWVAbGludXhm b3VuZGF0aW9uLm9yZz4KRGF0ZTogTW9uLCA2IE1hciAyMDE3IDIzOjQzOjMxICswMDAwClN1Ympl Y3Q6IGxpbnV4LXlvY3RvOiBBZGQgYmluZm10X2VsZiBmaXgKClNpZ25lZC1vZmYtYnk6IFJpY2hh cmQgUHVyZGllIDxyaWNoYXJkLnB1cmRpZUBsaW51eGZvdW5kYXRpb24ub3JnPgoKZGlmZiAtLWdp dCBhL21ldGEvcmVjaXBlcy1rZXJuZWwvbGludXgvZmlsZXMvYmluZm10X2VsZl9sb2FkX3NlY3Rp b25zX2ZpeC5wYXRjaCBiL21ldGEvcmVjaXBlcy1rZXJuZWwvbGludXgvZmlsZXMvYmluZm10X2Vs Zl9sb2FkX3NlY3Rpb25zX2ZpeC5wYXRjaApuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAw MDAwLi5mMWJhMWVmCi0tLSAvZGV2L251bGwKKysrIGIvbWV0YS9yZWNpcGVzLWtlcm5lbC9saW51 eC9maWxlcy9iaW5mbXRfZWxmX2xvYWRfc2VjdGlvbnNfZml4LnBhdGNoCkBAIC0wLDAgKzEsODUg QEAKK2JpbmZtdF9lbGY6IERvbid0IHJlcXVpcmUgUFRfTE9BRCBzZWN0aW9ucyB3aXRob3V0IGhv bGVzLgorCitUaGUga2VybmVsIG1ha2VzIHRoZSBhc3N1bXB0aW9uIHRoYXQgYWxsIFBUX0xPQUQg c2VjdGlvbnMgYXJlIGNvbnRpbm91cywgYXQKK2xlYXN0IHRvIHRoZSBwb2ludCB3aGVyZSBQVF9Q SERSIGlzIGNvbnRhaW5lZC4gSWYgdGhlcmUgYXJlIGhvbGVzIGJldHdlZW4KK3RoZSBQVF9MT0FE IHNlY3Rpb25zLCB0aGUgY2FsY3VsYXRpb24gbG9hZF9hZGRyICsgZXhlYy0+ZV9waG9mZiBmYWls cyB0bworY29ycmVjdGx5IGxvY2F0ZSB0aGUgcGhkcnMuCisKK1RoaXMgY2hhbmdlIG5vdGljZXMg d2hlcmUgcGhkcnMgYXJlIGxvY2F0ZWQgaW4gc29tZSBvdGhlciBQVF9MT0FEIHNlY3Rpb24KK290 aGVyIHRoYW4gdGhlIGZpcnN0IG9uZSBhbmQgY29ycmVjdHMgdGhlIG9mZnNldHMgdXNlZC4KKwor VGhpcyBpcyB1c2VmdWwgd2hlcmUgdXNlcnNwYWNlIHdhbnRzIHRvIHJld3JpdGUgZWxmIGZpbGVz IGJ1dCBjYW4ndCBndWFyYW50ZWUKK3RoYXQgYWxsIFBUX0xPQUQgc2VjdGlvbnMgYXJlIGNvbnRp bnVvdXMgd2l0aG91dCBob2xlcyB3aXRob3V0IHBvdGVudGlhbGx5CitjcmVhdGluZyBsYXJnZSBo b2xlcyBpbiB0aGUgYmluYXJpZXMuIEkgd2FzIHVuYWJsZSB0byBmaW5kIGFueSBFTEYgc3BlYwor d2hpY2ggcmVxdWlyZXMgY29udGludW91cyBQVF9MT0FEIHNlY3Rpb25zIG9yIHRoYXQgUFRfUEhE UiBiZSBpbiB0aGUgZmlyc3QKK1BUX0xPQUQgc2VjdGlvbi4KKworU2lnbmVkLW9mZi1ieTogUmlj aGFyZCBQdXJkaWUgPHJpY2hhcmQucHVyZGllQGxpbnV4Zm91bmRhdGlvbi5vcmc+CisKK2RpZmYg LS1naXQgYS9mcy9iaW5mbXRfZWxmLmMgYi9mcy9iaW5mbXRfZWxmLmMKK2luZGV4IDQyMjM3MDIu LjM5MmRhYjMgMTAwNjQ0CistLS0gYS9mcy9iaW5mbXRfZWxmLmMKKysrKyBiL2ZzL2JpbmZtdF9l bGYuYworQEAgLTE0OCw3ICsxNDgsNyBAQCBzdGF0aWMgaW50IHBhZHplcm8odW5zaWduZWQgbG9u ZyBlbGZfYnNzKQorIAorIHN0YXRpYyBpbnQKKyBjcmVhdGVfZWxmX3RhYmxlcyhzdHJ1Y3QgbGlu dXhfYmlucHJtICpicHJtLCBzdHJ1Y3QgZWxmaGRyICpleGVjLAorLQkJdW5zaWduZWQgbG9uZyBs b2FkX2FkZHIsIHVuc2lnbmVkIGxvbmcgaW50ZXJwX2xvYWRfYWRkcikKKysJCXVuc2lnbmVkIGxv bmcgbG9hZF9hZGRyLCB1bnNpZ25lZCBsb25nIGludGVycF9sb2FkX2FkZHIsIHVuc2lnbmVkIGxv bmcgcGhkcl9hZGRyKQorIHsKKyAJdW5zaWduZWQgbG9uZyBwID0gYnBybS0+cDsKKyAJaW50IGFy Z2MgPSBicHJtLT5hcmdjOworQEAgLTIzNCw3ICsyMzQsNyBAQCBjcmVhdGVfZWxmX3RhYmxlcyhz dHJ1Y3QgbGludXhfYmlucHJtICpicHJtLCBzdHJ1Y3QgZWxmaGRyICpleGVjLAorIAlORVdfQVVY X0VOVChBVF9IV0NBUCwgRUxGX0hXQ0FQKTsKKyAJTkVXX0FVWF9FTlQoQVRfUEFHRVNaLCBFTEZf RVhFQ19QQUdFU0laRSk7CisgCU5FV19BVVhfRU5UKEFUX0NMS1RDSywgQ0xPQ0tTX1BFUl9TRUMp OworLQlORVdfQVVYX0VOVChBVF9QSERSLCBsb2FkX2FkZHIgKyBleGVjLT5lX3Bob2ZmKTsKKysJ TkVXX0FVWF9FTlQoQVRfUEhEUiwgcGhkcl9hZGRyKTsKKyAJTkVXX0FVWF9FTlQoQVRfUEhFTlQs IHNpemVvZihzdHJ1Y3QgZWxmX3BoZHIpKTsKKyAJTkVXX0FVWF9FTlQoQVRfUEhOVU0sIGV4ZWMt PmVfcGhudW0pOworIAlORVdfQVVYX0VOVChBVF9CQVNFLCBpbnRlcnBfbG9hZF9hZGRyKTsKK0BA IC02NjgsNyArNjY4LDcgQEAgc3RhdGljIHVuc2lnbmVkIGxvbmcgcmFuZG9taXplX3N0YWNrX3Rv cCh1bnNpZ25lZCBsb25nIHN0YWNrX3RvcCkKKyBzdGF0aWMgaW50IGxvYWRfZWxmX2JpbmFyeShz dHJ1Y3QgbGludXhfYmlucHJtICpicHJtKQorIHsKKyAJc3RydWN0IGZpbGUgKmludGVycHJldGVy ID0gTlVMTDsgLyogdG8gc2h1dCBnY2MgdXAgKi8KKy0gCXVuc2lnbmVkIGxvbmcgbG9hZF9hZGRy ID0gMCwgbG9hZF9iaWFzID0gMDsKKysJdW5zaWduZWQgbG9uZyBsb2FkX2FkZHIgPSAwLCBsb2Fk X2JpYXMgPSAwLCBwaGRyX2FkZHIgPSAwLCBwaGRyX29mZnNldCA9IDA7CisgCWludCBsb2FkX2Fk ZHJfc2V0ID0gMDsKKyAJY2hhciAqIGVsZl9pbnRlcnByZXRlciA9IE5VTEw7CisgCXVuc2lnbmVk IGxvbmcgZXJyb3I7CitAQCAtNzk1LDYgKzc5NSwxMCBAQCBzdGF0aWMgaW50IGxvYWRfZWxmX2Jp bmFyeShzdHJ1Y3QgbGludXhfYmlucHJtICpicHJtKQorIAkJCWlmIChyZXR2YWwpCisgCQkJCWdv dG8gb3V0X2ZyZWVfZGVudHJ5OworIAkJCWJyZWFrOworKworKwkJY2FzZSBQVF9QSERSOgorKwkJ CXBoZHJfb2Zmc2V0ID0gZWxmX3BwbnQtPnBfb2Zmc2V0OworKwkJCWJyZWFrOworIAkJfQorIAor IAkvKiBTb21lIHNpbXBsZSBjb25zaXN0ZW5jeSBjaGVja3MgZm9yIHRoZSBpbnRlcnByZXRlciAq LworQEAgLTk0Nyw3ICs5NTEsMTMgQEAgc3RhdGljIGludCBsb2FkX2VsZl9iaW5hcnkoc3RydWN0 IGxpbnV4X2JpbnBybSAqYnBybSkKKyAJCQkJbG9hZF9hZGRyICs9IGxvYWRfYmlhczsKKyAJCQkJ cmVsb2NfZnVuY19kZXNjID0gbG9hZF9iaWFzOworIAkJCX0KKysJCQlwaGRyX2FkZHIgPSBsb2Fk X2FkZHI7CisrCQl9IGVsc2UgaWYgKHBoZHJfb2Zmc2V0ID49IGVsZl9wcG50LT5wX29mZnNldCAm JiBwaGRyX29mZnNldCA8IChlbGZfcHBudC0+cF9vZmZzZXQgKyBlbGZfcHBudC0+cF9maWxlc3op KSB7CisrCQkJcGhkcl9hZGRyID0gcGhkcl9hZGRyICsgZXJyb3IgLQorKwkJCQkgICAgRUxGX1BB R0VTVEFSVChsb2FkX2JpYXMgKyB2YWRkcikgKyAKKysJCQkJICAgIGVsZl9wcG50LT5wX3ZhZGRy IC0gZWxmX3BwbnQtPnBfb2Zmc2V0OworIAkJfQorKworIAkJayA9IGVsZl9wcG50LT5wX3ZhZGRy OworIAkJaWYgKGsgPCBzdGFydF9jb2RlKQorIAkJCXN0YXJ0X2NvZGUgPSBrOworQEAgLTEwNDUs OCArMTA1NSw5IEBAIHN0YXRpYyBpbnQgbG9hZF9lbGZfYmluYXJ5KHN0cnVjdCBsaW51eF9iaW5w cm0gKmJwcm0pCisgCQlnb3RvIG91dDsKKyAjZW5kaWYgLyogQVJDSF9IQVNfU0VUVVBfQURESVRJ T05BTF9QQUdFUyAqLworIAorKyAgICAgICAgcGhkcl9hZGRyID0gcGhkcl9hZGRyICsgbG9jLT5l bGZfZXguZV9waG9mZjsKKyAJcmV0dmFsID0gY3JlYXRlX2VsZl90YWJsZXMoYnBybSwgJmxvYy0+ ZWxmX2V4LAorLQkJCSAgbG9hZF9hZGRyLCBpbnRlcnBfbG9hZF9hZGRyKTsKKysJCQkgIGxvYWRf YWRkciwgaW50ZXJwX2xvYWRfYWRkciwgcGhkcl9hZGRyKTsKKyAJaWYgKHJldHZhbCA8IDApCisg CQlnb3RvIG91dDsKKyAJLyogTi5CLiBwYXNzZWRfZmlsZW5vIG1pZ2h0IG5vdCBiZSBpbml0aWFs aXplZD8gKi8KZGlmZiAtLWdpdCBhL21ldGEvcmVjaXBlcy1rZXJuZWwvbGludXgvbGludXgteW9j dG9fNC4xMC5iYiBiL21ldGEvcmVjaXBlcy1rZXJuZWwvbGludXgvbGludXgteW9jdG9fNC4xMC5i YgppbmRleCBkYjZmYTdlLi42NGZiMjAzIDEwMDY0NAotLS0gYS9tZXRhL3JlY2lwZXMta2VybmVs L2xpbnV4L2xpbnV4LXlvY3RvXzQuMTAuYmIKKysrIGIvbWV0YS9yZWNpcGVzLWtlcm5lbC9saW51 eC9saW51eC15b2N0b180LjEwLmJiCkBAIC0yNCw2ICsyNCw4IEBAIFNSQ1JFVl9tZXRhID89ICI4 MDVlYTQ0MGM3OTFhY2VlNDYxNzgyMGFiMzJlOTNmMWNjYmQ5YWUyIgogU1JDX1VSSSA9ICJnaXQ6 Ly9naXQueW9jdG9wcm9qZWN0Lm9yZy9saW51eC15b2N0by00LjEwLmdpdDtuYW1lPW1hY2hpbmU7 YnJhbmNoPSR7S0JSQU5DSH07IFwKICAgICAgICAgICAgZ2l0Oi8vZ2l0LnlvY3RvcHJvamVjdC5v cmcveW9jdG8ta2VybmVsLWNhY2hlO3R5cGU9a21ldGE7bmFtZT1tZXRhO2JyYW5jaD15b2N0by00 LjEwO2Rlc3RzdWZmaXg9JHtLTUVUQX0iCiAKK1NSQ19VUkkgKz0gImZpbGU6Ly9iaW5mbXRfZWxm X2xvYWRfc2VjdGlvbnNfZml4LnBhdGNoIgorCiBMSU5VWF9WRVJTSU9OID89ICI0LjEwIgogCiBQ ViA9ICIke0xJTlVYX1ZFUlNJT059K2dpdCR7U1JDUFZ9IgotLSAKY2dpdCB2MC4xMC4yCgo=