When we switched to recipe specific sysroots (rss), performance took a nose dive. Its easy to blame rss but it turns out not to be entirely at fault.

Three configurations are compared here:

a) Pre-RSS (revision 45df694a9f472ac2f684aadac4d864c3dfdc48a7)
b) Post-RSS (revision 226a508da955439b881b2f0a544a3aee76e59919)
c) as b) with a tweak to patchelf

Overall build times:

a) 22794.25user 2687.88system 30:32.84elapsed 1390%CPU (0avgtext+0avgdata 919056maxresident)k
b) 22677.25user 3238.79system 36:16.68elapsed 1190%CPU (0avgtext+0avgdata 918896maxresident)k
c) 23571.84user 3383.65system 31:36.83elapsed 1421%CPU (0avgtext+0avgdata 919068maxresident)k

For the overall build and sstate directories, du -s shows:
a)
3992588	build-pre-rss/sstate-cache
30804484	build-pre-rss/tmp
b)
4013272	build-with-rss/sstate-cache
36519084	build-with-rss/tmp
c)
4014744	build-with-rss2/sstate-cache
35336960	build-with-rss2/tmp

However more worryingly:

$ du -s build-pre-rss/tmp/sysroots/
2506092	build-pre-rss/tmp/sysroots/
$ du -s build-with-rss/tmp/sysroots-components/
3790712	build-with-rss/tmp/sysroots-components/
$ du -s build-with-rss2/tmp/sysroots-components/
2467544	build-with-rss2/tmp/sysroots-components/

These numbers *should* be equivalent but as you can see, b) is ~1.2GB larger. The reason turned out to be patchelf.

Taking a specific binary from a specific recipe, bc from bc-native, in a) its 82kb (stripped) yet in b) its 2.17MB.

$ ./patchelf --set-interpreter /bin/rp bc
warning: working around a Linux kernel bug by creating a hole of 2084864 bytes in ‘bc’

https://github.com/NixOS/patchelf/blob/master/src/patchelf.cc#L710 shows that this "hole" is just padded zeros using memset, its not a proper sparse hole.

My quick hack for c) is:

Subject: Make patchelf files sparse

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 410fb72..0d1063a 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -121,11 +121,8 @@ python uninative_changeinterp () {
             if not elf.isDynamic():
                 continue
 
-            try:
-                subprocess.check_output(("patchelf-uninative", "--set-interpreter",
-                                         d.getVar("UNINATIVE_LOADER"), f),
-                                        stderr=subprocess.STDOUT)
-            except subprocess.CalledProcessError as e:
-                bb.fatal("'%s' failed with exit code %d and the following output:\n%s" %
-                         (e.cmd, e.returncode, e.output))
+            subprocess.check_output(("patchelf-uninative", "--set-interpreter", d.getVar("UNINATIVE_LOADER"), f), stderr=subprocess.STDOUT)
+            subprocess.check_output(("cp", "--sparse=always", f, f + ".sparse"), stderr=subprocess.STDOUT)
+            os.unlink(f)
+            os.rename(f + ".sparse", f)
 }

which after patchelf does a cp --sparse=always on the file to turn the padding into sparseness, after which we get the performance back which seems linked to sysroot sizes.

Ideally, rather than my hack, we'd teach patchelf to seek() when it writes out the file, creating sparseness out the box and removing the need for my workaround. We'd need to ship a new uninative with the new patchelf to fix this all properly. In the meantime given the performance improvement, I'm tempted to add my workaround.