All of the system() calls to tar really impact performance as each system() call adds another pseudo load. We'll work with swupd upstream to add an optional libarchive backend, the methodology will be: * add tests for the tar calls * abstract out tar usage into a module/backend * implement a compatible libarchive module/backend * add configure machinery to choose backend
Took a quick look at the latest swupd-server master branch to see how widely tar is used. There are currently 11 different locations where TAR_COMMAND is used to construct a tar call, they seem to follow 4 or 5 different patterns. Should be simple enough to abstract these 4 or 5 patterns out into some functions which we can then look at providing a libarchive variant of.
Created attachment 3282 [details] multi-threaded benchmarks The attached program simulates the behavior of swupd_make_fullfiles: it runs 5000 tasks in 48 parallel threads. Each task is a series of creating/measuring/unlinking files (like moving a file to a bsdtar archive with different compression types and measuring the size in order to find the optimal compression type). Under pseudo it takes 10 times more time to run than with no pseudo. The reason is that pseudo serializes any parallelism of the program it's loaded into by setting a mutex on wrapped syscalls. Also the pseudo-server is a single-threaded application serializing access to sqlite. If to modify the benchmark to run tar with `system()` call in every task then the test gets ~15 time slower under pseudo, but again it could be much faster if pseudo used all available CPU cores.
This won't make M3 and based on Dmitry's analysis it may not even provide the big win we were hoping for.
Added Peter Seebach since I suspect he might be interested in a benchmark that makes pseudo perform badly...
It looks like a libarchive backend for swupd-server won't make as significant a difference as first expected. Most likely we should close this issue and open an enhancement request against Pseudo but deferring on that for now.
The slowdown on parallel system calls is probably very close to impossible to correct without MASSIVE increases in overhead and risk of serious errors. Note that actually distinct processes won't have that problem. And while the server is still serializing operations, that's actually usually a non-issue, especially when FASTOP is in use.
Performance of swupd_make_fullfiles gets much better under pseudo when invoking libarchive directly instead of invoking an external tar command. I have patches for that and various other enhancements. No enhancements of pseudo are needed. Currently I am trying to convince the swupd maintainers to accept those patches, to avoid forking the swupd code. See https://github.com/clearlinux/swupd-server/pull/34 and https://github.com/ostroproject/ostro-os/pull/198
Reassigning to Patrick as he's already made excellent progress here and has patches in progress. Patrick: please let me know if you're unhappy with this or if you feel the target milestone is inappropriate.
I just update meta-swupd master. The current revision (07ad71b789d) has implemented various performance improvements such that building update data for a build in which nothing changed takes only five minutes instead of more than two hours (in Ostro OS).