5854 2014-02-20 12:01:36 +0000 Investigate sstate package compression improvements 2023-06-29 16:43:06 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 1.7 All Multiple RESOLVED OBSOLETE Medium enhancement Future 1 bluelightning unassigned meta.mr.watcher meta.watcher randy.macleod richard.purdie --- --- Don't know oldest_to_newest 40489 0 bluelightning 2014-02-20 12:01:36 +0000 We should look to see if there are any benefits in using xz (LZMA) to compress sstate packages over the current gzip, weighing up not only disk usage but also time to decompress on average hardware (and presumably availability of xz packages on supported host distros as well.) 51747 1 bluelightning 2015-06-17 09:05:07 +0000 An old, old patch (that was not merged) that relates to this: http://lists.openembedded.org/pipermail/openembedded-core/2011-December/052656.html 76843 2 richard.purdie 2017-09-14 15:20:52 +0000 We did try xz but the issue is the build slow down in generating these. There is perhaps the idea of a server side process which converted .tgz -> tar.xz. This would mean the code checking sstate servers for multiple different compression types. Or we could have the code just decompress automatically and use a standard extension. Regardless, more thought needed... 95845 3 randy.macleod 2023-06-29 14:49:21 +0000 We switched to zstd. 95866 4 randy.macleod 2023-06-29 16:43:06 +0000 Already changed in October 2021: sstate: Switch to ZStandard compressor support This patch switches the compressor from Gzip to ZStandard for ssate cache files. Zstandard compression provides a significant improvement in decompression speed as well as improvement in compression speed and disk usage over the 'tgz' format in use. Furthermore, its configurable compression level offers a trade-off between time spent compressing sstate cache files and disk space used by those files. The reduced disk usage also contributes to saving network traffic for those sharing their sstate cache with others. Zstandard should therefore be a good choice when: * disk space is at a premium * network speed / resources are limited * the CI server can sstate packages can be created at high compression * less CPU on the build server should be used for sstate decompression Signed-off-by: Henry Kleynhans <hkleynhans@fb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> https://git.openembedded.org/openembedded-core/commit/?id=0710e98f40ec7394567ede1f88f7c7bae306b925