On particularly beefy build machines, it can happen that the system is stalled waiting for a particularly large ipkg being created. This turns out to be aggravated by gzip not exploiting the available parallelism. If its parallel version was used, it could take advantage of the other cores that otherwise might be left idling.
Igor, what was your build stalled on? do_package or do_package_write_ipk?
The build was stalled for 15 minutes on do_package and following that for 8 minutes on do_pakage_ipk. To give a reference, compiling the package (clang) took less than 7 minutes.
The clang repo: git clone -b krogoth https://github.com/kraj/meta-clang
There are several twisted paths here. gzip-native has a preferred provider of pigz, so if gzipnative is opted in then you get pigz instead. If not opted in then host gzip is used. opkg-native uses libarchive which uses zlib, which isn't parallel/tuned.
Hmm, I just noticed owning this bug. Probably I volunteered and I don't remember or something... Reading through the comments the requirement is for opkg-native to use parallelized implementation of zipping. Reading Ross' comment, using pigz is not an (easy or drop-in) alternative, but, opkg should implement parallel processing itself. AFAIU, zlib does not provide implement anything like that. I'm not that familiar with opkg or zlib internals so I have no idea how fundamental changes in opkg are required to utilize parallelism. How relevant is this still? I can take a quick look but I doubt that I find the time to implement anything for 2.2
It's a vague bug which needs clarification and splitting into the component parts: * opkg should support parallel zlib implementations * rpm -- ditto -- * dpkg -- ditto -- * Vet sstate to ensure parallel implementations are used * Vet fetcher to ensure parallel decompress is used Moving to 2.3.
The first step in this bug is to audit the current status of compression/decompression in the various pieces of the system. I'm nominating Ross+myself to figure out a more definitive list of what needs to be done, then we can reassign or open bugs accordingly.
When generating packages, rpm/opkg/dpkg all use xz now. sstate creation uses pigz gzip *cannot* be decompressed in parallel. pixz can only decompress in parallel if the xz was generated with pixz. I consider this fixed enough in general. Specific issues if found can be opened as new bugs.