Bug 9782

Summary: Add support for parallel gzip to package creation
Product: [Build System, Metadata & Runtime] BitBake Reporter: Igor Stoppa <igor.stoppa>
Component: bitbakeAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium CC: markus.lehtonen, mingli.yu, patrick.ohly, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie, ross.burton
Version: unspecified   
Target Milestone: 0.0.0   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Igor Stoppa 2016-06-15 11:43:01 UTC
On particularly beefy build machines, it can happen that the system is stalled waiting for a particularly large ipkg being created.

This turns out to be aggravated by gzip not exploiting the available parallelism.

If its parallel version was used, it could take advantage of the other cores that otherwise might be left idling.
Comment 1 Patrick Ohly 2016-06-15 12:02:16 UTC
Igor, what was your build stalled on? do_package or do_package_write_ipk?
Comment 2 Igor Stoppa 2016-06-15 12:05:00 UTC
The build was stalled for 15 minutes on do_package and following that for 8 minutes on do_pakage_ipk. To give a reference, compiling the package (clang) took less than 7 minutes.
Comment 3 Igor Stoppa 2016-06-15 12:08:49 UTC
The clang repo:  git clone -b krogoth https://github.com/kraj/meta-clang
Comment 4 Ross Burton 2016-06-16 15:27:44 UTC
There are several twisted paths here.

gzip-native has a preferred provider of pigz, so if gzipnative is opted in then you get pigz instead.  If not opted in then host gzip is used.

opkg-native uses libarchive which uses zlib, which isn't parallel/tuned.
Comment 5 Markus Lehtonen 2016-08-30 11:16:56 UTC
Hmm, I just noticed owning this bug. Probably I volunteered and I don't remember or something...

Reading through the comments the requirement is for opkg-native to use parallelized implementation of zipping. Reading Ross' comment, using pigz is not an (easy or drop-in) alternative, but, opkg should implement parallel processing itself. AFAIU, zlib does not provide implement anything like that.

I'm not that familiar with opkg or zlib internals so I have no idea how fundamental changes in opkg are required to utilize parallelism. How relevant is this still? I can take a quick look but I doubt that I find the time to implement anything for 2.2
Comment 6 Ross Burton 2016-09-08 08:37:57 UTC
It's a vague bug which needs clarification and splitting into the component parts:

* opkg should support parallel zlib implementations
* rpm  -- ditto --
* dpkg -- ditto --
* Vet sstate to ensure parallel implementations are used
* Vet fetcher to ensure parallel decompress is used

Moving to 2.3.
Comment 7 Richard Purdie 2018-11-29 16:14:39 UTC
The first step in this bug is to audit the current status of compression/decompression in the various pieces of the system. I'm nominating Ross+myself to figure out a more definitive list of what needs to be done, then we can reassign or open bugs accordingly.
Comment 8 Ross Burton 2019-10-24 22:59:06 UTC
When generating packages, rpm/opkg/dpkg all use xz now.

sstate creation uses pigz

gzip *cannot* be decompressed in parallel. pixz can only decompress in parallel if the xz was generated with pixz.

I consider this fixed enough in general.  Specific issues if found can be opened as new bugs.