| Summary: | Add support for parallel gzip to package creation | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Igor Stoppa <igor.stoppa> |
| Component: | bitbake | Assignee: | Ross Burton <ross.burton> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | Medium | CC: | markus.lehtonen, mingli.yu, patrick.ohly, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie, ross.burton |
| Version: | unspecified | ||
| Target Milestone: | 0.0.0 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
|
Description
Igor Stoppa
2016-06-15 11:43:01 UTC
Igor, what was your build stalled on? do_package or do_package_write_ipk? The build was stalled for 15 minutes on do_package and following that for 8 minutes on do_pakage_ipk. To give a reference, compiling the package (clang) took less than 7 minutes. The clang repo: git clone -b krogoth https://github.com/kraj/meta-clang There are several twisted paths here. gzip-native has a preferred provider of pigz, so if gzipnative is opted in then you get pigz instead. If not opted in then host gzip is used. opkg-native uses libarchive which uses zlib, which isn't parallel/tuned. Hmm, I just noticed owning this bug. Probably I volunteered and I don't remember or something... Reading through the comments the requirement is for opkg-native to use parallelized implementation of zipping. Reading Ross' comment, using pigz is not an (easy or drop-in) alternative, but, opkg should implement parallel processing itself. AFAIU, zlib does not provide implement anything like that. I'm not that familiar with opkg or zlib internals so I have no idea how fundamental changes in opkg are required to utilize parallelism. How relevant is this still? I can take a quick look but I doubt that I find the time to implement anything for 2.2 It's a vague bug which needs clarification and splitting into the component parts: * opkg should support parallel zlib implementations * rpm -- ditto -- * dpkg -- ditto -- * Vet sstate to ensure parallel implementations are used * Vet fetcher to ensure parallel decompress is used Moving to 2.3. The first step in this bug is to audit the current status of compression/decompression in the various pieces of the system. I'm nominating Ross+myself to figure out a more definitive list of what needs to be done, then we can reassign or open bugs accordingly. When generating packages, rpm/opkg/dpkg all use xz now. sstate creation uses pigz gzip *cannot* be decompressed in parallel. pixz can only decompress in parallel if the xz was generated with pixz. I consider this fixed enough in general. Specific issues if found can be opened as new bugs. |