<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>9782</bug_id>
          
          <creation_ts>2016-06-15 11:43:01 +0000</creation_ts>
          <short_desc>Add support for parallel gzip to package creation</short_desc>
          <delta_ts>2019-10-24 22:59:06 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>BitBake</product>
          <component>bitbake</component>
          <version>unspecified</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium</priority>
          <bug_severity>enhancement</bug_severity>
          <target_milestone>0.0.0</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Igor Stoppa">igor.stoppa</reporter>
          <assigned_to name="Ross Burton">ross.burton</assigned_to>
          <cc>markus.lehtonen</cc>
    
    <cc>mingli.yu</cc>
    
    <cc>patrick.ohly</cc>
    
    <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
    
    <cc>randy.macleod</cc>
    
    <cc>richard.purdie</cc>
    
    <cc>ross.burton</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>Don&apos;t know</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>62991</commentid>
    <comment_count>0</comment_count>
    <who name="Igor Stoppa">igor.stoppa</who>
    <bug_when>2016-06-15 11:43:01 +0000</bug_when>
    <thetext>On particularly beefy build machines, it can happen that the system is stalled waiting for a particularly large ipkg being created.

This turns out to be aggravated by gzip not exploiting the available parallelism.

If its parallel version was used, it could take advantage of the other cores that otherwise might be left idling.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>62992</commentid>
    <comment_count>1</comment_count>
    <who name="Patrick Ohly">patrick.ohly</who>
    <bug_when>2016-06-15 12:02:16 +0000</bug_when>
    <thetext>Igor, what was your build stalled on? do_package or do_package_write_ipk?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>62993</commentid>
    <comment_count>2</comment_count>
    <who name="Igor Stoppa">igor.stoppa</who>
    <bug_when>2016-06-15 12:05:00 +0000</bug_when>
    <thetext>The build was stalled for 15 minutes on do_package and following that for 8 minutes on do_pakage_ipk. To give a reference, compiling the package (clang) took less than 7 minutes.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>62994</commentid>
    <comment_count>3</comment_count>
    <who name="Igor Stoppa">igor.stoppa</who>
    <bug_when>2016-06-15 12:08:49 +0000</bug_when>
    <thetext>The clang repo:  git clone -b krogoth https://github.com/kraj/meta-clang</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>63047</commentid>
    <comment_count>4</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2016-06-16 15:27:44 +0000</bug_when>
    <thetext>There are several twisted paths here.

gzip-native has a preferred provider of pigz, so if gzipnative is opted in then you get pigz instead.  If not opted in then host gzip is used.

opkg-native uses libarchive which uses zlib, which isn&apos;t parallel/tuned.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>65608</commentid>
    <comment_count>5</comment_count>
    <who name="Markus Lehtonen">markus.lehtonen</who>
    <bug_when>2016-08-30 11:16:56 +0000</bug_when>
    <thetext>Hmm, I just noticed owning this bug. Probably I volunteered and I don&apos;t remember or something...

Reading through the comments the requirement is for opkg-native to use parallelized implementation of zipping. Reading Ross&apos; comment, using pigz is not an (easy or drop-in) alternative, but, opkg should implement parallel processing itself. AFAIU, zlib does not provide implement anything like that.

I&apos;m not that familiar with opkg or zlib internals so I have no idea how fundamental changes in opkg are required to utilize parallelism. How relevant is this still? I can take a quick look but I doubt that I find the time to implement anything for 2.2</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>65908</commentid>
    <comment_count>6</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2016-09-08 08:37:57 +0000</bug_when>
    <thetext>It&apos;s a vague bug which needs clarification and splitting into the component parts:

* opkg should support parallel zlib implementations
* rpm  -- ditto --
* dpkg -- ditto --
* Vet sstate to ensure parallel implementations are used
* Vet fetcher to ensure parallel decompress is used

Moving to 2.3.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>82360</commentid>
    <comment_count>7</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2018-11-29 16:14:39 +0000</bug_when>
    <thetext>The first step in this bug is to audit the current status of compression/decompression in the various pieces of the system. I&apos;m nominating Ross+myself to figure out a more definitive list of what needs to be done, then we can reassign or open bugs accordingly.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85440</commentid>
    <comment_count>8</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2019-10-24 22:59:06 +0000</bug_when>
    <thetext>When generating packages, rpm/opkg/dpkg all use xz now.

sstate creation uses pigz

gzip *cannot* be decompressed in parallel. pixz can only decompress in parallel if the xz was generated with pixz.

I consider this fixed enough in general.  Specific issues if found can be opened as new bugs.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>