<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>1166</bug_id>
          
          <creation_ts>2011-06-13 19:25:52 +0000</creation_ts>
          <short_desc>Reduce reparsing impact and cache dependencies calc results in bitbake</short_desc>
          <delta_ts>2011-10-07 06:28:57 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>BitBake</product>
          <component>bitbake</component>
          <version>unspecified</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Low</priority>
          <bug_severity>enhancement</bug_severity>
          <target_milestone>1.2</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="liping Ke">liping.ke</reporter>
          <assigned_to name="Richard Purdie">richard.purdie</assigned_to>
          <cc>jessica.zhang</cc>
    
    <cc>josh</cc>
    
    <cc>ke.yu</cc>
    
    <cc>lianhao.lu</cc>
    
    <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
    
    <cc>sgw</cc>
    
    <cc>shane.wang</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>---</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>14278</commentid>
    <comment_count>0</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-06-13 19:25:52 +0000</bug_when>
    <thetext>I am looking@Image Creator. Because of the current bitbake limitation, image creator is extremely slow because:
1) recipe files need reparsing always
2) Getting the packages dependency relationship is very slow.

When kicking off bitbake command line, we assume that one bitbake will run long time, so XX seconds are bearable. But for image creator, it&apos;s UI ops, this kind of response is really annoying.

From a outdoor user, I have below feelings:
1. Can bitbake reduce reparse chances, eg, some var only impact running configurations, such as thread numbers, we shouldn&apos;t reparse at all. We need to tell whether the changes need reparsing which means we need to consider the finer granularity of configuration management. Such as move smaller impact configurations into one files and need-reparsing vars into another files?
2. Can bitbake cache tree dependencies into local database (I mean all packages of an arch)? Users can explicitly run and get the fresh result. After that, if he changes recipes, he can run again. when using, he can request directly get the result from cache. This dependency calculation speed is really very slow.
4. Image creator in the long run should be the front-end UI of configuration, as well as an UI for triggering bitbake build. As an configuration UI, it should be fast and well integrated, such as modifying confs/recipes and updating bitbake parsing result cache (update tree dependencies, etc). 

Record my feelings after playing image creator and for all your reference.

Regards
criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14465</commentid>
    <comment_count>1</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2011-06-29 03:05:17 +0000</bug_when>
    <thetext>I would like to understand the information that the image creator UI is requiring which is taking so long to obtain.

Assuming the cache is hot, things should be very fast and for example runqueue can quickly compute dependency information about builds. Is the problem obtaining the data or it is turning it into a form the UI can use in its treeview?

I&apos;ve had a lot of experience over the years of tuning bitbake internals to be fast so perhaps if you can illustrate where the slowdowns are and how to demonstrate them, I can take a look and see if I have any specific ideas.

Regarding recipe parsing when variables like BB_NUMBER_THREADS change, the way the bitbake data model is structured, its very very hard to predict what effect changing a variable will have on the metadata. There are some clear cases we can make predictions but its not always clear. Now we have the checksum info with the dependency information about variables we would have a place to start considering things like this but its still a major amount of work. The problem is the details like some code which depends on a variable *not* being present.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14500</commentid>
    <comment_count>2</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-06-30 17:28:34 +0000</bug_when>
    <thetext>Hi, Richard

Thanks for the details information!

I will collect some data when hob branch is basically ready for presenting how much time to generate the dependency tree data (Showing the data into the treeview is very fast, the bottleneck is to generate the dependency tree data on the server side).

The data generation time is not a problem for bitbake commandline at all. But now there&apos;re more and more requests on UI, especially web UI, . For UI (web UI), it would be unbearable if the data generation needs more than 10s...


Thanks&amp; Regards,
criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14547</commentid>
    <comment_count>3</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-07-06 00:40:17 +0000</bug_when>
    <thetext>1. At least, we need to look more into the target tree generating phase on the server side. Decide which data is Hob needed instead of simply reuse generateDepTreeData function. (The data is too huge, 4M more +)... At least I feel that some over-small granularity data (tdepends? is not used till now)... This is one of the biggest bottleneck now. Need over 15+ seconds.

2. For selecting, deselecting one package item, we might be able to set up local hashset (tree data) to store part of the tree data structure when user select/deselect a package, instead of traverse the big data mentioned above, record package-ref count, accurately describing the package dependency relationship (brought in column can&apos;t record multiple dependency problem) ...

Some initial thoughts after read the code in the poky master...

Thanks&amp; Regards,
criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14548</commentid>
    <comment_count>4</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-07-06 01:47:40 +0000</bug_when>
    <thetext>(In reply to comment #3)
&gt; 1. At least, we need to look more into the target tree generating phase on the
&gt; server side. Decide which data is Hob needed instead of simply reuse
&gt; generateDepTreeData function. (The data is too huge, 4M more +)... At least I
&gt; feel that some over-small granularity data (tdepends? is not used till now)...
&gt; This is one of the biggest bottleneck now. Need over 15+ seconds.
&gt; 


More update for this:

Lianhao and I did some very initial try (remove some obviously useless data, such as inter-task deps do_compile, etc....), the data prepare process could be reduced from 11s to 9s... There are so much data there, we are not familiar with details... But I guess if we can look into details, we can gain something from this trial!

criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14549</commentid>
    <comment_count>5</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-07-06 01:55:08 +0000</bug_when>
    <thetext>Even more exciting:

We remove something more in the recrdeptask/runqueue.py together with tdepends. Reduced from 11s-&gt;5S!!!

criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14551</commentid>
    <comment_count>6</comment_count>
    <who name="Lianhao Lu">lianhao.lu</who>
    <bug_when>2011-07-06 02:15:49 +0000</bug_when>
    <thetext>Wait! I don&apos;t think we can aggressively remove those from runqueue.prepare() before we&apos;re 100% sure about what is being removed. My guessing is that removing recrdep and rundeps might change those content which the hob is interested in. 

(In reply to comment #5)
&gt; Even more exciting:
&gt; 
&gt; We remove something more in the recrdeptask/runqueue.py together with tdepends.
&gt; Reduced from 11s-&gt;5S!!!
&gt; 
&gt; criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14553</commentid>
    <comment_count>7</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2011-07-06 04:13:51 +0000</bug_when>
    <thetext>I just took a look at this code. It appears hob is just interested in &apos;rdepends&apos;, &apos;depends&apos; and packages. Looking at the generation code in cooker.py it would appear at first glance you can build all that data from taskdata without needing a runqueue object at all?

Certainly you never appear to use tdepends or rrecs-pkg which the rq would provide.

What might work out as a problem is that the taskdata has multiple providers listed (e.g. for virtual/libc). Its the up to runqueue to resolve those into a list of the preferred dependencies. The lists in taskdata are ordered you can probably take the first item though, just as runqueue does.

The bottom line is that you probably need a version of these functions that just works off taskdata...</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14675</commentid>
    <comment_count>8</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-07-12 00:19:23 +0000</bug_when>
    <thetext>Hi, Richard,

Today I made a try:

1. in prepareTreeData, there&apos;re some code about taskdata.add_provider() and then taskdata is used for constructing RunQueue. ( I am not sure this is the one you mentioned for multiple provider problem)

2. During rqdata.prepare, actually only self.runq.fnid.append logic is needed. Most of other logic (weight calc, sstate, etc) are not useful for hob at all.

3. in generateDepTreeData, recursive/tdepends data will be removed.

After removing all those logic which is not used in HOB currently, seems 20s could be reduced to 8s. And the data will be decreased from 4M -&gt; 1M

Yet I am really too fresh for the whole runqueue related things. Can you help to confirm that after removing those code, the original logic will not be hurt?


If it&apos;s ok, I will go on making a patch for this.

Regards,
criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14679</commentid>
    <comment_count>9</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2011-07-12 01:34:16 +0000</bug_when>
    <thetext>I can confirm that hob specifically should not be affected by removing the calls to that code and the simplifications we&apos;re discussing.

The dependency explorer does need that code, as do other users of the runqueue though so it can&apos;t be removed, we just need to stop hob using runqueue.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>14680</commentid>
    <comment_count>10</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-07-12 01:38:00 +0000</bug_when>
    <thetext>&gt; The dependency explorer does need that code, as do other users of the runqueue
&gt; though so it can&apos;t be removed, we just need to stop hob using runqueue.
Yes. Other users definitely still need all ruqqueue logic. 
OK, so I will make a patch. Let Hob use a special version of getting those data and Leave other users&apos; logic intact.

Thanks a lot!
criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>15166</commentid>
    <comment_count>11</comment_count>
    <who name="liping Ke">liping.ke</who>
    <bug_when>2011-08-02 19:34:07 +0000</bug_when>
    <thetext>Solved parts of the slow loading issue after remove not-used data-structure such as task dependencies. But still, we need to consider more about how to accelerate paring and loading process. It would be very important for the further version of image creator, especially web-based?

criping</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>16783</commentid>
    <comment_count>12</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2011-10-07 06:28:57 +0000</bug_when>
    <thetext>The specific issue this bug was opened for is resolved. Whilst I understand we&apos;d like to find further speed improvements I&apos;d suggest specific bugs are opened for areas when we have good ideas to speed them up. I&apos;m therefore marking this one as fixed.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>