| Summary: | Reduce reparsing impact and cache dependencies calc results in bitbake | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | liping Ke <liping.ke> |
| Component: | bitbake | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | Low | CC: | jessica.zhang, josh, ke.yu, lianhao.lu, poky.bs.watcher, poky.watcher, sgw, shane.wang |
| Version: | unspecified | ||
| Target Milestone: | 1.2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
|
Description
liping Ke
2011-06-13 19:25:52 UTC
I would like to understand the information that the image creator UI is requiring which is taking so long to obtain. Assuming the cache is hot, things should be very fast and for example runqueue can quickly compute dependency information about builds. Is the problem obtaining the data or it is turning it into a form the UI can use in its treeview? I've had a lot of experience over the years of tuning bitbake internals to be fast so perhaps if you can illustrate where the slowdowns are and how to demonstrate them, I can take a look and see if I have any specific ideas. Regarding recipe parsing when variables like BB_NUMBER_THREADS change, the way the bitbake data model is structured, its very very hard to predict what effect changing a variable will have on the metadata. There are some clear cases we can make predictions but its not always clear. Now we have the checksum info with the dependency information about variables we would have a place to start considering things like this but its still a major amount of work. The problem is the details like some code which depends on a variable *not* being present. Hi, Richard Thanks for the details information! I will collect some data when hob branch is basically ready for presenting how much time to generate the dependency tree data (Showing the data into the treeview is very fast, the bottleneck is to generate the dependency tree data on the server side). The data generation time is not a problem for bitbake commandline at all. But now there're more and more requests on UI, especially web UI, . For UI (web UI), it would be unbearable if the data generation needs more than 10s... Thanks& Regards, criping 1. At least, we need to look more into the target tree generating phase on the server side. Decide which data is Hob needed instead of simply reuse generateDepTreeData function. (The data is too huge, 4M more +)... At least I feel that some over-small granularity data (tdepends? is not used till now)... This is one of the biggest bottleneck now. Need over 15+ seconds. 2. For selecting, deselecting one package item, we might be able to set up local hashset (tree data) to store part of the tree data structure when user select/deselect a package, instead of traverse the big data mentioned above, record package-ref count, accurately describing the package dependency relationship (brought in column can't record multiple dependency problem) ... Some initial thoughts after read the code in the poky master... Thanks& Regards, criping (In reply to comment #3) > 1. At least, we need to look more into the target tree generating phase on the > server side. Decide which data is Hob needed instead of simply reuse > generateDepTreeData function. (The data is too huge, 4M more +)... At least I > feel that some over-small granularity data (tdepends? is not used till now)... > This is one of the biggest bottleneck now. Need over 15+ seconds. > More update for this: Lianhao and I did some very initial try (remove some obviously useless data, such as inter-task deps do_compile, etc....), the data prepare process could be reduced from 11s to 9s... There are so much data there, we are not familiar with details... But I guess if we can look into details, we can gain something from this trial! criping Even more exciting: We remove something more in the recrdeptask/runqueue.py together with tdepends. Reduced from 11s->5S!!! criping Wait! I don't think we can aggressively remove those from runqueue.prepare() before we're 100% sure about what is being removed. My guessing is that removing recrdep and rundeps might change those content which the hob is interested in. (In reply to comment #5) > Even more exciting: > > We remove something more in the recrdeptask/runqueue.py together with tdepends. > Reduced from 11s->5S!!! > > criping I just took a look at this code. It appears hob is just interested in 'rdepends', 'depends' and packages. Looking at the generation code in cooker.py it would appear at first glance you can build all that data from taskdata without needing a runqueue object at all? Certainly you never appear to use tdepends or rrecs-pkg which the rq would provide. What might work out as a problem is that the taskdata has multiple providers listed (e.g. for virtual/libc). Its the up to runqueue to resolve those into a list of the preferred dependencies. The lists in taskdata are ordered you can probably take the first item though, just as runqueue does. The bottom line is that you probably need a version of these functions that just works off taskdata... Hi, Richard, Today I made a try: 1. in prepareTreeData, there're some code about taskdata.add_provider() and then taskdata is used for constructing RunQueue. ( I am not sure this is the one you mentioned for multiple provider problem) 2. During rqdata.prepare, actually only self.runq.fnid.append logic is needed. Most of other logic (weight calc, sstate, etc) are not useful for hob at all. 3. in generateDepTreeData, recursive/tdepends data will be removed. After removing all those logic which is not used in HOB currently, seems 20s could be reduced to 8s. And the data will be decreased from 4M -> 1M Yet I am really too fresh for the whole runqueue related things. Can you help to confirm that after removing those code, the original logic will not be hurt? If it's ok, I will go on making a patch for this. Regards, criping I can confirm that hob specifically should not be affected by removing the calls to that code and the simplifications we're discussing. The dependency explorer does need that code, as do other users of the runqueue though so it can't be removed, we just need to stop hob using runqueue. > The dependency explorer does need that code, as do other users of the runqueue
> though so it can't be removed, we just need to stop hob using runqueue.
Yes. Other users definitely still need all ruqqueue logic.
OK, so I will make a patch. Let Hob use a special version of getting those data and Leave other users' logic intact.
Thanks a lot!
criping
Solved parts of the slow loading issue after remove not-used data-structure such as task dependencies. But still, we need to consider more about how to accelerate paring and loading process. It would be very important for the further version of image creator, especially web-based? criping The specific issue this bug was opened for is resolved. Whilst I understand we'd like to find further speed improvements I'd suggest specific bugs are opened for areas when we have good ideas to speed them up. I'm therefore marking this one as fixed. |