Bug 11303

Summary: Variable tracking spikes memory usage during up-front recipe parsing
Product: [Build System, Metadata & Runtime] BitBake Reporter: Christopher Larson <kergoth>
Component: bitbakeAssignee: Christopher Larson <kergoth>
Status: RESOLVED INVALID QA Contact:
Severity: normal    
Priority: Medium+ CC: poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: 2.5   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Christopher Larson 2017-04-06 16:29:42 UTC 
In my local testing with many small layers included, I'm seeing a spike to 5gb during up-front recipe parsing when doing a bitbake -e without a tmpdir, but it's only 2.5gb if I do a bitbake -p, and then bitbake -e after that. So it seems that the enabling of variable tracking is impacting the memory usage of the up-front recipe parsing, which should not be the case, as that parsing is only done to populate the recipe cache data / cooker data, not for the actual data emit for -e. This memory usage is temporary, as after the recipes are parsed, the usage drops back down, but spiking this high can cause OOM conditions on certain systems, so is still not ideal.
Comment 1 Christopher Larson 2017-04-06 19:40:23 UTC 
Currently verifying that this is a real issue, as memory info can be misleading, particularly anything rss based, since rss will count shared memory for each process. Testing with massif, since it gathers heap allocation info.
Comment 2 Christopher Larson 2017-04-07 17:44:20 UTC 
I re-tested using numbers from valgrind's massif rather than memory_profiler, and it does indicate this seems to be a real issue. My hacked up local tree seems to have fixed it, but not cleanly, so am digging further.
Comment 3 Christopher Larson 2017-07-28 20:47:29 UTC 
Revisit later, there are improvements to be made, but I'm uncertain as to whether this *particular* issue is valid or not.