11303 2017-04-06 16:29:42 +0000 Variable tracking spikes memory usage during up-front recipe parsing 2017-07-28 20:47:29 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED INVALID Medium+ normal 2.5 1 kergoth kergoth poky.bs.watcher poky.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 72142 0 kergoth 2017-04-06 16:29:42 +0000 In my local testing with many small layers included, I'm seeing a spike to 5gb during up-front recipe parsing when doing a bitbake -e without a tmpdir, but it's only 2.5gb if I do a bitbake -p, and then bitbake -e after that. So it seems that the enabling of variable tracking is impacting the memory usage of the up-front recipe parsing, which should not be the case, as that parsing is only done to populate the recipe cache data / cooker data, not for the actual data emit for -e. This memory usage is temporary, as after the recipes are parsed, the usage drops back down, but spiking this high can cause OOM conditions on certain systems, so is still not ideal. 72153 1 kergoth 2017-04-06 19:40:23 +0000 Currently verifying that this is a real issue, as memory info can be misleading, particularly anything rss based, since rss will count shared memory for each process. Testing with massif, since it gathers heap allocation info. 72186 2 kergoth 2017-04-07 17:44:20 +0000 I re-tested using numbers from valgrind's massif rather than memory_profiler, and it does indicate this seems to be a real issue. My hacked up local tree seems to have fixed it, but not cleanly, so am digging further. 75414 3 kergoth 2017-07-28 20:47:29 +0000 Revisit later, there are improvements to be made, but I'm uncertain as to whether this *particular* issue is valid or not.