In my local testing with many small layers included, I'm seeing a spike to 5gb during up-front recipe parsing when doing a bitbake -e without a tmpdir, but it's only 2.5gb if I do a bitbake -p, and then bitbake -e after that. So it seems that the enabling of variable tracking is impacting the memory usage of the up-front recipe parsing, which should not be the case, as that parsing is only done to populate the recipe cache data / cooker data, not for the actual data emit for -e. This memory usage is temporary, as after the recipes are parsed, the usage drops back down, but spiking this high can cause OOM conditions on certain systems, so is still not ideal.
Currently verifying that this is a real issue, as memory info can be misleading, particularly anything rss based, since rss will count shared memory for each process. Testing with massif, since it gathers heap allocation info.
I re-tested using numbers from valgrind's massif rather than memory_profiler, and it does indicate this seems to be a real issue. My hacked up local tree seems to have fixed it, but not cleanly, so am digging further.
Revisit later, there are improvements to be made, but I'm uncertain as to whether this *particular* issue is valid or not.