The vim.inc file checks to see if the DISTRO_FEATURES contains x11 and if so enables x11 and gtkgui as part of PACKAGECONFIG. A distribution can feature x11 but be building a command line only image such as core-image-full-cmdline. When including vim in such an image it then pulls in xt and gtk3. This probably should be checking IMAGE_FEATURES.
IMAGE_FEATURES won't work, as it doesn't get passed to all of the recipes that go into the image. If you want to build both a non-graphical vim and a graphical vim then you'll need to have two recipes, or alter the recipe to build a basic vim and a featureful vim.
Thanks. I just needed the non-graphical vim. I was caught by surprise that the final image included vim and gtk+3 libraries when there was no X server. If IMAGE_FEATURES can't be used then I wonder what is the intended use of IMAGE_FEATURES.
Image features are image-wide controls and don't impact individual recipe building: https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#ref-features-image