Since tasks are referred to almost exclusively via RDEPENDS, and bitbake will build an entire task recipe when only one of its task packages are actually needed, building a console-only image that uses task-core-apps-console (or less directly, has apps-console in its IMAGE_FEATURES) will cause a build of a whole list of X11 applications that aren't needed. Splitting the task-core recipe into X11 and console portions would prevent this issue.
Fix now merged into OE-core with commit 8c310ee0a94a7e15d815a2719272ffc704e857b0