Changing the default debug packaging behaviour to "debug-with-srcpkg" can give the best of both worlds; both the debug symbols and/or source can be included independently. It needs to be ensured that -src packages can be installed with a DISTRO_FEATURE that maps to a COMPLEMENTARY_GLOB.
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ba3aa531129186ce531c46acb68f329f6c8f376e