If you build Poky (with an empty tmp/ directory) and use pstaged packages, the kernel modules will not be rebuilt for any 'image' target. Building from scratch (no pstage packages) would of course build the modules.
Obsoleted by the new sstate class and relative changes. If problems still exist please report then against the new code.