I changed XSERVER in meta-yocto/conf/machine/atom-pc.conf, then did a bitbake -c clean task-core and then bitbake task-core, and what I got was a build from sstate without the change to XSERVER being reflected. Using cleansstate instead of clean worked. Something must be wrong with the variable dependencies in order for the sstate checksum not to have changed.
Fixed in master http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c7869b4d4cade8e87e9aab9e27c07bbc6fed39fe