The HOME var is set to '/' for anyting running under X. It appears that the /etc/X11/Xinit script that is supposed to set it is never executed; before the server is started the /etc/X11/Xsession is run, but that only set HOME_VOLATILE, not HOME. I am not sure whether the best solution is to add HOME to Xsession, or whether the Xinit script should be chained in.
I have added export HOME to the Xsession script at least for now.
Recent changes to Poky mean this is no longer a problem. The Xinit script has been removed and the Xserver script is run from su which makes sure the environment is sane.