It's entirely possible to end up with a valid image that doesn't start X because the server and drivers have different ABIs. The package system can help avoid this by the xserver-xorg package providing names such as xorg-input-abi-16, and then the driver packages depending on those. This will have to be implemented by the xorg-xserver package at build time writing the abi versions to staging, where they can be picked up driver builds and used in the dependencies. For example, in Debian xserver-xorg generates and packages /usr/share/xserver-xorg/xinputdep which contains "xorg-input-abi-16, xserver-xorg-core (>= 2:1.12.3.901)". Input drivers then add those dependencies to the package when they build.
Fixed in poky 662189deb638a538f814621f6ef62871d6e4084b and 662189deb638a538f814621f6ef62871d6e4084b.
Verified on master branch. Commit ID: 9ffee9add2d0380f4025e7a678a166f25a7d659e