Should we build the non default kernel as well as part of some configration on autobuilder as well? Perhaps keep separate defaults for altcfg so it's automatically built? Example, for current master - 5.4 for poky and 5.2 for altcfg. We can change defaults whenever things get removed/added ... This would ensure that the other kernel is also tested and not broken.
We're about to drop the 5.2 kernel from master so this becomes a non-issue. In future I would take a patch to poky-altcfg if we end up in this situation again.