Interestingly, after some experimentation with retroarch and KMS i've reached a conclusion about this:
This could be a worthwhile project, but there the distros aren't there yet.
Namely kmscon and its older project systemd-consoled were killed around the time of the anti-systemd hysteria back in 2014. This is important for GUIs on KMS because of two reasons:
Since the linux kernel drivers are supposed to be simple to maintain, distros reacted by putting a lot of hardware quirks in userspace libraries for X11 and Wayland DEs. Namely the current holder of this 'mess' is libinput, that at least on weston/wayland attaches libinput to the compositor between it and udev. When you use KMS directly you get the hardware sending straight to edev, and naturally everything is screwed up (most notably touchpads).
Another complicated problem is GPU permissions. At least on retroarch it's very easy to force a crash by losing and (failing) to re-acquire the opengl context due to lack of 'returning' permissions.
I think for sure the first problem, and maybe the second problem could be dealt with if the tty console was a wayland compositor (like kmscon or systemd-consoled). But again these projects have been dead for years as far as i can tell. If you know of a current alternative please let me know.