So the best way right now is to use the D-Bus display interface of QEMU.
I made a plugin for that which will submit as MR soon™. Many of the discussed here issues were fixed by just copying DMA buffers allocated by QEMU into ones allocated via GBM. This even fixed some of the modifier issues as GBM can just allocate buffers with specific modifiers.