What's the best way to push video from QEMU to GStreamer?

Guest os could be either Linux or Windows, host is Linux.

I need to capture video output from the QEMU (that is, from virtual display device) and push it into a GStreamer pipeline.

I’ve tried using libspice-client-glib, but video was very choppy, latency was pretty bad and quality wasn’t good as well. It’s probably due to multiple re-encodings though (qemu → spice-server [gstreamer|mjpeg] → socket → spice-client-glib [gstreamer|mjpeg] → (raw video) my app → gstreamer), but there doesn’t seem a better way to do it given the library API.

Other option seem to be rfbsrc but it doesn’t seem to be actively improved anymore. I’ve also heard some bad things about VNC, so not sure whether it’ll be any better.

Are there any other (potentially better) approaches?

Spice seems the most popular remote access to QEmu/KVM these days, unfortunatly we don’t have an easy source like rfbsrc. Perhaps you can learn how to implement a spice / gstreamer client from:

Spice is similar to RFB in the sense it can do JPEG but also H.264 encoding, that depends on the server (QEmu).

Is actually built from that repo. Funnily enough, it already makes use of gstreamer internally (appsrc+decodebin). I wonder if simply hacking it to instead push into my pipeline would make any difference in terms of snappiness.

Is there an easy way to create a plugin which basically is appsrc + something that pushes data into it? Creating a “proper” plugin seems to be too much work.

There is no issue with doing exactly the same but in your own program.

1 Like

I tried QEMU D-Bus display instead which appears to be a better option going forward.

However, I cannot get GStreamer to work with DMA Buffers, as per the issue.

Let’s continue that discussion here.

We on purpose don’t support implicit modifiers (0xffffffffffffff). Explicitly specify your modifiers.

How can I get that modifier? 0xffffffffffffff is what I got from QEMU.

For the try with “linear”, your GL stack reports not supporting it, so that’s also not a bug

I would expect to actually see some explicit references to that in the logs. Can you please point me to where it actually says that? I added an additional debug message to gst_glimage_sink_get_caps and looks like it only supports these formats:
video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], texture-target=(string){ 2D, external-oes };
video/x-raw(memory:GLMemory, meta:GstVideoOverlayComposition), format=(string)RGBA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], texture-target=(string){ 2D, external-oes }

That’s unfortunate and really strange. How is it even supposed to work then? This even contradicts with what docs and gst-inspect-1.0 says about glimagesink. Is there some specific hardware requirements for glimagesink to support DMA buffers? It’s not running in a virtualized environment btw, but rather as a host app.

Do you have any suggestions on how to make it work? In the future I would need to encode the video instead just displaying it, so it seems that I need some element that can handle DMA Buffers and convert them to whatever the next element would need (i.e. to system memory by mmap+read+unmap or something). Is there anything like that?

As the source is QEMU, GStreamer is the wrong channel to get answers to most of your questions.

The debug category GST_DEBUG="GL context:4" includes a dump of your GL stack capabilities. Note it is not because a format is supported that the underlying dmabuf will be compatible with your hardware.

Assuming this is the right dbus interface, for Unix host you have a SHM and a pixman format. This indicates that you don’t have a dmabuf. You can wrap this FD using GstFDAllocator, and negotiate as normal system memory.

https://www.qemu.org/docs/master/interop/dbus-display.html

I missed the dmabuf scannout event. But if you get invalid modifier (0xffffffffffffff) from that, there is nothing you can do.

There is a design flaw noted by Mathias Classens that will also makes this interface quite unreliable in GStreamer (and GTK4).

The interface assumes immediate rendering. To do that correctly you’ll need to block in that dbus call until the memory is released (use a weak ref). That can be difficult if by accident you have anything that needs more then one buffer to continue. You could easily hit some dbus timeout with that.

I consider interaction with QEMU via D-Bus a solved problem now, it’s the GStreamer that I have troubles getting the picture data into.

I found a (relatively simple) way to trigger QEMU to send Scanout/Update with pure bytes (aka turn off 3D acceleration), but it leads to quite bad experience even for opening the start menu in a VM.
It also seems that appsrc is not good for this task as it requires a constant data flow: for example, application cannot exit unless another buffer is pushed into it after.

Is there some kind of source that will be better suited for this task? Like maybe something with double-buffering: it’ll send the same picture unless new one is pushed outside.

Assuming this is the right dbus interface, for Unix host you have a SHM and a pixman format. This indicates that you don’t have a dmabuf. You can wrap this FD using GstFDAllocator, and negotiate as normal system memory.

Do you think shared memory scanouts would work any better?

Unfortunately, this isn’t true in general as GstFdAllocator does not handle offset and hardcodes it to 0 in gst_fd_mem_map. Can this be fixed?

We do support offsets of course. Even in two ways.

A. you can set an offset directly on the GstMemory object:

mem = gst_fd_allocator_alloc (allocator, fd, size, GST_FD_MEMORY_FLAG_KEEP_MAPPED  | GST_FD_MEMORY_FLAG_DONT_CLOSE);
gst_memory_resize (mem, offset, size - offset);

B. you can leave the full size and use GstVideoMeta. Handy if you have 1 FD for N planes:

mem = gst_fd_allocator_alloc (allocator, fd , size);
gst_buffer_append_memory (buffer, mem);
gst_buffer_add_video_meta_full (buffer, 0, format, width, height, n_planes,offset, stride);
1 Like

I don’t think it’s equivalent to properly passing the offset to mmap, is it?
Currently the offset is hardcoded to 0:

I guess I would need to actually do something like this:

mem = gst_fd_allocator_alloc (allocator, fd, size + offset, GST_FD_MEMORY_FLAG_KEEP_MAPPED  | GST_FD_MEMORY_FLAG_DONT_CLOSE);
gst_memory_resize (mem, offset, size);

But I’m still not sure whether it’s equivalent.

Do you know how I might debug this further? Many frames updates are being skipped here for no apparent reason (CPU usage isn’t even that high), so is there a better way to push data into GStreamer compared to what I’m doing currently? I.e. just pushing the part of the picture that was updated since last full scanout or something like that.

I’m getting weird artifacts like the following:
image
Even dragging windows takes a lot of time to properly redraw on the screen.

DMABuf or MemFD works fine with 0 as offset. The only usage of mmap offset that is relevant is when using V4L2 mmap, since it uses the offset as a cookie.

QEmu should provide a test application for this feature, running the test application should tell you what to expect in your GStreamer integration.

There is libmks but it’s not very useful here as it’s too much gui-related (cairo, gdk) and basically immediately sends data there. There just is an API to redraw parts of the screen and that’s it.

I am yet to try memfd though, but it seems that I have to support all of the scanout methods.

BTW, DMA buffers are exported via eglExportDMABUFImageQueryMESA/eglExportDMABUFImageMESA in QEMU. And, apparently, the former actually returns the implicit modifier, but libmks can handle it.

Actually, the only issue with DMA buffers I have currently is that, apparently, GStreamer just can’t use them on my machine with obscure “negotiation failed” error messages. If I set caps to memory:DMABuf then it fails to negotiate, if I set it to “normal” then mmaping fd fails (which is, I guess, expected for DMA buffers).

Okay, I think there is a bug in GStreamer regarding DMA buffers or maybe something is not implemented properly.

I was able to import the fd into an OpenGL texture (with eglCreateImageKHR/glEGLImageTargetTexture2DOES) and then get pixels using glGetTexImage. Performance is obviously bad, but at least there is a picture on the screen.

So my hardware supports DMA buffers, it’s just something with GStreamer that prevents it from using them.

BTW, GST_DEBUG="GL context:4" printed nothing, so I’m not sure what to do next.

The debug categories are case and space sensitive. Here’s a well spelled example and what you get on supported GPU drivers.

GST_DEBUG=glcontext:4 gst-launch-1.0 videotestsrc num-buffers=1 ! glimagesink

0:00:00.089317570 10552 0x7f0b14000b90 INFO               glcontext gstglcontext_egl.c:1583:_print_all_dma_formats:<glcontextegl0> 
============= All DMA Formats With Modifiers =============
| Gst Format   | DRM Format              | External Flag |
|========================================================|
| (null)       | R8                      |               |
|              | R8  :0x0100000000000001 |               |
|              | R8  :0x0100000000000002 |               |
|              | R8  :0x0100000000000006 |               |
|              | R8  :0x0100000000000008 |               |
|--------------------------------------------------------|
| (null)       | R16                     |               |
|              | R16 :0x0100000000000001 |               |
|              | R16 :0x0100000000000002 |               |
|              | R16 :0x0100000000000006 |               |
|              | R16 :0x0100000000000008 |               |
|--------------------------------------------------------|
| P010_10LE    | P010                    | external only |
|              | P010:0x0100000000000001 | external only |
|              | P010:0x0100000000000002 | external only |
|              | P010:0x0100000000000007 | external only |
|--------------------------------------------------------|
| Y210         | Y210                    | external only |
|              | Y210:0x0100000000000001 | external only |
|              | Y210:0x0100000000000002 | external only |
|--------------------------------------------------------|
| Y410         | Y410                    | external only |
|              | Y410:0x0100000000000001 | external only |
|              | Y410:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | P030                    | external only |
|              | P030:0x0100000000000001 | external only |
|              | P030:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | AB30                    |               |
|              | AB30:0x0100000000000001 |               |
|              | AB30:0x0100000000000002 |               |
|              | AB30:0x0100000000000006 |               |
|              | AB30:0x0100000000000008 |               |
|--------------------------------------------------------|
| BGR10A2_LE   | AR30                    |               |
|              | AR30:0x0100000000000001 |               |
|              | AR30:0x0100000000000002 |               |
|              | AR30:0x0100000000000006 |               |
|              | AR30:0x0100000000000008 |               |
|--------------------------------------------------------|
| (null)       | XR30                    |               |
|              | XR30:0x0100000000000001 |               |
|              | XR30:0x0100000000000002 |               |
|              | XR30:0x0100000000000006 |               |
|              | XR30:0x0100000000000008 |               |
|--------------------------------------------------------|
| Y41B         | YU11                    | external only |
|              | YU11:0x0100000000000001 | external only |
|              | YU11:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | YV11                    | external only |
|              | YV11:0x0100000000000001 | external only |
|              | YV11:0x0100000000000002 | external only |
|--------------------------------------------------------|
| NV21         | NV21                    | external only |
|              | NV21:0x0100000000000001 | external only |
|              | NV21:0x0100000000000002 | external only |
|--------------------------------------------------------|
| P012_LE      | P012                    | external only |
|              | P012:0x0100000000000001 | external only |
|              | P012:0x0100000000000002 | external only |
|              | P012:0x0100000000000007 | external only |
|--------------------------------------------------------|
| Y212_LE      | Y212                    | external only |
|              | Y212:0x0100000000000001 | external only |
|              | Y212:0x0100000000000002 | external only |
|--------------------------------------------------------|
| Y412_LE      | Y412                    | external only |
|              | Y412:0x0100000000000001 | external only |
|              | Y412:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | AR12                    |               |
|              | AR12:0x0100000000000001 |               |
|              | AR12:0x0100000000000002 |               |
|              | AR12:0x0100000000000006 |               |
|              | AR12:0x0100000000000008 |               |
|--------------------------------------------------------|
| I420         | YU12                    | external only |
|              | YU12:0x0100000000000001 | external only |
|              | YU12:0x0100000000000002 | external only |
|--------------------------------------------------------|
| NV12         | NV12                    | external only |
|              | NV12:0x0100000000000001 | external only |
|              | NV12:0x0100000000000002 | external only |
|              | NV12:0x0100000000000007 | external only |
|--------------------------------------------------------|
| YV12         | YV12                    | external only |
|              | YV12:0x0100000000000001 | external only |
|              | YV12:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | GR32                    |               |
|              | GR32:0x0100000000000001 |               |
|              | GR32:0x0100000000000002 |               |
|              | GR32:0x0100000000000006 |               |
|              | GR32:0x0100000000000008 |               |
|--------------------------------------------------------|
| RGBA         | AB24                    |               |
|              | AB24:0x0100000000000001 |               |
|              | AB24:0x0100000000000002 |               |
|              | AB24:0x0100000000000006 |               |
|              | AB24:0x0100000000000007 | external only |
|              | AB24:0x0100000000000008 |               |
|--------------------------------------------------------|
| RGBx         | XB24                    |               |
|              | XB24:0x0100000000000001 |               |
|              | XB24:0x0100000000000002 |               |
|              | XB24:0x0100000000000006 |               |
|              | XB24:0x0100000000000007 | external only |
|              | XB24:0x0100000000000008 |               |
|--------------------------------------------------------|
| BGRA         | AR24                    |               |
|              | AR24:0x0100000000000001 |               |
|              | AR24:0x0100000000000002 |               |
|              | AR24:0x0100000000000006 |               |
|              | AR24:0x0100000000000007 | external only |
|              | AR24:0x0100000000000008 |               |
|--------------------------------------------------------|
| BGRx         | XR24                    |               |
|              | XR24:0x0100000000000001 |               |
|              | XR24:0x0100000000000002 |               |
|              | XR24:0x0100000000000006 |               |
|              | XR24:0x0100000000000007 | external only |
|              | XR24:0x0100000000000008 |               |
|--------------------------------------------------------|
| Y444         | YU24                    | external only |
|              | YU24:0x0100000000000001 | external only |
|              | YU24:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | YV24                    | external only |
|              | YV24:0x0100000000000001 | external only |
|              | YV24:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | AR15                    |               |
|              | AR15:0x0100000000000001 |               |
|              | AR15:0x0100000000000002 |               |
|              | AR15:0x0100000000000006 |               |
|              | AR15:0x0100000000000008 |               |
|--------------------------------------------------------|
| (null)       | P016                    | external only |
|              | P016:0x0100000000000001 | external only |
|              | P016:0x0100000000000002 | external only |
|              | P016:0x0100000000000007 | external only |
|--------------------------------------------------------|
| (null)       | Y216                    | external only |
|              | Y216:0x0100000000000001 | external only |
|              | Y216:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | Y416                    | external only |
|              | Y416:0x0100000000000001 | external only |
|              | Y416:0x0100000000000002 | external only |
|--------------------------------------------------------|
| RGB16        | RG16                    |               |
|              | RG16:0x0100000000000001 |               |
|              | RG16:0x0100000000000002 |               |
|              | RG16:0x0100000000000006 |               |
|              | RG16:0x0100000000000008 |               |
|--------------------------------------------------------|
| Y42B         | YU16                    | external only |
|              | YU16:0x0100000000000001 | external only |
|              | YU16:0x0100000000000002 | external only |
|--------------------------------------------------------|
| NV16         | NV16                    | external only |
|              | NV16:0x0100000000000001 | external only |
|              | NV16:0x0100000000000002 | external only |
|              | NV16:0x0100000000000007 | external only |
|--------------------------------------------------------|
| (null)       | YV16                    | external only |
|              | YV16:0x0100000000000001 | external only |
|              | YV16:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | AB48                    |               |
|              | AB48:0x0100000000000001 |               |
|              | AB48:0x0100000000000002 |               |
|              | AB48:0x0100000000000006 |               |
|              | AB48:0x0100000000000008 |               |
|--------------------------------------------------------|
| (null)       | XB48                    |               |
|              | XB48:0x0100000000000001 |               |
|              | XB48:0x0100000000000002 |               |
|              | XB48:0x0100000000000006 |               |
|              | XB48:0x0100000000000008 |               |
|--------------------------------------------------------|
| (null)       | GR88                    |               |
|              | GR88:0x0100000000000001 |               |
|              | GR88:0x0100000000000002 |               |
|              | GR88:0x0100000000000006 |               |
|              | GR88:0x0100000000000008 |               |
|--------------------------------------------------------|
| YVU9         | YVU9                    | external only |
|              | YVU9:0x0100000000000001 | external only |
|              | YVU9:0x0100000000000002 | external only |
|--------------------------------------------------------|
| YUV9         | YUV9                    | external only |
|              | YUV9:0x0100000000000001 | external only |
|              | YUV9:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | AB4H                    |               |
|              | AB4H:0x0100000000000001 |               |
|              | AB4H:0x0100000000000002 |               |
|              | AB4H:0x0100000000000006 |               |
|              | AB4H:0x0100000000000008 |               |
|--------------------------------------------------------|
| (null)       | XB4H                    |               |
|              | XB4H:0x0100000000000001 |               |
|              | XB4H:0x0100000000000002 |               |
|              | XB4H:0x0100000000000006 |               |
|              | XB4H:0x0100000000000008 |               |
|--------------------------------------------------------|
| YVYU         | YVYU                    | external only |
|              | YVYU:0x0100000000000001 | external only |
|              | YVYU:0x0100000000000002 | external only |
|--------------------------------------------------------|
| VUYA         | AYUV                    | external only |
|              | AYUV:0x0100000000000001 | external only |
|              | AYUV:0x0100000000000002 | external only |
|--------------------------------------------------------|
| (null)       | XYUV                    | external only |
|              | XYUV:0x0100000000000001 | external only |
|              | XYUV:0x0100000000000002 | external only |
|--------------------------------------------------------|
| YUY2         | YUYV                    | external only |
|              | YUYV:0x0100000000000001 | external only |
|              | YUYV:0x0100000000000002 | external only |
|              | YUYV:0x0100000000000006 | external only |
|              | YUYV:0x0100000000000007 | external only |
|              | YUYV:0x0100000000000008 | external only |
|--------------------------------------------------------|
| VYUY         | VYUY                    | external only |
|              | VYUY:0x0100000000000001 | external only |
|              | VYUY:0x0100000000000002 | external only |
|--------------------------------------------------------|
| UYVY         | UYVY                    | external only |
|              | UYVY:0x0100000000000001 | external only |
|              | UYVY:0x0100000000000002 | external only |
|              | UYVY:0x0100000000000006 | external only |
|              | UYVY:0x0100000000000007 | external only |
|              | UYVY:0x0100000000000008 | external only |
==========================================================

I’m not seeing any of that though:

$ GST_DEBUG=glcontext:4 gst-launch-1.0 videotestsrc num-buffers=1 ! glimagesink
Setting pipeline to PAUSED ...
0:00:00.012290913 119240 0x5602c89ab6a0 INFO               glcontext gstglcontext.c:348:gst_gl_context_new: creating a context for display <gldisplayx11-0>, user choice:(null)
0:00:00.012443441 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext_glx.c:776:gst_gl_context_glx_choose_format: GLX Version: 1.4
0:00:00.093385232 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1319:gst_gl_context_create_thread:<glcontextglx0> Attempting to create opengl context. user chosen api(s) (any), compiled api support (opengl opengl3 gles2) display api (opengl opengl3 gles2)
0:00:00.094760313 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1333:gst_gl_context_create_thread:<glcontextglx0> created context
0:00:00.097011120 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1349:gst_gl_context_create_thread:<glcontextglx0> available GL APIs: opengl3
0:00:00.097236235 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1112:_create_context_info:<glcontextglx0> GL_VERSION: 4.6 (Core Profile) Mesa 24.3.1-arch1.3
0:00:00.097244431 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1114:_create_context_info:<glcontextglx0> GL_SHADING_LANGUAGE_VERSION: 4.60
0:00:00.097250021 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1117:_create_context_info:<glcontextglx0> GL_VENDOR: AMD
0:00:00.097254370 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1119:_create_context_info:<glcontextglx0> GL_RENDERER: AMD Radeon Graphics (radeonsi, raphael_mendocino, LLVM 18.1.8, DRM 3.59, 6.12.5-arch1-1-mtu-upstream)
0:00:00.097426485 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:2076:_gst_gl_context_debug_is_enabled:<glcontextglx0> Disabling GL context debugging (gldebug category debug level < warning)
0:00:00.097431425 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1172:_unlock_create_thread:<glcontextglx0> gl thread running
0:00:00.097445682 119240 0x5602c89ab6a0 INFO               glcontext gstglcontext.c:1079:gst_gl_context_create:<glcontextglx0> gl thread created
Pipeline is PREROLLING ...
Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
0:00:00.098537336 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:2076:_gst_gl_context_debug_is_enabled:<glcontextglx0> Disabling GL context debugging (gldebug category debug level < warning)
0:00:00.098632416 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:2076:_gst_gl_context_debug_is_enabled:<glcontextglx0> Disabling GL context debugging (gldebug category debug level < warning)
0:00:00.098763273 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:2076:_gst_gl_context_debug_is_enabled:<glcontextglx0> Disabling GL context debugging (gldebug category debug level < warning)
0:00:00.098800213 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:2076:_gst_gl_context_debug_is_enabled:<glcontextglx0> Disabling GL context debugging (gldebug category debug level < warning)
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.033442484
Setting pipeline to NULL ...
0:00:00.144641324 119240 0x5602c89ab6a0 INFO               glcontext gstglcontext.c:689:gst_gl_context_finalize:<glcontextglx0> send quit gl window loop
0:00:00.144647486 119240 0x5602c89ab6a0 INFO               glcontext gstglcontext.c:692:gst_gl_context_finalize:<glcontextglx0> joining gl thread
0:00:00.144653627 119240 0x5602c8ca8b40 INFO               glcontext gstglcontext.c:1403:gst_gl_context_create_thread:<glcontextglx0> loop exited
0:00:00.149316887 119240 0x5602c89ab6a0 INFO               glcontext gstglcontext.c:695:gst_gl_context_finalize:<glcontextglx0> gl thread joined
Freeing pipeline ...