What's the best way to push video from QEMU to GStreamer?

Application Development
41

Another thing is that only the following pipeline works:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glviewconvert input-flags-override=left-flipped input-mode-override=left ! glimagesink"

These do not work:
With this one, renderer window isn’t opened at all:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glvideoflip video-direction=vert ! glimagesink

What’s even worse is that this one only displays a mostly black picture:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! gldownload ! video/x-raw ! videoflip video-direction=vert ! glimagesink

image
image1280×829 6.7 KB

This is the most concerning issue. Is there a way to make it work? Is there maybe another way to get picture back to the system memory for further processing?

42

Does its work if the first element after glupload is glcolorconvert? If not, it’s important that normally implicit modifiers works by attaching some metadata to the kernel object, but in current case, mesa simply guess the format, so a slight change in your GL pipeline can break that guess.

43

glcolorconvert doesn’t change much for neither of the pipelines .
This shows a window with black picture (previously it wasn’t even opening the window):

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glcolorconvert ! glvideoflip video-direction=vert ! glimagesink

This one shows the same window, but with a few colorful pixels on the bottom:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glcolorconvert ! gldownload ! video/x-raw ! videoflip video-direction=vert ! glimagesink

And this one still sometimes only shows a black picture:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glcolorconvert ! glviewconvert input-flags-override=left-flipped input-mode-override=left ! glimagesink
44

I got testing results from an Intel platform where explicit modifiers work:

  1. glvideoflip works with glcolorconvert, but doesn’t even show a window without it (that is, the same behavior as on AMD)
  2. Pure black picture.
  3. Works.

So, apparently, there is a difference with how glvideoflip and glviewconvert handle implicit modifiers, not sure whether is this a bug though.

2nd pipeline not working is also quite bad, this doesn’t look related to implicit modifiers. Is there anything I can do to debug this further?

45

Just a note: “does not show a window” likely comes with a “not negotiated” bus error. You may want to monitor and trace these. That would mean the caps you provide on DMABuf don’t translate to video/x-raw\(memory:GLMemory\),format=RGBA in glupload. It is probably normal.

46

I don’t think there is a “not negotiated” bus error in that case as the application doesn’t exit (and it usually did when that error occured). Pipeline seems to be up and running, but no window without glcolorconvert.
With GST_DEBUG=4 the following lines are printed repeatedly (and a few more not very relevant ones as well):

01.949750684 848537 0x768054000b90 INFO               structure gststructure.c:3958:gst_structure_get_valist: Expected field 'gl-allocation-params' in structure: GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\(memory:GLMemory\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ texture-target\=\(string\)2D\,\ framerate\=\(fraction\)0/1", size=(uint)4096000, min-buffers=(uint)0, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)"GstAllocationParams\,\ flags\=\(GstMemoryFlags\)0\,\ align\=\(guint64\)0\,\ prefix\=\(guint64\)0\,\ padding\=\(guint64\)0\;", gl-min-free-queue-size=(uint)1, options=(string)< GstBufferPoolOptionGLSyncMeta, GstBufferPoolOptionGLTextureTarget2D >;
0:00:01.949767556 848537 0x768054000b90 INFO            glbasefilter gstglbasefilter.c:567:gst_gl_base_filter_find_gl_context_unlocked:<gltransformation0> found OpenGL context <glcontextegl0>
0:00:01.949782304 848537 0x7680a8016050 INFO            glbasefilter gstglbasefilter.c:400:_gl_set_caps:<gltransformation0> set GL caps input video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)1280, height=(int)800, texture-target=(string)2D
0:00:01.949790970 848537 0x7680a8016050 INFO            glbasefilter gstglbasefilter.c:402:_gl_set_caps:<gltransformation0> set GL caps output video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)1280, height=(int)800, texture-target=(string)2D, framerate=(fraction)0/1
0:00:01.949825827 848537 0x768054000b90 INFO               structure gststructure.c:3958:gst_structure_get_valist: Expected field 'gl-allocation-params' in structure: GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\(memory:GLMemory\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ texture-target\=\(string\)2D\,\ framerate\=\(fraction\)0/1", size=(uint)4096000, min-buffers=(uint)1, max-buffers=(uint)0, allocator=(GstAllocator)"\(GstGLMemoryPBOAllocator\)\ glmemorypboallocator0", params=(GstAllocationParams)"GstAllocationParams\,\ flags\=\(GstMemoryFlags\)0\,\ align\=\(guint64\)0\,\ prefix\=\(guint64\)0\,\ padding\=\(guint64\)0\;", gl-min-free-queue-size=(uint)1, options=(string)< GstBufferPoolOptionGLSyncMeta, GstBufferPoolOptionGLTextureTarget2D >;
0:00:01.949849442 848537 0x768054000b90 INFO               structure gststructure.c:3958:gst_structure_get_valist: Expected field 'gl-allocation-params' in structure: GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\(memory:GLMemory\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ texture-target\=\(string\)2D\,\ framerate\=\(fraction\)0/1", size=(uint)4096000, min-buffers=(uint)1, max-buffers=(uint)0, allocator=(GstAllocator)"\(GstGLMemoryPBOAllocator\)\ glmemorypboallocator0", params=(GstAllocationParams)"GstAllocationParams\,\ flags\=\(GstMemoryFlags\)0\,\ align\=\(guint64\)0\,\ prefix\=\(guint64\)0\,\ padding\=\(guint64\)0\;", gl-min-free-queue-size=(uint)1, options=(string)< GstBufferPoolOptionGLSyncMeta, GstBufferPoolOptionGLTextureTarget2D, GstBufferPoolOptionVideoMeta >;
0:00:01.949880741 848537 0x768054000b90 INFO               structure gststructure.c:3958:gst_structure_get_valist: Expected field 'gl-min-free-queue-size' in structure: GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\(memory:GLMemory\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ texture-target\=\(string\)2D", size=(uint)4096000, min-buffers=(uint)0, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)"GstAllocationParams\,\ flags\=\(GstMemoryFlags\)0\,\ align\=\(guint64\)0\,\ prefix\=\(guint64\)0\,\ padding\=\(guint64\)0\;";
0:00:01.949894327 848537 0x768054000b90 INFO               structure gststructure.c:3958:gst_structure_get_valist: Expected field 'gl-allocation-params' in structure: GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\(memory:GLMemory\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ texture-target\=\(string\)2D", size=(uint)4096000, min-buffers=(uint)0, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)"GstAllocationParams\,\ flags\=\(GstMemoryFlags\)0\,\ align\=\(guint64\)0\,\ prefix\=\(guint64\)0\,\ padding\=\(guint64\)0\;";
0:00:01.949904757 848537 0x768054000b90 INFO            glbasefilter gstglbasefilter.c:567:gst_gl_base_filter_find_gl_context_unlocked:<gluploadelement0> found OpenGL context <glcontextegl0>
0:00:01.949915748 848537 0x768054000b90 INFO               structure gststructure.c:3958:gst_structure_get_valist: Expected field 'gl-min-free-queue-size' in structure: GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\(memory:GLMemory\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ texture-target\=\(string\)2D", size=(uint)4096000, min-buffers=(uint)1, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)"GstAllocationParams\,\ flags\=\(GstMemoryFlags\)0\,\ align\=\(guint64\)0\,\ prefix\=\(guint64\)0\,\ padding\=\(guint64\)0\;";
0:00:01.949928702 848537 0x768054000b90 INFO               structure gststructure.c:3958:gst_structure_get_valist: Expected field 'gl-allocation-params' in structure: GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\(memory:GLMemory\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ texture-target\=\(string\)2D", size=(uint)4096000, min-buffers=(uint)1, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)"GstAllocationParams\,\ flags\=\(GstMemoryFlags\)0\,\ align\=\(guint64\)0\,\ prefix\=\(guint64\)0\,\ padding\=\(guint64\)0\;";
0:00:01.949942128 848537 0x768054000b90 WARN           basetransform gstbasetransform.c:2228:default_generate_output:<gluploadelement0> could not get buffer from pool: ok

What’s so special about glcolorconvert that fixes glvideoflip?

For the reference, I’ve also tried manually fixating the caps:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! video/x-raw(memory:GLMemory),format=RGBA ! glvideoflip name=flip ! glimagesink

The result is the same.

I guess I’ll just slap glcolorconvert everywhere after glupload, this probably won’t be a big deal performance-wise.

The main issue is still the following pipeline:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glcolorconvert ! gldownload ! video/x-raw ! videoflip name=flip ! glimagesink

Is there anything I can do to debug why it’s only showing black picture? Surely GStreamer can read pixels from the GPU back into the CPU memory, right?

47

More trace, there hundreds of debug categories you may enable with GST_DEBUG environment.

But any reason you want so achieve software flip? There is an element in GL for that, plus glimagesink can flip while rendering saving a render pass.

48

Rendering in glimagesink isn’t the end goal, so flip was just added as a visual example of processing the data on the CPU.
I’ll need to later encode video with different encoders (CPU, GPU, special hardware cards and so on), so need to get it back to the system memory.

I’ve captured logs at GST_DEBUG=5: logs5.txt - Google Drive

I can’t find anything odd there.

49

@ndufresne is there anything else I can try?

BTW, can you maybe review my MR?

50

Okay, so no matter what I do, gldownload is just broken. @ndufresne is this a known issue?
For the reference, the following pipeline works:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glcolorconvert ! glviewconvert input-mode-override=left name=flip ! glimagesink

But this one doesn’t:

appsrc format=time do-timestamp=true stream-type=stream is-live=true name=src ! glupload ! glcolorconvert ! gldownload ! video/x-raw ! videoflip name=flip ! glimagesink

Another thing is that the newly introduced ScanoutDMABUF2 may send more than 1 fd. Am I supposed to create GstMemory for each one and if so, how should offsets and strides be mapped to fds? Is there always one-to-one relationship if more than 1 fd is present?

51

Another thing is that the newly introduced ScanoutDMABUF2 may send more than 1 fd. Am I supposed to create GstMemory for each one and if so, how should offsets and strides be mapped to fds? Is there always one-to-one relationship if more than 1 fd is present?

The GstVideoMeta offsets are relative to the GstBuffer arrangement. So if you have two plane in two dmabuf, they both have their own GstMemory, the offeset will be 0 and sizeof(first_mem).

52

So GstVideoMeta offsets have nothing to do with the actual dmabuf offsets, do they? This is extremely confusing.

The current situation is as follows:
There are 2 planes and my program receives 1 fd, 2 offsets (0, 4587520) and 2 strides (5120, 1536), resolution is 1280x800. I create one GstMemory with that fd of size 4587520 + 1536*800 = 5816320 bytes (“max offset + height * stride for max offset”) , then put it into GstBuffer and add GstVideoMeta with these exact offsets and strides. Interestingly, the offset for the second plane is 4587520, which is bigger than 800 * 5120 for whatever reason. I guess some alignment?

I was told that it is possible to receive more than one fd and the same amount of offsets and strides (that is, n_planes of them). Would the dmabuf offsets would be different from what GstVideoMeta offsets in this case? If so, I’d have to create GstMemory for each fd (so n_planes of them), add them to GstBuffer and then calculate offsets? But these offsets would not be the same as the ones which were received and I expect importing the dmabuf to fail miserably because of that, so will GStreamer handle this somehow?

53

Since you have a single FD, the GStreamer and DMABuf offset are the same. They still correlate otherwise.

The offset you have indicates that your image has a vertical alignment of 128 lines, 896 pixels is the padded height).

54

Since you use appsrc, it’s up to you to translate.

55

But what if I’ll get more than one fd on a different machine? I don’t understand how to handle this situation with the GStreamer API. There doesn’t seem to be a way to pass both dmabuf and GStreamer offsets. Documentation doesn’t say anything about this, it’s just “offset of each plane” and that’s it.
I looked into the code and it indeed looks like API needs to be extended to be less confusing:

So the offset that will be passed for importing is stored in the GstMemory, but it’s not possible to set it when creating it as gst_dmabuf_allocator_alloc will call gst_fd_allocator_alloc which in turn calls gst_fd_allocator_alloc_full with hardcoded offset of 0. This is wrong in the general case, so, I guess, I have to update the offset by writing to the GstMemory’s offset field. Is this what I’m expected to do?

56

Then this is wrong:

Besides the file descriptors , there may be a GstVideoMeta data attached to each GstBuffer to describe more information such as the width, height, pitches, strides and plane offsets for that DMA buffer

(from DMA buffers)
These offsets not necessarily the same as dmabuf ones as per what you said.

57

Moderator hat on, please avoid ranting on the forum, the post is then no longer useful to others.

We have a lot of code that uses per plane GstMemory and it works. It’s not simple though, and highly specific to the format the original memory you received is. You are asking relatively vague question. I also find this thread be used as a chat, perhaps you should create threads for specific and clear questions. We also have matrix chat for that purpose.

Here’s my final answer, I’ll close this thread afterward. The video meta offsets are basically not changed even if the buffer is composed of two or more memory. There is GstBuffer to locate the memory. If you need to pass this to the lowest level, like through a Linux ioctl, the FD specific offset is the sum of the returned skip value and memory.offset. memory.offset is rarely used for DMABuf.

58

I’m still having issues with DMA and shmem buffers. Should I open another thread?
The current pipeline is

appsrc name=videosrc format=time do-timestamp=true stream-type=stream is-live=true ! queue leaky=downstream flush-on-eos=true max-size-time=40000000 max-size-buffers=0 max-size-bytes=0 ! vapostproc ! video/x-raw(memory:VAMemory),format=NV12 ! vah264enc name=vcodec bitrate=20000 ! video/x-h264,stream-format=byte-stream,profile=high ! appsink name=appsink

shmem is quite slow for some reason when encoding via vah264enc, but it seems totally fine with my second app (which just displays the video via glimagesink).

And dmabufs are still failing to negotiate with vah264enc:

0:00:00.152998911 35901 0x7ceb5c000d80 DEBUG          basetransform gstbasetransform.c:1260:gst_base_transform_acceptcaps_default:<vapostproc0> accept caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, width=(int)1280, height=(int)800, drm-format=(string)XB24:0x20000000056bb03
0:00:00.153004511 35901 0x7ceb5c000d80 DEBUG          basetransform gstbasetransform.c:1266:gst_base_transform_acceptcaps_default:<vapostproc0> intersect with pad template: video/x-raw(memory:VAMemory), width=(int)[ 1, 16384 ], height=(int)[ 1, 16384 ], format=(string){ NV12, YV12, I420, P010_10LE, P012_LE, GRAY8, UYVY, YUY2, Y444, RGBA, BGRA, RGBx, BGRx, BGR10A2_LE }; video/x-raw(memory:DMABuf), width=(int)[ 1, 16384 ], height=(int)[ 1, 16384 ], format=(string)DMA_DRM, drm-format=(string){ NV12, YV12, YU12, P010, P012, "R8\ \ ", YUYV, YU24, AB24, AR24, XB24, XR24, AR30 }; video/x-raw, width=(int)[ 1, 16384 ], height=(int)[ 1, 16384 ], format=(string){ NV12, P010_10LE, P012_LE, I420, YV12, YUY2, UYVY, GRAY8, Y444, RGBP, BGRA, RGBA, BGRx, RGBx, BGR10A2_LE }
0:00:00.153007878 35901 0x7ceb5c000d80 DEBUG          basetransform gstbasetransform.c:1288:gst_base_transform_acceptcaps_default:<vapostproc0> caps can't intersect with the template
0:00:00.153009350 35901 0x7ceb5c000d80 DEBUG          basetransform gstbasetransform.c:1278:gst_base_transform_acceptcaps_default:<vapostproc0> accept-caps result: 0
0:00:00.153011074 35901 0x7ceb5c000d80 DEBUG               GST_PADS gstpad.c:4256:gst_pad_query:<vapostproc0:sink> sent query 0x7cebc8430420 (accept-caps), result 1
0:00:00.153012777 35901 0x7ceb5c000d80 DEBUG               GST_PADS gstutils.c:2798:gst_pad_proxy_query_accept_caps:<queue0:sink> proxying accept caps query: 0
0:00:00.153014420 35901 0x7ceb5c000d80 DEBUG               GST_PADS gstpad.c:3554:gst_pad_query_default:<queue0:sink> not forwarding 0x7cebc8430420 (accept-caps) query
0:00:00.153016033 35901 0x7ceb5c000d80 DEBUG               GST_PADS gstpad.c:4256:gst_pad_query:<queue0:sink> sent query 0x7cebc8430420 (accept-caps), result 1
0:00:00.153017466 35901 0x7ceb5c000d80 DEBUG               GST_CAPS gstutils.c:3202:gst_pad_query_accept_caps:<queue0:sink> query returned 0
0:00:00.153019429 35901 0x7ceb5c000d80 WARN                GST_CAPS gstpad.c:5888:pre_eventfunc_check:<queue0:sink> caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, width=(int)1280, height=(int)800, drm-format=(string)XB24:0x20000000056bb03 not accepted
0:00:00.153021153 35901 0x7ceb5c000d80 DEBUG               GST_PADS gstpad.c:6162:gst_pad_send_event_unchecked:<queue0:sink> pre event check failed
0:00:00.153023227 35901 0x7ceb5c000d80 LOG                 GST_PADS gstpad.c:5670:gst_pad_push_event_unchecked:<videosrc:src> sent event 0x7cebc8498240 (caps) to peerpad <queue0:sink>, ret not-negotiated

Why are these even considered incompatible? Every part of the video caps seems to be compatible with vapostproc caps. Is this once again modifier issue?

It seems to go a bit further if I force 1 video plane and don’t add modifier to the caps:

0:00:00.061621115 38478 0x713b18000b70 WARN               vaencoder gstvaencoder.c:838:gst_va_encoder_has_trellis:<vaencoder1> Driver does not support trellis

(wormhole:38478): GStreamer-CRITICAL **: 00:11:11.778: _gst_util_uint64_scale: assertion 'denom != 0' failed

(wormhole:38478): GStreamer-Video-CRITICAL **: 00:11:11.778: gst_video_encoder_set_latency: assertion 'GST_CLOCK_TIME_IS_VALID (min_latency)' failed
0:00:00.062115331 38478 0x713b18000b70 ERROR              vadisplay vasurfaceimage.c:171:va_create_surfaces: vaCreateSurfaces: resource allocation failed
0:00:00.062187056 38478 0x713b18000b70 ERROR               fdmemory gstfdmemory.c:132:gst_fd_mem_map: 0x713b5c001890: fd 29: mmap failed: Bad file descriptor
0:00:00.062193689 38478 0x713b18000b70 ERROR               fdmemory gstfdmemory.c:132:gst_fd_mem_map: 0x713b5c001890: fd 29: mmap failed: Bad file descriptor
0:00:00.062200852 38478 0x713b18000b70 ERROR              videometa gstvideometa.c:427:default_map: cannot map memory range 0-1
0:00:00.062205151 38478 0x713b18000b70 ERROR                default video-frame.c:168:gst_video_frame_map_id: failed to map video frame plane 0
0:00:00.062209659 38478 0x713b18000b70 WARN         vabasetransform gstvabase.c:178:gst_va_buffer_importer_import:<vapostproc0> warning: invalid video buffer received
59

Apparently, only linear modifier is supported. Well, game over. Or did I get it wrong?

60

As we discuss the code got analysed out of context. The modifiers are supported with VA drivers that supports it. This is a fallback for older drivers. Your specific AMD mesa driver have no working support yet, early patches exists but do not work for VA.