GLcolorconvert and RTP raw payloder - performance issues

jpietek · December 4, 2024, 3:56pm

Hi,

I have a Gstreamer app that has appsrc input with GLMemory RGBA buffers taken from libobs based video mixer.

The pipeline I’m trying to run should convert those buffers to YUV422 and send them over R(C)TP. With the CPU-based pixel format conversion it’s fine:

gldownload ! videoconvert ! video/x-raw,width=1920,height=1080,framerate=60/1,format=UYVY ! rtpvrawpay ! udpsink

The problem starts at 4K60fps when I try to use glcolorconvert to speed up pixfmt conversion and not push raw RGBA over PCI-Express bus.

glcolorconvert ! video/x-raw(memory:GLMemory),width=3840,height=2160,framerate=60/1,format=UYVY ! gldownload ! rtpvrawpay ! udpsink

This leads to huge number of frames accumulating at the RTP payloader. Adding a leaky queue can make it run, but the framerate is only a few fps.

The question is as always: why?! After glcolorconvert the buffers have GstGLSyncMeta attached. Maybe this is problematic for the RTP payloader. Removing those metadata might help? Any hints or tips appreciated.

cheers,
Jan

tpm · December 4, 2024, 6:08pm

Couple of things you could try:

add a queue max-size-bytes=0 max-size-time=0 max-size-buffers=5 before and after the rtpvrawpay
use a higher rtpvrawpay chunks-per-frame=X than the default (shouldn’t change overall cpu usage, just gets out data quicker and might lubricate things a little by sending smaller chunks at a time)
increase the kernel send buffer size with e.g. sudo sysctl net.core.wmem_max=67108864 (might be excessive, but the default is very very low) and then use a higher value with udpsink buffer-size=X (won’t change cpu usage, but makes packet loss through “lumpy scheduling of data to send out” less likely)

tpm · December 4, 2024, 6:09pm

Also, videoconvert has an n-threads property which can be set to a non-1 value to parallelise the conversion, which might help push through frames on time with software conversion (depends on your CPU ofc).

jpietek · December 9, 2024, 12:18pm

@tpm many thanks for the hints. I’ve tuned the tcp stack and udpsink buffer-size, bumped payloader chunks-per-frame, but it was not (only) the case. Even with fakesink the payloader was still lagging.

What - did - help was commenting out this line:

github.com

GStreamer/gst-plugins-good/blob/20bbeb5e37666c53c254c7b08470ad8a00d97630/gst/rtp/gstrtpvrawpay.c#L577


      
            GST_LOG_OBJECT (rtpvrawpay, "field/frame complete, set marker");
            gst_rtp_buffer_set_marker (&rtp, TRUE);
            complete = TRUE;
          }
          gst_rtp_buffer_unmap (&rtp);
          if (left > 0) {
            GST_LOG_OBJECT (rtpvrawpay, "we have %u bytes left", left);
            gst_buffer_resize (out, 0, gst_buffer_get_size (out) - left);
          }
          
          gst_rtp_copy_video_meta (rtpvrawpay, out, buffer);
          
          /* Now either push out the buffer directly */
          if (!use_buffer_lists) {
            ret = gst_rtp_base_payload_push (payload, out);
            continue;
          }
          
          /* or add the buffer to buffer list ... */
          gst_buffer_list_add (list, out);

So seems like GstGLSyncMeta was the problem? I don’t understand why after gldownload it is still there? I would assume that for the payloader passing anything related to the OpenGL context makes no sense?