Hi! I’m using the gstreamer to receive RTP h264 video stream on the Oculus Quest 3 device.
Here is my pipeline
udpsrc port=5000 ! application/x-rtp, media=video ! queue ! rtpjitterbuffer ! rtph264depay ! h264parse ! avdec_h264 ! videorate ! video/x-raw,framerate=15/1 ! videoconvert ! autovideosink sync=false
The latency is smth like ~300-350ms. At some random moment, the stream freezes for 1-2 seconds and after that, the latency increases up to ~1 sec and sometimes the video is like in slow-motion.
I suppose that the problem is related to frames buffering.
The question is how can I fix the issue? And maybe you can advise me on how I can additionally optimize the gstreamer pipeline to reduce the latency?
Thanks!
Since you have disabled stream synchronization (sync property on video sync), the rendering delays are unreliable and depends on non-realtime scheduling of threads on both the sender and the receiver. Overrun are no longer compensated by skipping notably. I would recommend to start with time synchronization enabled.
From there, work on the sender, specially the encoding parameters. The encoding parameters will contribute to the latency needed to decode.
The rtpjitterbuffer has a latency configuration which is 200ms by default, you will likely want to reduce that.
You may want to adjust the udpsrc buffer-size. This will adjust the buffering in the kernel. With the load, non realtime process (like GStreamer) can loose some packet, which adds to the brokenness.
Finally, you may want to introduce RTCP using rtpbin (or the rtpsrc/sink element). This feedback channel will allow improving the timing (which is needed to reduce the latency) andwill also allow enabling retransmission, error correction, key frame recovery, etc.