I’m trying to get the lowest latency while still being rock solid. Both devices will always be on the same network and for my purposes we can assume the sender will be connected directly to ethernet and the receiver will be over wifi with direct line of sight to a router. I’m running this through a C++ plugin so I’m not sure if there are any command line debug arguments I can use to help or how I would implement them in code.
These pipelines work pretty well but there is still some clear jitter randomly every number of seconds if I watch closely while swinging a window around in a circle. I’ve been reading through the docs for every element trying to find what I can tune to eliminate this micro stutters and I’m struggling to find anything else.
One thing I’ve noticed is in my logs (which are at a fairly low log level) there’s a lot of
01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Assuming DONL field is not present 01-25 16:53:31.859 1663 2120 I : [GStreamer] Pushing frame buffer: 0xb400007692830c40, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 13572, offset none, offset_end none, flags 0x0
I’m not expert in anything video streaming but I wonder if pts and dts here are really supposed to be 99:99:99.999999999? It’s a live stream with sync disabled to maybe this makes sense but I figured I should still mention it.
Any ideas on how to further tune this would be great.
Since you seem to be controlling both sides, you might get better results if you add re-transmissions. see the doc for rtprtxreceive for examples of how to enable it. Or even better, you can use webrtcbin to send and receive which connects re-tranmissions and can do bitrate adaptation.
Depending on the WiFi quality and interference, as well as the scheduling priority on the devices, you should be able to do less than 60ms.
The jitter is cause because you have “sync=false” on the receiver, enable it and the jitter should go away. Otherwise, the jitter will indeed be random based on what happens in the network or elsewhere in the system.
On the sender side, you may want to enable sync=true as well, but also set max-bitrate=XX to a value slightly higher than your encoder bitrate, but not exceeding the network bitrate.
I’m going to play with rtprtxreceive and see if it helps at all.
I’m fairly confident webrtcbin will add extra latency (webrtc vs UDP). I’m sure it would still be fine but I’d really like to be as low low as possible which is why I’m trying just UDP and RTP.
The jitter is cause because you have “sync=false” on the receiver, enable it and the jitter should go away.
It still jitters with sync enabled on the receiver. It also increases latency substantially.
On the sender side, you may want to enable sync=true as well, but also set max-bitrate=XX to a value slightly higher than your encoder bitrate, but not exceeding the network bitrate.
If sync is enabled on both it’s unusable. Latency jumps up into a second and a half of delay.
Webrtc should have roughly the same pipeline inside the bit as you created, but webrtcsink adds a appsink/appsrc pair which could make your life worse indeed.
Setting sink=true on the receiver side make it use the configure latency (of 60ms) at all times, otherwise, it will have a ltency of up to 60 ms if there is a list packet, which will add visible jitter. There is definitely a trade-off there, you can also lower the latency on the jb (to like 15-20ms) and enable it and see how that works for you.
Hi @bluewave41 As you are working with a live video, I would initially give you the following suggestion:
Don’t use default queues, configure them to buffer maximum of 2 buffers and drop the rest, use this queue max-size-buffers=2 leaky=downstream . Make this change for all the queues, however, I would advise on removing queues from parts of the pipeline with encoded frames, if you are going to configure them as leaky, because that corrupts the video, and leaving them as default could add delays, so try removing the queue right before the udpsink on the sender pipeline and the one after the rtpjitterbuffer.
I’d previously read in general that queues before sinks are good practice so this is news to me. I removed the queues before the sinks.
ChatGPT was always trying to get me to use smaller queue sizes but I never saw a difference and just ignored it. I see the default size is like 200 though so these are probably good changes.
Syncing on the sender and receiver added tons of latency but I haven’t determined if that was due to my outdated pipewire not sending proper clock values or whatever else it may have been.
I think updating pipewire and GStreamer provided the greatest benefits. Ubuntu 24 pipewire is very outdated (1.0.7 vs 1.5.45) and GStreamer is still a number of versions behind. Installing those to a /usr/local prefix as well as tinkering with paths and library paths seems to have improved both quality and performance dramatically.