Synchronizing Multiple RTSP Camera Streams in GStreamer

Application Development
1

Hi everyone,

I’m currently working on a project where I need to synchronize four RTSP camera streams using GStreamer. Initially, the streams are in sync, but over time, they tend to drift out of synchronization. This is quite problematic for the application I’m working on.

Here’s a brief overview of my setup:

  • Four RTSP camera sources
  • Live streams (not pre-recorded)
  • GStreamer pipeline handling the streams (it doesn’t matter if it’s an empty pipeline with cameras going straight to fakesink or a DeepStream pipeline used in the production environment)

I’ve tried several methods to address the desynchronization issue, but none have been successful so far:

  • Flushing the pipeline
  • Sending latency messages
  • Stopping and playing the streams again
  • Resetting GST_CLOCK

Despite these efforts, the streams still go out of sync after some time. I suspect that the cameras themselves might be contributing to the desynchronization. However, I’m looking for a solution that can force the streams back into sync when desynchronization occurs (this should be possible due to them being live sources, I believe).

Has anyone encountered a similar issue or have any suggestions on how to maintain synchronization for live RTSP streams over extended periods? Additionally, if desync does happen, what steps can be taken to force the streams back into sync?

Also, I provided some graphs that explain my issue. It is a PTS probe graph and, as you can see, the cameras seem to lag for a bit and then never sync again. It can be seen to drift (PTS value) by around 1 second. After a while, the second and third cameras drift and seem to be in sync with the first desynced camera.

PTS graph
PTS graph1917×2721 404 KB

Any advice or insights on this matter would be greatly appreciated. Thanks in advance for you help.

2

Not sure for your case, but does enabling ntp-sync in rtspsrc improve ?

3

I believe i have tried it before on a full pipeline and it did not help but i guess i should try it with just current testing setup (4 cameras straight to fake sink). Thanks for you input.

EDIT: Tried it, no changes

4

Are the cameras configured to sync to a common NTP server?

What transport is being used for the data flow? RTSP interleaved or UDP?

What version of GStreamer are you using?

5

Hello,

yes cameras are configured to sync to a common NTP server. I tried using both TCP and UDP for data flow (in production pipeline i use nvurisrcbin to configure rtsp sources)

In production we are currently using Deepstream 6.2 so gstreamer 1.16

Graph shown above with “empty” pipeline were done using gstreamer 1.20 with no visible differences. It should be noted (probably not the issue here) that i set up packet loss messages for jitterbuffer using post-drop-messages flag in gstreamer 1.20 and they did not happen at all. I had a few tests done on a different camera that had these packet loss messages so the code for packet loss messages works.