Any specific considerations when handling multiple mixed RTP streams through webrtcbin

Curious if there are any special considerations when mixing multiple RTP streams through webrtcbin.

I’ve got a basic mixer setup that works fine for most cases, but incurs latency on the receiver side for some specific endpoint types (absent from others).

The basic pipeline structure is that each clients gets its own webrtcbin connection, its own audiomixer and its own compositor. All other client’s send audio and video into eachothers audiomixer | compositor, which is then output through that clients webrtcbin.

What’s interesting is that for most client types (browsers), glass to glass is pretty good (~250 ms). However, for other endpoint types (some android clients for example), glass to class is 550-650ms. When those same android endpoints call one another directly using TURN, glass to glass is about 200ms.

For example:

  • Chrome (250ms) ↔ Chrome (250ms)
  • Chrome (250ms) ↔ Android (550ms)

Something in the mixer flow is causing specific endpoint types to have higher latency than others. What are some things I should keep in mind for a setup like this? A clue i see is that one of the queues in webrtcbin is filling up.

Other’s are seeing the same thing: Large latency with webrtcbin (#1261) · Issues · GStreamer / gstreamer · GitLab

To recap on the pipeline:

  1. Each client has its own webrtcbin that is used for input and output.
  2. Each client has its own audiomixer.
  3. Each client has its own compositor.
  4. Each client sends its audio into every other clients audiomixer (not its own).
  5. Each client sends its video into every other clients compositor (including its own).
  6. OPUS and openh264 are used has the decoders/encoders.
  7. Clients are browsers, Android phones, iOS phones, etc.

Any insights would be lovely, as Im stumped! My suspicion is that it has something to do with the audio stream, as latency tends to be much lower when audio is disabled in the pipeline.

I noticed that there is a ts-offset on one of the jitter buffers as well. That makes me think that the JB is trying to sync audio/video, but the audio is behind the video. I’m guessing the ts-offset is adding time to the PTS on the buffers, which are being held at the sink.

Or at least, possibly.

I think I found the root cause. For some device pairings, the ts-offset value on the jitter buffer is huge. I guess its trying to do AV sync.

Is there any way to remediate this (such as disabling AV sync in webrtcbin)? The latency is quite high as you can see below.

I tried to set the max-ts-offset-adjustment value but it seems the ts-offset still climbs beyond it.

rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 30449551 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 908079013 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 30449551 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 815944206 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 25849607 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 669923523 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 25849607 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 489499346 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 11679512 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 344486083 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 11679512 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 256859119 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 21913902 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 213262411 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 21913902 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer1] offset: 0 
rtp-jitter-buffer [rtpjitterbuffer0] offset: 169741249 
rtp-jitter-buffer [rtpjitterbuffer3] offset: 10606273 
rtp-jitter-buffer [rtpjitterbuffer2] offset: 0