Curious if there are any special considerations when mixing multiple RTP streams through webrtcbin.
I’ve got a basic mixer setup that works fine for most cases, but incurs latency on the receiver side for some specific endpoint types (absent from others).
The basic pipeline structure is that each clients gets its own webrtcbin connection, its own audiomixer and its own compositor. All other client’s send audio and video into eachothers audiomixer | compositor, which is then output through that clients webrtcbin.
What’s interesting is that for most client types (browsers), glass to glass is pretty good (~250 ms). However, for other endpoint types (some android clients for example), glass to class is 550-650ms. When those same android endpoints call one another directly using TURN, glass to glass is about 200ms.
For example:
Chrome (250ms) ↔ Chrome (250ms)
Chrome (250ms) ↔ Android (550ms)
Something in the mixer flow is causing specific endpoint types to have higher latency than others. What are some things I should keep in mind for a setup like this? A clue i see is that one of the queues in webrtcbin is filling up.
Each client has its own webrtcbin that is used for input and output.
Each client has its own audiomixer.
Each client has its own compositor.
Each client sends its audio into every other clients audiomixer (not its own).
Each client sends its video into every other clients compositor (including its own).
OPUS and openh264 are used has the decoders/encoders.
Clients are browsers, Android phones, iOS phones, etc.
Any insights would be lovely, as Im stumped! My suspicion is that it has something to do with the audio stream, as latency tends to be much lower when audio is disabled in the pipeline.
I noticed that there is a ts-offset on one of the jitter buffers as well. That makes me think that the JB is trying to sync audio/video, but the audio is behind the video. I’m guessing the ts-offset is adding time to the PTS on the buffers, which are being held at the sink.
