Problems adding RED to audio in webrtc

General Discussion
1

I’ve asked this on SO (c++ - Enable RED on webrtcbin for audio - Stack Overflow), but got nowhere, so I’m trying here.

General description:
I’ve got a media server that receives SDP offers from endpoints, then acts as a middleman between them all, so I’ll have a pipeline a bit like this:

endpoint1 <--> server <--> endpoint2
                                     --> WatcherOfEndpoint1

So my server has components matches up to the various endpoints, which include a webrtcbin that it listens to, gets streams from, decodes them to video/x-raw or audio/x-raw and then the components communicate with each other via appsink/appsrc.

However, sometimes our audio from one of the endpoints (say endpoint1) isn’t great. The other endpoints can’t hear it very well, even though endpoint1 can hear everyone else just fine. So I would like to try and beef up the audio on that one endpoint a bit more, using RED (as suggested here - RED: Improving Audio Quality with Redundancy - webrtcHacks).

Making dot diagrams of my pipelines suggest that webrtcbin should be capable of doing RED internally, without me adding elements to my pipelines (though I’ve tried that, see the SO post). The trouble is, even if I munge the SDP offer from my Vivaldi browser so that the red line is above the opus line, my answer prefers opus. The only way I can get red to be the number one choice in my answer is by removing opus entirely from the m-line (although I wonder if that causes problems with it understanding what’s within the red-encoded stream?)

If I get-transceivers, initially kind isn’t set, so I can’t target just the audio, and if I set both of them to use fec-type and do-nack, and then revert the video one back, I get the webrtcbin emitting just the video pad-added signal, then complaining that it can’t stream because it’s not negotiated. If I do nothing, or set the audio transceiver when I finally have the kind, then I get video, but the audio pad-added signal never fires.

Any suggestions welcome, I’m pretty much out of ideas!

2

Sending RED from my server to the client (relatively simple):

Build pipeline elements as normal, hook up a listener on the on-new-transceiver signal, then connect my webrtcbin to video and audio payloaders (the latter is rtpopuspay). Because I know the order I do the linking, I know which transceiver is going to be the audio, and I set that transceiver up to use RED:

g_object_set(G_OBJECT(transceiver),
			 "fec-type", GST_WEBRTC_FEC_TYPE_ULP_RED,
			 "fec-percentage", 100,
			 "do-nack", true, NULL);

However, even with a munged SDP offer putting RED in front of vanilla OPUS, the answer has OPUS first. So we end up with the server transmitting RED, but being sent OPUS, which makes sense from understanding what each end sees in the offer and answer.

Sending RED in both directions

This is where it gets tricky, and I don’t know if this should be raised as a PR for gstreamer, or if someone with more knowledge than me wants to explain why what I’ve done is an awful idea.

The reason that OPUS always gets first billing in the SDP answer is because in gstwebrtcbin.c, in create_answer_task, we make a call to gst_sdp_media_set_media_from_caps, then _media_add_fec. Getting the media from caps looks first for codec_preferences, but those aren’t set, so it looks at the pad that we’re connected to. We’re connected to rtpopuspay, so we end up with vanilla OPUS. (If I set codec_preferences when adding the transceiver to try and force it to use RED, then the linking to rtpopuspay doesn’t work because of the mismatch). Then media_add_fec comes along, and adds RED.

I managed to get RED in first place by changing the order of these two calls - however, my browser still transmitted OPUS until I changed the rtpmap line, and added in the fmtp line.

# Old version
a=rtpmap:63 red/48000
a=rtpmap:111 OPUS/48000/2

# New version
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:111 OPUS/48000/2

Are these changes that suggest something needs changing in gstreamer, and if so how would I go about offering the patch up?