I’m using the webrtc example mentioned here in my tauri app with with the only change being that I’m only receiving video and audio, and using vaapivp9enc (will replace it with vavp9enc once 1.26 gets released) to force vp9 to achieve better quality. It starts off fine. After some time, lot of artifacts appear and the stream gets messed up.
I used the get-stats action available in webrtcbin and observed that this occurs when there is packet loss.
Packet loss can happen for any number of reasons and is a consequence of using a network. The only way to avoid this is by using mitigation techniques like RTX (retransmissions) or FEC. RTX is enabled by setting the ‘do-nack’ property to TRUE before SDP negotiation occurs. FEC is enabled by setting the ‘fec-type’ property on the transceiver to ‘ulp-red’ and configuring an appropriate ‘fec-percentage’
Another aspect I’ve just discovered is that we don’t support VP9 error_resiliant_mode, even in the new encoder wrapper.
/* We do not support error resilient mode now. */
.error_resilient_mode = 0,
Without this, the lost of one packet will lead to complete visual corruption of the following delta frame until the next keyframe is received. It would be nice to check with Intel if this is a limitation on their end, or if its just the GStreamer implementation. I do believe that without error_resilient_mode, VP9 is not going to work very well under webrtc, unless you have a lot of FEC/RTX and that you ensure to drop any corrupted frame.
To set the do-nack property, do I need to get the transceiver or add a new transceiver from the webrtcbin?
Edit: Nvm, I figured it out. Although for some reason it’s been removed from the current webrtc example. It used to be there before.
let pipeline = gstreamer::parse::launch(
"videotestsrc pattern=ball is-live=true ! vah264lpenc target-usage=1 ! rtph264pay name=vpay pt=96 ! webrtcbin. \
audiotestsrc is-live=true ! opusenc perfect-timestamp=true ! rtpopuspay name=apay pt=97 ! application/x-rtp,encoding-name=OPUS ! webrtcbin. \
webrtcbin name=webrtcbin latency=100"
).expect("Failed to launch");
let pipeline = pipeline.downcast::<gstreamer::Pipeline>().expect("Not a pipeline");
let webrtcbin = pipeline.by_name("webrtcbin").expect("Did not find webrtcbin");
let transceiver = webrtcbin.emit_by_name::<gstreamer_webrtc::WebRTCRTPTransceiver>("get-transceiver", &[&0i32]);
transceiver.set_property("do-nack", true);
I did this and it doesn’t seem to have any effect? nack-count is 0 and packets-lost is 10
Hmmm that’s interesting.
I tried with vah264lpenc and I’m still seeing artifacts after packet loss. Could it be an intel issue?
I’ll try with software encoding (x264enc) and check if it still occurs. If it’s still happening, can we conclude that this is an Intel issue?
Update: Using x264enc did not change anything. As soon as there is packet loss, there is artifacting
Well, packet lost == artifacts, unless you have a way to recover from it (FEC or RTX). What I was saying is that without error resiliance in VP9, the artifacts looks like random noizy images.
I noticed that in the answer sent by gstreamer, even though I’m using vah264lpenc, the answer contains vp8.
I also observed that the way in which do-nack is set is behaving differently.
// Sample pipeline
let pipeline = gstreamer::parse_launch(
"videotestsrc pattern=ball is-live=true ! vah264lpenc target-usage=1 ! rtph264pay name=vpay pt=96 ! webrtcbin. \
audiotestsrc is-live=true ! opusenc perfect-timestamp=true ! rtpopuspay name=apay pt=97 ! application/x-rtp,encoding-name=OPUS ! webrtcbin. \
webrtcbin name=webrtcbin latency=100"
let pipeline = pipeline.downcast::<gstreamer::Pipeline>().expect("Not a pipeline");
let webrtcbin = pipeline.by_name("webrtcbin").expect("Did not find webrtcbin");
// Method 1
// let transceivers = webrtcbin.emit_by_name::<gstreamer_webrtc::WebRTCRTPTransceiver>("get-transceivers", &[]);
// let transceiver = webrtcbin.emit_by_name::<gstreamer_webrtc::WebRTCRTPTransceiver>("get-transceiver", &[&0i32]);
// transceiver.set_property("do-nack", true);
// Method 2
webrtcbin.connect("on-new-transceiver", false, move |values| {
let transceiver = values[1].get::<gstreamer_webrtc::WebRTCRTPTransceiver>().expect("Could not get it");
println!("New transceiver {:?}", transceiver);
transceiver.set_property("do-nack", true);
None
});
Here are the answers along with behaviour I’m observing:
VP8 using Method 2: No packet loss / artifacting. But ideally want to avoid to use a better encoder and a HW accelerated one.
If anybody is struggling with this, I figured out how to set do-nack and fec-type by referring to the webrtcsrcimplementation.
But I’m still facing packet loss.
Despite setting do-nack to true, I can see in the stats that there are no nacks sent or received