Auto reconnect in whipeclientsink

I use the following to stream video and audio over an unstable mobile connection:

gst-launch-1.0 whipclientsink signaller::whip-endpoint="http://xxxx:xxx@xxx:8889/live/fly/whip" name=ws video-caps="video/x-vp8"  \
  v4l2src device=${VIDEO} ! tee name=v \
  v. ! queue leaky=1 ! autovideosink sync=false \
  v. ! queue ! ws.  \
  pulsesrc device=${MIC_AUDIO} ! queue ! gareus-org-oss-lv2-nodelay-mega delay=11000 ! audiomixer name=a \
  pulsesrc device=${HDMI_AUDIO} ! queue ! a. \
  a. ! ws.

The server is mediamtx.

This commands stream to the server and display a local monitor window with video.

I have two issues:

  • The quality is adapted to the connection, BUT, if the connection drops, it does not restart. I have to kill and relaunch the command.
  • Sometimes, the sound cracks for about 30 seconds

Hmm, I’m not sure what the expectation from the spec is on ICE connection failure, but it seems like it might make sense for the application to explicitly manage restarting the pipeline, rather than having the stream automatically restart?

For sound cracking, I might be interesting to stick in a tee after the audiomixer and dump the audio locally, make sure the send side hasn’t seen any issues. For example, if your system is under load, it could be that either the source or your filter is not able to provide data quickly enough for a short period, and that could manifest as dropped buffers?

I will try to tee but it seems to be related to the bandwidth management algorithm.

You can see it here https://www.kuon.ch/twitch/vods/2024-06-10-15-36-18-vol/ at 1:21:35 exactly. When I move the camera, I think the congestion algorithm kicks in, and we can see some frames are dropped and the sound start cracking, and after a few seconds it gets back to normal.

About the reconnection, I would expect gstreamer to handle it, as the congestion control algorithm is the source of the current bandwidth available, but if it is out of the scope of gstreamer, I can write a small script that monitor network interface datarate, and restart when it hit zero for a few seconds.

I notice similar glitches around 1:18:14 (entirely by accident!) that seem unrelated to the video, assuming the chat overlay is part of the video, but I haven’t looked too closely.

@taruntej was working on some ICE consent freshness feedback that might tie into restarting ICE, maybe that helps here (I’ll let him weigh in on that).

I have verified the behavior on my system with whipserversrc and there it is able to reconnect properly once the network is restored.

@kuon do you have the logs of the whipclient/webrtcsink somewhere so I can take a look?

It is hard to reproduce the real conditions as I had those issues while doing a mobile stream.

I am trying to reproduce this in my office, I will let you know and share the logs.