Freezes after adding WebRTC sender to already playing pipeline with v4l2src source

General Discussion
1

Hi!

Creating multiclient player here. The idea is to capture audio/video stream from one source and send to many (for now - webRTC) clients at once. Clients can connect and disconnect at any time.

Organized the system as follows:

  • source (for now - V4l2src) and sender (webrtcbin + required tools) are built using textual pipeline description with gst_parse_launch()
  • to connect source and sender i use proxysink/proxysrc pairs for video and audio

Source scheme looks as follows:

 v4l2src do-timestamp=true ! video/x-raw ! tee name=vtee
 autoaudiosrc ! audio/x-raw ! tee name=atee

Senders have following structure:

 proxysrc name=vsink
 ! ...

 proxysrc name=asink
 ! ...
  • when adding sender to source i add one more pad to tee, create and connect proxysink element to the pad, then connect proxysink and proxysrc elements.
    For example for video: I request pad from vtee element, create and add to the pipeline proxysink named “vsrc”, then connect “vsrc” and “vsink”.

WebRTC sender has following pipeline description:

webrtcbin name=webrtcbin stun-server=stun://stun.l.google.com:19302
 proxysrc name=vsink
 ! queue
 ! videoconvert
 ! video/x-raw,format=I420
 ! x264enc bitrate=600 speed-preset=ultrafast tune=zerolatency key-int-max=15
 ! video/x-h264
 ! rtph264pay config-interval=-1 aggregate-mode=zero-latency
 ! application/x-rtp,media=video,encoding-name=H264,payload=96 ! webrtcbin.

 proxysrc name=asink
 ! queue 
 ! audioconvert
 ! audioresample
 ! opusenc perfect-timestamp=true ! rtpopuspay pt=97
 ! application/x-rtp, encoding-name=OPUS
 ! webrtcbin.

(pretty standard as far as i can tell)

As result two pipelines start to work and everything is OK.

(Needless to mention that I use special WebRTC client to communicate with webrtcbin. )

Now, when I add second sender I have following issues:

  • long time for the system (all pipelines) to configure, video is stuck on the first sender (up to a minute)
  • video/audio “freezes” similar to not having enough CPU resources, however “top” utility does not show this issue

More detailed description.

  1. After creating and connecting new pipeline with WebRTC sender everything “freezes” (for up to minute) and then continues without errors or warnings. V4l2src logs about lost frames. I was unable to pinpoint exact element which performs active “something” and blocks all the pipelines.

  2. when the system has finished “something” and both senders are sending streams i observe following situation: data is passed for 3-4 seconds and then everything freezes for 8 seconds (using timestamps from “buffer” probes i put into pads in some places). I’m not sure why numbers are like this. I do not have any timer callbacks set up in gstreamer which could block the system.

The same issue is observed when I use “decklinksrc” plugins as data source.
On the other hand if I use as source playbin with a file everything works well.

playbin uri=file://[%FILEPATH%] name=pb
 tee name=vtee
 tee name=atee

Also if I modify sender to use fakesink instead of webrtcbin I also get nicely working system: second/third “client” connects in a seconds, no lags or freezes is observed.

What I have tried and done:

  • when creating second/next sender i copy already negotiated caps from already existing pads into newly created ones. The idea here - to reduce negotiating procedure. Didn’t help as far as i can tell.
  • linking pads with flag “no reconfiguration” - didn’t help. I see reconfiguration queries with my probes.
  • pause/resume source pipeline around appending new sender - didn’t help.
  • using queue elements between tee and proxysink - didn’t help.
  • using logging - unable to pinpoint exact element and/or activity which occurs when the system freezes.
  • using pad probes: no events/queries when streams are active, no heavy exchange when the system performs initial reconfiguration after connecting second/next sender. “Buffer” probes show no activity in these periods.
  • Playbin/synthetic data sources + webrtcbin work well
  • v4l2src/decklink plugins with webrtcbin - have described issue
  • v4l2src with fakesink senders - work well.

I’ve searched for solutions. Nothing has helped as for now (for example, about blocking nature of x264encoder and queues).

Now, I need some support if possible.
Possible directions:

  • better debugging techniques to find exact root of the issue
  • making sure reconfiguration procedure does not happen - some way for “fixing” caps when I have completed first negoitiation.
  • using playbin with sources like v4l2src/decklink plugins (if the element can do some magic, let it do when i need it)
  • … ?

Thanks