Freezes after adding WebRTC sender to already playing pipeline with v4l2src source

General Discussion
1

Hi!

Creating multiclient player here. The idea is to capture audio/video stream from one source and send to many (for now - webRTC) clients at once. Clients can connect and disconnect at any time.

Organized the system as follows:

  • source (for now - V4l2src) and sender (webrtcbin + required tools) are built using textual pipeline description with gst_parse_launch()
  • to connect source and sender i use proxysink/proxysrc pairs for video and audio

Source scheme looks as follows:

 v4l2src do-timestamp=true ! video/x-raw ! tee name=vtee
 autoaudiosrc ! audio/x-raw ! tee name=atee

Senders have following structure:

 proxysrc name=vsink
 ! ...

 proxysrc name=asink
 ! ...
  • when adding sender to source i add one more pad to tee, create and connect proxysink element to the pad, then connect proxysink and proxysrc elements.
    For example for video: I request pad from vtee element, create and add to the pipeline proxysink named “vsrc”, then connect “vsrc” and “vsink”.

WebRTC sender has following pipeline description:

webrtcbin name=webrtcbin stun-server=stun://stun.l.google.com:19302
 proxysrc name=vsink
 ! queue
 ! videoconvert
 ! video/x-raw,format=I420
 ! x264enc bitrate=600 speed-preset=ultrafast tune=zerolatency key-int-max=15
 ! video/x-h264
 ! rtph264pay config-interval=-1 aggregate-mode=zero-latency
 ! application/x-rtp,media=video,encoding-name=H264,payload=96 ! webrtcbin.

 proxysrc name=asink
 ! queue 
 ! audioconvert
 ! audioresample
 ! opusenc perfect-timestamp=true ! rtpopuspay pt=97
 ! application/x-rtp, encoding-name=OPUS
 ! webrtcbin.

(pretty standard as far as i can tell)

As result two pipelines start to work and everything is OK.

(Needless to mention that I use special WebRTC client to communicate with webrtcbin. )

Now, when I add second sender I have following issues:

  • long time for the system (all pipelines) to configure, video is stuck on the first sender (up to a minute)
  • video/audio “freezes” similar to not having enough CPU resources, however “top” utility does not show this issue

More detailed description.

  1. After creating and connecting new pipeline with WebRTC sender everything “freezes” (for up to minute) and then continues without errors or warnings. V4l2src logs about lost frames. I was unable to pinpoint exact element which performs active “something” and blocks all the pipelines.

  2. when the system has finished “something” and both senders are sending streams i observe following situation: data is passed for 3-4 seconds and then everything freezes for 8 seconds (using timestamps from “buffer” probes i put into pads in some places). I’m not sure why numbers are like this. I do not have any timer callbacks set up in gstreamer which could block the system.

The same issue is observed when I use “decklinksrc” plugins as data source.
On the other hand if I use as source playbin with a file everything works well.

playbin uri=file://[%FILEPATH%] name=pb
 tee name=vtee
 tee name=atee

Also if I modify sender to use fakesink instead of webrtcbin I also get nicely working system: second/third “client” connects in a seconds, no lags or freezes is observed.

What I have tried and done:

  • when creating second/next sender i copy already negotiated caps from already existing pads into newly created ones. The idea here - to reduce negotiating procedure. Didn’t help as far as i can tell.
  • linking pads with flag “no reconfiguration” - didn’t help. I see reconfiguration queries with my probes.
  • pause/resume source pipeline around appending new sender - didn’t help.
  • using queue elements between tee and proxysink - didn’t help.
  • using logging - unable to pinpoint exact element and/or activity which occurs when the system freezes.
  • using pad probes: no events/queries when streams are active, no heavy exchange when the system performs initial reconfiguration after connecting second/next sender. “Buffer” probes show no activity in these periods.
  • Playbin/synthetic data sources + webrtcbin work well
  • v4l2src/decklink plugins with webrtcbin - have described issue
  • v4l2src with fakesink senders - work well.

I’ve searched for solutions. Nothing has helped as for now (for example, about blocking nature of x264encoder and queues).

Now, I need some support if possible.
Possible directions:

  • better debugging techniques to find exact root of the issue
  • making sure reconfiguration procedure does not happen - some way for “fixing” caps when I have completed first negoitiation.
  • using playbin with sources like v4l2src/decklink plugins (if the element can do some magic, let it do when i need it)
  • … ?

Thanks

2

The problem solved.

To whomever it may concern below is the description of the issue.

webrtcbin creates elements clocksync which make sure frames re aligned with time. These elements internally use running time and base time:

  • running time - this is time since start of playing. Value is reported for each buffer (havent checked, it’s just my guess)
  • base time - this is time when the pipeline was started (switched to state PLAYING)

In short words base time + running time - current time = delay. If a buffer comes “too early” clocksync will sleep until appropriate time.

Now, as i have separate pipelines with webrtc they have non synchronised base time, and the same running time. And when buffer arrived “in time” first created clocksync detects it and passes the buffer further. However, clocksync in a “second” pipeline will have time of buffer “in the future” (for the difference in base time’s between pipelines) and sleeps for as many seconds as have passed before second pipeline was created. This makes the whole chain to the buffer source to collect buffers and at some point to just halt until this pause is over.

That was the root of the issue.

I have fixed the issue by manually setting base time in each second pipeline to value read from the first pipeline.

The case closed.