Why do I need to pause pipeline when adding branch to tee?

landersson · December 11, 2024, 12:06am

Hi, I’m playing around with some python code to dynamically add various types of branches to a tee in a running gstreamer pipeline. I want to toggle recording to mp4 files on and off by dynamically adding/removing recording branches, toggle local visualization on/off by adding/removing videosink branches, add a branch to stream video to web browser via a webrtc sink etc. The code is available here:

https://github.com/landersson/gstreamer-experiments/blob/master/dynamic_record.py

I’ve googled around and read as much as I’ve been able to find about the proper way to dynamically manipulate a PLAYING gstreamer pipeline. This seem to work fairly well, but I still run into issues unless I temporarily put the whole pipeline into the PAUSED state when linking the tee src pad to the branch sink pad.

github.com

landersson/gstreamer-experiments/blob/eb8ad6dffa6ae2d7f39ec597c33902570b741e16/dynamic_record.py#L251


      
          
              # Add new tee pad to tee element
              tee_src_pad_template = self.tee.get_pad_template(tee_src_pad_name)
              tee_pad = self.tee.request_pad(tee_src_pad_template, None, None)
          
              # ???: In certain cases, we need to set the pipeline to paused before
              # linking a branch to the tee and then set it back to playing. For
              # example, this is needed when adding a recording branch after having
              # added _and_ removed a display branch. Why?
          
              # self.pipeline.set_state(Gst.State.PAUSED)
              tee_pad.link(branch.queue_sink_pad())
              # self.pipeline.set_state(Gst.State.PLAYING)
              # ???: In general, do we need to use a blocking probe to add a branch to
              # a tee? Assuming no above.
          
              # tee_pad.add_probe(Gst.PadProbeType.IDLE, self.link_tee_pad_cb, branch)
              return tee_pad
          
          def object_name_short(self, object_name):
              return str(object_name).split("(")[1].split(")")[0]

For example, when I first add/remove an xvimagesink display branch and then try to add a mp4 recording branch, I get the following errors and the pipeline fails:

qtmux gstqtmux.c:5801:gst_qt_mux_can_renegotiate:<muxer> pad video_0 refused renegotiation to video/x-h264 ....... from video/x-h264 .......
basesrc gstbasesrc.c:3127:gst_base_src_loop:<src> error: Internal data stream error.
queue gstqueue.c:992:gst_queue_handle_sink_event:<recording_queue> error: streaming stopped, reason not-negotiated (-4)

When just trying to add a recording branch (scenario_2 in the code above), this works, but I get warnings like this when adding the branch:

WARN           GST_CAPS gstpad.c:3235:gst_pad_query_accept_caps_default:<timeoverlay:video_sink> caps: video/x-raw, width=(int)1024, height=(int)768, format=(string)ABGR64_LE, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive were not compatible with: EMPTY
WARN           basetransform gstbasetransform.c:1370:gst_base_transform_setcaps:<capsfilter> transform could not transform video/x-raw, format=(string)ABGR64_LE, width=(int)1024, height=(int)768, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive in anything we support
WARN           basetransform gstbasetransform.c:1431:gst_base_transform_reconfigure_unlocked:<capsfilter> warning: not negotiated
WARN           basetransform gstbasetransform.c:1431:gst_base_transform_reconfigure_unlocked:<capsfilter> warning: not negotiated

If I temporarily pause the pipeline when adding the recording branch and then resume it straight after (lines 251 and 253 in the code above), it works fine. I’ve seen other examples of adding/removing recording tees to a running pipeline though, but somehow I can’t get my particular scenario to work. Perhaps pausing temporarily is ok in my case, but I’m curious about what I might be doing wrong… what would need to be changed to add branches to the tee without pausing? Any suggestions?