Dynamically add/remove live streaming while start/stop recording on demand

General Discussion
1

Hi Experts,

We have a cam video source and a mic audio source. The pipeline has two branches, one for recording by Ridgerun prerecrod element but recording to file only on motion detection. The other branch is the live streaming (Webrtc) upon user’s requests. So the live streaming can start/stop at any time so as the filesink.

We have encountered a problem which blocks the video source (not calling the fill function any more) on certain case.

The file recording runs fine all the time regardless the live streaming is on or not.

The live streaming is working on its own without problem, i.e. if there is no motion detected, it can start/stop many times OK.

But as long as the live is started once, even it is stopped, if there is a motion detected (save to file regardless it is finished or not), next time starting the live streaming again will cause the video src to stall (after building the webrtc branch), i.e. not calling the overrided fill function any more. If no motion is ever detected, it is OK.

The simplified pipeline (omitted the audio tee):

mycamsrc ! h264parse ! tee name=t ! queue ! prerecord ! mux ! filesink

When live is connected it becomes:

mycamsrc ! h264parse ! tee name=t ! queue ! prerecord ! mux ! filesink t. ! queue ! rtph264pay ! webrtcbin

When adding/removing the live webrtc branch, we set a block probe on tee sink pad and release it after the branch is built and synced with parent.

We also set up a block probe on either the queue before the prerecord or on tee when the prerecord swithcing to debuffering /buffering, i.e. to save to file or stop saving. And release it after the switching.

The debug GST_DEBUG=basesrc:5,task:9 doesn’t print anything related to the video src once it happens. Sounds like the srcpad task being destroyed? We can’t see the state changed to either PAUSE or STOP.

Please help!

Many thanks

2

Here it part of the log when it goes wrong:

2025-04-29 14:40:42  Information [omletWebrtc]: on_connection - Entering on_connecttion...jim2 p2p test_token wss://media.omlet.com/OmletSmartCam/websocket 
2025-04-29 14:40:42  Information [omletWebrtc]: on_connection - Websocket connected.
2025-04-29 14:40:42  Information [omletWebrtc]: on_connection - Sending join command: {"command":"join","streamId":"jim2","multiPeer":false,"mode":"both"}.
2025-04-29 14:40:42  Information [omletWebrtc]: on_message - Received text data: {"streamId":"jim2","definition":"joined","command":"notification"}
2025-04-29 14:40:42  Information [omletWebrtc]: on_message - Received text data: {"streamId":"jim2","command":"start"}
2025-04-29 14:40:42  Information [omletWebrtc]: create_webrtc - Creating webrtc bin with id jim2. Offer: 1 type: p2p uri: wss://media.omlet.com/OmletSmartCam/websocket.
2025-04-29 14:40:42      Warning [omletWebrtc]: audio_tee_srcpad_block_probe_cb - audio_q: 0 rtpopuspay: 0 not ready.
2025-04-29 14:40:42      Warning [omletWebrtc]: video_tee_sinkpad_block_probe_cb - video_q: 0 rtph264pay: 0 or webrtc: 0 not ready.
2025-04-29 14:40:42      Warning [omletWebrtc]: audio_tee_sinkpad_block_probe_cb - audio_q: 0 rtpopuspay: 2027118272 not ready.
2025-04-29 14:40:42      Warning [omletWebrtc]: video_tee_srcpad_block_probe_cb - Get state results: video_q_state: 1 rtph264pay_state: 1 webrtc_state: 1.
0:01:12.461331652  3420 0x787683a0 DEBUG                   task gsttask.c:487:gst_task_new: Created task 0x78d383a0
0:01:12.468579369  3420 0x787683a0 INFO                    task gsttask.c:516:gst_task_set_lock: setting stream lock 0x78d38b8c on task 0x78d383a0
0:01:12.469893984  3420 0x787683a0 DEBUG                   task gsttask.c:729:gst_task_set_state_unlocked:<task14> Changing task 0x78d383a0 to state 0
0:01:12.472497883  3420 0x787683a0 DEBUG                   task gsttask.c:487:gst_task_new: Created task 0x78d3a6d0
0:01:12.474796460  3420 0x787683a0 INFO                    task gsttask.c:516:gst_task_set_lock: setting stream lock 0x78d34ed4 on task 0x78d3a6d0
0:01:12.476110408  3420 0x787683a0 DEBUG                   task gsttask.c:729:gst_task_set_state_unlocked:<task15> Changing task 0x78d3a6d0 to state 0
0:01:12.477439023  3420 0x6d0091f8 DEBUG                   task gsttask.c:339:gst_task_func: Entering task 0x78d383a0, thread 0x6d0091f8
0:01:12.480379575  3420 0x6d0091f8 DEBUG                   task gsttask.c:301:gst_task_configure_name:<audio_queue_stream:src> Setting thread name to 'audio_queue_stre'
0:01:12.483675442  3420 0x6d030818 DEBUG                   task gsttask.c:339:gst_task_func: Entering task 0x78d3a6d0, thread 0x6d030818
0:01:12.484370081  3420 0x6d030818 DEBUG                   task gsttask.c:301:gst_task_configure_name:<video_queue_stream:src> Setting thread name to 'video_queue_stre'
2025-04-29 14:40:43      Warning [omletWebrtc]: audio_tee_sinkpad_block_probe_cb - all required audio elements are in playing state so audio block pad (sink) probe is removed.
2025-04-29 14:40:43      Warning [omletWebrtc]: video_tee_sinkpad_block_probe_cb - all required video elements are in playing state so video block pad (sink) probe is removed.
2025-04-29 14:40:43  Information [omletWebrtc]: create_webrtc - [[[ webrtc has been created. ]]]
WARNING: Buffer Queue is FULL...
WARNING: Buffer Queue is FULL...
......repeat forever.....
3

And basesrc:5 prints last seen with video_source

0:01:17.158439320  3524 0x7775bf00 DEBUG                basesrc gstbasesrc.c:2107:gst_base_src_default_event:<video_source> handle event reconfigure event: 0x77b23260, time 99:99:99.999999999, seq-num 788, (NULL)
0:01:17.160811894  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:3444:gst_base_src_negotiate_unlocked:<video_source> starting negotiation
0:01:17.164363755  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:1386:gst_base_src_default_query:<video_source> query caps returns 1
0:01:17.167008985  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:3374:gst_base_src_default_negotiate:<video_source> caps of src: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, profile=(string){ baseline, main, high }
0:01:17.169767544  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:3384:gst_base_src_default_negotiate:<video_source> caps of peer: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, profile=(string){ baseline, main, high }
2025-04-29 14:52:37  Information [omletWebrtc]: on_negotiation_needed - >>>>>>>>>>>>>> Negotiation needed event emitted. <<<<<<<<<<<<<<
2025-04-29 14:52:37  Information [omletWebrtc]: create_webrtc - [[[ webrtc has been created. ]]]
0:01:17.182609695  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:3395:gst_base_src_default_negotiate:<video_source> have caps: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, profile=(string){ baseline, main, high }
0:01:17.194701222  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:1050:gst_base_src_default_fixate:<video_source> using default caps fixate function
0:01:17.195505857  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:3403:gst_base_src_default_negotiate:<video_source> fixated to: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, profile=(string)baseline
0:01:17.196393823  3524 0x6c501df8 DEBUG                basesrc gstbasesrc.c:1003:gst_base_src_set_caps:<video_source> New caps equal to old ones: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, profile=(string)baseline
4

We have further debug info about the case and found that whenever there is a prerecord element switching, either to buffering (drop the buffers in the queue probe implemented by prerecord) or debuffering (allow the buffers to pass), will cause the next time video src (basesrc) allocation query event to hang (never come back to the basesrc). This happens after the live webrtc branch has been constructed. It looks like it’s locked or blocked somewhere but we don’t understand why. Below are more logs:

2025-04-30 08:33:11  Information [omletWebrtc]: on_negotiation_needed - >>>>>>>>>>>>>> Negotiation needed event emitted. <<<<<<<<<<<<<<
......
2025-04-30 08:33:11  Information [media]: on_pad_probe_handler - Pad query: caps, type: 43523, parent: video_source mode: 1,
0:03:09.027435987  4641 0x6cf015a0 DEBUG               GST_PADS gstpad.c:3542:gst_pad_query_default:<mux:sink_66> not forwarding 0x62a11f00 (caps) query
0:03:09.028304619  4641 0x6cf015a0 DEBUG               GST_PADS gstpad.c:4229:gst_pad_query:<mux:sink_66> sent query 0x62a11f00 (caps), result 1
......
0:03:09.040920793  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3737:probe_hook_marshal:<video_source:src> asked to pass item
0:03:09.041127451  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3963:do_probe_callbacks:<video_source:src> data is passed
0:03:09.041340776  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4333:gst_pad_peer_query:<video_source:src> peer query 0x78978338 (allocation)
0:03:09.041452772  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3963:do_probe_callbacks:<video_source:src> data is passed
2025-04-30 08:33:11  Information [media]: on_pad_probe_handler - Pad probe handler called. probe type: 1200. Task state: 0
......
2025-04-30 08:33:11  Information [media]: on_pad_probe_handler - Pad query: allocation, type: 35846, parent: video_source mode: 1,
......
0:03:09.066673774  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3737:probe_hook_marshal:<video_source:src> asked to pass item
0:03:09.067207753  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3963:do_probe_callbacks:<video_source:src> data is passed
0:03:09.067294416  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4206:gst_pad_query:<capsfilter0:sink> doing query 0x78978338 (allocation)
0:03:09.067351747  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:1683:gst_pad_check_reconfigure:<capsfilter0:src> remove RECONFIGURE flag
0:03:09.067515074  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4206:gst_pad_query:<h264_parse:sink> doing query 0x78978620 (accept-caps)
0:03:09.067715065  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3228:gst_pad_query_accept_caps_default:<h264_parse:sink> query accept-caps accept-caps query: 0x78978620, GstQueryAcceptCaps, caps=(GstCaps)"video/x-h264\,\ stream-format\=\(string\)byte-stream\,\ alignment\=\(string\)au\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)25/1\,\ profile\=\(string\)baseline", result=(boolean)false;
0:03:09.068262377  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3253:gst_pad_query_accept_caps_default:<h264_parse:sink> allowed caps intersect video/x-h264, caps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, profile=(string)baseline
0:03:09.068414370  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3542:gst_pad_query_default:<h264_parse:sink> not forwarding 0x78978620 (accept-caps) query
0:03:09.068484368  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4229:gst_pad_query:<h264_parse:sink> sent query 0x78978620 (accept-caps), result 1
0:03:09.068608363  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4333:gst_pad_peer_query:<capsfilter0:src> peer query 0x78978338 (allocation)
0:03:09.068677026  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4206:gst_pad_query:<h264_parse:sink> doing query 0x78978338 (allocation)
0:03:09.068742357  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:3542:gst_pad_query_default:<h264_parse:sink> forwarding 0x78978338 (allocation) query
0:03:09.068789689  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:2996:gst_pad_iterate_internal_links_default:<h264_parse:sink> Making iterator
0:03:09.068861019  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4333:gst_pad_peer_query:<h264_parse:src> peer query 0x78978338 (allocation)
0:03:09.068923016  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4206:gst_pad_query:<video_tee:sink> doing query 0x78978338 (allocation)
0:03:09.069020346  4641 0x6cf013c8 DEBUG               GST_PADS gstpad.c:4206:gst_pad_query:<video_queue:sink> doing query 0x78979ff0 (allocation)
0:03:09.069213005  4641 0x6cf00d48 DEBUG               GST_PADS gstpad.c:4333:gst_pad_peer_query:<video_queue:src> peer query 0x78979ff0 (allocation)
0:03:09.069311667  4641 0x6cf00d48 DEBUG               GST_PADS gstpad.c:4206:gst_pad_query:<prerecord:sink0> doing query 0x78979ff0 (allocation)
0:03:09.069380998  4641 0x6cf00d48 DEBUG               GST_PADS gstpad.c:3542:gst_pad_query_default:<prerecord:sink0> forwarding 0x78979ff0 (allocation) query
0:03:09.069460328  4641 0x6cf00d48 DEBUG               GST_PADS gstpad.c:4333:gst_pad_peer_query:<sink0:proxypad0> peer query 0x78979ff0 (allocation)
0:03:09.069524325  4641 0x6cf00d48 DEBUG               GST_PADS gstpad.c:4206:gst_pad_query:<queue0:sink> doing query 0x78979ff0 (allocation)
......
WARNING: Buffer Queue is FULL...
......
5

Root Cause Analysis

Prerecord Mode Switches:

    When prerecord switches between buffering/debuffering, it manipulates buffer flow via pad probes, which can interfere with ongoing allocation queries.

    The logs show prerecord’s downstream queue (queue0) becoming full (WARNING: Buffer Queue is FULL...), blocking upstream elements and stalling allocation queries.

WebRTC Branch Activation:

    Adding the webrtcbin branch alters the pipeline’s buffer requirements (caps negotiation). If the prerecord branch is in an inconsistent state during this, allocation queries may deadlock.

Allocation Query Propagation:

    The allocation query from mycamsrc propagates downstream but gets stuck at prerecord’s queue due to:

        Blocked pads (probes not released)

        Full queues

Implement these fixes in order:

1. Fix Prerecord Queue Configuration

Configure queues in the prerecord branch to prevent blocking:

c

// Set leaky=downstream and reasonable buffer limits g_object_set(queue0, “leaky”, 2 /downstream/, “max-size-buffers”, 200, NULL);

This ensures buffers are dropped (not queued indefinitely) when downstream elements (e.g., prerecord) are inactive.

2. Reset Prerecord on Mode Switches

When switching prerecord modes, flush buffers to reset internal states:

c

// When stopping recording: gst_element_send_event(prerecord, gst_event_new_flush_start()); gst_element_send_event(prerecord, gst_event_new_flush_stop(TRUE)); // When restarting recording: gst_element_send_event(prerecord, gst_event_new_reconfigure());

This clears stale buffers and unblocks allocation queries.

Got these online. Not sure if they give you some hints. Could be craps as well.

6

Hi Joe,

Thank you so much for your excellent analysis and suggestions. That makes a lot sense!

We did try setting the leaky property to downstream but that didn’t seem to fix. We also tried setting the async-handling to true at the same time and that didn’t work either.

The second suggestion sounds very promising. We will try shortly.

Many thanks!

Jim

7

Hmm.. the above mentioned two approaches do not seem to work. The second one locks the video src (seeing Queue FULL prints) immediately after sending the event in either case. Any idea?

8

By the way, to clarify that the

WARNING: Buffer Queue is FULL...

message is printed by our video src internal frame grab thread. This thread gets the frames and pushes into a glib queue. The overrided “fill” function pops the queue and copies the data into gstbuffer then fed to downstream.

So when it is locked, it is this “fill” function never gets called.