How to restart pipeline programmatically

Hi, I have a Deepstream application that is monitoring multiple RTSP streams. Sometimes these cameras reset, but they don’t send EOS signal, so my application doesn’t stop. However, when cameras get back online, Deepstream attempts restart, but fails and doesn’t start inference (FPS is always 0).
The only solution that works so far is to restart entire application, so my question is, is there a way to restart inference programmatically?

I already asked this question on Nvidia forum, but they pointed me to Gstreamer forum.

Logs:

2024-05-04 08:24:32.344 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 29.99, '00-07-5f-c8-7e-84': 29.99}***
2024-05-04 08:24:37.346 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 16.6, '00-07-5f-c8-7e-84': 16.6}***
2024-05-04 08:24:42.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:24:45.599 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 10 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:24:45.600 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:24:47.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.4}***
2024-05-04 08:24:48.539 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 13 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:24:48.540 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:24:50.194 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:24:50.196 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:24:51.698 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:24:51.700 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:24:52.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:24:57.216 | ERROR    | utils.heartbeat_util:send_fps_heartbeat:86 - fps for stream 00-07-5f-e6-cb-5e is 0!
2024-05-04 08:24:57.220 | ERROR    | utils.heartbeat_util:send_fps_heartbeat:86 - fps for stream 00-07-5f-c8-7e-84 is 0!
2024-05-04 08:24:57.349 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:25:02.349 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:25:07.350 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.4}***
2024-05-04 08:25:12.350 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:17.351 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:22.351 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:27.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:32.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:37.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:42.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:47.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:47.987 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:25:47.988 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:25:48.978 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:25:48.979 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:25:52.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:57.354 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:02.355 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:05.311 | ERROR    | save_img:_reader:32 - Terminate camera thread is set!
2024-05-04 08:26:07.355 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:12.356 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:17.357 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:22.358 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:27.358 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:32.359 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:37.359 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:42.360 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:45.703 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: h264parse2
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.GstH264Parse object at 0xffff97b636c0 (GstH264Parse at 0xfffe10085260)>
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:45.705 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: capsfilter2
2024-05-04 08:26:45.706 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.706 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.GstCapsFilter object at 0xffff97b636c0 (GstCapsFilter at 0xfffe1c1af220)>
2024-05-04 08:26:45.707 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:45.711 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: nvv4l2decoder2
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.nvv4l2decoder object at 0xffff97b636c0 (nvv4l2decoder at 0xfffe100a3a30)>
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:47.360 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:52.361 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:57.362 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:27:02.362 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.4, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:28:58.001 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 10 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:28:58.002 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:28:58.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.25, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:03.001 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:08.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:13.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:18.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.4, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:23.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:28.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:33.004 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:38.004 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:43.005 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***

Hi
you can set the pipeline to null → play. it will restart pipeline.

Hi,

I added this code:

pipeline.set_state(Gst.State.NULL)
pipeline.set_state(Gst.State.PLAYING)

However I get this error:

(python:86219): GStreamer-CRITICAL **: 07:16:22.835: gst_element_get_static_pad: assertion 'GST_IS_ELEMENT (element)' failed

(python:86219): GStreamer-CRITICAL **: 07:16:22.835: gst_pad_link_full: assertion 'GST_IS_PAD (sinkpad)' failed
utils.bus_call:bus_call:33 - Error: gst-stream-error-quark: Failed to populate and link video elements (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1706): cb_newpad_rtspsrc (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
utils.probe_util:on_error:503 - ------> uri-decode-bin: err: gst-stream-error-quark: Failed to populate and link video elements (1), debug message: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1706): cb_newpad_rtspsrc (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin

I have very little experience with Gstreamer so I don’t know if I’m missing something in my code.

Can share the pipeline and some more details.

I cannot share you my application, but it is built from Nvidia’s deepstream-test3 example

The only difference in the pipeline is that I don’t have nvmultistreamtiler, but I do have secondary nvinfer and nvtracker

Right now, I have scheduler that checks FPS, and if FPS is 0 it sets pipeline to null and then to playing state.