How to restart pipeline programmatically

Application Development
1

Hi, I have a Deepstream application that is monitoring multiple RTSP streams. Sometimes these cameras reset, but they don’t send EOS signal, so my application doesn’t stop. However, when cameras get back online, Deepstream attempts restart, but fails and doesn’t start inference (FPS is always 0).
The only solution that works so far is to restart entire application, so my question is, is there a way to restart inference programmatically?

I already asked this question on Nvidia forum, but they pointed me to Gstreamer forum.

Logs:

2024-05-04 08:24:32.344 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 29.99, '00-07-5f-c8-7e-84': 29.99}***
2024-05-04 08:24:37.346 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 16.6, '00-07-5f-c8-7e-84': 16.6}***
2024-05-04 08:24:42.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:24:45.599 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 10 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:24:45.600 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:24:47.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.4}***
2024-05-04 08:24:48.539 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 13 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:24:48.540 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:24:50.194 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:24:50.196 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:24:51.698 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:24:51.700 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:24:52.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:24:57.216 | ERROR    | utils.heartbeat_util:send_fps_heartbeat:86 - fps for stream 00-07-5f-e6-cb-5e is 0!
2024-05-04 08:24:57.220 | ERROR    | utils.heartbeat_util:send_fps_heartbeat:86 - fps for stream 00-07-5f-c8-7e-84 is 0!
2024-05-04 08:24:57.349 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:25:02.349 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:25:07.350 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.4}***
2024-05-04 08:25:12.350 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:17.351 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:22.351 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:27.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:32.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:37.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:42.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:47.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:47.987 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:25:47.988 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:25:48.978 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:25:48.979 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:25:52.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:57.354 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:02.355 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:05.311 | ERROR    | save_img:_reader:32 - Terminate camera thread is set!
2024-05-04 08:26:07.355 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:12.356 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:17.357 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:22.358 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:27.358 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:32.359 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:37.359 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:42.360 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:45.703 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: h264parse2
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.GstH264Parse object at 0xffff97b636c0 (GstH264Parse at 0xfffe10085260)>
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:45.705 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: capsfilter2
2024-05-04 08:26:45.706 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.706 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.GstCapsFilter object at 0xffff97b636c0 (GstCapsFilter at 0xfffe1c1af220)>
2024-05-04 08:26:45.707 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:45.711 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: nvv4l2decoder2
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.nvv4l2decoder object at 0xffff97b636c0 (nvv4l2decoder at 0xfffe100a3a30)>
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:47.360 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:52.361 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:57.362 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:27:02.362 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.4, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:28:58.001 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 10 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:28:58.002 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:28:58.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.25, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:03.001 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:08.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:13.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:18.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.4, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:23.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:28.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:33.004 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:38.004 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:43.005 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2

Hi
you can set the pipeline to null → play. it will restart pipeline.

3

Hi,

I added this code:

pipeline.set_state(Gst.State.NULL)
pipeline.set_state(Gst.State.PLAYING)

However I get this error:

(python:86219): GStreamer-CRITICAL **: 07:16:22.835: gst_element_get_static_pad: assertion 'GST_IS_ELEMENT (element)' failed

(python:86219): GStreamer-CRITICAL **: 07:16:22.835: gst_pad_link_full: assertion 'GST_IS_PAD (sinkpad)' failed
utils.bus_call:bus_call:33 - Error: gst-stream-error-quark: Failed to populate and link video elements (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1706): cb_newpad_rtspsrc (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
utils.probe_util:on_error:503 - ------> uri-decode-bin: err: gst-stream-error-quark: Failed to populate and link video elements (1), debug message: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1706): cb_newpad_rtspsrc (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin

I have very little experience with Gstreamer so I don’t know if I’m missing something in my code.

4

Can share the pipeline and some more details.

5

I cannot share you my application, but it is built from Nvidia’s deepstream-test3 example

The only difference in the pipeline is that I don’t have nvmultistreamtiler, but I do have secondary nvinfer and nvtracker

Right now, I have scheduler that checks FPS, and if FPS is 0 it sets pipeline to null and then to playing state.

6

So in my application I used Gst-nvurisrcbin, but once I switched to uridecodebin and using this code my application can restart successfully when using one stream:

def restart_pipeline(pipeline):
    pipeline.set_state(Gst.State.NULL)
    pipeline.set_state(Gst.State.READY)
    pipeline.set_state(Gst.State.PLAYING)

However as soon as I include multiple sources, sometimes after the restart FPS will be 0 and on other times I’ll get this warning:

Source info not found for source 1. Maybe the GST_NVEVENT_PAD_ADDED event was never generated for the source.

7

When cameras are reset or network is done, there is no EOS message. You may need something like ONVIF to detect if a camera is available or not in the network. If a camera disappears, stop the pipeline. If a camera shows up, start the pipeline.

8

After much trial and error I managed to restart pipeline.
My solution is to set pipeline to NULL and then create a new one. That way I can continue to use Gst-nvurisrcbin element.

def main():
    create_pipeline()

    loop = GLib.MainLoop()
    loop.run()

def restart_pipeline(pipeline):
    pipeline.set_state(Gst.State.NULL)
    time.sleep(2)
    logger.error("Recreate pipeline")
    create_pipeline()

def create_pipeline():
    '''
        add all elements here
        and set pipeline to PLAYING state
    '''