I am still new with GStreamer and I am trying to add forward error correction to my pipeline because I have a lot of packet loss on my radio link. In my setup, I run a ROS2 application that starts an RTSP Server and adds a GStreamer pipeline. Then, I watch the video stream from a second computer which is connected via an IP radio.
The first example is working and I can open an RTSP client to watch the stream:
appsrc name=imagesrc is-live=true ! videoconvert ! x264enc ! rtph264pay name=pay0 pt=96
The second example with additional FEC results in a crash of the RTSP Server, the moment I connect a client:
appsrc name=imagesrc is-live=true ! videoconvert ! x264enc ! rtph264pay name=pay0 pt=96 ! rtpulpfecenc percentage=100 pt=122
Below you can find logs of the Server with GST_DEBUG=4. In my understanding, the linking of the error-corrected elements is where the pipeline falls apart. Do you have any ideas for solving this issue?
[rtp_server-1] [INFO] [1715253974.288815968] [rclcpp]: Initializing RTSP server
[rtp_server-1] 0:00:00.000146368 47513 0xaaab14508c00 INFO GST_INIT gst.c:586:init_pre: Initializing GStreamer Core Library version 1.16.3
[rtp_server-1] 0:00:00.000205408 47513 0xaaab14508c00 INFO GST_INIT gst.c:587:init_pre: Using library installed in /usr/lib/aarch64-linux-gnu
[rtp_server-1] 0:00:00.000223136 47513 0xaaab14508c00 INFO GST_INIT gst.c:605:init_pre: Linux AGX-ORIN-22 5.10.104-tegra #1 SMP PREEMPT Sun Mar 19 07:55:28 PDT 2023 aarch64
[rtp_server-1] 0:00:00.000603392 47513 0xaaab14508c00 INFO GST_INIT gstmessage.c:128:_priv_gst_message_initialize: init messages
[rtp_server-1] 0:00:00.001091072 47513 0xaaab14508c00 INFO GST_INIT gstcontext.c:84:_priv_gst_context_initialize: init contexts
[rtp_server-1] 0:00:00.001301056 47513 0xaaab14508c00 INFO GST_PLUGIN_LOADING gstplugin.c:318:_priv_gst_plugin_initialize: registering 0 static plugins
[rtp_server-1] 0:00:00.001427104 47513 0xaaab14508c00 INFO GST_PLUGIN_LOADING gstplugin.c:226:gst_plugin_register_static: registered static plugin "staticelements"
[rtp_server-1] 0:00:00.001439232 47513 0xaaab14508c00 INFO GST_PLUGIN_LOADING gstplugin.c:228:gst_plugin_register_static: added static plugin "staticelements", result: 1
[rtp_server-1] 0:00:00.001467520 47513 0xaaab14508c00 INFO GST_REGISTRY gstregistry.c:1733:ensure_current_registry: reading registry cache: /root/.cache/gstreamer-1.0/registry.aarch64.bin
[rtp_server-1] 0:00:00.019275008 47513 0xaaab14508c00 INFO GST_REGISTRY gstregistrybinary.c:621:priv_gst_registry_binary_read_cache: loaded /root/.cache/gstreamer-1.0/registry.aarch64.bin in 0.017764 seconds
[rtp_server-1] 0:00:00.019428576 47513 0xaaab14508c00 INFO GST_REGISTRY gstregistry.c:1592:scan_and_update_registry: Validating plugins from registry cache: /root/.cache/gstreamer-1.0/registry.aarch64.bin
[rtp_server-1] 0:00:00.022622432 47513 0xaaab14508c00 INFO GST_REGISTRY gstregistry.c:1691:scan_and_update_registry: Registry cache has not changed
[rtp_server-1] 0:00:00.022656960 47513 0xaaab14508c00 INFO GST_REGISTRY gstregistry.c:1768:ensure_current_registry: registry reading and updating done, result = 1
[rtp_server-1] 0:00:00.022668288 47513 0xaaab14508c00 INFO GST_INIT gst.c:806:init_post: GLib runtime version: 2.64.6
[rtp_server-1] 0:00:00.022677024 47513 0xaaab14508c00 INFO GST_INIT gst.c:808:init_post: GLib headers version: 2.64.6
[rtp_server-1] 0:00:00.022681152 47513 0xaaab14508c00 INFO GST_INIT gst.c:810:init_post: initialized GStreamer successfully
[rtp_server-1] [INFO] [1715253974.331270048] [robot_1_22.RtspServerNode]: Mountpoint: /cam, Pipeline: appsrc name=imagesrc is-live=true ! videoconvert ! x264enc ! rtph264pay name=pay0 pt=96 ! rtpulpfecenc percentage=100 pt=122
[rtp_server-1] 0:00:00.024011872 47513 0xaaab14508c00 INFO rtspmountpoints rtsp-mount-points.c:333:gst_rtsp_mount_points_add_factory: adding media factory 0xaaab14610960 for path /cam
[rtp_server-1] [INFO] [1715253974.331476864] [robot_1_22.RtspServerNode]: Mountpoint: /night, Pipeline: appsrc name=imagesrc is-live=true ! videoconvert ! x264enc ! rtph264pay name=pay0 pt=96 ! rtpulpfecenc percentage=100 pt=122
[rtp_server-1] 0:00:00.024067424 47513 0xaaab14508c00 INFO rtspmountpoints rtsp-mount-points.c:333:gst_rtsp_mount_points_add_factory: adding media factory 0xaaab14610a30 for path /night
[rtp_server-1] 0:00:02.902776512 47513 0xaaab1473eb90 INFO rtspclient rtsp-client.c:4157:gst_rtsp_client_set_connection: client 0xaaab14744730 connected to server ip 192.168.1.22, ipv6 = 0
[rtp_server-1] 0:00:02.902819264 47513 0xaaab1473eb90 INFO rtspclient rtsp-client.c:4161:gst_rtsp_client_set_connection: added new client 0xaaab14744730 ip 192.168.1.177:49388
[rtp_server-1] [INFO] [1715253977.210280320] [robot_1_22.RtspServerNode]: New RTSP client
[rtp_server-1] 0:00:02.903265440 47513 0xaaab1473eb90 INFO rtspclient rtsp-client.c:4763:gst_rtsp_client_attach: client 0xaaab14744730: attaching to context 0xffff8c00c9b0
[rtp_server-1] 0:00:02.903372000 47513 0xffff8c0022a0 INFO rtspthreadpool rtsp-thread-pool.c:329:do_loop: enter mainloop of thread 0xffff8c00bc40
[rtp_server-1] 0:00:02.904017312 47513 0xffff8c0022a0 INFO rtspclient rtsp-client.c:3533:handle_request: client 0xaaab14744730: received a request OPTIONS rtsp://192.168.1.22:8554/cam 1.0
[rtp_server-1] 0:00:02.912851872 47513 0xffff8c0022a0 INFO rtspclient rtsp-client.c:3533:handle_request: client 0xaaab14744730: received a request DESCRIBE rtsp://192.168.1.22:8554/cam 1.0
[rtp_server-1] 0:00:02.912961248 47513 0xffff8c0022a0 INFO rtspmountpoints rtsp-mount-points.c:300:gst_rtsp_mount_points_match: found media factory 0xaaab14610960 for path /cam
[rtp_server-1] 0:00:02.912992096 47513 0xffff8c0022a0 INFO GST_PIPELINE gstparse.c:336:gst_parse_launch_full: parsing pipeline description 'appsrc name=imagesrc is-live=true ! videoconvert ! x264enc ! rtph264pay name=pay0 pt=96 ! rtpulpfecenc percentage=100 pt=122'
[rtp_server-1] 0:00:02.913630208 47513 0xffff8c0022a0 INFO GST_PLUGIN_LOADING gstplugin.c:902:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstapp.so" loaded
[rtp_server-1] 0:00:02.913651168 47513 0xffff8c0022a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:363:gst_element_factory_create: creating element "appsrc"
[rtp_server-1] 0:00:02.914001376 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstBaseSrc@0xffff90010a90> adding pad 'src'
[rtp_server-1] 0:00:02.915542848 47513 0xffff8c0022a0 INFO GST_PLUGIN_LOADING gstplugin.c:902:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstvideoconvert.so" loaded
[rtp_server-1] 0:00:02.915568032 47513 0xffff8c0022a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:363:gst_element_factory_create: creating element "videoconvert"
[rtp_server-1] 0:00:02.915905280 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstBaseTransform@0xffff9001d800> adding pad 'sink'
[rtp_server-1] 0:00:02.915929408 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstBaseTransform@0xffff9001d800> adding pad 'src'
[rtp_server-1] 0:00:02.918303584 47513 0xffff8c0022a0 INFO x264enc gstx264enc.c:3002:plugin_init: linked against x264 build: 155
[rtp_server-1] 0:00:02.918392864 47513 0xffff8c0022a0 INFO GST_PLUGIN_LOADING gstplugin.c:902:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstx264.so" loaded
[rtp_server-1] 0:00:02.918405920 47513 0xffff8c0022a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:363:gst_element_factory_create: creating element "x264enc"
[rtp_server-1] 0:00:02.918646560 47513 0xffff8c0022a0 INFO x264enc gstx264enc.c:225:gst_x264_enc_add_x264_chroma_format: 8-bit depth supported
[rtp_server-1] 0:00:02.918660448 47513 0xffff8c0022a0 INFO x264enc gstx264enc.c:250:gst_x264_enc_add_x264_chroma_format: 10-bit depth supported
[rtp_server-1] 0:00:02.918781728 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstVideoEncoder@0xffff90024f50> adding pad 'sink'
[rtp_server-1] 0:00:02.918804256 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstVideoEncoder@0xffff90024f50> adding pad 'src'
[rtp_server-1] 0:00:02.920072416 47513 0xffff8c0022a0 INFO GST_PLUGIN_LOADING gstplugin.c:902:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstrtp.so" loaded
[rtp_server-1] 0:00:02.920090976 47513 0xffff8c0022a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:363:gst_element_factory_create: creating element "rtph264pay"
[rtp_server-1] 0:00:02.920310016 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstRTPBasePayload@0xffff900341f0> adding pad 'src'
[rtp_server-1] 0:00:02.920336384 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstRTPBasePayload@0xffff900341f0> adding pad 'sink'
[rtp_server-1] 0:00:02.920404928 47513 0xffff8c0022a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:363:gst_element_factory_create: creating element "rtpulpfecenc"
[rtp_server-1] 0:00:02.920484032 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstRtpUlpFecEnc@0xffff90039060> adding pad 'src'
[rtp_server-1] 0:00:02.920505376 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:671:gst_element_add_pad:<GstRtpUlpFecEnc@0xffff90039060> adding pad 'sink'
[rtp_server-1] 0:00:02.920540320 47513 0xffff8c0022a0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:363:gst_element_factory_create: creating element "bin"
[rtp_server-1] 0:00:02.920651552 47513 0xffff8c0022a0 INFO GST_PIPELINE grammar.y:648:gst_parse_perform_link: linking some pad of GstAppSrc named imagesrc to some pad of GstVideoConvert named videoconvert0 (0/0) with caps "(NULL)"
[rtp_server-1] 0:00:02.920664992 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstutils.c:1771:gst_element_link_pads_full: trying to link element imagesrc:(any) to element videoconvert0:(any)
[rtp_server-1] 0:00:02.920677728 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1034:gst_pad_check_link: trying to link imagesrc:src and videoconvert0:sink
[rtp_server-1] 0:00:02.920719360 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:4237:gst_pad_peer_query:<videoconvert0:src> pad has no peer
[rtp_server-1] 0:00:02.921737920 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1587:prepare_link_maybe_ghosting: imagesrc and videoconvert0 in same bin, no need for ghost pads
[rtp_server-1] 0:00:02.921766464 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2377:gst_pad_link_prepare: trying to link imagesrc:src and videoconvert0:sink
[rtp_server-1] 0:00:02.921781472 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:4237:gst_pad_peer_query:<videoconvert0:src> pad has no peer
[rtp_server-1] 0:00:02.922745088 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2585:gst_pad_link_full: linked imagesrc:src and videoconvert0:sink, successful
[rtp_server-1] 0:00:02.922755552 47513 0xffff8c0022a0 INFO GST_EVENT gstevent.c:1579:gst_event_new_reconfigure: creating reconfigure event
[rtp_server-1] 0:00:02.922772672 47513 0xffff8c0022a0 INFO GST_EVENT gstpad.c:5812:gst_pad_send_event_unchecked:<imagesrc:src> Received event on flushing pad. Discarding
[rtp_server-1] 0:00:02.922797888 47513 0xffff8c0022a0 INFO GST_PIPELINE grammar.y:648:gst_parse_perform_link: linking some pad of GstVideoConvert named videoconvert0 to some pad of GstX264Enc named x264enc0 (0/0) with caps "(NULL)"
[rtp_server-1] 0:00:02.922806240 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstutils.c:1771:gst_element_link_pads_full: trying to link element videoconvert0:(any) to element x264enc0:(any)
[rtp_server-1] 0:00:02.922817376 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1034:gst_pad_check_link: trying to link videoconvert0:src and x264enc0:sink
[rtp_server-1] 0:00:02.923783040 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1587:prepare_link_maybe_ghosting: videoconvert0 and x264enc0 in same bin, no need for ghost pads
[rtp_server-1] 0:00:02.923799136 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2377:gst_pad_link_prepare: trying to link videoconvert0:src and x264enc0:sink
[rtp_server-1] 0:00:02.924776800 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2585:gst_pad_link_full: linked videoconvert0:src and x264enc0:sink, successful
[rtp_server-1] 0:00:02.924791520 47513 0xffff8c0022a0 INFO GST_EVENT gstevent.c:1579:gst_event_new_reconfigure: creating reconfigure event
[rtp_server-1] 0:00:02.924799392 47513 0xffff8c0022a0 INFO GST_EVENT gstpad.c:5812:gst_pad_send_event_unchecked:<videoconvert0:src> Received event on flushing pad. Discarding
[rtp_server-1] 0:00:02.924817504 47513 0xffff8c0022a0 INFO GST_PIPELINE grammar.y:648:gst_parse_perform_link: linking some pad of GstX264Enc named x264enc0 to some pad of GstRtpH264Pay named pay0 (0/0) with caps "(NULL)"
[rtp_server-1] 0:00:02.924830016 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstutils.c:1771:gst_element_link_pads_full: trying to link element x264enc0:(any) to element pay0:(any)
[rtp_server-1] 0:00:02.924839904 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1034:gst_pad_check_link: trying to link x264enc0:src and pay0:sink
[rtp_server-1] 0:00:02.924857248 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:4237:gst_pad_peer_query:<pay0:src> pad has no peer
[rtp_server-1] 0:00:02.924873856 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1587:prepare_link_maybe_ghosting: x264enc0 and pay0 in same bin, no need for ghost pads
[rtp_server-1] 0:00:02.924883968 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2377:gst_pad_link_prepare: trying to link x264enc0:src and pay0:sink
[rtp_server-1] 0:00:02.924892416 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:4237:gst_pad_peer_query:<pay0:src> pad has no peer
[rtp_server-1] 0:00:02.924902208 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2585:gst_pad_link_full: linked x264enc0:src and pay0:sink, successful
[rtp_server-1] 0:00:02.924906848 47513 0xffff8c0022a0 INFO GST_EVENT gstevent.c:1579:gst_event_new_reconfigure: creating reconfigure event
[rtp_server-1] 0:00:02.924911936 47513 0xffff8c0022a0 INFO GST_EVENT gstpad.c:5812:gst_pad_send_event_unchecked:<x264enc0:src> Received event on flushing pad. Discarding
[rtp_server-1] 0:00:02.924924448 47513 0xffff8c0022a0 INFO GST_PIPELINE grammar.y:648:gst_parse_perform_link: linking some pad of GstRtpH264Pay named pay0 to some pad of GstRtpUlpFecEnc named rtpulpfecenc0 (0/0) with caps "(NULL)"
[rtp_server-1] 0:00:02.924931104 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstutils.c:1771:gst_element_link_pads_full: trying to link element pay0:(any) to element rtpulpfecenc0:(any)
[rtp_server-1] 0:00:02.924937920 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1034:gst_pad_check_link: trying to link pay0:src and rtpulpfecenc0:sink
[rtp_server-1] 0:00:02.924950336 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:4237:gst_pad_peer_query:<rtpulpfecenc0:src> pad has no peer
[rtp_server-1] 0:00:02.924959552 47513 0xffff8c0022a0 INFO GST_PADS gstutils.c:1587:prepare_link_maybe_ghosting: pay0 and rtpulpfecenc0 in same bin, no need for ghost pads
[rtp_server-1] 0:00:02.924967264 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2377:gst_pad_link_prepare: trying to link pay0:src and rtpulpfecenc0:sink
[rtp_server-1] 0:00:02.924974624 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:4237:gst_pad_peer_query:<rtpulpfecenc0:src> pad has no peer
[rtp_server-1] 0:00:02.924982400 47513 0xffff8c0022a0 INFO GST_PADS gstpad.c:2585:gst_pad_link_full: linked pay0:src and rtpulpfecenc0:sink, successful
[rtp_server-1] 0:00:02.924986624 47513 0xffff8c0022a0 INFO GST_EVENT gstevent.c:1579:gst_event_new_reconfigure: creating reconfigure event
[rtp_server-1] 0:00:02.924991360 47513 0xffff8c0022a0 INFO GST_EVENT gstpad.c:5812:gst_pad_send_event_unchecked:<pay0:src> Received event on flushing pad. Discarding
[rtp_server-1] 0:00:02.925138048 47513 0xffff8c0022a0 INFO GST_PARENTAGE gstbin.c:4478:gst_bin_get_by_name: [bin0]: looking up child element pay0
[rtp_server-1] 0:00:02.925156352 47513 0xffff8c0022a0 INFO rtspmedia rtsp-media.c:2066:gst_rtsp_media_collect_streams: found stream 0 with payloader 0xffff900341f0
[rtp_server-1] 0:00:02.925165856 47513 0xffff8c0022a0 INFO GST_ELEMENT_PADS gstelement.c:928:gst_element_get_static_pad: found pad pay0:src
[rtp_server-1]
[rtp_server-1] (rtp_server:47513): GStreamer-CRITICAL **: 13:26:17.232: gst_ghost_pad_new: assertion '!gst_pad_is_linked (target)' failed
[rtp_server-1]
[rtp_server-1] (rtp_server:47513): GStreamer-CRITICAL **: 13:26:17.232: gst_pad_set_active: assertion 'GST_IS_PAD (pad)' failed
[rtp_server-1]
[rtp_server-1] (rtp_server:47513): GStreamer-CRITICAL **: 13:26:17.232: gst_element_add_pad: assertion 'GST_IS_PAD (pad)' failed
[rtp_server-1]
[rtp_server-1] ** (rtp_server:47513): CRITICAL **: 13:26:17.232: gst_rtsp_stream_new: assertion 'GST_IS_PAD (pad)' failed
[rtp_server-1]
[rtp_server-1] ** (rtp_server:47513): CRITICAL **: 13:26:17.232: gst_rtsp_stream_set_multicast_iface: assertion 'GST_IS_RTSP_STREAM (stream)' failed
[rtp_server-1]
[rtp_server-1] ** (rtp_server:47513): CRITICAL **: 13:26:17.232: gst_rtsp_stream_set_max_mcast_ttl: assertion 'GST_IS_RTSP_STREAM (stream)' failed
[rtp_server-1]
[rtp_server-1] ** (rtp_server:47513): CRITICAL **: 13:26:17.232: gst_rtsp_stream_set_bind_mcast_address: assertion 'GST_IS_RTSP_STREAM (stream)' failed
[rtp_server-1]
[rtp_server-1] ** (rtp_server:47513): CRITICAL **: 13:26:17.232: gst_rtsp_stream_set_profiles: assertion 'GST_IS_RTSP_STREAM (stream)' failed
[rtp_server-1]
[rtp_server-1] ** (rtp_server:47513): CRITICAL **: 13:26:17.232: gst_rtsp_stream_set_protocols: assertion 'GST_IS_RTSP_STREAM (stream)' failed