GStreamer Pipeline Crash: "free(): invalid pointer" with Core Dump

Application Development
1

Hey ,

I am running a GStreamer pipeline to process an RTSP stream using NVIDIA hardware acceleration. However, I am encountering a core dump with the error “free(): invalid pointer”.

GST_DEBUG=3 gst-launch-1.0 rtspsrc location="rtsp://localhost:23101/cam01" retry=100 latency=1000 ! watchdog timeout=300000 ! rtph264depay ! h264parse ! h264timestamper ! queue leaky=downstream max-size-time=2000000000 max-size-buffers=0 ! h264parse ! nvh264dec ! glcolorconvert ! video/x-raw'(ANY)',format=RGBA ! videorate ! video/x-raw'(ANY)' ! gldownload ! videoflip ! fakesink

Output along with pipeline


0:00:00.024707615 13197 0x55cfdbe8da00 WARN               cudanvrtc gstcudanvrtc.cpp:152:gst_cuda_nvrtc_load_library_once: Could not open nvrtc library libnvrtc.so: cannot open shared object file: No such file or directory
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Got context from element 'nvh264dec0': gst.cuda.context=context, gst.cuda.context=(GstCudaContext)"\(GstCudaContext\)\ cudacontext1", cuda-device-id=(uint)0;
Progress: (open) Opening Stream
Pipeline is PREROLLED ...
Prerolled, waiting for progress to finish...
Progress: (connect) Connecting to rtsp://localhost:23101/MLITcam01
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Redistribute latency...
0:00:01.580019659 13197 0x7f7694001210 FIXME                default gstutils.c:4088:gst_element_decorate_stream_id_internal:<fakesrc0> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Progress: (request) Sending PLAY request
Redistribute latency...
Progress: (request) Sent PLAY request
Redistribute latency...
0:00:01.596581741 13197 0x7f7694001470 FIXME        rtpjitterbuffer gstrtpjitterbuffer.c:1756:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Unsupported media clock
Redistribute latency...
0:00:02.642330837 13197 0x7f7694001930 WARN        codectimestamper gstcodectimestamper.c:233:gst_codec_timestamper_set_caps:<h264timestamper0> Unknown frame rate, assume 25/1
Redistribute latency...
Redistribute latency...
Redistribute latency...
Redistribute latency...
0:00:02.667203210 13197 0x7f76440040f0 WARN               cudautils gstcudautils.cpp:479:gst_cuda_graphics_resource_register_gl_buffer: CUDA call failed: CUDA_ERROR_OUT_OF_MEMORY, out of memory
0:00:02.667223388 13197 0x7f76440040f0 WARN               nvdecoder gstnvdecoder.cpp:485:gst_nv_decoder_register_cuda_resource:<nvdecoder0> Failed to register memory
0:00:02.667233631 13197 0x7f76440040f0 WARN               nvdecoder gstnvdecoder.cpp:526:gst_nv_decoder_ensure_cuda_graphics_resource:<nvdecoder0> Couldn't register resource
0:00:02.667242005 13197 0x7f76440040f0 WARN               nvdecoder gstnvdecoder.cpp:577:gst_nv_decoder_copy_frame_to_gl_internal:<nvdecoder0> could not register 0th memory
0:00:02.667264622 13197 0x7f7694000b70 WARN               nvdecoder gstnvdecoder.cpp:983:gst_nv_decoder_output_picture:<nvh264dec0> Couldn't copy frame to GL memory, fallback to system memory
0:00:02.667325514 13197 0x7f76440040f0 ERROR                gldebug gstgldebug.c:306:_gst_gl_debug_callback:<glcontextglx0> high: GL error from API id:1285, GL_OUT_OF_MEMORY error generated. Failed to allocate memory for buffer object.
0:00:02.667334769 13197 0x7f76440040f0 ERROR               glmemory gstglmemorypbo.c:324:_pbo_download_transfer: Failed to map pbo
0:00:02.667395415 13197 0x7f76440040f0 ERROR                gldebug gstgldebug.c:306:_gst_gl_debug_callback:<glcontextglx0> high: GL error from API id:1285, GL_OUT_OF_MEMORY error generated. Failed to allocate memory for buffer object.
0:00:02.667405237 13197 0x7f76440040f0 ERROR               glmemory gstglmemorypbo.c:324:_pbo_download_transfer: Failed to map pbo

(gst-launch-1.0:13197): GStreamer-CRITICAL **: 19:49:59.502: gst_memory_unmap: assertion 'info->memory == mem' failed
free(): invalid pointer
Aborted (core dumped)

System Details:
OS: Ubuntu 24.04
CPU: Intel Xeon 5th Gen ( Gold 6542Y )
GPU: NVIDIA L4 (Driver Version: 565.57.01, CUDA 12.7)
GStreamer Version: 1.24.5

Output of gst-inspect-1.0 nvcodec

image
image1006×729 120 KB

image
image855×419 46.6 KB

Any suggestions for debugging or modifications to make the pipeline more stable?

2

The crash is unfortunate, though the problem seems to be

CUDA_ERROR_OUT_OF_MEMORY

Looking at the pipeline there is little reason to run out of memory, unless there is a leak. Try upgrading to a newer GStreamer version.

3

Thank you for your response.

We upgraded GStreamer to version 1.24.11, but unfortunately, the issue still persists.

Any insights or suggestions you can provide would be extremely helpful. Thank you in advance for your support!

4

I’d say, try and replace nvh264dec with a software decoder (avdec_h264 or openh264dec), you’ll need glupload after the decoder, that should confirm is the issue lay in that decoder. The leaky queue might be part of the problem, try removing that too. Let us know if that allow narrowing the search. As for the crash, use a debugger.

1 Like