How to deploy RTSP server using from OpenCv frames

Hi,

I’m trying to make an RTSP server from OpenCv frames using OpenCv VideoWriter with Gstreamer backend.

After rebuilding OpenCV with Gstreamer I’m getting some errors, does not matter even in C++ or Python. I’m getting errors on both.

It seems that using Python the error sounds more descriptive.

import os
os.add_dll_directory("C:\\gstreamer\\1.0\\msvc_x86_64\\bin")
os.add_dll_directory("C:\\gstreamer\\1.0\\msvc_x86_64\\lib\\gstreamer-1.0")
import cv2 as cv
import numpy
 
sourceUrl = "http://pendelcam.kip.uni-heidelberg.de/mjpg/video.mjpg"
#command = "appsrc ! videoconvert ! x264enc ! rtph264pay ! udpsink port=5004 host=127.0.0.1 "
command = "appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw, format=BGRx ! videoconvert ! x264enc speed-preset=veryfast tune=zerolatency bitrate=800 insert-vui=1 ! h264parse ! rtph264pay name=pay0 pt=96 config-interval=1 ! udpsink port=5004 host=127.0.0.1 auto-multicast=0"
 
cap = cv.VideoCapture(sourceUrl)
if not cap.isOpened():
    print("Capture Not Opened")
 
writer = cv.VideoWriter(command, cv.CAP_GSTREAMER, 0, 30, (1280, 720), True)
if not writer.isOpened():
    print("WRITER NOT OPENED!!!!!")
 
if cap.isOpened() and writer.isOpened():
    while True:
        ret, frame = cap.read()
        if not ret:
            print("Not Fram received")
            break
        cv.imshow("frame", frame)
        if cv.waitKey(1) > 0:
            break
   
    cap.release()
    writer.release()

I’m getting this ouput in console. It seems that plugins cannot be loaded event I have added the plugin directory.
I have installed MSVC 64-bit (VS 2019, Release CRT) runtime and development installers.

(python.exe:6512): GStreamer-WARNING **: 17:18:56.512: Failed to load plugin 'C:\gstreamer\1.0\msvc_x86_64\lib\gstreamer-1.0\gstx264.dll': The specified module could not be found.
This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
There are also some third-party GUIs to list and debug DLL dependencies recursively.
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.512: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GLib-GObject-WARNING **: 17:18:56.512: cannot retrieve class for invalid (unclassed) type '<invalid>'
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.512: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.512: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GLib-GObject-CRITICAL **: 17:18:56.512: g_object_class_find_property: assertion 'G_IS_OBJECT_CLASS (class)' failed
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.512: gst_object_unref: assertion 'object != NULL' failed
 
(python.exe:6512): GLib-GObject-CRITICAL **: 17:18:56.512: g_type_class_unref: assertion 'g_class != NULL' failed
 
(python.exe:6512): GStreamer-WARNING **: 17:18:56.512: Failed to load plugin 'C:\gstreamer\1.0\msvc_x86_64\lib\gstreamer-1.0\gstvideoparsersbad.dll': The specified module could not be found.
This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
There are also some third-party GUIs to list and debug DLL dependencies recursively.
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.512: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GLib-GObject-WARNING **: 17:18:56.525: cannot retrieve class for invalid (unclassed) type '<invalid>'
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.526: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.526: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GStreamer-WARNING **: 17:18:56.530: Failed to load plugin 'C:\gstreamer\1.0\msvc_x86_64\lib\gstreamer-1.0\gstvideoparsersbad.dll': The specified module could not be found.
This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
There are also some third-party GUIs to list and debug DLL dependencies recursively.
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.531: gst_object_unref: assertion 'object != NULL' failed
 
(python.exe:6512): GLib-GObject-CRITICAL **: 17:18:56.532: g_type_class_unref: assertion 'g_class != NULL' failed
 
(python.exe:6512): GStreamer-WARNING **: 17:18:56.535: Failed to load plugin 'C:\gstreamer\1.0\msvc_x86_64\lib\gstreamer-1.0\gstrtp.dll': The specified module could not be found.
This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
There are also some third-party GUIs to list and debug DLL dependencies recursively.
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.537: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GLib-GObject-WARNING **: 17:18:56.537: cannot retrieve class for invalid (unclassed) type '<invalid>'
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.539: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.540: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GLib-GObject-CRITICAL **: 17:18:56.540: g_object_class_find_property: assertion 'G_IS_OBJECT_CLASS (class)' failed
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.541: gst_object_unref: assertion 'object != NULL' failed
 
(python.exe:6512): GLib-GObject-CRITICAL **: 17:18:56.541: g_type_class_unref: assertion 'g_class != NULL' failed
 
(python.exe:6512): GStreamer-WARNING **: 17:18:56.545: Failed to load plugin 'C:\gstreamer\1.0\msvc_x86_64\lib\gstreamer-1.0\gstudp.dll': The specified module could not be found.
This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
There are also some third-party GUIs to list and debug DLL dependencies recursively.
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.546: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GLib-GObject-WARNING **: 17:18:56.547: cannot retrieve class for invalid (unclassed) type '<invalid>'
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.548: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.549: gst_element_factory_get_element_type: assertion 'GST_IS_ELEMENT_FACTORY (factory)' failed
 
(python.exe:6512): GLib-GObject-CRITICAL **: 17:18:56.551: g_object_class_find_property: assertion 'G_IS_OBJECT_CLASS (class)' failed
 
(python.exe:6512): GStreamer-CRITICAL **: 17:18:56.552: gst_object_unref: assertion 'object != NULL' failed
 
(python.exe:6512): GLib-GObject-CRITICAL **: 17:18:56.553: g_type_class_unref: assertion 'g_class != NULL' failed
[ WARN:0@1.151] global cap_gstreamer.cpp:2436 cv::CvVideoWriter_GStreamer::open OpenCV | GStreamer warning: error opening writer pipeline: no property "speed-preset" in element "x264enc"

Do you have any idea on what I’m doing wrong.
Thanks for your help. With regards,

There might be several issues for your case.

Maybe main one is setting environment. I have poor experience with Windows, but you may have to ensure that gstreamer DLLs location is in PATH.

Once this is ok, you would have to be careful about how fast the video from the link can be received.

I’d first suggest that you download the mjpg video with:

wget http://pendelcam.kip.uni-heidelberg.de/mjpg/video.mjpg

you can stop when size reaches 20 MB with Ctrl-C.
Then try to display with:

gst-play-1.0 video.mjpg

It will display, but may be so fast that you can’t see the scene.

Why ? Get details from the video file with:

gst-discoverer-1.0 video.mjpg
...
Properties:
  Duration: 0:00:00.000000000
  Seekable: no
  Live: no
  container #0: Multipart
    video #1: JPEG
      Stream ID: 324b400c2525037f9335f6a23a11b501f9a1c0b4dd9f0e812ea479b25822fb12/image/jpeg
      Width: 640
      Height: 360
      Depth: 24
      Frame rate: 0/1
      Pixel aspect ratio: 1/1
      Interlaced: false
      Bitrate: 0
      Max bitrate: 0

The framerate is unknown (0/1) that is async mode. Let’s try to set a framerate:

gst-launch-1.0 filesrc location=video.mjpg ! image/jpeg,width=640,height=480,framerate=30/1 ! jpegdec ! videoconvert ! autovideosink

If this is ok, you may also create the opencv VideoCapture explictly setting the video backend. Passing an url may use FFMPEG backend if available (if not available, may try other backends to no avail).
For a gstreamer backend capture, you would use:

cap_pipeline_str = 'filesrc location=video.mjpg ! image/jpeg,width=640,height=480,framerate=30/1 ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! queue ! appsink drop=1'
cap = cv2.VideoCapture(cap_pipeline_str, cv2.CAP_GSTREAMER)

Opencv writer pipeline may be:

writer_pipeline_str = 'appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! x264enc key-int-max=30 speed-preset=veryfast tune=zerolatency bitrate=800 insert-vui=1 ! h264parse ! rtph264pay ! udpsink port=5004 host=127.0.0.1'
writer = cv2.VideoWriter(writer_pipeline_str, cv2.CAP_GSTREAMER, 0, float(30), (640, 480), True)

This would stream the video with RTPH264 over UDP to localhost port 5004.
Then you can have a local RTSP server getting the RTPH264 stream from port 5004 and serving it as RTSP with test-launch example such as :

test-launch "udpsrc address=127.0.0.1 port=5004 ! application/x-rtp,encoding-name=H264 ! rtph264depay ! h264parse ! rtph264pay name=pay0"

If you have your own RTSP server running, you may just use rtspserversink instead.

Finally, note that you would have to push the frame into writer in the loop after/before/instead imshow:

writer.write(frame)

Hi,

Thanks for your reply!
My setup is simple, I have just installed runtime and development installers of MSVC 64-bit (VS 2019, Release CRT).

I have seen that I have a system environment variable called GSTREAMER_1_0_ROOT_MSVC_X86_64=C:\gstreamer\1.0\msvc_x86_64\

My code is just an example, because I wanted to test if the environment is working (OpenCv + Gstreamer).

What I want to to is capture images from IntelRealSense camera using OpenCv, do some process and stream frames as video so multiple clients can be able to show video streaming.

I don’t understand your example, If I undertsand weel you are proposing to re-stream the video twice? What is test-launch?

What do you want to say with my own RTSP server running? That why I want to use Gstreamer, I want to use Gstreamer to work as RTSP server.

I’ll try to add Gstreamer path to system environment PATH.

With regards,

test-launch is a basic rtsp server example that you may have a look at for starting.

I have added both paths to System PATH and I have same type of errors.
Also added GST_PLUGIN_SCANER environment variable and set to
C:\gstreamer\1.0\msvc_x86_64\lib\gstreamer-1.0
Didn’t work too

I don’t understand this type of errors all dlls mentioned in error messages are present in the plugin folder. And I have used standard installation…

With regards,

Someone better skilled with gstreamer with Windows may better help for your case.
Sorry I have mainly advice for Linux case.

1 Like

Thanks for your information in any case.

Hi @Honey_Patouceul

Finally it’s working. I have added gstreamer paths to Path environment variable following a reply on Stackoverflow

C:\gstreamer\1.0\msvc_x86_64\lib\
C:\gstreamer\1.0\msvc_x86_64\bin\ 
C:\gstreamer\1.0\msvc_x86_64\include\

I have also tried before this to copy msvc_x86_64\bin\dlls to msvc_x86_64\lib\gstreamer-1.0 because I seen that plugins variables was depending on gstreamer dlls on bin. It’s not a good option but works also.

Regarding to gstreamer pipeline with this command seems to work but with two inconvenients.

  • Sometimes VLC stops and needs to buffer, seems not to be very stable.
  • It doesn’t allow more than one clien.

command = "appsrc ! videoconvert ! x264enc speed-preset=veryfast tune=zerolatency bitrate=500 insert-vui=1 ! rtph264pay ! udpsink port=5004 host=127.0.0.1"

I’m not a Gstreamer and video expert but I was trying to read and testing multiple combinations, at the moment this is what I have achieved.

With regards and thanks for your time!

Do you have a local RTSP server such as test-launch ?
Or are you just reading RTPH264 from localhost into VLC using a SDP file ?
Not sure why you removed key-int-max=30 but it might take longer time for receiving a key frame allowing full decoding and ability to display.

Better share your code and commands so that it can easily be reproduced.
Also suggest using a recorded file as previously advised for ruling out source url transport issues.

Hi!

Sorry about key-int-max=30 I have been testing from scratch and maybe I forgot.

First I tried to see only video without RTSP …

appsrc ! videoconvert ! autovideosink

I don’t understand why you are talking about test-lauch (local RTSP server). With this code my program is working as RTSP server. Is that true?
I tried with key-int-max=30 It seems that VLC have also to buffer sometimes. I have been reading so much examples on the internet using after appsrc video/x-raw format=BGR or other things but with that doesn’t works …

import os
os.add_dll_directory("C:\\gstreamer\\1.0\\msvc_x86_64\\bin")
import cv2 as cv
 
sourceUrl = "http://pendelcam.kip.uni-heidelberg.de/mjpg/video.mjpg"
command = "appsrc ! videoconvert ! x264enc key-int-max=30 speed-preset=veryfast tune=zerolatency bitrate=800 insert-vui=1 ! rtph264pay ! udpsink port=5004 host=127.0.0.1"
 
cap = cv.VideoCapture(sourceUrl)
if not cap.isOpened():
    print("CAPTURE NOT OPENED")
 
fps = cap.get(cv.CAP_PROP_FPS)
w = int(cap.get(cv.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT))
size = (w, h)
print(f"Size: {(size)}. FPS: {fps}")
 
writer = cv.VideoWriter(command, cv.CAP_GSTREAMER, 0, fps, size, True)
if not writer.isOpened():
    print("WRITER NOT OPENED!!!!!")
 
if cap.isOpened() and writer.isOpened():
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            print("Not Frame received")
            break
        writer.write(frame)
        cv.imshow("frame", frame)
        if cv.waitKey(1) > 0:
            break
   
    cap.release()
    writer.release()

This is the sdp file which I’m using to play with VLC.

c=IN IP4 127.0.0.1
m=video 5004 RTP/AVP 96
a=rtpmap:96 H264/90000

EDIT:
I think that now I understand you about test-launch… Gstreamer only acts as RTP server sending media over UDP but not reallly as RTSP server with PLAY and PAUSE commands …
You mean using RTSP server to receive RTP from Gstreamer and making it available as RTSP server. Am I on the way?

Thanks for your time!

Yes, your understanding is now correct, opencv application would stream RTPH264 to localhost into port 5004, then test-launch RTSP server would read it and serve it as RTSP. (Note that I’ve edited my original post that was lacking quotes for passing the pipeline to test-launch).
So just build test-launch and try. Once both the RTSP server and your opencv application will be running, you should be able to read with VLC or a browser or gst-play-1.0 with url rtsp://127.0.0.1:8554/test from localhost or rtsp://<IP_of_RTSP_server>:8554/test from other hosts in same LAN.

opencvApp.py
import cv2

# You may check for opencv video backends support with the following
#print(cv2.getBuildInformation())

sourceUrl = "http://pendelcam.kip.uni-heidelberg.de/mjpg/video.mjpg"

# 1. Capture with FFMPEG backend from URL - Assumes opencv has support for FFMPEG video backend
cap = cv2.VideoCapture(sourceUrl, cv2.CAP_FFMPEG)

# 2. Capture with Gstreamer backend from URL - Assumes opencv has support for GStreamer video backend -  fails in my case because of nvjpegdec but may work with your case 
#cap = cv2.VideoCapture(sourceUrl, cv2.CAP_GSTREAMER)

# 3. The following pipeline using jpegdec instead works in my case with gstreamer backend. 
#cap_pipeline_str = 'souphttpsrc is-live=1 automatic-redirect=0 location=http://pendelcam.kip.uni-heidelberg.de/mjpg/video.mjpg ! multipartdemux ! image/jpeg,framerate=25/1 ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! queue ! appsink drop=1'
#cap = cv2.VideoCapture(cap_pipeline_str, cv2.CAP_GSTREAMER)

if not cap.isOpened():
    print("CAPTURE NOT OPENED")
    exit(-1)

fps = float(cap.get(cv2.CAP_PROP_FPS))
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
size = (w, h)
print(f"Size: {(size)}, FPS: {fps}")

rtph264_writer_pipeline_str = "appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! x264enc key-int-max=30 speed-preset=veryfast tune=zerolatency bitrate=800 insert-vui=1 ! rtph264pay ! udpsink port=5004 host=127.0.0.1"
rtph264_writer = cv2.VideoWriter(rtph264_writer_pipeline_str, cv2.CAP_GSTREAMER, 0, fps, size, True)
if not rtph264_writer.isOpened():
    print("WRITER NOT OPENED!!!!!")
 
if cap.isOpened() and rtph264_writer.isOpened():
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            print("Not Frame received")
            break
        rtph264_writer.write(frame)
        cv2.imshow("frame", frame)
        if cv2.waitKey(1) > 0:
            break
   
cap.release()
if rtph264_writer.isOpened():   
    rtph264_writer.release()

exit(0)

EDIT: Not sure if related to this post, but it seems that the source URL may now be down. It was working fine a few hours ago.
EDIT2: Seems working back now. You may first check with a browser and if it works test that code.

Hi,

I tried to build gstreamer to build gst-rtsp-server but fails using meson (I didn’t use mason before).

What will be de advantage of using test-launch? I day that because with RTP server I can play with VLC using sdp file.
Can gstreamer work as RTSP server only using VideoWriter? At the moment I need to have more than one client. I supose that gstreamer can do it.

Using test-launchI supose that I’m adding extra delay because I’m doing the streaming twice.

Regarding my test program you can ignore VideoCapture. I’m using it because with that I can get frames one by one to test VideoWriter. it’s just a simple program to test gstreamer. What I want to do finally is to get frames from videocamera process them and write them to VideWriter.

With regards,

Some clarifications that might help:

  • gstreamer is a framework for handling video (and images, audio, more) that can be used by many applications such as opencv in support for this backend for VideoCapture and VideoWriter. test-launch is a basic RTSP server also designed from using gstreamer, especially its gstrtspserver library.
  • The advantage of a RTSP server is that it will manage the clients, negociate transport protocols with them, and send the SDP file so that you don’t have to bother with creating it by hand (and copying to client), and more…
  • The gstreamer based opencv VideoWriter from the example above is only streaming RTPH264, not RTSP. Although it might be possible to write a full RTSP server example with opencv, for simplicity my example used test-launch RTSP server separate from opencv application. This wouldn’t add so much latency as this goes through local loopback interface, and is just depaying/parsing/repaying without decoding/re-encoding.

For trying to answer your initial question with RTSP, note that you may read RTPH264 from localhost with SDP file and re-stream as RTSP with VLC or FFMPEG or else if you’re not able to build test-launch.

This would be out of your original question’s scope, but maybe what you want would be using a multicast address for RTP. Instead of streaming to localhost, you would stream to the multicast group address:

rtph264_writer_pipeline_str = "... ! udpsink port=5004 host=224.1.1.1"

Then adjust a multicast SDP file with that IP address, any host in LAN should be able to read the RTPH264 stream with that SDP file (assuming it has SW for decoding, permission to get that port and no firewall blocking).
Note that multicast might have issues with some wifi networks.

1 Like

Hi @Honey_Patouceul

First of all I want to thanks you all your replies.
It’s first time that I have to deal with multimedia in development field and you are helping me :slight_smile:

Do you have a guide to build Gstreamer or only the gst-rtsp-server subproject (I think that test-launs is using this subproject).

As I understand RTP is only the transport protocol and RTSP server is the responsible to receive client connection requests and sending them the information to play RTP streaming.

Gstreamer backend of Opencv can only encode separated frames in video container and stream video in RTP protocol, so I cannot work as RTSP server, thats true? (unless I program my custom RTSP server).

I don’t think that develop an RTSP server have any sense if I can use alrededy developed ones that I’m ser will be more tested than custom one…

I know that FFMPEG has also an integration with OpenCv but I have read in OpenCv forum that the wrapper does not handle rtsp (or other) streaming (even if ffmpeg itself can do that).

Do you think that I’m on the correct way using Gstreamer? I think that I can try to use Gstreamer and them use FFMPEG to work as RTSP server reading from Gstreamer RTP streaming. I want to do as best as I can because I need to be efficient because the streamed video will be from camera and I need as less latency as possible.

Regarding VLC I know that it can work as RSTP server but in VLC discord channel they said to me that maybe VLC RTSP server is not very stable.

Thanks for your help!

Sorry to say that, but I’m just trying to share some knowledge (or my belief so far).
Though, don’t think you can throw your current thinking and that I’ll correct these :wink:
Better try what is advised, try it and understand it, and just report what doesn’t work out with code and errors.

Hi,

I’ve seen that maybe I’m having a problem with the pipeline configuration.

I’ve noticed that I have 2 or more seconds of latency or delay. I have seen that my executable is not using GPU, so I think that maybe this is causing latency.

I’ve been reading about Vaapi but seems that it’s not present on my installation.

With regards,

I have discovered one RTSP server which seems to work and it seems is commonly used. I have try to use MediaMtx (rtsp-simple-server) and it’s working :slight_smile:

Gstreamer have a sink for this server and I can use it from pipeline using rtspclientsink.

It seems to be working properly

Glad to see you’ve moved forward.

Yes as previously advised in my initial post in a different flavor, but YMMV:

Have fun !