Got data flow before segment event

ngoetting · February 12, 2024, 5:49pm

Hi all,

I tried to set up a simple video streaming pipeline using matroska and it seems to work fine. However, I receive dozens of Got data flow before segment event errors when starting the client side. What could cause this?

Here are the pipelines:

Server: gst-launch-1.0 v4l2src ! video/x-h264,width=640,height=360,framerate=24/1 ! queue ! h264parse ! mux. alsasrc ! audio/x-raw,rate=48000,channels=1 ! queue ! opusenc ! mux. matroskamux streamable=true name=mux ! tcpserversink host=$IP port=5000
Client: gst-launch-1.0 tcpclientsrc host=$IP port=5000 ! video/x-matroska ! matroskademux name=demux demux. ! queue ! h264parse ! avdec_h264 ! videoconvert ! queue ! autovideosink sync=false demux. ! queue ! opusdec ! audioconvert ! audioresample ! autoaudiosink sync=false

Both sides use GStreamer 1.22.2

Would be happy to hear your opinions!

Thanks,
Niclas

tpm · February 12, 2024, 6:27pm

This usually indicates a bug in an element, here probably either matroskademux or h264parse.

Does it cause any problems or is it just noisy/annoying?

ngoetting · February 19, 2024, 2:27pm

In this specific setup it’s just noise, however, I have a much more complicated RTSPS pipeline which this snippet is a part of and I suspected this to be the root of my problems. Maybe I should expand and explain the whole problem.

In the minimal setup, there are three devices: an edge device (raspberry), a server (debian) and a smartphone (in this case Android).

The raspberry communicates with the server by sending a matroska stream over its VPN tunnel and this works pretty well (I would like to use UDP, though, which I didn’t get to work with matroska so far, as it doesn’t send stream headers after the stream has already started; this isn’t the problem here, as it does work with TCP for now).

As there could be several smartphones connected to the server, I use RTSP(S) for this connection. The RTSP server demuxes the matroska stream and sends both video and audio via rtp to the smartphone, which authenticates via basic auth.

With GStreamer 1.14 I had no problems using this setup, with GStreamer 1.22 I get the following messages in chronological order:

rtspclient rtsp-client.c:1080:find_media: client 0x7f618cb40510: not authorized to see factory path
Got data flow before segment event for every element in the pipeline
rtpsession gstrtpsession.c:2514:gst_rtp_session_chain_send_rtp_common:<rtpsession0> Don't have a clock yet and can't determine NTP time for this packet
rtspstream rtsp-stream.c:5618:gst_rtsp_stream_query_position:<GstRTSPStream@0x7f615404a350> Couldn't obtain position: no segment event (the same for query_stop)
GStreamer-RTSP-Server:ERROR:../gst/rtsp-server/rtsp-media.c:2766:gst_rtsp_media_get_rates: assertion failed: (FALSE)
Bail out! GStreamer-RTSP-Server:ERROR:../gst/rtsp-server/rtsp-media.c:2766:gst_rtsp_media_get_rates: assertion failed: (FALSE)

I figured that the first error is actually not a problem, as this only fires for the OPTIONS RTSP request and also occurs on GStreamer 1.14. With the rest of the problems, I’m not sure which is the root of the stream not working at all.

The pipelines are:

Raspberry to server: v4l2src extra-controls="c,rotate=${videoflip_method},video_bitrate=1000000" ! video/x-h264,width=360,height=640,framerate=24/1 ! queue ! h264parse ! matroskamux streamable=true name=mux alsasrc ! audio/x-raw,rate=48000,channels=1 ! webrtcdsp probe=probe echo-suppression-level="high" gain-control="false" ! queue ! volume volume=${mic_volume} mute=true name=mic_volume ! opusenc ! queue ! mux. mux. ! tcpserversink host=0.0.0.0 port=${sbp0}
Server to Smartphone: tcpclientsrc port=${src_port} host=${raspberry_host} ! matroskademux name=demux demux. ! queue ! h264parse ! video/x-h264,stream-format=avc ! rtph264pay pt=96 name=pay0 demux. ! queue ! rtpopuspay pt=97 name=pay1
Smartphone: rtspsrc timeout=500000 latency=250 drop-on-latency=True name=stream ! rtph264depay ! h264parse ! avdec_h264 ! queue ! videoconvert ! glimagesink name=video-sink stream. ! rtpopusdepay ! opusdec ! queue ! audioconvert ! audioresample ! audio/x-raw,channels=1,channel-mask=(bitmask)0x1 ! openslessink stream-type=media name=audio-sink

There are also pipelines in the other direction, but I don’t believe they are relevant

ngoetting · February 19, 2024, 8:40pm

Now I got around to setup a minimal example. Consider the following pipeline as the sent stream from the raspberry

gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=360,framerate=24/1 ! queue ! x264enc ! matroskamux streamable=true name=mux ! tcpserversink host=127.0.0.1 port=5000

and the following minimal python script for RTSP with basic auth:

import sys
import gi

gi.require_version('Gst', '1.0')
gi.require_version('GstRtspServer', '1.0')
gi.require_version('GstRtsp', '1.0')
from gi.repository import Gst, GstRtsp, GstRtspServer, GLib

loop = GLib.MainLoop()
Gst.init(None)


class GstServer(GstRtspServer.RTSPServer):
    pipeline = "( tcpclientsrc host=127.0.0.1 port=5000 ! video/x-matroska ! matroskademux name=demux ! h264parse ! rtph264pay name=pay0 pt=96 )"
    def __init__(self, **properties):
        super(GstServer, self).__init__(**properties)
        self.set_address("0.0.0.0")
        self.set_service("8554")
        self.factory = GstRtspServer.RTSPMediaFactory()
        self.factory.set_launch(self.pipeline)
        self.factory.set_shared(True)

        self.permissions = GstRtspServer.RTSPPermissions()
        self.permissions.add_permission_for_role("user", "media.factory.access", True)
        self.permissions.add_permission_for_role("user", "media.factory.construct", True)
        self.factory.set_permissions(self.permissions)

        self.get_mount_points().add_factory("/test", self.factory)


server = GstServer()
auth = GstRtspServer.RTSPAuth()
token = GstRtspServer.RTSPToken()
token.set_string("media.factory.role", "user")
basic = GstRtspServer.RTSPAuth.make_basic("user", "password")
auth.add_basic(basic, token)
server.set_auth(auth)
server.attach(loop.get_context())
loop.run()

I can then just use ffplay rtsp://user:password@localhost:8554/test to connect to the stream and get

(python:3788): GStreamer-WARNING **: 21:33:52.606: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4731:gst_pad_push_data:<demux:video_0> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.607: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4458:gst_pad_chain_data_unchecked:<h264parse0:sink> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4731:gst_pad_push_data:<h264parse0:src> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4458:gst_pad_chain_data_unchecked:<pay0:sink> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4731:gst_pad_push_data:<pay0:src> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4458:gst_pad_chain_data_unchecked:<src_0:proxypad0> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4731:gst_pad_push_data:<bin0:src_0> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4458:gst_pad_chain_data_unchecked:<rtpbin0:send_rtp_sink_0> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4731:gst_pad_push_data:<send_rtp_sink_0:proxypad2> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4458:gst_pad_chain_data_unchecked:<rtpsession0:send_rtp_sink> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4731:gst_pad_push_data:<rtpsession0:send_rtp_src> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4458:gst_pad_chain_data_unchecked:<send_rtp_src_0:proxypad1> Got data flow before segment event

(python:3788): GStreamer-WARNING **: 21:33:52.608: ../gstreamer/subprojects/gstreamer/gst/gstpad.c:4731:gst_pad_push_data:<rtpbin0:send_rtp_src_0> Got data flow before segment event
**
GStreamer-RTSP-Server:ERROR:../gstreamer/subprojects/gst-rtsp-server/gst/rtsp-server/rtsp-media.c:2766:gst_rtsp_media_get_rates: assertion failed: (FALSE)
Bail out! GStreamer-RTSP-Server:ERROR:../gstreamer/subprojects/gst-rtsp-server/gst/rtsp-server/rtsp-media.c:2766:gst_rtsp_media_get_rates: assertion failed: (FALSE)

When I exchange the tcpclientsrc with a videotestsrc ! x264enc then the ffplay stream works fine.

tpm · February 24, 2024, 10:55am

Sounds like an issue in matroskademux.

What version are you using? (gst-inspect-1.0 matroskademux | grep Version)

ngoetting · February 26, 2024, 10:55am

It’s version 1.22.10. I’m pretty sure you’re right that it’s an issue with matroskademux as I reimplemented the whole thing with rtpbin and plain payloaded h264/opus in the meantime and get no error, even with authentication.