RTPS to mp4, h264parse dropping buffer PTS

Application Development
1

Hi all, I’m fairly new to Gstreamer so interested in solving my problem and any suggested debugging approaches. I have some approaches that seem to work but I’m not really sure they are right, so looking for some advice :slightly_smiling_face:

What I’m trying to do is pull from an RTSP video stream (from a remote IP camera), h264 encoded, and write to MP4 without decoding / re-encoding the video.

My pipeline looks like this:

pipeline = Gst.parse_launch(                                                                      
    f"rtspsrc location={url} ! "                                                                  
    "rtph264depay ! video/x-h264,stream-format=avc,alignment=au ! "                                
    "h264parse name=parser ! "                                                                
    "mp4mux ! "                                                                                   
    "filesink location=foo.mp4"                                                                  
)

When there are network constraints (and I can recreate this by tethering my laptop to my phone on 4G) I will at a some point get an error from muxer:

Could not multiplex stream
../gst/isomp4/gstqtmux.c(5402): gst_qt_mux_add_buffer (): /GstPipeline:pipeline0/GstMP4Mux:mp4mux0:
Buffer has no PTS.

I’ve added buffer probes to both the the sink and src pads on h264parse and what I see is that usually there are pts on the buffer going in and out (the same values) but at some point when the bandwidth is constrained I will start to get successive buffers on the sink with identical pts, and the then the buffers on the src will have Gst.CLOCK_TIME_NONE.

These are things I’ve tried:

  • Remove h264parse entirely – then I get a pipeline error streaming stopped, reason not-negotiated (-4) unless I add width=1280,height=720 to the caps filter.
    • In this case I still see the repeated pts values (probing an identity in place of the h264parse) but the muxer seems to happy and the resulting MP4 is playable.
    • In my test-case I know the width and height so this is OK, but in practise won’t always, so not sure it’s the right solution.
  • Using the src probe on the parser to drop any buffers with pts == Gst.CLOCK_TIME_NONE.
    • This also seems to work and produce a playable mp4, but I’m not sure it’s “the right answer”?
  • I’ve tried different options in the caps filter (e.g stream-format=byte-stream,alignment=nal) to see if that alters the behaviour of h264parse in some way, but doesn’t seem to make a difference.

My aim is to have a playable MP4, of the best quality I can achieve, accepting that there is some unavoidable degradation if the bandwidth is limited. Any advice appreciated, thanks!

1 Like
2

mp4mux wants proper dts/pts on the input buffers, but that’s not always available when coming from RTP. Theoretically it would be h264parse’s job to fix that up, but in practice it doesn’t do that (yet), so there’s a separate h264timestamper element which should hopefully reconstruct and set pts/dts correctly. So give that a try perhaps.

The other question is what happens when there are network constraints? is it just that you get framedrops and that causes it, or does the sender perhaps switch to a different video resolution/framerate?

2 Likes
3

Thanks, I’ve just tested that out and it seems to be working! Thanks so much!

I somehow completely missed that element as it’s not in the version Gstreamer version I have installed locally – danger of not keeping my system updated :man_facepalming: , but I do have 1.22 in a Docker container and it’s available there.

I’ve added h264timestamper between the h264parse and mp4mux, and using the probe I can see buffers with no PTS coming out of h264parse, and being fixed by h264timestamper :tada:

I’m not too sure what is happening when the network is constrained. I’d assumed the source is somehow dropping the bitrate. I think the framerate and resolution is fixed but it’s possible I’m getting frames dropped too. Certainly watching back the MP4 I get a moment of static video around these points where this is happening. I’ll have to dig into this some more, but in practice I’m not too worried in this case as (other than tethering to 4G) it only happens quite infrequently.