Seeking causes whipclientsink to reset PTS timestamp, regardless of video identity single-segment

Application Development
1

I have an application that reads video (h264) and audio (opus) streams from an mp4 file and sends them, without any kind of transcoding, via whipclientsink, to a MediaMTX WHIP endpoint. Despite the fact that my h264 stream does not have any B-frames, seek events cause MediaMTX to think that the stream suddenly has B-frames and kill the WebRTC connection.

(I am aware that this is an unusual setup. However, streaming pre-encoded video via RTC while seeking is non-negotiable. I am positive that there is nothing wrong with my video’s encoding settings.)

I am seeking using:

gst_element_seek_simple(
    pipeline,
    GST_FORMAT_TIME,
    GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_KEY_UNIT | GST_SEEK_FLAG_SNAP_BEFORE,
    new_position
);

After some debugging, the apparent cause of this is that whipclientsink (probably a rtph264pay element inside whipclientsink’s internal pipeline but I can’t confirm this) seems to restart PTS timestamps from segment running time. Is this expected behavior in webrtcsink/whipclientsink after a seek? If so, is there a way to disable this?

I know that this is a PTS reset issue due to the following code in MediaMTX triggering the error:

if !firstReceived {
    firstReceived = true
} else if u.PTS < lastPTS {
    return fmt.Errorf("WebRTC doesn't support H265 streams with B-frames")
}

My pipeline is, roughly (and without audio):

filesrc
  ↓
qtdemux
  ↓
queue2
  ↓
h264parse
  ↓
whipclientsink
  ↓
rtph264pay0
  ↓
WebRTC
  ↓
WHIP
  ↓
MediaMTX
  ↓
Browser
2

Yes, a flushing seek resets running time to 0.

Have you tried if a non-flushing seek works for your purposes?

Means data in the queue will still be sent out though. If you use a multiqueue that should keep the data in the queue as small as possible.

3

Fascinating. That does fix it, although if I seek while paused, it won’t start playing again.

I’m very unfamiliar with gstreamer so I’m not sure if I’m supposed to be handling paused seeks in some special way or something.

4

If your pipeline is paused the seek would be processed once you go into playing again.

With a flushing seek it would throw away all data in the pipeline and the pipeline would re-preroll again (fill the pipeline with data and end up in paused state ready to go), so that the first buffer is then from the new position.

A paused pipeline in a webrtc scenario seems somewhat unusual though.

5

Right, I’m aware that I’m using a webconferencing protocol for a low-latency watch party application, which is very much not what webrtc was intended for.

If I simply can’t make a paused pipeline work with webrtc, I suppose I could just create a fake paused state that mutes the audio and continuously loops one frame, although I’m not sure how I’d even begin doing that.

6

Do you need scalability here, or would adding decoding/encoding be acceptable? (Would make things considerably easier implementation wise at least)

7

The whole point of this project is to avoid the need for transcoding. The idea is that you could set this up on a dirt cheap 1 core VPS and have it work just fine.
If it’s “decoding exactly one frame while fake-paused”, then that might be acceptable, I’d have to test it out.

8

Right, if the goal is to pass through encoded data as-is and support pausing, I think you have two options:

  1. Handle pausing/freezing on the client (browser?) side by e.g. freezing the last frame. Maybe tear down and reinstate the webrtc session if the pausing is too long.
  2. Handle it on the sender side. I think that would require two pipelines, or two bins in a pipeline, where one pipeline is the webrtc sender and always continues. When you pause you would pause at a keyframe and then keep re-sending the encoded keyframe on the sender pipeline while keeping the “producer pipeline or bin” paused. This would require some kind of timestamp offsetting when you resume the producer and while you re-inject the same keyframe over and over again. Perhaps it could be done application side via appsinks (on the producer) and the application forwarding buffers to the sender/webrtc pipeline via appsrc elements or somesuch.
    .