Issues with Seeking and vp9alphadecodebin (on retimestamped buffers)

Application Development
1

Hi,

I’m encountering some issues with seeking and timestamp modifications on VP9 buffers when using vp9alphadecodebin.

Context:

  • I’ve implemented a custom GStreamer source bin that wraps a filesrc and matroskademux. It modifies the PTS/DTS of buffers to use Unix Epoch time on the matroskademux source pad.
  • This bin is used in a pipeline where I send frame-accurate seek events (in Unix timestamps) to process specific time intervals.
  • The setup works perfectly for:
    • H.264 buffers with avdec_h264.
    • VP9 buffers with vp9dec (but this loses alpha transparency, which I need).

Problem: When using vp9alphadecodebin, the pipeline hangs during seeks - no buffers are passed to the srcpad on vp9alphadecodebin and I fail to query the pipeline state.
I suspect this may be due to a queue filling up, but I see no obvious warnings or errors in the debug logs. If I skip the timestamp modifications on both buffers and seek events, the pipeline works fine. This to me suggests that the issue is related to the timestamp handling

Observations:

  1. Using vp9alphadecodebin, I receive two seek events with the same seqnum, whereas with other decoders, this does not happen. I currently ignore the second event.
  2. The problem seem to only occurs when modifying timestamps.

Implementation Details:

  • Seek Handling: On the srcpad, I intercept seek events, perform an internal seek (sending flush-start/stop), and dynamically add the filesrc-matroskademux elements for the requested time. After receiving the stream-start event, I forward the intercepted seek segment event downstream. Repeated seek events with the same seqnum are ignored.
  • Buffer Retimestamping: I adjust the PTS/DTS of buffers on the matroskademux source pad.

Questions:

  • Is there something specific to vp9 alpha buffers or vp9alphadecodebin that requires special handling during timestamp modification or seek events?
  • Does the alpha metadata also need some special modification other than setting pts/dts on the buffer?
  • Any other suggestions to get around my problem?

Pipeline:

compositor name=mixer ! videoconvert ! videoconvert ! x264enc ! queue ! mpegtsmux ! filesink location=out.ts bpfdemuxbin uri=playlist name=demuxer ! queue name=queue ! vp9parse name=parser ! vp9alphadecodebin name=dec ! videoconvert ! mixer.

Bpfdemuxbin is the custom bin that parses a playlist and wraps the filesrc-matroskademux

out
out10781×650 450 KB

Any help is greatly appreciated!

2

Edit: Just probed the branch responsible for handling the alpha and noticed the timestamps are not changed. So feels like the issue is that these are not properly re-timestamped.

3

About the two seek events, that’s because of the split happening in the bin to separate the color stream from the alpha stream. I don’t know for the rest, seems a bit specific to your software.

Demuxers usually cope with these duplicated seek, as you properly identified, the sequence number is there to allow deduplication.

4

Thanks for explaining the two seek events and how those should be handled.

Regarding the issue, I have boiled it down to the fact that the alpha stream does not get re-timestamped in my demuxer. I have a src pad probe on a matroskademuxer and re-write the pts/dts on the buffers:

if let Some(PadProbeData::Buffer(ref mut buffer)) = info.data {
    let buffer = buffer.make_mut();
    buffer.set_pts(new_pts);
    buffer.set_dts(new_pts);
}

The alpha buffers - which exist as an metadata on the color buffers videocodecalphameta - does not get re-timestamped.

When re-writing those, the issue is fixed. I don’t seem to get a mutable access to the buffer from safe rust api, so I am using unsafe here.

if let Some(mut meta) = buffer.meta_mut::<gst_video::video_meta::VideoCodecAlphaMeta>()
   {
   // Adjust the pts and dts of the alpha buffer.
    unsafe {
        let alpha = meta.as_mut_ptr();
        let buf = (*alpha).buffer;
        (*buf).pts = *adjusted_pts;
        (*buf).dts = *adjusted_pts;
   }
}

Perhaps it is possible using the safe rust-bindings? And is it correct that one should handle the alpha stream seperately like so or should the alpha pts be modfied when I change the pts on the parent buffer?

5

I’d be fine if codecalphademux ignores the alpha buffer timestamp and simply override it. I’d do that only if it’s different since otherwise it introduces malloc while making the GstBuffer writable.