Issues with Seeking and vp9alphadecodebin (on retimestamped buffers)

Hi,

I’m encountering some issues with seeking and timestamp modifications on VP9 buffers when using vp9alphadecodebin.

Context:

  • I’ve implemented a custom GStreamer source bin that wraps a filesrc and matroskademux. It modifies the PTS/DTS of buffers to use Unix Epoch time on the matroskademux source pad.
  • This bin is used in a pipeline where I send frame-accurate seek events (in Unix timestamps) to process specific time intervals.
  • The setup works perfectly for:
    • H.264 buffers with avdec_h264.
    • VP9 buffers with vp9dec (but this loses alpha transparency, which I need).

Problem: When using vp9alphadecodebin, the pipeline hangs during seeks - no buffers are passed to the srcpad on vp9alphadecodebin and I fail to query the pipeline state.
I suspect this may be due to a queue filling up, but I see no obvious warnings or errors in the debug logs. If I skip the timestamp modifications on both buffers and seek events, the pipeline works fine. This to me suggests that the issue is related to the timestamp handling

Observations:

  1. Using vp9alphadecodebin, I receive two seek events with the same seqnum, whereas with other decoders, this does not happen. I currently ignore the second event.
  2. The problem seem to only occurs when modifying timestamps.

Implementation Details:

  • Seek Handling: On the srcpad, I intercept seek events, perform an internal seek (sending flush-start/stop), and dynamically add the filesrc-matroskademux elements for the requested time. After receiving the stream-start event, I forward the intercepted seek segment event downstream. Repeated seek events with the same seqnum are ignored.
  • Buffer Retimestamping: I adjust the PTS/DTS of buffers on the matroskademux source pad.

Questions:

  • Is there something specific to vp9 alpha buffers or vp9alphadecodebin that requires special handling during timestamp modification or seek events?
  • Does the alpha metadata also need some special modification other than setting pts/dts on the buffer?
  • Any other suggestions to get around my problem?

Pipeline:

compositor name=mixer ! videoconvert ! videoconvert ! x264enc ! queue ! mpegtsmux ! filesink location=out.ts bpfdemuxbin uri=playlist name=demuxer ! queue name=queue ! vp9parse name=parser ! vp9alphadecodebin name=dec ! videoconvert ! mixer.    

Bpfdemuxbin is the custom bin that parses a playlist and wraps the filesrc-matroskademux

Any help is greatly appreciated!

Edit: Just probed the branch responsible for handling the alpha and noticed the timestamps are not changed. So feels like the issue is that these are not properly re-timestamped.

About the two seek events, that’s because of the split happening in the bin to separate the color stream from the alpha stream. I don’t know for the rest, seems a bit specific to your software.

Demuxers usually cope with these duplicated seek, as you properly identified, the sequence number is there to allow deduplication.

Thanks for explaining the two seek events and how those should be handled.

Regarding the issue, I have boiled it down to the fact that the alpha stream does not get re-timestamped in my demuxer. I have a src pad probe on a matroskademuxer and re-write the pts/dts on the buffers:

if let Some(PadProbeData::Buffer(ref mut buffer)) = info.data {
    let buffer = buffer.make_mut();
    buffer.set_pts(new_pts);
    buffer.set_dts(new_pts);
}

The alpha buffers - which exist as an metadata on the color buffers videocodecalphameta - does not get re-timestamped.

When re-writing those, the issue is fixed. I don’t seem to get a mutable access to the buffer from safe rust api, so I am using unsafe here.

if let Some(mut meta) = buffer.meta_mut::<gst_video::video_meta::VideoCodecAlphaMeta>()
   {
   // Adjust the pts and dts of the alpha buffer.
    unsafe {
        let alpha = meta.as_mut_ptr();
        let buf = (*alpha).buffer;
        (*buf).pts = *adjusted_pts;
        (*buf).dts = *adjusted_pts;
   }
}

Perhaps it is possible using the safe rust-bindings? And is it correct that one should handle the alpha stream seperately like so or should the alpha pts be modfied when I change the pts on the parent buffer?

I’d be fine if codecalphademux ignores the alpha buffer timestamp and simply override it. I’d do that only if it’s different since otherwise it introduces malloc while making the GstBuffer writable.