Gapless playback with multiple uridecodebin3 elements in a pipeline

Application Development
1

Hi!

Almost complete GStreamer beginner here. I’m trying to build a live video mixer for an art project where I want to be able to mix 2 video clips and later replace them individually.

Ideally the videos should be replaced on the fly so that when one is replaced, the other keeps playing. After reading through the docs, I went with using a couple of uridecodebin3 elements. Both also have instant-uri flag set to True, so I don’t have to restart the pipeline when replacing either clip.

My pipeline looks like this at the moment:

pipeline-all
pipeline-all11660×1571 1.34 MB

However, as I keep reloading videos on either uridecodebin3, it gets progressively slower, i.e. after 4-5 changes to uri property on the “foreground” decodebin, the playback freezes for more than 1 second (looks like entire pipeline freezes, as both videos are paused).

Another problem is that newly loaded videos aren’t played from the start, but rather get fast-forwarded in sync with the “background” uridecodebin3.

Neither of this happens if I only have one uridecodebin3 in the pipeline - switching clips is fast and smooth, new videos are always played from the beginning.

Is there anything I’m missing in terms of pipeline construction and/or logic around clip reloading (i’m only setting uri property on uridecodebins at the moment) that would fix the issue?

Is it actually allowed to have multiple uridecodebin3s in the same pipeline?

On a slightly offtopic matter, what elements would I normally put after each decodebin, to normalize the videos to the same size/format/framerate? I currently have videoscale with capsfilter, but it wouldn’t center the video or zoom it to fill the canvas entirely.

I’ve got GStreamer version 1.26.1, if that matters.

2

My first guess would be that what happens is something like this:

  • if you change the inputs while the pipeline is playing, the timestamps coming out of the uridecodebin3 will start from running time 0 again, since they come from a file (unlike a live input which is timestamped based on the pipeline clock).
  • The compositor will mix inputs based on the current running time, which started at 0 when the pipeline started up, and is at whatever position it’s at when you switch inputs (if the compositor output goes to a videosink, the videosink will output in realtime and block the compositor output, so compositor will operate roughly in real-time and not as fast as possible. So the compositor is probably slightly ahead of the video sink running time wise.
  • Long story short, if you play for 30secs the compositor is at roughly 30secs+ running-time wise, and expects inputs to be at that position too, and will discard any inputs that are before that.
  • How do you fix that? You can query the compositor for the current position and then use gst_pad_set_offset() on a source pad before the compositor input to move the new file input to the new position. Might be a bit tricky though if it’s “instantanous” since you’d have to make sure you do that only once the input switched over to the new file (which I would expect to be announced with a stream-start event and a segment event, but I didn’t check).

Another thing you could do, which might be easier to manage:

  • use three separate pipelines
  • pipeline 1: have two intervideosrc channel=inputX that feed into the compositor and your output
  • pipeline 2: uridecodebin3 ! videoconvert ! intervideosink channel=input1 (or playbin3 with video-sink=intervideosink)
  • pipeline 3: same, but intervideosink channel=input2

Then you just keep pipeline 1 running, and you can stop / shutdown / pause / seek / reverse-play pipeline 2 and 3 as/when you like.

Lastly, another possibility could be to do output to two sinks and do the compositing/mixing GTK-side (I don’t know how easy/possible it is, just something to check - maybe someone else knows).

1 Like
3

Thanks for the suggestions. All look like worthy leads to explore! Compositing on the GTK side doesn’t sound bad either, as there might be more possibilities for blending.

4

I have been working on a similar problem, here’s the approach that worked for me when you want to replace vid 1 with vid 2:

  1. Preroll video2, that means, create a uridecodebin3 branch up to the videoscale2 that you have.
  2. add a blocking pad on the videoscale2 sink pad
  3. once you receive an event inside the blocking pad, that means that branch is prerolled.
  4. request a new sink pad from the compositor, adjust the offset of the pad to be the current running time of the compositor. Link the videoscale2 with the new compositor pad.
  5. Remove the blocking pad so data will now flow
  6. Set vid1’s branch state to null, release it’s compositor sink pad, remove from the pipeline

Another way you can do it is to use the concat element, which basically does the same thing internally.

1 Like