Wpesrc causing backwards dts when used with video source

Application Development
1

Hello, trying to use the pipeline below

gst-launch-1.0 -f \
	uridecodebin3 name=decode uri=file:///app/vid.mp4 ! queue2 ! videoconvert ! videoscale  ! video/x-raw,width=1920,height=1080 ! \
		compositor name=comp  ! videoconvert ! x264enc bitrate=5120 tune=zerolatency speed-preset=veryfast key-int-max=30 ! queue2 ! flvmux name=mux streamable=TRUE ! rtmp2sink location=rtmp://localhost:1936/live/live \
	decode. ! queue2 ! audioconvert ! avenc_aac ! aacparse ! mux. \
	wpesrc location=http://localhost draw-background=FALSE ! video/x-raw,width=1920,height=1080,framerate=\(fraction\)30/1  ! videoconvert ! queue2 ! comp.

The main idea is that I’m trying to stream a video that has audio content while adding a web source overlay. So we combine the video branch from the decodebin with the wpesrc using the compositor element. However when passing the video + audio content to the flvmux, we get the following:

0:00:00.206039225  4444 0x60c6bc5968c0 FIXME             aggregator gstaggregator.c:1497:gst_aggregator_loop:<mux> Subclass should call gst_aggregator_selected_samples() from its aggregate implementation.
0:00:00.260199404  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.133000000 < 0:00:00.139000000)
0:00:00.325690836  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.200000000 < 0:00:00.208000000)
0:00:00.393012018  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.266000000 < 0:00:00.278000000)
0:00:00.434020596  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.300000000 < 0:00:00.301000000)
0:00:00.495217744  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.366000000 < 0:00:00.371000000)
0:00:00.556334828  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.433000000 < 0:00:00.441000000)
0:00:00.637664059  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.500000000 < 0:00:00.510000000)
0:00:00.661593407  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.533000000 < 0:00:00.534000000)
0:00:00.692337634  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.566000000 < 0:00:00.580000000)
0:00:00.722230563  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.600000000 < 0:00:00.603000000)
0:00:00.768306195  4444 0x60c6bc5968c0 WARN                  flvmux gstflvmux.c:1296:gst_flv_mux_buffer_to_tag_internal:<mux:video> Got backwards dts! (0:00:00.633000000 < 0:00:00.650000000)

I believe the muxer is not able to properly combine the video and audio content with the correct timestamps after the video+websource got passed to the compositor? Any idea how can I navigate this?

Edit:
I was able to solve this by delaying linking the wpe branch to the compositor for a few seconds in my rust code. Though the same problem happens when the video finishes playing and I switch to another video by updating the uri parameter on the uridecodebin3 on the about-to-finish signal. Thoughts?

snippet of the signal handler


        uridecodebin.set_property("instant-uri", true);
        let uridecodebin = Arc::new(uridecodebin);
        let decodebin_clone = Arc::clone(&uridecodebin);
        uridecodebin.connect("about-to-finish", true, move |data| {
            println!(
                "================= About to finish ================ {:?}",
                data
            );
            decodebin_clone.set_property("uri", FILE2_PATH);
            None
        });
2

I think this is happening because the wpevideosrc element waits until the first buffer is ready… this should be accounted in the latency but isn’t…

The workaround could be to set force-live=1 latency=100000000 on the compositor.

3

Also try ignore-inactive-pads=1 on the compositor…

4

Thank you! Tried that and it kinda worked for the first time video, but after I change the uri on the uridecodebin3, the video gets fast-forwarded until the end time of the previous video.
I’ve recorded the experiment here

And this is the source code for the experiment

Here I play a video and add the wpesrc as an overlay, and when the video is about to finish (listening to the about-to-finish signal and updating the uri), the pipeline kinda goes haywire. Not sure if there’s something that can be done there.

5

Is the rtmp stream playable anyway? Do you notice any visual issue?

6

Yes it’s playable, but whenever the file changes, the image gets kinda fast-forwarded to the last second that was played in the previous video.
For example if the first video is 10 seconds, it finished and then another video gets played, the first 10 seconds of the second video move very fast until it reaches the 10 sec mark at which the warning also disappears.