D3d11 pipeline, sync property and jitter on frame times

Application Development
1

Hi there,

I need your help. I’m working with a friend on an spare time art project using a d3d11 pipeline doing animation by rotating a video to be synchronous to a moving machine.

With your help this has progressed really far (Thx a lot!) but now I’m facing a new challenge.

The display is connected by HDMI is capable of doing 60 Hz refresh rate. When keeping the rendering in default settings (sync=true for d3d11videosink the rendering unfortunately is not smooth enough and visibly stuttering.

I added a handler to be called before rendering each frame like this:

// Add pad probe to the sink pad

gst_pad_add_probe(sinkpad, GST_PAD_PROBE_TYPE_BUFFER, pad_probe_cb, pipeline, NULL);

In this callback we adjust the angle by which the video shout be rotated (update-on-redraw=false). Here I added also capturing a time stamp. We see a significant difference from when sync=true vs false. False, has a perfectly smooth rendering. This also reflects looking at interval this call back is called.

55 FPS video with sync has nearly all the frames taken 17 and 18 ms.
sync=false

While with sync on 200 frames (about 15% get a big outlier to 30ms)

sync=true

This is very likely causing the stuttering video. Unfortunately I cannot easily turn of sync, because the adjustment the playback speed using gst_element_seek is not working any more. This is needed to keep the video speed in sync with (slightly varying speed of the machine).

I tried by now this:

  • With a 30 FPS video the outliers are also there (even with the frame rate far away from display limit).
  • Using 30 FPS and “update-on-redraw=true. Setting the property seems to be blocking on something (sync with frame?) as the execution time carries an even larger jitter
  • The “videorate” plug-in also generates a stuttering (just forcefully dropping frames?)

The pipeline is done in C/C++ but similar to this (angle dynamically changes).

gst-launch-1.0 filesrc location=“…” ! qtdemux name=demuxer demuxer.video_0 ! d3d11h264dec ! queue ! d3d11videosink fullscreen=false rotation-z=15.0 msaa=2 fullscreen-toggle-mode=4 redraw-on-update=false scale-x=1.4 scale-y=1.4 sync=false

Can you help me with possible approaches? Ideas I can imagine:

  • Is it possible to remove the jitter on sync=true?
  • Are there ways to adjust the playback speed of the video even with sync=false (sync=false has here by far superior looking results at the moment)
  • Any other ideas?

Your help would be greatly appreciated to bring this effort over the finish line.

Alex

2

Hi there,

just a short update how 've managed to solve this:

In my use case the video does not have any audio. I exploited that.

For the video loop I’m using I’ve created jpg files for each individual frame. Thankfully the video loop is short and all can be loaded into RAM as textures. These textures are then rotated and one by one rendered.

The v-sync here is done using the SDL framework. This is subject to only very little jitter on the frame times:

image

Not sure whether I was doing something wrong or there is an issue with vsync handling in GStreamer?

Just noticed my original post had a small but important word missing: the frame time w/o jitter is with sync==off.

Best regards
Alex