D3d11compositor with different frame rates

Application Development
1

Hello

I have been using the d3d11compositor.
I think it has great performance for multi views with network cameras.

But I have been encountered a problem.
When views with different frame rates are composited, all views have the lowest frame rate.

I would like to display views with each frame rate, how should I do?

It seems that if changing the frame rate lower midway, the same frame will be copied and it will work better, but I would like to display it at a different frame rate from the beginning.

Best regards,
Kamiya

2

videoaggregator subclasses (e.g., d3d11compositor) requires fixed output framerate, and will produce output frames with the configured framerate interval. Once output framerate is configured, input side change (caps update, pad add/remove) might not trigger output framerate update, but can be updated by downstream reconfigure.

I would like to display views with each frame rate, how should I do?

As mentioned the above, you will need to control output framerate of compositor explicitly. d3d11compositor ! capsfilter and setting caps (with high enough framerate) to the capsfilter is a way.

(The caps should be dynamically configurable even in the middle of streaming but if it does not work, please file an issue with an example code)

Then, compositor output with high enough framerate would be effectively/visually
similar to displaying each view with its original framerate.

1 Like
3

Hello seungha

Thank you for your reply.
I am happy to talk with you.

I tried your suggestion “d3d11compositor ! capsfilter”, but it did not work well.

I can get lower the frame rate with capsfilter.
But I can not get higher the frame rate with capsfilter.
It works the same whether I set the capsfilter at the beginning or in the middle.

My source code is multifunctional, So I would like to ask you again after making a simple example.

Thank you.
Kamiya

4

Hello seungha

I found one cause in the process of simplifying my source code.
My pipeline is the below.

gst-launch-1.0 d3d11compositor name=m `
		sink_0::xpos=0 sink_0::ypos=0 sink_0::width=640 sink_0::height=320 `
		sink_1::xpos=640 sink_1::ypos=0 sink_1::width=640 sink_1::height=320 `
		sink_2::xpos=0 sink_2::ypos=320 sink_2::width=640 sink_2::height=320 `
		sink_3::xpos=640 sink_3::ypos=320 sink_3::width=640 sink_3::height=320 `
		! d3d11videosink `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videorate max-rate=30 ! videocrop ! d3d11convert ! m.sink_0 `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videorate max-rate=30 ! videocrop ! d3d11convert ! m.sink_1 `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videorate max-rate=30 ! videocrop ! d3d11convert ! m.sink_2 `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videorate max-rate=30 ! videocrop ! d3d11convert ! m.sink_3

The problem does not occur in the command, but it occurs in the execution of the source code.

I tried removing the videorate. then it worked well.

I had used the videorate for lowering the rate when the PC specifications are insufficient.
Instead, I replaced the videorate with “d3d11compositor ! capsfilter” which you suggested.

I don’t know the reason, but I think I’ll move forward with this.
Please let me know if you notice anything.

Thank you for your advice.
Kamiya

5

videorate elements in front of compositor were lowering framerate already so setting higher framerate after compositor would not work as intended. So yes, removing the videorate is right approach. compositor will behave as if it’s videorate.

note that d3d11convert elements in front of compositor seems to be redundant unless you are doing 3D transformation (rotation for example)

6

Hello seungha

Thank you for your comments.
I can remove the videorate with relieving.

I am using d3d11convert for resizing view.
Because I found that it is better to reduce jaggies with d3d11convert than with d3d11compositor.

I had achieved a similar effect using texture Mipmap and anisotropic filtering of Direct3D9EX in the past.
So I supporse that d3d11convert also includes similar processing. The d3d11convert is great.

I forgot that there was capsfilter behind d3d11convert, so I updated the pipeline information.

gst-launch-1.0 d3d11compositor name=m `
		sink_0::xpos=0 sink_0::ypos=0 sink_0::width=640 sink_0::height=320 `
		sink_1::xpos=640 sink_1::ypos=0 sink_1::width=640 sink_1::height=320 `
		sink_2::xpos=0 sink_2::ypos=320 sink_2::width=640 sink_2::height=320 `
		sink_3::xpos=640 sink_3::ypos=320 sink_3::width=640 sink_3::height=320 `
		! capsfilter ! d3d11videosink `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videocrop ! d3d11convert ! capsfilter ! m.sink_0 `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videocrop ! d3d11convert ! capsfilter ! m.sink_1 `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videocrop ! d3d11convert ! capsfilter ! m.sink_2 `
	rtspsrc location=***** ! rtph265depay ! h265parse ! d3d11h265dec ! videocrop ! d3d11convert ! capsfilter ! m.sink_3

Unfortunately this effect seems to have disappeared in my latest code, so I might ask it in another topic if needed.

Thank you.
Kamiya.