Low FPS when v4l2src is used with appsink

Application Development
1

I have uvcvideo base HDMI frame grabbers that support 1080p60, put I only get ~50FPS when I use the appsink element. It can be reproduced with this pipeline, which takes, most of the time, 1.3 seconds to complete where it should in 1.0:

time gst-launch-1.0 v4l2src num-buffers=60 ! video/x-raw,width=1920,height=1080,framerate=60/1 ! appsink wait-on-eos=false
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Got EOS from element "pipeline0".
Execution ended after 0:00:01.369419161
Setting pipeline to NULL ...
Freeing pipeline ...

real	0m1,521s
user	0m0,070s
sys	0m0,143s

A pipeline using fakesink or xvimagesink completes in ~ one second:

$ time gst-launch-1.0 v4l2src num-buffers=60 ! video/x-raw,width=1920,height=1080,framerate=60/1 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Got EOS from element "pipeline0".
Execution ended after 0:00:01.044376869
Setting pipeline to NULL ...
Freeing pipeline ...

real	0m1,185s
user	0m0,034s
sys	0m0,042s

Note1: Above i said, most of the time. When i invoke the first pipeline above multiple times, there are invocations that hit 60fps thruought the while runtime of the pipeline, even if that is an hour.

Note2: When the FPS is not good, e.g. ~50FPS I get these warnings

0:00:00.476362251 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.406577540
0:00:00.560462500 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.490682242
0:00:00.644543693 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.574755291
0:00:00.733278163 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.663498816
0:00:00.803333435 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.733546165
0:00:00.849977969 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.780196091
0:00:00.919770738 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.849990994
0:00:00.966305003 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.896526152
0:00:01.036061973 55641 0x55697ae3b8c0 WARN                 v4l2src gstv4l2src.c:1338:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.966283048

thanks

2

What do you observe with appsink drop=true max-buffers=1?

It might be worth trying with a quick python script or C/Rust app that pulls buffers out of the appsink (which you can’t really do in gst-launch-1.0.

Also, I would expect quite a bit of start-up delay, but if it works fine with fakesink then that’s not it since that should be independent of the sink being used.

3

@tpm there is no difference in with this pipeline:

time GST_DEBUG=2 gst-launch-1.0 v4l2src num-buffers=600 ! video/x-raw,width=1920,height=1080,framerate=60/1 ! appsink drop=true max-buffers=1 wait-on-eos=false

Still not 60fps, still “lost frames detected” message, still some invocations reaching 60fps without the lost frame messages.

I do actually pull out the buffers in a C++ application using (hopefully correct):

g_signal_emit_by_name(sink, "pull-sample", &sample, NULL);

which blocks ~20ms, assuming this locks me at ~50FPS.

The timed gst-launch-1.0 commands above are just for reproduction purposes.

I now switched to fakesink inside my C++ app using the handoff signal to retrive the buffers, which gives me 60FPS. Is using fakesink with the handoff signal OK, or does this lead to other troubles?

Thanks

4

@tpm I found a lead: in invocations where the buffers are dropped and the FPS is not reaching 60FPS i see these two additional messages at the beginning:

0:00:00.077417237 39166 0x55bb832858c0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1336:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool0:src> Driver should never set v4l2_buffer.field to ANY
0:00:00.077437363 39166 0x55bb832858c0 WARN                 v4l2src gstv4l2src.c:1255:gst_v4l2src_create:<v4l2src0> Timestamp does not correlate with any clock, ignoring driver timestamps

in invocations where these messages are missing there are no lost frame messages and the FPS reaches 60.

5

Curious. This driver behaviour should not be affected by what sink you use.

You can also try appsink sync=false btw, which should match fakesink behaviour then.

Or try appsink processing-deadline=0 (which removes an additional 20ms wait to accommodate for processing delays on top of the capture latency which should be 1/fps)

6

On a side note, I would recommend using the GstAppSink API from the gstapp library in this case, i.e. gst_app_sink_pull_sample(sink). Not that there’s anything wrong with your existing code though.