Decoder cache 3 frames and introduce a latency of 3 frames

markus · March 8, 2025, 9:01am

Hi there dear GStreamer Community!

I’m transferring a video stream via TCP from one PC to another. I’m using gstreamer 1.24. The Stream gets encoded using to h265/hevc by the camera (hw) itself Rgb Encoding which makes it quite easy on compute-resources.

My goal is to decode that stream AS FAST as possible with the LOWEST possible latency.

tcpclientsrc host=robot-car port=8554 timeout=10
  ! application/x-rtp-stream
  ! rtpstreamdepay
  ! application/x-rtp,media=video,clock-rate=90000,encoding-name=H265
  ! rtph265depay
  ! queue
  ! h265parse config-interval=-1
  ! video/x-h265,stream-format=hvc1,alignment=au
  ! queue
  ! vah265dec qos=false
  ! video/x-raw(memory:DMABuf),drm-format=NV12
  ! queue max-size-buffers=1 leaky=1
  ! glimagesink sync=false qos=false

Using the environment variables

GST_TRACERS="latency(flags=pipeline+element)"
# GST_TRACERS="latency(flags=pipeline+element+reported)"
GST_DEBUG_FILE=trace.log
GST_DEBUG="1,GST_TRACER:7"

I’m able to detect that vah265dec is reporting and actually using 100ms for a framerate of 30/1. Which indicates to me, that the 100ms/3 frames-caching is happening by design.

Time: 0:00:09.549951361

Latency Statistics:
        0x55c61a9124d0.tcpclientsrc0.src|0x55c61a9023b0.glimagesinkbin0.sink: mean=0:00:00.100234675 min=0:00:00.050495205 max=0:00:00.121120488
        0x55c61a9124d0.tcpclientsrc0.src|0x55c61aaf81d0.sink.sink: mean=0:00:00.100317872 min=0:00:00.050545080 max=0:00:00.121196562

Element Latency Statistics:
        0x55c61aab37f0.capsfilter0.src: mean=0:00:00.000006225 min=0:00:00.000003737 max=0:00:00.000566180
        0x55c61aa24230.rtpstreamdepay0.src: mean=0:00:00.000018388 min=0:00:00.000011602 max=0:00:00.000102474
        0x55c61ab05ea0.capsfilter1.src: mean=0:00:00.000009136 min=0:00:00.000002785 max=0:00:00.000039395
        0x55c61a80ebf0.rtph265depay0.src: mean=0:00:00.000035421 min=0:00:00.000019036 max=0:00:00.001843301
        0x55c61a90bbd0.queue0.src: mean=0:00:00.000013551 min=0:00:00.000008907 max=0:00:00.000028313
        0x55c61aa337f0.h265parse0.src: mean=0:00:00.000118532 min=0:00:00.000057499 max=0:00:00.001058128
        0x55c61ab02100.capsfilter2.src: mean=0:00:00.000013271 min=0:00:00.000008676 max=0:00:00.000036559
        0x55c61a880640.queue1.src: mean=0:00:00.000020159 min=0:00:00.000008676 max=0:00:00.001672268
        0x55c61ab7d2e0.vah265dec0.src: mean=0:00:00.099842792 min=0:00:00.049800864 max=0:00:00.120715323
        0x55c61a6ff020.capsfilter3.src: mean=0:00:00.000017780 min=0:00:00.000007975 max=0:00:00.000036318
        0x55c61ab6b040.queue2.src: mean=0:00:00.000084689 min=0:00:00.000015750 max=0:00:00.000448497
        0x55c61a910a00.gluploadelement0.src: mean=0:00:00.000059767 min=0:00:00.000031881 max=0:00:00.000378325
        0x55c61a884b60.glcolorconvertelement0.src: mean=0:00:00.000011727 min=0:00:00.000008746 max=0:00:00.000036179
        0x55c61aae5390.glcolorbalance0.src: mean=0:00:00.000011701 min=0:00:00.000008977 max=0:00:00.000027973

Element Reported Latency:
        0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451577889
        0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451589761
        0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451596213
        0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451604469
        0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.451613756
        0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451619778
        0x55c61aa337f0.h265parse0: min=0:00:00.033333333 max=0:00:00.000000000 ts=0:00:00.451625208
        0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452703745
        0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452719985
        0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452726798
        0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452732198
        0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.452737779
        0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452743360
        0x55c61aa337f0.h265parse0: min=0:00:00.033333333 max=0:00:00.000000000 ts=0:00:00.452748509
        0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452753499
        0x55c61a880640.queue1: min=5124095:34:33.676218283 max=0:00:00.000000000 ts=0:00:00.452758789
        0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453004593
        0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453018700
        0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453027326
        0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453034800
        0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.453055419
        0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453063274
        0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453078442
        0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454061529
        0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454071748
        0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454077459
        0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454082459
        0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.454087879
        0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454097597
        0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454106734
        0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454113537
        0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454120901
        0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.454126211
        0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525850890
        0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525866730
        0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525873834
        0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525879454
        0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.525885065
        0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525894312
        0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525904942
        0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525912256
        0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525902348
        0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525918067
        0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.525935610
        0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525935650
        0x55c61a6ff020.capsfilter3: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525947172
        0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525956400
        0x55c61ab6b040.queue2: min=5124095:34:33.609551616 max=0:00:00.000000000 ts=0:00:00.525961940
        0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525968623
        0x55c61a9023b0.glimagesinkbin0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525975436
        0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.525983551
        0x55c61a910a00.gluploadelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525988240
        0x55c61a884b60.glcolorconvertelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525995904
        0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525999291
        0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526016032
        0x55c61aae5390.glcolorbalance0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526003128
        0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526024538
        0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526037162
        0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.526045558
        0x55c61a6ff020.capsfilter3: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526053062
        0x55c61ab6b040.queue2: min=5124095:34:33.609551616 max=0:00:00.000000000 ts=0:00:00.526060767
        0x55c61a9023b0.glimagesinkbin0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526068592
        0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526069313
        0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526082778
        0x55c61a910a00.gluploadelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526076697
        0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526089962
        0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526096905
        0x55c61a884b60.glcolorconvertelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526099730
        0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.526102175
        0x55c61aae5390.glcolorbalance0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526108878
        0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526117704
        0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526127473
        0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526132993
        0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526138263
        0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.526143343
        0x55c61a6ff020.capsfilter3: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526148242
        0x55c61ab6b040.queue2: min=5124095:34:33.609551616 max=0:00:00.000000000 ts=0:00:00.526153342
        0x55c61a9023b0.glimagesinkbin0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526158511
        0x55c61a910a00.gluploadelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526164553
        0x55c61a884b60.glcolorconvertelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526171486
        0x55c61aae5390.glcolorbalance0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526177688

I’m currently trying very hard to shake that latency of 100ms down to ~0ms
Has anyone any idea WHY vah265dec is doing so. I also tried:

avdec_h265
libde265dec
vaapih265dec

all with the same result.

I also tried a all-in-one pipeline

#!/usr/bin/env bash

set -e

(
  export GST_DEBUG="GST_TRACER:7"
  export GST_TRACERS='latency(flags=pipeline+element)'
  # export GST_TRACERS='latency(flags=pipeline+element)'
  export GST_DEBUG_FILE=trace.log
  # export GST_DEBUG="3"
  export GST_DEBUG_DUMP_DOT_DIR="$(pwd)"

  rm trace.log || true

  gst-launch-1.0 \
    videotestsrc num-buffers=30 is-live=true \
    ! video/x-raw,width=1280,height=720,framerate=10/1 \
    ! vah265enc b-frames=0 \
    ! h265parse \
    ! rtph265pay aggregate-mode=zero-latency config-interval=-1 pt=96 \
    ! rtpstreampay \
    ! application/x-rtp-stream \
    ! rtpstreamdepay \
    ! application/x-rtp,media=video,clock-rate=90000,encoding-name=H265 \
    ! rtph265depay \
    ! h265parse config-interval=1 \
    ! video/x-h265,stream-format=hvc1,alignment=au \
    ! vah265dec qos=false \
    ! fakesink sync=false qos=false
)

gst-stats-1.0 trace.log

which DOES NOT introduce these latencies.

Using GST_DEBUG_DUMP_DOT_DIR I checked the caps in front of the decoder elements, but nothing weird. aside that the all-in-one pipeline uses 10bits/main-10 and level 3.1.

So in conclusion the camera generates a h265 stream where all decoders need to cache 3 frames. I ensured that no B-frames are getting generated. Any other Ideas why this is happening, or how I can do further analysis? What else could I check? I am really hoping not needing to compile and debug the plugin myself

Thanks for your time!

bilboed · March 10, 2025, 3:39pm

The camera is most likely doing frame reordering. Make sure it uses a “zero-latency” preset. There’s no other reason that all 3 decoders would introduce 3 frame latency.

That’s what you’re doing in the completely local pipeline

markus · March 11, 2025, 7:47am

Thanks for your response! I guess you are correct. Annoyingly there is very few information about the camera’s encoder.
I’m gonna try to change that and get more information on that topic, by contacting them.

Thanks again.