Spatial artifacts with VAAPI H.264 (CQP/ICQ) in real-time screen capture

I’m building a zero-copy GPU screen recorder on Linux using GStreamer + VAAPI. The pipeline is stable, but I consistently get localized pixelated artifacts (often bottom regions or after motion) even at high bitrate and with quality-based modes (the bitrate is also unusually high no matter how much i adjust qp).

here is my pipeline:

  • low

appsrc name=src is-live=true format=time do-timestamp=false block=true ! queue max-size-buffers=8 max-size-bytes=0 max-size-time=0 ! vapostproc ! video/x-raw(memory:VAMemory),format=NV12,width=2880,height=1800,framerate=120/1,color-range=(string)full,colorimetry=(string)bt709 ! vah264enc name=enc rate-control=cqp bitrate=15000 key-int-max=90 ref-frames=1 target-usage=4 min-qp=22 max-qp=36 qpi=33 qpp=35 ! h264parse ! mp4mux faststart=true ! filesink location=output.mp4

- medium

appsrc name=src is-live=true format=time do-timestamp=false block=true ! queue max-size-buffers=8 max-size-bytes=0 max-size-time=0 ! vapostproc ! video/x-raw(memory:VAMemory),format=NV12,width=2880,height=1800,framerate=120/1,color-range=(string)full,colorimetry=(string)bt709 ! vah264enc name=enc rate-control=cqp bitrate=15000 key-int-max=60 ref-frames=1 target-usage=4 min-qp=28 max-qp=36 qpi=28 qpp=32 ! h264parse ! mp4mux faststart=true ! filesink location=output.mp4

- high

appsrc name=src is-live=true format=time do-timestamp=false block=true ! queue max-size-buffers=8 max-size-bytes=0 max-size-time=0 ! vapostproc ! video/x-raw(memory:VAMemory),format=NV12,width=2880,height=1800,framerate=120/1,color-range=(string)full,colorimetry=(string)bt709 ! vah264enc name=enc rate-control=cqp bitrate=15000 key-int-max=30 ref-frames=1 target-usage=3 min-qp=16 max-qp=32 qpi=20 qpp=24 ! h264parse ! mp4mux faststart=true ! filesink location=output.mp4

thank you.

i have no problems with vah265enc, QSV, or x264.

example:

I guess the first step would be to check if you have the latest version of the VA driver.