Audio is choppy in streaming to youtube with va-api

General Discussion
1

I have a pipeline to stream audio/video to youtube. With VAAPI, the pipeline works nicely on Ubuntu 18.04(gstreamer 1.14.5). But with VA-API, the audio is choppy on Ubuntu 22.04(GStreamer: 1.22.5) while video seems OK. Audio part in the pipeline is exactly the same. Video part uses vaapipostproc/vaapih264enc with VAAPI and vapostproc/vah264enc with VA-API.

Streaming audio/video to LAN with VAAPI/VA-API works fine. Computer is same in both cases(VAAPI and VA-API).

What can cause this issue? How to debug?

2

got following warnings from gstreamer debug output

0:00:22.385490503 23319 0x55c4fa04ec60 INFO         audio-converter audio-converter.c:1361:gst_audio_converter_new: unitsizes: 2 -> 4
0:00:22.385561575 23319 0x55c4fa04ec60 INFO         audio-converter audio-converter.c:728:chain_unpack: unpack format S16LE to S16LE
0:00:22.385608052 23319 0x55c4fa04ec60 INFO         audio-converter audio-converter.c:870:chain_mix: mix format S16LE, passthrough 0, in_channels 1, out_channels 2
0:00:22.385636635 23319 0x55c4fa04ec60 INFO         audio-resampler audio-resampler.c:1553:gst_audio_resampler_update: phase 0 out_rate 44100, in_rate 16000, gcd 100
0:00:22.385654650 23319 0x55c4fa04ec60 INFO         audio-resampler audio-resampler.c:1560:gst_audio_resampler_update: new phase 0/441
0:00:22.385669742 23319 0x55c4fa04ec60 INFO         audio-resampler audio-resampler.c:1566:gst_audio_resampler_update: have new options, reconfigure filter
0:00:22.385776582 23319 0x55c4fa04ec60 WARN         audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
0:00:22.385864081 23319 0x55c4fa04ec60 WARN         audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
0:00:22.385940913 23319 0x55c4fa04ec60 WARN         audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
0:00:22.386008837 23319 0x55c4fa04ec60 INFO         audio-converter audio-converter.c:966:chain_quantize: depth in 16, out 16
0:00:22.386027112 23319 0x55c4fa04ec60 INFO         audio-converter audio-converter.c:978:chain_quantize: using no dither and noise shaping
0:00:22.386053211 23319 0x55c4fa04ec60 INFO         audio-converter audio-converter.c:1035:chain_pack: pack format S16LE to S16LE

0:03:35.140304346 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
0:03:35.141001930 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
0:03:35.141314636 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500

and caps for resample is
audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2

3

added key-int-max=30 to vah264enc and set higher bitrate=384000 from 128000 for voaacenc. Still not working. Any help?

4

made some comparison and noticed
vaapipostproc uses format 2=I420 <==set
vapostproc uses format=NV12 <==auto set

I tried to add caps to vapostproc with
video/x-raw, width=1920, height=1080, framerate=25/1, pixel-aspect-ratio=1/1, format=I420
then vapostproc and vah264enc do not link anymore. Why?

vapostproc does not support I420?

5

I changed the format of vaapipostproc to 23 to NV12. The pipeline with vaapi still works fine.

6

the stream format of the va encoder is byte-stream
/GstPipeline:pipeline0/GstVaH264Enc:vah264enc0.GstPad:src: caps =
video/x-h264, profile=(string)high, level=(string)4, width=(int)1920,
height=(int)1080, alignment=(string)au, stream-format=(string)byte-stream,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)25/1, interlace-
mode=(string)progressive, colorimetry=(string)bt709, chroma-
site=(string)mpeg2

the stream format of the vaapi encoder is avc
/GstPipeline:pipeline0/GstVaapiEncodeH264:vaapiencodeh264-0.GstPad:src:
caps = video/x-h264, stream-format=(string)avc, alignment=(string)au,
profile=(string)high, width=(int)1920, height=(int)1080, pixel-aspect-
ratio=(fraction)1/1, framerate=(fraction)25/1,
codec_data=(buffer)01640028ffe1002967640028acd301e0089797ff00010001100000030010000003032e0000030112a80000337f86f7bdca01000468ee3cb0,
interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-
site=(string)mpeg2

not sure this matters.

7

tried to set caps with stream-format=avc to GstVaH264Enc and the pipeline will not work.
avc is packet-oriented streaming. Is it not supported in vah264enc anymore?

8

what are the pipelines?

Looks like in GstVA there’s more CPU consumption.

9

gst-inspect-1.0 should tell you what color formats does vaposproc support.

But the case is vah264enc only imports NV12 color format (at least on intel and radeonsi)

10

it’s not, but you can use h264parse to convert from byte-stream to avc

11

Thanks for your reply, vjaquez. I have confirmed vah264enc only imports NV12 color format for intel CPU.

12

I do not think byte-stream or avc should matter.

13

flvmux streamable=true name=mux ! rtmpsink location=“*** app=live2” ximagesrc use-damage=0 show-pointer=0 endx=*** endy=*** ! queue ! vapostproc ! vah264enc rate-control=vbr bitrate=1500 dct8x8=true cabac=true target-usage=7 ! h264parse ! follow an audio mixer.

the same audio mixer works fine with vaapipostproc and vaapih264enc. The only changes I made are to replace vaapipostproc with vapostproc and vaapih264enc with vah264enc. Also similar VA encoding code is applied for streaming to LAN and Wi-Fi without any problem. Gstreamer version is 1.22.5.

14

What is the cause for the following warning messages?

0:03:35.140304346 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
0:03:35.141001930 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
0:03:35.141314636 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
15

More warning messages:
m0:00:00.152869458 2203 0x5651d00831e0 WARN aggregator gstaggregator.c:2146:gst_aggregator_query_latency_unlocked: Latency query failed
0:00:00.152895028 2203 0x5651d00831e0 WARN aggregator gstaggregator.c:2146:gst_aggregator_query_latency_unlocked: Latency query failed
0:00:00.153425396 2203 0x5651d0083000 WARN aggregator gstaggregator.c:2146:gst_aggregator_query_latency_unlocked: Latency query failed
0:00:00.153868660 2203 0x5651d0083000 WARN aggregator gstaggregator.c:2146:gst_aggregator_query_latency_unlocked: Latency query failed
Pipeline is live and does not need PREROLL …
Got context from element ‘vah264enc0’: gst.va.display.handle=context, gst-display=(GstObject)“(GstVaDisplayDrm)\ vadisplaydrm3”, description=(string)“Intel\ iHD\ driver\ for\ Intel(R)\ Gen\ Graphics\ -\ 23.3.0\ (ca1098d8a)”, path=(string)/dev/dri/renderD128;
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstSystemClock
0:00:00.157079897 2203 0x5651cfbf5f00 WARN audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can’t find exact taps
Redistribute latency…
Redistribute latency…
0:00:00.460339510 2203 0x5651d00831e0 WARN flvmux gstflvmux.c:1241:gst_flv_mux_buffer_to_tag_internal:mux:sink_1 Got backwards dts! (0:00:00.128000000 < 0:00:00.135000000)

gst-inspect-1.0 audioresample shows S16LE is available.