Trouble using nvh264enc with webrtcsink (rs version)

Application Development
1

I have the following stream where I’m trying to hardware encode video/x-raw to h264 and send to webrtcsink.

my-src ! bayer2rgb ! capsfilter caps="video/x-raw,format=RGBx" ! videorate ! video/x-raw,framerate=30/1 ! videoconvert ! nvh264enc ! h264parse 
! capsfilter caps="video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)2.1, profile=(string)main, width=(int)480, height=(int)304, framerate=(fraction)30/1, coded-picture-structure=(string)frame, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8"
! webrtcsink video-caps="video/x-h264"

When I run the pipeline I get the error below and no frames via webrtc. Removing the h264parse element has no effect. Additionally, when I run nvidia-smi encodersessions I see that a session gets created and vram is used, but the average fps and latency are always zero.

0:00:22.222943224  1176 0x7f37740011a0 WARN       codecparsers_h264 gsth264parser.c:2473:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 0
0:00:22.223103477  1176 0x7f37740011a0 WARN               h264parse gsth264parse.c:2028:gst_h264_parse_handle_frame:<h264parse1> Failed to update backlog

I’ve replaced webrtcsink with avimux ! filesink location=output.avi and it works, but I suspect it’s still not using the hardware encoder because nvidia-smi encodersessions still shows average fps and latency as zero.

I’m pretty stumped after working on this all day.

2

Hi @cdelguercio ,

webrtcsink works well when passing xraw frames to it. What I believe happens is that inside the webrtcsink the encoder is dynamically created as the encoding parameters have to be adapted on the fly (to adjust bitrate, compression rate, etc.) depending on the network bandwidth.

I also haven’t figured out how to use it with a hardware_encoder, e.g. nvvh264enc. Did you manage to run a webrtcsink with a hardware encoder in a pipeline?

3

I would like to also join this discussion, as i am unable to find a solution for using nvh264enc together with webrtcsink.
Right now i am running this pipeline, streaming 3 GigE cameras to a Chrome Web UI

Bildschirmfoto vom 2026-09-15 17-43-02
Bildschirmfoto vom 2026-09-15 17-43-021948×1174 331 KB

With this i am loosing all the good stuff from webrtcsink controlling bitrate etc..
Since for now the sink and UI are on localhost at the same time, this is running ok, but unstable.
I am dropping frames both in pipeline and Browser it seems, i have no good idea how to really profile this.

4

Hi @apergios

Have you tried feeding the raw NVMM frames directly into webrtcsink and letting it create the H.264 encoder internally?

In your pipeline, each branch uses nvv4l2h264enc before webrtcsink, so the sink receives an already encoded H.264 stream. In that configuration, webrtcsink can packetize and transmit the stream, but it cannot control the bitrate of the external encoder.

As a first step, you could remove nvv4l2h264enc from one branch and pass the raw NVMM frames directly to the sink:

... ! video/x-raw(memory:NVMM),format=NV12,width=...,height=...,framerate=... ! \
queue ! ws.

Then configure the sink to negotiate H.264 and enable congestion control for bitrate adaptation:

webrtcsink name=ws \
  video-caps="video/x-h264" \
  congestion-control=gcc

It would then be useful to check which encoder webrtcsink selects internally through its encoder-setup signal.

For profiling the drops, you could compare the webrtcsink stats property with Chrome’s chrome://webrtc-internals/. This should help determine whether frames are dropped before reaching WebRTC or later in the browser.

Please let me know if you have any updates!

5

Hi @Julian_Camacho
in this topic i was able to solve my problem:

There was a reason for the “manual” encoder before webrtcsink.
The sink was not able to discover a fitting encoder with NVMM as input caps and the video caps set to h-264 encoding.
After applying patches to the nvidia encoder from the linux for tegra version of the element it works, even with proper congestion control.