I am unable to find a solution for using nvh264enc together with webrtcsink on a Jetson Orin AGX.
It seems webrtcsink is can not defer the needed encoder for my input caps.
Right now i am running this pipeline, streaming 3 GigE cameras to a Chrome Web UI
With this i am loosing all the good stuff from webrtcsink controlling bitrate etc..
Since for now the sink and UI are on localhost at the same time, this is running ok, but unstable.
I am dropping frames both in pipeline and Browser it seems, i have no good idea how to really profile this.
To make this work i had to add a pad probe to convert force-keyunit events to the custom action signal based NVIDIA encoder API ( i took this from gst-plugins-rs/net/webrtc/src/webrtcsink/imp.rs#L729:
namespace {
GstPadProbeReturn force_keyunit_probe(GstPad* pad, GstPadProbeInfo* info, gpointer user_data) {
auto* saw_buffer = static_cast<std::atomic<bool>*>(user_data);
if (info->type & (GST_PAD_PROBE_TYPE_BUFFER | GST_PAD_PROBE_TYPE_BUFFER_LIST)) {
saw_buffer->store(true, std::memory_order_seq_cst);
} else if (info->type & GST_PAD_PROBE_TYPE_EVENT_UPSTREAM) {
GstEvent* ev = GST_PAD_PROBE_INFO_EVENT(info);
if (gst_video_event_is_force_key_unit(ev) && saw_buffer->load(std::memory_order_seq_cst)) {
GstObject* parent = gst_pad_get_parent(pad); // new ref, NULL if unparented
if (parent) {
// dGPU nvv4l2h264enc exposes force-idr as a PROPERTY, not an action signal,
// FIX from:
// https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/0.15/net%2Fwebrtc%2Fsrc%2Fwebrtcsink%2Fimp.rs?ref_type=heads&blame=1#L729
if (g_object_class_find_property(G_OBJECT_GET_CLASS(parent), "force-idr") != nullptr) {
g_object_set(G_OBJECT(parent), "force-idr", TRUE, NULL);
}
gst_object_unref(parent);
}
}
}
return GST_PAD_PROBE_OK;
}
} // namespace
void add_nv4l2enc_force_keyunit_workaround(GstElement* enc) {
GstPad* srcpad = gst_element_get_static_pad(enc, "src");
g_assert(srcpad != nullptr);
auto* saw_buffer = new std::atomic<bool>(false);
gulong id = gst_pad_add_probe(
srcpad,
static_cast<GstPadProbeType>(GST_PAD_PROBE_TYPE_BUFFER | GST_PAD_PROBE_TYPE_BUFFER_LIST |
GST_PAD_PROBE_TYPE_EVENT_UPSTREAM),
force_keyunit_probe, saw_buffer, [](gpointer data) { delete static_cast<std::atomic<bool>*>(data); });
g_assert(id != 0);
gst_object_unref(srcpad);
}
Running the pipeline on a dGPU x86 without the extra encoder, the automatic encoder selection works fine with following warnings:
0:00:06.328453208 23639 0x7976b40031c0 WARN GST_PADS gstpad.c:4392:gst_pad_peer_query:<capsfilter7:src> could not send sticky events
0:00:06.328617623 23639 0x7973ec038a40 WARN nvenc gstnvbaseenc.c:2668:gst_nv_base_enc_set_property:<nvh265enc0> device does not support requested rate control mode 5
0:00:06.328634541 23639 0x7976b40031c0 WARN nvenc gstnvbaseenc.c:1745:gst_nv_base_enc_set_format:<nvh264enc0> error: Selected preset not supported
0:00:06.328653345 23639 0x7976b40031c0 WARN videoencoder gstvideoencoder.c:773:gst_video_encoder_setcaps:<nvh264enc0> rejected caps video/x-raw, width=(int)616, height=(int)514, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)15/1, batch-size=(uint)1, num-surfaces-per-frame=(int)1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:06.328861551 23639 0x7976b4003a40 WARN nvvideoconvert gstnvvideoconvert.c:1818:gst_nvvideoconvert_set_caps:<nvvideoconvert8> Cannot keep DAR
0:00:06.334418849 23639 0x7976b40031c0 WARN nvenc gstnvbaseenc.c:1745:gst_nv_base_enc_set_format:<nvh264enc0> error: Selected preset not supported
0:00:06.334454170 23639 0x7976b40031c0 WARN videoencoder gstvideoencoder.c:773:gst_video_encoder_setcaps:<nvh264enc0> rejected caps video/x-raw, width=(int)616, height=(int)514, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)15/1, batch-size=(uint)1, num-surfaces-per-frame=(int)1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:06.336743925 23639 0x7973ec038a40 WARN default gstdebugutils.c:881:gst_debug_bin_to_dot_file: Failed to open file '/app/data/dot_files/0.00.06.336727458-webrtcsink-discovery-pipeline3-Null-to-Ready.dot' for writing: No such file or directory
0:00:06.337048903 23639 0x7976b4003a40 WARN vpxenc gstvpxenc.c:1863:gst_vpx_enc_set_format:<vp9enc0> "Failed to set VP8E_SET_NOISE_SENSITIVITY": incapable (details: (NULL))
0:00:06.337089267 23639 0x7976b4003a40 WARN vpxenc gstvpxenc.c:1878:gst_vpx_enc_set_format:<vp9enc0> "Failed to set VP8E_SET_TOKEN_PARTIONS": incapable (details: (NULL))
0:00:06.337201799 23639 0x7976b4004e90 FIXME default gstutils.c:4088:gst_element_decorate_stream_id_internal:<appsrc3> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:06.337205644 23639 0x7973ec038a40 WARN default gstdebugutils.c:881:gst_debug_bin_to_dot_file: Failed to open file '/app/data/dot_files/0.00.06.337191404-webrtcsink-discovery-pipeline0-done.dot' for writing: No such file or directory
0:00:06.339079480 23639 0x7973ec038a40 WARN webrtcsink net/webrtc/src/webrtcsink/imp.rs:4425:gstrswebrtc::webrtcsink::imp::BaseWebRTCSink::run_discovery_pipeline::{{closure}}:<webrtcsink> Error in discovery pipeline: Error {
structure: Some(
GstMessageError {
gerror: (GError) ((GError*) 0x797310007030),
debug: (gchararray) "../sys/nvcodec/gstnvbaseenc.c(1745): gst_nv_base_enc_set_format (): /GstPipeline:pipeline1/GstNvH264Enc:nvh264enc0:\nSelected preset not supported",
},
),
source: Some(
(
Object {
inner: TypedObjectRef {
inner: 0x00007973ec7a0ad0,
type: GstNvH264Enc,
},
},
"nvh264enc0",
),
),
error: Error {
domain: gst-library-error-quark,
code: 5,
message: "Could not configure supporting library.",
},
debug: Some(
"../sys/nvcodec/gstnvbaseenc.c(1745): gst_nv_base_enc_set_format (): /GstPipeline:pipeline1/GstNvH264Enc:nvh264enc0:\nSelected preset not supported",
),
details: None,
}
0:00:06.339116165 23639 0x7973ec038a40 WARN default gstdebugutils.c:881:gst_debug_bin_to_dot_file: Failed to open file '/app/data/dot_files/0.00.06.339107731-webrtcsink-discovery-pipeline1-error.dot' for writing: No such file or directory
0:00:06.339260734 23639 0x7976b4004670 WARN nvvideoconvert gstnvvideoconvert.c:1818:gst_nvvideoconvert_set_caps:<nvvideoconvert10> Cannot keep DAR
0:00:06.339683056 23639 0x7976b4004670 WARN nvenc gstnvbaseenc.c:1745:gst_nv_base_enc_set_format:<nvh265enc0> error: Selected preset not supported
0:00:06.339709152 23639 0x7976b4004670 WARN videoencoder gstvideoencoder.c:773:gst_video_encoder_setcaps:<nvh265enc0> rejected caps video/x-raw, width=(int)616, height=(int)514, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)15/1, batch-size=(uint)1, num-surfaces-per-frame=(int)1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:06.339999822 23639 0x7976b4004670 WARN nvenc gstnvbaseenc.c:1745:gst_nv_base_enc_set_format:<nvh265enc0> error: Selected preset not supported
0:00:06.340020406 23639 0x7976b4004670 WARN videoencoder gstvideoencoder.c:773:gst_video_encoder_setcaps:<nvh265enc0> rejected caps video/x-raw, width=(int)616, height=(int)514, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)15/1, batch-size=(uint)1, num-surfaces-per-frame=(int)1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:06.340032414 23639 0x7976b4004670 WARN GST_PADS gstpad.c:4392:gst_pad_peer_query:<capsfilter9:src> could not send sticky events
0:00:06.340801249 23639 0x7976b4004670 WARN nvenc gstnvbaseenc.c:1745:gst_nv_base_enc_set_format:<nvh265enc0> error: Selected preset not supported
0:00:06.340827455 23639 0x7976b4004670 WARN videoencoder gstvideoencoder.c:773:gst_video_encoder_setcaps:<nvh265enc0> rejected caps video/x-raw, width=(int)616, height=(int)514, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)15/1, batch-size=(uint)1, num-surfaces-per-frame=(int)1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:06.341680860 23639 0x7973ec038a40 WARN default gstdebugutils.c:881:gst_debug_bin_to_dot_file: Failed to open file '/app/data/dot_files/0.00.06.341674634-webrtcsink-discovery-pipeline2-done.dot' for writing: No such file or directory
0:00:06.342349467 23639 0x7976b4004670 WARN nvenc gstnvbaseenc.c:1745:gst_nv_base_enc_set_format:<nvh265enc0> error: Selected preset not supported
0:00:06.342379200 23639 0x7976b4004670 WARN videoencoder gstvideoencoder.c:773:gst_video_encoder_setcaps:<nvh265enc0> rejected caps video/x-raw, width=(int)616, height=(int)514, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)15/1, batch-size=(uint)1, num-surfaces-per-frame=(int)1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:06.353201737 23639 0x7973ec038a40 WARN webrtcsink net/webrtc/src/webrtcsink/imp.rs:4425:gstrswebrtc::webrtcsink::imp::BaseWebRTCSink::run_discovery_pipeline::{{closure}}:<webrtcsink> Error in discovery pipeline: Error {
structure: Some(
GstMessageError {
gerror: (GError) ((GError*) 0x7972f8006770),
debug: (gchararray) "../sys/nvcodec/gstnvbaseenc.c(1745): gst_nv_base_enc_set_format (): /GstPipeline:pipeline3/GstNvH265Enc:nvh265enc0:\nSelected preset not supported",
},
),
source: Some(
(
Object {
inner: TypedObjectRef {
inner: 0x00007973ec580dd0,
type: GstNvH265Enc,
},
},
"nvh265enc0",
),
),
error: Error {
domain: gst-library-error-quark,
code: 5,
message: "Could not configure supporting library.",
},
debug: Some(
"../sys/nvcodec/gstnvbaseenc.c(1745): gst_nv_base_enc_set_format (): /GstPipeline:pipeline3/GstNvH265Enc:nvh265enc0:\nSelected preset not supported",
),
details: None,
}
0:00:06.353240489 23639 0x7973ec038a40 WARN default gstdebugutils.c:881:gst_debug_bin_to_dot_file: Failed to open file '/app/data/dot_files/0.00.06.353229745-webrtcsink-discovery-pipeline3-error.dot' for writing: No such file or directory
0:00:06.354988637 23639 0x7973ec038a40 WARN webrtcsink net/webrtc/src/webrtcsink/imp.rs:4571:gstrswebrtc::webrtcsink::imp::BaseWebRTCSink::lookup_caps::{{closure}}:<webrtcsink> Codec discovery pipeline failed: Could not configure supporting library.
0:00:06.354996274 23639 0x7973ec038a40 WARN webrtcsink net/webrtc/src/webrtcsink/imp.rs:4571:gstrswebrtc::webrtcsink::imp::BaseWebRTCSink::lookup_caps::{{closure}}:<webrtcsink> Codec discovery pipeline failed: Could not configure supporting library.
I am just unsure if this really uses hardware accelerated encoding.