Hi there, I have been trying to get the pi 5 hardware decoder working to display video in a Qt QML app using the gstreamer qml6glsink. I am using Raspberry Pi OS (based on Debian Trixie)
First, I confirmed the hardware decoder works using a basic pipeline:
gst-launch-1.0 udpsrc port=5602 caps=\"application/x-rtp,media=video,encoding-name=H265,payload=96\" ! rtph265depay ! h265parse ! v4l2slh265dec ! autovideosink
(this seems to use glimagesink)
Next I made a little cpp app to draw the video. It accepts a gstreamer pipeline from the command line, and draws the video sink into a Qt QML GUI.
#include <QGuiApplication>
#include <QQmlApplicationEngine>
#include <QQmlContext>
#include <QQuickItem>
#include <QQuickWindow>
#include <gst/gst.h>
#include <iostream>
#include <QOpenGLContext>
int main(int argc, char *argv[]) {
gst_init(&argc, &argv);
QQuickWindow::setGraphicsApi(QSGRendererInterface::OpenGL);
QGuiApplication app(argc, argv);
QQmlApplicationEngine engine;
std::cout << argv[1] << std::endl;
GstElement *pipeline = gst_parse_launch(
argv[1],
nullptr
);
GstElement *sink = gst_bin_get_by_name(GST_BIN(pipeline), "sink");
engine.loadData(R"(
import QtQuick 2.15
import QtQuick.Window 2.15
import org.freedesktop.gstreamer.Qt6GLVideoItem 1.0
Window {
width: 1280; height: 720
visible: true
GstGLQt6VideoItem {
id: video
objectName: "videoItem"
anchors.fill: parent
}
}
)");
QObject *rootObj = engine.rootObjects().first();
QQuickItem *videoItem = rootObj->findChild<QQuickItem*>("videoItem");
QQuickWindow *window = qobject_cast<QQuickWindow*>(rootObj);
QObject::connect(window, &QQuickWindow::sceneGraphInitialized, [&]() {
g_object_set(sink, "widget", videoItem, nullptr);
gst_element_set_state(pipeline, GST_STATE_PLAYING);
});
int ret = app.exec();
gst_element_set_state(pipeline, GST_STATE_NULL);
gst_object_unref(sink);
gst_object_unref(pipeline);
return ret;
}
when i run the pipeline with this command
WAYLAND_DISPLAY=wayland-0 ./build/gstqml 'udpsrc port=5602 caps="application/x-rtp,media=video,encoding-name=H265,payload=96" ! rtph265depay ! h265parse ! v4l2slh265dec ! glupload ! qml6glsink name=sink sync=false'
It throws this error:
0:00:00.165007381 2005 0x5555fb60f090 ERROR GST_PIPELINE gst/parse/grammar.y:1206:gst_parse_perform_link: could not link v4l2slh265dec0 to sink 0:00:01.276511833 2005 0x7ffeac000b70 WARN v4l2codecs-h265dec gstv4l2codech265dec.c:434:gst_v4l2_codec_h265_dec_negotiate:<v4l2slh265dec0> error: Unsupported pixel format 0:00:01.276549166 2005 0x7ffeac000b70 WARN v4l2codecs-h265dec gstv4l2codech265dec.c:434:gst_v4l2_codec_h265_dec_negotiate:<v4l2slh265dec0> error: No support for 1920x1080 format UNKNOWN 0:00:01.276574481 2005 0x7ffeac000b70 ERROR v4l2codecs-h265dec gstv4l2codech265dec.c:991:gst_v4l2_codec_h265_dec_new_sequence:<v4l2slh265dec0> Failed to negotiate with downstream 0:00:01.276580036 2005 0x7ffeac000b70 WARN h265decoder gsth265decoder.c:600:gst_h265_decoder_process_sps:<v4l2slh265dec0> subclass does not want accept new sequence 0:00:01.276584351 2005 0x7ffeac000b70 WARN h265decoder gsth265decoder.c:874:gst_h265_decoder_process_slice:<v4l2slh265dec0> Failed to process sps 0:00:01.276602944 2005 0x7ffeac000b70 WARN basesrc gstbasesrc.c:3187:gst_base_src_loop:<udpsrc0> error: Internal data stream error. 0:00:01.276608610 2005 0x7ffeac000b70 WARN basesrc gstbasesrc.c:3187:gst_base_src_loop:<udpsrc0> error: streaming stopped, reason not-negotiated (-4) 0:00:01.276630017 2005 0x7ffeac000b70 WARN videodecoder gstvideodecoder.c:1421:gst_video_decoder_sink_event_default:<v4l2slh265dec0> error: No valid frames decoded before end of stream 0:00:01.276636036 2005 0x7ffeac000b70 WARN videodecoder gstvideodecoder.c:1421:gst_video_decoder_sink_event_default:<v4l2slh265dec0> error: no valid frames found
I was able to dig a little deeper into the debug by using the env variable GST_DEBUG=2,v4l2codecs-h265dec:7,glupload:5 . I saw a negotiation error between v4l2slh265 sink and glupload
0:00:03.359846385 6761 0x7fff0c000b70 DEBUG v4l2codecs-h265dec gstv4l2codech265dec.c:426:gst_v4l2_codec_h265_dec_negotiate:<v4l2slh265dec0> Supported output formats: video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:03.359890921 6761 0x7fff0c000b70 DEBUG glupload gstglupload.c:1446:_direct_dma_buf_upload_transform_caps:<glupload0> direction sink, fails to transformed DMA caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:03.359924754 6761 0x7fff0c000b70 DEBUG glupload gstglupload.c:1053:_dma_buf_upload_transform_caps:<glupload0> direction sink, fails to transformed DMA caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:03.359951050 6761 0x7fff0c000b70 DEBUG glupload gstglupload.c:2069:_raw_data_upload_drm_transform_caps:<glupload0> direction sink, fails to transformed DMA caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:03.359977475 6761 0x7fff0c000b70 DEBUG glupload gstglupload.c:475:_gl_memory_upload_transform_caps:<glupload0> direction src, transformed EMPTY into EMPTY
0:00:03.359992012 6761 0x7fff0c000b70 DEBUG glupload gstglupload.c:1499:_direct_dma_buf_upload_transform_caps:<glupload0> direction src, fails to transformed DMA caps EMPTY
0:00:03.360010808 6761 0x7fff0c000b70 DEBUG glupload gstglupload.c:1081:_dma_buf_upload_transform_caps:<glupload0> direction src, fails to transformed DMA caps EMPTY
0:00:03.360024956 6761 0x7fff0c000b70 DEBUG glupload gstglupload.c:2082:_raw_data_upload_drm_transform_caps:<glupload0> direction src, fails to transformed DMA caps EMPTY
0:00:03.360040141 6761 0x7fff0c000b70 DEBUG v4l2codecs-h265dec gstv4l2codech265dec.c:430:gst_v4l2_codec_h265_dec_negotiate:<v4l2slh265dec0> Peer supported formats: EMPTY
So from this output, it looks like the decoder can only output DMABUF video with drm-format NV12:0x0700000000000004. But for some reason glupload can’t work with this - I suspect this is due to the drm-format.
I also have the same debug output from the good working negotiation between v4l2slh265dec and glimagesink from my first good pipeline:
0:00:05.395777998 2669 0x7fff00000b70 DEBUG glupload gstglupload.c:1446:_direct_dma_buf_upload_transform_caps:<glupload0> direction sink, fails to transformed DMA caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:05.395835368 2669 0x7fff00000b70 DEBUG glupload gstglupload.c:1513:_direct_dma_buf_upload_transform_caps:<glupload0> direction sink, transformed video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ] into video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ], texture-target=(string)external-oes
0:00:05.395879257 2669 0x7fff00000b70 DEBUG glupload gstglupload.c:1053:_dma_buf_upload_transform_caps:<glupload0> direction sink, fails to transformed DMA caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:05.395916682 2669 0x7fff00000b70 DEBUG glupload gstglupload.c:2069:_raw_data_upload_drm_transform_caps:<glupload0> direction sink, fails to transformed DMA caps video/x-raw(memory:DMABuf), format=(string)DMA_DRM, drm-format=(string)NV12:0x0700000000000004, width=(int)1920, height=(int)1080, framerate=(fraction)[ 0/1, 2147483647/1
In this negotiation, we see that the _direct_dma_buf_upload_transform_caps function is actually called TWICE - this is different from my failed example. On the 2nd function execution, we get texture-target=external-oes in the output caps. Not sure what that means, but that is probably what we need to handle the drm-format outputted by the decoder.
I had a look at Using dmabuf with qml6glsink here on the gstreamer forums - seems like qml6glsink needs EGL to properly handle the DMA conversions. I made sure to confirm my Qt app was running using the wayland QPA backend, which should hopefully use EGL.
From here, my debugging skills are exhausted. Does anybody have any ideas how to make this work? Any help would be much appreciated!
Thanks, Jack