How to display a h264 video file using appsink in windows

gorogoro · February 19, 2024, 5:56pm

I’m trying to setup an application in C++ with gstreamer to read a .mov file encoded in h264 format.

I’ve try the following pipelines with success:

gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux name=demux ! h264parse ! decodebin ! videoconvert ! videoscale ! autovideosink

gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux name=demux ! h264parse ! qsvh264dec ! videoconvert ! videoscale ! autovideosink

gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux name=demux ! h264parse ! d3d11h264dec ! videoconvert ! videoscale ! autovideosink

gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux name=demux ! h264parse ! d3d11h264device1dec ! videoconvert ! videoscale ! autovideosink

gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux name=demux ! h264parse ! openh264dec  ! videoconvert ! videoscale ! autovideosink

gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux name=demux ! h264parse ! nvh264dec ! videoconvert ! videoscale ! autovideosink

gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux name=demux ! h264parse ! nvh264sldec  ! videoconvert ! videoscale ! autovideosink

My code is as follow:


void Foo::init()
{
    gst_init(0, NULL);

    _pipelineEments.pipeline = gst_pipeline_new("video-player");
    _pipelineEments._videoSrc = gst_element_factory_make("filesrc", "video_src");
    _pipelineEments._qtdemux = gst_element_factory_make("qtdemux", "qtdemux_src");
    _pipelineEments._h264parse = gst_element_factory_make("h264parse", "h264parse_src");
    _pipelineEments._decodeBin = gst_element_factory_make("decodebin", "video_decode");
    _pipelineEments._videoConvert = gst_element_factory_make("videoconvert", "video_convert");
    _pipelineEments._videoScale = gst_element_factory_make("videoscale", "video_scale");
    _pipelineEments._sink = GST_APP_SINK(gst_element_factory_make("appsink", "video_sink"));

    VERIFY_GST_NULL(_pipelineEments.pipeline)
    VERIFY_GST_NULL(_pipelineEments._videoSrc)
    VERIFY_GST_NULL(_pipelineEments._qtdemux)
    VERIFY_GST_NULL(_pipelineEments._h264parse)
    VERIFY_GST_NULL(_pipelineEments._decodeBin)
    VERIFY_GST_NULL(_pipelineEments._videoConvert)
    VERIFY_GST_NULL(_pipelineEments._videoScale)
    VERIFY_GST_NULL(_pipelineEments._sink)
 gst_bin_add_many(GST_BIN (_pipelineEments.pipeline),
                         _pipelineEments._videoSrc,
                         _pipelineEments._qtdemux,
                         _pipelineEments._h264parse,
                         _pipelineEments._decodeBin,
                         _pipelineEments._videoConvert,
                         _pipelineEments._videoScale,
                         _pipelineEments._sink,
                         NULL);

        gst_element_link_many(_pipelineEments._videoSrc,
                              _pipelineEments._qtdemux,
                              _pipelineEments._h264parse,
                              _pipelineEments._decodeBin,
                              _pipelineEments._videoConvert,
                              _pipelineEments._videoScale,
                              _pipelineEments._sink,
                              NULL);

        gst_app_sink_set_emit_signals(_pipelineEments._sink, true);
        gst_app_sink_set_drop(_pipelineEments._sink, true);
        gst_app_sink_set_max_buffers(_pipelineEments._sink, 1);
        gst_app_sink_set_caps(_pipelineEments._sink,   gst_caps_new_simple("video/x-raw",
                                                         "format", G_TYPE_STRING, "BGRA",
                                                         "width", G_TYPE_INT, _videoWidth,
                                                         "height", G_TYPE_INT, _videoHeight,
                                                         NULL));
        g_object_set (_pipelineEments._videoSrc, "location", file.c_str(), NULL);

        g_signal_connect (_pipelineEments._qtdemux, "pad-added", G_CALLBACK (Foo::pad_added_handler), &_pipelineEments);

gst_element_set_state(_pipelineEments.pipeline, GST_STATE_PLAYING);
}

void Foo::pad_added_handler(GstElement *src, GstPad *newPad, PipelineElements *data)
{
    auto padName = GST_PAD_NAME(newPad);
    auto elementName = GST_ELEMENT_NAME(src);
    LOG_STATIC(Logger::Level::Info, "Received new pad: " << padName << " from: " << elementName)

    auto sink_pad = gst_element_get_static_pad(data->_h264parse, "sink");
    if (gst_pad_is_linked(sink_pad))
    {
        LOG_STATIC(Logger::Level::Info, "Sink pad of qtdemux already linked")
        return;
    }

    /* Check the new pad's type */
    auto new_pad_caps = gst_pad_get_current_caps (newPad);
    auto new_pad_struct = gst_caps_get_structure (new_pad_caps, 0);
    auto new_pad_type = gst_structure_get_name (new_pad_struct);
    if (!g_str_has_prefix (new_pad_type, "video/x-h264"))
    {
        LOG_STATIC(Logger::Level::Info, "New pad: '" << new_pad_type << "' is not video! Descarting it")
        return;
    }

    /* Attempt the link */
    auto ret = gst_pad_link (newPad, sink_pad);
    if (GST_PAD_LINK_FAILED (ret))
    {
        LOG_STATIC(Logger::Level::Error, "Type is '" << new_pad_type << "' but link failed!")
    }
    else
    {
        LOG_STATIC(Logger::Level::Info, "Link succeeded. Type: " << new_pad_type)
    }
}

I’ve been playing around in this pipeline and just changing the _pipelineElements._decodeBin and loading different decoders.

For the following decoders I always got: “Internal data stream error.” , I also can see the link being created beetween the qtdemux and h264parse.

nvh264sldec
nvh264dec
openh264dec
decodebin

So it seems that there’s something wrong on my pipeline, but I can’t figure out what it will be.

Any help will be very much appreciated.

Thank you

Honey_Patouceul · February 19, 2024, 8:08pm

What if using gst_parse_launch for building the pipeline ?
You can get any element by name for setting callbacks.
You may also try adding a queue after source or after decoder.

tpm · February 24, 2024, 11:18am

The problem is with your gst_element_link_many() call I think. It will fail/stop when it tries to link qtdemux to h264parse and then not link the rest, but even if it did it would fail again linking decodebin to videoconvert, because decodebin has no source pads yet at that point, and then it won’t continue to link videoconvert to videoscale and videoscale to appsink, so those remain unlinked even if you handle the pad-added signal on decodebin later.

Try:

 gst_element_link_many(_pipelineEments._videoSrc,
                              _pipelineEments._qtdemux,
                              NULL);

// qtdemux ! h264parse will be linked in the pad-added signal handler

 gst_element_link_many(
                              _pipelineEments._h264parse,
                              _pipelineEments._decodeBin,
                              NULL);

// decodebin ! videoconvert will be linked in the pad-added signal handler

 gst_element_link_many(
                              _pipelineEments._videoConvert,
                              _pipelineEments._videoScale,
                              _pipelineEments._sink,
                              NULL);

You could just use uridecodebin3 uri=file:/// ! videoconvert ! videoscale ! appsink for what it’s worth, then you only have to handle pad-added once, and it will plug qtdemux and h264parse for you.

gorogoro · February 26, 2024, 5:19pm

Thank you all for the help.

I’m using decodebin because it can do the work.
However I also have try with another decoder that works on gst-launch: openh264dec, nvh264sldec, nvh264dec. But the stream gives me same error: ‘Internal data stream error’. This message is not very helpfull

Regarding the pipeline not being linked, I’ve also added the g_signal_connect to all other elements to see if any other pad was added but nothing was called except for the qtdemux source ones.

Using videotestsrc inste of filesrc everything is fine.

I’m a litle lost here. Thank you

tpm · February 26, 2024, 5:54pm

The “Internal data stream error” will have a “debug message” attached to it which will contain a “reason” like “not-linked” or “not-negotiated” (caps problem) or so.

You can also check the GST_DEBUG log for more information.

gorogoro · February 28, 2024, 1:42pm

You were right.

The problem was that I was only linking the debux to the parser and the rest of the pads were not linked.

Thank you very much