The pipeline stop working when the h264bytesStream change the resolution

Application Development
1

Hello, everyone. I am facing a problem: when the h264 bytes stream changes the resolution, my pipeline stop working. How can I solve this problem?

    // Init Gstreamer basic
    gst_init(nullptr,nullptr);

    // init pipeline element
    pipeline = gst_pipeline_new("pipeline");
    appsrc = gst_element_factory_make("appsrc", "appsrc");
    h264parse = gst_element_factory_make("h264parse", "h264parse");
    decoder = gst_element_factory_make("nvv4l2decoder", "decoder");
    nvvidconv = gst_element_factory_make("nvvidconv", "nvvidconv");
    videoconvert = gst_element_factory_make("videoconvert", "videoconvert");
    appsink = gst_element_factory_make("appsink", "appsink");

    if (!pipeline || !appsrc || !h264parse || !decoder || !nvvidconv || !videoconvert || !appsink) {
        ERROR("Create element fail in Gstreamer!\n");
        return -2;
    }
    // add all element to the pipeline
    gst_bin_add_many(GST_BIN(pipeline), appsrc, h264parse, decoder, nvvidconv, videoconvert, appsink, NULL);
    gst_element_link_many(h264parse, decoder, nvvidconv, NULL);

    // Add Caps in appsrc
    GstCaps *appsrc_caps = gst_caps_new_simple(
                            "video/x-h264",
                            "width", G_TYPE_INT, 1920,
                            "height", G_TYPE_INT, 1080,
                            "framerate", GST_TYPE_FRACTION, 30, 1,
                            "stream-format", G_TYPE_STRING, "byte-stream",
                            NULL);

    //g_object_set(G_OBJECT(appsrc), "caps", appsrc_caps, NULL);
    if (!gst_element_link_filtered(appsrc, h264parse, appsrc_caps)) {
        ERROR("Failed to link appsrc to h264parse with filter.");
        gst_caps_unref(appsrc_caps);
        return -3;
    }
    gst_caps_unref(appsrc_caps);

    // change features in h264parse
    g_object_set(G_OBJECT(h264parse), "config-interval", 1, NULL);

    // add caps between nvvidconv and videoconvert
    GstCaps *nvvidconv_caps = gst_caps_new_simple("video/x-raw",
                                                  "format", G_TYPE_STRING, "BGRx",
                                                  NULL);
    
    if (!gst_element_link_filtered(nvvidconv, videoconvert, nvvidconv_caps)) {
        ERROR("Failed to link nvvidconv to videoconvert with filter.");
        gst_caps_unref(nvvidconv_caps);
        return -3;
    }
    gst_caps_unref(nvvidconv_caps);

    // add caps between videoconvert and appsink
    GstCaps *appsink_caps = gst_caps_new_simple("video/x-raw",
                                                "format", G_TYPE_STRING, "BGR",
                                                NULL);
    if (!gst_element_link_filtered(videoconvert, appsink, appsink_caps)) {
        ERROR("Failed to link videoconvert to appsink with filter.");
        gst_caps_unref(appsink_caps);
        return -4;
    }
    gst_caps_unref(appsink_caps);

    // set appsink festures
    g_object_set(appsink, "emit-signals", TRUE, "sync", FALSE, NULL);
    //g_signal_connect(appsink, "new-sample", G_CALLBACK(on_new_image_static), NULL);
    g_signal_connect(appsink, "new-sample", G_CALLBACK(on_new_image_static), this);
    GstStateChangeReturn ret = gst_element_set_state(pipeline, GST_STATE_PLAYING);
    if (ret == GST_STATE_CHANGE_FAILURE) {
        ERROR("Failed to set pipeline to PLAYING state.");
        return -1; 
    }

Also, there has no callback show me what 's wrong in the bus. When the pipeline doesn’t work, there is no any callback back in the function.

    GstBus *bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
    gst_bus_add_watch(bus, on_message, NULL);
2

appsrc has a caps property and/or API to set the caps, and you should use that in order to specify the format of the data you’re going to push into it. There’s no need for a capsfilter element after the appsrc, but it’s probably also not the cause of your issues.

Next, I would recommend to see if it works with software decoders first (avdec_h264, videoconvert, etc), to remove the nvidia elements from the equation.

3

There is no avdec_h264 in nvidia nx. Even I use apt install gstreamer1.0-libav, and I still cannot see it.

4

It should work. You may tell if you’re using Xavier NX or Orin NX and what’s the L4T release (see /etc/nv_tegra_release).

Did you build a custom ffmpeg or gstreamer and have set custom environment for that ?

Do you see unmet dependencies (not found) from:

ldd /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so

?
If yes, first fix these.
If not, you may try to clear the cache:

# Clear gstreamer cache
rm ~/.cache/gstreamer-1.0/registry.aarch64.bin

# This will rebuild the cache and tell what plugins are blacklisted if any
/usr/bin/gst-inspect-1.0 --version -b
5

I try the following command, and I can see avdec_h264 in gst-inspect-1.0 and gst-launch-1.0 now.

rm ~/.cache/gstreamer-1.0/registry.aarch64.bin
LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 gst-inspect-1.0
gst-inspect-1.0 | grep h264

But in C++ version gstreamer, I still cannot make the ‘avdec_h264’ element. My CMakeList is

find_package(PkgConfig REQUIRED)
pkg_check_modules(GSTREAMER REQUIRED gstreamer-1.0 gstreamer-app-1.0 gstreamer-base-1.0)
include_directories(${GSTREAMER_INCLUDE_DIRS})
link_directories(${GSTREAMER_LIBRARY_DIRS})

And the output of ${GSTREAMER_INCLUDE_DIRS} is
/usr/include.gstreamer-1.0;/usr/include/glib-2.0;/usr/lib/aarch64-linux-gun-glib-2.0/include

And the output of ${GSTREAMER_LIBRARY_DIRS} is empty

Also, I ask this question in nvidia forum, they say

“When resolution changes, the buffer has to be destroyed and re-allocated. This may not be well considered in the plugins. So we would suggest fix the resolution.”

. It seems the problem is from nvv4l2decoder and nvvidconv. How can I fix these problem. :smiling_face_with_tear:

6

It might help if you can answer the asked questions.

Looks like there have been some edits in your system config… Would rather be:

/usr/include/gstreamer-1.0;/usr/include/glib-2.0;/usr/lib/aarch64-linux-gnu/glib-2.0/include`

This preload might not be mandatory for your case.

Note that my previous post was just about having avdec_h264 working on Jetson.

I also think that nvv4l2decoder and nvvidconv don’t expect dynamic resolution/profile change.
When your appsrc detects some resolution or else changing, it may stop and destroy current pipeline and recreate a new one with new caps.

As said in NVIDIA forum, you can also customize these NV plugins from public sources for your needs.

Or try CPU based avdec_h264, as advised before by @tpm and see if this case is better managed by this implementation. If yes you may have a look to its sources and adapt to NV plugins customization.