Overlay Element Reference count

Application Development
1

In a h264 video pipeline, we are dynamically adding(and removing) a text overlay when video is in playing condition.
Please see the details
olcamerasrc->capsfilter->queue->appsink

olcamersrc is custom element - will produce H264 encoded video in its src
pad. It also have a sink pad to accept overlay buffer to be encoded with the
video.

After setting initial pipeline’s state to PLAYING I am trying to add below
mentioned pipeline dynamically to above one to display an overlay on the
video. overlay buffer will be fed to appsrc once new pipeline is added.

appsrc–>alphacolor–>queue–>capsfilter->overlaysink( sink pad of
olcamerasrc element)

After setting new elements state to PLAYING I printed the reference count of
each elements. It is suddenly changed to 6 from 1. When I trying to remove
these elements from pipeline reference count never reaching to zero. Seems
like it causing memory leak. I have done a performance testing by adding and
removing new pipeline elements and application got killed by OS due to
increase in memory usage.

My steps to add pipeline elements:

  1. Add a probe in source pad of olcamerasrc
  2. Inside the call back method of probe
    a) Add new elements to pipeline using -gst_bin_add_many
    b) Link new elements using - gst_element_link_many
    c) Request new sink pad from olcamerasrc.
    d) Link overlay sink pad of olcamerasrc and src pad of capsfilter
    e) Set state to PLAYING for new elements
  3. return pad drop

My steps to remove pipeline elements:

  1. Add a probe in source pad of olcamerasrc
  2. Inside the call back method of probe
    a) Unlink overlay sink pad of olcamerasrc and src pad of capsfilter
    b) Unlink new elements using - gst_element_unlink_many
    c) Remove new elements from pipeline using -gst_bin_remove_many
    d) Release new sink pad from olcamerasrc.
    e) Set state to NULL for new elements
  3. return pad drop

I have a few confusions here .

  1. Is it required to add any probe( in pad of olcamerasrc) in order to
    add/remove overlay ( above pipeline ) dynamically to initial pipeline?

  2. What would be the reason for the spike in reference count when state
    changing to PLAYING?

  3. How can I properly reach zero reference count for elements after
    removing from pipeline?

Could you please help me to identify the above mentioned issues ?

2

Generally speaking a probe on the camerasrc’s source pad shouldn’t be required to add/remove inputs to the camerasrc element, but of course I don’t know anything about the implementation.

The spike/drop in reference count could be from bus messages. Element state changes will generate state change messages on the bus, and these messages will carry a ref to the element that posted the message. When the application has processed / popped these messages off the bus, those refs will get dropped. Just a guess of course.

3

This extra references for all the added elements stays even during long run. When this overlay is removed, this extra references are not getting removed. We are calling
gst_object_unref as per API reference manaul and due to the these extra references created during gst_element_set_state we are seeing memory leak during continuous overlay add\remove operation

4

And you are handling/processing GstBus messages in your application? (if not they’ll pile up and take more and more memory over time, at east until the pipeline gets shut down again)

5

Yes. We are seeing leak issues only during add and remove operation. There are no memory leak issues if we update the overlay (by pushing new buffer etc)

During remove overlay, after all our unref call, reference count of overlay elements are not reaching zero.

We referred gstreamer api reference

6

Can you share any references to dynamic overlay adding\removing samples?