Pull sample on demand from appsink

General Discussion
1

hi! I have created a sink abstraction to read from cameras StreamCapture where I have implemented a stream.grab() method that each time is called will return the latest available frame coming from the gst pipeline.

In the way is implemented now, we have to set a callback that stores in an optional arc optional latest frame which requires to have an additional running thread in order to make the pipeline to flow.

Is there any good practice to avoid to create that extra thread running in the background and set the pipeline to flow as a non-blocking method ? Ideally I would like to be able to call the pull_sample method and extract (if available) the latest frame directly from the appsink component instead of having to set a callback.

Is that even possible to do in gstreamer ?

Thanks in advance ! Edgar

2

You can call pull_sample() from any thread you want. The callback is just there to notify you that a new sample is available. pull_sample() would block until one is available or the appsink is shut down. You could for example use the callback to notify your application code that a sample is available and then pull it (there’s appsink.stream() which gives you an async Stream that does basically that).

You currently use a separate thread for handling the bus messages. You can also do that via a) a callback (bus.set_sync_handler(), careful because you can’t change any element states from that thread), b) an async stream (bus.stream(), which is just a channel around a sync handler), or c) do your own integration into your event loop in one of the many ways (sync handler, fd readiness, …).

3

Thanks for the answer ! I’ve seen also that there’s a try_pull_sample() with a timeout. I believe this is the functionality I need as for the use case I’m targeting to grab frames at a constant rate for robotics applications.

I have the doubt thought, if the user calls try_pull_sample() let’s say with the minimum timeout (1ns) every 30Hz and misses the sample, what happens internally ? Does gstreamer drop the sample or just puts into a queue until the next pull_sample call ?

Ideally I want that each time the use call the grab() method it gets the latest available frame served by the stream and give ownership to the user with zero-copy (if you could comment here).

I tried, bus.set_sync_handler() and seems to work well. I guess it’s running a job in the background in any case, right ?

For now I got the following solution

pub fn grab(&self) -> Result<Option<Image<u8, 3>>, StreamCaptureError> {
    self.appsink
        .try_pull_sample(gst::ClockTime::ZERO)
        .map(Self::extract_image_frame)
        .transpose()
}
4

It’s a timeout. If there’s a sample available it returns immediately, otherwise it returns at latest after the timeout and if there was a sample in that time it returns that or otherwise NULL.

If samples are arriving faster than you pull them, they will pile up internally in the appsink. There’s a queue (and you can configure its size).

What you describe doesn’t sound like a good solution to me. You generally don’t want to poll, ever. And instead get notified whenever a frame is available and then react to that.

You could always pull the frame from the new-sample callback and store it in some shared state, then from grab() you get the latest sample that is currently stored there. That’s also how e.g. gtk4paintablesink works (GTK is notified when a new frame is available, whenever GTK wants to render it gets the latest frame at that moment).

GStreamer is running (one or more depending on the pipeline) threads in the background for doing media processing. None of that is happening in your application’s thread.

The sync handler gets called directly from whatever thread is posting the message.