`max-buffers` Ignored in GStreamer Pipeline with IP Cameras on Raspberry Pi

Application Development
1

We have a network of IP cameras (IMX291) connected to Raspberry Pis, used for meteor observation. The application handling the video stream is written in Python. Our GStreamer pipeline is as follows:

rtspsrc protocols=tcp tcp-timeout=5000000 retry=5 location="rtsp://192.168.42.10:554/user=admin&password=&channel=1&stream=0.sdp/" ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw,format=BGR ! appsink max-buffers=25 drop=true sync=1 name=appsink

The issue we’re facing is that the max-buffers=25 setting appears to be ignored. On slower machines, the application cannot keep up, and the appsink buffer fills up with several minutes worth of 1080p video at 25 fps, until it suddenly empties completely. This results in the loss of several minutes worth of data. Ideally, we would prefer a mechanism where frames are dropped gradually rather than in large chunks.

I would greatly appreciate any insights or help with this issue. Thank you!

2

Appsink being placed after your slow elements, it’s “leaky” feature won’t work and data will accumulate inside rtspsrc jitter buffer. The jitter buffer have a “drop-on-latency” feature, which is one option. And the queue element have a leaky property too. Note that dropping before the decoder will cause corruption.

Have you considered using v4l2h264dec, the HW decoder?

3

Thank you for the help.

Yes, we tried using v4l2h264dec, but couldn’t get it to work. I’m not sure why, but it failed on all our Pis, from RPi3 running Buster to RPi5 running Bookworm.

I added ...h264parse ! queue max-size-buffers=25 ! avdec_h264... to the pipeline, but the buffer still grows indefinitely.

I also tried queue leaky=downstream max-size-buffers=25, but this configuration significantly slowed down the pipeline.

Another requirement we have is to precisely timestamp each frame, with the timestamp being as close as possible to the capture time. Currently, we add the presentation timestamp (pts) to the time at which the pipeline is launched. The pts is remarkably accurate, but capturing the correct start time is challenging. There’s probably a better way to do this that I’m unaware of. Regardless, maintaining accurate pts is essential.

4

On Pi5, there is no HW decoders (except for HEVC, but the driver is not currently usable by GStreamer). If your issue also exist on Pi5, then performance is not the real issue. Perhaps what you are doing to timestamp is causing trouble. On Pi3, you are out of luck if you really don’t want to debug the HW decoder, its a very slow CPU.

At the moment, you seem to drop a lot of your requirements, and what does not work, but for anyone on the forum to usefully help you, you’ll have to share things that we can reproduce.

5

No performance issues with RPi5 and most RPi4 models. However, some RPi4 and RPi3 are experiencing issues with to low-performing or worn-out SD cards.

Putting the performance problem aside, the issue at hand is the inability to control pipeline buffers effectively. When I artificially underclock the CPU to simulate slower machines, even with queue max-size-buffers=X placed between each element, retrieved frames become progressively older, eventually reaching about 10 minutes in delay. At that point, the buffer flushes, resulting in the loss of that chunk of data.

Adding leaky=downstream or leaky=upstream does seem to drop frames as desired, but it significantly slows down the pipeline. Meaning, a pipeline that was previously able to keep up is now unable to keep up.

For instance, the following line, which serves as a good stress test since converting video to grayscale with videoconvert doesn’t use fast-path, shows an ever-increasing buffer delay:

gst-launch-1.0 rtspsrc location='rtsp://192.168.42.10:554/user=admin&password=&channel=1&stream=0.sdp/' ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw,format=GRAY8 ! fakesink

The same happens with queues added between each element:

gst-launch-1.0 rtspsrc location='rtsp://192.168.42.10:554/user=admin&password=&channel=1&stream=0.sdp/' ! queue max-size-buffers=25 ! rtph264depay ! queue max-size-buffers=25 ! h264parse ! queue max-size-buffers=25 ! avdec_h264 ! queue max-size-buffers=25 ! videoconvert ! queue max-size-buffers=25 ! video/x-raw,format=GRAY8 ! queue max-size-buffers=25 ! fakesink

This setup below does drop frames but results in extreme slowness:

gst-launch-1.0 rtspsrc location='rtsp://192.168.42.10:554/user=admin&password=&channel=1&stream=0.sdp/' ! rtph264depay ! h264parse ! avdec_h264 ! queue leaky=downstream max-size-buffers=25 ! videoconvert ! video/x-raw,format=GRAY8 ! fakesink

I don’t understand how to make this work.

Edit: GStreamer 1.22.0

6

What you describe doesn’t really match what one would expect, so it’s all a bit hard to follow.

What does “dropping frames results in extreme slowness” mean? Slowness measured/observed in what way? Dropping raw video frames should be cheap and shouldn’t make anything worse.

Regarding the queue configuration, keep in mind that there are three limits: max-size-buffers, max-size-bytes, max-size-time. Whichever is hit first “wins”. The byte limit is 1M or so by default, which is easy to hit with raw video frames. So you may want to set the other limits to 0 if you configure the queue.

In case your stream contains B-frames you could configure avdec_h264 skip-frame=1 on slower machines to skip those frames there.

Otherwise you could also try sink sync=true qos=true max-lateness=20000000 or somesuch, if that helps (that let’s the decoder know how early/late frames arrive at the sink, so it can skip frames that will be late anyway, but how well that will work in practice for you is to be seen.

I don’t quite follow the " the appsink buffer fills up with several minutes worth of 1080p video at 25 fps, until it suddenly empties completely" either, that sounds very odd. Not only the sudden emptying, but also why does it fill up in the first place? Are you not pulling out data fast enough?

1 Like
7

Thank you for all the good info and I apologize for the lack of clarity.
Your explanation about the three limits was very helpful and I was finally able to get it to work (see end of post).

Regarding my previous statement:

Adding leaky=downstream or leaky=upstream does seem to drop frames as desired, but it significantly slows down the pipeline. Meaning, a pipeline that was previously able to keep up is now unable to keep up.

What I observe is that if I add a leaky parameter to queue leaky=downstream max-size-buffers=25 anywhere in the pipeline, it takes more CPU resources, or more time if the CPU resource is limited, for the GStreamer pipeline to process each frame. This is not what is expected. I can see that simply by measuring the time between each frame’s retrieval. When adding leaky=downstream or leaky=upstream , the interval between frame retrievals from the pipeline exceeds 1/(camera fps). I suspect leaky breaks something in the pipeline in my case.

Regarding this previous statement:

…retrieved frames become progressively older, eventually reaching about 10 minutes in delay. At that point, the buffer flushes, resulting in the loss of that chunk of data.

What I observe is that when the interval between frame retrievals is longer than 1/(camera fps), the elapsed time between when a frame is captured and when it is retrieved from the pipeline gets longer. That interval grows to several minutes. The interval peaks at approximately 10 minutes and then suddenly drops to about 2 seconds. This means that several minutes’ worth of video frames are lost.

Now, thanks to your explanations, I finally got it to work.
The key was to put a queue on each side of videoconvert, specifying all three limits on each side, and leaky on the left side. I admittedly have no idea why, but by process of elimination, this is the winning pipeline:

rtspsrc protocols=tcp tcp-timeout=5000000 retry=5 location="rtsp://192.168.42.10:554/user=admin&password=&channel=1&stream=0.sdp/" ! rtph264depay ! h264parse ! avdec_h264 ! queue leaky=downstream max-size-buffers=250 max-size-bytes=0 max-size-time=0 ! videoconvert ! video/x-raw,format=BGR ! queue max-size-buffers=250 max-size-bytes=0 max-size-time=0 ! appsink max-buffers=250 drop=true sync=1 name=appsink

If I remove or add anything to both queues’ parameters, the pipeline misbehave. Now, this pipeline has a 10s buffer after which it starts dropping frames piecemeal. Sweet!