I’m working on an embedded device that incorporates a relatively complex GST pipeline including interpipe objects (from Ridge Run). We’re running GStreamer 1.20.3 on an NXP i.MX 8M mini SoC running a Linux 5.15.71 kernel.
Here’s a stripped-down version of the pipeline that (I hope) can be used to reproduce the problem:
The main app pipeline looks like this:
! video/x-raw, width=1280, height=720, framerate=30/1, format=GRAY8
! tee name=t1
t1. ! appsink max=buffers=1 drop=true async=false
t1. ! queue max-size-buffers=1 max-size-bytes=0 leaky=downstream
! tee name=t2
t2. ! queue max-size-buffers=1 max-size-bytes=0 leaky=downstream
! udpsink host=host.example.com port=5000 sync=false async=false
t2. ! h264parse ! mpegtsmux
! interpipesink name=logger_sink sync=false async=false
appsinkis used to shunt video frames into application code for processing. No problems with that part.
cairooverlayis used for us to draw data based on the frames that have been processed. We’re having no problems with that part. (There is some latency where the app-sink processes frames faster than they can go through the overlay. We have code to deal with that. No problems there).
- The overlayed video is then send to our SoC’s hardware H.264 encode (
- The first branch of
t2wraps the raw H.264 video and sends it out over UDP, where we can stream it using VLC. No problems with this, other than some latency (to be expected).
- The second branch of
t2wraps the H.264 video into an MPEG-TS stream. In our product, we mux in audio as well (not shown here) and send the resulting data to a RidgeRun
interpipesink. This is so we can save the stream to an MPEG file, but selectively decide when to turn the file-saving on and off without halting the rest of the stream.
We have a second pipeline which receives the output of that interpipe to write the data to a file:
interpipesrc listen-to=logger_sink allow-renegotiation=false accept-events=false format=time
! filesink sync=false async=false
When we want to start saving the MPEG data to a file, we set the filesink’s
location property to the filename, then set that secondary pipeline into PLAYING state. When we want to stop saving MPEG data, we send an EOS the secondary pipeline and then wait for the bus to report EOS, after which we set its state to NULL.
In general, this all works.
We can stream live video (via RTP) from the UDP port. We see all this come through without problem and without interruption.
When we turn on the file-save pipeline, we get MP4 files. And we can play them to see the same video that we saw on the live stream.
But we find that some of the saved MP4 files begin with a period of black frames (no image) before the video content starts. This period may range between nothing and about 3 seconds. After which time, the video is all good.
On an older build, using GStreamer version 1.20.0 on Linux kernel 5.15.31, we see the same problem, but instead of black video during that initial period, it is a single frame from the video stream, duplicated for up to about 3s.
And just to reiterate, the output of the H264 encoder is not getting messed up because the RTP stream never shows these glitches. And the MPEG-TS mux isn’t stopped. The only thing changing when we record new files is starting and stopping that secondary pipeline.
I’m not sure what could be causing this. I suspect it has something to do with the stream generating key frames at about one every 3s, so the video file is unable to show anything until it his that first key frame. But that’s just a suspicion. I’m not certain about how to debug this problem, let alone how to go about fixing it.
Alternatively, is there a better approach to saving the MPEG data to files. Maybe it can be done without the interpipe? Would that change anything?