Hi all. I’ve got a live stream from a MIPI camera. I want my app to be able to save 5s clips from the stream on an on-demand fashion. What I’ve tried so far (see my other question Missing daa at the start of new MP4 file for a description there) isn’t quite working out that well. So instead of asking “why is this broken”, I’m going to see if “how should I do this” might be more meaningful.
My app is running on an embedded Linux device (an NXP i.MX 8M mini) which has a MIPI video input an a hardware H.264 encoder.
The core of the application is based on this pipeline fragment:
v4l2src device=/dev/video0
! video/x-raw,width=1280,height=720,framerate=30/1,format=GRAY8
! appsink max-buffers=1 drop=true async=false
With this, my application has a thread that repeatedly gets frames from the appsink (by emiting “pull-sample” signals) and processes them. It performs some object detection logic and logs the results, retaining the results from the most recent frames, indexed by their timestamp. No problems with any of this.
For debugging purposes, we tee the video and use a cairooverlay to draw the processing results on the frame (using the timestamps to keep it all in sync). Then we encode the result to H.264, wrap that in RTP and send it out over UDP, whee it can be viewed by VLC or other similar receivers:
v4l2src device=/dev/video0
! video/x-raw,width=1280,height=720,framerate=30/1,format=GRAY8
! tee name=t1
t1. ! appsink max-buffers=1 drop=true async=false
t1. ! queue max-size-buffers=1 max-size-bytes=0 leaky=downstream
! videoconvert
! cairooverlay
! videoconvert
! vpuenc_h264
! rtph264pay
! udpsink host=192.168.5.2 port=5000 sync=false async=false
The vpuenc_h264 element is provided by NXP and is the hardware CODEC for the chip’s VPU.
Again, this works. I can launch a client on the host at 192.168.5.2 and stream the video from port 5000.
So now, what I want to be able to do is to have a way to, at run-time, save the video to a file. What I’ve tried so far is to add another tee after the H.264 encoder, with that stream going into an MPEG-TS mux, then send the result to a RidgeRun interpipe, where another pipeline saves the stream to a file. By playing and stopping that secondary pipeline, I can start/stop the data saving to the file:
The primary pipeline:
v4l2src device=/dev/video0
! video/x-raw,width=1280,height=720,framerate=30/1,format=GRAY8
! tee name=t1
t1. ! appsink max-buffers=1 drop=true async=false
t1. ! queue max-size-buffers=1 max-size-bytes=0 leaky=downstream
! videoconvert
! cairooverlay
! videoconvert
! vpuenc_h264
! tee name=t2
t2. ! queue max-size-buffers=1 max-size-bytes=0 leaky=downstream
! rtph264pay
! udpsink host=192.168.5.2 port=5000 sync=false async=false
t2. ! queue max-size-buffers=1 max-size-bytes=0 leaky=downstream
! h264parse
! mpegtsmux
! interpipesink name=ip1 sync=false async=false
And the secondary pipeline:
interpipesrc listen-to=ip1 allow-renegotiation=false accept-events=false format=time
! filesink sync=false async=false
So when it’s time to save a video clip, I can set the filename on the filesink and then set the pipelne with the interpipesrc to PLAYING, causing it to save the MPEG-TS buffers to a file. And then stop that pipeline to close the file.
It works, but the resulting video file ends up with up to 3s of black video before the stream plays normally.
I suspect this is because H.264 uses key-frames and we’re just dumping data from the middle of the stream to a file. It syncs up, thanks to the MPEG-TS encoding, but not until the next key frame arrives, causing the player to show a black screen until then.
So the big question is: What is the best way to shunt video into a file in a way that can be turned on and off?
Stopping and restarting the pipeline itself isn’t an option, because that would interrupt the application reading from the appsink.
I assume I wouldn’t see this if I teed the raw frames (before the vpuenc_h264 element, and then encoded them in the secondary pipeline (the interpipesrc), performing the MPEG-TS muxing there. But now there’s two instances of the VPU encoder running.
That in itself might not be a problem, but the above fragment is part of a larger system. There may actually be multiple video files being recorded from the above pipeline, each with its own respective interpipe sink/src. If each one needs its own VPU encoding step, that might consume more CPU/VPU power than the hardware has available.
Any ideas?