I’m building a video analysis application in Rust using GStreamer where users can choose different processing scales (e.g., analyze every frame, every 2nd frame, every 10th frame, etc.). I’ve attempted to implement frame dropping using the videorate element, but it seems to only affect the display of progress and not the actual frame processing.
For example, with a 30fps video that’s 2 minutes long (~3600 frames):
When selecting 10x speed (process every 10th frame), the progress shows “360/360 frames”.
When it reaches “360/360”, it says completed but continues processing in the background.
The actual processing time remains the same as if processing all frames.
The backend appears to still decode and process all 3600 frames.
Here’s my current pipeline setup:
fn create_single_pipeline(
path: &PathBuf,
data_tx: &Sender<FrameCmd>,
video_index: usize,
orientation: Orientation,
is_muxed: bool,
processing_scale: f32,
) -> Result<gst::Pipeline> {
let file_info = MediaInfo::from(path)?;
let duration = file_info.duration;
let video = file_info
.video
.first()
.ok_or(anyhow!("No video streams found in file"))?;
let video_width = video.width;
let video_height = video.height;
let input_fps = f64::from(video.framerate.numer()) / f64::from(video.framerate.denom());
let target_fps = (input_fps * processing_scale as f64).max(1.0); // Ensure minimum 1 fps
// Adjust estimated frames based on target fps
let video_estimated_total_frames: u64 = (target_fps * duration.as_secs_f64()).floor() as u64;
debug!(
"Video {}: input {} x {}, input fps {}, target fps {}, estimated frames {}",
video_index, video_width, video_height, input_fps, target_fps, video_estimated_total_frames
);
let pipeline = gst::Pipeline::new();
let src = gst::ElementFactory::make("filesrc")
.property("location", path.to_str().ok_or(anyhow!("Invalid path"))?)
.build()?;
let decodebin = gst::ElementFactory::make("decodebin").build()?;
pipeline.add_many([&src, &decodebin])?;
gst::Element::link_many([&src, &decodebin])?;
let pipeline_weak = pipeline.downgrade();
let data_tx_clone = data_tx.clone();
decodebin.connect_pad_added(move |dbin, src_pad| {
let pipeline = match pipeline_weak.upgrade() {
Some(pipeline) => pipeline,
None => return,
};
let (is_audio, is_video) = {
let media_type = src_pad.current_caps().and_then(|caps| {
caps.structure(0).map(|s| {
let name = s.name();
(name.starts_with("audio/"), name.starts_with("video/"))
})
});
match media_type {
None => {
element_warning!(
dbin,
gst::CoreError::Negotiation,
("Failed to get media type from pad {}", src_pad.name())
);
return;
}
Some(media_type) => media_type,
}
};
let insert_sink = |is_audio: bool, is_video: bool| -> Result<(), Error> {
if is_audio {
Self::setup_audio_pipeline(&pipeline, src_pad, data_tx_clone.clone(), video_index)
} else if is_video {
let queue1 = gst::ElementFactory::make("queue")
.property("max-size-buffers", 2u32)
.build()?;
let videorate = gst::ElementFactory::make("videorate")
.property("drop-only", true)
.property("skip-to-first", true)
.build()?;
// Set target framerate caps
let rate_caps = gst::Caps::builder("video/x-raw")
.field("framerate", gst::Fraction::new(
(target_fps * 1000.0) as i32,
1000
))
.build();
let rate_filter = gst::ElementFactory::make("capsfilter")
.property("caps", &rate_caps)
.build()?;
let convert = gst::ElementFactory::make("videoconvert").build()?;
let scale = gst::ElementFactory::make("videoscale").build()?;
pipeline.add_many([&queue1, &videorate, &rate_filter, &convert, &scale])?;
gst::Element::link_many([&queue1, &videorate, &rate_filter, &convert, &scale])?;
src_pad.link(&queue1.static_pad("sink").unwrap())?;
if is_muxed {
Self::setup_muxed_video_pipeline(
&pipeline,
&scale,
data_tx_clone.clone(),
video_width,
video_height,
video_estimated_total_frames,
video_index,
)
} else { //others for the different cases...
And here’s how I handle the video sink:
fn setup_video_sink(
sink: gst_app::AppSink,
width: u32,
height: u32,
estimated_total_frames: u64,
side: VideoFrameSide,
video_index: usize,
data_tx: Sender<FrameCmd>,
) -> Result<(), Error> {
sink.set_callbacks(
gst_app::AppSinkCallbacks::builder()
.new_sample(move |appsink| {
let sample = appsink.pull_sample().map_err(|_| gst::FlowError::Eos)?;
let buffer = sample.buffer().ok_or_else(|| {
element_error!(
appsink,
gst::ResourceError::Failed,
("Failed to get buffer from appsink")
);
gst::FlowError::Error
})?;
let map = buffer.map_readable()?;
let samples = map.as_slice();
let frame = match side {
VideoFrameSide::Left => VideoFrameData::left(
width,
height,
estimated_total_frames,
samples,
video_index,
),
VideoFrameSide::Right => VideoFrameData::right(
width,
height,
estimated_total_frames,
samples,
video_index,
),
}?;
data_tx.send(FrameCmd::video_frame(frame))?;
Ok(gst::FlowSuccess::Ok)
})
.build(),
);
Ok(())
}
Is my approach to frame dropping using videorate correct? The pipeline seems to still process all frames despite the rate limiting.
Should I be looking at a different approach or different elements to achieve actual frame dropping at decode time?
Is there something specific about using AppSink that might be bypassing the frame dropping?
Environment:
GStreamer version: Latest
Operating System: macOS
Rust GStreamer bindings version: Latest
Any guidance would be greatly appreciated.
It’s been quite a while since then, and I’ve made my pipeline much more stable and implemented many of the improvements I mentioned above.. Of course I did many sanity checks/tests to make sure… and you are correct… My old assumption about dropping frames before decoding was actually flawed 
. Because of how GOP structures and P-frames work, standard decoders must process the stream sequentially to prevent video corruption; they don’t do any magic partial frame merging. In my current architecture, the