Hello, I am trying to develop an application which brings in live HLS streams, extracts the captions, converts to CEA-708, and turns them into JSON. Currently, I’m able to bring in HLS, SRT, DASH, etc. and generate a thumbnail every second of the decoded video. But so far, I am unable to successfully see caption data. Here is an example stream I am testing with:
https://content.uplynk.com/channel/3324f2467c414329b3b0cc5cd987b6be.m3u8
What we see is that in our AppSinkCallback, we are never successfully pulling a new sample or even falling into that code block. We confirmed the caption pad of ccextractor is linked successfully. The ccconverter element is disabled for now, just trying to at least get a successful extraction.
Sample of code:
let vid_bin = Bin::builder().name("vid_bin").build();
// setup the tee elements to allow thumbnailing and monitoring
let vid_tee = ElementFactory::make("tee").name("vid_tee").build().unwrap();
vid_bin.add(&vid_tee).unwrap();
//create caption extraction and conversion elements
let cc_extract = ElementFactory::make("ccextractor")
.name("cc_extract")
.build()
.unwrap();
let cc_app_callback = AppSinkCallbacks::builder()
.new_sample(move |cc_app| {
println!("trying to pull sample");
println!("{:?}", cc_app.pull_sample());
Ok(gstreamer::FlowSuccess::Ok)
}).new_event(move |cc_app| {
println!("trying to pull object");
println!("{:?}", cc_app.pull_object());
true
}).new_preroll(move |cc_app| {
println!("trying to pull preroll");
println!("{:?}", cc_app.pull_preroll());
Ok(gstreamer::FlowSuccess::Ok)
})
.build();
let cc_app = AppSink::builder()
.name("cc_app")
.callbacks(cc_app_callback)
.caps(&Caps::builder("closedcaption/x-cea-708").field("format", "cc_data").build())
.async_(false)
.build();
let cc_que = ElementFactory::make("queue")
.name("cc_que")
.build()
.unwrap();
let cc_sink = ElementFactory::make("fakesink")
.name("cc_sink")
.build()
.unwrap();
let cc_convert = ElementFactory::make("ccconverter")
.name("cc_convert")
.build()
.unwrap();
// add the CCs to the bin
vid_bin.add(&cc_extract).unwrap();
vid_bin.add(&cc_que).unwrap();
vid_bin.add(&cc_sink).unwrap();
vid_bin.add(&cc_app).unwrap();
vid_bin.add(&cc_convert).unwrap();
vid_tee
.link_pads(Some("src_%u"), &cc_extract, Some("sink"))
.unwrap();
//cc_que
// .link_pads(Some("src"), &cc_extract, Some("sink"))
// .unwrap();
//cc_convert
// .link_pads(Some("src"), &cc_app, Some("sink"))
// .unwrap();
cc_extract.connect_pad_added(move |src, src_pad| {
println!("Received new pad {} from {}", src_pad.name(), src.name());
// Received new pad caption from cc_extract
if src_pad.name() == "caption" {
src
.link_pads(Some("caption"), &cc_app, Some("sink"))
.unwrap();
}
});
cc_extract
.link_pads(Some("src"), &cc_sink, Some("sink"))
.unwrap();
cc_sink.dynamic_cast::<BaseSink>().unwrap().set_async(false);
To better understand what might be wrong, I also tried to create this workflow using the command line. Based on reading other posts, this was the closest I came up with as we are using playbin3 within the application:
gst-launch-1.0 playbin3 uri=https://content.uplynk.com/channel/3324f2467c414329b3b0cc5cd987b6be.m3u8 video-sink="ccextractor name=extractor extractor.src ! fakesink async=false extractor.caption ! ccconverter ! closedcaption/x-cea-708,format=cdp ! fakesink dump=true async=false" -v
But even this does not successfully output real captions. Can anyone help point me in the right direction as to what could be wrong?