How to use ccextractor element?

Application Development
1

Hello

I am planning to extend my GStreamer tool a bit.
Currently I use GStreamer to demux video streams.
At the end, after the tsdemux, there are two AppSinks that provide video and audio packets separately.
The decode is NOT done with GStreamer.

Now I would like to get the ClosedCaption (CC) data.
But I am not quite sure how to implement this.
My first idea was to use the ccextract element.
Unfortunately, I have not found an example of how and where to include it in my pipeline.
I assume that I have to add it to my branch after the tsdemux.
Is that correct?
If I have understood correctly, the CC data at CEA 708 is in the Picure User Data of the video data. So I should be able to access the data after the tsdemux, right?
However, the documentation says that this element should be behind a decode. Do I understand this wrong? Can I still get the Picture User Data after the decode? Are they not on the NAL unit level?

2

These captions are usually embedded in the encoded video bitstream, so you will usually need at least a video parser to extract them (in form of metas on the buffers).

The problem with encoded video is that if it contains B-frames the encoded video frames might be in a different order than the decoded video frames. A video decoder will reorder the frames to make sure you get them in display order.

The question then is what happens with any captions that are on B-frames? should they be in encoding order and then get reordered by the decoder or should their order be maintained as they’re in the bitstream? I don’t know if this is really speced properly. Other people will know. I think often B-frames simply don’t carry caption data to avoid this problem, perhaps.

Anyway, this is all to say that your mileage may vary, as the saying goes, and that this is something to look out for. You can see how it works for you with your streams.

3

Thanks for the answer

do you have a running example for the ccextractor element?

Whether as gst-launch-1.0 or as code is irrelevant.
I just want to see the data first. Whether in a file or as a log doesn’t matter to me.

4

Does anyone have an idea how I can get the CC data with GStreamer?
I really can’t find an approach or examples.

Otherwise my solution will be to use ffmpeg.

5

ccextractor will split the GstVideoCaptionMeta from incoming video buffers into a separate stream. The GstVideoCaptionMeta is usually set by parsers (when decoding) but the order of them is not really a specified thing. Use is like this:

filesrc location=path-to-video-with-captions ! demuxer ! parser ! ccextractor name=extractor extractor.src ! fakesink async=false extractor.caption ! fakesink dump=true async=false -v

6

The element’s documentation also mentions where to place the element in the pipeline (after the decoder): ccextractor

7

Thank you for your answer.
Now I am one step further, but unfortunately I get an error message.

gst-launch-1.0 filesrc location=GVH_full_capture_ts.ts ! tsdemux  ! tsparse ! ccextractor name=extractor extractor.src ! fakesink async=false extractor.caption ! fakesink dump=true async=false -v

Output

Setting pipeline to PAUSED ...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
WARNING: from element /GstPipeline:pipeline0/GstTSDemux:tsdemux0: Delayed linking failed.
Additional debug info:
subprojects/gstreamer/gst/parse/grammar.y(859): gst_parse_no_more_pads (): /GstPipeline:pipeline0/GstTSDemux:tsdemux0:
failed delayed linking some pad of GstTSDemux named tsdemux0 to some pad of MpegTSParse2 named mpegtsparse2-0
ERROR: from element /GstPipeline:pipeline0/GstTSDemux:tsdemux0: Internal data stream error.
Additional debug info:
../subprojects/gst-plugins-bad/gst/mpegtsdemux/mpegtsbase.c(1759): mpegts_base_loop (): /GstPipeline:pipeline0/GstTSDemux:tsdemux0:
streaming stopped, reason not-linked (-1)
Execution ended after 0:00:00.002103000
Setting pipeline to NULL ...
Freeing pipeline ...

File Discover

gst-discoverer-1.0 GVH_full_capture_ts.ts -v
Analyzing file:///Users/patrickfischer/Video/GVH_full_capture_ts.ts
Done discovering file:///Users/patrickfischer/Video/GVH_full_capture_ts.ts

Properties:
  Duration: 0:03:12.422129777
  Seekable: yes
  Live: no
  Tags: 
      Sprachcode: es
      Audio-Codec: AC-3 (ATSC A/52)
      Video-Codec: H.264
  container #0: video/mpegts, systemstream=(boolean)true, packetsize=(int)188
    Tags:
      None
    
    video #1: video/x-h264, stream-format=(string)avc, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)60000/1001, coded-picture-structure=(string)frame, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)high, level=(string)3.2, codec_data=(buffer)01640020ffe1002567640020ac2ca5014016ec068408080900000303e90001d4c0e0c00030724003075ab9c02801000468ef3cb0
      Tags:
        Video-Codec: H.264
      
      Codec:
        video/x-h264, stream-format=(string)avc, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)60000/1001, coded-picture-structure=(string)frame, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)high, level=(string)3.2, codec_data=(buffer)01640020ffe1002567640020ac2ca5014016ec068408080900000303e90001d4c0e0c00030724003075ab9c02801000468ef3cb0
      Stream ID: 3157a63877ee13071bb60a3fa97c39844cb2b71af3d16f53c24bd47e8492491e:1/00000031
      Width: 1280
      Height: 720
      Depth: 24
      Frame rate: 60000/1001
      Pixel aspect ratio: 1/1
      Interlaced: false
      Bitrate: 0
      Max bitrate: 0
    audio #2: audio/x-ac3, framed=(boolean)true, rate=(int)48000, channels=(int)6, alignment=(string)frame
      Tags:
        Sprachcode: en
        Audio-Codec: AC-3 (ATSC A/52)
      
      Codec:
        audio/x-ac3, framed=(boolean)true, rate=(int)48000, channels=(int)6, alignment=(string)frame
      Stream ID: 3157a63877ee13071bb60a3fa97c39844cb2b71af3d16f53c24bd47e8492491e:1/00000032
      Language: en
      Channels: 6 (front-left, front-right, front-center, lfe1, side-left, side-right)
      Sample rate: 48000
      Depth: 32
      Bitrate: 0
      Max bitrate: 0
    audio #3: audio/x-ac3, framed=(boolean)true, rate=(int)48000, channels=(int)2, alignment=(string)frame
      Tags:
        Sprachcode: es
        Audio-Codec: AC-3 (ATSC A/52)
      
      Codec:
        audio/x-ac3, framed=(boolean)true, rate=(int)48000, channels=(int)2, alignment=(string)frame
      Stream ID: 3157a63877ee13071bb60a3fa97c39844cb2b71af3d16f53c24bd47e8492491e:1/00000033
      Language: es
      Channels: 2 (front-left, front-right)
      Sample rate: 48000
      Depth: 32
      Bitrate: 0
      Max bitrate: 0

file ffprobe

Input #0, mpegts, from 'GVH_full_capture_ts.ts':
  Duration: 00:03:14.93, start: 51264.085278, bitrate: 3779 kb/s
  Program 1 
  Stream #0:0[0x31]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn
  Stream #0:1[0x32](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 384 kb/s
  Stream #0:2[0x33](spa): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 96 kb/s
  Stream #0:3[0x34]: Data: scte_35

it contains an CEA 708 CC

gst-launch-1.0 --version
gst-launch-1.0 version 1.22.10
GStreamer 1.22.10
8

I get data with this pipeline.
That’s already very good. However, I get an extremely large amount of data.
I’ll have to see what happens.
Maybe I still need a caps filter

gst-launch-1.0 urisourcebin uri=file:///Users/patrickfischer/Video/GVH_full_capture_ts.ts ! parsebin ! ccextractor name=extractor extractor.src ! fakesink async=false extractor.caption ! fakesink dump=true async=false -v
9

That looks correct. It depends on the exact format you’re dealing with as to what the output will look like.

You might see data that starts with 0x96 0x69, or contains some 0xfc 0x80 0x80, or 0xfa 0x00 0x00 or some variations.

10

Exactly

00000000 (0x600002bccdb0): fd 80 80 fc 80 80 fa 00 00 fa 00 00 fa 00 00 fa  ................
00000010 (0x600002bccdc0): 00 00 fa 00 00 fa 00 00 fa 00 00 fa 00 00 fa 00  ................
00000020 (0x600002bccdd0): 00 fa 00 00 fa 00 00 fa 00 00 fa 00 00 fa 00 00  ................
00000030 (0x600002bccde0): fa 00 00 fa 00 00 fa 00 00 fa 00 00              ............    

/GstPipeline:pipeline0/GstCCConverter:ccconverter0.GstPad:src: caps = closedcaption/x-cea-708, format=(string)cc_data, framerate=(fraction)60000/1001
/GstPipeline:pipeline0/GstFakeSink:fakesink1.GstPad:sink: caps = closedcaption/x-cea-708, format=(string)cc_data, framerate=(fraction)60000/1001
/GstPipeline:pipeline0/GstCCConverter:ccconverter0.GstPad:sink: caps = closedcaption/x-cea-708, format=(string)cc_data, framerate=(fraction)60000/1001

I can really work with that!
I could now decode the data with a CEA-708 decoder, but does GStreamer also have an element that can take over this task?

11

There is a cc708overlay element that contains a 708 decoder internally: subprojects/gst-plugins-bad/ext/closedcaption/gstcea708decoder.c · main · GStreamer / gstreamer · GitLab

If you want to write your own, then probably using something like cea708-types is a good idea.

1 Like
12

back on topic :slight_smile:

after a while i still want to extract the cc from a stream.

as you said, i need a decoder to get the CC in correct order.

I thought i can use the decodebin element.
Unfortunately I don’t know how to bind it right now.
If I attach it after the decodebin element, then the CC are no longer present and I only have a video/x-raw
Where and how do I have to attach it?
I write the app in C

pipeline_1721306328687
pipeline_17213063286879059Ă—984 529 KB