Inserting SEI metadata in h264

General Discussion
1

Hello!

I ran into an issue when trying to use the gst_buffer_add_video_sei_user_data_unregistered_meta function from https://gstreamer.freedesktop.org/documentation//video/gstvideosei.html?gi-language=c#GstVideoSEIUserDataUnregisteredMeta.

I’m using gstreamer 1.22.8 with docker container livekit/gstreamer:1.22.8-dev:

gst-inspect-1.0 --version

gst-inspect-1.0 version 1.22.8
GStreamer 1.22.8
Unknown package origin

As a basis, I took this repository gstreamer-boilerplate-cpp which describes the steps for creating your own plugin in C++.

I changed the gst_my_filter_chain function gstreamer-boilerplate-cpp/gst-plugin/src/gstmyfilter.cpp at master · ozankaraali/gstreamer-boilerplate-cpp · GitHub
in the following way:

static GstFlowReturn
gst_my_filter_chain (GstPad * pad, GstObject * parent, GstBuffer * buf)
{
  GstMyFilter *filter;

  filter = GST_MYFILTER (parent);

  /*if (filter->silent == FALSE){
    g_print ("Loaded!");
    // Now we can use iostream C++:
    std::cout<< "Test" <<std::endl;
  }*/

  // Create a UUID for the SEI data
  guint8 uuid[16] = {0x4d, 0x4f, 0x4d, 0x4f, 0x54, 0x4f, 0x00, 0x00,
                   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};

  // Create some SEI data (for example, a string)
  const gchar *sei_data = "MOMOTO";
  gsize sei_size = strlen (sei_data);

  // Add the SEI data to the buffer as metadata
  GstVideoSEIUserDataUnregisteredMeta* ret = gst_buffer_add_video_sei_user_data_unregistered_meta(buf, uuid, (guint8 *) sei_data, sei_size);

  // std::cout<< "RET VALUE" << ret <<std::endl;

  // GST_DEBUG_OBJECT (filter, "SEI data added to buffer");
  // GST_DEBUG_OBJECT (filter, "SEI data added to buffer: %.*s", sei_size, (gchar*)sei_data);


  /* just push out the incoming buffer without touching it */
  return gst_pad_push (filter->srcpad, buf);
}

also I included gstreamer-video-1.0 in configure.ac:

PKG_CHECK_MODULES(GST, [
  gstreamer-1.0 >= $GST_REQUIRED
  gstreamer-base-1.0 >= $GST_REQUIRED
  gstreamer-controller-1.0 >= $GST_REQUIRED
  gstreamer-audio-1.0 >= $GST_REQUIRED
  gstreamer-video-1.0 >= $GST_REQUIRED
]

after building the source code:

cd gst-plugin
./autogen.sh

Libraries have been installed in:
/usr/local/lib/gstreamer-1.0

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the ‘-LLIBDIR’
flag during linking and do at least one of the following:

  • add LIBDIR to the ‘LD_LIBRARY_PATH’ environment variable
    during execution
  • add LIBDIR to the ‘LD_RUN_PATH’ environment variable
    during linking
  • use the ‘-Wl,-rpath -Wl,LIBDIR’ linker flag
  • have your system administrator add LIBDIR to ‘/etc/ld.so.conf’

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

make[2]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin/src’
make[1]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin/src’
make[1]: Entering directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’
make[2]: Entering directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’
make[2]: Nothing to be done for ‘install-exec-am’.
make[2]: Nothing to be done for ‘install-data-am’.
make[2]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’
make[1]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’

I see a message about the successful build of the plugin at the following path:
/usr/local/lib/gstreamer-1.0/libgstmyfilter.so

Plugin verification is successful:
gst-inspect-1.0 /usr/local/lib/gstreamer-1.0/libgstmyfilter.so

Plugin Details:
Name myfilter
Description Template myfilter
Filename /usr/local/lib/gstreamer-1.0/libgstmyfilter.so
Version 1.0.0
License LGPL
Source module my-filter
Binary package GStreamer

myfilter: MyFilter

1 features:
±- 1 elements

I’m running a pipeline that uses my built plugin. The debug messages show that a GstVideoSEIUserDataUnregisteredMeta object of size 48 is being created:

GST_DEBUG=6 gst-launch-1.0 videotestsrc num-buffers=30 ! x264enc tune=zerolatency ! myfilter ! h264parse ! matroskamux ! filesink location=1.mkv

0:00:00.678695478 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:880:gst_buffer_new: new 0x7f35001187f0
0:00:00.678701669 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:570:gst_buffer_copy_into: copy 0x7f3500144140 to 0x7f35001187f0, offset 0-6204/6204
0:00:00.678707139 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:463:_memory_add: buffer 0x7f35001187f0, idx -1, mem 0x7f3500132440
0:00:00.678712580 136769 0x7f3508000b70 DEBUG GST_PERFORMANCE gstminiobject.c:540:ensure_priv_data: allocating private data GstMemory miniobject 0x7f3500132440
0:00:00.678717328 136769 0x7f3508000b70 DEBUG video-sei video-sei.c:110:gst_video_sei_user_data_unregistered_meta_transform: copy SEI User Data Unregistered metadata
0:00:00.678722508 136769 0x7f3508000b70 DEBUG GST_BUFFER gstbuffer.c:2337:gst_buffer_add_meta: alloc metadata 0x7f350011ef30 (GstVideoSEIUserDataUnregisteredMeta) of size 48
0:00:00.678728760 136769 0x7f3508000b70 DEBUG GST_PERFORMANCE gstminiobject.c:440:gst_mini_object_make_writable: copy GstBuffer miniobject 0x7f3500144140 → 0x7f35001187f0
0:00:00.678737917 136769 0x7f3508000b70 LOG baseparse gstbaseparse.c:2248:gst_base_parse_handle_buffer: handling buffer of size 6204 with dts 1000:00:00.866666666, pts 1000:00:00.866666666, duration 99:99:99.999999999
0:00:00.678745822 136769 0x7f3508000b70 LOG baseparse gstbaseparse.c:2211:gst_base_parse_prepare_frame: preparing frame at offset 18446744073709551615 (0xffffffffffffffff) of size 6204
0:00:00.678751482 136769 0x7f3508000b70 LOG baseparse gstbaseparse.c:804:gst_base_parse_update_frame: marking as new frame
0:00:00.678757954 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:1863:gst_buffer_map_range: buffer 0x7f35001187f0, idx 0, length -1, flags 0001
0:00:00.678764436 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:306:_get_merged_memory: buffer 0x7f35001187f0, idx 0, length 1
0:00:00.678770007 136769 0x7f3508000b70 LOG h264parse gsth264parse.c:1252:gst_h264_parse_handle_frame_packetized: processing packet buffer of size 6204
0:00:00.678775597 136769 0x7f3508000b70 DEBUG codecparsers_h264 gsth264parser.c:250:gst_h264_parse_nalu_header: Nal type 9, ref_idc 0
0:00:00.678780546 136769 0x7f3508000b70 DEBUG h264parse gsth264parse.c:1264:gst_h264_parse_handle_frame_packetized: AVC nal offset 6
0:00:00.678788110 136769 0x7f3508000b70 DEBUG h264parse gsth264parse.c:971:gst_h264_parse_process_nal: processing nal of type 9 AU delimiter, size 2
0:00:00.678794242 136769 0x7f3508000b70 LOG h264parse gsth264parse.c:1174:gst_h264_parse_process_nal: collecting NAL in AVC frame
0:00:00.678800383 136769 0x7f3508000b70 DEBUG h264parse gsth264parse.c:488:gst_h264_parse_wrap_nal: nal length 2
0:00:00.678806615 136769 0x7f3508000b70 DEBUG GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x7f3500126140, maxsize:13 offset:0 size:6

after that I try to find my metadata in the video either through the console output:

gst-launch-1.0 filesrc location=1.mkv ! matroskademux ! myfilter ! filesink location=/dev/stdout | hexdump -C | grep MOMO

0:00:00.029950355 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received stream-start event: stream-start event: 0x7f3728013900, time 99:99:99.999999999, seq-num 21, GstEventStreamStart, stream-id=(string)d4370fbf6e250a336cc468433b1516904107e477db6e0eb715f3f082dc02c2b3/001:6486666301394315289, flags=(GstStreamFlags)GST_STREAM_FLAG_SELECT, group-id=(uint)1;
0:00:00.030068758 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received caps event: caps event: 0x7f37280123b0, time 99:99:99.999999999, seq-num 22, GstEventCaps, caps=(GstCaps)“video/x-h264,\ level=(string)1.3,\ profile=(string)high-4:4:4,\ codec_data=(buffer)01f4000dffe1001b67f4000d919640507ec05a83030320000003002000000791e2854901000668ebcc448440,\ stream-format=(string)avc,\ alignment=(string)au,\ width=(int)320,\ height=(int)240,\ framerate=(fraction)30/1,\ interlace-mode=(string)progressive,\ colorimetry=(string)bt601”;
0:00:00.030135974 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received tag event: tag event: 0x7f3728013d00, time 99:99:99.999999999, seq-num 23, GstTagList-global, taglist=(taglist)“taglist,\ datetime=(datetime)2024-01-18T06:24:30.298886Z,\ container-format=(string)Matroska;”;
0:00:00.030179226 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received tag event: tag event: 0x7f3728014420, time 99:99:99.999999999, seq-num 28, GstTagList-stream, taglist=(taglist)“taglist,\ video-codec=(string)H264,\ encoder=(string)x264,\ bitrate=(uint)1431656;”;
0:00:00.030366628 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received segment event: segment event: 0x7f3728014030, time 99:99:99.999999999, seq-num 33, GstEventSegment, segment=(GstSegment)“segment, flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1, applied-rate=(double)1, format=(GstFormat)time, base=(guint64)0, offset=(guint64)0, start=(guint64)0, stop=(guint64)18446744073709551615, time=(guint64)0, position=(guint64)0, duration=(guint64)999999999;”;
0:00:00.060276106 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received eos event: eos event: 0x7f372801a2c0, time 99:99:99.999999999, seq-num 48, (NULL)

or via ffmpeg pyav:

import av
from uuid import UUID

container = av.open("./1.mkv")
stream = container.streams.video[0]

for packet in container.demux(stream):
    for frame in packet.decode():
        for sd in list(frame.side_data.keys()):
            bts = bytes(sd)
            uuid_bts = bts[:16]
            uuid_str = str(UUID(bytes=uuid_bts))

            message = bts[16:].decode('utf-8')
            print(f"message: {message}")
            print(f"uuid: {uuid_str}")

python avpipeline.py

message: x264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=3 lookahead_threads=3 sliced_threads=1 slices=3 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=300 keyint_min=30 scenecut=40 intra_refresh=0 rc_lookahead=0 rc=cbr mbtree=0 bitrate=2048 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=2048 vbv_bufsize=1228 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:1.00
uuid: dc45e9bd-e6d9-48b7-962c-d820d923eeef

I don’t see any metadata inserted. Even if you use x264enc:

gst-launch-1.0 videotestsrc ! x264enc ! myfilter ! fakesink dump=true | xxd -c 48 | grep MOMOTO

0:00:00.081584538 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received stream-start event: stream-start event: 0x7f3100001c00, time 99:99:99.999999999, seq-num 23, GstEventStreamStart, stream-id=(string)787c660313c22f4e77702a3f9a299cf0, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1;
0:00:00.081736314 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received caps event: caps event: 0x7f310021f830, time 99:99:99.999999999, seq-num 31, GstEventCaps, caps=(GstCaps)“video/x-h264,\ stream-format=(string)byte-stream,\ alignment=(string)au,\ level=(string)1.3,\ profile=(string)high-4:4:4,\ width=(int)320,\ height=(int)240,\ pixel-aspect-ratio=(fraction)1/1,\ framerate=(fraction)30/1,\ interlace-mode=(string)progressive,\ colorimetry=(string)bt601,\ chroma-site=(string)jpeg,\ multiview-mode=(string)mono,\ multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono”;
0:00:00.081812728 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received segment event: segment event: 0x7f310014c9c0, time 99:99:99.999999999, seq-num 32, GstEventSegment, segment=(GstSegment)“segment, flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1, applied-rate=(double)1, format=(GstFormat)time, base=(guint64)0, offset=(guint64)0, start=(guint64)3600000000000000, stop=(guint64)18446744073709551615, time=(guint64)0, position=(guint64)3600000000000000, duration=(guint64)18446744073709551615;”;
0:00:00.081850710 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received tag event: tag event: 0x7f31002210a0, time 99:99:99.999999999, seq-num 33, GstTagList-stream, taglist=(taglist)“taglist,\ encoder=(string)x264,\ encoder-version=(uint)164,\ maximum-bitrate=(uint)2097152,\ nominal-bitrate=(uint)2097152;”;

Perhaps I’m using the gst_buffer_add_video_sei_user_data_unregistered_meta function incorrectly and I need to do something else in my plugin code?