Inserting SEI metadata in h264

General Discussion
1

Hello!

I ran into an issue when trying to use the gst_buffer_add_video_sei_user_data_unregistered_meta function from https://gstreamer.freedesktop.org/documentation//video/gstvideosei.html?gi-language=c#GstVideoSEIUserDataUnregisteredMeta.

I’m using gstreamer 1.22.8 with docker container livekit/gstreamer:1.22.8-dev:

gst-inspect-1.0 --version

gst-inspect-1.0 version 1.22.8
GStreamer 1.22.8
Unknown package origin

As a basis, I took this repository gstreamer-boilerplate-cpp which describes the steps for creating your own plugin in C++.

I changed the gst_my_filter_chain function gstreamer-boilerplate-cpp/gst-plugin/src/gstmyfilter.cpp at master · ozankaraali/gstreamer-boilerplate-cpp · GitHub
in the following way:

static GstFlowReturn
gst_my_filter_chain (GstPad * pad, GstObject * parent, GstBuffer * buf)
{
  GstMyFilter *filter;

  filter = GST_MYFILTER (parent);

  /*if (filter->silent == FALSE){
    g_print ("Loaded!");
    // Now we can use iostream C++:
    std::cout<< "Test" <<std::endl;
  }*/

  // Create a UUID for the SEI data
  guint8 uuid[16] = {0x4d, 0x4f, 0x4d, 0x4f, 0x54, 0x4f, 0x00, 0x00,
                   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};

  // Create some SEI data (for example, a string)
  const gchar *sei_data = "MOMOTO";
  gsize sei_size = strlen (sei_data);

  // Add the SEI data to the buffer as metadata
  GstVideoSEIUserDataUnregisteredMeta* ret = gst_buffer_add_video_sei_user_data_unregistered_meta(buf, uuid, (guint8 *) sei_data, sei_size);

  // std::cout<< "RET VALUE" << ret <<std::endl;

  // GST_DEBUG_OBJECT (filter, "SEI data added to buffer");
  // GST_DEBUG_OBJECT (filter, "SEI data added to buffer: %.*s", sei_size, (gchar*)sei_data);


  /* just push out the incoming buffer without touching it */
  return gst_pad_push (filter->srcpad, buf);
}

also I included gstreamer-video-1.0 in configure.ac:

PKG_CHECK_MODULES(GST, [
  gstreamer-1.0 >= $GST_REQUIRED
  gstreamer-base-1.0 >= $GST_REQUIRED
  gstreamer-controller-1.0 >= $GST_REQUIRED
  gstreamer-audio-1.0 >= $GST_REQUIRED
  gstreamer-video-1.0 >= $GST_REQUIRED
]

after building the source code:

cd gst-plugin
./autogen.sh

Libraries have been installed in:
/usr/local/lib/gstreamer-1.0

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the ‘-LLIBDIR’
flag during linking and do at least one of the following:

  • add LIBDIR to the ‘LD_LIBRARY_PATH’ environment variable
    during execution
  • add LIBDIR to the ‘LD_RUN_PATH’ environment variable
    during linking
  • use the ‘-Wl,-rpath -Wl,LIBDIR’ linker flag
  • have your system administrator add LIBDIR to ‘/etc/ld.so.conf’

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

make[2]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin/src’
make[1]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin/src’
make[1]: Entering directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’
make[2]: Entering directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’
make[2]: Nothing to be done for ‘install-exec-am’.
make[2]: Nothing to be done for ‘install-data-am’.
make[2]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’
make[1]: Leaving directory ‘/mnt/arhiv/data/gstreamer-boilerplate-cpp/gst-plugin’

I see a message about the successful build of the plugin at the following path:
/usr/local/lib/gstreamer-1.0/libgstmyfilter.so

Plugin verification is successful:
gst-inspect-1.0 /usr/local/lib/gstreamer-1.0/libgstmyfilter.so

Plugin Details:
Name myfilter
Description Template myfilter
Filename /usr/local/lib/gstreamer-1.0/libgstmyfilter.so
Version 1.0.0
License LGPL
Source module my-filter
Binary package GStreamer

myfilter: MyFilter

1 features:
±- 1 elements

I’m running a pipeline that uses my built plugin. The debug messages show that a GstVideoSEIUserDataUnregisteredMeta object of size 48 is being created:

GST_DEBUG=6 gst-launch-1.0 videotestsrc num-buffers=30 ! x264enc tune=zerolatency ! myfilter ! h264parse ! matroskamux ! filesink location=1.mkv

0:00:00.678695478 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:880:gst_buffer_new: new 0x7f35001187f0
0:00:00.678701669 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:570:gst_buffer_copy_into: copy 0x7f3500144140 to 0x7f35001187f0, offset 0-6204/6204
0:00:00.678707139 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:463:_memory_add: buffer 0x7f35001187f0, idx -1, mem 0x7f3500132440
0:00:00.678712580 136769 0x7f3508000b70 DEBUG GST_PERFORMANCE gstminiobject.c:540:ensure_priv_data: allocating private data GstMemory miniobject 0x7f3500132440
0:00:00.678717328 136769 0x7f3508000b70 DEBUG video-sei video-sei.c:110:gst_video_sei_user_data_unregistered_meta_transform: copy SEI User Data Unregistered metadata
0:00:00.678722508 136769 0x7f3508000b70 DEBUG GST_BUFFER gstbuffer.c:2337:gst_buffer_add_meta: alloc metadata 0x7f350011ef30 (GstVideoSEIUserDataUnregisteredMeta) of size 48
0:00:00.678728760 136769 0x7f3508000b70 DEBUG GST_PERFORMANCE gstminiobject.c:440:gst_mini_object_make_writable: copy GstBuffer miniobject 0x7f3500144140 → 0x7f35001187f0
0:00:00.678737917 136769 0x7f3508000b70 LOG baseparse gstbaseparse.c:2248:gst_base_parse_handle_buffer: handling buffer of size 6204 with dts 1000:00:00.866666666, pts 1000:00:00.866666666, duration 99:99:99.999999999
0:00:00.678745822 136769 0x7f3508000b70 LOG baseparse gstbaseparse.c:2211:gst_base_parse_prepare_frame: preparing frame at offset 18446744073709551615 (0xffffffffffffffff) of size 6204
0:00:00.678751482 136769 0x7f3508000b70 LOG baseparse gstbaseparse.c:804:gst_base_parse_update_frame: marking as new frame
0:00:00.678757954 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:1863:gst_buffer_map_range: buffer 0x7f35001187f0, idx 0, length -1, flags 0001
0:00:00.678764436 136769 0x7f3508000b70 LOG GST_BUFFER gstbuffer.c:306:_get_merged_memory: buffer 0x7f35001187f0, idx 0, length 1
0:00:00.678770007 136769 0x7f3508000b70 LOG h264parse gsth264parse.c:1252:gst_h264_parse_handle_frame_packetized: processing packet buffer of size 6204
0:00:00.678775597 136769 0x7f3508000b70 DEBUG codecparsers_h264 gsth264parser.c:250:gst_h264_parse_nalu_header: Nal type 9, ref_idc 0
0:00:00.678780546 136769 0x7f3508000b70 DEBUG h264parse gsth264parse.c:1264:gst_h264_parse_handle_frame_packetized: AVC nal offset 6
0:00:00.678788110 136769 0x7f3508000b70 DEBUG h264parse gsth264parse.c:971:gst_h264_parse_process_nal: processing nal of type 9 AU delimiter, size 2
0:00:00.678794242 136769 0x7f3508000b70 LOG h264parse gsth264parse.c:1174:gst_h264_parse_process_nal: collecting NAL in AVC frame
0:00:00.678800383 136769 0x7f3508000b70 DEBUG h264parse gsth264parse.c:488:gst_h264_parse_wrap_nal: nal length 2
0:00:00.678806615 136769 0x7f3508000b70 DEBUG GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x7f3500126140, maxsize:13 offset:0 size:6

after that I try to find my metadata in the video either through the console output:

gst-launch-1.0 filesrc location=1.mkv ! matroskademux ! myfilter ! filesink location=/dev/stdout | hexdump -C | grep MOMO

0:00:00.029950355 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received stream-start event: stream-start event: 0x7f3728013900, time 99:99:99.999999999, seq-num 21, GstEventStreamStart, stream-id=(string)d4370fbf6e250a336cc468433b1516904107e477db6e0eb715f3f082dc02c2b3/001:6486666301394315289, flags=(GstStreamFlags)GST_STREAM_FLAG_SELECT, group-id=(uint)1;
0:00:00.030068758 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received caps event: caps event: 0x7f37280123b0, time 99:99:99.999999999, seq-num 22, GstEventCaps, caps=(GstCaps)“video/x-h264,\ level=(string)1.3,\ profile=(string)high-4:4:4,\ codec_data=(buffer)01f4000dffe1001b67f4000d919640507ec05a83030320000003002000000791e2854901000668ebcc448440,\ stream-format=(string)avc,\ alignment=(string)au,\ width=(int)320,\ height=(int)240,\ framerate=(fraction)30/1,\ interlace-mode=(string)progressive,\ colorimetry=(string)bt601”;
0:00:00.030135974 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received tag event: tag event: 0x7f3728013d00, time 99:99:99.999999999, seq-num 23, GstTagList-global, taglist=(taglist)“taglist,\ datetime=(datetime)2024-01-18T06:24:30.298886Z,\ container-format=(string)Matroska;”;
0:00:00.030179226 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received tag event: tag event: 0x7f3728014420, time 99:99:99.999999999, seq-num 28, GstTagList-stream, taglist=(taglist)“taglist,\ video-codec=(string)H264,\ encoder=(string)x264,\ bitrate=(uint)1431656;”;
0:00:00.030366628 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received segment event: segment event: 0x7f3728014030, time 99:99:99.999999999, seq-num 33, GstEventSegment, segment=(GstSegment)“segment, flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1, applied-rate=(double)1, format=(GstFormat)time, base=(guint64)0, offset=(guint64)0, start=(guint64)0, stop=(guint64)18446744073709551615, time=(guint64)0, position=(guint64)0, duration=(guint64)999999999;”;
0:00:00.060276106 136798 0x7f3730000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received eos event: eos event: 0x7f372801a2c0, time 99:99:99.999999999, seq-num 48, (NULL)

or via ffmpeg pyav:

import av
from uuid import UUID

container = av.open("./1.mkv")
stream = container.streams.video[0]

for packet in container.demux(stream):
    for frame in packet.decode():
        for sd in list(frame.side_data.keys()):
            bts = bytes(sd)
            uuid_bts = bts[:16]
            uuid_str = str(UUID(bytes=uuid_bts))

            message = bts[16:].decode('utf-8')
            print(f"message: {message}")
            print(f"uuid: {uuid_str}")

python avpipeline.py

message: x264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=3 lookahead_threads=3 sliced_threads=1 slices=3 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=300 keyint_min=30 scenecut=40 intra_refresh=0 rc_lookahead=0 rc=cbr mbtree=0 bitrate=2048 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=2048 vbv_bufsize=1228 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:1.00
uuid: dc45e9bd-e6d9-48b7-962c-d820d923eeef

I don’t see any metadata inserted. Even if you use x264enc:

gst-launch-1.0 videotestsrc ! x264enc ! myfilter ! fakesink dump=true | xxd -c 48 | grep MOMOTO

0:00:00.081584538 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received stream-start event: stream-start event: 0x7f3100001c00, time 99:99:99.999999999, seq-num 23, GstEventStreamStart, stream-id=(string)787c660313c22f4e77702a3f9a299cf0, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1;
0:00:00.081736314 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received caps event: caps event: 0x7f310021f830, time 99:99:99.999999999, seq-num 31, GstEventCaps, caps=(GstCaps)“video/x-h264,\ stream-format=(string)byte-stream,\ alignment=(string)au,\ level=(string)1.3,\ profile=(string)high-4:4:4,\ width=(int)320,\ height=(int)240,\ pixel-aspect-ratio=(fraction)1/1,\ framerate=(fraction)30/1,\ interlace-mode=(string)progressive,\ colorimetry=(string)bt601,\ chroma-site=(string)jpeg,\ multiview-mode=(string)mono,\ multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono”;
0:00:00.081812728 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received segment event: segment event: 0x7f310014c9c0, time 99:99:99.999999999, seq-num 32, GstEventSegment, segment=(GstSegment)“segment, flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1, applied-rate=(double)1, format=(GstFormat)time, base=(guint64)0, offset=(guint64)0, start=(guint64)3600000000000000, stop=(guint64)18446744073709551615, time=(guint64)0, position=(guint64)3600000000000000, duration=(guint64)18446744073709551615;”;
0:00:00.081850710 136818 0x7f3108000b70 LOG myfilter gstmyfilter.cpp:209:gst_my_filter_sink_event: Received tag event: tag event: 0x7f31002210a0, time 99:99:99.999999999, seq-num 33, GstTagList-stream, taglist=(taglist)“taglist,\ encoder=(string)x264,\ encoder-version=(uint)164,\ maximum-bitrate=(uint)2097152,\ nominal-bitrate=(uint)2097152;”;

Perhaps I’m using the gst_buffer_add_video_sei_user_data_unregistered_meta function incorrectly and I need to do something else in my plugin code?

2

Did you ever figure this out? I’m trying to do a similar thing and I’ve run into the same sort of issues

3

Probably, necroposting, related.

My goal was to insert current timestamp to SEI NAL units.
In order to achieve that, I came upon in this forum to Sebastian Droege’s answer, where he mentioned “appending probe to h264parse source pad”.

I was unable to make things work with codecparsers header file from Gstreamer, and came up to own ugly implementation.
Here is the code:

 static GstPadProbeReturn sei_injection_probe(GstPad *pad, GstPadProbeInfo *info, gpointer user_data) {
        RTSPgwhisper *self = static_cast<RTSPgwhisper*>(user_data);
        static uint64_t frame_counter = 0;
        if (++frame_counter % 60 != 0)
        {
            return GST_PAD_PROBE_OK;
        }

        g_print("SEI injection probe start\n");
        GstBuffer* buffer;
        GstBuffer* new_buffer;
        GstMapInfo map_info;
        bool found = false;
        
        buffer = GST_PAD_PROBE_INFO_BUFFER(info);
        std::string timestamp = self->generate_timestamp();
        
        // Map buffer for reading
        if (!gst_buffer_map(buffer, &map_info, GST_MAP_READ)) {
            g_print("Can't get buffer\n");
            return GST_PAD_PROBE_OK;
        }
        
        // Find first slice NAL unit and insert SEI before it
        guint8* data = map_info.data;
        gsize size = map_info.size;
        gsize sei_insert_pos = 0;
#ifdef DEBUG        
        g_print("SEI probe: buffer size = %zu\n", size);
        if (size > 0) {
            g_print("First 32 bytes: ");
            for (int i = 0; i < MIN(32, size); i++) {
                g_print("%02x ", data[i]);
            }
            g_print("\n");
        }
        g_print("Found NAL types in buffer: ");
#endif
        for (gsize i = 0; i < size - 4;) {
           if (i + 4 <= size &&
               data[i] == 0 && data[i+1] == 0 && data[i+2] == 0 && data[i+3] == 1) {
               if (i + 5 <= size) {
                   guint8 nal_type = data[i+4] & 0x1F;
                   g_print("%d ", nal_type);
                   if (nal_type >= 1 && nal_type <= 5) {
#ifdef DEBUG
                       g_print("SET insert pos to %zu in case of 0x00 0x00 0x00 0x01 \n", i);
#endif
                       sei_insert_pos = i;
                       found = true;
                       break;
                   }
               }
               i += 4;
           }
           else if (i + 3 <= size &&
                    data[i] == 0 && data[i+1] == 0 && data[i+2] == 1) {
               if (i + 4 <= size) {
                   guint8 nal_type = data[i+3] & 0x1F;
                   g_print("%d ", nal_type);
                   if (nal_type >= 1 && nal_type <= 5) {
#ifdef DEBUG
                    g_print("SET insert pos to %zu in case of  in case of 0x00 0x00 0x01 \n", i);
#endif
                         sei_insert_pos = i;
                         found = true;
                         break;
                   }
               }
               i += 3;
           } else {
               i++;
           }
       }
#ifdef DEBUG       
       g_print("\n");
       g_print("After loop: sei_insert_pos = %zu\n", sei_insert_pos);
#endif
       gst_buffer_unmap(buffer, &map_info);
        
        if (!found) {
            // No suitable position found, return original buffer
            g_print("No suitable position found\n");
            return GST_PAD_PROBE_OK;
        }
        
        // Create SEI NAL unit with absolutely random UUUD
        const guint8 uuid_data[16] = {
            0x8c, 0x5c, 0x6e, 0x6a, 0x13, 0x6a, 0x4a, 0x4e,
            0x8f, 0x7d, 0x9b, 0x4c, 0x8f, 0x6d, 0x5e, 0x5f
        };
        
        // Calculate SEI payload size
        size_t timestamp_len = timestamp.length();
        size_t sei_payload_size = 16 + timestamp_len; // UUID + data
        
        // Calculate total SEI NAL unit size
        size_t sei_size_bytes = 1; // For sizes < 255
        if (sei_payload_size >= 255) {
            sei_size_bytes = (sei_payload_size / 255) + 1;
        }
        
        size_t total_sei_size = 4 + 1 + 1 + sei_size_bytes + sei_payload_size + 1;
        
        // Create new buffer with space for SEI
        new_buffer = gst_buffer_new_and_alloc(gst_buffer_get_size(buffer) + total_sei_size);
        
        if (!new_buffer) {
            g_print("Can't create buffer and allocate");
            return GST_PAD_PROBE_OK;
        }
        
        // Map both buffers
        GstMapInfo old_map, new_map;
        if (!gst_buffer_map(buffer, &old_map, GST_MAP_READ) ||
            !gst_buffer_map(new_buffer, &new_map, GST_MAP_WRITE)) {
            if (old_map.data) gst_buffer_unmap(buffer, &old_map);
            gst_buffer_unref(new_buffer);
            g_print("Can't map buffer\n");
            return GST_PAD_PROBE_OK;
        }
        
        // Copy data before SEI insertion point
        memcpy(new_map.data, old_map.data, sei_insert_pos);
        guint8* write_ptr = new_map.data + sei_insert_pos;
        
        // Start code
        write_ptr[0] = 0x00; write_ptr[1] = 0x00; write_ptr[2] = 0x00; write_ptr[3] = 0x01;
        write_ptr += 4;
        
        // NAL unit type: 6 (SEI)
        *write_ptr++ = 0x06;
        
        // Payload type: 5 (user_data_unregistered)
        *write_ptr++ = 0x05;
        
        // Payload size
        size_t payload_size = 16 + timestamp.length();
        size_t temp = payload_size;
        do {
            guint8 val = (temp > 255) ? 255 : temp;
            *write_ptr++ = val;
            temp -= val;
        } while (temp > 0);
        
        // UUID
        memcpy(write_ptr, uuid_data, 16);
        write_ptr += 16;
        
        // Timestamp
        memcpy(write_ptr, timestamp.c_str(), timestamp.length());
        write_ptr += timestamp.length();
        
        // RBSP trailing
        *write_ptr++ = 0x80;
        
        // Copy remaining data
        memcpy(write_ptr, old_map.data + sei_insert_pos, old_map.size - sei_insert_pos);
        write_ptr += old_map.size - sei_insert_pos;
        
#ifdef DEBUG
        // Show us what we've done
        g_print("First 64 bytes after SEI injection:\n");
        for (int j = 0; j < MIN(64, write_ptr - new_map.data); j++) {
            if (j % 8 == 0) g_print(" ");
            if (j % 16 == 0) g_print("\n");
            g_print("%02x ", new_map.data[j]);
        }
        g_print("\n");
#endif DEBUG
        // Unmap buffers
        gst_buffer_unmap(buffer, &old_map);
        gst_buffer_unmap(new_buffer, &new_map);
        
        // Copy buffer metadata
        gst_buffer_copy_into(new_buffer, buffer, GST_BUFFER_COPY_METADATA, 0, -1);
        
        // Replace buffer in probe info
        info->data = new_buffer;
        gst_buffer_unref(buffer);
                
        return GST_PAD_PROBE_OK;
    }

Then you need to make “pad probe” on the h264enc’s source pad.

 GstPad *encoder_src = gst_element_get_static_pad(current_h264enc, "src");
        if (encoder_src) {
            gst_pad_add_probe(encoder_src, GST_PAD_PROBE_TYPE_BUFFER, sei_injection_probe, this, NULL);
            gst_object_unref(encoder_src);
            g_print("SEI injection probe added to openh264enc src pad\n");
        }

Verifying:

image
image1121×395 51.2 KB