How to decode raw h264 frames using appsrc?

Application Development
1

Hello! I’m receiving raw h264 i- and p-frames from RTSP stream using RtspClientSharp (C# library). I’m trying to push that frames to appsrc and convert them into JPEG, but something goes wrong and appsink doesn’t emit new-sample signal. Here is my code:

using System.Net;
using Gst;
using Gst.App;
using RtspClientSharp;
using RtspClientSharp.RawFrames;
using Format = Gst.Format;
using Task = System.Threading.Tasks.Task;
using Uri = System.Uri;

Application.Init();

var uri = new Uri("rtsp://x.x.x.x/onvif/media?profile=Profile1");
var credentials = new NetworkCredential("xxx", "xxx");
var rtspClient = new RtspClient(new ConnectionParameters(uri, credentials));

using var pipeline = new Pipeline("ppl");

using var source = ElementFactory.Make("appsrc", "source");
using var h264parse = ElementFactory.Make("h264parse", "h264parse");
using var avdec_h264 = ElementFactory.Make("avdec_h264", "avdec_h264");
using var jpegenc = ElementFactory.Make("jpegenc", "jpegenc");
using var sink = new AppSink("sink");

pipeline.Add(source, h264parse, avdec_h264, jpegenc, sink);

source.Link(h264parse);
h264parse.Link(avdec_h264);
avdec_h264.Link(jpegenc);
jpegenc.Link(sink);

rtspClient.FrameReceived += async (sender, frame) =>
{
    if (frame.Type != FrameType.Video) return;
    using var gstBuffer = new Gst.Buffer(frame.FrameSegment.Array!);
    using var caps = Caps.FromString("video/x-h264");
    var segment = new Segment();
    using var structure = new Structure("frame");
    structure.SetValue("width", new GLib.Value(1280));
    structure.SetValue("height", new GLib.Value(720));
    using var gstSample = new Sample(gstBuffer, caps, segment, structure);
    source.Emit("push-sample", gstSample);

    if (pipeline.CurrentState is not State.Playing)
        pipeline.SetState(State.Playing);
    
    await Task.CompletedTask;
};

sink.NewSample += (object sender, NewSampleArgs args) =>
{
    var sink = sender as AppSink;
    using var sample = sink!.PullSample();
    using var buffer = sample.Buffer;
    var size = buffer.Size;

    // Read the JPEG image from the buffer
    var jpegData = new byte[size];
    buffer.Extract(0, ref jpegData);
    
    Console.WriteLine("Received JPEG image data.");
};

await rtspClient.ConnectAsync(CancellationToken.None);
await rtspClient.ReceiveAsync(CancellationToken.None);

while (true)
{
    await Task.Delay(1000);
}

If I set GST_DEBUG to 2 I get following output:

0:00:00.302714100  8860 0000029AFA55E180 WARN               h264parse gsth264parse.c:4072:gst_h264_parse_set_caps:<h264parse> video/x-h264 caps without codec_data or stream-format
0:00:00.304448000  8860 0000029AFA55E180 WARN       codecparsers_h264 gsth264parser.c:2474:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 0
0:00:00.304749700  8860 0000029AFA55E180 WARN               h264parse gsth264parse.c:2028:gst_h264_parse_handle_frame:<h264parse> Failed to update backlog
0:00:00.305084200  8860 0000029AFA55E180 WARN       codecparsers_h264 gsth264parser.c:2474:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 0
0:00:00.305365700  8860 0000029AFA55E180 WARN               h264parse gsth264parse.c:2028:gst_h264_parse_handle_frame:<h264parse> Failed to update backlog
0:00:00.305703600  8860 0000029AFA55E180 WARN       codecparsers_h264 gsth264parser.c:2474:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 0
0:00:00.305973300  8860 0000029AFA55E180 WARN               h264parse gsth264parse.c:2028:gst_h264_parse_handle_frame:<h264parse> Failed to update backlog

What am I doing wrong?

2

The first line in your debug log tells you what is wrong. The caps you pass to appsrc are missing codec_data or stream-format field.

3

so which codec_data or stream-format I should pass to it?

4

That depends on your input H.264 format. However it is likely stream-format=byte-stream format for a RT(S)P source.

5

so now I get this logs:

0:00:00.556115700 20092 0000013E3737E2A0 WARN               structure gststructure.c:2371:priv_gst_structure_parse_fields: Failed to find delimiter, r=stream-format=byte-stream
0:00:00.591275100 20092 0000013E3737E2A0 WARN                  appsrc gstappsrc.c:2619:gst_app_src_push_sample_internal:<source> received sample without caps
0:00:00.600652100 20092 0000013E3737E2A0 WARN               structure gststructure.c:2371:priv_gst_structure_parse_fields: Failed to find delimiter, r=stream-format=byte-stream
0:00:00.601709800 20092 0000013E3737E2A0 WARN                  appsrc gstappsrc.c:2619:gst_app_src_push_sample_internal:<source> received sample without caps
0:00:00.602194300 20092 0000013E3737E2A0 WARN               structure gststructure.c:2371:priv_gst_structure_parse_fields: Failed to find delimiter, r=stream-format=byte-stream
0:00:00.602233800 20092 0000013E37382A80 WARN       codecparsers_h264 gsth264parser.c:2474:gst_h264_parser_parse_slice_hdr: couldn't find associated picture parameter set with id: 0
0:00:00.603096600 20092 0000013E3737E2A0 WARN                  appsrc gstappsrc.c:2619:gst_app_src_push_sample_internal:<source> received sample without caps
6

Your caps string seems to be invalid.

I would suggest reading the GStreamer documentation.

7

I changed caps to video/x-h264,stream-format=byte-stream,alignment=nal

Also removed h264parse from pipeline and changed emit-signals property on appsink to true.

And finally changed FrameReceived event handler to

rtspClient.FrameReceived += async (sender, frame) =>
{
    if (frame.Type != FrameType.Video) return;

    var bufferSize = frame.FrameSegment.Count;
    var iFrame = frame as RawH264IFrame;
    if (iFrame is not null)
    {
        bufferSize += iFrame.SpsPpsSegment.Count;
    }

    var buf = new byte[bufferSize];
    if (iFrame is not null)
    {
        iFrame.SpsPpsSegment.CopyTo(buf);
        iFrame.FrameSegment.CopyTo(buf, iFrame.SpsPpsSegment.Count);
    }
    else
    {
        frame.FrameSegment.CopyTo(buf);
    }
    
    using var gstBuffer = new Gst.Buffer(buf);
    using var caps = Caps.FromString("video/x-h264,stream-format=byte-stream,alignment=nal");
    var segment = new Segment();
    using var structure = new Structure("frame");
    structure.SetValue("width", new GLib.Value(1280));
    structure.SetValue("height", new GLib.Value(720));
    using var gstSample = new Sample(gstBuffer, caps, segment, structure);
    source.Emit("push-sample", gstSample);
};

Now it works and I receive jpeg images. Thanks!