How to Render GStreamer Video Stream to a WPF Element (e.g., Image) Using AppSink?

Newcomers
,
1

I am using C# (.NET 7) and GStreamer to handle an RTSP stream and render the video to a WPF Image element. I aim to use appsink to process the video frames. Below is the pipeline definition I am using: string pipelineDescription = $"rtspsrc location={rtspUrl} ! decodebin ! videoconvert ! video/x-raw,format=RGB ! appsink name=mysink";

The pipeline initializes successfully, and when I list the elements using ListPipelineElements, I can see that mysink exists. However, when I attempt to retrieve the mysink element using GetChildByName(“mysink”) or manually iterate through the pipeline elements with IterateElements, the mysink element is recognized as a generic Element and not an AppSink. This results in the following error during type casting: System.InvalidCastException: Unable to cast object of type 'Gst.Element' to type 'Gst.App.AppSink'

I used the following method to list pipeline elements:

private void ListPipelineElements(Pipeline pipeline)
{
    System.Diagnostics.Debug.WriteLine("Pipeline elements:");
    foreach (var obj in pipeline.IterateElements())
    {
        if (obj is Element element)
        {
            System.Diagnostics.Debug.WriteLine($" - Element Name: {element.Name}, Type: {element.GetType().Name}");
        }
        else
        {
            System.Diagnostics.Debug.WriteLine("Unknown element type found.");
        }
    }
}

Output: Pipeline initialized successfully. Pipeline elements:

  • Element Name: capsfilter0, Type: Element
  • Element Name: mysink, Type: Element
  • Element Name: videoconvert0, Type: Element
  • Element Name: decodebin0, Type: Bin
  • Element Name: rtspsrc0, Type: Bin Checked element: capsfilter0 Checked element: mysink Element ‘mysink’ is not an AppSink, type: Element Checked element: videoconvert0 Checked element: decodebin0 Checked element: rtspsrc0 Could not find ‘mysink’ as an AppSink in the pipeline.

Attempts:

  1. Retrieving Element from Pipeline:
  • Used GetChildByName(“mysink”), but it was detected as a generic Element.
  1. Listing Pipeline Elements:
  • Used IterateElements() to manually list pipeline elements and confirmed that mysink was present but detected as Element.
  1. Type Casting:
  • Attempted to cast the element manually: var appSink = (AppSink)sinkElement; Resulted in: System.InvalidCastException
  1. Manual Pipeline Creation:
  • Manually created and added AppSink to the pipeline:
var appSink = new AppSink("mysink");
_pipeline.Add(appSink);

While this allowed defining the element, dynamic linking within the pipeline failed.

  1. Basic Pipeline Testing:
  • Tested a simpler pipeline with videotestsrc: string pipelineDescription = "videotestsrc ! videoconvert ! video/x-raw,format=RGB ! appsink name=mysink"; Faced the same type casting issue with AppSink.
  1. Rendering Video to WPF Image:
  • Attempted to update the Image control with GStreamer frame data:
private void UpdateBGImage(int width, int height, IntPtr data)
{
    int size = width * height * 4;
    int stride = ((width * 32 + 31) & ~31) / 8;

    this.Dispatcher.Invoke(() =>
    {
        BitmapSource source = BitmapSource.Create(width, height, 96, 96, PixelFormats.Bgra32, null, data, size, stride);
        BGImage.Source = source;
        this.InvalidateVisual();
    });
}

However, the Image control did not update.

What I Expected:

  • Successfully retrieve the mysink element as an AppSink.
  • Process frames using AppSink and update a WPF Image control.

What Happened:

  • mysink was detected as a generic Element.
  • Type casting to AppSink failed.

Additional Information and Attempts:

  1. Alternative Binding Libraries:
  • Tested gstream-sharp and gstream-sharp-core without success.
  • Attempted to reference gstapp-1.0-0.dll, but received “invalid reference” errors.
  1. Working Example with Windows Forms: The following pipeline worked in a Windows Forms project: rtspsrc location=rtsp://127.0.0.1:8554/mystream latency=300 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink However, I need this to work in WPF with .NET 7.

My Questions:

  1. How can I retrieve the AppSink element from a GStreamer pipeline in C#?
  2. Is there a compatibility issue with the libraries or binding? If so, what is the recommended GStreamer binding for .NET 7?
  3. How can I efficiently render GStreamer video streams to a WPF Image element?
2

You have to create the appsink using the binding constructor, otherwise the binding is unable to make the association. It’s a limitation of the bindings. Some pseudo code, I haven’t written CSharp for a long time.

string pipelineDescription = $"rtspsrc location={rtspUrl} ! decodebin ! videoconvert ! caps setter caps=video/x-raw,format=RGB name=end";
var pipeline = ...
var appSink = new AppSink("mysink");

elem =pipeline.GetByName("end");
pipeline.Add(appSink);

! Element.Link(elem, appSink);