Webrtcbin leaking sender/receiver/transceiver?

uler3161 · November 13, 2023, 6:51pm

I’m using gstreamer 1.22.6 with gstreamer-sharp. I have a pipeline to broadcast streams over webrtc. I’ve noticed ever increasing unmanaged memory as clients connect/disconnect.

I used GST_DEBUG value of GST_REFCOUNTING:5. With this, what I’ve noticed is that my webrtcbin objects do seem to get disposed and finalized. However, they seem to leave around sender, receiver, and transceiver objects that don’t get disposed. I’m guessing they are contributing to at least some of the leaking memory.

I can’t see anything in my code that I think would be maintaining references to these objects. I would have thought that successful dispose/finalize of webrtcbin would clean them up.

Here is my code:

using System.Text;
using BusinessObjects;
using Common;
using GLib;
using Gst;
using Gst.Sdp;
using Gst.WebRTC;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Services.SignalR;
using Task = System.Threading.Tasks.Task;

namespace Services.CameraConnections.Broadcast;

public class CameraWebRtcConnection : IAsyncDisposable
{
    private static int ReferenceIndex;
    private readonly AtomicBoolean _isDisposing = new();

    public CameraWebRtcConnection(
        ILogger<CameraWebRtcConnection> logger,
        IDbContextFactory<CameraContext> cameraContextFactory,
        IVideoHubService videoHubService,
        CameraConnectionID id,
        Pipeline pipeline)
    {
        ID = id;
        Pipeline = pipeline;
        Logger = logger;
        CameraContextFactory = cameraContextFactory;
        VideoHubService = videoHubService;
        Reference = Interlocked.Increment(ref ReferenceIndex);

        WebRtcBinName = $"webrtcbin_{Reference}";
        QueueName = $"queue_{Reference}";
        RtpH264PayName = $"rtph264pay_{Reference}";

        WebRtcPipeline = $@"queue name={QueueName} max-size-bytes=20971520 max-size-time=0 leaky=downstream !
            rtph264pay name={RtpH264PayName} aggregate-mode=zero-latency config-interval=-1 timestamp-offset=0 !
            webrtcbin name={WebRtcBinName} bundle-policy=max-bundle";
    }

    private Bin? Bin
    {
        get;
        set;
    }

    private IDbContextFactory<CameraContext> CameraContextFactory
    {
        get;
    }

    private CameraConnectionID ID
    {
        get;
    }

    private ILogger<CameraWebRtcConnection> Logger
    {
        get;
    }

    private ManualResetEvent OfferSent
    {
        get;
    } = new(false);

    private Pipeline Pipeline
    {
        get;
    }

    private string QueueName
    {
        get;
    }

    private int Reference
    {
        get;
    }

    private string RtpH264PayName
    {
        get;
    }

    private IVideoHubService VideoHubService
    {
        get;
    }

    private string WebRtcBinName
    {
        get;
    }

    private string WebRtcPipeline
    {
        get;
    }

    public async ValueTask DisposeAsync()
    {
        if (Bin == null)
        {
            return;
        }

        if (_isDisposing.TrySetValue(true))
        {
            await Task.Run(RemoveFromPipeline);
        }
    }

    public void AddIceCandidate(
        string iceCandidate)
    {
        Logger.LogInformation("Adding ice candidate from client: {iceCandidate}", iceCandidate);

        try
        {
            var msg = JsonConvert.DeserializeObject<dynamic>(iceCandidate)!;
            string candidate = msg.candidate;
            uint sdpMLineIndex = msg.sdpMLineIndex;

            Pipeline.GetByName(WebRtcBinName)
                ?.Emit("add-ice-candidate", sdpMLineIndex, candidate);
        }
        catch (Exception ex)
        {
            Logger.LogError(ex, ex.Message);
        }
    }

    public void Answer(
        string answer)
    {
        Logger.LogInformation("Answer from client: {answer}", answer);

        var sdp = JObject.Parse(answer)["sdp"]!.Value<string>()!;

        try
        {
            SDPMessage.New(out var sdpMsg);
            SDPMessage.ParseBuffer(Encoding.Default.GetBytes(sdp), (uint)sdp.Length, sdpMsg);
            var streamDescription = WebRTCSessionDescription.New(WebRTCSDPType.Answer, sdpMsg);
            var promise = new Promise();

            Pipeline.GetByName(WebRtcBinName)
                ?.Emit("set-remote-description", streamDescription, promise);
        }
        catch (Exception ex)
        {
            Logger.LogError(ex, ex.Message);
        }
    }

    public async Task StartAsync()
    {
        //Issues with getting subsequent connections to work. Race condition in gstreamer?
        //https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/1055
        //https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/1448
        //Note: apparently need the probe in addition to a queue ! fakesink on the initial pipeline.

        Logger.LogInformation("Client connected to broadcast: {id}, {pipelineText}", ID, WebRtcPipeline);

        Bin = Parse.BinFromDescription(WebRtcPipeline, true);
        await InitializeWebRtcBinAsync();
        Bin.SetState(State.Playing);

        var tee = Pipeline.GetByName("t");
        var teeSource = tee.GetRequestPad("src_%u");
        var binPad = Bin.GetStaticPad("sink");

        teeSource.AddProbe(PadProbeType.Blocking, (
            _,
            _) => Bin.CurrentState != State.Playing ? PadProbeReturn.Drop : PadProbeReturn.Remove);

        Pipeline.Add(Bin);

        teeSource.Link(binPad);
    }

    private async Task<TurnServerConfiguration[]> GetTurnUrlsAsync()
    {
        await using var cameraContext = await CameraContextFactory.CreateDbContextAsync();

        return await cameraContext.TurnServerConfigurations.AsNoTracking()
            .ToArrayAsync();
    }

    private async Task InitializeWebRtcBinAsync()
    {
        var webRtcBin = Bin?.GetByName(WebRtcBinName);

        if (webRtcBin == null)
        {
            Logger.LogError("Webrtcbin element not found!");
            return;
        }

        webRtcBin.Connect("on-negotiation-needed", OnNegotiationNeeded);
        webRtcBin.Connect("on-ice-candidate", OnIceCandidate);

        var turnServers = await GetTurnUrlsAsync();
        foreach (var server in turnServers)
        {
            var result = (bool)webRtcBin.Emit("add-turn-server", server.URL);
            Logger.LogInformation("Added turn server {url}. Successful? {result}", server.URL, result);
        }
    }

    private void OnIceCandidate(
        object o,
        SignalArgs args)
    {
        var sdpMLineIndex = (uint)args.Args[0];
        var candidate = (string)args.Args[1];
        var obj = new
        {
            ice = new
            {
                sdpMLineIndex,
                candidate
            }
        };
        var iceCandidate = JsonConvert.SerializeObject(obj);

        Task.Run(async () =>
        {
            OfferSent.WaitOne();

            if (_isDisposing.Value)
            {
                Logger.LogWarning("Disposing web rtc connection, so not sending ice candidate");
                return;
            }

            Logger.LogInformation("Sending ice candidate to {id}: {iceCandidate}", ID, iceCandidate);
            await VideoHubService.SendIceCandidateAsync(ID, iceCandidate);
        });
    }

    private void OnNegotiationNeeded(
        object o,
        SignalArgs args)
    {
        Logger.LogInformation("Negotiation needed, creating offer");

        var promise = new Promise(OnOfferCreated);
        var structure = new Structure($"struct{Reference}");

        Pipeline.GetByName(WebRtcBinName)
            ?.Emit("create-offer", structure, promise);
    }

    private void OnOfferCreated(
        Promise promise)
    {
        Logger.LogInformation("Offer created.");

        promise.Wait();
        var reply = promise.RetrieveReply();
        var offerValue = reply.GetValue("offer");
        var offer = (WebRTCSessionDescription)offerValue.Val;
        promise = new Promise();

        Pipeline.GetByName(WebRtcBinName)
            ?.Emit("set-local-description", offer, promise);
        promise.Interrupt();
        SendSdpOffer(offer);
    }

    private void RemoveFromPipeline()
    {
        try
        {
            OfferSent.Set();

            var tee = Pipeline.GetByName("t");
            var teeSource = Bin!.GetStaticPad("sink")
                .Peer;
            tee.Unlink(Bin);
            tee.ReleaseRequestPad(teeSource);

            var queue = Bin!.GetByName(QueueName);
            var rtpH264Pay = Bin!.GetByName(RtpH264PayName);
            var webRtcBin = Bin!.GetByName(WebRtcBinName);

            Pipeline.Remove(webRtcBin);
            webRtcBin.Disconnect("on-negotiation-needed", OnNegotiationNeeded);
            webRtcBin.Disconnect("on-ice-candidate", OnIceCandidate);

            webRtcBin.SetState(State.Null);
            webRtcBin.Dispose();

            Pipeline.Remove(rtpH264Pay);
            rtpH264Pay.SetState(State.Null);
            rtpH264Pay.Dispose();

            Pipeline.Remove(queue);
            queue.SetState(State.Null);
            queue.Dispose();

            Pipeline.Remove(Bin);
            Bin.SetState(State.Null);
            Bin.Dispose();

            Logger.LogInformation("Disposed connection from {id}", ID.ToString());
        }
        catch (Exception ex)
        {
            Logger.LogError(ex, ex.Message);
        }
    }

    private void SendSdpOffer(
        WebRTCSessionDescription description)
    {
        var text = description.Sdp.AsText();
        var obj = new
        {
            sdp = new
            {
                type = "offer",
                sdp = text
            }
        };
        var offer = JsonConvert.SerializeObject(obj);

        Task.Run(async () =>
        {
            try
            {
                Logger.LogInformation("Sending offer to {id}: {offer}", ID, offer);

                var turnServers = await GetTurnUrlsAsync();
                await VideoHubService.SendOfferAsync(ID, offer, turnServers);

                OfferSent.Set();
                Logger.LogInformation("Offer sent");
            }
            catch (Exception ex)
            {
                Logger.LogError(ex, ex.Message);
            }
        });
    }
}

uler3161 · November 14, 2023, 10:00pm

I’ve been playing around with the leak tracer. Using GST_DEBUG=GST_TRACER:7 and GST_TRACERS=leaks(check-refs=true)

First, there seems to be some weirdness with getting this tracer to work:

Neither of these seemed to work:

Global.Deinit
log-live-objects

I don’t know if they don’t work because I’m on Windows or because I’m using gstreamer-sharp or some other reason.

That said, when the app starts, I can do this:

Loop through all active tracers and emit activity-start-tracking.
When the app shuts down, loop through all active tracers and emit activity-log-checkpoint followed by activity-stop-tracking

I don’t know if that’s the most accurate because I thought Global.Deinit might do extra cleanup which would show up in the log.

Ignoring the issues I had getting the tracer working, the output seems kind of contradictory to the GST_REFCOUNTING output.

I assume with GST_REFCOUNTING:

When I see something like gst_object_set_parent, that’s implying an object creation.
And I’d expect later on to see a gst_element_dispose message when that object is disposed.

If that’s true, it’s clear from GST_REFCOUNTING that I continually get undisposed webrtctransceiver (among many other types). However, if I apply the same kind of logic to leak tracing object-added and object-removed I don’t seem to have leaks. My log is currently showing only one WebRTCTransceiver. I actually don’t think it should be showing that, but at least it’s not an ever growing list of them.

joaoantoniocardoso · January 30, 2024, 6:24am

Hi @uler3161 , did you made any progress? I am facing the same issue when using the webrtcbin, but in Rust.

uler3161 · January 30, 2024, 4:10pm

I have not. I simplified my code to where it just creates the main pipeline (the one passed to CameraWebRtcConnection in my code above). I found no leaks at that point.

But, I added code to properly handle RTSP shutdown and then I started to see leaks again. Here’s the code:

        private void AddTeardownFix()
        {
            Logger.LogInformation("Adding teardown fix. Getting iterator by factory name: rtspsrc");
            using var iterator = Pipeline.IterateAllByElementFactoryName("rtspsrc");

            foreach (Element e in iterator)
            {
                Logger.LogInformation("Connecting before-send to rtspsrc");
                e.Connect("before-send", OnRtspBeforeSend);
            }

            Logger.LogInformation("Adding teardown fix. Disposing iterator by factory name: rtspsrc");
        }

        private void RemoveTeardownFix()
        {
            Logger.LogInformation("Removing teardown fix. Getting iterator by factory name: rtspsrc");
            using var iterator = Pipeline.IterateAllByElementFactoryName("rtspsrc");

            foreach (Element e in iterator)
            {
                Logger.LogInformation("Disconnecting before-send to rtspsrc");
                e.Disconnect("before-send", OnRtspBeforeSend);
            }
            Logger.LogInformation("Removing teardown fix. Disposing iterator by factory name: rtspsrc");
        }

The leak detector in gstreamer says I have a few objects leaking, but I think it’s just the rtspsrc. It is reporting that the rtspsrc has 2 extra ref counts. This is after calling the gstreamer-sharp equivalent of deinit() and C# GC collection/finalization where I’d expect everything to be cleaned up.

I think possibly that the gstreamer leak tracker isn’t reporting everything. It only works when gst_object_ref and gst_object_unref are called. To properly trace ref/unref, glib needs patched and code needs added to g_object_ref and g_object_unref. I’ve done that with a gstreamer 1.22 build as well as dumping a stack every time those are called. I’m also seeing two extra ref calls with this. Ideally I’d be able to easily match up each unref stack trace to the corresponding ref, but that’s not as easy as I’d hoped. If I could, then I could figure out where the two extra refs came from and look for leaks.

For reference, after I figured out that the RTSP shutdown code above was leaking, I created an issue on the mono repo, but I really didn’t get much help. But, you can see a workaround I found (calling Unref() which you shouldn’t have to do):

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3162

Since I wasn’t getting help there, I went to the IRC channel. Found that they have some new place for chat at:https://app.element.io/#/room/#gstreamer:gstreamer.org

I only asked about this yesterday, so haven’t had too much of an opportunity for the main developers of gstreamer to take a look. And maybe they passed right by it, I don’t know.

I don’t know if it’d help, but I’d be willing to share what I put together to add glib debug output for g_object_ref/g_object_unref. It’s pretty messy, but it works (on Windows anyway).

tpm · January 30, 2024, 4:24pm

It might be worth making a C-only test application to make sure it’s not a bindings issue.

That will also make it easier for others to take a quick look.

uler3161 · February 5, 2024, 11:28pm

I’m a bit slow at getting test apps put together. It’s been ages since I’ve written C. I do, however, have my first test done. I decided since I’m both rusty at writing C as well as still not quite up to speed on some GStreamer basics that the thing to do would be to test the built in examples.

So, first thing, I’m building 1.22 source and enabling the examples. Specifically, I’m testing the multiparty-sendrecv. I’ve added a few logging statements (unnecessary) and a call to deinit (I believe this is necessary) to mp-webrtc-sendrecv.c. So, basically right before the return statement in the main method, I added this:

gst_deinit();

Next is setting up an environment to show the leaks. I wrote a python script to make it easier for me to do this:

import subprocess
import time 
import os
import sys

signalling_server_path = os.path.abspath("./gstreamer/subprojects/gst-examples/webrtc/signalling")
mp_webrtc_sendrecv_path = os.path.abspath("./release/bin")
log_file_path = os.path.join(os.getcwd(), "log.txt")

signalling_command = [
    sys.executable,
    "simple_server.py"
]

room_command = [
    os.path.join(mp_webrtc_sendrecv_path, "mp-webrtc-sendrecv.exe"),
    "--room-id=MemoryTest",
    "--server=https://localhost:8443"
]

def build_certs():
    print("Building signalling server certs...")
    
    cert_pem_path = os.path.join(signalling_server_path, "cert.pem")
    key_pem_path = os.path.join(signalling_server_path, "key.pem")
    if os.path.exists(cert_pem_path) and os.path.exists(key_pem_path):
        print("Certs already created. Returning.")
        return
        
    windows_path = os.path.join(signalling_server_path, "generate_cert.bat")
    linux_path = os.path.join(signalling_server_path, "generate_cert.sh")
    
    if os.path.exists(windows_path) and os.name == "nt":
        print("Creating certs with:", windows_path)
        create_certs = subprocess.Popen(windows_path, cwd=signalling_server_path)
        create_certs.wait()
        
    if os.path.exists(linux_path) and os.name == "posix":
        print("Creating certs with:", linux_path)
        create_certs = subprocess.Popen(linux_path, cwd=signalling_server_path)
        create_certs.wait()

    print("Signalling server certs built...")

def run_signalling_server():
    print("Starting signalling server...")
    
    return subprocess.Popen(signalling_command, cwd=signalling_server_path)
    
def create_room():
    print("Creating room...")
    
    new_env = os.environ.copy()
    new_env["GST_DEBUG"] = "GST_TRACER:7,*leak*:7"
    new_env["GST_DEBUG_FILE"] = log_file_path
    new_env["GST_TRACERS"] = "leaks(check-refs=true)"
    
    return subprocess.Popen(room_command, env=new_env, cwd=mp_webrtc_sendrecv_path)
    
def join_room(num_joins, seconds_per_join):
    for loop in range(num_joins):
        print(f"Join: {loop}")

        process = subprocess.Popen(room_command, cwd=mp_webrtc_sendrecv_path)
        time.sleep(seconds_per_join)
        process.terminate()
        process.wait()

build_certs()
signalling_process = run_signalling_server()
create_room_process = create_room()

join_room(3, 60)

print("Stopping signalling server...")
signalling_process.terminate()
signalling_process.wait()
print("Signalling server stopped.")

print("Waiting for room server process to end...")
create_room_process.wait()
print("Room server process finished.")

A few notes about this:

It’s only been tested on Windows.
You may need to change signalling_server_path = os.path.abspath("./gstreamer/subprojects/gst-examples/webrtc/signalling") to be wherever the gstreamer example signalling server simple_server.py script is
You may need to change mp_webrtc_sendrecv_path = os.path.abspath("./release/bin") to be wherever the mp-webrtc-sendrecv executable is
If you’re not on Windows, you may need to change os.path.join(mp_webrtc_sendrecv_path, "mp-webrtc-sendrecv.exe") to point to whatever the executable for mp-webrtc-sendrecv is called.

I think that’s pretty much all that needs done. Or just run the equivalent logic in the command line.

Also for reference, is my version of the generate_cert.sh file for windows (generate_cert.bat, goes in same location as simple_server.py):

set BASE_DIR=%~dp0

set OUTDIR=""
if "%~1" neq "" (
  set OUTDIR=%~1\
)

openssl req -x509 -newkey rsa:4096 -keyout "key.pem" -out "cert.pem" -days 365 -nodes -subj "/CN=example.com" 2>&1

if not %errorlevel% equ 0 (
  exit /b %errorlevel%
)

If this all works, after the script runs, it will notify you of leaks. Here is my console output:

**(mp-webrtc-sendrecv.exe:23152): WARNING **: 15:08:19.953: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

And in my case, here’s what is in log.txt:

0:00:00.859201000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:169:gst_tracer_register:<tracerfactory0> new tracer factory for latency
0:00:00.860367000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:177:gst_tracer_register:<latency> tracer factory for 3962761632:GstLatencyTracer
0:00:00.861365000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:169:gst_tracer_register:<tracerfactory1> new tracer factory for log
0:00:00.862389000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:177:gst_tracer_register:<log> tracer factory for 3962756512:GstLogTracer
0:00:00.862930000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:169:gst_tracer_register:<tracerfactory2> new tracer factory for stats
0:00:00.864996000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:177:gst_tracer_register:<stats> tracer factory for 3962761120:GstStatsTracer
0:00:00.865616000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:169:gst_tracer_register:<tracerfactory3> new tracer factory for leaks
0:00:00.866071000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:177:gst_tracer_register:<leaks> tracer factory for 3962760608:GstLeaksTracer
0:00:00.866670000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:169:gst_tracer_register:<tracerfactory4> new tracer factory for factories
0:00:00.867166000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:177:gst_tracer_register:<factories> tracer factory for 3962757792:GstFactoriesTracer
0:00:06.926749000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:169:gst_tracer_register:<tracerfactory5> new tracer factory for validate
0:00:06.927364000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracer.c:177:gst_tracer_register:<validate> tracer factory for 4077695200:GstValidateRunner
0:00:07.730254000 23152 000001B3EC38FB10 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-alive.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;", description=(structure)"value\,\ type\=\(type\)gchararray\;", ref-count=(structure)"value\,\ type\=\(type\)guint\;", trace=(structure)"value\,\ type\=\(type\)gchararray\;";
0:00:07.730304000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-alive, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:07.730798000 23152 000001B3EC38FB10 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-refings.class, ts=(structure)"value\,\ type\=\(type\)guint64\;", type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;", description=(structure)"value\,\ type\=\(type\)gchararray\;", ref-count=(structure)"value\,\ type\=\(type\)guint\;", trace=(structure)"value\,\ type\=\(type\)gchararray\;";
0:00:07.730836000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-refings, ts=(guint64)%llu, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;
0:00:07.731262000 23152 000001B3EC38FB10 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-added.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;";
0:00:07.731296000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-added, type-name=(string)%s, address=(gpointer)%p;
0:00:07.731659000 23152 000001B3EC38FB10 TRACE             GST_TRACER gsttracerrecord.c:109:gst_tracer_record_build_format: object-removed.class, type-name=(structure)"value\,\ type\=\(type\)gchararray\;", address=(structure)"value\,\ type\=\(type\)gpointer\;";
0:00:07.731690000 23152 000001B3EC38FB10 DEBUG             GST_TRACER gsttracerrecord.c:123:gst_tracer_record_build_format: new format string: object-removed, type-name=(string)%s, address=(gpointer)%p;
0:03:01.010655000 23152 000001B3EC38FB10 DEBUG                  leaks gstleaks.c:765:gst_leaks_tracer_finalize:<leakstracer0> destroying tracer, checking for leaks
0:03:01.010696000 23152 000001B3EC38FB10 TRACE                  leaks gstleaks.c:732:process_leaks:<leakstracer0> start listing currently alive objects
0:03:01.010736000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)<'':sink>, ref-count=(uint)1, trace=(string);
0:03:01.010763000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)65892916000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)2, trace=(string);
0:03:01.010786000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)65898086000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)1, trace=(string);
0:03:01.010807000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66328154000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)2, trace=(string);
0:03:01.010829000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66335755000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.010850000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66660765000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)4, trace=(string);
0:03:01.010871000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66666740000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)3, trace=(string);
0:03:01.010892000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66674864000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.010913000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66750087000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.010933000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66795425000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.010954000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66818911000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.010983000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66828852000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.011005000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66831881000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.011026000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66918339000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.011046000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66987678000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.011121000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67115826000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.011146000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67147912000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.011167000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67154197000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.011188000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67184938000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.011209000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67194038000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.011230000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67330053000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.011251000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67338131000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.011272000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180219128000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.011293000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180219806000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)4, trace=(string);
0:03:01.011314000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180220739000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011335000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180221245000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011356000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180222132000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)3, trace=(string);
0:03:01.011377000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180222844000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.011398000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180474867000, type-name=(string)GstPad, address=(gpointer)000001B3F608B8C0, description=(string)unreffed, ref-count=(uint)1, trace=(string);
0:03:01.011437000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstDtlsAgent, address=(gpointer)000001B3F5D61630, description=(string)<dtlsagent0>, ref-count=(uint)1, trace=(string);
0:03:01.011460000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-alive, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1, ref-count=(uint)1, trace=(string);
0:03:01.011484000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)54281098000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)2, trace=(string);
0:03:01.011519000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)54281098000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)1, trace=(string);
0:03:01.011541000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)54767984000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)2, trace=(string);
0:03:01.011562000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)54767988000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)3, trace=(string);
0:03:01.011583000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)54806001000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)4, trace=(string);
0:03:01.011604000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66732821000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011625000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66804859000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011645000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66839800000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011666000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66891486000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011687000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66896415000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011708000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66897790000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011729000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)66963223000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011749000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67126758000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011770000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67129009000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011791000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67129009000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011812000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67129010000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011832000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67164914000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011853000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67334656000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)5, trace=(string);
0:03:01.011892000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67334664000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)reffed, ref-count=(uint)6, trace=(string);
0:03:01.011914000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)67334667000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)5, trace=(string);
0:03:01.011943000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180081503000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)4, trace=(string);
0:03:01.011964000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180130337000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)3, trace=(string);
0:03:01.011984000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180139010000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)2, trace=(string);
0:03:01.012005000 23152 000001B3EC38FB10 TRACE             GST_TRACER :0:: object-refings, ts=(guint64)180139010000, type-name=(string)GstCaps, address=(gpointer)000001B3F6391BE0, description=(string)unreffed, ref-count=(uint)1, trace=(string);
0:03:01.012026000 23152 000001B3EC38FB10 TRACE                  leaks gstleaks.c:752:process_leaks:<leakstracer0> listed 3 alive objects

My understanding is that when deinit happens, the app should:

Clean up any resources left over.
Tell the leak tracer to list currently alive objects.

Obviously #2 works based on the log.txt output. But if there aren’t supposed to be any live objects at this point, it seems like something is wrong. Don’t know if it’s gstreamer itself or if it’s the example code.

I’m eventually going to try to put together an example closer to what I’ve been trying to work with. I may try to modify this example since I don’t think it’d require much.

uler3161 · February 12, 2024, 4:23pm

I haven’t been able to get a native implementation of adding rtsp teardown signal handler working as I had hoped, so maybe that particular leak is in gstreamer-sharp. I was able to come up with this C# example that leaks. I’ve narrowed it down to iterating through the iterator created by IterateAllByElementFactoryName. Calling that method itself doesn’t seem to cause a leak, you actually have to iterate through it.

using GLib;
using Gst;
using Gst.Rtsp;
using Gst.WebRTC;
using Application = Gst.Application;
using Global = Gst.Global;
using ObjectManager = GtkSharp.GstreamerSharp.ObjectManager;
using Task = System.Threading.Tasks.Task;

namespace LeakyGStreamer
{
    internal class Program
    {
        private const string PipelineText = "rtspsrc location=rtsp://localhost:8554/bbb ! fakesink";

        private static MainLoop Loop
        {
            get;
            set;
        }

        private static Pipeline Pipeline
        {
            get;
            set;
        }

        static void Main(
            string[] args)
        {
            Environment.SetEnvironmentVariable("GST_DEBUG", "GST_TRACER:7,*leaks*:7");
            Environment.SetEnvironmentVariable("GST_TRACERS", "leaks");
            Environment.SetEnvironmentVariable("GST_DEBUG_FILE", "leak_trace.txt");

            Application.Init();
            ObjectManager.Initialize();
            GType.Register(WebRTCSessionDescription.GType, typeof(WebRTCSessionDescription));
            GType.Register(RTSPMessage.GType, typeof(RTSPMessage));
            ExceptionManager.UnhandledException += OnUnhandledGlibException;

            Pipeline = (Pipeline)Parse.Launch(PipelineText);

            TestIterate();

            Pipeline.SetState(State.Playing);

            Loop = new MainLoop();

            Task.Run(async () =>
            {
                var seconds = 10;
                Console.WriteLine($"Running for {seconds} seconds, then closing...");
                await Task.Delay(seconds * 1000);
                Loop?.Quit();
            });

            Console.WriteLine("Starting loop");
            Loop.Run();

            Console.WriteLine("Stopping");
            Pipeline.SetState(State.Null);

            Pipeline.Dispose();

            GC.Collect();
            GC.WaitForPendingFinalizers();
            Global.Deinit();
            GC.Collect();
            GC.WaitForPendingFinalizers();
        }

        private static void TestIterate()
        {
            Console.WriteLine("Testing Iterator...");
            using var iterator = Pipeline.IterateAllByElementFactoryName("rtspsrc");
            foreach (Element e in iterator)
            {
                Console.WriteLine("Found item");
            }

            Console.WriteLine("Iterator done.");
        }

        private static void OnUnhandledGlibException(
            UnhandledExceptionArgs args)
        {
            Console.WriteLine(args.ExceptionObject.GetType()
                .Name);
        }
    }
}

uler3161 · February 12, 2024, 5:42pm

@tpm Do you know the GStreamer bindings very well? I think I may have found one of the issues, but I just don’t have the GStreamer knowledge to know if it’s the problem and the correct fix.

The change is to Iterator.cs (the one in the custom folder):

// Iterator.cs - Custom iterator wrapper for IEnumerable
//
// Authors:
//     Maarten Bosmans <mkbosmans@gmail.com>
//     Sebastian Dröge <slomo@circular-chaos.org>
//     Stephan Sundermann <stephansundermann@gmail.com>
//
// Copyright (c) 2009 Maarten Bosmans
// Copyright (c) 2009 Sebastian Dröge
// Copyright (c) 2013 Stephan Sundermann
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of version 2 of the Lesser GNU General 
// Public License as published by the Free Software Foundation.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this program; if not, write to the
// Free Software Foundation, Inc., 59 Temple Place - Suite 330,
// Boston, MA 02111-1307, USA.

namespace Gst {

	using GLib;
	using System;
	using System.Collections;
	using System.Collections.Generic;
	using System.Runtime.InteropServices;

	public partial class Iterator : IEnumerable {

		[DllImport("gobject-2.0-0.dll", CallingConvention = CallingConvention.Cdecl)]
		static extern IntPtr g_value_reset(ref GLib.Value val);

		[DllImport("gstreamer-1.0-0.dll", CallingConvention = CallingConvention.Cdecl)]
		static extern int gst_iterator_next(IntPtr raw, ref GLib.Value elem);

		public Gst.IteratorResult Next(ref GLib.Value elem) {
			int raw_ret = gst_iterator_next(Handle, ref elem);
			Gst.IteratorResult ret = (Gst.IteratorResult)raw_ret;
			return ret;
		}

        public void Reset(ref GLib.Value elem) {
            g_value_reset(ref elem);
        }

		private class Enumerator : IEnumerator {
			Iterator iterator;
			Hashtable seen = new Hashtable();

			private object current = null;
			public object Current {
				get {
					return current;
				}
			}

			public bool MoveNext() {
				IntPtr raw_ret;
				bool retry = false;

				if (iterator.Handle == IntPtr.Zero)
					return false;

				do {
					GLib.Value value = new GLib.Value(GLib.GType.Boolean);
					value.Dispose();

					IteratorResult ret = iterator.Next(ref value);

					switch (ret) {
						case IteratorResult.Done:
							return false;
						case IteratorResult.Ok:
							if (seen.Contains(value)) {
								retry = true;
								break;
							}
							seen.Add(value, null);
							current = value.Val;
                            iterator.Reset(ref value);
							return true;
						case IteratorResult.Resync:
							iterator.Resync();
							retry = true;
							break;
						default:
						case IteratorResult.Error:
							throw new Exception("Error while iterating");
					}
				} while (retry);

				return false;
			}

			public void Reset() {
				seen.Clear();
				if (iterator.Handle != IntPtr.Zero)
					iterator.Resync();
			}

			public Enumerator(Iterator iterator) {
				this.iterator = iterator;
			}
		}

		private Enumerator enumerator = null;

		public IEnumerator GetEnumerator() {
			if (this.enumerator == null)
				this.enumerator = new Enumerator(this);
			return this.enumerator;
		}
	}
}

The change is basically just to call g_value_reset at the end of each loop when the iterator result is Ok.

I based this change off this comment in gstiterator.c:

        case GST_ITERATOR_OK:
          ...get/use/change item here...
          g_value_reset (&item);
          break;

uler3161 · February 12, 2024, 8:36pm

Well, the fix worked for my example project, but not my real project. Back to the drawing board I guess.

uler3161 · February 13, 2024, 9:47pm

I put together a simple C only example that leaks. I tried to mimic what the Iterator.cs class does for iterating through the elements:

#include <gst/gst.h>

static GstElement *pipeline;
static GMainLoop *loop;

static gboolean stop_pipeline(gpointer data) 
{
    g_print("Stopping pipeline after 10 seconds.\n");
    gst_element_set_state(pipeline, GST_STATE_NULL);
    g_main_loop_quit(loop);
    return G_SOURCE_REMOVE;
}

static void test_iteration (const GValue * item, gpointer unused)
{
    g_print("Found rtspsrc.\n");
}

static gboolean iterate()
{
    GstIterator *it = gst_bin_iterate_all_by_element_factory_name (GST_BIN (pipeline), "rtspsrc");
    gboolean retry;
    
    retry = FALSE;
    
    do 
    {
        GValue value = { 0, };
        switch(gst_iterator_next(it, &value))
        {
            case GST_ITERATOR_OK:
                gst_iterator_free (it);
                return TRUE;
            case GST_ITERATOR_RESYNC:
                gst_iterator_resync (it);
                retry = TRUE;
                break;
            case GST_ITERATOR_ERROR:
                g_print("Iterator error.\n");
                gst_iterator_free (it);
                return FALSE;
            case GST_ITERATOR_DONE:
                gst_iterator_free (it);
                return FALSE;
        }
        
    } while (retry);
    
    gst_iterator_free (it);
    return FALSE;
}

int
main (int argc, char *argv[])
{
    gst_init (&argc, &argv);

    GError *error = NULL;
    
    loop = g_main_loop_new(NULL, FALSE);

    pipeline = gst_parse_launch ("rtspsrc location=rtsp://localhost:8554/bbb ! decodebin ! autovideosink", &error);
    if (!pipeline) 
    {
        g_print ("Parse error: %s\n", error->message);
        exit (1);
    }

    iterate();

    //gst_iterator_foreach (it, (GstIteratorForeachFunction) test_iteration, NULL);
    //gst_iterator_free (it);

    gst_element_set_state (pipeline, GST_STATE_PLAYING);

    g_timeout_add_seconds(10, stop_pipeline, NULL);

    g_main_loop_run(loop);

    gst_element_set_state (pipeline, GST_STATE_NULL);
    gst_object_unref (pipeline);

    gst_deinit();

    return 0;
}

ystreet00 · February 14, 2024, 12:06am

You are missing a g_value_unset/g_value_reset in the GST_ITERATOR_OK case as outlined in the gst_iterator_next documentation and in the example in the GstIterator documentation

uler3161 · February 14, 2024, 12:56am

I figured that was at least part of the problem, but it wasn’t clear that not doing that would cause an object ref problem. I did try to implement the reset in the gstreamer-sharp Iterator.cs. It fixed the sample app, but it did not fix the real app.

ystreet00 · February 14, 2024, 6:42am

That would indicate that you may have a reference counting issue in your application.

uler3161 · February 14, 2024, 1:43pm

If there is a ref count issue in my app, it’s definitely not obvious. Regardless of whether I’m using the original Iterator implementation or the one I added the reset code to, the following code is what causes a ref count issue:

using var iterator = Pipeline.IterateAllByElementFactoryName("rtspsrc");
foreach (Element e in iterator)
{
}

Originally I thought it was the code I had within the loop which was adding a signal handler. However, with more testing I found out that just iterating through the iterator caused the issue.

I know if I add e.Unref() in that loop it fixes the problem. However, it’s recommended that you never call that method in C# code and the person who recommended that also suggested Unref/Ref should be changed to private methods in the API.

Another point of interest is that it only leaks whatever elements are returned by IterateAllByElementFactoryName. Other elements in the pipeline don’t leak.

Perhaps I’m not using the API correctly? It’s hard to find good examples or docs on the gstreamer-sharp side of things. But really, I don’t see why iterating through an iterator does this. It really seems like a gstreamer or gstreamer-sharp bug rather than an issue in my app.

uler3161 · February 23, 2024, 5:33pm

Came up with another test, or actually multiple tests.

First, a change on the gstreamer-sharp side. Specifically to add unset/reset/unref calls in various places:

namespace Gst {

	using GLib;
	using System;
	using System.Collections;
	using System.Collections.Generic;
	using System.Runtime.InteropServices;

	public partial class Iterator : IEnumerable {

		[DllImport("gobject-2.0-0.dll", CallingConvention = CallingConvention.Cdecl)]
		static extern IntPtr g_value_reset(ref GLib.Value val);

		[DllImport("gobject-2.0-0.dll", CallingConvention = CallingConvention.Cdecl)]
		static extern void g_value_unset(ref GLib.Value val);

		[DllImport ("gobject-2.0-0.dll", CallingConvention = CallingConvention.Cdecl)]
		static extern void g_object_unref (IntPtr val);

		[DllImport("gstreamer-1.0-0.dll", CallingConvention = CallingConvention.Cdecl)]
		static extern int gst_iterator_next(IntPtr raw, ref GLib.Value elem);

		public Gst.IteratorResult Next(ref GLib.Value elem) {
			int raw_ret = gst_iterator_next(Handle, ref elem);
			Gst.IteratorResult ret = (Gst.IteratorResult)raw_ret;
			return ret;
		}

        public bool UnrefOnDispose {
            get;set;
        } = false;

        public void Reset(ref GLib.Value value) {
            g_value_reset(ref value);
        }

        public void Unset(ref GLib.Value value) {
            g_value_unset(ref value);
        }

        public void Unref(IntPtr value) {
            g_object_unref(value);
        }

		private class Enumerator : IEnumerator, IDisposable {
			Iterator iterator;
			Hashtable seen = new Hashtable();

			private object _current = null;
			public object Current {
				get {
					return _current;
				}
                set {
                    if (_current is GLib.Object glibObject) {
                        iterator.Unref(glibObject.Handle);
                    }
                    _current = value;
                }
			}

            public void Dispose()
            {
                Console.WriteLine("Iterator Dispose Called");

                if (iterator.UnrefOnDispose && _current is GLib.Object glibObject) {
                    Console.WriteLine("Unref:" + glibObject.Handle.ToString("x8"));
                    iterator.Unref(glibObject.Handle);
                }
                
                _current = null;
            }

			public bool MoveNext() {
				IntPtr raw_ret;
				bool retry = false;

				if (iterator.Handle == IntPtr.Zero)
					return false;

				do {
					GLib.Value value = new GLib.Value(GLib.GType.Boolean);
					value.Dispose();

					IteratorResult ret = iterator.Next(ref value);

					switch (ret) {
						case IteratorResult.Done:
                            iterator.Unset(ref value);
							return false;
						case IteratorResult.Ok:
							if (seen.Contains(value)) {
                                iterator.Reset(ref value);
								retry = true;
								break;
							}
							seen.Add(value, null);
							Current = value.Val;
                            iterator.Reset(ref value);
							return true;
						case IteratorResult.Resync:
							iterator.Resync();
							retry = true;
							break;
						default:
						case IteratorResult.Error:
							throw new Exception("Error while iterating");
					}
				} while (retry);

				return false;
			}

			public void Reset() {
				seen.Clear();
				if (iterator.Handle != IntPtr.Zero)
					iterator.Resync();
			}

			public Enumerator(Iterator iterator) {
				this.iterator = iterator;
			}
		}

		private Enumerator enumerator = null;

		public IEnumerator GetEnumerator() {
			if (this.enumerator == null)
				this.enumerator = new Enumerator(this);
			return this.enumerator;
		}
	}
}

Now the test harness in C#:

using GLib;
using Gst;
using Gst.Rtsp;
using Gst.WebRTC;
using Application = Gst.Application;
using Global = Gst.Global;
using ObjectManager = GtkSharp.GstreamerSharp.ObjectManager;
using Process = System.Diagnostics.Process;
using Task = System.Threading.Tasks.Task;

namespace LeakyGStreamer
{
    internal class Program
    {
        private const string PipelineText = "rtspsrc location=rtsp://localhost:8554/bbb ! fakesink";

        private static MainLoop Loop
        {
            get;
            set;
        }

        private static Pipeline Pipeline
        {
            get;
            set;
        }

        private static void Cleanup()
        {
            Console.WriteLine("Stopping");
            Pipeline.SetState(State.Null);

            Pipeline.Dispose();

            GC.Collect();
            GC.WaitForPendingFinalizers();
            Global.Deinit();
            GC.Collect();
            GC.WaitForPendingFinalizers();
        }

        private static void InitializeGStreamer()
        {
            Environment.SetEnvironmentVariable("GST_DEBUG", "GST_TRACER:7,*leaks*:7");
            Environment.SetEnvironmentVariable("GST_TRACERS", "leaks");
            Environment.SetEnvironmentVariable("GST_DEBUG_FILE", "leak_trace.txt");

            Application.Init();
            ObjectManager.Initialize();
            GType.Register(WebRTCSessionDescription.GType, typeof(WebRTCSessionDescription));
            GType.Register(RTSPMessage.GType, typeof(RTSPMessage));
            ExceptionManager.UnhandledException += OnUnhandledGlibException;
        }

        static void Main(
            string[] args)
        {
            if (args.Length != 3)
            {
                RunTests();
                return;
            }

            var iterations = int.Parse(args[0]);
            var dispose = bool.Parse(args[1]);
            var unref = bool.Parse(args[2]);

            InitializeGStreamer();

            Pipeline = (Pipeline)Parse.Launch(PipelineText);

            for (var i = 0; i < iterations; i++)
            {
                TestIterate(dispose, unref);
            }

            Pipeline.SetState(State.Playing);

            Loop = new MainLoop();

            StartExitTask();
            Loop.Run();
            Cleanup();
        }

        private static void OnUnhandledGlibException(
            UnhandledExceptionArgs args)
        {
            Console.WriteLine(args.ExceptionObject.GetType()
                .Name);
        }

        private static (bool leaks, string errors) RunTest(
            int iterations,
            bool dispose,
            bool unref)
        {
            var process = new Process();
            process.StartInfo.FileName = AppDomain.CurrentDomain.FriendlyName;
            process.StartInfo.Arguments = $"{iterations} {dispose} {unref}";
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.RedirectStandardOutput = true;
            process.StartInfo.RedirectStandardError = true;
            process.StartInfo.CreateNoWindow = true;
            process.Start();
            process.WaitForExit();

            var output = process.StandardOutput.ReadToEnd();
            var error = process.StandardError.ReadToEnd();
            var leaks = error.Contains("Leaks detected");
            return (leaks, error);
        }

        private static void RunTests()
        {
            Console.WriteLine("Running test suite...");

            for (var iterations = 1; iterations <= 2; iterations++)
            {
                foreach (var dispose in new[] { false, true })
                {
                    foreach (var unref in new[] { false, true })
                    {
                        Console.WriteLine($"Testing: {iterations} iterations, Dispose: {dispose}, Unref: {unref}");
                        var (leaks, errors) = RunTest(iterations, dispose, unref);
                        Console.WriteLine($"  Leaks? {leaks}");
                        Console.WriteLine("  Errors:");
                        Console.WriteLine($"    {errors.Replace(Environment.NewLine, Environment.NewLine + "    ")}");
                    }
                }
            }

            Console.WriteLine("Test suite done.");
        }

        private static void StartExitTask()
        {
            Task.Run(async () =>
            {
                var seconds = 2;
                await Task.Delay(seconds * 1000);
                Loop.Quit();
            });
        }

        private static void TestIterate(
            bool dispose,
            bool unref)
        {
            var iterator = Pipeline.IterateAllByElementFactoryName("rtspsrc");
            if (unref)
            {
                iterator.UnrefOnDispose = unref;
            }

            foreach (Element e in iterator)
            {
            }

            if (dispose)
            {
                iterator.Dispose();
            }
        }
    }
}

I decided on the Gstreamer-Sharp Iterator.cs changes based on what I’ve read in the docs. Specifically, the GstIterator docs shows to do a reset when the iterator returns OK, and unset when DONE. Also mentioned in those docs is that ref counted objects don’t have their ref count increased unless you explicitly do that, but that’s not what I’m seeing. Hence the reason I added unref code.

Here’s the output from running the tests:

Running test suite...
Testing: 1 iterations, Dispose: False, Unref: False
  Leaks? True
  Errors:

    (LeakyGStreamer:33036): GStreamer-WARNING **: 09:07:54.656: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.
    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    ** (LeakyGStreamer:33036): WARNING **: 09:07:56.814: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

Testing: 1 iterations, Dispose: False, Unref: True
  Leaks? True
  Errors:

    (LeakyGStreamer:12460): GStreamer-WARNING **: 09:07:57.036: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.
    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    ** (LeakyGStreamer:12460): WARNING **: 09:07:59.230: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

Testing: 1 iterations, Dispose: True, Unref: False
  Leaks? True
  Errors:

    (LeakyGStreamer:21112): GStreamer-WARNING **: 09:07:59.492: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.
    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    (LeakyGStreamer:21112): GStreamer-CRITICAL **: 09:07:59.671: gst_iterator_free: assertion 'it != NULL' failed

    ** (LeakyGStreamer:21112): WARNING **: 09:08:01.654: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

Testing: 1 iterations, Dispose: True, Unref: True
  Leaks? False
  Errors:

    (LeakyGStreamer:1268): GStreamer-WARNING **: 09:08:01.868: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.
    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    (LeakyGStreamer:1268): GStreamer-CRITICAL **: 09:08:02.051: gst_iterator_free: assertion 'it != NULL' failed

Testing: 2 iterations, Dispose: False, Unref: False
  Leaks? True
  Errors:

    (LeakyGStreamer:25208): GStreamer-WARNING **: 09:08:04.344: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.
    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    ** (LeakyGStreamer:25208): WARNING **: 09:08:06.530: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

Testing: 2 iterations, Dispose: False, Unref: True
  Leaks? True
  Errors:

    (LeakyGStreamer:10988): GStreamer-WARNING **: 09:08:06.731: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.

    (LeakyGStreamer:10988): GStreamer-CRITICAL **: 09:08:06.837:
    Trying to dispose object "rtspsrc0", but it still has a parent "pipeline0".
    You need to let the parent manage the object instead of unreffing the object directly.

    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    ** (LeakyGStreamer:10988): WARNING **: 09:08:08.893: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

Testing: 2 iterations, Dispose: True, Unref: False
  Leaks? True
  Errors:

    (LeakyGStreamer:27444): GStreamer-WARNING **: 09:08:09.102: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.
    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    (LeakyGStreamer:27444): GStreamer-CRITICAL **: 09:08:09.272: gst_iterator_free: assertion 'it != NULL' failed

    (LeakyGStreamer:27444): GStreamer-CRITICAL **: 09:08:09.277: gst_iterator_free: assertion 'it != NULL' failed

    ** (LeakyGStreamer:27444): WARNING **: 09:08:11.270: Leaks detected and logged under GST_DEBUG=GST_TRACER:7

Testing: 2 iterations, Dispose: True, Unref: True
  Leaks? False
  Errors:

    (LeakyGStreamer:25536): GStreamer-WARNING **: 09:08:11.529: Failed to load plugin 'D:\GStreamerBuild\release\lib\gstreamer-1.0\gstpng.dll': The specified module could not be found.
    This usually means Windows was unable to find a DLL dependency of the plugin. Please check that PATH is correct.
    You can run 'dumpbin -dependents' (provided by the Visual Studio developer prompt) to list the DLL deps of any DLL.
    There are also some third-party GUIs to list and debug DLL dependencies recursively.

    (LeakyGStreamer:25536): GStreamer-CRITICAL **: 09:08:11.667:
    Trying to dispose object "rtspsrc0", but it still has a parent "pipeline0".
    You need to let the parent manage the object instead of unreffing the object directly.

    'g_io_module_load': The specified procedure could not be found.
    Failed to load module: D:\GStreamerBuild\release\lib\gio\modules\gioenvironmentproxy.dll

    (LeakyGStreamer:25536): GStreamer-CRITICAL **: 09:08:11.734: gst_iterator_free: assertion 'it != NULL' failed

    (LeakyGStreamer:25536): GStreamer-CRITICAL **: 09:08:11.736: gst_iterator_free: assertion 'it != NULL' failed

Test suite done.

The best situation appears to be that I need to:

Only iterate once. Clearly a problem if I need to iterate again somewhere else in my code.
Dispose of the Iterator after I’m done.
Unref objects as I iterate through them and unref the Current object in the Iterator Enumerator object when I dispose the Iterator.

This does result in a critical error saying that the iterator is null. That’s likely because I already disposed it by the time the pipeline is disposed and the pipeline is trying to dispose it. If I don’t dispose it, a whole bunch of other elements are leaked. I’m not sure what side effects disposing the iterator has though.

I’m just stuck on why this is working the way it is and how I’m actually supposed to be handling this. I’m still not convinced I have anything wrong in my code.

joaoantoniocardoso · May 18, 2024, 12:10am

Hi, I didn’t read all the messages above, but I want to point out that my problem was happening inside the webrtcbin threads (libnice), and it was solved when I updated to 1.24.0, so you might want to try it.