[blocking] v4l2h264dec: fallback buffer pool leak (~6 MB per decode session) when downstream does not answer ALLOCATION query

General Discussion
1

GStreamer version: 1.26.2
Platform: Raspberry Pi 3, Debian Trixie, bcm2835-codec V4L2 driver
Element: v4l2h264dec
Component: gst-plugins-good / sys/v4l2

Bug description

Each time a video is decoded with v4l2h264dec, approximately 6–7 MB of virtual memory is permanently leaked and never reclaimed, even after the pipeline goes to NULL. On a 32-bit process (Pi3, 1 GB RAM), the accumulation causes the player to hit the ~1.9 GB virtual address space limit and be killed by the OOM killer.

Root cause

When v4l2h264dec sends a GST_QUERY_ALLOCATION downstream to negotiate buffer allocators, no downstream sink responds to the query. The decoder log confirms:

didn't get downstream ALLOCATION hints → no pool, making new pool

With no answer, v4l2h264dec creates a fallback copy pool for each decode session. On pipeline teardown, exactly 2 frames from this fallback pool are never released — 2 × ~3.1 MB (allocated via malloc/mmap), leaked permanently per session.

Reproduction

Reproduced both with GStreamer alone (no Qt, no NeoPlayer) and in production on Raspberry Pi 3.

Minimal reproduction — GStreamer only:

# Repeat N times, observe VmSize growing by ~6-7 MB per run
gst-launch-1.0 filesrc location=test.mp4 \
  ! qtdemux ! h264parse ! v4l2h264dec ! fakesink

Neither fakesink nor Qt’s qgstvideorenderersink respond to the ALLOCATION query — both reproduce the leak. This is not specific to any particular sink; the leak is systematic whenever downstream is silent on ALLOCATION.

In production (NeoPlayer, Raspberry Pi 3, 32-bit):

VmSize grows by ~7 MB per video played. RSS remains stable until the process approaches 1.9 GB virtual memory and is killed by the OOM killer or earlyoom (SIGTERM at 15% free RAM).

Additional context — no common sink responds to ALLOCATION

In our setup we tested fakesink and Qt’s qgstvideorenderersink (QGstreamerVideoSink). Neither responds to GST_QUERY_ALLOCATION. This means the fallback pool is triggered systematically in any standard pipeline on this platform.

A sink that properly responds with GstVideoMeta support should prevent the fallback pool from being created — this is exactly what our workaround demonstrates (see below). However, the silent sink behavior is not illegal per the GStreamer spec, so v4l2h264dec should handle it without leaking.

Workaround / proposed fix

Intercept the ALLOCATION query on the src pad of v4l2h264dec before it reaches downstream, and answer it by advertising GstVideoMeta support. This causes the decoder to push its own V4L2 buffers directly, bypassing the fallback pool entirely.

static GstPadProbeReturn alloc_probe(GstPad *pad, GstPadProbeInfo *info, gpointer data) {
    GstQuery *q = GST_PAD_PROBE_INFO_QUERY(info);
    if (!q || GST_QUERY_TYPE(q) != GST_QUERY_ALLOCATION) return GST_PAD_PROBE_OK;
    if (!(GST_PAD_PROBE_INFO_TYPE(info) & GST_PAD_PROBE_TYPE_PUSH)) return GST_PAD_PROBE_OK;
    if (!gst_query_find_allocation_meta(q, GST_VIDEO_META_API_TYPE, NULL))
        gst_query_add_allocation_meta(q, GST_VIDEO_META_API_TYPE, NULL);
    return GST_PAD_PROBE_HANDLED; /* answer before forwarding downstream */
}

The V4L2 buffers are then immediately copied into a proper GstVideoBufferPool (explicitly released on PAUSED→READY) and returned to the firmware right away.

Result: leak drops to 0.00 MB per playback in isolated GStreamer tests (measured on 300-run loop).

Expected behavior

v4l2h264dec should either:

  • fully release the fallback pool on pipeline teardown when no downstream reply was received, or
  • treat a silent downstream as implicit GstVideoMeta support and avoid creating the fallback pool altogether

Actual behavior

~6–7 MB of virtual memory leaked per decode session, cumulative, never freed → causes freez ! appCrash !

Is the expected behavior for v4l2h264dec to fully release the fallback pool on teardown when downstream did not respond to ALLOCATION, or should downstream sinks always provide an answer to this query?

If the latter, should fakesink and similar generic sinks be updated to respond with GstVideoMeta support by default?

Thank you for your time ( & patience , it was a lot of text)