Inquiry: AV2 Multistream Support in GStreamer

Plugin and library development
1

Hi all,

I’m looking into the technical roadmap for supporting AV2’s multistream feature in GStreamer and would appreciate your insights on the implementation details.

1. Parser & Demuxer

AV2 multistream introduces a new layer structure where a single temporal unit can contain multiple coded extended layer units. This implies the need for a new parser/demuxer element that can:

  • Identify and split the different streams within the AV2 bitstream

  • Expose multiple source pads (1-to-N), similar to how existing demuxers work

  • Handle the multistream_decoder_mode flag and the associated MSDO OBU

Would this be implemented as a dedicated av2parse element, or could it be integrated into a broader av2demux? How should the stream headers and OBU parsing be structured to expose each layer as a separate stream?

2. Decoder

With dav2d already available as an AV2 software decoder, integrating it seems straightforward for single-stream decoding. However, for multistream:

  • Does dav2d currently support multistream decoding mode, or would this require extension?

  • How should the decoder handle multiple layers – as separate decoder instances, or a single instance producing multiple output pads?

  • Would we need a new element like av2multidec that explicitly handles multistream, or can the existing av2dec be extended?

3. Display & Rendering

For displaying multiple streams (e.g., multi-angle, VR/AR layers, or PiP), what’s the recommended approach?

  • Option A: Use separate sink branches (e.g., multiple autovideosink elements) for each stream

  • Option B: Composite streams into a single output using elements like compositor or videomixer for tiled display

  • Option C: Leverage GstStreamCollection and GST_EVENT_SELECT_STREAMS for application-level stream selection and switching

4. Pipeline Construction

What would a typical pipeline look like? My initial thought:

text

filesrc location=file.av2 ! av2parse ! av2multidec ! ...

But how would the branching work? With a 1-to-N demuxer/parser, the pipeline would need to dynamically link pads as they are created:

text

filesrc ! av2parse name=p
p.src_0 ! queue ! av2dec ! videoconvert ! autovideosink
p.src_1 ! queue ! av2dec ! videoconvert ! autovideosink

Would decodebin3/playbin3 be able to autoplug such a multistream AV2 source, or would custom bin/pipeline construction be required?

Summary of Questions

Area Key Questions
Parser How to parse OBUs and expose multiple streams? New element or extend existing?
Decoder Does dav2d support multistream? Single or multiple decoder instances?
Display Separate sinks vs. compositor vs. stream selection API?
Pipeline How to dynamically link pads? Autoplugging with decodebin3?

I’d appreciate any thoughts, references to similar implementations (e.g., how HEVC MVC or VP9 spatial layers are handled), or pointers to ongoing work.

Thanks