After OMX Removal, How about using C2 for DSP Audio processing?

Unlike video which has V4L2 as a well-established codec abstraction layer,
Audio currently has no equivalent framework, only ALSA (transport) exists..

For embedded Linux devices with external DSP cores (HiFi, Hexagon, etc.),
each vendor ends up writing their own GStreamer element from scratch.
There is no reusable abstraction like for video.

I’ve ported Android’s Codec2 (C2) framework to run on glibc Linux as a GStreamer subproject (gst-plugins-android).

Today it runs AOSP’s SW audio decoders (Opus, AAC, FLAC, Vorbis, MP3, IAMF),

but the architecture is designed so that HW-accelerated C2 components can be dropped in without changing the GStreamer layer.

This could serve as a common abstraction for ARM-based Linux devices to leverage their audio DSPs through a standardized interface.

Would there be interest in reviewing this as a new subproject?

There is no reply about this topic :sweat_smile:

I expect those features can be used for…

  • TV OSs

    • Can use various SoCs easily
  • Car Infotainment Systems

    • Can use various SoCs / OSs easily
    • Reduce the cost & difficulty of adopting OTT Services
  • XR Glasses / Potable Gaming Device OSs ( Like Steam OS )

    • Can use ARM based Chips’ External DSP Easliy.
  • SoC Vendors

    • Reduce Bring-Up Costs for OS Customized

OTTs : Can Expect Various OSs’ / Chips performance / Quality

Hello!

Sorry for the late response. We’re all doing this in our spare time, which is an especially rare commodity these days.

This does sound quite interesting to me. Can you talk about which vendors you have already got on board for this initiative? That might be privileged information, in which case we can talk in private: nirbheek@centricular.com

We would love to see a gstreamer plugin for C2 audio upstream, but I think the android bits should reside elsewhere? I expect other projects will also make use of those parts? Have you given some thought to that?

Hello @nirbheek !

Thanks for the interest despite your busy schedule

Regarding vendors — happy to share details privately.
I’ll reach out to nirbheek@centricular.com.

I gonna update some architecture to enhance dlopen first.
Now I’m working on a plugin loader so HW vendors can drop .so files without touching the GStreamer layer.

Regarding the android bits: I agree they should be a standalone linux library.
That idea allow other projects consume the C2 layer directly.

I expect other projects will also make use of those parts? Have you given some thought to that?

-> Sure, Here’s the target architecture:

libcodec2-linux/              ← Standalone lib ( non - GStreamer )
├── porting/
├── aosp/
├── C2Store_linux.cpp
├── dlopen scanner
└── Output : libcodec2-linux.so + pkg-config

gst-plugins-c2/               ← GStreamer plugin (wrapper)
├── src/gstc2audiodec.c
└── Dependency: libcodec2-linux

Dear All,

Thanks to @nirbheek ,

Now I’m updating architecture and testing decoding with both gstreamer and non - gstreamer applications :
I am looking forward to the day I can share this with you all.

/*
 * c2l_player
 * Copyright (c) 2026 Changyong Ahn <changyong.ahn@lge.com>
 */

#include <codec2-linux/codec2_linux.h>
#include <alsa/asoundlib.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>


static int find_mp3_frame(const uint8_t *buf, size_t len, size_t *frame_size) {
    /* MP3 sync: 0xFF 0xE0+ (11 sync bits) */
    for (size_t i = 0; i + 3 < len; i++) {
        if (buf[i] == 0xFF && (buf[i+1] & 0xE0) == 0xE0) {

            int version = (buf[i+1] >> 3) & 3;    /* 0=2.5, 2=2, 3=1 */
            int layer = (buf[i+1] >> 1) & 3;      /* 1=III, 2=II, 3=I */
            int br_idx = (buf[i+2] >> 4) & 0xF;
            int sr_idx = (buf[i+2] >> 2) & 3;
            int padding = (buf[i+2] >> 1) & 1;

            if (version == 1 || layer == 0 || br_idx == 0 || br_idx == 15 || sr_idx == 3)
                continue;

            static const int bitrates[16] = {0,32,40,48,56,64,80,96,112,128,160,192,224,256,320,0};
            static const int samplerates[4] = {44100, 48000, 32000, 0};

            int bitrate = bitrates[br_idx] * 1000;
            int samplerate = samplerates[sr_idx];
            if (version == 0) samplerate /= 4;      /* MPEG 2.5 */
            else if (version == 2) samplerate /= 2;  /* MPEG 2 */

            int fsize;
            if (layer == 3) /* Layer I */
                fsize = (12 * bitrate / samplerate + padding) * 4;
            else /* Layer II, III */
                fsize = 144 * bitrate / samplerate + padding;

            if (fsize > 0 && i + fsize <= len) {
                *frame_size = fsize;
                return (int)i;
            }
        }
    }
    return -1;
}

int main(int argc, char *argv[]) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <file.mp3> [speed]\n", argv[0]);
        fprintf(stderr, "  speed: 1.0=normal, 2.0=double, 0.5=half\n");
        return 1;
    }

    float speed = (argc > 2) ? atof(argv[2]) : 1.0f;
    if (speed <= 0) speed = 1.0f;

    FILE *f = fopen(argv[1], "rb");
    if (!f) { perror("fopen"); return 1; }
    fseek(f, 0, SEEK_END);
    size_t file_size = ftell(f);
    fseek(f, 0, SEEK_SET);
    uint8_t *file_data = malloc(file_size);
    fread(file_data, 1, file_size, f);
    fclose(f);
    printf("Loaded %s (%zu bytes)\n", argv[1], file_size);

    C2LContext *ctx = c2l_context_create();
    c2l_context_set_resource_capacity(ctx, "cpu.audio", 100);

    C2LSessionRequest req = {
        .component_name = "c2.android.mp3.decoder",
        .resource_id = "cpu.audio",
        .resource_units = 1,
        .priority = C2L_PRIORITY_FOREGROUND,
    };
    C2LSession *session = NULL;
    C2LStatus st = c2l_session_create(ctx, &req, &session);
    if (st != C2L_STATUS_OK) {
        fprintf(stderr, "session_create failed: %s\n", c2l_status_string(st));
        return 1;
    }
    printf("Decoder: %s\n", c2l_session_component_name(session));

    c2l_session_start(session);

    C2LAudioConfig config = { .sample_rate = 44100, .channels = 2 };
    c2l_session_configure_audio(session, &config);

    size_t pos = 0;
    uint8_t *pcm_buf = NULL;
    size_t pcm_total = 0;
    size_t pcm_cap = 0;
    int detected_rate = 0, detected_ch = 0;
    int frame_count = 0;

    while (pos < file_size) {
        size_t frame_size = 0;
        int offset = find_mp3_frame(file_data + pos, file_size - pos, &frame_size);
        if (offset < 0) break;
        pos += offset;

        C2LInputBuffer input = {
            .data = file_data + pos,
            .size = frame_size,
            .pts_ns = C2L_TIME_NONE,
            .flags = 0,
        };
        c2l_session_queue(session, &input);
        pos += frame_size;
        frame_count++;

        C2LOutputBuffer output = {0};
        while (c2l_session_pull(session, &output) == C2L_STATUS_OK && output.size > 0) {
            if (pcm_total + output.size > pcm_cap) {
                pcm_cap = (pcm_total + output.size) * 2;
                pcm_buf = realloc(pcm_buf, pcm_cap);
            }
            memcpy(pcm_buf + pcm_total, output.data, output.size);
            pcm_total += output.size;
            if (!detected_rate && output.sample_rate) {
                detected_rate = output.sample_rate;
                detected_ch = output.channels;
            }
            c2l_output_buffer_release(&output);
            memset(&output, 0, sizeof(output));
        }
    }

    c2l_session_drain(session);
    C2LOutputBuffer output = {0};
    while (c2l_session_pull(session, &output) == C2L_STATUS_OK && output.size > 0) {
        if (pcm_total + output.size > pcm_cap) {
            pcm_cap = (pcm_total + output.size) * 2;
            pcm_buf = realloc(pcm_buf, pcm_cap);
        }
        memcpy(pcm_buf + pcm_total, output.data, output.size);
        pcm_total += output.size;
        if (!detected_rate && output.sample_rate) {
            detected_rate = output.sample_rate;
            detected_ch = output.channels;
        }
        c2l_output_buffer_release(&output);
        memset(&output, 0, sizeof(output));
    }

    printf("Decoded: %d frames → %zu bytes PCM (%dHz, %dch)\n",
           frame_count, pcm_total, detected_rate, detected_ch);

    c2l_session_stop(session);
    c2l_session_destroy(session);
    c2l_context_destroy(ctx);

    if (!detected_rate) detected_rate = 44100;
    if (!detected_ch) detected_ch = 2;

    int playback_rate = (int)(detected_rate * speed);
    printf("Playing at %dx%dch @ %dHz (speed=%.1fx)\n",
           16, detected_ch, playback_rate, speed);

    snd_pcm_t *pcm_handle;
    int err = snd_pcm_open(&pcm_handle, "default", SND_PCM_STREAM_PLAYBACK, 0);
    if (err < 0) {
        fprintf(stderr, "ALSA open: %s\n", snd_strerror(err));
        fprintf(stderr, "Writing to /tmp/c2l_out.raw instead\n");
        FILE *out = fopen("/tmp/c2l_out.raw", "wb");
        fwrite(pcm_buf, 1, pcm_total, out);
        fclose(out);
        printf("Saved %zu bytes to /tmp/c2l_out.raw\n", pcm_total);
        goto cleanup;
    }

    snd_pcm_set_params(pcm_handle,
        SND_PCM_FORMAT_S16_LE,
        SND_PCM_ACCESS_RW_INTERLEAVED,
        detected_ch,
        playback_rate,
        1,          /* allow ALSA software resampling */
        500000);    /* desired latency in microseconds (500ms) */

    size_t frame_bytes = detected_ch * 2; /* bytes per frame (S16LE = 2 bytes/sample) */
    size_t total_frames = pcm_total / frame_bytes;
    snd_pcm_sframes_t written = snd_pcm_writei(pcm_handle, pcm_buf, total_frames);
    if (written < 0) {
        snd_pcm_recover(pcm_handle, written, 0);
        written = snd_pcm_writei(pcm_handle, pcm_buf, total_frames);
    }
    printf("Played %ld/%zu frames\n", (long)written, total_frames);

    snd_pcm_drain(pcm_handle);
    snd_pcm_close(pcm_handle);

cleanup:
    free(pcm_buf);
    free(file_data);
    printf("Done.\n");
    return 0;
}