Can't tee AAC-encoded audio, but can tee raw audio

Application Development Embedded
1

I’m developing an embedded app for a custom hardware device (using an NXP i.MX 8M+ running Linux, for those who are interested). The hardware is being developed in-house as a part of the same project.

The BSP is including GStreamer version 1.20.3. Compiled via a Yocto project based on NXP’s imx-5.15.71-2.2.1.xml BSP manifest.

In my app, I need to receive video from its two MIPI camera devices and one PDM microphone. I’m encoding the video to H.264 (using a vendor-supplied hardware encoder plugin, vpuenc_h264) and passing each stream to mpegtsmux, and then saving to MP4 files (one for each camera’s feed).

A subset of the GST pipeline this app uses (content not needed to illustrate the problem omitted) is:

v4l2src device=/dev/video0
! video/x-raw, width=1280, height=720, framerate=30/1, format=GRAY8
! queue ! vpuenc_h264 ! h264parse ! mpegtsmux name=mux1
! filesink
v4l2src device=/dev/video1
! video/x-raw, width=1280, height=720, framerate=30/1, format=GRAY8
! queue ! vpuenc_h264 ! h264parse ! mpegtsmux name=mux2
! filesink

This part works great.

When I want to mux the audio in to the stream to be included with the video in the files, I can add it to one of the muxes this way:

alsasrc device=\"hw:0,0\"
! audio/x-raw, format=S16LE, rate=22050, channels=1
! queue ! audioconvert ! avenc_aac name=audio_aac
! mux1.

This also works great. But if I try to take the AAC-encoded audio and split it with a tee so it goes to both muxes, it doesn’t work:

alsasrc device=\"hw:0,0\"
! audio/x-raw, format=S16LE, rate=22050, channels=1
! queue ! audioconvert ! avenc_aac name=audio_aac
! tee name=taudio
  taudio. ! mux1.
  taudio. ! mux2.

The above fails. If I turn on GST debugging, I see these messages (plus an identical set of messages for the second mux).

DEBUG  GST_ELEMENT_PADS gstutils.c:1948:gst_element_link_pads_full: linked pad audio_aac:src to pad taudio:sink
INFO      GST_PARENTAGE gstbin.c:4377:gst_bin_get_by_name: [pipeline0]: looking up child element taudio
INFO      GST_PARENTAGE gstbin.c:4377:gst_bin_get_by_name: [pipeline0]: looking up child element mux1
INFO       GST_PIPELINE gst/parse/grammar.y:683:gst_parse_perform_link: linking pad  taudio of GstTee named taudio to pad  mux1 of GstMpegTsMux named mux1 (0/0) with caps "(NULL)"
INFO   GST_ELEMENT_PADS gstutils.c:1816:gst_element_link_pads_full: trying to link element taudio:(any) to element mux1:(any)
DEBUG  GST_ELEMENT_PADS gstutils.c:1925:gst_element_link_pads_full: looping through allowed src and dest pads
DEBUG  GST_ELEMENT_PADS gstutils.c:1928:gst_element_link_pads_full: trying src pad taudio:sink
DEBUG  GST_ELEMENT_PADS gstutils.c:1991:gst_element_link_pads_full: trying dest pad mux1:src
DEBUG  GST_ELEMENT_PADS gstutils.c:1991:gst_element_link_pads_full: trying dest pad mux1:sink_65
DEBUG  GST_ELEMENT_PADS gstutils.c:2043:gst_element_link_pads_full: we might have request pads on both sides, checking...
DEBUG  GST_ELEMENT_PADS gstutils.c:2097:gst_element_link_pads_full: no link possible from taudio to mux1
INFO            default gstutils.c:2206:gst_element_link_pads_filtered: Could not link pads: taudio:(null) - mux1:(null)
ERROR      GST_PIPELINE gst/parse/grammar.y:773:gst_parse_perform_link: could not link taudio to mux1
DEBUG  GST_ELEMENT_PADS gstutils.c:1948:gst_element_link_pads_full: linked pad audio_aac:src to pad taudio:sink
INFO      GST_PARENTAGE gstbin.c:4377:gst_bin_get_by_name: [pipeline0]: looking up child element taudio
INFO      GST_PARENTAGE gstbin.c:4377:gst_bin_get_by_name: [pipeline0]: looking up child element mux1
INFO       GST_PIPELINE gst/parse/grammar.y:683:gst_parse_perform_link: linking pad  taudio of GstTee named taudio to pad  mux1 of GstMpegTsMux named mux1 (0/0) with caps "(NULL)"
INFO   GST_ELEMENT_PADS gstutils.c:1816:gst_element_link_pads_full: trying to link element taudio:(any) to element mux1:(any)
DEBUG  GST_ELEMENT_PADS gstutils.c:1925:gst_element_link_pads_full: looping through allowed src and dest pads
DEBUG  GST_ELEMENT_PADS gstutils.c:1928:gst_element_link_pads_full: trying src pad taudio:sink
DEBUG  GST_ELEMENT_PADS gstutils.c:1991:gst_element_link_pads_full: trying dest pad mux1:src
DEBUG  GST_ELEMENT_PADS gstutils.c:1991:gst_element_link_pads_full: trying dest pad mux1:sink_65
DEBUG  GST_ELEMENT_PADS gstutils.c:2043:gst_element_link_pads_full: we might have request pads on both sides, checking...
DEBUG  GST_ELEMENT_PADS gstutils.c:2097:gst_element_link_pads_full: no link possible from taudio to mux1
INFO            default gstutils.c:2206:gst_element_link_pads_filtered: Could not link pads: taudio:(null) - mux1:(null)
ERROR      GST_PIPELINE gst/parse/grammar.y:773:gst_parse_perform_link: could not link taudio to mux1

If, however, I tee the audio before the AAC encoding, the result is good:

alsasrc device=\"hw:0,0\"
! audio/x-raw, format=S16LE, rate=22050, channels=1
! queue
! tee name=taudio
  taudio. ! audioconvert ! avenc_aac name=audio_aac1 ! mux1.
  taudio. ! audioconvert ! avenc_aac name=audio_aac2 ! mux2.

The above works as expected. I get two MP4 files with audio.

Any idea why I can tee raw audio but not AAC-encoded audio? Is it a bug in GST? Or am I doing something wrong?

2

There seems to be something funky going on when both sides of a link (here: tee and mpegtsmux need request pads to be created by parse-launch).

This works for me (adding a queue for each branch after the tee, which you should do anyway):

gst-launch-1.0 videotestsrc ! 'video/x-raw, width=1280, height=720, framerate=30/1' ! x264enc tune=zerolatency ! h264parse ! mpegtsmux name=mux1 ! fakesink  \
videotestsrc ! 'video/x-raw, width=1280, height=720, framerate=30/1' ! x264enc tune=zerolatency ! h264parse ! mpegtsmux name=mux2 ! fakesink \
audiotestsrc ! 'audio/x-raw,format=S16LE,rate=22050,channels=1' ! audioconvert ! avenc_aac ! tee name=t  \
t. ! queue ! mux1.  \
t. ! queue ! mux2.
1 Like
3

Thanks. That change works. I don’t know why it should be necessary, but that’s a different discussion.