Why my setting to feed audio RTP packets to GST works for Opus but not for M4A?

[braveyao]

Hi,

I’m working on a project in Golang and want to feed RTP packets from a RTSP audio stream to GStreamer. My practice works for Opus, but fails with M4A. Below are some code snippets and caps setting to each codec.

Appsrc creatation and settings:

elem, err := gst.NewElementWithName("appsrc", RTSPAppSourceLabel)) 
elem.SetProperty("caps", gst.NewCapsFromString(caps))
elem.SetProperty("is-live", true)
elem.SetArg("format", "time")

The source will work with a decodeBin and a custom sink, same to all codecs.

The caps for opus is like

application/x-rtp,media=audio,payload=97,encoding-name=OPUS,clock-rate=48000

The caps for M4A is like

application/x-rtp,media=audio,payload=97,encoding-name=MPEG4-GENERIC,clock-rate=44100,mode=aac-hbr

Except the caps setting, all other parts/processing are same.
Now the Opus stream works well. And to the M4A stream, I got an error as “[rtspAppSrc_audio] ERROR - Internal data stream error.” So wondering what’s missing for M4A packets?

BTW: I also used rtspsrc to verify the M4A stream and print out the caps setting from the pad, which is like:

map[clock-base:1685764825 clock-rate:44100 config:1210 encoding-name:MPEG4-GENERIC encoding-params:2 indexdeltalength:3 indexlength:3 media:audio mode:AAC-hbr npt-start:0 onvif-mode:false payload:97 play-scale:1 play-speed:1 profile-level-id:1 seqnum-base:7639 sizelength:13 ssrc:2858605283 streamtype:5]

I even tried to set the same to my codes and still failed. So there must be something else that matters to M4a, while not to Opus?

Also referred to the doc for Opus and M4A at:
https://gstreamer.freedesktop.org/documentation/rtp/rtpmp4gdepay.html?gi-language=c
https://gstreamer.freedesktop.org/documentation/rtp/rtpopusdepay.html?gi-language=c

Any helps are very appreciated!

[slomo]

Whenever you get an “internal data stream error” without more useful details, you’ll have to check the debug logs what exactly the cause of it is. Can you get a debug log with GST_DEBUG=6 and check / provide it here? Some lines above the first occurrence of the “internal data stream error” should be the real cause.

[braveyao]

@slomo, many thanks for the tips!
Successfully ran my app with gST_DEBUG=6 against same stream with either Opus or M4a encoding and collected the gst logs.
Saw that the two logs are almost identical all the way to PLAYING state. Then m4a logs show it fails with no negotiation.

0:00:00.583755000 93327 0x6000035159f0 DEBUG               GST_CAPS gstpad.c:2743:gst_pad_get_current_caps:<rtspAppSrc_audio:src> get current pad caps application/x-rtp, media=(string)audio, payload=(int)97, encoding-name=(string)MPEG4-GENERIC, clock-rate=(int)44100, mode=(string)aac-hbr
0:00:00.583775000 93327 0x6000035159f0 DEBUG               GST_PADS gstpad.c:4311:gst_pad_peer_query:<rtspAppSrc_audio:src> peer query 0x60000357c320 (allocation)
0:00:00.584485000 93327 0x6000035159f0 DEBUG               GST_PADS gstpad.c:4373:gst_pad_peer_query:<rtspAppSrc_audio:src> query failed
0:00:00.584487000 93327 0x6000035159f0 DEBUG                basesrc gstbasesrc.c:3314:gst_base_src_prepare_allocation:<rtspAppSrc_audio> peer ALLOCATION query failed
0:00:00.584718000 93327 0x6000035159f0 DEBUG                basesrc gstbasesrc.c:3321:gst_base_src_prepare_allocation:<rtspAppSrc_audio> ALLOCATION (1) params: allocation query: 0x60000357c320, GstQueryAllocation, caps=(GstCaps)"application/x-rtp\,\ media\=\(string\)audio\,\ payload\=\(int\)97\,\ encoding-name\=\(string\)MPEG4-GENERIC\,\ clock-rate\=\(int\)44100\,\ mode\=\(string\)aac-hbr", need-pool=(boolean)true, allocator=(GArray)NULL, pool=(GArray)NULL;
0:00:00.584736000 93327 0x6000035159f0 LOG                  basesrc gstbasesrc.c:2908:gst_base_src_loop:<rtspAppSrc_audio> next_ts 99:99:99.999999999 size 4096
0:00:00.584814000 93327 0x6000035159f0 DEBUG                basesrc gstbasesrc.c:2586:gst_base_src_get_range:<rtspAppSrc_audio> calling create offset 18446744073709551615 length 4096, time 0
0:00:00.584822000 93327 0x6000035159f0 LOG                   appsrc gstappsrc.c:1389:gst_app_src_update_queued_pop:<rtspAppSrc_audio> have buffer 0x1440080f0 of size 964
0:00:00.584829000 93327 0x6000035159f0 DEBUG                 appsrc gstappsrc.c:1455:gst_app_src_update_queued_pop:<rtspAppSrc_audio> Currently queued: 6880 bytes, 6 buffers, 0:00:00.301859411
0:00:00.584831000 93327 0x6000035159f0 DEBUG                basesrc gstbasesrc.c:2440:gst_base_src_do_sync:<rtspAppSrc_audio> no sync needed
0:00:00.584833000 93327 0x6000035159f0 DEBUG                basesrc gstbasesrc.c:2677:gst_base_src_get_range:<rtspAppSrc_audio> buffer ok
0:00:00.586413000 93327 0x6000035159f0 DEBUG               GST_PADS gstpad.c:1658:gst_pad_needs_reconfigure:<rtspAppSrc_audio:src> peeking RECONFIGURE flag 0
0:00:00.586426000 93327 0x6000035159f0 DEBUG                basesrc gstbasesrc.c:3070:gst_base_src_loop:<rtspAppSrc_audio> Not negotiated
0:00:00.586474000 93327 0x6000035159f0 DEBUG                basesrc gstbasesrc.c:3085:gst_base_src_loop:<rtspAppSrc_audio> pausing task, reason not-negotiated
0:00:00.586477000 93327 0x6000035159f0 DEBUG               GST_PADS gstpad.c:6382:gst_pad_pause_task:<rtspAppSrc_audio:src> pause task
0:00:00.586492000 93327 0x6000035159f0 DEBUG                   task gsttask.c:729:gst_task_set_state_unlocked:<rtspAppSrc_audio:src> Changing task 0x142f056e0 to state 2
0:00:00.586505000 93327 0x6000035159f0 WARN                 basesrc gstbasesrc.c:3132:gst_base_src_loop:<rtspAppSrc_audio> error: Internal data stream error.
0:00:00.586601000 93327 0x6000035159f0 WARN                 basesrc gstbasesrc.c:3132:gst_base_src_loop:<rtspAppSrc_audio> error: streaming stopped, reason not-negotiated (-4)

The Opus logs seems no issue.

0:00:00.664667000 93534 0x60000203d720 DEBUG               GST_CAPS gstpad.c:2743:gst_pad_get_current_caps:<rtspAppSrc_audio:src> get current pad caps application/x-rtp, media=(string)audio, payload=(int)97, encoding-name=(string)OPUS, clock-rate=(int)48000
0:00:00.664721000 93534 0x60000203d720 DEBUG               GST_PADS gstpad.c:4311:gst_pad_peer_query:<rtspAppSrc_audio:src> peer query 0x60000202bde0 (allocation)
0:00:00.665515000 93534 0x60000203d720 DEBUG               GST_PADS gstpad.c:4373:gst_pad_peer_query:<rtspAppSrc_audio:src> query failed
0:00:00.665519000 93534 0x60000203d720 DEBUG                basesrc gstbasesrc.c:3314:gst_base_src_prepare_allocation:<rtspAppSrc_audio> peer ALLOCATION query failed
0:00:00.665652000 93534 0x60000203d720 DEBUG                basesrc gstbasesrc.c:3321:gst_base_src_prepare_allocation:<rtspAppSrc_audio> ALLOCATION (1) params: allocation query: 0x60000202bde0, GstQueryAllocation, caps=(GstCaps)"application/x-rtp\,\ media\=\(string\)audio\,\ payload\=\(int\)97\,\ encoding-name\=\(string\)OPUS\,\ clock-rate\=\(int\)48000", need-pool=(boolean)true, allocator=(GArray)NULL, pool=(GArray)NULL;
0:00:00.678988000 93534 0x60000205d9f0 LOG                  basesrc gstbasesrc.c:761:gst_base_src_query_latency:<rtspAppSrc_audio> latency: live 1, min 0:00:00.000000000, max 0:00:00.000000000
0:00:00.678994000 93534 0x60000205d9f0 DEBUG               GST_PADS gstpad.c:4207:gst_pad_query:<rtspAppSrc_audio:src> sent query 0x600002004c80 (latency), result 1
0:00:00.684284000 93534 0x60000205d9f0 LOG                 GST_PADS gstpad.c:5570:gst_pad_push_event_unchecked:<decodebin3-0:sink> sending event 0x6000028192d0 (latency) to peerpad <rtspAppSrc_audio:src>
0:00:00.684287000 93534 0x60000205d9f0 DEBUG              GST_EVENT gstpad.c:5861:gst_pad_send_event_unchecked:<rtspAppSrc_audio:src> have event type latency event: 0x6000028192d0, time 99:99:99.999999999, seq-num 237, GstEventLatency, latency=(guint64)40000000;
0:00:00.684502000 93534 0x60000205d9f0 DEBUG                basesrc gstbasesrc.c:2107:gst_base_src_default_event:<rtspAppSrc_audio> handle event latency event: 0x6000028192d0, time 99:99:99.999999999, seq-num 237, GstEventLatency, latency=(guint64)40000000;
0:00:00.684542000 93534 0x60000205d9f0 DEBUG               GST_PADS gstpad.c:5954:gst_pad_send_event_unchecked:<rtspAppSrc_audio:src> sent event, ret ok
0:00:00.684546000 93534 0x60000205d9f0 LOG                 GST_PADS gstpad.c:5578:gst_pad_push_event_unchecked:<decodebin3-0:sink> sent event 0x6000028192d0 (latency) to peerpad <rtspAppSrc_audio:src>, ret ok
0:00:00.695600000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2286:update_degree:<input> element rtspAppSrc_audio not linked on any sinkpads
0:00:00.695656000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2263:update_degree:<input> change element rtspAppSrc_audio, degree 0->1, linked to decodebin3-0
0:00:00.701559000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2263:update_degree:<input> change element rtspAppSrc_audio, degree 1->0, linked to decodebin3-0
0:00:00.701602000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2146:add_to_queue:<input> adding 'rtspAppSrc_audio' to queue
0:00:00.705071000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2353:gst_bin_sort_iterator_next:<input> queue head gives rtspAppSrc_audio
0:00:00.705095000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2286:update_degree:<input> element rtspAppSrc_audio not linked on any sinkpads
0:00:00.713577000 93534 0x60000205d9f0 INFO              GST_STATES gstbin.c:2484:gst_bin_element_set_state:<rtspAppSrc_audio> current PLAYING pending VOID_PENDING, desired next PLAYING
0:00:00.713636000 93534 0x60000205d9f0 INFO              GST_STATES gstbin.c:2608:gst_bin_element_set_state:<rtspAppSrc_audio> skipping transition from PLAYING to  PLAYING
0:00:00.713683000 93534 0x60000205d9f0 INFO              GST_STATES gstbin.c:2939:gst_bin_change_state_func:<input> child 'rtspAppSrc_audio' changed state to 4(PLAYING) successfully
.
.
.
0:00:01.092291000 93534 0x60000205d9f0 LOG                 GST_PADS gstpad.c:5570:gst_pad_push_event_unchecked:<decodebin3-0:sink> sending event 0x60000281f1e0 (latency) to peerpad <rtspAppSrc_audio:src>
0:00:01.092310000 93534 0x60000205d9f0 DEBUG              GST_EVENT gstpad.c:5861:gst_pad_send_event_unchecked:<rtspAppSrc_audio:src> have event type latency event: 0x60000281f1e0, time 99:99:99.999999999, seq-num 498, GstEventLatency, latency=(guint64)219999998;
0:00:01.092571000 93534 0x60000205d9f0 DEBUG                basesrc gstbasesrc.c:2107:gst_base_src_default_event:<rtspAppSrc_audio> handle event latency event: 0x60000281f1e0, time 99:99:99.999999999, seq-num 498, GstEventLatency, latency=(guint64)219999998;
0:00:01.092621000 93534 0x60000205d9f0 DEBUG               GST_PADS gstpad.c:5954:gst_pad_send_event_unchecked:<rtspAppSrc_audio:src> sent event, ret ok
0:00:01.092704000 93534 0x60000205d9f0 LOG                 GST_PADS gstpad.c:5578:gst_pad_push_event_unchecked:<decodebin3-0:sink> sent event 0x60000281f1e0 (latency) to peerpad <rtspAppSrc_audio:src>, ret ok
0:00:01.153830000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2286:update_degree:<input> element rtspAppSrc_audio not linked on any sinkpads
0:00:01.153839000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2263:update_degree:<input> change element rtspAppSrc_audio, degree 0->1, linked to decodebin3-0
0:00:01.155442000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2263:update_degree:<input> change element rtspAppSrc_audio, degree 1->0, linked to decodebin3-0
0:00:01.155486000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2146:add_to_queue:<input> adding 'rtspAppSrc_audio' to queue
0:00:01.158799000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2353:gst_bin_sort_iterator_next:<input> queue head gives rtspAppSrc_audio
0:00:01.158801000 93534 0x60000205d9f0 DEBUG                    bin gstbin.c:2286:update_degree:<input> element rtspAppSrc_audio not linked on any sinkpads
0:00:01.158818000 93534 0x60000205d9f0 INFO              GST_STATES gstbin.c:2484:gst_bin_element_set_state:<rtspAppSrc_audio> current PLAYING pending VOID_PENDING, desired next PLAYING
0:00:01.158855000 93534 0x60000205d9f0 INFO              GST_STATES gstbin.c:2608:gst_bin_element_set_state:<rtspAppSrc_audio> skipping transition from PLAYING to  PLAYING
0:00:01.158873000 93534 0x60000205d9f0 INFO              GST_STATES gstbin.c:2939:gst_bin_change_state_func:<input> child 'rtspAppSrc_audio' changed state to 4(PLAYING) successfully

Still can’t see what’s the reason that will cause the difference between the two codecs.

BTW: How to enclose the whole logs here?

[slomo]

@braveyao

BTW: How to enclose the whole logs here?

Good question. I guess the easiest would be to upload them on dropbox or similar.

As you probably guessed, some more lines of the log would be useful.

The caps are technically wrong though: it should be AAC-hbr instead of aac-hbr. It’s case sensitive. However, since 1.20 or so, both variants are accepted.

[braveyao]

Hi,

Changing to AAC-hbr moved one step further: it asks for the codec_data config! After I attached “config=(string)1210” from the stream FMTP, no more luck. Same “not-negotiated” reason to the “Internal data stream error”. Lost clue again.

From what I can tell, the current pad caps is:

GST_CAPS gstpad.c:2715:gst_pad_has_current_caps:<aacparse0:src> check current pad caps audio/mpeg, mpegversion=(int)4, stream-format=(string)raw, codec_data=(buffer)1210, framed=(boolean)true, level=(string)2, base-profile=(string)lc, profile=(string)lc, rate=(int)44100, channels=(int)2

This is identical with the output of testing with rtspsrc.
What else of the caps/setting matter then?

Here is the link to the full log of the failure case: gst-log-AAC.txt - Google Drive. Let me know if you can’t access.

Regards

[braveyao]

Ah, finally made it work. Some facts:

• AAC-hbr and aac-hbr both works in my case
• config=(string)1210,indexdeltalength=(string)3,indexlength=(string)3,sizelength=(string)13, it seems that these 4 parameters are must-to-have for M4a. The fmtp attributes should be the source
• Last time when I tried to apply all the capabilities extracted from rtspsrc pads, I didn’t add “(string)” to some numeric values. So they are interpreted as (int) instead.

So remaining question is how to easily find out critical capability configs to a specific codec? Any documents?

@slomo, so many thanks! At least I figured out how to check into gst issues and find out a solution! So grateful!

[slomo]

@braveyao

So remaining question is how to easily find out critical capability configs to a specific codec? Any documents?

You’d either check the caps of the payloader, the SDP or the corresponding RFC. In this case that would be RFC3640. Section 3.3.6 says

In addition to the required MIME format parameters, the following
parameters MUST be present: sizeLength, indexLength, and
indexDeltaLength. AAC frames always have a fixed duration per Access
Unit; when interleaving in this mode, this specific duration MUST be
signaled by the MIME format parameter constantDuration. In addition,
the parameter maxDisplacement MUST be present when interleaving.
[…]
The hexadecimal value of the “config” parameter is the
AudioSpecificConfig(), as defined in ISO/IEC 14496-3.
AudioSpecificConfig() specifies a 5.1 channel AAC stream with a
sampling rate of 48 kHz. For the description of MIME parameters, see
section 4.1.

[braveyao]

That’s also my idea!

Thanks again! Now I’m more comfortable to work with GST!