Webrtc doesn't seem to interprete the sdp (video recvonly, audio sendrecv) properly

General Discussion
1

Hi experts,

We have a camera is streaming both video and audio to the remote p2p connection via ant media server. This camera has a speaker so the remote web app can send audio to it. Hence we defined the web app end to provide the sdp (either offer or anwser) to have video to be recvonly and the audio to be sendrecv. When the camera c code receives this sdp, we expect the camera streams the live video and audio to the web app which is fine. Yet for the audio coming from the web app to the camera, we expected the caps of the newly added webrtc srcpad to be an audio stream. Instead it gives a video stream which we don’t understand.

Any thoughts on this please? We may have done things wrong of course.

Many thanks

2

Just to attach more info:

camera → web offer

2025-04-08 08:25:16 Information [omletWebrtc]: Offer created:
v=0
o=- 6720459800550893366 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE video0 audio1
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:VpOtk5i4nO5ZJI8Cly3tCE7USxTbBQkM
a=ice-pwd:4hcX+wbL4eJasQwKuZN4PhbScbDvNLC6
a=rtcp-mux
a=rtcp-rsize
a=sendrecv
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 transport-cc
a=framerate:25
a=ssrc:3585540792 msid:user1075587677@host-8d87616d webrtctransceiver0
a=ssrc:3585540792 cname:user1075587677@host-8d87616d
a=mid:video0
a=fingerprint:sha-256 83:F9:3E:D7:4A:4E:C1:E2:FB:3E:C0:7A:6C:FC:63:92:CC:CC:A9:F7:75:BA:AA:81:C9:05:AA:D3:89:48:8F:A4
a=rtcp-mux-only
m=audio 0 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:VpOtk5i4nO5ZJI8Cly3tCE7USxTbBQkM
a=ice-pwd:4hcX+wbL4eJasQwKuZN4PhbScbDvNLC6
a=bundle-only
a=rtcp-mux
a=rtcp-rsize
a=sendrecv
a=rtpmap:96 OPUS/48000/2
a=rtcp-fb:96 transport-cc
a=fmtp:96 sprop-stereo=0;sprop-maxcapturerate=48000
a=ssrc:3946784428 msid:user1075587677@host-8d87616d webrtctransceiver1
a=ssrc:3946784428 cname:user1075587677@host-8d87616d
a=mid:audio1
a=fingerprint:sha-256 83:F9:3E:D7:4A:4E:C1:E2:FB:3E:C0:7A:6C:FC:63:92:CC:CC:A9:F7:75:BA:AA:81:C9:05:AA:D3:89:48:8F:A4
a=rtcp-mux-only

3

web → camera anwser
{
“streamId”: “jimtest7”,
“linkSession”: null,
“subscriberId”: “”,
“type”: “answer”,
“command”: “takeConfiguration”,
“sdp”: “v=0”,
“o=- 6030414970636399316 2 IN IP4 127.0.0.1”,
“s=-”,
“t=0 0”,
“a=group:BUNDLE video0 audio1”,
“a=msid-semantic: WMS 8d0625e7-3ff4-43bd-ac54-461736eab429”,
“m=video 9 UDP/TLS/RTP/SAVPF 96”,
“c=IN IP4 0.0.0.0”,
“a=rtcp:9 IN IP4 0.0.0.0”,
“a=ice-ufrag:4qYK”,
“a=ice-pwd:Lp3MwtES/TJI0tO4FeKzvGfS”,
“a=ice-options:trickle”,
“a=fingerprint:sha-256 62:4E:4A:7E:0E:63:2D:20:73:A2:61:5B:3C:22:AD:DF:1F:E1:1A:2D:26:C4:B6:B3:A0:9E:6C:F9:D4:9A:E1:69”,
“a=setup:active”,
“a=mid:video0”,
“a=recvonly”,
“a=rtcp-mux”,
“a=rtcp-rsize”,
“a=rtpmap:96 H264/90000”,
“a=rtcp-fb:96 transport-cc”,
“a=rtcp-fb:96 ccm fir”,
“a=rtcp-fb:96 nack pli”,
“a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f”,
“m=audio 9 UDP/TLS/RTP/SAVPF 96”,
“c=IN IP4 0.0.0.0”,
“a=rtcp:9 IN IP4 0.0.0.0”,
“a=ice-ufrag:4qYK”,
“a=ice-pwd:Lp3MwtES/TJI0tO4FeKzvGfS”,
“a=ice-options:trickle”,
“a=fingerprint:sha-256 62:4E:4A:7E:0E:63:2D:20:73:A2:61:5B:3C:22:AD:DF:1F:E1:1A:2D:26:C4:B6:B3:A0:9E:6C:F9:D4:9A:E1:69”,
“a=setup:active”,
“a=mid:audio1”,
“a=sendrecv”,
“a=rtcp-mux”,
“a=rtcp-rsize”,
“a=rtpmap:96 OPUS/48000/2”,
“a=rtcp-fb:96 transport-cc”,
“a=fmtp:96 minptime=10;useinbandfec=1; stereo=1”,
“a=ssrc:416345079 cname:DI8YBGxdrp77kusx”,
“a=ssrc:416345079 msid:8d0625e7-3ff4-43bd-ac54-461736eab429 204307fa-9b6c-408b-8c45-2b3c617fd452”,
“”
}

4

Here is the GST_DEBUG=4 log (partial) to show the wrong pad for webrtc is created:

0:00:23.202844512 3288 0x61ba2a10 INFO GST_PADS gstpad.c:2618:gst_pad_link_full: linked rtpbin:recv_rtp_src_0_766230453_96 and src_0:proxypad26, successful
0:00:23.202928509 3288 0x61ba2a10 INFO GST_EVENT gstevent.c:1687:gst_event_new_reconfigure: creating reconfigure event
0:00:23.203506486 3288 0x61ba2a10 INFO GST_ELEMENT_PADS gstelement.c:758:gst_element_add_pad: adding pad ‘src_0’
[09:46:34.298] ==== returned caps: application/x-rtp, media=(string)video, payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)0, profile=(string)constrained-baseline, rtcp-fb-transport-cc=(boolean)true, rtcp-fb-ccm-fir=(boolean)true, rtcp-fb-nack-pli=(boolean)true, ssrc=(uint)766230453 pad: src_0
0:00:23.205147755 3288 0x6c27bdb8 INFO h264parse gsth264parse.c:3845:gst_h264_parse_src_event:<h264_parse> received upstream force-key-unit event, seqnum 561 running_time 99:99:99.999999999 all_headers 0 count 0
2025-04-08 09:46:34 Information [omletWebrtc]: on_incoming_stream - --------------------------video stream received ----------------------------------
0:00:23.222501077 3288 0x6c27bdb8 INFO gstrtpfunnel gstrtpfunnel.c:488:gst_rtp_funnel_src_event:rtpfunnel0:src Sending custom-upstream event: 0x61bb27a8, time 99:99:99.999999999, seq-num 567, GstForceKeyUnit, ssrc=(uint)2664031496, all-headers=(boolean)false; to rtpfunnel0:sink_0
0:00:23.224545664 3288 0x6c27bdb8 INFO h264parse gsth264parse.c:3845:gst_h264_parse_src_event:<h264_parse> received upstream force-key-unit event, seqnum 567 running_time 99:99:99.999999999 all_headers 0 count 0
0:00:23.224929649 3288 0x61ba2a10 INFO GST_ELEMENT_FACTORY gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element “fakesink”
0:00:23.231513392 3288 0x61ba2a10 INFO GST_ELEMENT_PADS gstelement.c:758:gst_element_add_pad:GstBaseSink@0x71f87e68 adding pad ‘sink’
0:00:23.232808008 3288 0x61ba2a10 INFO GST_ELEMENT_PADS gstelement.c:1016:gst_element_get_static_pad: found pad fakesink0:sink
0:00:23.237177837 3288 0x61ba2a10 INFO GST_PADS gstpad.c:2437:gst_pad_link_prepare: trying to link jimtest7:src_0 and fakesink0:sink
0:00:23.239324420 3288 0x61ba2a10 INFO GST_PADS gstpad.c:2424:gst_pad_link_check_relations: pads have wrong hierarchy
0:00:23.239927730 3288 0x61ba2a10 INFO GST_PADS gstpad.c:2560:gst_pad_link_full: link between jimtest7:src_0 and fakesink0:sink failed: wrong hierarchy
2025-04-08 09:46:34 Warning [omletWebrtc]: on_incoming_stream - Failed to link pad (ret=-1)
0:00:23.261808208 3288 0x61ba2a10 INFO GST_PADS gstpad.c:4390:gst_pad_peer_query:jimtest7:src_0 pad has no peer
0:00:23.263133490 3288 0x61ba2a10 INFO GST_PADS gstpad.c:4390:gst_pad_peer_query:jimtest7:src_0 pad has no peer
0:00:23.265298739 3288 0x61ba2a10 INFO GST_PADS gstpad.c:4390:gst_pad_peer_query:jimtest7:src_0 pad has no peer
0:00:23.265704056 3288 0x61ba2a10 INFO task gsttask.c:368:gst_task_func:rtpjitterbuffer0:src Task going to paused
0:00:23.266669352 3288 0x6c27bdb8 INFO task gsttask.c:368:gst_task_func:queue2:src Task going to paused
0:00:23.294210276 3288 0x61b57cf8 INFO basesrc gstbasesrc.c:3042:gst_base_src_loop: pausing after gst_pad_push() = not-linked
0:00:23.294354937 3288 0x61b57cf8 WARN basesrc gstbasesrc.c:3132:gst_base_src_loop: error: Internal data stream error.
0:00:23.294408268 3288 0x61b57cf8 WARN basesrc gstbasesrc.c:3132:gst_base_src_loop: error: streaming stopped, reason not-linked (-1)
0:00:23.294554263 3288 0x61b57cf8 INFO GST_ERROR_SYSTEM gstelement.c:2281:gst_element_message_full_with_details: posting message: Internal data stream error.
2025-04-08 09:46:34 Warning [media]: OnMessage - ERROR from element nicesrc0: “Internal data stream error.”, code: 1, (GST: 1994667532).
0:00:23.307877742 3288 0x61b57cf8 INFO GST_ERROR_SYSTEM gstelement.c:2308:gst_element_message_full_with_details: posted error message: Internal data stream error.
0:00:23.309597008 3288 0x61b57cf8 WARN queue gstqueue.c:992:gst_queue_handle_sink_event: error: Internal data stream error.
0:00:23.311328274 3288 0x61b57cf8 WARN queue gstqueue.c:992:gst_queue_handle_sink_event: error: streaming stopped, reason not-linked (-1)
0:00:23.315905429 3288 0x61b57cf8 INFO GST_ERROR_SYSTEM gstelement.c:2281:gst_element_message_full_with_details: posting message: Internal data stream error.
2025-04-08 09:46:34 Warning [media]: OnMessage - ERROR from element queue2: “Internal data stream error.”, code: 1, (GST: 1994667532).
0:00:23.329492898 3288 0x61b57cf8 INFO GST_ERROR_SYSTEM gstelement.c:2308:gst_element_message_full_with_details: posted error message: Internal data stream error.
0:00:23.331289494 3288 0x61b57cf8 INFO task gsttask.c:368:gst_task_func:nicesrc0:src Task going to paused
0:00:25.653678047 3288 0x6c201058 INFO baseparse gstbaseparse.c:4108:gst_base_parse_set_latency:<h264_parse> min/max latency 0:00:00.000000000, 0:00:00.000000000
0:00:25.673422609 3288 0x61b97a10 INFO GST_EVENT gstevent.c:918:gst_event_new_caps: creating caps event application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, sprop-parameter-sets=(string)“J00AM+dAPAET8s1AQEB8AAADAAQAAAMAyMgAA2/gAAUn03//AoA=,KO48gA==”, payload=(int)96, seqnum-offset=(uint)30045, timestamp-offset=(uint)702313346, ssrc=(uint)2664031496, a-framerate=(string)25
0:00:25.679146386 3288 0x61b97a10 INFO rtpulpfecenc gstrtpulpfecenc.c:683:gst_rtp_ulpfec_enc_event_sink: TWCC extension ID: 0

I can attach the full log if needed.

Thanks in advance!

5

Your pipeline is invalid. You are configuring both VP8 encoding and OPUS encoding for payload 96. You must have a different payload identifier for each distinct format sent over a single RTP session.

1 Like
6

Thanks @ystreet00 for the quick response and excellent spot! I’ll check that!

7

Hi @ystreet00, hmm…we do have separete rtps for the video (H264) and audio (OPUS) in the pipeline on the camera end. The offer (above) is generated by the gstreamer webrtcbin. So I don’t understand why it is using the same payload type? I guess that causes the other end (web ant media javascript webrtc adaptor or chrome webrtc component) to anwser the same payload number back?

8

Ah, OK. I think @ystreet00 you are absolutely right! We didn’t specify the pt for the rtps. Once I set the video pt=96 and the audio pt=97, it starts working! Thanks a lot!