Webrtcbin with custom signaling doesn't produce video

anonymix007 · July 19, 2024, 11:48am

The application is heavily based on the sendrecv example.
Client (browser in this case) asks the server to provide ICE servers via /iceServers POST, then proceeds to the creation of peer connection using /createPeerConnection POST with the SDP offer. It appears to receive an SDP answer (though no field looks odd as it seems to be the same as in the offer itself). Then, upon button press, video is requested via /startVideo POST.
Both media tracks (audio and video) seem to be created, but there’s nothing on the canvas. Data channel doesn’t start as well.
These are the logs with GST_DEBUG=3:

0:00:08.184445493 734765 0x737a88000b90 FIXME                default gstutils.c:4088:gst_element_decorate_stream_id_internal:<nicesrc0> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:08.184472767 734765 0x737a88000b90 WARN                GST_PADS gstpad.c:4392:gst_pad_peer_query:<nicesrc0:src> could not send sticky events
0:00:08.184877634 734765 0x737a88000db0 FIXME                default gstutils.c:4088:gst_element_decorate_stream_id_internal:<nicesrc1> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:08.184900130 734765 0x737a88000db0 WARN                GST_PADS gstpad.c:4392:gst_pad_peer_query:<nicesrc1:src> could not send sticky events

(wh:734765): GStreamer-WARNING **: 14:25:36.995: Promise result isn't PENDING or INTERRUPTED
0:00:08.185506635 734765 0x737a88000fd0 FIXME                default gstutils.c:4088:gst_element_decorate_stream_id_internal:<nicesrc2> Creating random stream-id, consider implementing a deterministic way of creating a stream-id

(wh:734765): GStreamer-CRITICAL **: 14:25:36.995: gst_mini_object_unref: assertion 'GST_MINI_OBJECT_REFCOUNT_VALUE (mini_object) > 0' failed
0:00:08.185522602 734765 0x737a88000fd0 WARN                GST_PADS gstpad.c:4392:gst_pad_peer_query:<nicesrc2:src> could not send sticky events
Sending answer
ICE gathering state changed to gathering
ICE gathering state changed to complete
0:00:08.389725753 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin2> did not really configure latency of 0:00:00.000000000
0:00:08.389762305 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin1> did not really configure latency of 0:00:00.000000000
0:00:08.389787710 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin0> did not really configure latency of 0:00:00.000000000
0:00:08.390133940 734765 0x737a88002590 FIXME                default gstutils.c:4088:gst_element_decorate_stream_id_internal:<audiotestsrc0> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:08.390261944 734765 0x737a88002c20 FIXME                default gstutils.c:4088:gst_element_decorate_stream_id_internal:<ximagesrc0> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:08.390291867 734765 0x737aa0081a10 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin2> did not really configure latency of 0:00:00.000000000
0:00:08.390316482 734765 0x737aa0081a10 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin1> did not really configure latency of 0:00:00.000000000
0:00:08.390335839 734765 0x737aa0081a10 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin0> did not really configure latency of 0:00:00.000000000
0:00:08.390599995 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin2> did not really configure latency of 0:00:00.000000000
0:00:08.390634998 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin1> did not really configure latency of 0:00:00.000000000
0:00:08.390669411 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin0> did not really configure latency of 0:00:00.000000000
0:00:08.390707083 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin2> did not really configure latency of 0:00:00.000000000
0:00:08.390725079 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin1> did not really configure latency of 0:00:00.000000000
0:00:08.390741576 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin0> did not really configure latency of 0:00:00.000000000
0:00:08.391306141 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin2> did not really configure latency of 0:00:00.000000000
0:00:08.391338175 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin1> did not really configure latency of 0:00:00.000000000
0:00:08.391374017 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin0> did not really configure latency of 0:00:00.000000000
0:00:08.391407780 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin2> did not really configure latency of 0:00:00.000000000
0:00:08.391426347 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin1> did not really configure latency of 0:00:00.000000000
0:00:08.391442993 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin0> did not really configure latency of 0:00:00.000000000
0:00:08.391791572 734765 0x737a88002130 WARN              rtprtxsend gstrtprtxsend.c:942:gst_rtp_rtx_send_sink_event:<rtprtxsend0> Payload 96 not in rtx-pt-map
0:00:08.391883364 734765 0x737a88002130 WARN              rtpsession gstrtpsession.c:2484:gst_rtp_session_chain_send_rtp_common:<rtpsession0> Can't determine running time for this packet without knowing configured latency
0:00:08.395822603 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin2> did not really configure latency of 0:00:00.000000000
0:00:08.395852877 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin1> did not really configure latency of 0:00:00.000000000
0:00:08.395927592 734765 0x56934cfe4120 WARN    webrtctransportsendbin transportsendbin.c:457:gst_transport_send_bin_element_query:<transportsendbin0> did not really configure latency of 0:00:00.000000000
0:00:08.423461557 734765 0x737a880027c0 WARN              rtprtxsend gstrtprtxsend.c:942:gst_rtp_rtx_send_sink_event:<rtprtxsend1> Payload 96 not in rtx-pt-map
0:00:08.423539381 734765 0x737a880027c0 WARN              rtpsession gstrtpsession.c:2484:gst_rtp_session_chain_send_rtp_common:<rtpsession1> Can't determine running time for this packet without knowing configured latency

The full code is available here with build instructions in the README.

What is done wrong here and how to fix it?

ystreet00 · July 20, 2024, 12:53am

It’s hard to say what is wrong. You can try a couple of things.

Use VP8 instead of H264 which has less configuration requirements in browsers.
Look at the negotiated SDPs and ice candidates and see if they are as expected.
Try a test using webrtcsink to ensure that WebRTC as a whole does at least work for you with your specific browser.

anonymix007 · July 22, 2024, 5:56pm

No change
Offer:

{
  "sdp": "v=0\r\no=- 7702972628904207480 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:Lw8z\r\na=ice-pwd:2fa8p9m2mv9JbM72baGkSk7d\r\na=ice-options:trickle\r\na=fingerprint:sha-256 1D:D7:75:9E:06:E8:04:96:37:08:1C:4C:E6:63:D4:51:9F:E5:C6:63:A6:23:E0:7F:6F:24:D3:AF:36:C0:86:B6\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:63 red/48000/2\r\na=fmtp:63 111/111\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:126 telephone-event/8000\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 35 36 37 38 102 103 104 105 106 107 108 109 127 125 39 40 41 42 43 44 45 46 47 48 112 113 114 49\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:Lw8z\r\na=ice-pwd:2fa8p9m2mv9JbM72baGkSk7d\r\na=ice-options:trickle\r\na=fingerprint:sha-256 1D:D7:75:9E:06:E8:04:96:37:08:1C:4C:E6:63:D4:51:9F:E5:C6:63:A6:23:E0:7F:6F:24:D3:AF:36:C0:86:B6\r\na=setup:actpass\r\na=mid:1\r\na=extmap:14 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:13 urn:3gpp:video-orientation\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:100 VP9/90000\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=fmtp:100 profile-id=2\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:35 VP9/90000\r\na=rtcp-fb:35 goog-remb\r\na=rtcp-fb:35 transport-cc\r\na=rtcp-fb:35 ccm fir\r\na=rtcp-fb:35 nack\r\na=rtcp-fb:35 nack pli\r\na=fmtp:35 profile-id=1\r\na=rtpmap:36 rtx/90000\r\na=fmtp:36 apt=35\r\na=rtpmap:37 VP9/90000\r\na=rtcp-fb:37 goog-remb\r\na=rtcp-fb:37 transport-cc\r\na=rtcp-fb:37 ccm fir\r\na=rtcp-fb:37 nack\r\na=rtcp-fb:37 nack pli\r\na=fmtp:37 profile-id=3\r\na=rtpmap:38 rtx/90000\r\na=fmtp:38 apt=37\r\na=rtpmap:102 H264/90000\r\na=rtcp-fb:102 goog-remb\r\na=rtcp-fb:102 transport-cc\r\na=rtcp-fb:102 ccm fir\r\na=rtcp-fb:102 nack\r\na=rtcp-fb:102 nack pli\r\na=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f\r\na=rtpmap:103 rtx/90000\r\na=fmtp:103 apt=102\r\na=rtpmap:104 H264/90000\r\na=rtcp-fb:104 goog-remb\r\na=rtcp-fb:104 transport-cc\r\na=rtcp-fb:104 ccm fir\r\na=rtcp-fb:104 nack\r\na=rtcp-fb:104 nack pli\r\na=fmtp:104 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f\r\na=rtpmap:105 rtx/90000\r\na=fmtp:105 apt=104\r\na=rtpmap:106 H264/90000\r\na=rtcp-fb:106 goog-remb\r\na=rtcp-fb:106 transport-cc\r\na=rtcp-fb:106 ccm fir\r\na=rtcp-fb:106 nack\r\na=rtcp-fb:106 nack pli\r\na=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\r\na=rtpmap:107 rtx/90000\r\na=fmtp:107 apt=106\r\na=rtpmap:108 H264/90000\r\na=rtcp-fb:108 goog-remb\r\na=rtcp-fb:108 transport-cc\r\na=rtcp-fb:108 ccm fir\r\na=rtcp-fb:108 nack\r\na=rtcp-fb:108 nack pli\r\na=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f\r\na=rtpmap:109 rtx/90000\r\na=fmtp:109 apt=108\r\na=rtpmap:127 H264/90000\r\na=rtcp-fb:127 goog-remb\r\na=rtcp-fb:127 transport-cc\r\na=rtcp-fb:127 ccm fir\r\na=rtcp-fb:127 nack\r\na=rtcp-fb:127 nack pli\r\na=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f\r\na=rtpmap:125 rtx/90000\r\na=fmtp:125 apt=127\r\na=rtpmap:39 H264/90000\r\na=rtcp-fb:39 goog-remb\r\na=rtcp-fb:39 transport-cc\r\na=rtcp-fb:39 ccm fir\r\na=rtcp-fb:39 nack\r\na=rtcp-fb:39 nack pli\r\na=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f\r\na=rtpmap:40 rtx/90000\r\na=fmtp:40 apt=39\r\na=rtpmap:41 H264/90000\r\na=rtcp-fb:41 goog-remb\r\na=rtcp-fb:41 transport-cc\r\na=rtcp-fb:41 ccm fir\r\na=rtcp-fb:41 nack\r\na=rtcp-fb:41 nack pli\r\na=fmtp:41 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=f4001f\r\na=rtpmap:42 rtx/90000\r\na=fmtp:42 apt=41\r\na=rtpmap:43 H264/90000\r\na=rtcp-fb:43 goog-remb\r\na=rtcp-fb:43 transport-cc\r\na=rtcp-fb:43 ccm fir\r\na=rtcp-fb:43 nack\r\na=rtcp-fb:43 nack pli\r\na=fmtp:43 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=f4001f\r\na=rtpmap:44 rtx/90000\r\na=fmtp:44 apt=43\r\na=rtpmap:45 AV1/90000\r\na=rtcp-fb:45 goog-remb\r\na=rtcp-fb:45 transport-cc\r\na=rtcp-fb:45 ccm fir\r\na=rtcp-fb:45 nack\r\na=rtcp-fb:45 nack pli\r\na=fmtp:45 level-idx=5;profile=0;tier=0\r\na=rtpmap:46 rtx/90000\r\na=fmtp:46 apt=45\r\na=rtpmap:47 AV1/90000\r\na=rtcp-fb:47 goog-remb\r\na=rtcp-fb:47 transport-cc\r\na=rtcp-fb:47 ccm fir\r\na=rtcp-fb:47 nack\r\na=rtcp-fb:47 nack pli\r\na=fmtp:47 level-idx=5;profile=1;tier=0\r\na=rtpmap:48 rtx/90000\r\na=fmtp:48 apt=47\r\na=rtpmap:112 red/90000\r\na=rtpmap:113 rtx/90000\r\na=fmtp:113 apt=112\r\na=rtpmap:114 ulpfec/90000\r\na=rtpmap:49 flexfec-03/90000\r\na=rtcp-fb:49 goog-remb\r\na=rtcp-fb:49 transport-cc\r\na=fmtp:49 repair-window=10000000\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:Lw8z\r\na=ice-pwd:2fa8p9m2mv9JbM72baGkSk7d\r\na=ice-options:trickle\r\na=fingerprint:sha-256 1D:D7:75:9E:06:E8:04:96:37:08:1C:4C:E6:63:D4:51:9F:E5:C6:63:A6:23:E0:7F:6F:24:D3:AF:36:C0:86:B6\r\na=setup:actpass\r\na=mid:2\r\na=sctp-port:5000\r\na=max-message-size:262144\r\n",
  "type": "offer"
}

Answer:

{
  "sdp": "v=0\r\no=- 7702972628904207480 2 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:UurFq7YGPNkVDr1RTiralI9UV5VO2DEa\r\na=ice-pwd:2OVmroiBK5wkQ9O6FV0iWwp7KKNIT9SQ\r\na=mid:0\r\na=rtcp-mux\r\na=setup:active\r\na=rtpmap:111 OPUS/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=sendonly\r\na=fingerprint:sha-256 29:13:FF:63:17:F4:5B:E2:D3:70:C9:2B:7F:19:89:A2:14:2E:51:42:DD:F5:BC:D6:FE:0F:4A:8F:EC:47:3A:32\r\nm=video 9 UDP/TLS/RTP/SAVPF 96\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:UurFq7YGPNkVDr1RTiralI9UV5VO2DEa\r\na=ice-pwd:2OVmroiBK5wkQ9O6FV0iWwp7KKNIT9SQ\r\na=mid:1\r\na=rtcp-mux\r\na=setup:active\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 nack pli\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 transport-cc\r\na=sendonly\r\na=fingerprint:sha-256 29:13:FF:63:17:F4:5B:E2:D3:70:C9:2B:7F:19:89:A2:14:2E:51:42:DD:F5:BC:D6:FE:0F:4A:8F:EC:47:3A:32\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:UurFq7YGPNkVDr1RTiralI9UV5VO2DEa\r\na=ice-pwd:2OVmroiBK5wkQ9O6FV0iWwp7KKNIT9SQ\r\na=mid:2\r\na=setup:active\r\na=sctp-port:5000\r\na=fingerprint:sha-256 29:13:FF:63:17:F4:5B:E2:D3:70:C9:2B:7F:19:89:A2:14:2E:51:42:DD:F5:BC:D6:FE:0F:4A:8F:EC:47:3A:32\r\n",
  "type": "answer"
}

Candidate:

"candidate:1382198187 1 udp 1677729535 xx.xx.xx.xx 1239 typ srflx raddr 0.0.0.0 rport 0 generation 0 ufrag Lw8z network-cost 999"

Not sure whether any of those are correct though. IP address (masked with xx.xx.xx.xx) seems to be correct (not just localhost/127.0.0.1 as in SDP).
3. Seems to work just fine.