Send screencapture data from Pipewire to WebRTC

Application Development
1

I’m attempting to stream a Freedesktop ScreenCast capture over WebRTC via GStreamer using its pipewiresrc and webrtcbin elements.

I’ve made a decent amount of progress so far: I can stream the videotestsrc with my current configuration, and I can also stream a screen capture when using the higher level webrtcsink element. Finally, when I run my current configuration with the pipewiresrc, I do actually get some stream data: usually I only get the first frame before the stream freezes, but very occasionally, seemingly at random, do I get a very slow, very choppy stream.

Here’s my current GStreamer pipeline:

pipewiresrc path={self.pipewire_node_id} do-timestamp=true
! videoconvert
! videoscale
! videorate
! video/x-raw,format=I420,width=2560,height=1600,framerate=30/1
! vp8enc
! rtpvp8pay pt=96 mtu=1200
! application/x-rtp,media=video,encoding-name=VP8,payload=96
! webrtcbin name=webrtc stun-server=stun://stun.l.google.com:19302

And here’s the GStreamer debug output when I run my code at log level 2:

0:00:04.468006339  3556 0x7992ec000db0 WARN                GST_PADS gstpad.c:4392:gst_pad_peer_query:<nicesrc0:src> could not send sticky events
0:00:05.388609402  3556 0x7992ec000b70 WARN              rtprtxsend gstrtprtxsend.c:942:gst_rtp_rtx_send_sink_event:<rtprtxsend0> Payload 96 not in rtx-pt-map
0:00:05.388847235  3556 0x7992ec000b70 WARN              rtpsession gstrtpsession.c:2484:gst_rtp_session_chain_send_rtp_common:<rtpsession0> Can't determine running time for this packet without knowing configured latency

I’m thinking it’s a timing related thing since depending on how I tweak my pipeline I get various timing related errors, but I’m new to GStreamer so that’s just an intuition.

I’m running all of this with Python via PyGObject. I’ve tried just about everything I can imagine, or find online, so please feel free to ask for more information.

2

Do I understand correctly that the same pipeline works with webrtcsink?. One might get more information via debug logs with GST_DEBUG or capturing pipeline dot graphs. See Debugging tools.

For testing pipewiresrc, using WHIP with a pipeline like below in Python, works locally here.

pipewiresrc fd={fd} path={node_id} ! queue ! videoconvert ! queue ! whipclientsink name=ws video-caps=video/x-h264 signaller::whip-endpoint="http://127.0.0.1:8889/streamtest/whip"
3

Sanchayan, thanks for your help.

With webrtcsink I was able to get it working with a much simpler pipeline even, here’s what was confirmed working:

gst-launch-1.0 pipewiresrc path={node_id} ! webrtcsink run-signalling-server=true run-web-server=true

I have used the debugging logs, the only logs returned are in the original post, if that’s at all helpful.

I did capture some dot graphs, both of my pipeline, and the webrtcsink pipeline, and then attempted to replicated the known working webrtcsink pipeline in my own setup, but I didn’t have any success.

I tried to embed some of the pipeline graphs I captured, but since my account is new I can only upload one image, so I put the graphs in an Imgur album: https://imgur.com/a/hNFqQsp

The album contains a graph of my non-working webrtcbin pipeline, and a few different graphs from the working webrtcsink pipeline.

The webrtcsink pipeline graphs are definitely a bit over my head so it’s definitely possible I just missed something that should be obvious. My current thought process is that because webrtcsink internally wraps webrtcbin, then it’s obviously possible to get pipewiresrc and webrtcbin working on my own as well.

I have verified that pipewiresrc is working both with autovideosink and with webrtcsink.

If I had to guess, I think the issue is with the VP8 encoding on the pipewiresrc, but that’s just conjecture on my part.

Thank you for taking the time to reply, hopefully this extra information will make it easier for you to help me.

4

The webrtcsink pipeline is also running a signalling server. What is the signalling setup for webrtcbin case?

If you suspect VP8 or negotiation to be an issue, verbose logs with GST_DEBUG=3,webrtcbin:6 or such might point to the problems during negotiation.

Any reason for specifically wanting to use webrtcbin when webrtcsink works? webrtcsink/webrtcsrc were written to simplify the handling around webrtcbin when the use case allows.

5

Yes, I’m aware that it’s running a signalling server. It’s not possible to use webrtcbinwithout one, therefore I’ve implemented my own that is verified working when I test with videotestsrc.

I’m using webrtcbinas this is part of a larger piece of software with some very stringent regulatory requirements, and the extra granularity/flexibility is important.

I’m fairly certain that the rest of my setup is working fine, as my current code works fine with videotestsrc, pipewiresrcworks fine with webrtcsink, and my only problem is combining the two.

I’m really fairly confident that it’s just my pipeline that isn’t quite right, as this is really the only piece of the equation that I don’t have experience with, so I’m hoping someone can help me debug the pipeline. Surely there must be a way to verify each piece of the pipeline is getting the data it’s expecting to.

Thanks again for replying, I appreciate anyone’s insight.

6

Can you share the logs using pastebin or such by running the pipeline with GST_DEBUG=3,webrtcbin:6?

7

Sanchayan,

Thanks for your continued help, I really appreciate your time and effort.

Here’s the log produced from running with the debug settings you suggested: GStreamer pipwiresrc to webrtcbin pipeline logs - Pastebin.com

To recap, when I run my code, I do actually get some stream, usually the first frame of the screencast. However, it doesn’t update beyond that.

Additionally, here’s a screenshot of the Firefox WebRTC statistics when running my code:

screenshot
screenshot2469×1487 254 KB

8

ICE gathering was completed but seems there was a sender timeout.

You probably want to use deadline=1 with vp8enc.

Would have to look at logs upstream to webrtcbin. Is pipewiresrc pushing buffers? Would have to look at debug logs for something like GST_DEBUG=3,pipewiresrc:6,rtpvp8pay:6. Do things change if the explicit caps video/x-raw,format=I420,width=2560,height=1600,framerate=30/1 are removed, so the pipeline to check.

pipewiresrc path={self.pipewire_node_id} do-timestamp=true
! videoconvert
! vp8enc deadline=1
! rtpvp8pay pt=96
! application/x-rtp,media=video,encoding-name=VP8,payload=96
! webrtcbin name=webrtc stun-server=stun://stun.l.google.com:19302

Difficult to check further without a sample reproducer.

1 Like
9

Thanks for the suggestion, I believe that’s getting me somewhere!

After running the pipeline you suggested with the debug options, I can see what I believe is a timestamp issue with the buffers vp8encis getting from pipewiresrc. I’m getting these lines repeatedly in the logs:

0:00:06.735064785 16093 0x7e610c000b70 LOG              pipewiresrc gstpipewiresrc.c:571:dequeue_buffer:<pipewiresrc0> got new buffer 0x7e61040040f0
0:00:06.735070249 16093 0x7e610c000b70 LOG              pipewiresrc gstpipewiresrc.c:581:dequeue_buffer:<pipewiresrc0> pts 21897571245484, dts_offset 0
0:00:06.735084461 16093 0x7e610c000b70 LOG              pipewiresrc gstpipewiresrc.c:1239:gst_pipewire_src_create:<pipewiresrc0> popped buffer 0x7e61100c69f0
0:00:06.735093177 16093 0x7e610c000b70 LOG              pipewiresrc gstpipewiresrc.c:1174:gst_pipewire_src_get_times:<pipewiresrc0> start 6:04:57.571245484 (21897571245484), end 99:99:99.999999999 (18446744073709551615)
0:00:06.750396980 16093 0x7e610c000b70 LOG              pipewiresrc gstpipewiresrc.c:483:buffer_recycle:<pipewiresrc0> recycle buffer 0x7e61040040f0
0:00:06.750429399 16093 0x7e610c000b70 WARN                  vpxenc gstvpxenc.c:2267:gst_vpx_enc_handle_frame:<vp8enc0> decreasing pts 0:00:02.466624298 previous buffer was 5124089:29:38.604930445 enforce increasing pts
0:00:06.773944547 16093 0x7e610c000b70 WARN                  vpxenc gstvpxenc.c:2028:gst_vpx_enc_process:<vp8enc0> vpx pts 1660204996074443 does not match input frames, discarding

So I’m assuming if I can fix the timing issues the stream will start working? It seems pretty clear from these logs that the subsequent frames are being dropped by vp8enc.

I feel like I’m really close, is there something I can add to my pipeline to resolve the timing issues, or am I misunderstanding the problem here?

10

That definitely points to buffers being dropped by vp8enc.

What is the PipeWire version? There were improvements done in the GStreamer PipeWire elements with the last few releases.

What is the behavior if do-timestamp is not set on pipewiresrc? Do run the pipeline with GST_DEBUG=3,pipewiresrc:6,vp8enc:6,rtpvp8pay:6.

1 Like
11

I’m using the version of PipeWire in Ubuntu 24.04, version 1.0.5. At your suggestion I compiled and installed the latest version, 1.4.8, and miraculously, it worked! I had to remove do-timestampin order for it to work. Previously, the behavior didn’t seem to change regardless of whether do-timestamp was set or not.

While this is an acceptable solution, and there is a Ubuntu PPA providing later PipeWire versions, the programmer in me is still wanting to know why it wasn’t working before. One thing I noticed is the timestamps in the logs changed. Previously, the pipewiresrc timestamps had 99:99:99.999999999 as the end point, like so:

0:00:21.690293228 10824 0x7ddf7c000db0 LOG pipewiresrc gstpipewiresrc.c:1174:gst_pipewire_src_get_times:<pipewiresrc0> start 1:37:14.659300355 (5834659300355), end 99:99:99.999999999

When I run the log with PipeWire 1.4.8, the timestamps look more like one would expect, with a valid start and end timestamp:

0:00:03.265656909  3548 0x70a18c000b70 LOG              pipewiresrc gstpipewiresrc.c:1320:gst_pipewire_src_get_times:<pipewiresrc0> start 0:40:20.321222522 (2420321222522), end 0:40:20.338171674 (2420338171674)

Does this indicate something I could have done to make my pipeline work with PipeWire 1.0.5? I totally understand if you don’t want to deal with this issue any more, I have really appreciated your help!

12

So looks like the buffers were not timestamped correctly with the older version.

We did quite a few fixes and improvements for the GStreamer PipeWire elements. Even if somehow the timestamp could be compensated for, there are others. Would recommend the latest PipeWire/elements.