GStreamer and PipeWire

mikesmith · November 16, 2024, 7:59am

Has anyone gotten GStreamer to stream video to PipeWire? Is that possible and if so has anyone ever successfully done so?

Currently the best I can do is this script:

gphoto2 --stdout --capture-movie | gst-launch-1.0 fdsrc ! decodebin3 name=dec ! queue ! videoconvert ! v4l2sink device=/dev/video1

This works but obviously doesn’t use PipeWire.

I’ve tried this:

gphoto2 --stdout --capture-movie | gst-launch-1.0 fdsrc ! decodebin3 name=dec ! queue ! videoconvert ! pipewiresink

But that doesn’t work. I’m using Fedora KDE if that matters. Is there any kind of how to guide for using Gstreamer? Most of what I’ve need has been more programming centric not end user centric.

Do I need to set something up in PipeWire before being able to stream to it?

I’ve done:

pw-mon | grep /dev/video
                api.v4l2.path = "/dev/video1"
                object.path = "v4l2:/dev/video1"
                api.v4l2.path = "/dev/video2"
                object.path = "v4l2:/dev/video2"
                api.v4l2.path = "/dev/video0"
                object.path = "v4l2:/dev/video0"
              String "/dev/video1"
              String "/dev/video1"
                api.v4l2.path = "/dev/video1"
                object.path = "v4l2:/dev/video1"
                object.path = "v4l2:/dev/video1:capture_0"
              String "/dev/video2"
              String "/dev/video2"
                api.v4l2.path = "/dev/video2"
                object.path = "v4l2:/dev/video2"
                object.path = "v4l2:/dev/video2:capture_0"

and

pw-mon | grep v4l2
                api.v4l2.path = "/dev/video1"
                device.api = "v4l2"
                device.name = "v4l2_device._sys_devices_virtual_video4linux_video1"
                object.path = "v4l2:/dev/video1"
                api.v4l2.cap.driver = "v4l2 loopback"
                api.v4l2.cap.card = "Canon EOS M200"
                api.v4l2.cap.bus_info = "platform:v4l2loopback-001"
                api.v4l2.cap.version = "6.11.7"
                api.v4l2.cap.capabilities = "85200003"
                api.v4l2.cap.device-caps = "05200003"
                api.v4l2.path = "/dev/video2"
                device.api = "v4l2"
                device.name = "v4l2_device._sys_devices_virtual_video4linux_video2"
                object.path = "v4l2:/dev/video2"
                api.v4l2.cap.driver = "v4l2 loopback"
                api.v4l2.cap.card = "Dummy video device (0x0002)"
                api.v4l2.cap.bus_info = "platform:v4l2loopback-002"
                api.v4l2.cap.version = "6.11.7"
                api.v4l2.cap.capabilities = "85200003"
                api.v4l2.cap.device-caps = "05200003"
                api.v4l2.path = "/dev/video0"
                device.api = "v4l2"
                device.name = "v4l2_device._sys_devices_virtual_video4linux_video0"
                object.path = "v4l2:/dev/video0"
                api.v4l2.cap.driver = "v4l2 loopback"
                api.v4l2.cap.card = "OBS Virtual Camera"
                api.v4l2.cap.bus_info = "platform:v4l2loopback-000"
                api.v4l2.cap.version = "6.11.7"
                api.v4l2.cap.capabilities = "85200002"
                api.v4l2.cap.device-caps = "05200002"
                api.v4l2.cap.bus_info = "platform:v4l2loopback-001"
                api.v4l2.cap.capabilities = "85200003"
                api.v4l2.cap.card = "Canon EOS M200"
                api.v4l2.cap.device-caps = "05200003"
                api.v4l2.cap.driver = "v4l2 loopback"
                api.v4l2.cap.version = "6.11.7"
                api.v4l2.path = "/dev/video1"
                device.api = "v4l2"
                factory.name = "api.v4l2.source"
                node.name = "v4l2_input._sys_devices_virtual_video4linux_video1"
                object.path = "v4l2:/dev/video1"
                object.path = "v4l2:/dev/video1:capture_0"
                api.v4l2.cap.bus_info = "platform:v4l2loopback-002"
                api.v4l2.cap.capabilities = "85200003"
                api.v4l2.cap.card = "Dummy video device (0x0002)"
                api.v4l2.cap.device-caps = "05200003"
                api.v4l2.cap.driver = "v4l2 loopback"
                api.v4l2.cap.version = "6.11.7"
                api.v4l2.path = "/dev/video2"
                device.api = "v4l2"
                factory.name = "api.v4l2.source"
                node.name = "v4l2_input._sys_devices_virtual_video4linux_video2"
                object.path = "v4l2:/dev/video2"
                object.path = "v4l2:/dev/video2:capture_0"

I’m at a complete loss at this point, any help would be fantastic, even if it’s to simply point me to some useful user documentation.

Sanchayan · November 18, 2024, 7:14am

Can you try with mode=2 or mode=provide on the pipewiresink?

Do not have gphoto2 but here are the test pipeline/commands I tried locally.

wf-recorder --muxer=v4l2 --codec=rawvideo --file=/dev/video11 -x yuv420p

This uses wf-recorder to screencast to /dev/video11 which is a v4l2loopback device on my system.

GST_DEBUG=3,pipewiresink:5 gst-launch-1.0 v4l2src device=/dev/video11 ! queue ! videoconvert ! pipewiresink mode=provide client-name=SCREENCAST

This captures the screencast from the V4L2 loopback device and sends it to pipewiresink. Note the mode and client-name properties being set on the pipewiresink.

GST_DEBUG=3,pipewiresrc:5 gst-launch-1.0 pipewiresrc target-object=SCREENCAST ! videoconvert ! autovideosink

This above pipeline then plays back what was being written to pipewiresink using pipewiresrc. Note the target-object property setting on pipewiresrc here.

This also shows up in the output of wpctl status as below.

 └─ Streams:
       123. SCREENCAST
            132. output_1        > gst-launch-1.0:input_1       [active]
       209. gst-launch-1.0
            208. input_1         < SCREENCAST:output_1  [active]

GStreamer version used for this testing was 1.24.9.

mikesmith · November 18, 2024, 5:34pm

Wow thanks for this. So this is what I did.
I ran my gphoto2 scrip and outputted the stream to video1 like this:

gphoto2 --stdout --capture-movie | ffmpeg -hwaccel vaapi -c:v mjpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 2 -f v4l2 -s:v 1920x1080 /dev/video1
Then ran your script:
GST_DEBUG=3,pipewiresink:5 gst-launch-1.0 v4l2src device=/dev/video1 ! queue ! videoconvert ! pipewiresink mode=provide client-name=SCREENCAST
And finnaly your last script:
GST_DEBUG=3,pipewiresrc:5 gst-launch-1.0 pipewiresrc target-object=SCREENCAST ! videoconvert ! autovideosink

And up popped my video feed, so that is great!

I assume I’ll be able to change my script and remove the call to ffmpeg but that is an exercise for the class so to speak.

My new problem is how to I work with this Pipewire stream? I assumed now that I’m using Pipewire I’d be able to open Firefox go to google meet and simple see the video stream, like I used to be able to with /dev/video1 but I see nothing, same goes for VLC and OBS.

Using Helvum I can see the SCREENCAST item and it’s output_1.

So how do I access it other than your last script?

mikesmith · November 18, 2024, 6:34pm

Doing more research I believe the issue is the video stream is under “Streams” and should be under “Devices” when I do a wpctl status I get:

ipeWire 'pipewire-0' [1.2.6, p@cyanic, cookie:1905147376]
 └─ Clients:
        32. uresourced                          [1.2.6, p@cyanic, pid:2330]
        33. WirePlumber                         [1.2.6, p@cyanic, pid:2335]
        37. pipewire                            [1.2.6, p@cyanic, pid:2339]
        38.                                     [1.2.6, p@cyanic, pid:2654]
        41. Firefox                             [1.2.6, p@cyanic, pid:4252]
        49. WirePlumber [export]                [1.2.6, p@cyanic, pid:2335]
        52. kwin_wayland                        [1.2.6, p@cyanic, pid:2345]
        80. QtPulseAudio:2438                   [1.2.6, p@cyanic, pid:2438]
        83. libcanberra                         [1.2.6, p@cyanic, pid:2537]
        84.                                     [1.2.6, peppi@cyanic, pid:2537]
        85. xdg-desktop-portal                  [1.2.6, p@cyanic, pid:2365]
        88. libcanberra                         [1.2.6, p@cyanic, pid:2654]
        89. plasmashell                         [1.2.6, p@cyanic, pid:2654]
        91. Chromium input                      [1.2.6, p@cyanic, pid:153]
        94. Chromium input                      [1.2.6, p@cyanic, pid:129]
       101. Firefox                             [1.2.6, p@cyanic, pid:4252]
       108. wpctl                               [1.2.6, p@cyanic, pid:52631]
       113. Thunderbird                         [1.2.6, p@cyanic, pid:2]
       115. helvum                              [1.2.6, p@cyanic, pid:44339]
       116. mailnotification Plugin             [1.2.6, p@cyanic, pid:43185]
       118. obs                                 [1.2.6, p@cyanic, pid:50301]
       124. firefox                             [1.2.6, p@cyanic, pid:4252]
       180. gst-launch-1.0                      [1.2.6, p@cyanic, pid:49893]
       243.                                     [1.2.6, p@cyanic, pid:2537]
       313. OBS                                 [1.2.6, p@cyanic, pid:50301]

Audio
 ├─ Devices:
 │      53. Yeti Stereo Microphone              [alsa]
 │      54. Rembrandt Radeon High Definition Audio Controller [alsa]
 │      55. Family 17h/19h HD Audio Controller  [alsa]
 │
 ├─ Sinks:
 │  *   60. Yeti Stereo Microphone Analog Stereo [vol: 1.00]
 │      62. Family 17h/19h HD Audio Controller Digital Stereo (IEC958) [vol: 0.91]
 │      65. Rembrandt Radeon High Definition Audio Controller Digital Stereo (HDMI) [vol: 0.91]
 │
 ├─ Sources:
 │  *   59. Yeti Stereo Microphone Analog Stereo [vol: 0.75]
 │      63. Family 17h/19h HD Audio Controller Analog Stereo [vol: 0.85]
 │
 ├─ Filters:
 │
  └─ Streams:
        50. OBS
             92. monitor_FR
             95. monitor_FL
            112. input_FR        < Yeti Stereo Microphone:capture_FR    [active]
            114. input_FL        < Yeti Stereo Microphone:capture_FL    [active]
       143. OBS
            110. monitor_FR
            111. input_FL        < ALC897 Analog:capture_FL     [active]
            120. monitor_FL
            130. input_FR        < ALC897 Analog:capture_FR     [active]
       179. OBS
             90. monitor_FL
            248. input_FL        < Yeti Stereo Microphone:monitor_FL    [active]
            337. input_FR        < Yeti Stereo Microphone:monitor_FR    [active]
            338. monitor_FR

Video
 ├─ Devices:
 │      45. Canon EOS M200                      [v4l2]
 │      47. OBS Virtual Camera                  [v4l2]
 │
 ├─ Sinks:
 │
 ├─ Sources:
 │  *   86. Canon EOS M200 (V4L2)
 │
 ├─ Filters:
 │
 └─ Streams:
        81. SCREENCAST
            368. output_1

Settings
 └─ Default Configured Devices:
         0. Audio/Sink    alsa_output.pci-0000_10_00.6.analog-stereo
         1. Audio/Source  alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_LT_190923214456F39900DF_111000-00.analog-stereo

SCREENCAST being under Streams I think means that Firefox and OBS will not see it.

Any ideas how to resolve this?

Sanchayan · November 19, 2024, 5:35am

If you just want to show that stream in something like OBS,

This

wf-recorder --muxer=v4l2 --codec=rawvideo --file=/dev/video11 -x yuv420p

OR this

gst-launch-1.0 videotestsrc animation-mode=1 flip=true pattern=ball ! video/x-raw,width=1920,height=1080 ! clockoverlay ! videoconvert ! v4l2sink device=/dev/video11

By adding a V4L2 Capture Device as the source in OBS with the device selected to be /dev/video11, makes the stream being written to available in OBS.

This is also visible in Firefox. If I try the WebRTC landing page
WebRTC Test Landing Page for testing using getUserMedia example, when the Camera example asks for selecting an input, the V4L2 device can be selected and it works.

Note the device might show up a bit differently in the drop down menu.

# ~ λ: v4l2-ctl --list-devices
Dummy video device (0x000B) (platform:v4l2loopback-011):
        /dev/video11

Dummy video device (0x000C) (platform:v4l2loopback-012):
        /dev/video12

Dummy video device (0x000D) (platform:v4l2loopback-013):
        /dev/video13

Dummy video device (0x000E) (platform:v4l2loopback-014):
        /dev/video14

On my system, the hex numbers 0x000B are available for selection as camera input.

mikesmith · November 21, 2024, 2:02am

So I don’t just want to show the stream to something else. I want something else to use the SCREENCAST target. Aka I want other applications to use the PipeWire stream not the v4l2sink stream.

If you just use GST to stream to v4l2sink well I’m already doing that with my original script.

But I guess you’ve proven that GST can creat a PipeWire Video Stream. Is there a way to get GST to make a PipeWire Video Device or Video Source?

I believe if I created a PipeWire Video Device / Video Source: OBS, VLC and FireFox would then be able to use the PipeWire stream.

But I could be completely wrong. Happy for any correction or help you or anyone can give.

ystreet00 · November 21, 2024, 6:48am

gst-launch-1.0 videotestsrc animation-mode=1 flip=true pattern=ball is-live=true ! video/x-raw,width=1920,height=1080 ! clockoverlay ! videoconvert ! video/x-raw,format=I420 ! pipewiresink mode=provide stream-properties="properties,media.class=Video/Source,media.role=Camera"

Seems to work here with firefox.

mikesmith · November 22, 2024, 9:50pm

Wow! This is fantastic! Thanks so much.

So what I did was run this script: gphoto2 --stdout --capture-movie | ffmpeg -hwaccel vaapi -c:v mjpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 2 -f v4l2 -s:v 1920x1080 /dev/video1

Then I rant this GST pipeline: GST_DEBUG=3,pipewiresink:5 gst-launch-1.0 v4l2src device=/dev/video1 ! queue ! videoconvert ! pipewiresink mode=provide stream-properties="properties,media.class=Video/Source,media.rol e=Camera" client-name=EOS

When I did a wpctl status I see:

Video
 ├─ Devices:
 │      62. OBS Virtual Camera                  [v4l2]
 │      63. Canon EOS M200                      [v4l2]
 │  
 ├─ Sinks:
 │  
 ├─ Sources:
 │  *   86. Canon EOS M200 (V4L2)              
 │     293. EOS                                
 │  
 ├─ Filters:
 │  
 └─ Streams:

As you can see EOS is now included in Sources and not in Streams.

If I open Firefox I can now see the video stream!

Some questions:

How did you know to use: mode=provide stream-properties="properties,media.class=Video/Source,media.role=Camera"? Is there any documentation I can learn from?
Though FireFox can see the pipewire stream, both Google Chrome and OBS still do not see the stream, any ideas why?
There is an issue I have when I connect to the stream and then close FireFox. The GST stream dies, as there is no longer anything listening to to stream. Is there any way to change this behavior and not have the stream die when all clients (GST in this case) disconnect from the pipewire stream?

Here is the output from GST when I close FireFox:

GST_DEBUG=3,pipewiresink:5 gst-launch-1.0 v4l2src device=/dev/video1 ! queue ! videoconvert ! pipewiresink mode=provide stream-properties="properties,media.class=Video/Source,media.role=Camera" client-name=EOS
0:00:00.043178059 151049 0x55b73e11bf20 WARN                vafilter gstvafilter.c:1728:gst_va_filter_has_compose:<vafilter0> VPP does not support alpha blending
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
0:00:00.095985297 151048 0x7fa6b0000d90 WARN                    v4l2 gstv4l2object.c:4687:gst_v4l2_oPipeline is PREROLLED ...
bject_get_crop_rect:<v4l2src0:src> VIDIOC_CROPCAP failed
Setting pipeline to PLAYING ...
New clock: pipewireclock0
0:00:00.096978539 151048 0x7fa6b0000b90 DEBUG           pipewiresink gstpipewiresink.c:554:on_state_changed:<pipewiresink0> got stream state "connecting" (1)
0:00:00.097506730 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:554:on_state_changed:<pipewiresink0> got stream state "paused" (2)
0:00:00.097586810 151048 0x7fa6b0000d90 WARN                    v4l2 gstv4l2object.c:4687:gst_v4l2_object_get_crop_rect:<v4l2src0:src> VIDIOC_CROPCAP failed
0:00:00.097678400 151048 0x7fa6b0000d90 WARN           v4l2allocator gstv4l2allocator.c:276:gst_v4l2_memory_group_new:<v4l2src0:pool0:src:allocator> Driver pretends buffer 0 is queued even if freshly created, this indicates a bug in the driver.
0:00:00.097684350 151048 0x7fa6b0000d90 WARN           v4l2allocator gstv4l2allocator.c:276:gst_v4l2_memory_group_new:<v4l2src0:pool0:src:allocator> Driver pretends buffer 1 is queued even if freshly created, this indicates a bug in the driver.
Redistribute latency...
0:00:00.098306641 151048 0x7fa6b0000f90 WARN              bufferpool gstbufferpool.c:1429:gst_buffer_pool_set_flushing:<pipewirepool0> can't change flushing state of inactive pool
0:00:00.110077456 151048 0x7fa6b0000d90 WARN                 v4l2src gstv4l2src.c:1344:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1760 - ts: 0:00:00.014027340
0:00:00.151131960 151048 0x7fa6b0000b90 DEBUG           pipewiresink gstpipewiresink.c:299:pool_activated:<pipewirepool0> activated
0:04:57.541693422 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4dd8
0:04:57.541713222 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4e10
0:04:57.541717092 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4e48
0:04:57.541719902 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4e80
0:04:57.541724272 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4eb8
0:04:57.541726902 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4ef0
0:04:57.541731742 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4f28
0:04:57.541736742 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:461:on_add_buffer:<pipewiresink0> add pw_buffer 0x55f3064b4f60
0:04:57.542098372 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:554:on_state_changed:<pipewiresink0> got stream state "streaming" (3)
0:10:24.430237028 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:554:on_state_changed:<pipewiresink0> got stream state "paused" (2)
0:10:24.430273028 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4dd8
0:10:24.430281288 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4e10
0:10:24.430285778 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4e48
0:10:24.430289698 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4e80
0:10:24.430293668 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4eb8
0:10:24.430297358 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4ef0
0:10:24.430306818 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4f28
0:10:24.430311988 151048 0x7fa69800a060 DEBUG           pipewiresink gstpipewiresink.c:469:on_remove_buffer:<pipewiresink0> remove pw_buffer 0x55f3064b4f60
0:10:24.430319708 151048 0x7fa69800a060 WARN            pipewiresink gstpipewiresink.c:474:on_remove_buffer:<pipewiresink0> error: all buffers have been removed
0:10:24.430325428 151048 0x7fa69800a060 WARN            pipewiresink gstpipewiresink.c:474:on_remove_buffer:<pipewiresink0> error: PipeWire link to remote node was destroyed
ERROR: from element /GstPipeline:pipeline0/GstPipeWireSink:pipewiresink0: all buffers have been removed
Additional debug info:
../src/gst/gstpipewiresink.c(474): on_remove_buffer (): /GstPipeline:pipeline0/GstPipeWireSink:pipewiresink0:
PipeWire link to remote node was destroyed
Execution ended after 0:10:24.334405882
Setting pipeline to NULL ...
0:10:24.434665797 151048 0x55f30665dd90 DEBUG           pipewiresink gstpipewiresink.c:554:on_state_changed:<pipewiresink0> got stream state "unconnected" (0)
Freeing pipeline ...

Thanks for all the help so far much appreciated.

I figure I can always use GST Tee, to stream to PipeWire and v4l2, but would really like to be able to just use PipeWire with OBS and Chrome.