Pipeline with multiple video streams overlayed and alpha masked

[ti-guru]

Hi,

I’m trying to build a pipeline in gstreamer that overlays multiple video streams from v4l2src and udpsrc+rtpvrawdepay on a background image where one of the streams is alpha masked with an image. Digging through the documentation and Stack Overflow didn’t show any (obvious) plugins or examples that describe this case. E.g. the alpha plugin does chroma keying, but I’d like to use a png image to define the regions being masked in the stream.

In ffmpeg I would use a complex filter to overlay video streams onto a background image with alphaextract and alphamerge to mask one of the stream:

ffmpeg -i input1.mp4 -i input2.mp4 -i background.png -i mask.png -filter_complex "[0:v]scale=640:480:force_original_aspect_ratio=decrease[camera];[3]alphaextract[mask];[1][mask]alphamerge[crop];[crop]scale=1800:1012:force_original_aspect_ratio=decrease[stream];[2][camera]overlay=40:300[canvas];[canvas][stream]overlay=650:100" output.mp4

Any help pointing me in the right direction would be much appreciated!

[Honey_Patouceul]

Sorry that I’m not so familiar with ffmpeg for fully understanding your case, you may better tell your case for better advice.

Most of what you’re looking for should be available from compositor element:

gst-inspect-1.0 compositor

This element allows to compose video buffers from several input streams. For alpha composing have inputs in RGBA format, and set zorder property as well as location and size for each input. Searching this forum for compositor should lead you to a few examples for syntax or see compositor.

For receiving raw video from RTP over UDP, you may have to set sampling/depth and resolution (maybe also framerate) on receiver side, such as:

gst-launch-1.0 -v udpsrc port=5000 ! 'application/x-rtp, media=(string)video, encoding-name=(string)RAW, sampling=(string)RGBA, depth=(string)8, width=(string)640, height=(string)480' ! queue ! rtpjitterbuffer latency=0 ! rtpvrawdepay ! video/x-raw,format=RGBA ! autovideoconvert ! autovideosink

Note that latency=0 property of rtpjitterbuffer may only work on localhost. For real network you would increase (default is 2000 ms).

You can use a png file as mask, but as it is a single image, you may have to use element imagefreeze for making a still video from it, such as:

gst-launch-1.0 filesrc location=test_alpha_mask.png ! pngdec ! video/x-raw,format=RGBA ! imagefreeze ! video/x-raw,framerate=30/1 ! autovideoconvert ! autovideosink

# If the command above fails you may try instead
gst-launch-1.0 filesrc location=test_alpha_mask.png ! pngdec ! video/x-raw,format=RGBA ! imagefreeze ! video/x-raw,framerate=30/1 ! videoconvert ! autovideosink

[ti-guru]

Thanks for reaching out and sorry for not explaining my use case better. Specifically I’m trying to do the following:

Currently, I’m not sure on how to do the masking in gstreamer (i.e. how to make the hexagonal crop in the image above). What plugin should I look into?

[Honey_Patouceul]

Assuming that your background image and masking image have the same resolution, and that your V4L cameras output raw video @30 fps (not encoded such as JPEG or H264) you would try something like:

gst-launch-1.0 -v \
filesrc location=background.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_0 \
v4l2src device=/dev/video0 ! videoconvert ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_1 \
v4l2src device=/dev/video1 ! videoconvert ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_2 \
filesrc location=masking_image.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_3 \
compositor name=comp \
 sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 \
 sink_1::xpos=200 sink_1::ypos=300 sink_1::width=640 sink_1::height=480 sink_1::zorder=2 \
 sink_2::xpos=1100 sink_2::ypos=300 sink_2::width=640 sink_2::height=480 sink_2::zorder=2 \
 sink_3::xpos=0 sink_3::ypos=0 sink_3::width=1920 sink_3::height=1080 sink_3::zorder=3 \
 ! autovideoconvert ! autovideosink

[ti-guru]

Thanks again for the great example of how to use the compositor plugin; that saved me a bunch of time! However, running this pipeline only overlays the masking picture. I’m not able to extract parts of the stream coming from the V4L2 device. Se below for images:

[ti-guru]

This is an example frame from a V4L2 device:

[ti-guru]

This is an example masking file:

[ti-guru]

What I’m currently getting with the compositor example above:

[ti-guru]

What I would like to have:

So what plugin do I need to add to the pipeline in your example to get the masking effect shown in last image?

[Honey_Patouceul]

You would invert alpha channel from your masking_image.png file (alpha=1-alpha). This can be done with any image editing tool supporting png with alpha.

[ti-guru]

As always the devil is in the details! The original masking_image.png was exported from Gimp as RGB 8bpc and not RGBA 8bpc. Switching this and leaving out the green color in the RGB channels correctly masked the region of interest. However, the masked area is covered by black color and the background is not visible using the following pipeline:

gst-launch-1.0 -v \
filesrc location=background.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_0 \
v4l2src device=/dev/video11 ! videoconvert ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_1 \
filesrc location=masking_image.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_2 \
compositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 sink_1::zorder=2 \
sink_2::xpos=0 sink_2::ypos=0 sink_2::width=1920 sink_2::height=1080 sink_2::zorder=3 \
! autovideoconvert ! autovideosink

This yields the following stream:

Is there any way to make the black areas transparent so it’s possible to see the background?

Looking into the GStreamer documentation, alphacombine kind of does the masking I’m looking for. However, in the example pipeline in the docs, the videotestsrc is still partly visible outside the ball pattern. Is it possible to set the areas outside the ball pattern to be completely transparent?

[Honey_Patouceul]

Yes, you can use alphacombine for that. You would need a png mask with same resolution as the camera you want to mask (here assuming 640x480@30):

gst-launch-1.0 -v \
filesrc location=background.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_0 \
videotestsrc pattern=ball ! videoconvert ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_1 \
videotestsrc ! videoconvert ! video/x-raw,format=I420,framerate=30/1,width=640,height=480 ! acomb.sink \
filesrc location=mask.png ! pngdec ! imagefreeze ! videoconvert ! videoscale ! video/x-raw,format=I420,framerate=30/1,width=640,height=480 ! acomb.alpha \
alphacombine name=acomb ! video/x-raw,format=A420,framerate=30/1 ! comp.sink_2 \
compositor name=comp \
 sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 \
 sink_1::xpos=200 sink_1::ypos=200 sink_1::width=640 sink_1::height=480 sink_1::zorder=2 \
 sink_2::xpos=1100 sink_2::ypos=200 sink_2::width=640 sink_2::height=480 sink_2::zorder=2 \
 ! autovideoconvert ! autovideosink

[ti-guru]

Thanks again for taking your time! So close to getting the pipeline I need. I used the following pipeline:

gst-launch-1.0 -v \
filesrc location=background.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_0 \
v4l2src device=/dev/video11 ! queue ! videoconvert ! videoscale ! video/x-raw,format=I420,framerate=30/1,width=1920,height=1080 ! acomb.sink \
filesrc location=masking_image.png ! pngdec ! imagefreeze ! videoconvert ! videoscale ! video/x-raw,format=I420,framerate=30/1,width=1920,height=1080 ! acomb.alpha \
alphacombine name=acomb ! video/x-raw,format=A420,framerate=30/1 ! comp.sink_1 \
compositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 sink_1::zorder=2  \
! autovideoconvert ! autovideosink

This results in the following output:

Alphacombine makes the correct masking and the background is visible. However, areas outside the transparent part of the alpha mask is still visible, e.g. you can still see the left part of the island behind the climber.

Did I miss some setting in the alphacombine element that can adjust this? Or is it something wrong in my masking_image.png?

[Honey_Patouceul]

@ti-guru

So close to getting the pipeline I need.

The 80/20 rules here. I may have helped the easy part… I also see the issue with alphacombine but I’m currently unable to provide any solution or workaround.
For the remaining 20% someone better skilled may better help.

[ti-guru]

No worries! I’m just happy you pointed me in the right direction and forced me to really think this one over.

I’ve finally manged to get the pipeline I need: By nesting two compositors and using the alpha plugin I’m able to mask the v4l2 source and place it on top of the background:

gst-launch-1.0 \
v4l2src device=/dev/video11 ! queue ! videoconvert ! videoscale ! videorate ! video/x-raw,format=I420,framerate=30/1,width=1920,height=1080 ! alphacomp.sink_0 \
filesrc location=masking_image.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! alphacomp.sink_1 \
compositor name=alphacomp ! alpha method=green  ! comp.sink_1 \
filesrc location=background.png ! pngdec ! imagefreeze ! video/x-raw,format=RGBA,framerate=30/1 ! comp.sink_0 \
compositor name=comp \
sink_0::zorder=1 sink_1::zorder=2 \
! autovideoconvert ! autovideosink

This yields the following output:

It’s probably not the most elegant or resource efficient solution. So if someone knows how to shorten this or use alphacombine properly please leave a comment!