Black screen/video when using compositor to merge two video sources

General Discussion
1

I am unable to get compositor to work when mixing two video streams. The simplest case is to merge two test video sources together, eg:

gst-launch-1.0 compositor name=comp sink_1::xpos=640 ! queue ! videoconvert ! videoscale ! x264enc ! mp4mux ! filesink location=/tmp/testcomp.mp4 -e \
videotestsrc pattern=snow ! videorate ! video/x-raw,width=640,height=480,framerate=10/1,format=RGBx ! queue ! comp.sink_0 \
videotestsrc pattern=pinwheel ! videorate ! video/x-raw,width=640,height=480,framerate=10/1,format=RGBx ! queue ! comp.sink_1

When viewing the output video in the file browser you can see in the thumbnail the two test patterns, but when playing the video back it is just black.

If I don’t use compositor and just encode a single test source everything is ok.

gst-launch-1.0 videotestsrc pattern=snow ! videorate ! video/x-raw,width=640,height=480,framerate=10/1 !  videoconvert ! videoscale ! x264enc ! mp4mux ! filesink location=/tmp/testcomp-sing.mp4 -e

Do you get a black screen video from the top compositor command? Any ideas on how to fix this?

2

Both commands produce valid output here with GStreamer 1.24. You may need to specify what GStreamer version you are using.

3

Thanks for confirming.

I tried on two different machines, the first running 1.18.4 (Raspberry Pi/Debian Bullseye) and the second 1.22.8 (Fedora).

Tried on Ubuntu 22.04 with version 1.20.3 and it works.

Using VLC on all platforms to playback.

So its probably not a gstreamer version thing, but something else with those platforms. Will need to play around and see.

4

I discovered the the MP4 videos produced by compositor all played back ok on VLC under Ubuntu but not under Fedora.

Using ffmpeg to probe the video codec of the compositor video it is encoding with yuv444p

  Stream #0:0[0x1](und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p(tv, smpte170m/bt709/iec61966-2-1, progressive), 1280x480 [SAR 1:1 DAR 8:3], 1765 kb/s, 10 fps, 10 tbr, 1k tbn (default)

When outputing a single stream without compositor the encoding is yuv444p10le

  Stream #0:0[0x1](und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p10le(tv, smpte170m, progressive), 1280x480 [SAR 1:1 DAR 8:3], 1649 kb/s, 10 fps, 10 tbr, 1k tbn (default)

I see this is a problem with the VLC packaging under Fedora 39 which has broken playback of h264.

A work around is to force the format of gstreamer by adding video/x-raw,format=Y444_10LE, eg:

gst-launch-1.0 compositor name=comp sink_1::xpos=640 ! queue ! video/x-raw,format=Y444_10LE ! videoconvert ! videoscale ! x264enc ! mp4mux ! filesink location=/tmp/testcomp-codec.mp4 -e \
videotestsrc pattern=snow ! videorate ! video/x-raw,width=640,height=480,framerate=10/1 ! queue ! comp.sink_0 \
videotestsrc pattern=pinwheel ! videorate ! video/x-raw,width=640,height=480,framerate=10/1 ! queue ! comp.sink_1