Short version is: for some reason when I am adding compositor into my pipeline, it completely looses the frames from appsrc, showing transparent emptiness instead. If you can provide me with an example for such pipeline, it will resolve my question.
appsrc name=source ! compositor ! videoconvert ! autovideosink
The longer story is that I am trying to receive an RTP stream and overlay an image on top of the received frames.
I managed to do such thing with cairo and cairoverlay, but I was not impressed with the performance. So, I thought of a better way - to generate a transparent frame and overlay it with compositor. This will decouple frame size from drawing performance, especially with upscaling done by the same compositor. Great, I thought!
But instead I am trying to make it work for like third day straight.
The full pipeline with v4l2src for testing is as follows:
"v4l2src ! video/x-raw,width=640,height=480,framerate=30/1 ! queue ! mix. "
"appsrc name=appsrc is-live=true format=GST_FORMAT_TIME caps=video/x-raw,format=RGBA,width=640,height=480,framerate=30/1 ! queue ! "
"videoconvert ! compositor name=mix sink_1::width=640 sink_1::height=480 sink_0::width=640 sink_0::height=480 sink_1::xpos=0 sink_0::xpos=640 !"
"videoconvert ! autovideosink "
But the result is just an empty left side with webcam stream on the right half. (I moved them side by side just to make sure that this is not a zheight issue)
If I remove the compositor and webcam streams so that appsrc is outputting directly, I get the image corrrectly.
"appsrc name=appsrc is-live=true format=GST_FORMAT_TIME caps=video/x-raw,format=RGBA,width=640,height=480,framerate=30/1 ! queue ! "
"videoconvert ! autovideosink "
Interesting part is that it does not matter what I am generating frames with - I tried both Python PIL and cairo, and both were outputting correctly with direct pipeline and where showing transparent emptiness with composer.
Also interesting point is that if I leave only appsrc as an input for compositor, the result window is of 0 size. If I resize it manually it still shows transparent rectangle with right proportions for (640*2)x480.
Logs are also not that usefull. In one test I took working example from here to make sure that the issue is at least reproducible, added composer to the pipeline, and it got different logs 
Here are the two lines of logs from not my code (I didn’t find where to plug in to increase debug level):
GStreamer-Video:ERROR:../gstreamer/subprojects/gst-plugins-base/gst-libs/gst/video/gstvideoaggregator.c:2635:gst_video_aggregator_sink_event: assertion failed: (seg.format == GST_FORMAT_TIME)
Bail out! GStreamer-Video:ERROR:../gstreamer/subprojects/gst-plugins-base/gst-libs/gst/video/gstvideoaggregator.c:2635:gst_video_aggregator_sink_event: assertion failed: (seg.format == GST_FORMAT_TIME)
Yet in code, fixes and my fixes formats are defined seemingly correctly…
And here is the log from my code with webcamera and cairo for generating a frame:
0:00:00.138525009 22092 0x72a080000b90 WARN v4l2src gstv4l2src.c:993:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.138675055 22092 0x72a080000b90 WARN aggregator gstaggregator.c:2304:gst_aggregator_query_latency_unlocked:<mix> Latency query failed
0:00:00.138804231 22092 0x72a080001490 FIXME default gstutils.c:4089:gst_element_decorate_stream_id_internal:<appsrc> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:00.138899844 22092 0x72a080000b90 WARN v4l2src gstv4l2src.c:993:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.139657721 22092 0x72a080000b90 WARN aggregator gstaggregator.c:2304:gst_aggregator_query_latency_unlocked:<mix> Latency query failed
0:00:00.139846453 22092 0x72a080000b90 WARN v4l2src gstv4l2src.c:993:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.139963435 22092 0x72a080000b90 WARN aggregator gstaggregator.c:2304:gst_aggregator_query_latency_unlocked:<mix> Latency query failed
0:00:00.144573799 22092 0x72a080000b90 WARN v4l2src gstv4l2src.c:993:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.144598188 22092 0x72a080000b90 WARN aggregator gstaggregator.c:2304:gst_aggregator_query_latency_unlocked:<mix> Latency query failed
0:00:00.152366599 22092 0x72a080000b90 WARN v4l2src gstv4l2src.c:993:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.152409830 22092 0x72a080000b90 WARN aggregator gstaggregator.c:2304:gst_aggregator_query_latency_unlocked:<mix> Latency query failed
0:00:00.152383399 22092 0x72a080001250 WARN v4l2 gstv4l2object.c:4610:gst_v4l2_object_set_crop:<v4l2src0:src> VIDIOC_S_CROP failed
0:00:00.176471145 22092 0x72a080001250 WARN v4l2 gstv4l2object.c:3273:gst_v4l2_object_reset_compose_region:<v4l2src0:src> Failed to get default compose rectangle with VIDIOC_G_SELECTION: Invalid argument
0:00:00.250037014 22092 0x72a080000b90 FIXME basesink gstbasesink.c:3399:gst_base_sink_default_event:<autovideosink0-actual-sink-xvimage> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:00.583695721 22092 0x72a080001250 WARN v4l2src gstv4l2src.c:1344:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:00.412880774
Here is the full main function:
def main():
Gst.init(None)
Gst.debug_set_default_threshold(3)
pipeline_description = (
"v4l2src ! video/x-raw,width=640,height=480,framerate=30/1 ! queue ! mix. "
"appsrc name=appsrc is-live=true format=GST_FORMAT_TIME caps=video/x-raw,format=RGBA,width=640,height=480,framerate=30/1 ! queue ! "
"videoconvert ! compositor name=mix background=1 sink_1::width=640 sink_1::height=480 sink_0::width=640 sink_0::height=480 sink_1::xpos=0 sink_0::xpos=640 !"
"videoconvert ! autovideosink "
)
pipeline = Gst.parse_launch(pipeline_description)
appsrc = pipeline.get_by_name("appsrc")
if not appsrc:
print("Error: Unable to find appsrc in the pipeline.")
return
thread = threading.Thread(target=appsrc_feed, args=(appsrc, 640, 480), daemon=True)
thread.start()
pipeline.set_state(Gst.State.PLAYING)
# Run the GLib main loop to keep the pipeline running
loop = GLib.MainLoop()
try:
loop.run()
except KeyboardInterrupt:
print("Exiting...")
pipeline.set_state(Gst.State.NULL)
I am not that versed in gstreamer app programming (I mostly configured it from shell before), so my code originated from ChatGPT. I obviously modified it, but that will be the answer for weird decisions, if there are.
So, what is your verdict? Have I missed something, or is that some kind of a bug? Any help would be usefull, even just confirmation that I haven’t lost my mind, at this point.