I am building a program which can take multiple video inputs, compose them into one image and display on screen. I want to be able to dynamically add and remove input branches without pausing the pipeline.
So I have this:
udpsrc→decoder→queue→glvideomixer::sink_0
udpsrc→decoder→queue→glvideomixer::sink_1
glvideomixer→glimagesink
udpsrc→decoder→queue is put into its own bin, and linked with compositor using ghost pads.
I want to remove and add new input bins on demand, to have at most 4 and at least 0 linked. But I run into issues.
Initially I was creating all 4 bins and linked them and reused them by just changing the RTP port on each udpsrc element (putting it to NULL and then back to PLAYING) to the new stream. And for the unused bins I just put them to NULL and hide them in glvideomixer. This was working fine, but ran into issues.
Now I want to actually unlink and remove each bin when they are no longer needed, and create new one and link when adding new. I have tried this using pad blocking on the bin’s src-pad, and then unlink with glvideomixer sink-pad. But when I add a new bin, sometimes the video is not showing for that bin, and when I generate a graph for this, I can see that the src-pad has “Pad-Task: has paused task”, i.e. is says “[t]” instead of “[T]”. I have been following guides for dynamically unlinking and linking, but still it is not working great. How can I figure out what is causing the video to not show?
I have been searching for how to best do this, feels like I have a very common usecase.. but have not been able to find any good examples that cover my scenario.
What I do today:
add pad-probe of type IDLE on the queue-element’s src-pad. In the callback, I unlink the pad with the compostor’s pad. I signal to the application thread that it can start to dispose of the bin. That is to set it to null, remove from pipeline and unref the bin. I also release the compositor sink-pad.
When adding a new bin, I create a new compositor request pad, I create the bin itself, link it with the new compositor sink pad, set the port of the udpsrc and then I set the state to playing (gst_element_sync_state_with_parent(bin) followed by gst_bin_sync_children_states(bin).
What am I doing wrong here, what am I missing?