I’m using tauri and I’m trying to embed a gstreamer window using videooverlay in it. I’ve been successful so far with mac. But I was going through this and noticed the comment that says “wayland cannot provide the API for this”. If that’s the case, how do I embed the window for wayland?
If tauri provides enough API for that, you could do the same as subprojects/gst-plugins-bad/tests/examples/waylandsink/main.c · main · GStreamer / gstreamer · GitLab and use waylandsink.
However, embedding the video via the GstVideoOverlay interface is more or less deprecated. It completely steps around the toolkit and by that introduces a lot of problems.
A better approach is to directly integrate video rendering from GStreamer with your toolkit-specific APIs, which is what e.g. gtksink / gtkglsink (GTK3), gtkwaylandsink (GTK3), gtk4paintablesink, qmlglsink (QML 5) and qml6glsink are doing.
Note that e.g. GTK4 does not provide any API at all anymore to embed arbitrary Wayland surfaces into the UI because of the problems mentioned before.
Correct me if I’m wrong at any point here. AFAIK, gtk can only be used on the main thread. Since I’m using async Commands (functions that you can call from the webview frontend) on tauri, the functions run on a separate thread. So ig gtk based sinks can’t be used?
I tried using glimagesink with VideoOverlay and I’m getting this error multiple times: Gstreamer-GL-Critical: Failed to flush Wayland connection.
I tried using waylandsink and I’m getting:
Application did not provide a wayland display handle cannot use an externally-supplied-surface without an externally-supplied display handle. Consider providing a display handle from your application with GstContext
And also since tauri used GTK3, embedding should be fine ig?
What do you recommend I do?
That’s correct. I mentioned those sinks as examples for achieving the same goal with other UI toolkits. You would have to do something similar to those sinks, just with whatever APIs tauri provides.
For as long as they use GTK3 on Linux, yes. That will probably change sooner or later, which GTK3 being basically EOL.
See the GTK3/Wayland example that I linked. Just sharing the surface is not sufficient for embedding, you also need to handle the display connection/handle.
Thanks for your guidance!
By any chance, is there a rust example for wayland? I’m not able to find it in the repo.
I’m basically struggling with these lines in the example
// Line 96 in the example
context = gst_wl_display_handle_context_new (display_handle);
gst_element_set_context (GST_ELEMENT (GST_MESSAGE_SRC (message)), context);
gst_context_unref (context);
I don’t exactly know which method and the way to typecast this in rust.
Right now I have this:
match msg.view() {
MessageView::NeedContext(needed_context) => {
if "GstWlDisplayHandleContextType" == needed_context.context_type() {
// tauri provides a way to get the display handle
if let RawDisplayHandle::Wayland(handle) = result.display_handle().unwrap().as_raw() {
// handle: NonNull<c_void>
// TODO: Setting the context here
}
}
}
}
And another question
Since you mentioned that GstVideoOverlay should preferrably not be used, should I avoid it on macOS/Windows? Or does this apply only for linux-based distros?
The gst_wl_display_handle_context_new() API is not available via the bindings yet. You’d have to link your application to libgstwayland-1.0 and declare the function locally and then call it via FFI.
That applies to any platform. You want proper integration of the video rendering into the rendering infrastructure of your toolkit.
Hey sorry for bothering you!
I was able to get waylandsink working. But unfortunately, it renders it’s own window which I discovered very late. So I’m back to using gtk ig.
I was trying to refer to Basic Tutorial 5 equivalent example in rust. But it doesn’t exist in GitLab.
I’m basically trying to get the GtkWidget and then use it and call a function to set the window for it.
In the C walkthrough, the code is:
g_object_get (videosink, "widget", &data.sink_widget, NULL);
What is the equivalent in rust?
Ok I figured out that I have to get it through
let widget: gstreamer::glib::Type = sink.property("widget");
But how do I convert the type to the gtk’s Widget type ?
Ok I actually made a mistake for the wayland sink. I didn’t handle the prepare-window-handle.
MessageView::NeedContext(needed_context) => {
if "GstWlDisplayHandleContextType" == needed_context.context_type() {
if let RawDisplayHandle::Wayland(display_handle) =
result.display_handle().unwrap().as_raw()
{
unsafe {
let gst_context = gst_wl_display_handle_context_new(
display_handle.display.as_ptr(),
);
let context = gstreamer::Context::from_glib_full(gst_context);
println!("Context type mate {}", context.context_type());
pipeline.set_context(&context);
}
}
}
}
MessageView::Element(element) => {
let foo = element.structure().unwrap();
if foo.has_name("prepare-window-handle") {
if let RawWindowHandle::Wayland(window_handle) = handle.as_raw() {
unsafe {
video_overlay
.set_window_handle(window_handle.surface.as_ptr() as usize);
}
}
}
I’ve defined the FFI like so
extern "C" {
pub fn gst_wl_display_handle_context_new(display: *mut c_void) -> *mut GstContext;
}
But I’m still getting error
Application did not provide a wayland display handle cannot use an externally-supplied-surface without an externally-supplied display handle. Consider providing a display handle from your application with GstContext
I’m not sure how you want to use the GTK sink in tauri but the correct way would be
let widget = sink.property::<gtk::Widget>("widget");
You’ll have to share your code at this point or debug it yourself. One possibility I can think of here is that you’re not providing the display handle from a sync bus handler but from an async one.
I just realised that I was setting the context for the pipeline instead of the sink. That’s the reason why it was failing. Sorry for the inconvenience!
I did try
let widget = sink.property::<gtk::Widget>("widget");
But I got the error: rustc: the trait bound for<'b> gtk::Widget: gstreamer::glib::value::FromValue<'b> is not satisfied
Also for providing context, instead of using the FFI, I’ve done the following looking at this repo:
let mut context = gstreamer::Context::new("GstWlDisplayHandleContextType", true);
let context_ref = context.get_mut().unwrap();
let structure_ref = context_ref.structure_mut();
let mut value = MaybeUninit::zeroed();
let send_value = unsafe {
gstreamer::glib::gobject_ffi::g_value_init(
value.as_mut_ptr(),
gstreamer::glib::gobject_ffi::G_TYPE_POINTER,
gstreamer::glib::gobject_ffi::g_value_set_pointer(
value.as_mut_ptr(),
display_handle.display.as_ptr(),
);
gstreamer::glib::SendValue::from_glib_none(&value.assume_init() as *const _)
};
structure_ref.set_value("handle", send_value);
sink.set_context(&context
Thanks a lot for the assistance!
That means you have incompatible versions of the GTK and GStreamer bindings. Note that the GTK3 bindings are not maintained (and updated) anymore since more than a year, so you can’t use the latest version of the GStreamer bindings.
That’s basically the same in more lines, yes.
Also, if you got it working maybe you can put an example up somewhere. You’re probably not the only one running into this problem 
Will do it over the weekend! Thanks again!
Quick question: Do you guys have a roadmap or like a planner which I can look at? I was just wondering, what it takes for example a bad-plugin to become a good-plugin, the release cycle, etc