RFC: New gst-python-analytics project under FDO GStreamer group

General Discussion
1

Over the past 6 months, Collabora has developed an extensive Python framework to improve GStreamer support for various ML frameworks and models.

Supported functionality includes:

  1. object detection
  2. tracking
  3. video captioning
  4. translation
  5. transcription
  6. speech to text
  7. text to speech
  8. text to image
  9. LLMs
  10. serializing to cloud via Kafka server

Many of these new GStreamer features are not available anywhere else, neither in downstream open source forks or in commercial forks such as DeepStream.

The code started as an MR for the gst-python bindings subproject and has now moved here

As this project is tied to the version of gst analytics in upstream GStreamer, it may make sense to move it into the GStreamer fdo umbrella, either as a separate project or as part of the monorepo. Moving it will also make it easier for the community to contribute to development.

I am opening this topic to discuss a proposal to move the project to fdo. Looking forward to hearing your feedback.

cc @slomo @tpm @bilboed @thiblahute @ocrete @dmorin

2 Likes
2

I just catch-up on MR 7615 thread. If I can bring inputs to this discussion is I think bringing these classes under the GStreamer umbrella would be great in the caveat they don’t obstruct native development. I think an easy way to do this would be to prefix element name. I’m still thinking about pros and cons monorepo vs own project, but looking forward to read other’s opinion on this.

1 Like
3

Spent a bit of time looking at it (will try running some in a bit). In general the code is quite clean and a good example of integrating such systems in gstreamer plugins.

I concur with @dmorin regarding namespace. Maybe prefix the elements with pyml for example ? And maybe gst-python-ml would be a better namespace than gst-python-analytics , since it’s more than just analytics

1 Like
4

Naming it gst-python-ml or gst-python-ai or something like that is also good for helping people find it.

1 Like
5

Might also want to make it clear it won’t work with python > 3.11 (like most ml/ai sadly)

6

In fact it does work with 3.12 - this is the distro version for Ubuntu 24, and it works fine.

7

pypi packages are confusing. Definitely fails with 3.13

1 Like
8

thanks, will document that. We can enforce the version in the requirements.txt file. 3.13 is only 3 months old, so…

9

you can always set up a venv with 3.12

10

I see potential with GES by aiding techniques such as motion tracking or rotoscoping with CV algorithms, something that Kdenlive and Shotcut already do with OpenCV in MLT. By adding GESEffect elements based on a CV library of choice it could bring such tools to GES-based editors.

Another use could be AI-based frame interpolation, many people would be interested in that. On the side of analytics perhaps a histogram element based on Matplotlib can be of use.

2 Likes
11

Vox Populi, Vox Dei

I’ve changed the project name to gst-python-ml. Also, will be adding _pyml post-fix to all ML elements in the project.

New project location: GitHub - collabora/gst-python-ml

What needs to happen to move this to fdo ?

1 Like
12

The general rule is to prefix elements, to make it clear. Ex the rs variants of some elements are prefixed with rs.

13

makes sense - I have changed to prefix. Also, in case you tried the docker instance, there was a bug that prevented python element loading - now fixed.