I spoke to a bunch of folks at the conference after my talk, and it looks like there are many people doing this sort of thing. Feel free to share your experiences, questions, comments, brickbats over here!
Slides: https://indico.freedesktop.org/event/5/contributions/251/attachments/114/172/gst-on-the-server.pdf
Recording: Server-side Media Processing with GStreamer - GStreamer conferences