Do awss3sink or gssink stream data directly to cloud storage in gstreamer?

General Discussion
1

There are 2 possible approaches:

  1. Save data to local files and upload each to cloud storage.
  2. Stream incoming data to cloud storage directly.

Which one is used by gssink and awss3sink internally?

2

gssink and awss3sink both write to cloud storage directly by using the relevant APIs. gssink uses the relevant APIs via google-cloud-storage library, and awss3sink uses AWS SDK for Rust.

3

I’ve not looked at gssink, but awss3sink uses the multi-part upload API so in principle will stream into S3. You can use the part-size parameter to control the part-sizes.

I say “in principle” because I got caught out in the following way: Since awss3sink streams the data, is it not seekable (unlike, say, filesink) so if you have something like mp4mux upstream of the awss3sink (and depending on the properties you have set on the muxer) it may observe that the downstream sink is not seekable, and decide to keep everything buffered in memory or on disk until eos in order to write the atoms at the start of the file correctly.

4

You might want to look at Fragmented MP4 muxer when working with awss3sink.

5

Thanks! Yeah, I did look at fragmented mp4 and do some testing. Unfortunately for me the resulting files didn’t work for my use-case, so had to stick with standard mp4.