Concatenating multiple streams containing multiple streams

Application Development
1

I’m looking for a way on how to concatenate two streams, both containing audio and video inside. I was following examples and advice found here Concatenate multiple streams gaplessly with GStreamer – coaxion.net – slomo's blog and prepared these pipelines:

  1. 1st media file generation:
gst-launch-1.0 -e \
    webmmux name=mux ! filesink location=input1.webm \
    videotestsrc pattern=ball num-buffers=100  ! video/x-raw,width=1280,height=720 ! vp9enc ! mux. \
    audiotestsrc wave=ticks num-buffers=300 ! audioconvert ! opusenc ! mux.
  1. 2nd media file generation:
gst-launch-1.0 -e \
    webmmux name=mux ! filesink location=input2.webm \
    videotestsrc pattern=smpte num-buffers=100  ! video/x-raw,width=1280,height=720 ! vp9enc ! mux. \
    audiotestsrc wave=sine num-buffers=300 ! audioconvert ! opusenc ! mux.
  1. A failed attempt to concatenate it and produce mpeg-ts:
gst-launch-1.0 \
    mpegtsmux name=mux ! filesink location=output.ts \
    concat name=ac ! queue ! audioconvert ! avenc_aac ! queue ! mux. \
    concat name=vc ! queue ! videoconvert ! x264enc ! h264parse ! queue ! mux. \
    \
    streamsynchronizer name=ss1 ! vc. \
    ss1. ! ac. \
    filesrc location=input1.webm ! decodebin name=dec1 \
    dec1.src_0 ! queue  ! ss1. \
    dec1.src_1 ! queue  ! ss1. \
    \
    streamsynchronizer name=ss2 ! videoconvert ! vc. \
    ss2. ! audioconvert ! ac. \
    filesrc location=input2.webm ! decodebin name=dec2 \
    dec2.src_0 ! queue ! ss2. \
    dec2.src_1 ! queue ! ss2.

The above pipeline hangs. Playing around further with it I came up with something like the following, It produces two separate output files:

gst-launch-1.0 \
    matroskamux name=mux1 ! filesink location=out1.mp4 \
    matroskamux name=mux2 ! filesink location=out2.mp4 \
    concat name=ac ! queue ! audioconvert ! avenc_aac ! queue ! mux1. \
    concat name=vc ! queue ! videoconvert ! x264enc ! h264parse ! queue ! mux2. \
    \
    streamsynchronizer name=ss1 ! vc. ss1. ! ac. \
    filesrc location=input1.webm ! decodebin name=dec1 \
    dec1.src_0 ! queue ! ss1. \
    dec1.src_1 ! queue ! ss1. \
    \
    streamsynchronizer name=ss2 ! vc. ss2. ! ac. \
    filesrc location=input2.webm ! decodebin name=dec2 \
    dec2.src_0 ! queue ! ss2. \
    dec2.src_1 ! queue ! ss2. \

Although audio output (out1.mp4) seems to be fine, the video output (out2.mp4) is not valid and when playing gstreamer complains that:

WARNING A lot of buffers are being dropped.
WARNING debug information: ../subprojects/gstreamer/libs/gst/base/gstbasesink.c(3145): gst_base_sink_is_too_late (): /GstPlayBin:playbin/GstPlaySink:playsink/GstBin:vbin/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink:
There may be a timestamping problem, or this computer is too slow.

Could you help me here? TIA

2

Hello again,

Simplifying the problem further I prepared the following pipeline:

gst-launch-1.0 -v \
    mpegtsmux name=mux ! filesink location=concatenated_output.ts \
    concat name=vc ! identity silent=false ! x264enc ! h264parse ! mux. \
    \
    videotestsrc num-buffers=100 pattern=smpte ! video/x-raw,width=1280,height=720 ! vc. \
    videotestsrc num-buffers=100 pattern=ball ! video/x-raw,width=1280,height=720 ! vc.

When displaying the concatenated video (with autovideosink ) instead of serializing it to the file all seems to be fine, but when playing the generated concatenated_output.ts there is definitely something wrong. ffprobe complains as well:

ffprobe concatenated_output.ts
ffprobe version 6.0 Copyright (c) 2007-2023 the FFmpeg developers
  built with Apple clang version 14.0.3 (clang-1403.0.22.14.1)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/6.0_1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324302999, dts=324596999, size=68
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324308999, dts=324596999, size=80
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324311999, dts=324596999, size=84
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324314999, dts=324596999, size=84
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324317999, dts=324596999, size=81
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324323999, dts=324596999, size=137
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324320999, dts=324596999, size=87
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324326999, dts=324596999, size=108
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324329999, dts=324596999, size=109
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324332999, dts=324596999, size=87
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324335999, dts=324596999, size=92
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324338999, dts=324596999, size=106
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324341999, dts=324596999, size=169
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324344999, dts=324596999, size=154
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324347999, dts=324596999, size=165
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324350999, dts=324596999, size=211
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324353999, dts=324596999, size=190
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324356999, dts=324596999, size=168
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324359999, dts=324596999, size=200
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324362999, dts=324596999, size=150
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324365999, dts=324596999, size=135
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324368999, dts=324596999, size=193
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324371999, dts=324596999, size=246
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324377999, dts=324596999, size=198
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324374999, dts=324596999, size=80
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324383999, dts=324596999, size=179
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324380999, dts=324596999, size=161
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324395999, dts=324596999, size=268
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324389999, dts=324596999, size=141
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324386999, dts=324596999, size=145
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324392999, dts=324596999, size=135
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324407999, dts=324596999, size=314
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324401999, dts=324596999, size=113
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324398999, dts=324596999, size=95
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324404999, dts=324596999, size=99
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324419999, dts=324596999, size=275
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324413999, dts=324596999, size=180
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324410999, dts=324596999, size=107
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324416999, dts=324596999, size=125
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324431999, dts=324596999, size=374
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324425999, dts=324596999, size=163
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324422999, dts=324596999, size=126
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324428999, dts=324596999, size=125
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324443999, dts=324596999, size=453
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324437999, dts=324596999, size=154
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324434999, dts=324596999, size=164
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324440999, dts=324596999, size=158
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324455999, dts=324596999, size=443
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324449999, dts=324596999, size=206
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324446999, dts=324596999, size=177
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324452999, dts=324596999, size=136
[mpegts @ 0x128f050d0] Invalid timestamps stream=0, pts=324467999, dts=324596999, size=505
Input #0, mpegts, from 'concatenated_output.ts':
  Duration: 00:00:06.67, start: 3600.000000, bitrate: 1258 kb/s
  Program 1
  Stream #0:0[0x41]: Video: h264 (High 4:4:4 Predictive) (HDMV / 0x564D4448), yuv444p10le(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn

When using matroskamux instead of mpegtsmux:

gst-launch-1.0 \
    matroskamux name=mux ! filesink location=concatenated_output.mp4 \
    concat name=vc ! x264enc ! h264parse ! mux. \
    \
    videotestsrc num-buffers=100 pattern=smpte ! video/x-raw,width=1280,height=720 ! vc. \
    videotestsrc num-buffers=100 pattern=ball ! video/x-raw,width=1280,height=720 ! vc.

the generated file is also invalid (easier to observe when playing with ffplay instead of gst-play-1.0 but still).

Do you know if this is a problem in concat element itself, or perhaps in how I use it and for some reason using it in that pipeline is not legitimate?

3

Set single-segment=TRUE in identity element’s property and check.

1 Like
4

Yes, setting offsets helped. Thank you!