# Does setting pad offset require mixer latency?

**URL:** <https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229>\
**Category:** Application Development\
**Created:** [October 5, 2023, 1:33pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229 "2023-10-05T13:33:18Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 1:33pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/1 "2023-10-05T13:33:18Z")

</div>

In my application I use pad offsets to synchronize audio streams. These offsets are applied to the source pads linked to the sink pads of an audiomixer element.

Do I need to account for these offsets by increasing the latency property of the audiomixer? Or are pad offsets automatically taken into account into the pipeline latency?

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 2:12pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/2 "2023-10-05T14:12:53Z")

</div>

Additional question: does it matter whether the pad offset is on the mixer sinkpad, or on its peer?

---

<div class="post-metadata">

**Author:** ![slomo](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/slomo/32/4_2.png) [@slomo](https://discourse.gstreamer.org/u/slomo)\
**Post date:** [October 5, 2023, 2:41pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/3 "2023-10-05T14:41:59Z")

</div>

> Do I need to account for these offsets by increasing the latency property of the audiomixer? Or are pad offsets automatically taken into account into the pipeline latency?

You need to take them into account yourself if the addition of the pad offset actually increases the latency. It often doesn’t.

> Additional question: does it matter whether the pad offset is on the mixer sinkpad, or on its peer?

Theoretically no, but in practice pad offsets on sink pads misbehave due to hard to fix bugs and I would recommend putting the pad offset on a source pad.

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 2:48pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/4 "2023-10-05T14:48:24Z")

</div>

Awesome, thanks!

> [@slomo](#):
>
> You need to take them into account yourself if the addition of the pad offset actually increases the latency. It often doesn’t.

When/why wouldn’t it? If the offset is not on the “critical path”? In my case I can expect each of the inputs to have nearly identical latency, so then delaying any of them would increase the effective latency, right?

> [@slomo](#):
>
> Theoretically no, but in practice pad offsets on sink pads misbehave due to hard to fix bugs and I would recommend putting the pad offset on a source pad.

I recall trying that and things not working, so this is very good to know, thanks!

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 2:52pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/5 "2023-10-05T14:52:51Z")

</div>

How does `min_upstream_latency` factor into this? I set this to a worst case amount. My inputs are SRT streams, and it looks like srtsrc actually runs at lower latency than its latency property indicates, as long as network conditions are good. Does the mixer always delay its output by `min_upstream_latency` or is that only how long it is willing/able to buffer internally _if needed_?

---

<div class="post-metadata">

**Author:** ![slomo](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/slomo/32/4_2.png) [@slomo](https://discourse.gstreamer.org/u/slomo)\
**Post date:** [October 5, 2023, 2:52pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/6 "2023-10-05T14:52:53Z")

</div>

> [@mkonstapel](#):
>
> When/why wouldn’t it? If the offset is not on the “critical path”? In my case I can expect each of the inputs to have nearly identical latency, so then delaying any of them would increase the effective latency, right?

If you delay them you increase the running time of each buffer. By increasing the running time you effectively reduce the latency (or I am confused and need more coffee 😅 )

---

<div class="post-metadata">

**Author:** ![slomo](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/slomo/32/4_2.png) [@slomo](https://discourse.gstreamer.org/u/slomo)\
**Post date:** [October 5, 2023, 2:54pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/7 "2023-10-05T14:54:50Z")

</div>

> [@mkonstapel](#):
>
> Does the mixer always delay its output by `min_upstream_latency` or is that only how long it is willing/able to buffer internally _if needed_?

It doesn’t delay anything as long as buffers arrive early enough. It only sets a deadline until when buffers must arrive before the mixer times out and produces output anyway.

The value is taken into account by sinks and other elements that sync to the clock though. They will always delay by that much at least (unless one of the inputs of the mixer or any of the other streams going to any sink has a higher latency, then they delay by that much instead).

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 2:58pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/8 "2023-10-05T14:58:43Z")

</div>

> [@slomo](#):
>
> If you delay them you increase the running time of each buffer. By increasing the running time you effectively reduce the latency (or I am confused and need more coffee 😅 )

😅

I keep confusing myself with this as well!

I apply only positive offsets (negative offsets didn’t work, it just goes silent) so that adds to the running time of the buffers. That means they are due to be aggregated later, so the other inputs have to be buffered until then…? No, maybe you are right, delaying any input actually buys you extra time for the other inputs to arrive.

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 3:03pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/9 "2023-10-05T15:03:07Z")

</div>

> [@slomo](#):
>
> The value is taken into account by sinks and other elements that sync to the clock though.

None of my sinks are sync (hlssink, ie.e. filesink, and rtpbin into udpsinks to another process), but there are multiple levels of mixers and muxers, which do sync to the clock in live mode if I understand correctly.

So if an upstream mixer has a `min_upstream_latency` already, I don’t need to also set it on a downstream mixer or muxer?

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 3:05pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/10 "2023-10-05T15:05:36Z")

</div>

> [@mkonstapel](#):
>
> negative offsets didn’t work, it just goes silent

💡 Maybe that was because I did “run out” of latency then. I asked it to aggregate one source’s buffers early, and then others weren’t there yet.

What is leading for a live aggregator, the timestamps of its input buffers, or the pipeline clock time?

---

<div class="post-metadata">

**Author:** ![slomo](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/slomo/32/4_2.png) [@slomo](https://discourse.gstreamer.org/u/slomo)\
**Post date:** [October 5, 2023, 3:06pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/11 "2023-10-05T15:06:52Z")

</div>

> [@mkonstapel](#):
>
> No, maybe you are right, delaying any input actually buys you extra time for the other inputs to arrive.

Yes that’s what I was thinking.

> [@mkonstapel](#):
>
> negative offsets didn’t work, it just goes silent

Because then they all arrive even more too late (they were supposed to be rendered before you even received the buffer) and more latency needs to be configured to give them more time.

> [@mkonstapel](#):
>
> which do sync to the clock in live mode if I understand correctly.

No, they only use the latency as a deadline for timing out in live pipelines.

> [@mkonstapel](#):
>
> What is leading for a live aggregator, the timestamps of its input buffers, or the pipeline clock time?

The running time of the buffers and the latency make up the deadline for timing out, and it determines when that deadline is reached based on the pipeline clock.

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 3:08pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/12 "2023-10-05T15:08:45Z")

</div>

Thanks Sebastian! I am going to digest this info 😁 It might be just what I need, or I might be back with more questions 😇

---

<div class="post-metadata">

**Author:** ![slomo](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/slomo/32/4_2.png) [@slomo](https://discourse.gstreamer.org/u/slomo)\
**Post date:** [October 5, 2023, 3:14pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/13 "2023-10-05T15:14:58Z")

</div>

I think the important part to understand about latency in GStreamer is that in a live pipeline the running time of a buffer is the capture time. So at that point the buffer is already exactly just in time and delaying it any further would make it too late. The latency then gives an additional budget for the buffer to be processed further downstream until it arrives at its destination, so each element on the way (including the source as it needs a moment from capture to actually having the buffer ready for sending downstream) adds its own latency to the (minimum!) latency in the latency query.

When configuring the latency on a pipeline (by default) all sinks are queried for the latency, and the maximum of all minimum latencies is then configured on the whole pipeline so that streams with lower latency are delayed inside the sink by as much as is necessary to make them in sync with the stream with the highest latency.

The minimum latency in the latency query is also kind of confusingly named. It’s not the minimum latency that is introduced, but it is the maximum latency that is introduced in the worst case. It is the minimum latency that downstream has to compensate for to allow for buffers to be in time and not consider them all too late.

The maximum latency in the latency query on the other hand is the amount of buffering that is provided by the elements along the way. As delaying streams for a higher latency (e.g. video stream has 200ms latency, audio stream has 100ms latency, then you need to delay the audio for another 100ms to have both streams in sync) requires buffering somehow this always needs to be higher than the configured (and maximum of all minimum) latencies.

---

<div class="post-metadata">

**Author:** ![mkonstapel](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/mkonstapel/32/73_2.png) [@mkonstapel](https://discourse.gstreamer.org/u/mkonstapel)\
**Post date:** [October 5, 2023, 8:27pm UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/14 "2023-10-05T20:27:31Z")

</div>

> [@slomo](#):
>
> Yes that’s what I was thinking.

That’s good to hear, because I had convinced myself that was correct, then drove home thinking about it and found myself thinking “no wait it’s the other way around or is it” 😆

But let me think. If I have one input from say a live testsrc, that has no latency so it sets the deadline for the rest. The mixer knows its upstream latency from the pipeline latency query (which can be “artificially” increased by min-upstream-latency), so it will queue the buffers from the testsrc while they wait for “matching” buffers on the other pads. The latency property configured on the mixer is **“extra”** to account for imperfect timing upstream?

Now, if I set an offset on one of the inputs, those buffers also spend some time in the queue, because their running time is ahead of the other live inputs. But (and only) if I were to set the offset greater than the upstream latency, I’ll be making buffers “FROM THE FUTURE” and the mixer won’t provide enough queuing and I need to increase the mixer latency?

---

<div class="post-metadata">

**Author:** ![slomo](https://discourse.gstreamer.org/user_avatar/discourse.gstreamer.org/slomo/32/4_2.png) [@slomo](https://discourse.gstreamer.org/u/slomo)\
**Post date:** [October 6, 2023, 7:35am UTC](https://discourse.gstreamer.org/t/does-setting-pad-offset-require-mixer-latency/229/15 "2023-10-06T07:35:24Z")

</div>

> [@mkonstapel](#):
>
> The latency property configured on the mixer is **“extra”** to account for imperfect timing upstream?

It’s in addition to allow for wrongly reported latency from upstream, yes. It will allow for buffers to arrive later than they should without discarding them (or whatever the aggregator subclass’ timeout behaviour is).

> [@mkonstapel](#):
>
> But (and only) if I were to set the offset greater than the upstream latency, I’ll be making buffers “FROM THE FUTURE” and the mixer won’t provide enough queuing and I need to increase the mixer latency

No, any positive offset would delay the buffers and they would be “from the future”. There would have to be enough possible buffering (see maximum latency from the query) upstream of the mixer (e.g. a big enough `queue`) to allow for delaying by that much.

The mixer’s latency property only allows for buffers to arrive later than the configured latency. You could make use of that when setting a negative pad offset to compensate for buffers all arriving late according to their timestamp and latency budget.