So! After making really really sure I follow your dynamic-tee-vsink example removing of the outgoing path works fine!
Now our stress test ran further, but it never ran more than ~100 or ~200 iterations, then it fails when removing the src part of the pipeline.
I dug down and read documentation and thought I got everything right. But it was hanging on setting the bin with the src to state null.
This happened in the open media room at FOSDEM and @ndufresne was there and helped me find the relevant stack trace in GDB:
Thread 13 (Thread 0x7f649f5fa6c0 (LWP 61797) "rocket-worker-t"):
#0 0x00007f64a52893c0 in __lll_lock_wait () at /lib64/libc.so.6
#1 0x00007f64a528fe97 in pthread_mutex_lock@@GLIBC_2.2.5 () at /lib64/libc.so.6
#2 0x00007f64a51b5948 in gst_base_transform_activate () at /lib64/libgstbase-1.0.so.0
#3 0x00007f64a51b5a07 in gst_base_transform_sink_activate_mode () at /lib64/libgstbase-1.0.so.0
#4 0x00007f64a58a6807 in activate_mode_internal () at /lib64/libgstreamer-1.0.so.0
#5 0x00007f64a58a733b in gst_pad_set_active () at /lib64/libgstreamer-1.0.so.0
#6 0x00007f64a588c69b in activate_pads.lto_priv.1.lto_priv () at /lib64/libgstreamer-1.0.so.0
#7 0x00007f64a5898ae5 in gst_iterator_fold () at /lib64/libgstreamer-1.0.so.0
#8 0x00007f64a590c27e in iterator_activate_fold_with_resync.constprop () at /lib64/libgstreamer-1.0.so.0
#9 0x00007f64a588c7fe in gst_element_pads_activate () at /lib64/libgstreamer-1.0.so.0
#10 0x00007f64a588ca21 in gst_element_change_state_func.lto_priv () at /lib64/libgstreamer-1.0.so.0
#11 0x00007f649c6d7960 in gst_identity_change_state () at /lib64/gstreamer-1.0/libgstcoreelements.so
#12 0x00007f64a588ba94 in gst_element_change_state () at /lib64/libgstreamer-1.0.so.0
#13 0x00007f64a588bad8 in gst_element_change_state () at /lib64/libgstreamer-1.0.so.0
#14 0x00007f64a588c349 in gst_element_set_state_func.lto_priv () at /lib64/libgstreamer-1.0.so.0
#15 0x0000559d33c62139 in gstreamer::auto::element::{impl#3}::set_state<gstreamer::auto::element::Element> (self=0x7f649f5f2e58, state=gstreamer::auto::enums::State::Null) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gstreamer-0.20.7/src/auto/element.rs:547
#16 0x0000559d33c2f8bc in switchboard::jack::SwitchboardJack::remove_line (bin=0x7f649f5f3938, index=0) at spiideo-switchboard/src/jack.rs:825
#17 0x0000559d33c30151 in switchboard::jack::SwitchboardJack::remove_incoming_jack_line_thread_fn (bin=0x7f649f5f3938, index=0, pipeline=0x7f649f5f3940) at spiideo-switchboard/src/jack.rs:868
#18 0x0000559d33c31835 in switchboard::jack::SwitchboardJack::remove_incoming (self=0x7f6468324ae0, pipeline=0x7f64a00c9510) at spiideo-switchboard/src/jack.rs:947
#19 0x0000559d33c34332 in switchboard::jack::SwitchboardJack::remove (self=0x7f6468324ae0, pipeline=0x7f64a00c9510) at spiideo-switchboard/src/jack.rs:1072
#20 0x0000559d33bb2dd6 in switchboard::Switchboard::remove_jack (self=0x7f64a00c9510, name=...) at spiideo-switchboard/src/lib.rs:499
#21 0x0000559d331b7b08 in shibuya::switchboard_actor::SwitchboardActor::remove_incoming_jack (self=0x559d36e7d8b0) at shibuya/src/switchboard_actor.rs:372
#22 0x0000559d331b7f54 in shibuya::switchboard_actor::SwitchboardActor::handle_message (self=0x559d36e7d8b0, msg=...) at shibuya/src/switchboard_actor.rs:448
#23 0x0000559d331d001c in shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn#0} () at shibuya/src/switchboard_actor.rs:228
#24 0x0000559d330cbd40 in tokio::runtime::task::core::{impl#6}::poll::{closure#0}<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (ptr=0x559d36e7d8b0) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334
#25 0x0000559d330c8efb in tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}>>::with_mut<tokio::runtime::task::core::Stage<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>> (self=0x559d36e7d8b0, f=...) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16
#26 tokio::runtime::task::core::Core<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>::poll<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (self=0x559d36e7d8a0, cx=...) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323
#27 0x0000559d33183b11 in tokio::runtime::task::harness::poll_future::{closure#0}<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> () at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485
#28 0x0000559d3323bae3 in core::panic::unwind_safe::{impl#23}::call_once<core::task::poll::Poll<()>, tokio::runtime::task::harness::poll_future::{closure_env#0}<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>> (self=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/panic/unwind_safe.rs:271
#29 0x0000559d3308690b in std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>> (data=0x7f649f5f83a8) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panicking.rs:526
#30 0x0000559d3308d70b in __rust_try ()
#31 0x0000559d33082ba8 in std::panicking::try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>> (f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panicking.rs:490
#32 0x0000559d3303bbca in std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>> (f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panic.rs:142
#33 0x0000559d3318212e in tokio::runtime::task::harness::poll_future<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (core=0x559d36e7d8a0, cx=...) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473
#34 0x0000559d3318775f in tokio::runtime::task::harness::Harness<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>::poll_inner<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (self=0x7f649f5f8
5c0) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208
#35 0x0000559d3318bea3 in tokio::runtime::task::harness::Harness<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>::poll<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (self=...) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153
#36 0x0000559d3335740b in tokio::runtime::task::raw::poll<shibuya::switchboard_actor::{impl#1}::run_actor::{async_fn_env#0}, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (ptr=...) at /home/jonasdn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:276
#37 0x0000559d35524d77 in tokio::runtime::task::raw::RawTask::poll (self=...) at src/runtime/task/raw.rs:200
#38 0x0000559d35563502 in tokio::runtime::task::LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>::run<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (self=...) at src/runtime/task/mod.rs:400
#39 0x0000559d3555a0f0 in tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:639
#40 0x0000559d35559754 in tokio::runtime::coop::with_budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> (budget=..., f=...) at src/runtime/coop.rs:107
#41 tokio::runtime::coop::budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> (f=...) at src/runtime/coop.rs:73
#42 tokio::runtime::scheduler::multi_thread::worker::Context::run_task (self=0x7f649f5f8cb8, task=..., core=0x559d36e86b60) at src/runtime/scheduler/multi_thread/worker.rs:575
#43 0x0000559d35558c7a in tokio::runtime::scheduler::multi_thread::worker::Context::run (self=0x7f649f5f8cb8, core=0x559d36e86b60) at src/runtime/scheduler/multi_thread/worker.rs:526
#44 0x0000559d35558789 in tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:491
#45 0x0000559d35543d90 in tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>::set<tokio::runtime::scheduler::Context, tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}, ()> (self=0x7f649f5fa3b0, t=0x7f649f5f8cb0, f=...) at src/runtime/context/scoped.rs:40
#46 0x0000559d355132cb in tokio::runtime::context::set_scheduler::{closure#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}> (c=0x7f649f5fa378) at src/runtime/context.rs:176
#47 0x0000559d355653fe in std::thread::local::LocalKey<tokio::runtime::context::Context>::try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>, ()> (self=0x559d362d6188, f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/thread/local.rs:270
#48 0x0000559d3556433b in std::thread::local::LocalKey<tokio::runtime::context::Context>::with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>, ()> (self=0x559d362d6188, f=Python Exception <class 'gdb.MemoryError'>: Cannot access memory at address 0x80
#49 0x0000559d35513244 in tokio::runtime::context::set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}> (v=0x7f649f5f8cb0, f=...) at src/runtime/context.rs:176
#50 0x0000559d35558691 in tokio::runtime::scheduler::multi_thread::worker::run::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:486
#51 0x0000559d35521888 in tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::worker::run::{closure_env#0}, ()> (handle=0x7f649f5f8eb8, allow_block_in_place=true, f=...) at src/runtime/context/runtime.rs:65
#52 0x0000559d3555841c in tokio::runtime::scheduler::multi_thread::worker::run (worker=...) at src/runtime/scheduler/multi_thread/worker.rs:478
#53 0x0000559d3555828b in tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:447
#54 0x0000559d3555ca4e in tokio::runtime::blocking::task::{impl#2}::poll<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}, ()> (self=..., _cx=0x7f649f5f9040) at src/runtime/blocking/task.rs:42
#55 0x0000559d3552ebc3 in tokio::runtime::task::core::{impl#6}::poll::{closure#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (ptr=0x559d36e9c028) at src/runtime/task/core.rs:334
#56 0x0000559d3552ea3b in tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>>::with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> (self=0x559d36e9c028, f=...) at src/loom/std/unsafe_cell.rs:16
#57 tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (self=0x559d36e9c020, cx=...) at src/runtime/task/core.rs:323
#58 0x0000559d3554bf75 in tokio::runtime::task::harness::poll_future::{closure#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> () at src/runtime/task/harness.rs:485
#59 0x0000559d3552b743 in core::panic::unwind_safe::{impl#23}::call_once<core::task::poll::Poll<()>, tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> (self=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/panic/unwind_safe.rs:271
#60 0x0000559d3552950c in std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> (data=0x7f649f5f91a8) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panicking.rs:526
#61 0x0000559d3552b54b in __rust_try ()
#62 0x0000559d35528e08 in std::panicking::try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>> (f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panicking.rs:490
#63 0x0000559d355776eb in std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runti
#63 0x0000559d355776eb in std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runti--Type <RET> for more, q to quit, c to continue without paging--
me::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> (f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panic.rs:142
#64 0x0000559d3554bd4f in tokio::runtime::task::harness::poll_future<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (core=0x559d36e9c020, cx=...) at src/runtime/task/harness.rs:473
#65 0x0000559d3554aa99 in tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>::poll_inner<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (self=0x7f649f5f93c0) at src/runtime/task/harness.rs:208
#66 0x0000559d3554a817 in tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (self=...) at src/runtime/task/harness.rs:153
#67 0x0000559d3552502d in tokio::runtime::task::raw::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (ptr=...) at src/runtime/task/raw.rs:276
#68 0x0000559d35524d77 in tokio::runtime::task::raw::RawTask::poll (self=...) at src/runtime/task/raw.rs:200
#69 0x0000559d355635c7 in tokio::runtime::task::UnownedTask<tokio::runtime::blocking::schedule::BlockingSchedule>::run<tokio::runtime::blocking::schedule::BlockingSchedule> (self=...) at src/runtime/task/mod.rs:437
#70 0x0000559d35525687 in tokio::runtime::blocking::pool::Task::run (self=...) at src/runtime/blocking/pool.rs:159
#71 0x0000559d35527ce9 in tokio::runtime::blocking::pool::Inner::run (self=0x559d36e81b50, worker_thread_id=11) at src/runtime/blocking/pool.rs:513
#72 0x0000559d35527a14 in tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure#0} () at src/runtime/blocking/pool.rs:471
#73 0x0000559d3553ac76 in std::sys_common::backtrace::__rust_begin_short_backtrace<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> (f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/sys_common/backtrace.rs:154
#74 0x0000559d3553c232 in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> () at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/thread/mod.rs:529
#75 0x0000559d3552b5c2 in core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>> (self=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/panic/unwind_safe.rs:271
#76 0x0000559d35529429 in std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()> (data=0x7f649f5f9830) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panicking.rs:526
#77 0x0000559d3552b54b in __rust_try ()
#78 0x0000559d35529081 in std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>> (f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panicking.rs:490
#79 0x0000559d3553c05a in std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()> (f=...) at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panic.rs:142
#80 std::thread::{impl#0}::spawn_unchecked_::{closure#1}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> () at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/thread/mod.rs:528
#81 0x0000559d3553c3df in core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked_::{closure_env#1}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>, ()> () at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/ops/function.rs:250
#82 0x0000559d35658295 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2007
#83 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2007
#84 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#85 0x00007f64a528c897 in start_thread () at /lib64/libc.so.6
#86 0x00007f64a53136fc in clone3 () at /lib64/libc.so.6
It looked like it could perhaps be some interaction with tokio (thread sharing?) and the streaming thread?
When I did the actual removal (set to null, remove from pipeline) from a separate thread, pretty much:
let bin_weak = bin.downgrade();
let pipeline_weak = pipeline.downgrade();
std::thread::spawn(move || {
let bin = bin_weak.upgrade().unwrap();
let pipeline = pipeline_weak.upgrade().unwrap();
Self::remove_incoming_jack_line_thread_fn(&bin, index, &pipeline)?;
});
I did not see the hang anymore. It is a bit hard to describe our setup … it is Rust and rocket webserver that gets requests to add / remove sinks / srcs to a pipeline.
It is an actor pattern where we send messages to the thread that runs the pipeline mainloop. But here it looks like the streaming thread and the tokio threads are intermingled? It is difficult for me to follow.
Do you have any insight here?
There is a rtpjitterbuffer in the mix and our feeling is that this more prevaliant when we set an higher latency ~1000 ms +
