Based on this post I used a valve to drop the audio packets, and then stopped the audio_sink.
When I detect that the usb device is back I can close the valve and reactivate the audio_sink. Sometimes it works, sometimes the sound is a bit “dirty”. I am not sure yet how to properly do this.