  With its strong modeling capacity that comes from a multi-head and
multi-layer structure, Transformer is a very powerful model for learning a
sequential representation and has been successfully applied to speech
separation recently. However, multi-channel speech separation sometimes does
not necessarily need such a heavy structure for all time frames especially when
the cross-talker challenge happens only occasionally. For example, in
conversation scenarios, most regions contain only a single active speaker,
where the separation task downgrades to a single speaker enhancement problem.
It turns out that using a very deep network structure for dealing with signals
with a low overlap ratio not only negatively affects the inference efficiency
but also hurts the separation performance. To deal with this problem, we
propose an early exit mechanism, which enables the Transformer model to handle
different cases with adaptive depth. Experimental results indicate that not
only does the early exit mechanism accelerate the inference, but it also
improves the accuracy.
