  Real-time target speaker extraction (TSE) is intended to extract the desired
speaker's voice from the observed mixture of multiple speakers in a streaming
manner. Implementing real-time TSE is challenging as the computational
complexity must be reduced to provide real-time operation. This work introduces
to Conv-TasNet-based TSE a new architecture based on state space modeling (SSM)
that has been shown to model long-term dependency effectively. Owing to SSM,
fewer dilated convolutional layers are required to capture temporal dependency
in Conv-TasNet, resulting in the reduction of model complexity. We also enlarge
the window length and shift of the convolutional (TasNet) frontend encoder to
reduce the computational cost further; the performance decline is compensated
by over-parameterization of the frontend encoder. The proposed method reduces
the real-time factor by 78% from the conventional causal Conv-TasNet-based TSE
while matching its performance.
