  Single-channel, speaker-independent speech separation methods have recently
seen great progress. However, the accuracy, latency, and computational cost of
such methods remain insufficient. The majority of the previous methods have
formulated the separation problem through the time-frequency representation of
the mixed signal, which has several drawbacks, including the decoupling of the
phase and magnitude of the signal, the suboptimality of time-frequency
representation for speech separation, and the long latency in calculating the
spectrograms. To address these shortcomings, we propose a fully-convolutional
time-domain audio separation network (Conv-TasNet), a deep learning framework
for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder
to generate a representation of the speech waveform optimized for separating
individual speakers. Speaker separation is achieved by applying a set of
weighting functions (masks) to the encoder output. The modified encoder
representations are then inverted back to the waveforms using a linear decoder.
The masks are found using a temporal convolutional network (TCN) consisting of
stacked 1-D dilated convolutional blocks, which allows the network to model the
long-term dependencies of the speech signal while maintaining a small model
size. The proposed Conv-TasNet system significantly outperforms previous
time-frequency masking methods in separating two- and three-speaker mixtures.
Additionally, Conv-TasNet surpasses several ideal time-frequency magnitude
masks in two-speaker speech separation as evaluated by both objective
distortion measures and subjective quality assessment by human listeners.
Finally, Conv-TasNet has a significantly smaller model size and a shorter
minimum latency, making it a suitable solution for both offline and real-time
speech separation applications.
