  Speaker extraction is to extract a target speaker's voice from multi-talker
speech. It simulates humans' cocktail party effect or the selective listening
ability. The prior work mostly performs speaker extraction in frequency domain,
then reconstructs the signal with some phase approximation. The inaccuracy of
phase estimation is inherent to the frequency domain processing, that affects
the quality of signal reconstruction. In this paper, we propose a time-domain
speaker extraction network (TseNet) that doesn't decompose the speech signal
into magnitude and phase spectrums, therefore, doesn't require phase
estimation. The TseNet consists of a stack of dilated depthwise separable
convolutional networks, that capture the long-range dependency of the speech
signal with a manageable number of parameters. It is also conditioned on a
reference voice from the target speaker, that is characterized by speaker
i-vector, to perform the selective listening to the target speaker. Experiments
show that the proposed TseNet achieves 16.3% and 7.0% relative improvements
over the baseline in terms of signal-to-distortion ratio (SDR) and perceptual
evaluation of speech quality (PESQ) under open evaluation condition.
