  Deep gated convolutional networks have been proved to be very effective in
single channel speech separation. However current state-of-the-art framework
often considers training the gated convolutional networks in time-frequency
(TF) domain. Such an approach will result in limited perceptual score, such as
signal-to-distortion ratio (SDR) upper bound of separated utterances and also
fail to exploit an end-to-end framework. In this paper we present an integrated
simple and effective end-to-end approach to monaural speech separation, which
consists of deep gated convolutional neural networks (GCNN) that takes the
mixed utterance of two speakers and maps it to two separated utterances, where
each utterance contains only one speaker's voice. In addition long short-term
memory (LSTM) is employed for long term temporal modeling. For the objective,
we propose to train the network by directly optimizing utterance level SDR in a
permutation invariant training (PIT) style. Our experiments on the public
WSJ0-2mix data corpus demonstrate that this new scheme can produce more
discriminative separated utterances and leading to performance improvement on
the speaker separation task.
