  Despite recent advances in voice separation methods, many challenges remain
in realistic scenarios such as noisy recording and the limits of available
data. In this work, we propose to explicitly incorporate the phonetic and
linguistic nature of speech by taking a transfer learning approach using an
end-to-end automatic speech recognition (E2EASR) system. The voice separation
is conditioned on deep features extracted from E2EASR to cover the long-term
dependence of phonetic aspects. Experimental results on speech separation and
enhancement task on the AVSpeech dataset show that the proposed method
significantly improves the signal-to-distortion ratio over the baseline model
and even outperforms an audio visual model, that utilizes visual information of
lip movements.
