  Single-channel speech enhancement approaches do not always improve automatic
recognition rates in the presence of noise, because they can introduce
distortions unhelpful for recognition. Following a trend towards end-to-end
training of sequential neural network models, several research groups have
addressed this problem with joint training of front-end enhancement module with
back-end recognition module. While this approach ensures enhancement outputs
are helpful for recognition, the enhancement model can overfit to the training
data, weakening the recognition model in the presence of unseen noise. To
address this, we used a pre-trained acoustic model to generate a perceptual
loss that makes speech enhancement more aware of the phonetic properties of the
signal. This approach keeps some benefits of joint training, while alleviating
the overfitting problem. Experiments on Voicebank + DEMAND dataset for
enhancement show that this approach achieves a new state of the art for some
objective enhancement scores. In combination with distortion-independent
training, our approach gets a WER of 2.80\% on the test set, which is more than
20\% relative better recognition performance than joint training, and 14\%
relative better than distortion-independent mask training.
