  It has been shown that the intelligibility of noisy speech can be improved by
speech enhancement (SE) algorithms. However, monaural SE has not been
established as an effective frontend for automatic speech recognition (ASR) in
noisy conditions compared to an ASR model trained on noisy speech directly. The
divide between SE and ASR impedes the progress of robust ASR systems,
especially as SE has made major advances in recent years. This paper focuses on
eliminating this divide with an ARN (attentive recurrent network) time-domain
and a CrossNet time-frequency domain enhancement models. The proposed systems
fully decouple frontend enhancement and backend ASR trained only on clean
speech. Results on the WSJ, CHiME-2, LibriSpeech, and CHiME-4 corpora
demonstrate that ARN and CrossNet enhanced speech both translate to improved
ASR results in noisy and reverberant environments, and generalize well to real
acoustic scenarios. The proposed system outperforms the baselines trained on
corrupted speech directly. Furthermore, it cuts the previous best word error
rate (WER) on CHiME-2 by $28.4\%$ relatively with a $5.57\%$ WER, and achieves
$3.32/4.44\%$ WER on single-channel CHiME-4 simulated/real test data without
training on CHiME-4.
