  Compensation for channel mismatch and noise interference is essential for
robust automatic speech recognition. Enhanced speech has been introduced into
the multi-condition training of acoustic models to improve their generalization
ability. In this paper, a noise-aware training framework based on two cascaded
neural structures is proposed to jointly optimize speech enhancement and speech
recognition. The feature enhancement module is composed of a multi-task
autoencoder, where noisy speech is decomposed into clean speech and noise. By
concatenating its enhanced, noise-aware, and noisy features for each frame, the
acoustic-modeling module maps each feature-augmented frame into a triphone
state by optimizing the lattice-free maximum mutual information and cross
entropy between the predicted and actual state sequences. On top of the
factorized time delay neural network (TDNN-F) and its convolutional variant
(CNN-TDNNF), both with SpecAug, the two proposed systems achieve word error
rate (WER) of 3.90% and 3.55%, respectively, on the Aurora-4 task. Compared
with the best existing systems that use bigram and trigram language models for
decoding, the proposed CNN-TDNNF-based system achieves a relative WER reduction
of 15.20% and 33.53%, respectively. In addition, the proposed CNN-TDNNF-based
system also outperforms the baseline CNN-TDNNF system on the AMI task.
