  The end-to-end (E2E) automatic speech recognition (ASR) systems are often
required to operate in reverberant conditions, where the long-term sub-band
envelopes of the speech are temporally smeared. In this paper, we develop a
feature enhancement approach using a neural model operating on sub-band
temporal envelopes. The temporal envelopes are modeled using the framework of
frequency domain linear prediction (FDLP). The neural enhancement model
proposed in this paper performs an envelope gain based enhancement of temporal
envelopes. The model architecture consists of a combination of convolutional
and long short term memory (LSTM) neural network layers. Further, the envelope
dereverberation, feature extraction and acoustic modeling using transformer
based E2E ASR can all be jointly optimized for the speech recognition task. The
joint optimization ensures that the dereverberation model targets the ASR cost
function. We perform E2E speech recognition experiments on the REVERB challenge
dataset as well as on the VOiCES dataset. In these experiments, the proposed
joint modeling approach yields significant improvements compared to the
baseline E2E ASR system (average relative improvements of 21% on the REVERB
challenge dataset and about 10% on the VOiCES dataset).
