  Recently audio-visual speech recognition (AVSR), which better leverages video
modality as additional information to extend automatic speech recognition
(ASR), has shown promising results in complex acoustic environments. However,
there is still substantial space to improve as complex computation of visual
modules and ineffective fusion of audio-visual modalities. To eliminate these
drawbacks, we propose a down-up sampling-based AVSR model (Hourglass-AVSR) to
enjoy high efficiency and performance, whose time length is scaled during the
intermediate processing, resembling an hourglass. Firstly, we propose a context
and residual aware video upsampling approach to improve the recognition
performance, which utilizes contextual information from visual representations
and captures residual information between adjacent video frames. Secondly, we
introduce a visual-audio alignment approach during the upsampling by explicitly
incorporating boundary constraint loss. Besides, we propose a cross-layer
attention fusion to capture the modality dependencies within each visual
encoder layer. Experiments conducted on the MISP-AVSR dataset reveal that our
proposed Hourglass-AVSR model outperforms ASR model by 12.9% and 20.8% relative
concatenated minimum permutation character error rate (cpCER) reduction on
far-field and middle-field test sets, respectively. Moreover, compared to other
state-of-the-art AVSR models, our model exhibits the highest improvement in
cpCER for the visual module. Furthermore, on the benefit of our down-up
sampling approach, Hourglass-AVSR model reduces 54.2% overall computation costs
with minor performance degradation.
