  Automatic speech recognition (ASR) technologies have been significantly
advanced in the past few decades. However, recognition of overlapped speech
remains a highly challenging task to date. To this end, multi-channel
microphone array data are widely used in current ASR systems. Motivated by the
invariance of visual modality to acoustic signal corruption and the additional
cues they provide to separate the target speaker from the interfering sound
sources, this paper presents an audio-visual multi-channel based recognition
system for overlapped speech. It benefits from a tight integration between a
speech separation front-end and recognition back-end, both of which incorporate
additional video input. A series of audio-visual multi-channel speech
separation front-end components based on TF masking, Filter&Sum and mask-based
MVDR neural channel integration approaches are developed. To reduce the error
cost mismatch between the separation and recognition components, the entire
system is jointly fine-tuned using a multi-task criterion interpolation of the
scale-invariant signal to noise ratio (Si-SNR) with either the connectionist
temporal classification (CTC), or lattice-free maximum mutual information
(LF-MMI) loss function. Experiments suggest that: the proposed audio-visual
multi-channel recognition system outperforms the baseline audio-only
multi-channel ASR system by up to 8.04% (31.68% relative) and 22.86% (58.51%
relative) absolute WER reduction on overlapped speech constructed using either
simulation or replaying of the LRS2 dataset respectively. Consistent
performance improvements are also obtained using the proposed audio-visual
multi-channel recognition system when using occluded video input with the face
region randomly covered up to 60%.
