  In this study, we try to address the problem of leveraging visual signals to
improve Automatic Speech Recognition (ASR), also known as visual context-aware
ASR (VC-ASR). We explore novel VC-ASR approaches to leverage video and text
representations extracted by a self-supervised pre-trained text-video embedding
model. Firstly, we propose a multi-stream attention architecture to leverage
signals from both audio and video modalities. This architecture consists of
separate encoders for the two modalities and a single decoder that attends over
them. We show that this architecture is better than fusing modalities at the
signal level. Additionally, we also explore leveraging the visual information
in a second pass model, which has also been referred to as a `deliberation
model'. The deliberation model accepts audio representations and text
hypotheses from the first pass ASR and combines them with a visual stream for
an improved visual context-aware recognition. The proposed deliberation scheme
can work on top of any well trained ASR and also enabled us to leverage the
pre-trained text model to ground the hypotheses with the visual features. Our
experiments on HOW2 dataset show that multi-stream and deliberation
architectures are very effective at the VC-ASR task. We evaluate the proposed
models for two scenarios; clean audio stream and distorted audio in which we
mask out some specific words in the audio. The deliberation model outperforms
the multi-stream model and achieves a relative WER improvement of 6% and 8.7%
for the clean and masked data, respectively, compared to an audio-only model.
The deliberation model also improves recovering the masked words by 59%
relative.
