  Multimodal speech recognition aims to improve the performance of automatic
speech recognition (ASR) systems by leveraging additional visual information
that is usually associated to the audio input. While previous approaches make
crucial use of strong visual representations, e.g. by finetuning pretrained
image recognition networks, significantly less attention has been paid to its
counterpart: the speech component. In this work, we investigate ways of
improving the base speech recognition system by following similar techniques to
the ones used for the visual encoder, namely, transferring representations and
data augmentation. First, we show that starting from a pretrained ASR
significantly improves the state-of-the-art performance; remarkably, even when
building upon a strong unimodal system, we still find gains by including the
visual modality. Second, we employ speech data augmentation techniques to
encourage the multimodal system to attend to the visual stimuli. This technique
replaces previously used word masking and comes with the benefits of being
conceptually simpler and yielding consistent improvements in the multimodal
setting. We provide empirical results on three multimodal datasets, including
the newly introduced Localized Narratives.
