  Multimodal automatic speech recognition systems integrate information from
images to improve speech recognition quality, by grounding the speech in the
visual context. While visual signals have been shown to be useful for
recovering entities that have been masked in the audio, these models should be
capable of recovering a broader range of word types. Existing systems rely on
global visual features that represent the entire image, but localizing the
relevant regions of the image will make it possible to recover a larger set of
words, such as adjectives and verbs. In this paper, we propose a model that
uses finer-grained visual information from different parts of the image, using
automatic object proposals. In experiments on the Flickr8K Audio Captions
Corpus, we find that our model improves over approaches that use global visual
features, that the proposals enable the model to recover entities and other
related words, such as adjectives, and that improvements are due to the model's
ability to localize the correct proposals.
