  Retrieval is a widely adopted approach for improving language models
leveraging external information. As the field moves towards multi-modal large
language models, it is important to extend the pure text based methods to
incorporate other modalities in retrieval as well for applications across the
wide spectrum of machine learning tasks and data types. In this work, we
propose multi-modal retrieval with two approaches: kNN-LM and cross-attention
techniques. We demonstrate the effectiveness of our retrieval approaches
empirically by applying them to automatic speech recognition tasks with access
to external information. Under this setting, we show that speech-based
multi-modal retrieval outperforms text based retrieval, and yields up to 50 %
improvement in word error rate over the multi-modal language model baseline.
Furthermore, we achieve state-of-the-art recognition results on the
Spoken-Squad question answering dataset.
