  Multimodal learning is a recent challenge that extends unimodal learning by
generalizing its domain to diverse modalities, such as texts, images, or
speech. This extension requires models to process and relate information from
multiple modalities. In Information Retrieval, traditional retrieval tasks
focus on the similarity between unimodal documents and queries, while
image-text retrieval hypothesizes that most texts contain the scene context
from images. This separation has ignored that real-world queries may involve
text content, image captions, or both. To address this, we introduce Multimodal
Retrieval on Representation of ImaGe witH Text (Mr. Right), a novel and
comprehensive dataset for multimodal retrieval. We utilize the Wikipedia
dataset with rich text-image examples and generate three types of text-based
queries with different modality information: text-related, image-related, and
mixed. To validate the effectiveness of our dataset, we provide a multimodal
training paradigm and evaluate previous text retrieval and image retrieval
frameworks. The results show that proposed multimodal retrieval can improve
retrieval performance, but creating a well-unified document representation with
texts and images is still a challenge. We hope Mr. Right allows us to broaden
current retrieval systems better and contributes to accelerating the
advancement of multimodal learning in the Information Retrieval.
