  Currently, dialogue systems have achieved high performance in processing
text-based communication. However, they have not yet effectively incorporated
visual information, which poses a significant challenge. Furthermore, existing
models that incorporate images in dialogue generation focus on discussing the
image itself. Our proposed approach presents a novel perspective on multi-modal
dialogue systems, which interprets the image in the context of the dialogue. By
doing so, we aim to expand the capabilities of current dialogue systems and
transition them from single modality (text) to multi-modality. However, there
is a lack of validated English datasets that contain both images and dialogue
contexts for this task. Thus, we propose a two-stage approach to automatically
construct a multi-modal dialogue dataset. In the first stage, we utilize
text-to-image similarity and sentence similarity to identify which utterances
could be replaced with an image. In the second stage, we replace those
utterances by selecting a subset of relevant images and filtering them with a
visual question answering model. We used this approach, along with additional
labeling, to create the IMage Augmented multi-modal Dialogue dataset (IMAD),
which can serve as a validated dataset for this task. Furthermore, we propose a
baseline model trained on this dataset, which outperforms model trained on the
same data without images and BlenderBot.
