  Multimodal machine translation (MMT) systems have been shown to outperform
their text-only neural machine translation (NMT) counterparts when visual
context is available. However, recent studies have also shown that the
performance of MMT models is only marginally impacted when the associated image
is replaced with an unrelated image or noise, which suggests that the visual
context might not be exploited by the model at all. We hypothesize that this
might be caused by the nature of the commonly used evaluation benchmark, also
known as Multi30K, where the translations of image captions were prepared
without actually showing the images to human translators. In this paper, we
present a qualitative study that examines the role of datasets in stimulating
the leverage of visual modality and we propose methods to highlight the
importance of visual signals in the datasets which demonstrate improvements in
reliance of models on the source images. Our findings suggest the research on
effective MMT architectures is currently impaired by the lack of suitable
datasets and careful consideration must be taken in creation of future MMT
datasets, for which we also provide useful insights.
