  Research in massively multilingual image captioning has been severely
hampered by a lack of high-quality evaluation datasets. In this paper we
present the Crossmodal-3600 dataset (XM3600 in short), a geographically diverse
set of 3600 images annotated with human-generated reference captions in 36
languages. The images were selected from across the world, covering regions
where the 36 languages are spoken, and annotated with captions that achieve
consistency in terms of style across all languages, while avoiding annotation
artifacts due to direct translation. We apply this benchmark to model selection
for massively multilingual image captioning models, and show superior
correlation results with human evaluations when using XM3600 as golden
references for automatic metrics.
