  Image captioning as a multimodal task has drawn much interest in recent
years. However, evaluation for this task remains a challenging problem.
Existing evaluation metrics focus on surface similarity between a candidate
caption and a set of reference captions, and do not check the actual relation
between a caption and the underlying visual content. We introduce a new
diagnostic evaluation framework for the task of image captioning, with the goal
of directly assessing models for grammaticality, truthfulness and diversity
(GTD) of generated captions. We demonstrate the potential of our evaluation
framework by evaluating existing image captioning models on a wide ranging set
of synthetic datasets that we construct for diagnostic evaluation. We
empirically show how the GTD evaluation framework, in combination with
diagnostic datasets, can provide insights into model capabilities and
limitations to supplement standard evaluations.
