  We introduce a new multi-modal task for computer systems, posed as a combined
vision-language comprehension challenge: identifying the most suitable text
describing a scene, given several similar options. Accomplishing the task
entails demonstrating comprehension beyond just recognizing "keywords" (or
key-phrases) and their corresponding visual concepts. Instead, it requires an
alignment between the representations of the two modalities that achieves a
visually-grounded "understanding" of various linguistic elements and their
dependencies. This new task also admits an easy-to-compute and well-studied
metric: the accuracy in detecting the true target among the decoys.
  The paper makes several contributions: an effective and extensible mechanism
for generating decoys from (human-created) image captions; an instance of
applying this mechanism, yielding a large-scale machine comprehension dataset
(based on the COCO images and captions) that we make publicly available; human
evaluation results on this dataset, informing a performance upper-bound; and
several baseline and competitive learning approaches that illustrate the
utility of the proposed task and dataset in advancing both image and language
comprehension. We also show that, in a multi-task learning setting, the
performance on the proposed task is positively correlated with the end-to-end
task of image captioning.
