  Visual Dialog is a multimodal task of answering a sequence of questions
grounded in an image, using the conversation history as context. It entails
challenges in vision, language, reasoning, and grounding. However, studying
these subtasks in isolation on large, real datasets is infeasible as it
requires prohibitively-expensive complete annotation of the 'state' of all
images and dialogs.
  We develop CLEVR-Dialog, a large diagnostic dataset for studying multi-round
reasoning in visual dialog. Specifically, we construct a dialog grammar that is
grounded in the scene graphs of the images from the CLEVR dataset. This
combination results in a dataset where all aspects of the visual dialog are
fully annotated. In total, CLEVR-Dialog contains 5 instances of 10-round
dialogs for about 85k CLEVR images, totaling to 4.25M question-answer pairs.
  We use CLEVR-Dialog to benchmark performance of standard visual dialog
models; in particular, on visual coreference resolution (as a function of the
coreference distance). This is the first analysis of its kind for visual dialog
models that was not possible without this dataset. We hope the findings from
CLEVR-Dialog will help inform the development of future models for visual
dialog. Our dataset and code are publicly available.
