  Visual dialog is challenging since it needs to answer a series of coherent
questions based on understanding the visual environment. How to ground related
visual objects is one of the key problems. Previous studies utilize the
question and history to attend to the image and achieve satisfactory
performance, however these methods are not sufficient to locate related visual
objects without any guidance. The inappropriate grounding of visual objects
prohibits the performance of visual dialog models. In this paper, we propose a
novel approach to Learn to Ground visual objects for visual dialog, which
employs a novel visual objects grounding mechanism where both prior and
posterior distributions over visual objects are used to facilitate visual
objects grounding. Specifically, a posterior distribution over visual objects
is inferred from both context (history and questions) and answers, and it
ensures the appropriate grounding of visual objects during the training
process. Meanwhile, a prior distribution, which is inferred from context only,
is used to approximate the posterior distribution so that appropriate visual
objects can be grounded even without answers during the inference process.
Experimental results on the VisDial v0.9 and v1.0 datasets demonstrate that our
approach improves the previous strong models in both generative and
discriminative settings by a significant margin.
