  Visual Dialog is a challenging vision-language task since the visual dialog
agent needs to answer a series of questions after reasoning over both the image
content and dialog history. Though existing methods try to deal with the
cross-modal understanding in visual dialog, they are still not enough in
ranking candidate answers based on their understanding of visual and textual
contexts. In this paper, we analyze the cross-modal understanding in visual
dialog based on the vision-language pre-training model VD-BERT and propose a
novel approach to improve the cross-modal understanding for visual dialog,
named ICMU. ICMU enhances cross-modal understanding by distinguishing different
pulled inputs (i.e. pulled images, questions or answers) based on four-way
contrastive learning. In addition, ICMU exploits the single-turn visual
question answering to enhance the visual dialog model's cross-modal
understanding to handle a multi-turn visually-grounded conversation.
Experiments show that the proposed approach improves the visual dialog model's
cross-modal understanding and brings satisfactory gain to the VisDial dataset.
