  Video-and-Language Inference is a recently proposed task for joint
video-and-language understanding. This new task requires a model to draw
inference on whether a natural language statement entails or contradicts a
given video clip. In this paper, we study how to address three critical
challenges for this task: judging the global correctness of the statement
involved multiple semantic meanings, joint reasoning over video and subtitles,
and modeling long-range relationships and complex social interactions. First,
we propose an adaptive hierarchical graph network that achieves in-depth
understanding of the video over complex interactions. Specifically, it performs
joint reasoning over video and subtitles in three hierarchies, where the graph
structure is adaptively adjusted according to the semantic structures of the
statement. Secondly, we introduce semantic coherence learning to explicitly
encourage the semantic coherence of the adaptive hierarchical graph network
from three hierarchies. The semantic coherence learning can further improve the
alignment between vision and linguistics, and the coherence across a sequence
of video segments. Experimental results show that our method significantly
outperforms the baseline by a large margin.
