  We study video-grounded dialogue generation, where a response is generated
based on the dialogue context and the associated video. The primary challenges
of this task lie in (1) the difficulty of integrating video data into
pre-trained language models (PLMs) which presents obstacles to exploiting the
power of large-scale pre-training; and (2) the necessity of taking into account
the complementarity of various modalities throughout the reasoning process.
Although having made remarkable progress in video-grounded dialogue generation,
existing methods still fall short when it comes to integrating with PLMs in a
way that allows information from different modalities to complement each other.
To alleviate these issues, we first propose extracting pertinent information
from videos and turning it into reasoning paths that are acceptable to PLMs.
Additionally, we propose a multi-agent reinforcement learning method to
collaboratively perform reasoning on different modalities (i.e., video and
dialogue context). Empirical experiment results on two public datasets indicate
that the proposed model can significantly outperform state-of-the-art models by
large margins on both automatic and human evaluations.
