  Video paragraph captioning is the task of automatically generating a coherent
paragraph description of the actions in a video. Previous linguistic studies
have demonstrated that coherence of a natural language text is reflected by its
discourse structure and relations. However, existing video captioning methods
evaluate the coherence of generated paragraphs by comparing them merely against
human paragraph annotations and fail to reason about the underlying discourse
structure. At UCLA, we are currently exploring a novel discourse based
framework to evaluate the coherence of video paragraphs. Central to our
approach is the discourse representation of videos, which helps in modeling
coherence of paragraphs conditioned on coherence of videos. We also introduce
DisNet, a novel dataset containing the proposed visual discourse annotations of
3000 videos and their paragraphs. Our experiment results have shown that the
proposed framework evaluates coherence of video paragraphs significantly better
than all the baseline methods. We believe that many other multi-discipline
Artificial Intelligence problems such as Visual Dialog and Visual Storytelling
would also greatly benefit from the proposed visual discourse framework and the
DisNet dataset.
