  Understanding video content and generating caption with context is an
important and challenging task. Unlike prior methods that typically attempt to
generate generic video captions without context, our architecture
contextualizes captioning by infusing extracted information from relevant text
data. We propose an end-to-end sequence-to-sequence model which generates video
captions based on visual input, and mines relevant knowledge such as names and
locations from contextual text. In contrast to previous approaches, we do not
preprocess the text further, and let the model directly learn to attend over
it. Guided by the visual input, the model is able to copy words from the
contextual text via a pointer-generator network, allowing to produce more
specific video captions. We show competitive performance on the News Video
Dataset and, through ablation studies, validate the efficacy of contextual
video captioning as well as individual design choices in our model
architecture.
