  Video description is one of the most challenging problems in vision and
language understanding due to the large variability both on the video and
language side. Models, hence, typically shortcut the difficulty in recognition
and generate plausible sentences that are based on priors but are not
necessarily grounded in the video. In this work, we explicitly link the
sentence to the evidence in the video by annotating each noun phrase in a
sentence with the corresponding bounding box in one of the frames of a video.
Our dataset, ActivityNet-Entities, augments the challenging ActivityNet
Captions dataset with 158k bounding box annotations, each grounding a noun
phrase. This allows training video description models with this data, and
importantly, evaluate how grounded or "true" such model are to the video they
describe. To generate grounded captions, we propose a novel video description
model which is able to exploit these bounding box annotations. We demonstrate
the effectiveness of our model on our dataset, but also show how it can be
applied to image description on the Flickr30k Entities dataset. We achieve
state-of-the-art performance on video description, video paragraph description,
and image description and demonstrate our generated sentences are better
grounded in the video.
