  Learning how to generate descriptions of images or videos received major
interest both in the Computer Vision and Natural Language Processing
communities. While a few works have proposed to learn a grounding during the
generation process in an unsupervised way (via an attention mechanism), it
remains unclear how good the quality of the grounding is and whether it
benefits the description quality. In this work we propose a movie description
model which learns to generate description and jointly ground (localize) the
mentioned characters as well as do visual co-reference resolution between pairs
of consecutive sentences/clips. We also propose to use weak localization
supervision through character mentions provided in movie descriptions to learn
the character grounding. At training time, we first learn how to localize
characters by relating their visual appearance to mentions in the descriptions
via a semi-supervised approach. We then provide this (noisy) supervision into
our description model which greatly improves its performance. Our proposed
description model improves over prior work w.r.t. generated description quality
and additionally provides grounding and local co-reference resolution. We
evaluate it on the MPII Movie Description dataset using automatic and human
evaluation measures and using our newly collected grounding and co-reference
data for characters.
