  Stories are essential for genealogy research since they can help build
emotional connections with people. A lot of family stories are reserved in
historical photos and albums. Recent development on image captioning models
makes it feasible to "tell stories" for photos automatically. The attention
mechanism has been widely adopted in many state-of-the-art encoder-decoder
based image captioning models, since it can bridge the gap between the visual
part and the language part. Most existing captioning models implicitly trained
attention modules with word-likelihood loss. Meanwhile, lots of studies have
investigated intrinsic attentions for visual models using gradient-based
approaches. Ideally, attention maps predicted by captioning models should be
consistent with intrinsic attentions from visual models for any given visual
concept. However, no work has been done to align implicitly learned attention
maps with intrinsic visual attentions. In this paper, we proposed a novel model
that measured consistency between captioning predicted attentions and intrinsic
visual attentions. This alignment loss allows explicit attention correction
without using any expensive bounding box annotations. We developed and
evaluated our model on COCO dataset as well as a genealogical dataset from
Ancestry.com Operations Inc., which contains billions of historical photos. The
proposed model achieved better performances on all commonly used language
evaluation metrics for both datasets.
