  Recent neural models for image captioning usually employ an encoder-decoder
framework with an attention mechanism. However, the attention mechanism in such
a framework aligns one single (attended) image feature vector to one caption
word, assuming one-to-one mapping from source image regions and target caption
words, which is never possible. In this paper, we propose a novel attention
model, namely Adaptive Attention Time (AAT), to align the source and the target
adaptively for image captioning. AAT allows the framework to learn how many
attention steps to take to output a caption word at each decoding step. With
AAT, an image region can be mapped to an arbitrary number of caption words
while a caption word can also attend to an arbitrary number of image regions.
AAT is deterministic and differentiable, and doesn't introduce any noise to the
parameter gradients. In this paper, we empirically show that AAT improves over
state-of-the-art methods on the task of image captioning. Code is available at
https://github.com/husthuaan/AAT.
