  Image captioning has been recently gaining a lot of attention thanks to the
impressive achievements shown by deep captioning architectures, which combine
Convolutional Neural Networks to extract image representations, and Recurrent
Neural Networks to generate the corresponding captions. At the same time, a
significant research effort has been dedicated to the development of saliency
prediction models, which can predict human eye fixations. Even though saliency
information could be useful to condition an image captioning architecture, by
providing an indication of what is salient and what is not, research is still
struggling to incorporate these two techniques. In this work, we propose an
image captioning approach in which a generative recurrent neural network can
focus on different parts of the input image during the generation of the
caption, by exploiting the conditioning given by a saliency prediction model on
which parts of the image are salient and which are contextual. We show, through
extensive quantitative and qualitative experiments on large scale datasets,
that our model achieves superior performances with respect to captioning
baselines with and without saliency, and to different state of the art
approaches combining saliency and captioning.
